Hi Thanuja,
+1, although I'm again concerned with the structural complexity. However
when weighed against other options available (having the YAML files in the
same folder, enabling deep merge [1], etc) this is the best option as I
see.
`kubernetes/default.yaml` for WSO2 AM 1.10.0 would look something like the
following. This would get duplicated for each product version which is a
bit complex than needed and is unnecessary duplication IMO.
[image: Inline image 1]
[1] - [Dev] [Hiera] Deeper hash merge instead of Native hash merge?
Regards,
Chamila de Alwis
Committer and PMC Member - Apache Stratos
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
On Mon, Apr 11, 2016 at 4:15 PM, Thanuja Uruththirakodeeswaran <
thanu...@wso2.com> wrote:
>
>
> On Mon, Apr 11, 2016 at 2:06 PM, Chamila De Alwis <chami...@wso2.com>
> wrote:
>
>> Hi Pubudu,
>>
>> That's a good idea, however, we can't exactly map `environment` with the
>> `platform` concept. How about using a new level on top of the profile
>> specific level mapping to `platform`?
>>
>> :hierarchy:
>> - "node/%{::clientcert}"
>> * -
>> "wso2/%{::product_name}/%{::product_version}/%{::product_profile}-%{::platform}"*
>> - "wso2/%{::product_name}/%{::product_version}/%{::product_profile}"
>> - "wso2/%{::product_name}/%{::product_version}/default"
>> - "osfamily/%{::osfamily}"
>> - "vm_type/%{::vm_type}"
>> - wso2/common
>> - common
>>
>
> +1 for introducing a new level in hierarchy to lookup for platform
> specific data on top of profile specific data per product.
>
>>
>> This will allow us to extract the platform specific data only to one
>> layer up. An example would be,
>>
>> puppet-modules/hieradata/dev/wso2/wso2am/1.10.0/default-kubernetes.yaml
>>
>
> It would be better to have spearate folders for platform specific yaml
> files inside each product versions like below with changing the above hiera
> level to: *-
> "wso2/%{::product_name}/%{::product_version}/%{::product_profile}/%{::platform}"*
>
> hieradata/dev/wso2/wso2as/
>
> └── 5.3.0
>
> ├── default.yaml
>
> ├── <*platform*>
>
> │ ├── default.yaml
>
> │ ├── manager.yaml
>
> │ └── worker.yaml
>
> ├── kubernetes
>
> │ ├── default.yaml
>
> │ ├── manager.yaml
>
> │ └── worker.yaml
>
> ├── manager.yaml
>
> └── worker.yaml
>
>
> With the above structure, configs common to all platform will be there in
> the products' root level hierafiles. Only the platform specific configs
> related to a particular profile will be there in <platform>/<profile>.yaml
> (ex: proxy_ports).
>
>
>
>> WDYT?
>>
>>
>> Regards,
>> Chamila de Alwis
>> Committer and PMC Member - Apache Stratos
>> Software Engineer | WSO2 | +94772207163
>> Blog: code.chamiladealwis.com
>>
>>
>>
>> On Mon, Apr 11, 2016 at 1:36 PM, Pubudu Gunatilaka <pubu...@wso2.com>
>> wrote:
>>
>>> Hi Isuru,
>>>
>>> Without duplicating files and adding new files, I would rather prefer to
>>> use the following hierarchy. For each and every different environment we
>>> can have product profiles and include the environment specific values. For
>>> Kubernetes, we can include the clustering section only in manager.yaml file
>>> which resides under environment Kubernetes. And the rest of the other
>>> configurations can be included in the generic file folder, i.e "
>>> wso2/%{::product_name}/%{::product_version}/%{::product_profile}".
>>>
>>>
>>> :hierarchy:
>>> - "node/%{::clientcert}" - - "
>>> wso2/%{::product_name}/%{::product_version}/%{::environment}
>>> /%{::product_profile}"
>>> - "wso2/%{::product_name}/%{::product_version}/%{::product_profile}"
>>> - "wso2/%{::product_name}/%{::product_version}/default"
>>> - "osfamily/%{::osfamily}"
>>> - "vm_type/%{::vm_type}"
>>> - wso2/common
>>> - common
>>>
>>>
>>> Thank you!
>>>
>>> On Mon, Apr 11, 2016 at 1:07 PM, Isuru Haththotuwa <isu...@wso2.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Apr 11, 2016 at 12:47 PM, Gayan Gunarathne <gay...@wso2.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 11, 2016 at 12:39 PM, Lakmal Warusawithana <
>>>>> lak...@wso2.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 11, 2016 at 12:22 PM, Imesh Gunaratne <im...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Chamila,
>>>>>>>
>>>>>>> On Mon, Apr 11, 2016 at 7:19 AM, Chamila De Alwis <chami...@wso2.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi Isuru, Imesh,
>>>>>>>>
>>>>>>>> IMO we shouldn't have any platform specific restructuring in
>>>>>>>> wso2/puppet-modules. This should be done at the end user's setup.
>>>>>>>> Another
>>>>>>>> point is that we have now decoupled wso2/puppet-modules from
>>>>>>>> wso2/dockerfiles and users are not required to incorporate
>>>>>>>> puppet-modules
>>>>>>>> in their container setup.
>>>>>>>>
>>>>>>>
>>>>>>> A very good concern! The way I see this is little different. Let's
>>>>>>> evaluate the options we have:
>>>>>>>
>>>>>>> 1. Ship generic product/profile Hiera YAML files and let the
>>>>>>> users configure them according to their platform (VM, AWS, Azure,
>>>>>>> OpenStack, K8S, Mesos, OpenShift, CF, etc)
>>>>>>> 2. Ship product/profile/platform Hiera YAML files and let users
>>>>>>> use them OOB with very few changes.
>>>>>>>
>>>>>>> +1 for 2nd option. Yes, it has some duplication on configurations,
>>>>>> but customer POV, it is very easy to look at single place to do the
>>>>>> minimum
>>>>>> changes. (rather looking for many files in deferent folders to do the
>>>>>> changes)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Which one would be the best option? IMO 2nd option would provide a
>>>>>>> much better user experience compared to 1 as it provides platform
>>>>>>> specific
>>>>>>> values such as clustering configuration & port mappings OOB. User will
>>>>>>> only
>>>>>>> need to provide values such as database hosts, passwords, identity
>>>>>>> management, etc which are user specific.
>>>>>>>
>>>>>>> The whole idea of this effort is to provide a better user experience.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>>
>>>>>>>> IMO Docker images will not be able to run OOB on Kubernetes using
>>>>>>>> wso2/puppet-modules and wso2/kubernetes-artifacts. There will anyway be
>>>>>>>> changes related to the Kubernetes Membership Scheme in
>>>>>>>> wso2/puppet-modules
>>>>>>>> and in wso2/kubernetes-artifacts where environment dependent changes
>>>>>>>> such
>>>>>>>> as image names, SecureVault passwords, etc. need to be adjusted.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Chamila de Alwis
>>>>>>>> Committer and PMC Member - Apache Stratos
>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 11, 2016 at 1:36 AM, Imesh Gunaratne <im...@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Gayan,
>>>>>>>>>
>>>>>>>>> On Sun, Apr 10, 2016 at 5:02 PM, Gayan Gunarathne <gay...@wso2.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> IMO this will create maintainability issue. We need to maintain
>>>>>>>>>> all the separate hieradata structure for each scenarios.For the one
>>>>>>>>>> particular alternation we need to change whole set of files.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In this scenario user experience is much more important than the
>>>>>>>>> maintainability of few yaml files. If we do not do this, users will
>>>>>>>>> not be
>>>>>>>>> able to use puppet modules OOB until they manually update
>>>>>>>>> configuration
>>>>>>>>> values in above files. The whole idea of this effort is to let users
>>>>>>>>> do
>>>>>>>>> following:
>>>>>>>>>
>>>>>>>>> - Setup a K8S cluster
>>>>>>>>> - Download puppet modules zip file(s).
>>>>>>>>> - Download docker files
>>>>>>>>> - Build docker images using puppet for different product
>>>>>>>>> profiles
>>>>>>>>> - Deploy WSO2 product on K8S using K8S artifacts
>>>>>>>>>
>>>>>>>>> The above process will allow users to deploy any WSO2 product
>>>>>>>>> (with mutlitple deployment patterns) on K8S with zero configurations.
>>>>>>>>> This
>>>>>>>>> will be true for any VM based platform or any other container cluster
>>>>>>>>> management system.
>>>>>>>>>
>>>>>>>>
>>>>> Mainly the target users group of the puppet/hiera files will be system
>>>>> administrators/Dev Ops. So those guys will be consider the fact the
>>>>> maintainability of puppet/hiera files. So if this is a maintainability
>>>>> issue, it will become bad experience for the end user in end of the day.
>>>>>
>>>>>
>>>>>>
>>>>>>>>>> Why can't we do this by using defined types in Hiera and lookup
>>>>>>>>>> parameters for a given instance? Based on the identify keys we
>>>>>>>>>> set for each vm, docker, K8S etc we can select the appropriate data
>>>>>>>>>> set from Hiera file.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Will you be able to provide a sample?
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>> I think we can make this with with Defined Types[1][2] without
>>>>> creating duplicate set of YAML files for each platform. We can do the same
>>>>> as the example given in the document.
>>>>>
>>>> I do not think we need to duplicate the yaml files here. Sorry that the
>>>> sample in the first reply sent by me implied so. What we can do is refactor
>>>> out the hiera data so that data which is changing according to the platform
>>>> (ex.: clustering section) can be moved to a different yaml file(s). For an
>>>> example, clustering for kubernetes scenario can be included in
>>>> *puppet-modules/hieradata/dev/wso2/kubernetes/clustering.yaml*. Here,
>>>> 'kubernetes' is the value of the facter which is used to identify the
>>>> environment. Similarly, other such data can be refactored out.
>>>>
>>>
> If we follow this approach, then we can't specify product and profile
> specific data (ex: kubernetes service name varies per product and
> proxy_ports varies per poduct/profile) in the common platform hiera file:
> *puppet-modules/hieradata/dev/wso2/kubernetes/clustering.yaml.*
>
>
>>>>> [1]
>>>>> https://docs.puppet.com/puppet/latest/reference/lang_defined_types.html
>>>>> [2]http://puppetlunch.com/puppet/hiera.html
>>>>>
>>>>>
>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Gayan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Apr 9, 2016 at 8:28 AM, Imesh Gunaratne <im...@wso2.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 8, 2016 at 7:48 PM, Isuru Haththotuwa <
>>>>>>>>>>> isu...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> hieradata
>>>>>>>>>>>> |--- dev
>>>>>>>>>>>> |--- wso2
>>>>>>>>>>>> |---- <product_name>
>>>>>>>>>>>> |--- <product_version>
>>>>>>>>>>>> |-- *vm*
>>>>>>>>>>>> |--
>>>>>>>>>>>> default.yaml
>>>>>>>>>>>> |--
>>>>>>>>>>>> manager.yaml
>>>>>>>>>>>> |--
>>>>>>>>>>>> worker.yaml
>>>>>>>>>>>> |--*
>>>>>>>>>>>> docker*
>>>>>>>>>>>> |--
>>>>>>>>>>>> default.yaml
>>>>>>>>>>>> |--
>>>>>>>>>>>> manager.yaml
>>>>>>>>>>>> |--
>>>>>>>>>>>> worker.yaml
>>>>>>>>>>>> |--
>>>>>>>>>>>> *kubernetes*
>>>>>>>>>>>> |--
>>>>>>>>>>>> default.yaml
>>>>>>>>>>>> |--
>>>>>>>>>>>> manager.yaml
>>>>>>>>>>>> |--
>>>>>>>>>>>> worker.yaml
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> +1 for the suggestion Isuru, will proceed with this. We can add
>>>>>>>>>>> other platforms such as OpenShift, Mesos, Cloud Foundry on the same
>>>>>>>>>>> level.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *Imesh Gunaratne*
>>>>>>>>>>>>> Senior Technical Lead
>>>>>>>>>>>>> WSO2 Inc: http://wso2.com
>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057
>>>>>>>>>>>>> W: http://imesh.io
>>>>>>>>>>>>> Lean . Enterprise . Middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Isuru H.
>>>>>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Imesh Gunaratne*
>>>>>>>>>>> Senior Technical Lead
>>>>>>>>>>> WSO2 Inc: http://wso2.com
>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057
>>>>>>>>>>> W: http://imesh.io
>>>>>>>>>>> Lean . Enterprise . Middleware
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Dev mailing list
>>>>>>>>>>> Dev@wso2.org
>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>> email : gay...@wso2.com | mobile : +94 775030545
>>>>>>>>>> <%2B94%20766819985>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Dev mailing list
>>>>>>>>>> Dev@wso2.org
>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Imesh Gunaratne*
>>>>>>>>> Senior Technical Lead
>>>>>>>>> WSO2 Inc: http://wso2.com
>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057
>>>>>>>>> W: http://imesh.io
>>>>>>>>> Lean . Enterprise . Middleware
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Dev mailing list
>>>>>>>>> Dev@wso2.org
>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Imesh Gunaratne*
>>>>>>> Senior Technical Lead
>>>>>>> WSO2 Inc: http://wso2.com
>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057
>>>>>>> W: http://imesh.io
>>>>>>> Lean . Enterprise . Middleware
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> Dev@wso2.org
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lakmal Warusawithana
>>>>>> Director - Cloud Architecture; WSO2 Inc.
>>>>>> Mobile : +94714289692
>>>>>> Blog : http://lakmalsview.blogspot.com/
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list
>>>>>> Dev@wso2.org
>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Gayan Gunarathne
>>>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>>>> Committer & PMC Member, Apache Stratos
>>>>> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list
>>>>> Dev@wso2.org
>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> +94 716 358 048* <http://wso2.com/>*
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Dev mailing list
>>>> Dev@wso2.org
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>>
>>> --
>>> *Pubudu Gunatilaka*
>>> Committer and PMC Member - Apache Stratos
>>> Software Engineer
>>> WSO2, Inc.: http://wso2.com
>>> mobile : +94774078049 <%2B94772207163>
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>> _______________________________________________
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> Thanuja Uruththirakodeeswaran
> Software Engineer
> WSO2 Inc.;http://wso2.com
> lean.enterprise.middleware
>
> mobile: +94 774363167
>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
