+1 IMO it is better to proceed with this and later improve it as needed. One more point regarding the file list, I think we can copy all the files found under repository/components/lib without specifically defining them in the Hiera file. WDYT?
Thanks On Tue, Apr 12, 2016 at 9:56 AM, Chamila De Alwis <[email protected]> wrote: > Hi Thanuja, > > +1, although I'm again concerned with the structural complexity. However > when weighed against other options available (having the YAML files in the > same folder, enabling deep merge [1], etc) this is the best option as I > see. > > `kubernetes/default.yaml` for WSO2 AM 1.10.0 would look something like the > following. This would get duplicated for each product version which is a > bit complex than needed and is unnecessary duplication IMO. > > [image: Inline image 1] > > [1] - [Dev] [Hiera] Deeper hash merge instead of Native hash merge? > > > Regards, > Chamila de Alwis > Committer and PMC Member - Apache Stratos > Software Engineer | WSO2 | +94772207163 > Blog: code.chamiladealwis.com > > > > On Mon, Apr 11, 2016 at 4:15 PM, Thanuja Uruththirakodeeswaran < > [email protected]> wrote: > >> >> >> On Mon, Apr 11, 2016 at 2:06 PM, Chamila De Alwis <[email protected]> >> wrote: >> >>> Hi Pubudu, >>> >>> That's a good idea, however, we can't exactly map `environment` with the >>> `platform` concept. How about using a new level on top of the profile >>> specific level mapping to `platform`? >>> >>> :hierarchy: >>> - "node/%{::clientcert}" >>> * - >>> "wso2/%{::product_name}/%{::product_version}/%{::product_profile}-%{::platform}"* >>> - "wso2/%{::product_name}/%{::product_version}/%{::product_profile}" >>> - "wso2/%{::product_name}/%{::product_version}/default" >>> - "osfamily/%{::osfamily}" >>> - "vm_type/%{::vm_type}" >>> - wso2/common >>> - common >>> >> >> +1 for introducing a new level in hierarchy to lookup for platform >> specific data on top of profile specific data per product. >> >>> >>> This will allow us to extract the platform specific data only to one >>> layer up. An example would be, >>> >>> puppet-modules/hieradata/dev/wso2/wso2am/1.10.0/default-kubernetes.yaml >>> >> >> It would be better to have spearate folders for platform specific yaml >> files inside each product versions like below with changing the above hiera >> level to: *- >> "wso2/%{::product_name}/%{::product_version}/%{::product_profile}/%{::platform}"* >> >> hieradata/dev/wso2/wso2as/ >> >> └── 5.3.0 >> >> ├── default.yaml >> >> ├── <*platform*> >> >> │ ├── default.yaml >> >> │ ├── manager.yaml >> >> │ └── worker.yaml >> >> ├── kubernetes >> >> │ ├── default.yaml >> >> │ ├── manager.yaml >> >> │ └── worker.yaml >> >> ├── manager.yaml >> >> └── worker.yaml >> >> >> With the above structure, configs common to all platform will be there in >> the products' root level hierafiles. Only the platform specific configs >> related to a particular profile will be there in <platform>/<profile>.yaml >> (ex: proxy_ports). >> >> >> >>> WDYT? >>> >>> >>> Regards, >>> Chamila de Alwis >>> Committer and PMC Member - Apache Stratos >>> Software Engineer | WSO2 | +94772207163 >>> Blog: code.chamiladealwis.com >>> >>> >>> >>> On Mon, Apr 11, 2016 at 1:36 PM, Pubudu Gunatilaka <[email protected]> >>> wrote: >>> >>>> Hi Isuru, >>>> >>>> Without duplicating files and adding new files, I would rather prefer >>>> to use the following hierarchy. For each and every different environment we >>>> can have product profiles and include the environment specific values. For >>>> Kubernetes, we can include the clustering section only in manager.yaml file >>>> which resides under environment Kubernetes. And the rest of the other >>>> configurations can be included in the generic file folder, i.e " >>>> wso2/%{::product_name}/%{::product_version}/%{::product_profile}". >>>> >>>> >>>> :hierarchy: >>>> - "node/%{::clientcert}" - - " >>>> wso2/%{::product_name}/%{::product_version}/%{::environment} >>>> /%{::product_profile}" >>>> - "wso2/%{::product_name}/%{::product_version}/%{::product_profile}" >>>> - "wso2/%{::product_name}/%{::product_version}/default" >>>> - "osfamily/%{::osfamily}" >>>> - "vm_type/%{::vm_type}" >>>> - wso2/common >>>> - common >>>> >>>> >>>> Thank you! >>>> >>>> On Mon, Apr 11, 2016 at 1:07 PM, Isuru Haththotuwa <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Apr 11, 2016 at 12:47 PM, Gayan Gunarathne <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Apr 11, 2016 at 12:39 PM, Lakmal Warusawithana < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Apr 11, 2016 at 12:22 PM, Imesh Gunaratne <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Chamila, >>>>>>>> >>>>>>>> On Mon, Apr 11, 2016 at 7:19 AM, Chamila De Alwis < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Isuru, Imesh, >>>>>>>>> >>>>>>>>> IMO we shouldn't have any platform specific restructuring in >>>>>>>>> wso2/puppet-modules. This should be done at the end user's setup. >>>>>>>>> Another >>>>>>>>> point is that we have now decoupled wso2/puppet-modules from >>>>>>>>> wso2/dockerfiles and users are not required to incorporate >>>>>>>>> puppet-modules >>>>>>>>> in their container setup. >>>>>>>>> >>>>>>>> >>>>>>>> A very good concern! The way I see this is little different. Let's >>>>>>>> evaluate the options we have: >>>>>>>> >>>>>>>> 1. Ship generic product/profile Hiera YAML files and let the >>>>>>>> users configure them according to their platform (VM, AWS, Azure, >>>>>>>> OpenStack, K8S, Mesos, OpenShift, CF, etc) >>>>>>>> 2. Ship product/profile/platform Hiera YAML files and let users >>>>>>>> use them OOB with very few changes. >>>>>>>> >>>>>>>> +1 for 2nd option. Yes, it has some duplication on configurations, >>>>>>> but customer POV, it is very easy to look at single place to do the >>>>>>> minimum >>>>>>> changes. (rather looking for many files in deferent folders to do the >>>>>>> changes) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Which one would be the best option? IMO 2nd option would provide a >>>>>>>> much better user experience compared to 1 as it provides platform >>>>>>>> specific >>>>>>>> values such as clustering configuration & port mappings OOB. User will >>>>>>>> only >>>>>>>> need to provide values such as database hosts, passwords, identity >>>>>>>> management, etc which are user specific. >>>>>>>> >>>>>>>> The whole idea of this effort is to provide a better user >>>>>>>> experience. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>>> >>>>>>>>> IMO Docker images will not be able to run OOB on Kubernetes using >>>>>>>>> wso2/puppet-modules and wso2/kubernetes-artifacts. There will anyway >>>>>>>>> be >>>>>>>>> changes related to the Kubernetes Membership Scheme in >>>>>>>>> wso2/puppet-modules >>>>>>>>> and in wso2/kubernetes-artifacts where environment dependent changes >>>>>>>>> such >>>>>>>>> as image names, SecureVault passwords, etc. need to be adjusted. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Chamila de Alwis >>>>>>>>> Committer and PMC Member - Apache Stratos >>>>>>>>> Software Engineer | WSO2 | +94772207163 >>>>>>>>> Blog: code.chamiladealwis.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Apr 11, 2016 at 1:36 AM, Imesh Gunaratne <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Gayan, >>>>>>>>>> >>>>>>>>>> On Sun, Apr 10, 2016 at 5:02 PM, Gayan Gunarathne < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> IMO this will create maintainability issue. We need to maintain >>>>>>>>>>> all the separate hieradata structure for each scenarios.For the one >>>>>>>>>>> particular alternation we need to change whole set of files. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> In this scenario user experience is much more important than the >>>>>>>>>> maintainability of few yaml files. If we do not do this, users will >>>>>>>>>> not be >>>>>>>>>> able to use puppet modules OOB until they manually update >>>>>>>>>> configuration >>>>>>>>>> values in above files. The whole idea of this effort is to let users >>>>>>>>>> do >>>>>>>>>> following: >>>>>>>>>> >>>>>>>>>> - Setup a K8S cluster >>>>>>>>>> - Download puppet modules zip file(s). >>>>>>>>>> - Download docker files >>>>>>>>>> - Build docker images using puppet for different product >>>>>>>>>> profiles >>>>>>>>>> - Deploy WSO2 product on K8S using K8S artifacts >>>>>>>>>> >>>>>>>>>> The above process will allow users to deploy any WSO2 product >>>>>>>>>> (with mutlitple deployment patterns) on K8S with zero >>>>>>>>>> configurations. This >>>>>>>>>> will be true for any VM based platform or any other container cluster >>>>>>>>>> management system. >>>>>>>>>> >>>>>>>>> >>>>>> Mainly the target users group of the puppet/hiera files will be >>>>>> system administrators/Dev Ops. So those guys will be consider the fact >>>>>> the >>>>>> maintainability of puppet/hiera files. So if this is a maintainability >>>>>> issue, it will become bad experience for the end user in end of the day. >>>>>> >>>>>> >>>>>>> >>>>>>>>>>> Why can't we do this by using defined types in Hiera and lookup >>>>>>>>>>> parameters for a given instance? Based on the identify keys we >>>>>>>>>>> set for each vm, docker, K8S etc we can select the appropriate data >>>>>>>>>>> set from Hiera file. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Will you be able to provide a sample? >>>>>>>>>> >>>>>>>>> >>>>>> >>>>>> I think we can make this with with Defined Types[1][2] without >>>>>> creating duplicate set of YAML files for each platform. We can do the >>>>>> same >>>>>> as the example given in the document. >>>>>> >>>>> I do not think we need to duplicate the yaml files here. Sorry that >>>>> the sample in the first reply sent by me implied so. What we can do is >>>>> refactor out the hiera data so that data which is changing according to >>>>> the >>>>> platform (ex.: clustering section) can be moved to a different yaml >>>>> file(s). For an example, clustering for kubernetes scenario can be >>>>> included >>>>> in *puppet-modules/hieradata/dev/wso2/kubernetes/clustering.yaml*. >>>>> Here, 'kubernetes' is the value of the facter which is used to identify >>>>> the >>>>> environment. Similarly, other such data can be refactored out. >>>>> >>>> >> If we follow this approach, then we can't specify product and profile >> specific data (ex: kubernetes service name varies per product and >> proxy_ports varies per poduct/profile) in the common platform hiera file: >> *puppet-modules/hieradata/dev/wso2/kubernetes/clustering.yaml.* >> >> >>>>>> [1] >>>>>> https://docs.puppet.com/puppet/latest/reference/lang_defined_types.html >>>>>> [2]http://puppetlunch.com/puppet/hiera.html >>>>>> >>>>>> >>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Gayan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Apr 9, 2016 at 8:28 AM, Imesh Gunaratne <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Apr 8, 2016 at 7:48 PM, Isuru Haththotuwa < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> hieradata >>>>>>>>>>>>> |--- dev >>>>>>>>>>>>> |--- wso2 >>>>>>>>>>>>> |---- <product_name> >>>>>>>>>>>>> |--- <product_version> >>>>>>>>>>>>> |-- >>>>>>>>>>>>> *vm* >>>>>>>>>>>>> >>>>>>>>>>>>> |-- default.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> |-- manager.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> |-- worker.yaml >>>>>>>>>>>>> |--* >>>>>>>>>>>>> docker* >>>>>>>>>>>>> >>>>>>>>>>>>> |-- default.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> |-- manager.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> |-- worker.yaml >>>>>>>>>>>>> |-- >>>>>>>>>>>>> *kubernetes* >>>>>>>>>>>>> >>>>>>>>>>>>> |-- default.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> |-- manager.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> |-- worker.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> +1 for the suggestion Isuru, will proceed with this. We can add >>>>>>>>>>>> other platforms such as OpenShift, Mesos, Cloud Foundry on the >>>>>>>>>>>> same level. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>> W: http://imesh.io >>>>>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Isuru H. >>>>>>>>>>>>> +94 716 358 048* <http://wso2.com/>* >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>> W: http://imesh.io >>>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Gayan Gunarathne >>>>>>>>>>> Technical Lead, WSO2 Inc. (http://wso2.com) >>>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>>> email : [email protected] | mobile : +94 775030545 >>>>>>>>>>> <%2B94%20766819985> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>> Senior Technical Lead >>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>> W: http://imesh.io >>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Imesh Gunaratne* >>>>>>>> Senior Technical Lead >>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>> W: http://imesh.io >>>>>>>> Lean . Enterprise . Middleware >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Lakmal Warusawithana >>>>>>> Director - Cloud Architecture; WSO2 Inc. >>>>>>> Mobile : +94714289692 >>>>>>> Blog : http://lakmalsview.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Gayan Gunarathne >>>>>> Technical Lead, WSO2 Inc. (http://wso2.com) >>>>>> Committer & PMC Member, Apache Stratos >>>>>> email : [email protected] | mobile : +94 775030545 <%2B94%20766819985> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> >>>>> Isuru H. >>>>> +94 716 358 048* <http://wso2.com/>* >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Pubudu Gunatilaka* >>>> Committer and PMC Member - Apache Stratos >>>> Software Engineer >>>> WSO2, Inc.: http://wso2.com >>>> mobile : +94774078049 <%2B94772207163> >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Thanuja Uruththirakodeeswaran >> Software Engineer >> WSO2 Inc.;http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 774363167 >> > > -- *Imesh Gunaratne* Senior Technical Lead WSO2 Inc: http://wso2.com T: +94 11 214 5345 M: +94 77 374 2057 W: http://imesh.io Lean . Enterprise . Middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
