Hi, I think we can copy all the files found under repository/components/lib > without specifically defining them in the Hiera file. WDYT?
This is possible if we are not copying any profile specific files. Since we are not copying profile specific files I think we can copy all the files found under components folder. On Tue, Apr 12, 2016 at 12:24 PM, Imesh Gunaratne <[email protected]> wrote: > +1 IMO it is better to proceed with this and later improve it as needed. > > One more point regarding the file list, I think we can copy all the files > found under repository/components/lib without specifically defining them in > the Hiera file. WDYT? > > Thanks > > On Tue, Apr 12, 2016 at 9:56 AM, Chamila De Alwis <[email protected]> > wrote: > >> Hi Thanuja, >> >> +1, although I'm again concerned with the structural complexity. However >> when weighed against other options available (having the YAML files in the >> same folder, enabling deep merge [1], etc) this is the best option as I >> see. >> >> `kubernetes/default.yaml` for WSO2 AM 1.10.0 would look something like >> the following. This would get duplicated for each product version which is >> a bit complex than needed and is unnecessary duplication IMO. >> >> [image: Inline image 1] >> >> [1] - [Dev] [Hiera] Deeper hash merge instead of Native hash merge? >> >> >> Regards, >> Chamila de Alwis >> Committer and PMC Member - Apache Stratos >> Software Engineer | WSO2 | +94772207163 >> Blog: code.chamiladealwis.com >> >> >> >> On Mon, Apr 11, 2016 at 4:15 PM, Thanuja Uruththirakodeeswaran < >> [email protected]> wrote: >> >>> >>> >>> On Mon, Apr 11, 2016 at 2:06 PM, Chamila De Alwis <[email protected]> >>> wrote: >>> >>>> Hi Pubudu, >>>> >>>> That's a good idea, however, we can't exactly map `environment` with >>>> the `platform` concept. How about using a new level on top of the profile >>>> specific level mapping to `platform`? >>>> >>>> :hierarchy: >>>> - "node/%{::clientcert}" >>>> * - >>>> "wso2/%{::product_name}/%{::product_version}/%{::product_profile}-%{::platform}"* >>>> - "wso2/%{::product_name}/%{::product_version}/%{::product_profile}" >>>> - "wso2/%{::product_name}/%{::product_version}/default" >>>> - "osfamily/%{::osfamily}" >>>> - "vm_type/%{::vm_type}" >>>> - wso2/common >>>> - common >>>> >>> >>> +1 for introducing a new level in hierarchy to lookup for platform >>> specific data on top of profile specific data per product. >>> >>>> >>>> This will allow us to extract the platform specific data only to one >>>> layer up. An example would be, >>>> >>>> puppet-modules/hieradata/dev/wso2/wso2am/1.10.0/default-kubernetes.yaml >>>> >>> >>> It would be better to have spearate folders for platform specific yaml >>> files inside each product versions like below with changing the above hiera >>> level to: *- >>> "wso2/%{::product_name}/%{::product_version}/%{::product_profile}/%{::platform}"* >>> >>> hieradata/dev/wso2/wso2as/ >>> >>> └── 5.3.0 >>> >>> ├── default.yaml >>> >>> ├── <*platform*> >>> >>> │ ├── default.yaml >>> >>> │ ├── manager.yaml >>> >>> │ └── worker.yaml >>> >>> ├── kubernetes >>> >>> │ ├── default.yaml >>> >>> │ ├── manager.yaml >>> >>> │ └── worker.yaml >>> >>> ├── manager.yaml >>> >>> └── worker.yaml >>> >>> >>> With the above structure, configs common to all platform will be there >>> in the products' root level hierafiles. Only the platform specific configs >>> related to a particular profile will be there in <platform>/<profile>.yaml >>> (ex: proxy_ports). >>> >>> >>> >>>> WDYT? >>>> >>>> >>>> Regards, >>>> Chamila de Alwis >>>> Committer and PMC Member - Apache Stratos >>>> Software Engineer | WSO2 | +94772207163 >>>> Blog: code.chamiladealwis.com >>>> >>>> >>>> >>>> On Mon, Apr 11, 2016 at 1:36 PM, Pubudu Gunatilaka <[email protected]> >>>> wrote: >>>> >>>>> Hi Isuru, >>>>> >>>>> Without duplicating files and adding new files, I would rather prefer >>>>> to use the following hierarchy. For each and every different environment >>>>> we >>>>> can have product profiles and include the environment specific values. For >>>>> Kubernetes, we can include the clustering section only in manager.yaml >>>>> file >>>>> which resides under environment Kubernetes. And the rest of the other >>>>> configurations can be included in the generic file folder, i.e " >>>>> wso2/%{::product_name}/%{::product_version}/%{::product_profile}". >>>>> >>>>> >>>>> :hierarchy: >>>>> - "node/%{::clientcert}" - - " >>>>> wso2/%{::product_name}/%{::product_version}/%{::environment} >>>>> /%{::product_profile}" >>>>> - "wso2/%{::product_name}/%{::product_version}/%{::product_profile}" >>>>> - "wso2/%{::product_name}/%{::product_version}/default" >>>>> - "osfamily/%{::osfamily}" >>>>> - "vm_type/%{::vm_type}" >>>>> - wso2/common >>>>> - common >>>>> >>>>> >>>>> Thank you! >>>>> >>>>> On Mon, Apr 11, 2016 at 1:07 PM, Isuru Haththotuwa <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Apr 11, 2016 at 12:47 PM, Gayan Gunarathne <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Apr 11, 2016 at 12:39 PM, Lakmal Warusawithana < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 11, 2016 at 12:22 PM, Imesh Gunaratne <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Chamila, >>>>>>>>> >>>>>>>>> On Mon, Apr 11, 2016 at 7:19 AM, Chamila De Alwis < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Isuru, Imesh, >>>>>>>>>> >>>>>>>>>> IMO we shouldn't have any platform specific restructuring in >>>>>>>>>> wso2/puppet-modules. This should be done at the end user's setup. >>>>>>>>>> Another >>>>>>>>>> point is that we have now decoupled wso2/puppet-modules from >>>>>>>>>> wso2/dockerfiles and users are not required to incorporate >>>>>>>>>> puppet-modules >>>>>>>>>> in their container setup. >>>>>>>>>> >>>>>>>>> >>>>>>>>> A very good concern! The way I see this is little different. Let's >>>>>>>>> evaluate the options we have: >>>>>>>>> >>>>>>>>> 1. Ship generic product/profile Hiera YAML files and let the >>>>>>>>> users configure them according to their platform (VM, AWS, Azure, >>>>>>>>> OpenStack, K8S, Mesos, OpenShift, CF, etc) >>>>>>>>> 2. Ship product/profile/platform Hiera YAML files and let >>>>>>>>> users use them OOB with very few changes. >>>>>>>>> >>>>>>>>> +1 for 2nd option. Yes, it has some duplication on configurations, >>>>>>>> but customer POV, it is very easy to look at single place to do the >>>>>>>> minimum >>>>>>>> changes. (rather looking for many files in deferent folders to do the >>>>>>>> changes) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Which one would be the best option? IMO 2nd option would provide a >>>>>>>>> much better user experience compared to 1 as it provides platform >>>>>>>>> specific >>>>>>>>> values such as clustering configuration & port mappings OOB. User >>>>>>>>> will only >>>>>>>>> need to provide values such as database hosts, passwords, identity >>>>>>>>> management, etc which are user specific. >>>>>>>>> >>>>>>>>> The whole idea of this effort is to provide a better user >>>>>>>>> experience. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>>> >>>>>>>>>> IMO Docker images will not be able to run OOB on Kubernetes using >>>>>>>>>> wso2/puppet-modules and wso2/kubernetes-artifacts. There will anyway >>>>>>>>>> be >>>>>>>>>> changes related to the Kubernetes Membership Scheme in >>>>>>>>>> wso2/puppet-modules >>>>>>>>>> and in wso2/kubernetes-artifacts where environment dependent changes >>>>>>>>>> such >>>>>>>>>> as image names, SecureVault passwords, etc. need to be adjusted. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Chamila de Alwis >>>>>>>>>> Committer and PMC Member - Apache Stratos >>>>>>>>>> Software Engineer | WSO2 | +94772207163 >>>>>>>>>> Blog: code.chamiladealwis.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Apr 11, 2016 at 1:36 AM, Imesh Gunaratne <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Gayan, >>>>>>>>>>> >>>>>>>>>>> On Sun, Apr 10, 2016 at 5:02 PM, Gayan Gunarathne < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> IMO this will create maintainability issue. We need to maintain >>>>>>>>>>>> all the separate hieradata structure for each scenarios.For the one >>>>>>>>>>>> particular alternation we need to change whole set of files. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> In this scenario user experience is much more important than the >>>>>>>>>>> maintainability of few yaml files. If we do not do this, users will >>>>>>>>>>> not be >>>>>>>>>>> able to use puppet modules OOB until they manually update >>>>>>>>>>> configuration >>>>>>>>>>> values in above files. The whole idea of this effort is to let >>>>>>>>>>> users do >>>>>>>>>>> following: >>>>>>>>>>> >>>>>>>>>>> - Setup a K8S cluster >>>>>>>>>>> - Download puppet modules zip file(s). >>>>>>>>>>> - Download docker files >>>>>>>>>>> - Build docker images using puppet for different product >>>>>>>>>>> profiles >>>>>>>>>>> - Deploy WSO2 product on K8S using K8S artifacts >>>>>>>>>>> >>>>>>>>>>> The above process will allow users to deploy any WSO2 product >>>>>>>>>>> (with mutlitple deployment patterns) on K8S with zero >>>>>>>>>>> configurations. This >>>>>>>>>>> will be true for any VM based platform or any other container >>>>>>>>>>> cluster >>>>>>>>>>> management system. >>>>>>>>>>> >>>>>>>>>> >>>>>>> Mainly the target users group of the puppet/hiera files will be >>>>>>> system administrators/Dev Ops. So those guys will be consider the fact >>>>>>> the >>>>>>> maintainability of puppet/hiera files. So if this is a maintainability >>>>>>> issue, it will become bad experience for the end user in end of the day. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>>>>> Why can't we do this by using defined types in Hiera and lookup >>>>>>>>>>>> parameters for a given instance? Based on the identify keys we >>>>>>>>>>>> set for each vm, docker, K8S etc we can select the appropriate data >>>>>>>>>>>> set from Hiera file. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Will you be able to provide a sample? >>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>> I think we can make this with with Defined Types[1][2] without >>>>>>> creating duplicate set of YAML files for each platform. We can do the >>>>>>> same >>>>>>> as the example given in the document. >>>>>>> >>>>>> I do not think we need to duplicate the yaml files here. Sorry that >>>>>> the sample in the first reply sent by me implied so. What we can do is >>>>>> refactor out the hiera data so that data which is changing according to >>>>>> the >>>>>> platform (ex.: clustering section) can be moved to a different yaml >>>>>> file(s). For an example, clustering for kubernetes scenario can be >>>>>> included >>>>>> in *puppet-modules/hieradata/dev/wso2/kubernetes/clustering.yaml*. >>>>>> Here, 'kubernetes' is the value of the facter which is used to identify >>>>>> the >>>>>> environment. Similarly, other such data can be refactored out. >>>>>> >>>>> >>> If we follow this approach, then we can't specify product and profile >>> specific data (ex: kubernetes service name varies per product and >>> proxy_ports varies per poduct/profile) in the common platform hiera file: >>> *puppet-modules/hieradata/dev/wso2/kubernetes/clustering.yaml.* >>> >>> >>>>>>> [1] >>>>>>> https://docs.puppet.com/puppet/latest/reference/lang_defined_types.html >>>>>>> [2]http://puppetlunch.com/puppet/hiera.html >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Gayan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Apr 9, 2016 at 8:28 AM, Imesh Gunaratne <[email protected] >>>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Apr 8, 2016 at 7:48 PM, Isuru Haththotuwa < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> hieradata >>>>>>>>>>>>>> |--- dev >>>>>>>>>>>>>> |--- wso2 >>>>>>>>>>>>>> |---- <product_name> >>>>>>>>>>>>>> |--- <product_version> >>>>>>>>>>>>>> |-- >>>>>>>>>>>>>> *vm* >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- default.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- manager.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- worker.yaml >>>>>>>>>>>>>> |--* >>>>>>>>>>>>>> docker* >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- default.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- manager.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- worker.yaml >>>>>>>>>>>>>> |-- >>>>>>>>>>>>>> *kubernetes* >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- default.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- manager.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> |-- worker.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> +1 for the suggestion Isuru, will proceed with this. We can >>>>>>>>>>>>> add other platforms such as OpenShift, Mesos, Cloud Foundry on >>>>>>>>>>>>> the same >>>>>>>>>>>>> level. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>> W: http://imesh.io >>>>>>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Isuru H. >>>>>>>>>>>>>> +94 716 358 048* <http://wso2.com/>* >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>> W: http://imesh.io >>>>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Gayan Gunarathne >>>>>>>>>>>> Technical Lead, WSO2 Inc. (http://wso2.com) >>>>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>>>> email : [email protected] | mobile : +94 775030545 >>>>>>>>>>>> <%2B94%20766819985> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>> Senior Technical Lead >>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>> W: http://imesh.io >>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Imesh Gunaratne* >>>>>>>>> Senior Technical Lead >>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>> W: http://imesh.io >>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Lakmal Warusawithana >>>>>>>> Director - Cloud Architecture; WSO2 Inc. >>>>>>>> Mobile : +94714289692 >>>>>>>> Blog : http://lakmalsview.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Gayan Gunarathne >>>>>>> Technical Lead, WSO2 Inc. (http://wso2.com) >>>>>>> Committer & PMC Member, Apache Stratos >>>>>>> email : [email protected] | mobile : +94 775030545 >>>>>>> <%2B94%20766819985> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks and Regards, >>>>>> >>>>>> Isuru H. >>>>>> +94 716 358 048* <http://wso2.com/>* >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Pubudu Gunatilaka* >>>>> Committer and PMC Member - Apache Stratos >>>>> Software Engineer >>>>> WSO2, Inc.: http://wso2.com >>>>> mobile : +94774078049 <%2B94772207163> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Thanuja Uruththirakodeeswaran >>> Software Engineer >>> WSO2 Inc.;http://wso2.com >>> lean.enterprise.middleware >>> >>> mobile: +94 774363167 >>> >> >> > > > -- > *Imesh Gunaratne* > Senior Technical Lead > WSO2 Inc: http://wso2.com > T: +94 11 214 5345 M: +94 77 374 2057 > W: http://imesh.io > Lean . Enterprise . Middleware > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Thanks and Regards,* Anuruddha Lanka Liyanarachchi Software Engineer - WSO2 Mobile : +94 (0) 712762611 Tel : +94 112 145 345 a <[email protected]>[email protected]
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
