Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Imesh Gunaratne]

On Thu, Sep 8, 2016 at 5:56 AM, Akila Ravihansa Perera 
wrote:

> Hi Imesh,
>
> On Thu, Sep 8, 2016 at 3:47 AM, Imesh Gunaratne  wrote:
>
>> What would be the reason for using rsync in the Vagrant setup [2]? Do we
>> update files inside the VM?
>>
>> [2] https://github.com/wso2/puppet-common/blob/56b34c4c2db11f07d
>> ca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L53
>>
>
> We ran into an issue when mounting Hieradata folder (which is a symlink
> now) to the VM. It seems Vagrant does not fully support mounting symlinks
> so we had to rsync it.
>

I talked to Anuruddha on this. This is what has happened in ESB module:

   - The setup.sh bash script has tried to create symlinks to hieradata
   folders from product puppet module and those have failed.
   - However the bash script has not detected that and continued rest of
   the logic.
   - As a result of this, file synchronization in Vagrant has failed with
   the above error.

IMO if a such error occurs in setup.sh, it should stop at that point and
raise an error.

@Akila: I don't think using rsync as a workaround for some other problem is
a good solution. If the root cause of the issue is with symlinks better to
fix that.

[3] https://www.vagrantup.com/docs/synced-folders/rsync.html

Thanks

​
> Thanks.
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>



-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
lean. enterprise. middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Pubudu Gunatilaka]

Hi Akila,

Great. I will remove unused directory_list variable in [1].

[1] -
https://github.com/wso2/puppet-base/blob/master/manifests/configure.pp#L21

Thank you!

On Thu, Sep 8, 2016 at 2:57 PM, Akila Ravihansa Perera 
wrote:

> Hi Pubudu,
>
> That feature is still available but moved to install stage (install.pp),
> because all the resources in configure stage depend on that.
>
> Thanks.
>
>
> On Thursday, 8 September 2016, Pubudu Gunatilaka  wrote:
>
>> Hi Akila,
>>
>> In puppet modules 2.1.0 release we had the capability to create
>> directories for a given folder structure [1] and it has used in some
>> products [2]. In latest puppet changes I could not find that option except
>> the directory list value [3]. Did we remove that feature on purpose? If not
>> shall we add that?
>>
>> [1] - https://github.com/wso2/puppet-modules/blob/master/modules
>> /wso2base/manifests/configure.pp#L19
>> [2] - https://github.com/wso2/puppet-modules/blob/master/hierada
>> ta/dev/wso2/wso2das/3.0.1/mesos/default.yaml#L44
>> [3] - https://github.com/wso2/puppet-base/blob/master/manifests/
>> configure.pp#L21
>>
>> Thank you!
>>
>> On Thu, Sep 8, 2016 at 6:23 AM, Akila Ravihansa Perera <
>> raviha...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Adding few more details I'd like to highlight;
>>>
>>> * - Removed Java 3rd party Puppet module dependency*
>>> In earlier versions we used an external 3rd party Puppet module for Java
>>> installation which we thought is not necessary anymore. Therefore we have
>>> implemented our own Java class without any dependencies.
>>>
>>> The only contract between Java module and WSO2 Puppet modules is it
>>> should install Java JDK on "JAVA_HOME" path. JAVA_HOME is expected as a
>>> class parameter by all WSO2 Puppet modules and defaults to "/opt/java".
>>> Usually this is a symlink and actual JDK installation path may differ.
>>> As a convenience for users we have provided a basic java class in
>>> wso2base module which takes the JDK from Puppet file bucket, extracts it
>>> and sets the JAVA_HOME symlink. Java class is *not *contained in any of
>>> the WSO2 Puppet module classes so it has to be added from site.pp. For eg:
>>>
>>> ## site.pp
>>> class { 'wso2base::java' } -> class { 'wso2as' }
>>>
>>> This offers greater flexibility for users to plug-in their own Java
>>> module.
>>>
>>>
>>> * - Use of Puppet file server*
>>> Earlier users had to copy jdk and product packs into respective Puppet
>>> module's files directory. We have improved Puppet scripts to lookup in
>>> Puppet file server path as well. For eg: packs are searched in
>>> /files/packs directory which contains packs for all the
>>> products. This is again a convenience for the users. This applies to
>>> patches and configs files as well.
>>> You can try this feature with Vagrant. By default it takes Puppet file
>>> server path as /files directory and pass that as a Puppet
>>> options parameter [1].
>>>
>>>
>>> * - Patches structure*
>>> With the introduction of "patch_list" parameter together with Puppet
>>> file server support, now you do not need to copy patches to each and every
>>> Puppet module! Patches are grouped by platform version therefore patches
>>> dir will be as follows;
>>>
>>> /files/patches
>>> - 4.2.0
>>>- patch0001
>>>  ...
>>>
>>> - 4.4.0
>>>- patch0005
>>>  ...
>>>
>>> Note that "patch_list" parameter is optional. Meaning, if you keep it
>>> empty or undef, Puppet will copy all the patches you have placed inside
>>> respective Puppet module's files/patches//* directory.
>>> If you set the "patch_list" as an array, it will search the Puppet file
>>> server and Puppet module's files bucket for the specified patches. The
>>> difference is, the latter method will error if any of the specified patches
>>> are missing. This is useful when you want to ensure that all required
>>> patches are copied.
>>>
>>>
>>> [1] https://github.com/wso2/puppet-common/blob/56b34c4c2db11
>>> f07dca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L78
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Sep 8, 2016 at 5:56 AM, Akila Ravihansa Perera <
>>> raviha...@wso2.com> wrote:
>>>
 Hi Imesh,


 On Thu, Sep 8, 2016 at 3:47 AM, Imesh Gunaratne  wrote:

> What would be the reason for using rsync in the Vagrant setup [2]? Do
> we update files inside the VM?
>
> [2] https://github.com/wso2/puppet-common/blob/56b34c4c2db11f07d
> ca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L53
>

 We ran into an issue when mounting Hieradata folder (which is a symlink
 now) to the VM. It seems Vagrant does not fully support mounting symlinks
 so we had to rsync it.

 Thanks.

 --
 Akila Ravihansa Perera
 WSO2 Inc.;  http://wso2.com/

 Blog: http://ravihansa3000.blogspot.com

>>>
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> WSO2 Inc.;  http://wso2.com/
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Akila Ravihansa Perera]

Hi Pubudu,

That feature is still available but moved to install stage (install.pp),
because all the resources in configure stage depend on that.

Thanks.

On Thursday, 8 September 2016, Pubudu Gunatilaka  wrote:

> Hi Akila,
>
> In puppet modules 2.1.0 release we had the capability to create
> directories for a given folder structure [1] and it has used in some
> products [2]. In latest puppet changes I could not find that option except
> the directory list value [3]. Did we remove that feature on purpose? If not
> shall we add that?
>
> [1] - https://github.com/wso2/puppet-modules/blob/master/
> modules/wso2base/manifests/configure.pp#L19
> [2] - https://github.com/wso2/puppet-modules/blob/master/
> hieradata/dev/wso2/wso2das/3.0.1/mesos/default.yaml#L44
> [3] - https://github.com/wso2/puppet-base/blob/master/
> manifests/configure.pp#L21
>
> Thank you!
>
> On Thu, Sep 8, 2016 at 6:23 AM, Akila Ravihansa Perera  > wrote:
>
>> Hi,
>>
>> Adding few more details I'd like to highlight;
>>
>> * - Removed Java 3rd party Puppet module dependency*
>> In earlier versions we used an external 3rd party Puppet module for Java
>> installation which we thought is not necessary anymore. Therefore we have
>> implemented our own Java class without any dependencies.
>>
>> The only contract between Java module and WSO2 Puppet modules is it
>> should install Java JDK on "JAVA_HOME" path. JAVA_HOME is expected as a
>> class parameter by all WSO2 Puppet modules and defaults to "/opt/java".
>> Usually this is a symlink and actual JDK installation path may differ.
>> As a convenience for users we have provided a basic java class in
>> wso2base module which takes the JDK from Puppet file bucket, extracts it
>> and sets the JAVA_HOME symlink. Java class is *not *contained in any of
>> the WSO2 Puppet module classes so it has to be added from site.pp. For eg:
>>
>> ## site.pp
>> class { 'wso2base::java' } -> class { 'wso2as' }
>>
>> This offers greater flexibility for users to plug-in their own Java
>> module.
>>
>>
>> * - Use of Puppet file server*
>> Earlier users had to copy jdk and product packs into respective Puppet
>> module's files directory. We have improved Puppet scripts to lookup in
>> Puppet file server path as well. For eg: packs are searched in
>> /files/packs directory which contains packs for all the
>> products. This is again a convenience for the users. This applies to
>> patches and configs files as well.
>> You can try this feature with Vagrant. By default it takes Puppet file
>> server path as /files directory and pass that as a Puppet
>> options parameter [1].
>>
>>
>> * - Patches structure*
>> With the introduction of "patch_list" parameter together with Puppet file
>> server support, now you do not need to copy patches to each and every
>> Puppet module! Patches are grouped by platform version therefore patches
>> dir will be as follows;
>>
>> /files/patches
>> - 4.2.0
>>- patch0001
>>  ...
>>
>> - 4.4.0
>>- patch0005
>>  ...
>>
>> Note that "patch_list" parameter is optional. Meaning, if you keep it
>> empty or undef, Puppet will copy all the patches you have placed inside
>> respective Puppet module's files/patches//* directory.
>> If you set the "patch_list" as an array, it will search the Puppet file
>> server and Puppet module's files bucket for the specified patches. The
>> difference is, the latter method will error if any of the specified patches
>> are missing. This is useful when you want to ensure that all required
>> patches are copied.
>>
>>
>> [1] https://github.com/wso2/puppet-common/blob/56b34c4c2db11
>> f07dca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L78
>>
>> Thanks.
>>
>>
>> On Thu, Sep 8, 2016 at 5:56 AM, Akila Ravihansa Perera <
>> raviha...@wso2.com >
>> wrote:
>>
>>> Hi Imesh,
>>>
>>>
>>> On Thu, Sep 8, 2016 at 3:47 AM, Imesh Gunaratne >> > wrote:
>>>
 What would be the reason for using rsync in the Vagrant setup [2]? Do
 we update files inside the VM?

 [2] https://github.com/wso2/puppet-common/blob/56b34c4c2db11f07d
 ca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L53

>>>
>>> We ran into an issue when mounting Hieradata folder (which is a symlink
>>> now) to the VM. It seems Vagrant does not fully support mounting symlinks
>>> so we had to rsync it.
>>>
>>> Thanks.
>>>
>>> --
>>> Akila Ravihansa Perera
>>> WSO2 Inc.;  http://wso2.com/
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.;  http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> *Pubudu Gunatilaka*
> Committer and PMC Member - Apache Stratos
> Software Engineer
> WSO2, Inc.: http://wso2.com
> mobile : +94774078049 <%2B94772207163>
>
>

-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/list

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Pubudu Gunatilaka]

Hi Akila,

In puppet modules 2.1.0 release we had the capability to create directories
for a given folder structure [1] and it has used in some products [2]. In
latest puppet changes I could not find that option except the directory
list value [3]. Did we remove that feature on purpose? If not shall we add
that?

[1] -
https://github.com/wso2/puppet-modules/blob/master/modules/wso2base/manifests/configure.pp#L19
[2] -
https://github.com/wso2/puppet-modules/blob/master/hieradata/dev/wso2/wso2das/3.0.1/mesos/default.yaml#L44
[3] -
https://github.com/wso2/puppet-base/blob/master/manifests/configure.pp#L21

Thank you!

On Thu, Sep 8, 2016 at 6:23 AM, Akila Ravihansa Perera 
wrote:

> Hi,
>
> Adding few more details I'd like to highlight;
>
> * - Removed Java 3rd party Puppet module dependency*
> In earlier versions we used an external 3rd party Puppet module for Java
> installation which we thought is not necessary anymore. Therefore we have
> implemented our own Java class without any dependencies.
>
> The only contract between Java module and WSO2 Puppet modules is it should
> install Java JDK on "JAVA_HOME" path. JAVA_HOME is expected as a class
> parameter by all WSO2 Puppet modules and defaults to "/opt/java". Usually
> this is a symlink and actual JDK installation path may differ.
> As a convenience for users we have provided a basic java class in wso2base
> module which takes the JDK from Puppet file bucket, extracts it and sets
> the JAVA_HOME symlink. Java class is *not *contained in any of the WSO2
> Puppet module classes so it has to be added from site.pp. For eg:
>
> ## site.pp
> class { 'wso2base::java' } -> class { 'wso2as' }
>
> This offers greater flexibility for users to plug-in their own Java module.
>
>
> * - Use of Puppet file server*
> Earlier users had to copy jdk and product packs into respective Puppet
> module's files directory. We have improved Puppet scripts to lookup in
> Puppet file server path as well. For eg: packs are searched in
> /files/packs directory which contains packs for all the
> products. This is again a convenience for the users. This applies to
> patches and configs files as well.
> You can try this feature with Vagrant. By default it takes Puppet file
> server path as /files directory and pass that as a Puppet
> options parameter [1].
>
>
> * - Patches structure*
> With the introduction of "patch_list" parameter together with Puppet file
> server support, now you do not need to copy patches to each and every
> Puppet module! Patches are grouped by platform version therefore patches
> dir will be as follows;
>
> /files/patches
> - 4.2.0
>- patch0001
>  ...
>
> - 4.4.0
>- patch0005
>  ...
>
> Note that "patch_list" parameter is optional. Meaning, if you keep it
> empty or undef, Puppet will copy all the patches you have placed inside
> respective Puppet module's files/patches//* directory.
> If you set the "patch_list" as an array, it will search the Puppet file
> server and Puppet module's files bucket for the specified patches. The
> difference is, the latter method will error if any of the specified patches
> are missing. This is useful when you want to ensure that all required
> patches are copied.
>
>
> [1] https://github.com/wso2/puppet-common/blob/
> 56b34c4c2db11f07dca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L78
>
> Thanks.
>
>
> On Thu, Sep 8, 2016 at 5:56 AM, Akila Ravihansa Perera  > wrote:
>
>> Hi Imesh,
>>
>>
>> On Thu, Sep 8, 2016 at 3:47 AM, Imesh Gunaratne  wrote:
>>
>>> What would be the reason for using rsync in the Vagrant setup [2]? Do we
>>> update files inside the VM?
>>>
>>> [2] https://github.com/wso2/puppet-common/blob/56b34c4c2db11f07d
>>> ca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L53
>>>
>>
>> We ran into an issue when mounting Hieradata folder (which is a symlink
>> now) to the VM. It seems Vagrant does not fully support mounting symlinks
>> so we had to rsync it.
>>
>> Thanks.
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.;  http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>



-- 
*Pubudu Gunatilaka*
Committer and PMC Member - Apache Stratos
Software Engineer
WSO2, Inc.: http://wso2.com
mobile : +94774078049 <%2B94772207163>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Akila Ravihansa Perera]

Hi,

Adding few more details I'd like to highlight;

* - Removed Java 3rd party Puppet module dependency*
In earlier versions we used an external 3rd party Puppet module for Java
installation which we thought is not necessary anymore. Therefore we have
implemented our own Java class without any dependencies.

The only contract between Java module and WSO2 Puppet modules is it should
install Java JDK on "JAVA_HOME" path. JAVA_HOME is expected as a class
parameter by all WSO2 Puppet modules and defaults to "/opt/java". Usually
this is a symlink and actual JDK installation path may differ.
As a convenience for users we have provided a basic java class in wso2base
module which takes the JDK from Puppet file bucket, extracts it and sets
the JAVA_HOME symlink. Java class is *not *contained in any of the WSO2
Puppet module classes so it has to be added from site.pp. For eg:

## site.pp
class { 'wso2base::java' } -> class { 'wso2as' }

This offers greater flexibility for users to plug-in their own Java module.


* - Use of Puppet file server*
Earlier users had to copy jdk and product packs into respective Puppet
module's files directory. We have improved Puppet scripts to lookup in
Puppet file server path as well. For eg: packs are searched in
/files/packs directory which contains packs for all the
products. This is again a convenience for the users. This applies to
patches and configs files as well.
You can try this feature with Vagrant. By default it takes Puppet file
server path as /files directory and pass that as a Puppet
options parameter [1].


* - Patches structure*
With the introduction of "patch_list" parameter together with Puppet file
server support, now you do not need to copy patches to each and every
Puppet module! Patches are grouped by platform version therefore patches
dir will be as follows;

/files/patches
- 4.2.0
   - patch0001
 ...

- 4.4.0
   - patch0005
 ...

Note that "patch_list" parameter is optional. Meaning, if you keep it empty
or undef, Puppet will copy all the patches you have placed inside
respective Puppet module's files/patches//* directory.
If you set the "patch_list" as an array, it will search the Puppet file
server and Puppet module's files bucket for the specified patches. The
difference is, the latter method will error if any of the specified patches
are missing. This is useful when you want to ensure that all required
patches are copied.


[1]
https://github.com/wso2/puppet-common/blob/56b34c4c2db11f07dca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L78

Thanks.


On Thu, Sep 8, 2016 at 5:56 AM, Akila Ravihansa Perera 
wrote:

> Hi Imesh,
>
>
> On Thu, Sep 8, 2016 at 3:47 AM, Imesh Gunaratne  wrote:
>
>> What would be the reason for using rsync in the Vagrant setup [2]? Do we
>> update files inside the VM?
>>
>> [2] https://github.com/wso2/puppet-common/blob/56b34c4c2db11f07d
>> ca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L53
>>
>
> We ran into an issue when mounting Hieradata folder (which is a symlink
> now) to the VM. It seems Vagrant does not fully support mounting symlinks
> so we had to rsync it.
>
> Thanks.
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>



-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Akila Ravihansa Perera]

Hi Imesh,


On Thu, Sep 8, 2016 at 3:47 AM, Imesh Gunaratne  wrote:

> What would be the reason for using rsync in the Vagrant setup [2]? Do we
> update files inside the VM?
>
> [2] https://github.com/wso2/puppet-common/blob/
> 56b34c4c2db11f07dca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L53
>

We ran into an issue when mounting Hieradata folder (which is a symlink
now) to the VM. It seems Vagrant does not fully support mounting symlinks
so we had to rsync it.

Thanks.

-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Imesh Gunaratne]

What would be the reason for using rsync in the Vagrant setup [2]? Do we
update files inside the VM?

[2]
https://github.com/wso2/puppet-common/blob/56b34c4c2db11f07dca739f0e92bf1d955cfcb46/vagrant/Vagrantfile#L53

Thanks

On Wed, Sep 7, 2016 at 10:01 PM, Pubudu Gunatilaka  wrote:

> Hi Rushmin,
>
> We have only completed wso2as puppet module [1]. We have not completed
> wso2esb and it is not upto date. Currently we are working on other puppet
> modules including wso2esb.
>
> [1] - https://github.com/wso2/puppet-as
>
> Thank you!
>
> On Wed, Sep 7, 2016 at 7:31 PM, Rushmin Fernando  wrote:
>
>> Akila I got the following error while I was trying with ESB.
>>
>>
>> ==> esb.dev.wso2.org: Rsyncing folder: 
>> /home/rushmin/wso2/dev/env/puppet/hieradata/
>> => /puppet/hieradata
>> There was an error when attempting to rsync a synced folder.
>> Please inspect the error message below for more info.
>>
>> Host path: /home/rushmin/wso2/dev/env/puppet/hieradata/
>> Guest path: /puppet/hieradata
>> Command: rsync --verbose --archive --delete -z --copy-links --no-owner
>> --no-group --rsync-path sudo rsync -e ssh -p  -o
>> StrictHostKeyChecking=no -o IdentitiesOnly=true -o
>> UserKnownHostsFile=/dev/null -i '/home/rushmin/wso2/dev/rnd/sr
>> c/git/puppet-commons/vagrant/.vagrant/machines/esb.dev.wso2.
>> org/virtualbox/private_key' --exclude .vagrant/
>> /home/rushmin/wso2/dev/env/puppet/hieradata/ vagrant@127.0.0.1:
>> /puppet/hieradata
>> Error: Warning: Permanently added '[127.0.0.1]:' (ECDSA) to the list
>> of known hosts.
>> symlink has no referent: "/home/rushmin/wso2/dev/env/pu
>> ppet/hieradata/dev/wso2/wso2apim"
>> rsync error: some files/attrs were not transferred (see previous errors)
>> (code 23) at main.c(1183) [sender=3.1.0]
>>
>> What am I doing wrong here ?
>>
>> Best Regards
>> Rushmin
>>
>> On Wed, Sep 7, 2016 at 2:49 PM, Akila Ravihansa Perera <
>> raviha...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> This thread is to raise awareness around some of the implementation
>>> level decisions that were taken when refactoring Puppet modules and
>>> splitting them into their own respective GitHub repos.
>>>
>>> * - Decouple Hiera from Puppet modules*
>>> In last Puppet-Modules release Hiera lookups were done in >> module>/manifests/init.pp which made it mandatory for users to use Hiera.
>>> As part of the refactoring Hiera is made optional and configuration
>>> parameters are defined as class parameters which can be passed by site.pp
>>>
>>>
>>> * - Use Hiera only as a data store*
>>> There were some discussions about creating hooks in Puppet manifest to
>>> which Puppet resources can be plugged in via Hiera data. However, going
>>> forward we have come to a conclusion that Hiera should only be used as a
>>> data store and nothing more. Any customizations to Puppet manifest can be
>>> done directly in init.pp.
>>>
>>> * - Break Puppet configurations into stages*
>>> We have identified following stages which can be considered completely
>>> independent from each other;
>>> system - set operating system configs, install packages etc.
>>> clean - remove any existing installation
>>> install - extract the pack and set user permissions
>>> configure - populate templates, copy config files, import certs
>>> service - start the service
>>>
>>> Here is a snippet of wso2as/manifests/init.pp
>>>
>>> contain wso2base
>>> contain wso2base::system
>>> contain wso2base::clean
>>> contain wso2base::install
>>> contain wso2base::configure
>>> contain wso2base::service
>>>
>>> Class['::wso2base'] -> Class['::wso2base::system']
>>> -> Class['::wso2base::clean'] -> Class['::wso2base::install']
>>> -> Class['::wso2base::configure'] ~> Class['::wso2base::service']
>>>
>>> As you can see Puppet class containment [1] is used to ensure the
>>> dependency ordering. This is useful when users combine WSO2 Puppet modules
>>> with other modules (for eg: when creating roles and profiles).
>>>
>>> * - Remove inter module class inheritance*
>>> As per official Puppet docs class inheritance is to be avoided whenever
>>> possible [2]. Considering this we have removed all such relationships
>>> except for params.pp which contains default values.
>>>
>>> * - Move default values to params.pp and remove wso2base/params.pp*
>>> After giving much thought, we decided to keep default values in
>>> /manifests/params.pp and only in that place. Having
>>> common default values in /manifests/params.pp would minimize
>>> duplication but would be less readable for end users. Therefore we have
>>> come to a conclusion that wso2base module should only be used as a common
>>> resources/classes pool. It should not contain any data.
>>>
>>> *Try it out!*
>>>
>>> 1. Git clone https://github.com/wso2/puppet-common
>>> 2. Run puppet-common/setup.sh -p as
>>> 3. Goto puppet-common/vagrant
>>> 4. Run vagrant up
>>>
>>>
>>> [1] https://docs.puppet.com/puppet/latest/reference/lang_con
>>> tainment.html#containing-classes
>>> [2] https://docs.puppet.com/guides/style_

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Pubudu Gunatilaka]

Hi Rushmin,

We have only completed wso2as puppet module [1]. We have not completed
wso2esb and it is not upto date. Currently we are working on other puppet
modules including wso2esb.

[1] - https://github.com/wso2/puppet-as

Thank you!

On Wed, Sep 7, 2016 at 7:31 PM, Rushmin Fernando  wrote:

> Akila I got the following error while I was trying with ESB.
>
>
> ==> esb.dev.wso2.org: Rsyncing folder: 
> /home/rushmin/wso2/dev/env/puppet/hieradata/
> => /puppet/hieradata
> There was an error when attempting to rsync a synced folder.
> Please inspect the error message below for more info.
>
> Host path: /home/rushmin/wso2/dev/env/puppet/hieradata/
> Guest path: /puppet/hieradata
> Command: rsync --verbose --archive --delete -z --copy-links --no-owner
> --no-group --rsync-path sudo rsync -e ssh -p  -o
> StrictHostKeyChecking=no -o IdentitiesOnly=true -o
> UserKnownHostsFile=/dev/null -i '/home/rushmin/wso2/dev/rnd/
> src/git/puppet-commons/vagrant/.vagrant/machines/esb.
> dev.wso2.org/virtualbox/private_key' --exclude .vagrant/
> /home/rushmin/wso2/dev/env/puppet/hieradata/ vagrant@127.0.0.1:/puppet/
> hieradata
> Error: Warning: Permanently added '[127.0.0.1]:' (ECDSA) to the list
> of known hosts.
> symlink has no referent: "/home/rushmin/wso2/dev/env/
> puppet/hieradata/dev/wso2/wso2apim"
> rsync error: some files/attrs were not transferred (see previous errors)
> (code 23) at main.c(1183) [sender=3.1.0]
>
> What am I doing wrong here ?
>
> Best Regards
> Rushmin
>
> On Wed, Sep 7, 2016 at 2:49 PM, Akila Ravihansa Perera  > wrote:
>
>> Hi,
>>
>> This thread is to raise awareness around some of the implementation level
>> decisions that were taken when refactoring Puppet modules and splitting
>> them into their own respective GitHub repos.
>>
>> * - Decouple Hiera from Puppet modules*
>> In last Puppet-Modules release Hiera lookups were done in > module>/manifests/init.pp which made it mandatory for users to use Hiera.
>> As part of the refactoring Hiera is made optional and configuration
>> parameters are defined as class parameters which can be passed by site.pp
>>
>>
>> * - Use Hiera only as a data store*
>> There were some discussions about creating hooks in Puppet manifest to
>> which Puppet resources can be plugged in via Hiera data. However, going
>> forward we have come to a conclusion that Hiera should only be used as a
>> data store and nothing more. Any customizations to Puppet manifest can be
>> done directly in init.pp.
>>
>> * - Break Puppet configurations into stages*
>> We have identified following stages which can be considered completely
>> independent from each other;
>> system - set operating system configs, install packages etc.
>> clean - remove any existing installation
>> install - extract the pack and set user permissions
>> configure - populate templates, copy config files, import certs
>> service - start the service
>>
>> Here is a snippet of wso2as/manifests/init.pp
>>
>> contain wso2base
>> contain wso2base::system
>> contain wso2base::clean
>> contain wso2base::install
>> contain wso2base::configure
>> contain wso2base::service
>>
>> Class['::wso2base'] -> Class['::wso2base::system']
>> -> Class['::wso2base::clean'] -> Class['::wso2base::install']
>> -> Class['::wso2base::configure'] ~> Class['::wso2base::service']
>>
>> As you can see Puppet class containment [1] is used to ensure the
>> dependency ordering. This is useful when users combine WSO2 Puppet modules
>> with other modules (for eg: when creating roles and profiles).
>>
>> * - Remove inter module class inheritance*
>> As per official Puppet docs class inheritance is to be avoided whenever
>> possible [2]. Considering this we have removed all such relationships
>> except for params.pp which contains default values.
>>
>> * - Move default values to params.pp and remove wso2base/params.pp*
>> After giving much thought, we decided to keep default values in
>> /manifests/params.pp and only in that place. Having
>> common default values in /manifests/params.pp would minimize
>> duplication but would be less readable for end users. Therefore we have
>> come to a conclusion that wso2base module should only be used as a common
>> resources/classes pool. It should not contain any data.
>>
>> *Try it out!*
>>
>> 1. Git clone https://github.com/wso2/puppet-common
>> 2. Run puppet-common/setup.sh -p as
>> 3. Goto puppet-common/vagrant
>> 4. Run vagrant up
>>
>>
>> [1] https://docs.puppet.com/puppet/latest/reference/lang_con
>> tainment.html#containing-classes
>> [2] https://docs.puppet.com/guides/style_guide.html#class-inheritance
>>
>> Thanks.
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.;  http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> *Best Regards*
>
> *Rushmin Fernando*
> *Technical Lead*
>
> WSO2 Inc.  - Lean . Enterprise . Midd

Re: [Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Rushmin Fernando]

Akila I got the following error while I was trying with ESB.


==> esb.dev.wso2.org: Rsyncing folder:
/home/rushmin/wso2/dev/env/puppet/hieradata/ => /puppet/hieradata
There was an error when attempting to rsync a synced folder.
Please inspect the error message below for more info.

Host path: /home/rushmin/wso2/dev/env/puppet/hieradata/
Guest path: /puppet/hieradata
Command: rsync --verbose --archive --delete -z --copy-links --no-owner
--no-group --rsync-path sudo rsync -e ssh -p  -o
StrictHostKeyChecking=no -o IdentitiesOnly=true -o
UserKnownHostsFile=/dev/null -i
'/home/rushmin/wso2/dev/rnd/src/git/puppet-commons/vagrant/.vagrant/machines/
esb.dev.wso2.org/virtualbox/private_key' --exclude .vagrant/
/home/rushmin/wso2/dev/env/puppet/hieradata/ vagrant@127.0.0.1:
/puppet/hieradata
Error: Warning: Permanently added '[127.0.0.1]:' (ECDSA) to the list of
known hosts.
symlink has no referent:
"/home/rushmin/wso2/dev/env/puppet/hieradata/dev/wso2/wso2apim"
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1183) [sender=3.1.0]

What am I doing wrong here ?

Best Regards
Rushmin

On Wed, Sep 7, 2016 at 2:49 PM, Akila Ravihansa Perera 
wrote:

> Hi,
>
> This thread is to raise awareness around some of the implementation level
> decisions that were taken when refactoring Puppet modules and splitting
> them into their own respective GitHub repos.
>
> * - Decouple Hiera from Puppet modules*
> In last Puppet-Modules release Hiera lookups were done in  module>/manifests/init.pp which made it mandatory for users to use Hiera.
> As part of the refactoring Hiera is made optional and configuration
> parameters are defined as class parameters which can be passed by site.pp
>
>
> * - Use Hiera only as a data store*
> There were some discussions about creating hooks in Puppet manifest to
> which Puppet resources can be plugged in via Hiera data. However, going
> forward we have come to a conclusion that Hiera should only be used as a
> data store and nothing more. Any customizations to Puppet manifest can be
> done directly in init.pp.
>
> * - Break Puppet configurations into stages*
> We have identified following stages which can be considered completely
> independent from each other;
> system - set operating system configs, install packages etc.
> clean - remove any existing installation
> install - extract the pack and set user permissions
> configure - populate templates, copy config files, import certs
> service - start the service
>
> Here is a snippet of wso2as/manifests/init.pp
>
> contain wso2base
> contain wso2base::system
> contain wso2base::clean
> contain wso2base::install
> contain wso2base::configure
> contain wso2base::service
>
> Class['::wso2base'] -> Class['::wso2base::system']
> -> Class['::wso2base::clean'] -> Class['::wso2base::install']
> -> Class['::wso2base::configure'] ~> Class['::wso2base::service']
>
> As you can see Puppet class containment [1] is used to ensure the
> dependency ordering. This is useful when users combine WSO2 Puppet modules
> with other modules (for eg: when creating roles and profiles).
>
> * - Remove inter module class inheritance*
> As per official Puppet docs class inheritance is to be avoided whenever
> possible [2]. Considering this we have removed all such relationships
> except for params.pp which contains default values.
>
> * - Move default values to params.pp and remove wso2base/params.pp*
> After giving much thought, we decided to keep default values in
> /manifests/params.pp and only in that place. Having common
> default values in /manifests/params.pp would minimize duplication
> but would be less readable for end users. Therefore we have come to a
> conclusion that wso2base module should only be used as a common
> resources/classes pool. It should not contain any data.
>
> *Try it out!*
>
> 1. Git clone https://github.com/wso2/puppet-common
> 2. Run puppet-common/setup.sh -p as
> 3. Goto puppet-common/vagrant
> 4. Run vagrant up
>
>
> [1] https://docs.puppet.com/puppet/latest/reference/lang_
> containment.html#containing-classes
> [2] https://docs.puppet.com/guides/style_guide.html#class-inheritance
>
> Thanks.
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
*Best Regards*

*Rushmin Fernando*
*Technical Lead*

WSO2 Inc.  - Lean . Enterprise . Middleware

mobile : +94772891266
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [Puppet] Deep dive into WSO2 Puppet module implementation

[Akila Ravihansa Perera]

Hi,

This thread is to raise awareness around some of the implementation level
decisions that were taken when refactoring Puppet modules and splitting
them into their own respective GitHub repos.

* - Decouple Hiera from Puppet modules*
In last Puppet-Modules release Hiera lookups were done in /manifests/init.pp which made it mandatory for users to use Hiera.
As part of the refactoring Hiera is made optional and configuration
parameters are defined as class parameters which can be passed by site.pp


* - Use Hiera only as a data store*
There were some discussions about creating hooks in Puppet manifest to
which Puppet resources can be plugged in via Hiera data. However, going
forward we have come to a conclusion that Hiera should only be used as a
data store and nothing more. Any customizations to Puppet manifest can be
done directly in init.pp.

* - Break Puppet configurations into stages*
We have identified following stages which can be considered completely
independent from each other;
system - set operating system configs, install packages etc.
clean - remove any existing installation
install - extract the pack and set user permissions
configure - populate templates, copy config files, import certs
service - start the service

Here is a snippet of wso2as/manifests/init.pp

contain wso2base
contain wso2base::system
contain wso2base::clean
contain wso2base::install
contain wso2base::configure
contain wso2base::service

Class['::wso2base'] -> Class['::wso2base::system']
-> Class['::wso2base::clean'] -> Class['::wso2base::install']
-> Class['::wso2base::configure'] ~> Class['::wso2base::service']

As you can see Puppet class containment [1] is used to ensure the
dependency ordering. This is useful when users combine WSO2 Puppet modules
with other modules (for eg: when creating roles and profiles).

* - Remove inter module class inheritance*
As per official Puppet docs class inheritance is to be avoided whenever
possible [2]. Considering this we have removed all such relationships
except for params.pp which contains default values.

* - Move default values to params.pp and remove wso2base/params.pp*
After giving much thought, we decided to keep default values in
/manifests/params.pp and only in that place. Having common
default values in /manifests/params.pp would minimize duplication
but would be less readable for end users. Therefore we have come to a
conclusion that wso2base module should only be used as a common
resources/classes pool. It should not contain any data.

*Try it out!*

1. Git clone https://github.com/wso2/puppet-common
2. Run puppet-common/setup.sh -p as
3. Goto puppet-common/vagrant
4. Run vagrant up


[1]
https://docs.puppet.com/puppet/latest/reference/lang_containment.html#containing-classes
[2] https://docs.puppet.com/guides/style_guide.html#class-inheritance

Thanks.

-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev