Jira (PUP-9707) Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins
Title: Message Title Josh Cooper updated an issue Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins Change By: Josh Cooper Epic Link: PUP-11032 Add Comment This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.308771.1557941093000.23135.1645670220034%40Atlassian.JIRA.
Jira (PUP-9707) Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins
Title: Message Title Jorie Tappa updated an issue Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins Change By: Jorie Tappa Team: Coremunity Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.308771.1557941093000.12270.1558370700463%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-9707) Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins
Title: Message Title
Nick Walker updated an issue
Puppet / PUP-9707
Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins
Change By:
Nick Walker
h1. BackgroundIn PE we added the ability for the agent to bulk download plugins before installing the agent via the frictionless installation script. On the master, a tarball of the plugins is created and placed in a directory for jetty to serve so the frictionless agent install script can pull it down. h1. Suggested ImprovementThe instead of only doing this via the installation script we could move the logic into the puppet-agent so that the agent could detect that no plugins are present and instead of doing a one by one download the bulk pluginsync should be performed. h1. Out of scope My suggestion would be that this functionality is added first with an assumption that the tarball will already be present on the master and it can be put there by a separate process other than puppetserver itself making the tarball. Then in the future we can improve puppetserver to make the tarball if we want to. SERVER-2539 h1. ImpactIn order to show the impact of using the current bulk_pluginsync implementation we install the puppetlabs-windows module and then perform pluginsync via puppet or via bulk_pluginsync. When we look at the results we can see a puppet agent run without bulk_pluginsync takes around 90 seconds and with bulk_pluginsync it takes about 14 seconds. The download and extraction of the pluginsync tarball takes under 1 second and a reasonable part of the agent run after bulk_pluginsync seems to be validating the plugins. Puppet-agent ( one file at a time )```/etc/puppetlabs/code/environments/production/modules└─┬ puppetlabs-windows (v6.1.0) ├── puppet-download_file (v3.2.0) ├── puppet-windows_env (v3.2.0) ├── puppet-windowsfeature (v3.2.2) ├── puppetlabs-acl (v2.1.0) ├─┬ puppetlabs-chocolatey (v3.3.0) │ ├── puppetlabs-powershell (v2.3.0) │ ├── puppetlabs-registry (v2.1.0) │ └── puppetlabs-stdlib (v5.2.0) ├─┬ puppetlabs-dsc (v1.9.2) │ └── puppetlabs-reboot (v2.1.2) ├── puppetlabs-iis (v4.5.1) └── puppetlabs-wsus_client (v1.1.0)[root@master201910-centos ~]#[root@master201910-centos ~]#[root@master201910-centos ~]#[root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/facts.d/ lib/ locales/ preview/ puppet-module/ state/[root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/[root@master201910-centos ~]# time puppet agent -tInfo: Using configured environment 'production'Info: Retrieving pluginfactsInfo: Retrieving pluginNotice: /File[/opt/puppetlabs/puppet/cache/lib/facter]/ensure: createdNotice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_build.rb]/ensure: defined content as '{md5}cdcc1ff07bc245c66cc1d46be56b3af5'Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_version.rb]/ensure: defined content as '{md5}d05c8cbf788f47d33efd46a935dda61e'Notice:
Jira (PUP-9707) Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins
Title: Message Title
Nick Walker updated an issue
Puppet / PUP-9707
Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins
Change By:
Nick Walker
h1. BackgroundIn PE we added the ability for the agent to bulk download plugins before installing the agent via the frictionless installation script. On the master, a tarball of the plugins is created and placed in a directory for jetty to serve so the frictionless agent install script can pull it down. h1. Suggested ImprovementThe instead of only doing this via the installation script we could move the logic into the puppet-agent so that the agent could detect that no plugins are present and instead of doing a one by one download the bulk pluginsync should be performed. h1. Out of scope My suggestion would be that this functionality is added first with an assumption that the tarball will already be present on the master and it can be put there by a separate process other than puppetserver itself making the tarball. Then in the future we can improve puppetserver to make the tarball if we want to. SERVER-2539h1. ImpactIn order to show the impact of using the current bulk_pluginsync implementation we install the puppetlabs-windows module and then perform pluginsync via puppet or via bulk_pluginsync. When we look at the results we can see a puppet agent run without bulk_pluginsync takes around 90 seconds and with bulk_pluginsync it takes about 14 seconds. The download and extraction of the pluginsync tarball takes under 1 second and a reasonable part of the agent run after bulk_pluginsync seems to be validating the plugins. Puppet-agent ( one file at a time ) ``` {code} /etc/puppetlabs/code/environments/production/modules└─┬ puppetlabs-windows (v6.1.0) ├── puppet-download_file (v3.2.0) ├── puppet-windows_env (v3.2.0) ├── puppet-windowsfeature (v3.2.2) ├── puppetlabs-acl (v2.1.0) ├─┬ puppetlabs-chocolatey (v3.3.0) │ ├── puppetlabs-powershell (v2.3.0) │ ├── puppetlabs-registry (v2.1.0) │ └── puppetlabs-stdlib (v5.2.0) ├─┬ puppetlabs-dsc (v1.9.2) │ └── puppetlabs-reboot (v2.1.2) ├── puppetlabs-iis (v4.5.1) └── puppetlabs-wsus_client (v1.1.0)[root@master201910-centos ~]#[root@master201910-centos ~]#[root@master201910-centos ~]#[root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/facts.d/ lib/ locales/ preview/ puppet-module/ state/[root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/[root@master201910-centos ~]# time puppet agent -tInfo: Using configured environment 'production'Info: Retrieving pluginfactsInfo: Retrieving pluginNotice: /File[/opt/puppetlabs/puppet/cache/lib/facter]/ensure: createdNotice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_build.rb]/ensure: defined content as '{md5}cdcc1ff07bc245c66cc1d46be56b3af5'Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_version.rb]/ensure: defined content as '{md5}d05c8cbf788f47d33efd46a935dda61e'Notice:
Jira (PUP-9707) Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins
Title: Message Title Nick Walker created an issue Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins Issue Type: Improvement Assignee: Unassigned Created: 2019/05/15 10:24 AM Priority: Normal Reporter: Nick Walker Background In PE we added the ability for the agent to bulk download plugins before installing the agent via the frictionless installation script. On the master, a tarball of the plugins is created and placed in a directory for jetty to serve so the frictionless agent install script can pull it down. Suggested Improvement The instead of only doing this via the installation script we could move the logic into the puppet-agent so that the agent could detect that no plugins are present and instead of doing a one by one download the bulk pluginsync should be performed. Out of scope My suggestion would be that this functionality is added first with an assumption that the tarball will already be present on the master and it can be put there by a separate process other than puppetserver itself making the tarball. Then in the future we can improve puppetserver to make the tarball if we want to. SERVER-2539 Add Comment
