Jira (PUP-10540) Performance of puppet grows worse as the size of /proc/mounts increases
Title: Message Title
Michele Baldessari updated an issue
Puppet / PUP-10540
Performance of puppet grows worse as the size of /proc/mounts increases
Change By:
Michele Baldessari
*Puppet Version: puppet-5.5.10-5.el8ost**Puppet Server Version: N/A**OS Name/Version: Centos 8*A system that has many containers (~1000) will make /proc/mounts rather large (700-900 kbytes). Running any trivial puppet command will take a very long time there due to the following line:https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/selinux.rb#L210In there we're reading /proc/mounts 1024 bytes at the time, which is clearly suboptimal in this case (but also in most default cases as /proc/mounts is rarely < 1024 bytes these days on most modern linux systems).On such an environment we will observe:[root@database-0 ~]# time puppet apply --noop --detailed-exitcodes --color=false -e 'file {"/tmp/foobar": content => "baz" }'Notice: Compiled catalog for database-0.redhat.local in environment production in 0.16 secondsNotice: /Stage[main]/Main/File[/tmp/foobar]/ensure: current_value absent, should be file (noop)Notice: Class[Main]: Would have triggered 'refresh' from 1 eventsNotice: Stage[main]: Would have triggered 'refresh' from 1 eventsNotice: Applied catalog in 0.05 secondsreal5m43.258suser5m7.929ssys 0m5.603sIncreasing the read size to 1024 ** 2 provides immediate benefits:[root@database-0 ~]# time puppet apply --noop --detailed-exitcodes --color=false -e 'file {"/tmp/foobar": content => "baz" }'Notice: Compiled catalog for database-0.redhat.local in environment production in 0.11 secondsNotice: /Stage[main]/Main/File[/tmp/foobar]/ensure: current_value absent, should be file (noop)Notice: Class[Main]: Would have triggered 'refresh' from 1 eventsNotice: Stage[main]: Would have triggered 'refresh' from 1 eventsNotice: Applied catalog in 0.13 secondsreal1m22.954suser0m50.423ssys 0m5.115sIn such an environment strace confirms that the number of small read() calls goes from:[root@database-0 ~]# grep -ir -e 'read(7.*1024' /tmp/strace-puppet.txt |wc -l231381To:[root@database-0 ~]# grep -ir -e 'read(7.*1024' /tmp/strace-puppet-after.txt |wc -l810
Add Comment
Jira (PUP-10540) Performance of puppet grows worse as the size of /proc/mounts increases
Title: Message Title
Michele Baldessari created an issue
Puppet / PUP-10540
Performance of puppet grows worse as the size of /proc/mounts increases
Issue Type:
Bug
Assignee:
Unassigned
Created:
2020/06/08 9:40 AM
Priority:
Normal
Reporter:
Michele Baldessari
Puppet Version: puppet-5.5.10-5.el8ost Puppet Server Version: N/A OS Name/Version: Centos 8 A system that has many containers (~1000) will make /proc/mounts rather large (700-900 kbytes). Running any trivial puppet command will take a very long time there due to the following line: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/selinux.rb#L210 In there we're reading /proc/mounts 1024 bytes at the time, which is clearly suboptimal in this case (but also in most default cases as /proc/mounts is rarely < 1024 bytes these days on most modern linux systems). On such an environment we will observe: [root@database-0 ~]# time puppet apply --noop --detailed-exitcodes --color=false -e 'file {"/tmp/foobar": content => "baz" } ' Notice: Compiled catalog for database-0.redhat.local in environment production in 0.16 seconds Notice: /Stage[main]/Main/File[/tmp/foobar]/ensure: current_value absent, should be file (noop) Notice: Class[Main]: Would have triggered 'refresh' from 1 events Notice: Stage[main]: Would have triggered 'refresh' from 1 events Notice: Applied catalog in 0.05 seconds real 5m43.258s user 5m7.929s sys 0m5.603s Increasing the read size to 1024 ** 2 provides immediate benefits: [root@database-0 ~]# time puppet apply --noop --detailed-exitcodes --color=false -e 'file {"/tmp/foobar": content => "baz" } ' Notice: Compiled catalog for database-0.redhat.local in environment production in 0.11 seconds Notice: /Stage[main]/Main/File[/tmp/foobar]/ensure: current_value absent, should be file (noop) Notice: Class[Main]: Would have triggered 'refresh' from 1 events Notice: Stage[main]: Would have triggered 'refresh' from 1 events Notice: Applied catalog in 0.13 seconds real 1m22.954s user 0m50.423s sys 0m5.115s In such an environment strace confirms that the number of small read() calls goes from: [root@database-0 ~]# grep -ir -e 'read(7.*1024' /tmp/strace-puppet.txt |wc -l 231381 To: [root@database-0 ~]#
