Jira (PUP-3866) File checksum should not be triggered when 'replace => false'

[Josh Cooper (JIRA)]

Title: Message Title

Josh Cooper commented on PUP-3866   Re: File checksum should not be triggered when 'replace => false' When replace is false and checksum is mtime, then puppet will open and read big_file_01 3 times, once for each copy operation to 02, 03, 04. It appears puppet is reading the entire file contents into memory. This is due to the file serving code [root@pnfo90tpwvcp5v9 ~]# cat manifest.pp exec { '/usr/bin/dd if=/dev/urandom of=/tmp/big_file_01 bs=1MB count=100': creates => '/tmp/big_file_01', } ->   file { [ '/tmp/big_file_02', '/tmp/big_file_03', '/tmp/big_file_04' ]: ensure => file, replace => false, checksum => 'mtime', source => '/tmp/big_file_01', owner => 'root', group => 'root', mode => '0644', } [root@pnfo90tpwvcp5v9 ~]# strace -e open,read -P /tmp/big_file_01 -P /tmp/big_file_02 -P /tmp/big_file_03 -P /tmp/big_file_04 puppet apply manifest.pp 2> strace.log Notice: Compiled catalog for pnfo90tpwvcp5v9.delivery.puppetlabs.net in environment production in 0.19 seconds Notice: /Stage[main]/Main/Exec[/usr/bin/dd if=/dev/urandom of=/tmp/big_file_01 bs=1MB count=100]/returns: executed successfully Notice: /Stage[main]/Main/File[/tmp/big_file_02]/ensure: defined content as '{mtime}2017-08-03 07:38:29 +0000' Notice: /Stage[main]/Main/File[/tmp/big_file_03]/ensure: defined content as '{mtime}2017-08-03 07:38:29 +0000' Notice: /Stage[main]/Main/File[/tmp/big_file_04]/ensure: defined content as '{mtime}2017-08-03 07:38:29 +0000' Notice: Applied catalog in 9.35 seconds [root@pnfo90tpwvcp5v9 ~]# grep -e open -e read strace.log open("/tmp/big_file_01", O_RDONLY|O_CLOEXEC) = 8 read(8, "\324\367\213\213\16\332\320\302\257\227.\274\236\3551\304^.|\223.\365V\0r)\2310\344\f\v\342"..., 100000000) = 100000000 read(8, "", 8192) = 0 open("/tmp/big_file_01", O_RDONLY|O_CLOEXEC) = 8 read(8, "\324\367\213\213\16\332\320\302\257\227.\274\236\3551\304^.|\223.\365V\0r)\2310\344\f\v\342"..., 100000000) = 100000000 read(8, "", 8192) = 0 open("/tmp/big_file_01", O_RDONLY|O_CLOEXEC) = 8 read(8, "\324\367\213\213\16\332\320\302\257\227.\274\236\3551\304^.|\223.\365V\0r)\2310\344\f\v\342"..., 100000000) = 100000000 read(8, "", 8192) When the default checksum is used (md5), then puppet will open big_file_01 6 times. The first three times are to see if the file is insync?. The other three times are to copy the file as above. For example: open("/tmp/big_file_01", O_RDONLY|O_CLOEXEC) = 7 read(7, "j.z\177T\17\307\221\327$W2\267\334\250?2\350\207\337\340\34m\357\20Uh\ 345\362B\223\25"..., 4096) = 4096 ... read(7, "\17\365]\244g$\313\17\304\3\270\276\375\336\251\376\304\24+\365\32\321 $B#\216\26\206\213\335\233\333"..., 4096) = 4096 read(7, "\257,\22N\27w+a,3\273\27*h\206ZV\341^\0019\217\355z\261\317\340\334\24 0?[\314"..., 4096) = 256 read(7, "", 3840) = 0 read(7, "", 4096) = 0 open("/tmp/big_file_01", O_RDONLY|O_CLOEXEC) = 8 read(8, "j.z\177T\17\307\221\327$W2\267\334\250?2\350\207\337\340\34m\357\20Uh\345\362B\223\25"..., 100000000) = 100000000 read(8, "", 8192) = 0 The default buffer size when checksumming the file is 4k: def checksum_file(digest, filename, lite = false) buffer = lite ? 512 : 4096 If replace is false, then we should only create the destination file, but never update it. And in that case, we shouldn't need to checksum the source file (since the insync check is solely based on the presence/absence of the destination file). Thanks for reporting Marc Villacorta. Add Comment   This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.