Issue #1469 has been updated by luke.
Subject changed from Recursive file resource with lots of local files is
horribly slow. to File recursion with a remote source should not recurse locally
Priority changed from Normal to High
I guess we need to change the behaviour here, but we need to provide backward
compatibility, probably through some kind of recursion selection attribute:
<pre>
file { "/etc/ssh":
source => "...",
recurse => true,
recurseselect => <remote|both>
}
</pre>
I don't like that attribute name, but we need something, and at least for now,
it should default to the current behaviour (probably with a deprecation
warning).
----------------------------------------
Bug #1469: File recursion with a remote source should not recurse locally
http://projects.reductivelabs.com/issues/show/1469
Author: masterzen
Status: Accepted
Priority: High
Assigned to: luke
Category: file
Target version:
Complexity: Unknown
Patch: None
Affected version: 0.25.0
Keywords: file recursive slow
Hi,
I have the following manifests:
<pre>
file {
"${homeroot}/$name":
source => [ "puppet:///files/users/$name/skel/",
"puppet:///files/users/skel/"],
sourceselect => all,
checksum => mtime,
recurse => true, owner => $name, group => $name, replace => no,
require => [ User["$name"], Group["$name"] ];
}
</pre>
that is used to copy various files to the home directory of a new user.
If the home directory of this user contains a deep hierarchy full of files,
puppet takes ages and consume a lot of memory to finally copy over a handful
of files.
My understanding of the issue is the following:
* Puppet builds localrecurse first (see puppet/type/file.rb). This parts can
take a long time, each local present file has to be stat'ed and a new
(implicit) resource is created in the catalog.
* then Puppet builds the sourcerecurse
If Puppet would build the sourcerecurse first, and prune the localrecurse in
function of the availability of the sourced file/dir it would be way much
faster. Moreover I don't think people are using file{} to distribute more than
a few files, but I bet they like me have more local files than distributed.
Also note, that there is another issue if filebucket and checksum are enabled
in the aforementioned manifest: it will copy over all files, and filebucket the
previous one, as seen in this log extract:
<pre>
debug: Calling fileserver.list
debug: Calling fileserver.describe
debug: Calling fileserver.retrieve
debug: //all_users/Account[brice]/File[/home/brice/.zshrc]/source: Executing
'diff -u /home/brice/.zshrc /tmp/user/0/puppet-diffing20080704-15708-glfegc-0'
debug: //all_users/Account[brice]/File[/home/brice/.zshrc]: Changing
checksum,source
debug: //all_users/Account[brice]/File[/home/brice/.zshrc]: 2 change(s)
debug: //all_users/Account[brice]/File[/home/brice/.zshrc]/checksum: Replacing
/home/brice/.zshrc checksum {md5}1395a1f669e2c86443b142cc5ad10f55 with
{mtime}Fri Jul 04 15:17:39 +0200 2008
notice: //all_users/Account[brice]/File[/home/brice/.zshrc]/checksum: checksum
changed '{md5}1395a1f669e2c86443b142cc5ad10f55' to '{mtime}Fri Jul 04 15:17:39
+0200 2008'
debug: Calling puppetbucket.addfile
info: //all_users/Account[brice]/File[/home/brice/.zshrc]: Filebucketed to
server with sum 1395a1f669e2c86443b142cc5ad10f55
debug: //all_users/Account[brice]/File[/home/brice/.zshrc]/checksum: Replacing
/home/brice/.zshrc checksum {mtime}Fri Jul 04 15:17:39 +0200 2008 with
{md5}1395a1f669e2c86443b142cc5ad10f55
notice: //all_users/Account[brice]/File[/home/brice/.zshrc]/source: replacing
from source puppet:///files/users/brice/skel/.zshrc with contents
{md5}1395a1f669e2c86443b142cc5ad10f55
</pre>
This is because default file checksum is md5, but my file resource above said
"mtime". It then compares apples to oranges.
----------------------------------------
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://reductivelabs.com/redmine/my/account
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en
-~----------~----~----~----~------~----~------~--~---
