Issue #17230 has been updated by Andrew Parker. Status changed from Needs Decision to Closed
As Eric says, I don't see a point in the near future where we'll try to tackle this. I find it more likely we'll tackle this by fixing up the algorithm we use for applying catalogs. The proposed solution is, as Eric says, very similar to what puppet already does internally for other cases and, although it works, the creation of dummy resources has caused issues in other parts of the system (these are the "whits" that show up sometimes). So I don't see us trying to specifically fix this case. Thanks for the report and glad that you found a way around the problem. ---------------------------------------- Bug #17230: Chaining resource collections is unnecessarily slow https://projects.puppetlabs.com/issues/17230#change-77901 Author: Josh Shabtai Status: Closed Priority: Normal Assignee: Andrew Parker Category: Target version: Affected Puppet version: Keywords: Branch: Hey, Puppet master version: 2.7.18 Puppet agent version: 2.7.1 It seems that when chaining resource collections (e.g. <|collection1|> -> <|collection2|>), puppet creates dependencies between each resource in each side. If, for example, you have 100 of each resource, this creates 10,000 dependencies and seems to take a long time to handle. We have a situation similar to this (though I don't know exactly how many resources we actually have). I added a dummy resource in between each pair of collections though, which sped things up significantly (e.g. <|collection1|> -> Exec["dummy"] -> <|collection2|>). I would suggest that you do something similar internally. Here's the diff that got my runtime from over an hour to about half that: + # Dummy resources for chaining + # The idea is that when puppet chains collections of sizes x and y, you get + # x*y requirement. By adding a dummy resource in the middle, we should be + # able to get it down to x+y, which will drastically speed things up. + exec { "dummy-domain-subdomain": command => "/bin/true" } + exec { "dummy-mailbox-set-status": command => "/bin/true" } + exec { "dummy-set-status-child-mailbox": command => "/bin/true" } + exec { "dummy-child-mailbox-set-cos": command => "/bin/true" } # Resource chaining - Exec["cache domains"] -> Exec<| tag == "domain" |> -> Exec<| tag == "subdomain" |> -> Exec["cache cos"] -> Exec<| tag == "cos" |> -> Exec["cache mailboxes"] -> Exec<| tag == "mailbox" |> -> Exec<| tag == "set-status" |> -> Exec<| tag == "child-mailbox" |> -> Exec<| tag == "set-cos" |> + Exec["cache domains"] -> Exec<| tag == "domain" |> -> Exec["dummy-domain-subdomain"] -> Exec<| tag == "subdomain" |> -> Exec["cache cos"] -> Exec<| tag == "cos" |> -> Exec["cache mailboxes"] -> Exec<| tag == "mailbox" |> -> Exec["dummy-mailbox-set-status"] -> Exec<| tag == "set-status" |> -> Exec["dummy-set-status-child-mailbox"] -> Exec<| tag == "child-mailbox" |> -> Exec["dummy-child-mailbox-set-cos"] -> Exec<| tag == "set-cos" |>` -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to puppet-bugs@googlegroups.com. To unsubscribe from this group, send email to puppet-bugs+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
