Issue #17230 has been updated by eric sorenson. Status changed from Unreviewed to Needs Decision Assignee set to Andrew Parker
Hi Josh, that is indeed a good optimization. It is what puppet does internally for dependencies between groups of contained resources (like Class[a] -> Class[b]) but I don't know that it makes sense to try to detect the collection case and handle it specially. Your workaround seems like a fine option that doesn't require core puppet changes. Passing to lead developer for a second opinion, but I'm inclined to say unless someone submits a patch that implements this it's not something we'd spend time on implementing. ---------------------------------------- Bug #17230: Chaining resource collections is unnecessarily slow https://projects.puppetlabs.com/issues/17230#change-77842 Author: Josh Shabtai Status: Needs Decision Priority: Normal Assignee: Andrew Parker Category: Target version: Affected Puppet version: Keywords: Branch: Hey, Puppet master version: 2.7.18 Puppet agent version: 2.7.1 It seems that when chaining resource collections (e.g. <|collection1|> -> <|collection2|>), puppet creates dependencies between each resource in each side. If, for example, you have 100 of each resource, this creates 10,000 dependencies and seems to take a long time to handle. We have a situation similar to this (though I don't know exactly how many resources we actually have). I added a dummy resource in between each pair of collections though, which sped things up significantly (e.g. <|collection1|> -> Exec["dummy"] -> <|collection2|>). I would suggest that you do something similar internally. Here's the diff that got my runtime from over an hour to about half that: + # Dummy resources for chaining + # The idea is that when puppet chains collections of sizes x and y, you get + # x*y requirement. By adding a dummy resource in the middle, we should be + # able to get it down to x+y, which will drastically speed things up. + exec { "dummy-domain-subdomain": command => "/bin/true" } + exec { "dummy-mailbox-set-status": command => "/bin/true" } + exec { "dummy-set-status-child-mailbox": command => "/bin/true" } + exec { "dummy-child-mailbox-set-cos": command => "/bin/true" } # Resource chaining - Exec["cache domains"] -> Exec<| tag == "domain" |> -> Exec<| tag == "subdomain" |> -> Exec["cache cos"] -> Exec<| tag == "cos" |> -> Exec["cache mailboxes"] -> Exec<| tag == "mailbox" |> -> Exec<| tag == "set-status" |> -> Exec<| tag == "child-mailbox" |> -> Exec<| tag == "set-cos" |> + Exec["cache domains"] -> Exec<| tag == "domain" |> -> Exec["dummy-domain-subdomain"] -> Exec<| tag == "subdomain" |> -> Exec["cache cos"] -> Exec<| tag == "cos" |> -> Exec["cache mailboxes"] -> Exec<| tag == "mailbox" |> -> Exec["dummy-mailbox-set-status"] -> Exec<| tag == "set-status" |> -> Exec["dummy-set-status-child-mailbox"] -> Exec<| tag == "child-mailbox" |> -> Exec["dummy-child-mailbox-set-cos"] -> Exec<| tag == "set-cos" |>` -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
