Issue #3362 has been updated by James Turnbull. Status changed from In Topic Branch Pending Merge to Needs Decision Assignee changed from Markus Roberts to Nigel Kersten
This probably needs review and comment from you Nigel. ---------------------------------------- Bug #3362: splay drift occurs when passenger/mongrel get too much load. https://projects.puppetlabs.com/issues/3362 Author: Dan Bode Status: Needs Decision Priority: Normal Assignee: Nigel Kersten Category: plumbing Target version: 2.7.x Affected Puppet version: 0.25.4 Keywords: passenger load splay mongrel connection timeouts Branch: http://github.com/MarkusQ/puppet/tree/ticket/0.25.x/3362 not sure if this counts as a bug... I could not concretely prove the assumptions below. I did some investigation and this is my best guess as to the cause. Splay was drifting for hundreds of machines so that over time, most were checking in at the same time, while at other times none were checking in. here is my theory as to why. splay only runs the first time after puppet starts. Assumption: runinterval starts counting only after the client finishes its last run? Here is the chain of events that I think causes this: 1. passenger or mongrel is under heavy load. 2. processes get used up, they start queuing hosts. 3. Once a machine falls into the queue, it gets stuck with the group of machines that cause the queue to fill up, since it will now use runinterval and check in at the same time as the other machines that were running at that same time. 4. Over time, splay drifts so that most machines are checking in at the same time. Basically, once performance starts getting bad, the splaying falls apart so that it gets much worse. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
