Jira (PUP-11163) Prevent thundering herd via deterministic run scheduling

[Reid Vandewiele (Jira)]

Title: Message Title

Reid Vandewiele updated an issue   Puppet / PUP-11163 Prevent thundering herd via deterministic run scheduling Change By: Reid Vandewiele h2. Problem One of the most-viewed articles in Open Source Puppet Assist between September of 2020 and March of 2021 was [determine a thundering herd condition in Puppet|https://ospassist.puppet.com/hc/en-us/articles/360040327753-determine-a-thundering-herd-condition-in-puppet-] (OSPA-21). Thundering herds have been an out-of-box problem in Puppet for more than a decade, and for users "in the know", there is a clear, bulletproof solution: use splayed, deterministic run start times to evenly distribute agent checkins. Problematically, this doesn't work with our daemonized agent. Customers must instead disable the agent, and use OS schedulers such as Cron (Posix) and Scheduled Tasks (Windows). See [this module|https://forge.puppet.com/modules/reidmv/puppet_run_scheduler] for full automation of the configuration. h2. Desired Solution The Puppet agent should provide a deterministic run scheduling mode which evenly distributes agent runs automatically, and eliminates the possibility of thundering herds developing. This mode should become the new default mode. Characteristics: * Scheduled run start times should be based on computed wall clock times. Not on variable-time events such as previous run end times. * An agent's individual run start time should include a random offset element, seeded on the agent's certname setting * The agent should schedule and perform a first-start run quickly (immediately, or splayed if splay is configured) on daemon startup, after which subsequent run starts should adhere to the computed wall-clock schedule On startup, an agent might schedule runs thusly. # Schedule a startup run for {{now() + $splay}} # Schedule the first regular daemon run for {{now() + ((($run_interval - (now() % $run_interval)) + fqdn_rand($certname)) % $run_interval)}} # Every subsequent run should be scheduled exactly {{$run_interval}} after of the previously scheduled run * Optionally, a provision can be made to skip a scheduled run if the previous scheduled run completed too soon within a configured proximity. This may happen after the special startup run, for example, or if a scheduled run entered into a backoff/retry loop based on server 503 responses and so did not start/complete for much longer than expected     We advise our largest customers to achieve this scheduling algorithm today in order to better distribute load, and to prevent thundering herds. We should eliminate the onerous workaround by incorporating it into our product, and in so doing deliver the same benefits to all Puppet users, as well as eliminate one of our most common troubleshooting exasperations. Add Comment   This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.406175.1625677218000.76190.1625682420042%40Atlassian.JIRA.