With 16k hosts, it's possible that you never actually see the max 50 forks in use. We've made some pretty big performance improvements to large host counts, but you might only see somewhere around 30 forks in use, maybe less, even though you have specified a higher count.
The forking limitation is caused by having a high fork count, with a CPU (single CPU core) bound process. Effectively, we have a single process that orchestrates forking, and processing results. The more forks, the more CPU bound that process is. The more hosts you have in inventory, the more CPU is used to calculate variables when spawning workers and processing results. Once the first sets of results start coming in, the process becomes busier, and is unable to process results and spawn new processes to keep the fork count up. So what happens is results take longer to process, which delays spawning new workers, and before you reach the theoretical max of your 50 configured forks, other forks are exiting, and the process never gets back to the max. So ultimately, what you have to do is baseline your use case, and adjust forks to a more appropriate level. Decrease forks, or split your inventory into smaller chunks, or do both. On Tue, Nov 1, 2022 at 12:28 PM Mike Eggleston <[email protected]> wrote: > I’ll look at the docs you mention. My command is: ansible all -I list -m > command -m ‘rpm -qa openssl’. Other than the forks setting and the log > setting I’ve not changed the configuration. (forks = 50 and log = > /tmp/${USER}-ansible.log > > Mike > > On Nov 1, 2022, at 12:22, Andrew Latham <[email protected]> wrote: > > > Mike > > There are dedicated docs pages[1][2] for this topic. I must admit there is > a fair bit of missing info in your query. > > So to recap what we know: > * ~16k hosts > * forks set to 50 > * unknown strategy > * unknown config for async, throttle, serial and the other settings that > can affect > * unknown number of running ansible forks/processes on the control node > * unknown if still polling for async > * unknown other things as I am typing this in a hurry > > 1. https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html > 2. > https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html > > On Tue, Nov 1, 2022 at 10:48 AM Mike Eggleston <[email protected]> > wrote: > >> I have my forks set to 50 in ~/.Ansible.CTG and the forks setting seems >> to work for the first set of boxes, but doesn’t seem to work for the rest. >> I thought the forks setting would keep $forks connections going until all >> boxes in the inventory have been touched. >> >> Was I thinking wrong? (I’m running an ad hoc command on 16k boxes.) >> >> Mike >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ansible Project" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/ansible-project/1EB21F73-144F-4F43-A696-0BA08D4CB2BA%40gmail.com >> . >> > > > -- > - Andrew "lathama" Latham - > > -- > You received this message because you are subscribed to the Google Groups > "Ansible Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ansible-project/CA%2Bqj4S-7xuwUmb02hjO2WbiVqr7PBbZgsoYm7tSMaheBxrO7wQ%40mail.gmail.com > <https://groups.google.com/d/msgid/ansible-project/CA%2Bqj4S-7xuwUmb02hjO2WbiVqr7PBbZgsoYm7tSMaheBxrO7wQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "Ansible Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ansible-project/4D5A0D53-A4E5-450B-938F-BEA5DC69C3FF%40gmail.com > <https://groups.google.com/d/msgid/ansible-project/4D5A0D53-A4E5-450B-938F-BEA5DC69C3FF%40gmail.com?utm_medium=email&utm_source=footer> > . > -- Matt Martz @sivel sivel.net -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAD8N0v-ZA8brU%3DCrVr7zp-NrimvhSw3RchZkznb6uZkiHSzESw%40mail.gmail.com.
