Thank you for the all the responses so far. I had a look at mitogen, but in our setup almost all modules are run on the controller (templates, lookups etc), in fact we hardly use SSH at all (as targets are usually set up using XML-RPC or REST APIs), so I'm not sure how much it's going to help here. I'll run some evaluations later today.
I also looked at running multiple instances of ansible-playbook (for example one per target device). The biggest challenge is that for us - if the deployment fails on one device - we have to stop deployment on all others and roll them back (using a separate role). We're deploying configuration (for example for a L3VPN) across a number of devices. If the deployment fails on one of them - we have roll back the lot. That's relatively easy to do with only ansible-playbook running, but I couldn't make this work reliably with multiple ones. That's also the reason 'async' is not going to work for us. I think for now we'll have to optimise the playbooks further. It looks like the following areas take the most time: - include_* statements - single template operations (so consolidate the templates, for now a single device config can come from many templates that first get populated, then consolidated) - loops - reduce the number, particularly with include inside them Any further suggestions are appreciated. kind regards Pshem On Wed, 8 Aug 2018 at 03:26 Andrew Latham <lath...@gmail.com> wrote: > Just in the case you have not found this resource try > https://docs.ansible.com/ansible/2.5/user_guide/playbooks_async.html > > On Mon, Aug 6, 2018 at 6:46 PM Pshem Kowalczyk <pshe...@gmail.com> wrote: > >> Hi, >> >> We have a scenario where we pre-generate a lot of configuration on the >> controller host (targets are generally network devices, so not capable of >> running python). >> Typical generation process involves pulling data from a number odelef >> different systems (over APIs), running some local modules - mostly >> 'template'. A typical run for a single devices can have up to 500-700 >> tasks. >> >> A single device can be done under 5 minutes in most cases (including >> deployment). All the generation already runs with "strategy: free". Once we >> go to bigger deployments - 10-15 devices the time that's need gets >> significantly longer (40-50 minutes is not unusual). We tried throwing CPUs >> at the problem, but it looks like only one-two CPUs get ever to 100% whilst >> the rest of them seems to be near-idle (regardless of "fork" values). >> There's plenty of RAM too (utilisation hardly ever goes over 3-4GB). >> >> Is there a way for Ansible to utilise all the CPUs? I realise this might >> not be a typical case, but we're looking now at deployments that have 30-40 >> devices and waiting 3h for completion is not something we'd want to see. >> >> kind regards >> Pshem >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ansible Project" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to ansible-project+unsubscr...@googlegroups.com. >> To post to this group, send email to ansible-project@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/ansible-project/CAEaZiRV1pLwu-F-%3D4uzx6_zOy78HaXP-XC2G634c1s66t2x3rw%40mail.gmail.com >> <https://groups.google.com/d/msgid/ansible-project/CAEaZiRV1pLwu-F-%3D4uzx6_zOy78HaXP-XC2G634c1s66t2x3rw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . > > >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > - Andrew "lathama" Latham - > > -- > You received this message because you are subscribed to the Google Groups > "Ansible Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ansible-project+unsubscr...@googlegroups.com. > To post to this group, send email to ansible-project@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ansible-project/CA%2Bqj4S8SuZMSuNbrM9EvsY%2B4e-hZL3OOg64OsoDCKaumhsXvWw%40mail.gmail.com > <https://groups.google.com/d/msgid/ansible-project/CA%2Bqj4S8SuZMSuNbrM9EvsY%2B4e-hZL3OOg64OsoDCKaumhsXvWw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscr...@googlegroups.com. To post to this group, send email to ansible-project@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEaZiRXV5BFYe1y0vJ3m6ZZMhp%2BO-GVSFYeHxgiZdnYMdRL84g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
