Re: [ansible-project] Multi threading support in Ansible

2020-06-11 Thread Matt Martz
You have 1 worker process. One ansible-playbook process is the control process, the other is the worker. On Thu, Jun 11, 2020 at 9:43 AM Jagadeeshkumar Dittakavi < d.jagadeeshku...@gmail.com> wrote: > Thank you Matt! > In the above example I have explicitly passed --forks=1 but still there > are

Re: [ansible-project] Multi threading support in Ansible

2020-06-11 Thread Jagadeeshkumar Dittakavi
Thank you Matt! In the above example I have explicitly passed --forks=1 but still there are 2 worker processes(PIDs 69484 and 69520) were spawned, that means there will be minimum 2 workers get spawned and we can't limit that to one? I understand that there is no control to limit the total

Re: [ansible-project] Multi threading support in Ansible

2020-06-11 Thread Matt Martz
There are a number of steps involved here. 1. The primary playbook process spawns a worker 2. The worker executes the async_wrapper for the command module 3. The async_wrapper forks to daemonize 4. The async_wrapper executes the transferred module 5. The actual module is contained within what we

Re: [ansible-project] Multi threading support in Ansible

2020-06-11 Thread Jagadeeshkumar Dittakavi
@Matt, Got another question in concurrency support in Ansible. Is there any way to limit the number of processes that could be spawned on a given host? My requirement is not to execute the commands/scripts remotely. In my case, the whole play needs to be executed on locahost only. I have tried

Re: [ansible-project] Multi threading support in Ansible

2020-06-04 Thread Jagadeeshkumar Dittakavi
Thank you Matt for the detailed and quick reply.. Much appreciated the support from the community. On Friday, June 5, 2020 at 12:29:30 AM UTC+5:30, Matt Martz wrote: > > Yes, it would utilize the threading library in Python. The GIL is a > primary cause to the CPU restrictions. Our main

Re: [ansible-project] Multi threading support in Ansible

2020-06-04 Thread Matt Martz
Yes, it would utilize the threading library in Python. The GIL is a primary cause to the CPU restrictions. Our main process that orchestrates all of the task executions is already heavily CPU bound, so adding additional threads to the same core can cause a decrease in performance. Assuming we

Re: [ansible-project] Multi threading support in Ansible

2020-06-04 Thread Jagadeeshkumar Dittakavi
Thank you for the prompt reply.. Just a curious question: Is the threading work that is underway based on python threads or pthreads or any other threading mechanism? As you mentioned that the threading model is not going to be performant, was the reason being the python's GIL? On Friday,

Re: [ansible-project] Multi threading support in Ansible

2020-06-04 Thread Matt Martz
The only current process model is forking. There has been some work done to add a threaded process model, but there are some large hurdles to overcome. In practice, it is not necessarily more performant, and in many cases it was less performant, as it causes more CPU contention on a single core