Re: [ansible-project] Multi threading support in Ansible

[Matt Martz]

You have 1 worker process. One ansible-playbook process is the control
process, the other is the worker.

On Thu, Jun 11, 2020 at 9:43 AM Jagadeeshkumar Dittakavi <
d.jagadeeshku...@gmail.com> wrote:

> Thank you Matt!
> In the above example I have explicitly passed --forks=1 but still there
> are 2 worker processes(PIDs 69484 and 69520) were spawned, that means
> there will be minimum 2 workers get spawned and we can't limit that to one?
> I understand that there is no control to limit the total number of
> processes will be spawned by the workers.
>
> On Thursday, June 11, 2020 at 7:54:01 PM UTC+5:30, Matt Martz wrote:
>>
>> There are a number of steps involved here.
>>
>> 1. The primary playbook process spawns a worker
>> 2. The worker executes the async_wrapper for the command module
>> 3. The async_wrapper forks to daemonize
>> 4. The async_wrapper executes the transferred module
>> 5. The actual module is contained within what we call AnsiballZ which is
>> a compressed archive, and it extracts and executes the actual python code
>> 6. Actual module executing.
>>
>> `forks` only limits how many workers can be launched by the primary
>> playbook process, not how many processes will be spawned as a result of the
>> worker.
>>
>> On Thu, Jun 11, 2020 at 7:01 AM Jagadeeshkumar Dittakavi <
>> d.jagad...@gmail.com> wrote:
>>
>>> @Matt,
>>>
>>> Got another question in concurrency support in Ansible.
>>> Is there any way to limit the number of processes that could be spawned
>>> on a given host?
>>> My requirement is not to execute the commands/scripts remotely. In my
>>> case, the whole play needs to be executed on locahost only.
>>> I have tried a simple test program and noticed that there are as many as
>>> 6 processes are spawned to execute 'sleep 20' asynchronously.
>>>
>>> Please kindly revert. Thank you inadvance.
>>>
>>> *Command:* ansible-playbook test_playbook.yml --forks=1
>>>
>>>
>>>
>>> *Processes:*
>>>
>>> root  69484  34309  9 04:50 pts/10   00:00:00 /usr/bin/python2
>>> /usr/bin/*ansible*-playbook test_playbook.yml --forks=1
>>>
>>> root  69509  1  0 04:50 ?00:00:00 /usr/bin/python2
>>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py
>>> 198806654079 50 
>>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py
>>> _
>>>
>>> root  69510  69509  0 04:50 ?00:00:00 /usr/bin/python2
>>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py
>>> 198806654079 50 
>>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py
>>> _
>>>
>>> root  69511  69510  0 04:50 ?00:00:00 /usr/bin/python2
>>> /root/.*ansible*/tmp/*ansible*
>>> -tmp-1591876209.82-38354017880191/command.py
>>>
>>> root  69512  69511  1 04:50 ?00:00:00 /usr/bin/python2 /tmp/
>>> *ansible*_f9ckPD/*ansible*_module_command.py
>>>
>>> root  69520  69484  3 04:50 pts/10   00:00:00 /usr/bin/python2
>>> /usr/bin/*ansible*-playbook test_playbook.yml --forks=1
>>>
>>>
>>>
>>>
>>>
>>> *Code:*
>>>
>>> [root@oracle-siha file_copy_test]# cat test_playbook.yml
>>>
>>> - name: Testing processes
>>>
>>>   gather_facts: no
>>>
>>>   hosts: localhost
>>>
>>>   tasks:
>>>
>>> - name: run sleep command
>>>
>>>   async: 50
>>>
>>>   poll: 0
>>>
>>>   command: sleep 20
>>>
>>>   register: res
>>>
>>> - name: wait for the completion
>>>
>>>   async_status:
>>>
>>> jid: "{{ res.ansible_job_id }}"
>>>
>>>   register: output
>>>
>>>   until: output.finished
>>>
>>>   delay: 5
>>>
>>>   retries: 10
>>>
>>>
>>>
>>> On Friday, June 5, 2020 at 9:50:19 AM UTC+5:30, Jagadeeshkumar Dittakavi
>>> wrote:

 Thank you Matt for the detailed and quick reply.. Much appreciated the
 support from the community.

 On Friday, June 5, 2020 at 12:29:30 AM UTC+5:30, Matt Martz wrote:
>
> Yes, it would utilize the threading library in Python.  The GIL is a
> primary cause to the CPU restrictions.  Our main process that orchestrates
> all of the task executions is already heavily CPU bound, so adding
> additional threads to the same core can cause a decrease in performance.
> Assuming we create a process model plugin type, other process models are
> possible, such as using asyncio, concurrent.futures, gevent, etc.  But I
> don't expect this work to be complete any time soon.
>
> So for now, consider forking the only process model for the near
> future.
>
> On Thu, Jun 4, 2020 at 1:51 PM Jagadeeshkumar Dittakavi <
> d.jagad...@gmail.com> wrote:
>
>> Thank you for the prompt reply.. Just a curious question: Is the
>> threading work that is underway based on python threads or pthreads or 
>> any
>> other threading mechanism? As you mentioned that the threading model is 
>> not
>> going to be performant, was the reason being the python's GIL?
>>
>>
>> On Friday, 

Re: [ansible-project] Multi threading support in Ansible

[Jagadeeshkumar Dittakavi]

Thank you Matt! 
In the above example I have explicitly passed --forks=1 but still there are 
2 worker processes(PIDs 69484 and 69520) were spawned, that means there 
will be minimum 2 workers get spawned and we can't limit that to one? I 
understand that there is no control to limit the total number of processes 
will be spawned by the workers.

On Thursday, June 11, 2020 at 7:54:01 PM UTC+5:30, Matt Martz wrote:
>
> There are a number of steps involved here.
>
> 1. The primary playbook process spawns a worker
> 2. The worker executes the async_wrapper for the command module
> 3. The async_wrapper forks to daemonize
> 4. The async_wrapper executes the transferred module
> 5. The actual module is contained within what we call AnsiballZ which is a 
> compressed archive, and it extracts and executes the actual python code
> 6. Actual module executing.
>
> `forks` only limits how many workers can be launched by the primary 
> playbook process, not how many processes will be spawned as a result of the 
> worker.
>
> On Thu, Jun 11, 2020 at 7:01 AM Jagadeeshkumar Dittakavi <
> d.jagad...@gmail.com > wrote:
>
>> @Matt, 
>>
>> Got another question in concurrency support in Ansible.
>> Is there any way to limit the number of processes that could be spawned 
>> on a given host?
>> My requirement is not to execute the commands/scripts remotely. In my 
>> case, the whole play needs to be executed on locahost only.
>> I have tried a simple test program and noticed that there are as many as 
>> 6 processes are spawned to execute 'sleep 20' asynchronously. 
>>
>> Please kindly revert. Thank you inadvance.
>>
>> *Command:* ansible-playbook test_playbook.yml --forks=1
>>
>>  
>>
>> *Processes:*
>>
>> root  69484  34309  9 04:50 pts/10   00:00:00 /usr/bin/python2 
>> /usr/bin/*ansible*-playbook test_playbook.yml --forks=1
>>
>> root  69509  1  0 04:50 ?00:00:00 /usr/bin/python2 
>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py
>>  
>> 198806654079 50 
>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py 
>> _
>>
>> root  69510  69509  0 04:50 ?00:00:00 /usr/bin/python2 
>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py
>>  
>> 198806654079 50 
>> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py 
>> _
>>
>> root  69511  69510  0 04:50 ?00:00:00 /usr/bin/python2 
>> /root/.*ansible*/tmp/*ansible*
>> -tmp-1591876209.82-38354017880191/command.py
>>
>> root  69512  69511  1 04:50 ?00:00:00 /usr/bin/python2 /tmp/
>> *ansible*_f9ckPD/*ansible*_module_command.py
>>
>> root  69520  69484  3 04:50 pts/10   00:00:00 /usr/bin/python2 
>> /usr/bin/*ansible*-playbook test_playbook.yml --forks=1
>>
>>  
>>
>>  
>>
>> *Code:*
>>
>> [root@oracle-siha file_copy_test]# cat test_playbook.yml 
>>
>> - name: Testing processes
>>
>>   gather_facts: no
>>
>>   hosts: localhost
>>
>>   tasks:
>>
>> - name: run sleep command
>>
>>   async: 50
>>
>>   poll: 0
>>
>>   command: sleep 20
>>
>>   register: res
>>
>> - name: wait for the completion
>>
>>   async_status:
>>
>> jid: "{{ res.ansible_job_id }}"
>>
>>   register: output
>>
>>   until: output.finished
>>
>>   delay: 5
>>
>>   retries: 10
>>
>>  
>>
>> On Friday, June 5, 2020 at 9:50:19 AM UTC+5:30, Jagadeeshkumar Dittakavi 
>> wrote:
>>>
>>> Thank you Matt for the detailed and quick reply.. Much appreciated the 
>>> support from the community.
>>>
>>> On Friday, June 5, 2020 at 12:29:30 AM UTC+5:30, Matt Martz wrote:

 Yes, it would utilize the threading library in Python.  The GIL is a 
 primary cause to the CPU restrictions.  Our main process that orchestrates 
 all of the task executions is already heavily CPU bound, so adding 
 additional threads to the same core can cause a decrease in performance.  
 Assuming we create a process model plugin type, other process models are 
 possible, such as using asyncio, concurrent.futures, gevent, etc.  But I 
 don't expect this work to be complete any time soon.

 So for now, consider forking the only process model for the near future.

 On Thu, Jun 4, 2020 at 1:51 PM Jagadeeshkumar Dittakavi <
 d.jagad...@gmail.com> wrote:

> Thank you for the prompt reply.. Just a curious question: Is the 
> threading work that is underway based on python threads or pthreads or 
> any 
> other threading mechanism? As you mentioned that the threading model is 
> not 
> going to be performant, was the reason being the python's GIL?
>
>
> On Friday, June 5, 2020 at 12:01:14 AM UTC+5:30, Matt Martz wrote:
>>
>> The only current process model is forking.  There has been some work 
>> done to add a threaded process model, but there are some large hurdles 
>> to 
>> overcome.
>>
>> In practice, it is not necessarily more 

Re: [ansible-project] Multi threading support in Ansible

[Matt Martz]

There are a number of steps involved here.

1. The primary playbook process spawns a worker
2. The worker executes the async_wrapper for the command module
3. The async_wrapper forks to daemonize
4. The async_wrapper executes the transferred module
5. The actual module is contained within what we call AnsiballZ which is a
compressed archive, and it extracts and executes the actual python code
6. Actual module executing.

`forks` only limits how many workers can be launched by the primary
playbook process, not how many processes will be spawned as a result of the
worker.

On Thu, Jun 11, 2020 at 7:01 AM Jagadeeshkumar Dittakavi <
d.jagadeeshku...@gmail.com> wrote:

> @Matt,
>
> Got another question in concurrency support in Ansible.
> Is there any way to limit the number of processes that could be spawned on
> a given host?
> My requirement is not to execute the commands/scripts remotely. In my
> case, the whole play needs to be executed on locahost only.
> I have tried a simple test program and noticed that there are as many as 6
> processes are spawned to execute 'sleep 20' asynchronously.
>
> Please kindly revert. Thank you inadvance.
>
> *Command:* ansible-playbook test_playbook.yml --forks=1
>
>
>
> *Processes:*
>
> root  69484  34309  9 04:50 pts/10   00:00:00 /usr/bin/python2
> /usr/bin/*ansible*-playbook test_playbook.yml --forks=1
>
> root  69509  1  0 04:50 ?00:00:00 /usr/bin/python2 /root/.
> *ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py
> 198806654079 50 
> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py
> _
>
> root  69510  69509  0 04:50 ?00:00:00 /usr/bin/python2 /root/.
> *ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py
> 198806654079 50 
> /root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py
> _
>
> root  69511  69510  0 04:50 ?00:00:00 /usr/bin/python2 /root/.
> *ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py
>
> root  69512  69511  1 04:50 ?00:00:00 /usr/bin/python2 /tmp/
> *ansible*_f9ckPD/*ansible*_module_command.py
>
> root  69520  69484  3 04:50 pts/10   00:00:00 /usr/bin/python2
> /usr/bin/*ansible*-playbook test_playbook.yml --forks=1
>
>
>
>
>
> *Code:*
>
> [root@oracle-siha file_copy_test]# cat test_playbook.yml
>
> - name: Testing processes
>
>   gather_facts: no
>
>   hosts: localhost
>
>   tasks:
>
> - name: run sleep command
>
>   async: 50
>
>   poll: 0
>
>   command: sleep 20
>
>   register: res
>
> - name: wait for the completion
>
>   async_status:
>
> jid: "{{ res.ansible_job_id }}"
>
>   register: output
>
>   until: output.finished
>
>   delay: 5
>
>   retries: 10
>
>
>
> On Friday, June 5, 2020 at 9:50:19 AM UTC+5:30, Jagadeeshkumar Dittakavi
> wrote:
>>
>> Thank you Matt for the detailed and quick reply.. Much appreciated the
>> support from the community.
>>
>> On Friday, June 5, 2020 at 12:29:30 AM UTC+5:30, Matt Martz wrote:
>>>
>>> Yes, it would utilize the threading library in Python.  The GIL is a
>>> primary cause to the CPU restrictions.  Our main process that orchestrates
>>> all of the task executions is already heavily CPU bound, so adding
>>> additional threads to the same core can cause a decrease in performance.
>>> Assuming we create a process model plugin type, other process models are
>>> possible, such as using asyncio, concurrent.futures, gevent, etc.  But I
>>> don't expect this work to be complete any time soon.
>>>
>>> So for now, consider forking the only process model for the near future.
>>>
>>> On Thu, Jun 4, 2020 at 1:51 PM Jagadeeshkumar Dittakavi <
>>> d.jagad...@gmail.com> wrote:
>>>
 Thank you for the prompt reply.. Just a curious question: Is the
 threading work that is underway based on python threads or pthreads or any
 other threading mechanism? As you mentioned that the threading model is not
 going to be performant, was the reason being the python's GIL?


 On Friday, June 5, 2020 at 12:01:14 AM UTC+5:30, Matt Martz wrote:
>
> The only current process model is forking.  There has been some work
> done to add a threaded process model, but there are some large hurdles to
> overcome.
>
> In practice, it is not necessarily more performant, and in many cases
> it was less performant, as it causes more CPU contention on a single core
> that is already resource constrained.
>
> On Thu, Jun 4, 2020 at 1:18 PM Jagadeeshkumar Dittakavi <
> d.jagad...@gmail.com> wrote:
>
>> I am a newbie to ansible but I got to explore how to run a tasks in
>> parallel by spawing a thread for each task instead of a process. My
>> requirement is to run the playbook on my localhost and there is no remote
>> task execution needed.
>> I also would like to wait for all threads to complete before I move
>> on to a task that has to be 

Re: [ansible-project] Multi threading support in Ansible

[Jagadeeshkumar Dittakavi]

@Matt, 

Got another question in concurrency support in Ansible.
Is there any way to limit the number of processes that could be spawned on 
a given host?
My requirement is not to execute the commands/scripts remotely. In my case, 
the whole play needs to be executed on locahost only.
I have tried a simple test program and noticed that there are as many as 6 
processes are spawned to execute 'sleep 20' asynchronously. 

Please kindly revert. Thank you inadvance.

*Command:* ansible-playbook test_playbook.yml --forks=1

 

*Processes:*

root  69484  34309  9 04:50 pts/10   00:00:00 /usr/bin/python2 /usr/bin/
*ansible*-playbook test_playbook.yml --forks=1

root  69509  1  0 04:50 ?00:00:00 /usr/bin/python2 /root/.
*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py 
198806654079 50 
/root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py 
_

root  69510  69509  0 04:50 ?00:00:00 /usr/bin/python2 /root/.
*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/async_wrapper.py 
198806654079 50 
/root/.*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py 
_

root  69511  69510  0 04:50 ?00:00:00 /usr/bin/python2 /root/.
*ansible*/tmp/*ansible*-tmp-1591876209.82-38354017880191/command.py

root  69512  69511  1 04:50 ?00:00:00 /usr/bin/python2 /tmp/
*ansible*_f9ckPD/*ansible*_module_command.py

root  69520  69484  3 04:50 pts/10   00:00:00 /usr/bin/python2 /usr/bin/
*ansible*-playbook test_playbook.yml --forks=1

 

 

*Code:*

[root@oracle-siha file_copy_test]# cat test_playbook.yml 

- name: Testing processes

  gather_facts: no

  hosts: localhost

  tasks:

- name: run sleep command

  async: 50

  poll: 0

  command: sleep 20

  register: res

- name: wait for the completion

  async_status:

jid: "{{ res.ansible_job_id }}"

  register: output

  until: output.finished

  delay: 5

  retries: 10

 

On Friday, June 5, 2020 at 9:50:19 AM UTC+5:30, Jagadeeshkumar Dittakavi 
wrote:
>
> Thank you Matt for the detailed and quick reply.. Much appreciated the 
> support from the community.
>
> On Friday, June 5, 2020 at 12:29:30 AM UTC+5:30, Matt Martz wrote:
>>
>> Yes, it would utilize the threading library in Python.  The GIL is a 
>> primary cause to the CPU restrictions.  Our main process that orchestrates 
>> all of the task executions is already heavily CPU bound, so adding 
>> additional threads to the same core can cause a decrease in performance.  
>> Assuming we create a process model plugin type, other process models are 
>> possible, such as using asyncio, concurrent.futures, gevent, etc.  But I 
>> don't expect this work to be complete any time soon.
>>
>> So for now, consider forking the only process model for the near future.
>>
>> On Thu, Jun 4, 2020 at 1:51 PM Jagadeeshkumar Dittakavi <
>> d.jagad...@gmail.com> wrote:
>>
>>> Thank you for the prompt reply.. Just a curious question: Is the 
>>> threading work that is underway based on python threads or pthreads or any 
>>> other threading mechanism? As you mentioned that the threading model is not 
>>> going to be performant, was the reason being the python's GIL?
>>>
>>>
>>> On Friday, June 5, 2020 at 12:01:14 AM UTC+5:30, Matt Martz wrote:

 The only current process model is forking.  There has been some work 
 done to add a threaded process model, but there are some large hurdles to 
 overcome.

 In practice, it is not necessarily more performant, and in many cases 
 it was less performant, as it causes more CPU contention on a single core 
 that is already resource constrained.

 On Thu, Jun 4, 2020 at 1:18 PM Jagadeeshkumar Dittakavi <
 d.jagad...@gmail.com> wrote:

> I am a newbie to ansible but I got to explore how to run a tasks in 
> parallel by spawing a thread for each task instead of a process. My 
> requirement is to run the playbook on my localhost and there is no remote 
> task execution needed.
> I also would like to wait for all threads to complete before I move on 
> to a task that has to be serialised.
>
> Can I chose thread vs process when it comes to parallel task execution?
> If it is possible to spawn threads from ansible, are they equivalent 
> to python greenthreads or pthreads or something else?
>
> Thank you in advance!
>
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to ansible...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/ansible-project/0d59cf96-b053-4390-8dbb-663c70403104o%40googlegroups.com
>  
> 

Re: [ansible-project] Multi threading support in Ansible

[Jagadeeshkumar Dittakavi]

Thank you Matt for the detailed and quick reply.. Much appreciated the 
support from the community.

On Friday, June 5, 2020 at 12:29:30 AM UTC+5:30, Matt Martz wrote:
>
> Yes, it would utilize the threading library in Python.  The GIL is a 
> primary cause to the CPU restrictions.  Our main process that orchestrates 
> all of the task executions is already heavily CPU bound, so adding 
> additional threads to the same core can cause a decrease in performance.  
> Assuming we create a process model plugin type, other process models are 
> possible, such as using asyncio, concurrent.futures, gevent, etc.  But I 
> don't expect this work to be complete any time soon.
>
> So for now, consider forking the only process model for the near future.
>
> On Thu, Jun 4, 2020 at 1:51 PM Jagadeeshkumar Dittakavi <
> d.jagad...@gmail.com > wrote:
>
>> Thank you for the prompt reply.. Just a curious question: Is the 
>> threading work that is underway based on python threads or pthreads or any 
>> other threading mechanism? As you mentioned that the threading model is not 
>> going to be performant, was the reason being the python's GIL?
>>
>>
>> On Friday, June 5, 2020 at 12:01:14 AM UTC+5:30, Matt Martz wrote:
>>>
>>> The only current process model is forking.  There has been some work 
>>> done to add a threaded process model, but there are some large hurdles to 
>>> overcome.
>>>
>>> In practice, it is not necessarily more performant, and in many cases it 
>>> was less performant, as it causes more CPU contention on a single core that 
>>> is already resource constrained.
>>>
>>> On Thu, Jun 4, 2020 at 1:18 PM Jagadeeshkumar Dittakavi <
>>> d.jagad...@gmail.com> wrote:
>>>
 I am a newbie to ansible but I got to explore how to run a tasks in 
 parallel by spawing a thread for each task instead of a process. My 
 requirement is to run the playbook on my localhost and there is no remote 
 task execution needed.
 I also would like to wait for all threads to complete before I move on 
 to a task that has to be serialised.

 Can I chose thread vs process when it comes to parallel task execution?
 If it is possible to spawn threads from ansible, are they equivalent to 
 python greenthreads or pthreads or something else?

 Thank you in advance!


 -- 
 You received this message because you are subscribed to the Google 
 Groups "Ansible Project" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to ansible...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/ansible-project/0d59cf96-b053-4390-8dbb-663c70403104o%40googlegroups.com
  
 
 .

>>>
>>>
>>> -- 
>>> Matt Martz
>>> @sivel
>>> sivel.net
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Ansible Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to ansible...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/ansible-project/48f0ed5f-83c7-4bf4-a991-94ec78ce832eo%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Matt Martz
> @sivel
> sivel.net
>

-- 
You received this message because you are subscribed to the Google Groups 
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/ansible-project/04da1e45-01fc-4842-9a4e-b53d4073ca38o%40googlegroups.com.

Re: [ansible-project] Multi threading support in Ansible

[Matt Martz]

Yes, it would utilize the threading library in Python.  The GIL is a
primary cause to the CPU restrictions.  Our main process that orchestrates
all of the task executions is already heavily CPU bound, so adding
additional threads to the same core can cause a decrease in performance.
Assuming we create a process model plugin type, other process models are
possible, such as using asyncio, concurrent.futures, gevent, etc.  But I
don't expect this work to be complete any time soon.

So for now, consider forking the only process model for the near future.

On Thu, Jun 4, 2020 at 1:51 PM Jagadeeshkumar Dittakavi <
d.jagadeeshku...@gmail.com> wrote:

> Thank you for the prompt reply.. Just a curious question: Is the threading
> work that is underway based on python threads or pthreads or any other
> threading mechanism? As you mentioned that the threading model is not going
> to be performant, was the reason being the python's GIL?
>
>
> On Friday, June 5, 2020 at 12:01:14 AM UTC+5:30, Matt Martz wrote:
>>
>> The only current process model is forking.  There has been some work done
>> to add a threaded process model, but there are some large hurdles to
>> overcome.
>>
>> In practice, it is not necessarily more performant, and in many cases it
>> was less performant, as it causes more CPU contention on a single core that
>> is already resource constrained.
>>
>> On Thu, Jun 4, 2020 at 1:18 PM Jagadeeshkumar Dittakavi <
>> d.jagad...@gmail.com> wrote:
>>
>>> I am a newbie to ansible but I got to explore how to run a tasks in
>>> parallel by spawing a thread for each task instead of a process. My
>>> requirement is to run the playbook on my localhost and there is no remote
>>> task execution needed.
>>> I also would like to wait for all threads to complete before I move on
>>> to a task that has to be serialised.
>>>
>>> Can I chose thread vs process when it comes to parallel task execution?
>>> If it is possible to spawn threads from ansible, are they equivalent to
>>> python greenthreads or pthreads or something else?
>>>
>>> Thank you in advance!
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Ansible Project" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to ansible...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/ansible-project/0d59cf96-b053-4390-8dbb-663c70403104o%40googlegroups.com
>>> 
>>> .
>>>
>>
>>
>> --
>> Matt Martz
>> @sivel
>> sivel.net
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-project+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/48f0ed5f-83c7-4bf4-a991-94ec78ce832eo%40googlegroups.com
> 
> .
>


-- 
Matt Martz
@sivel
sivel.net

-- 
You received this message because you are subscribed to the Google Groups 
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/ansible-project/CAD8N0v9LWHdjHK7KLP-AsJt1CuO0GBVzpdzJg6Ha8MiftgFvUQ%40mail.gmail.com.

Re: [ansible-project] Multi threading support in Ansible

[Jagadeeshkumar Dittakavi]

Thank you for the prompt reply.. Just a curious question: Is the threading 
work that is underway based on python threads or pthreads or any other 
threading mechanism? As you mentioned that the threading model is not going 
to be performant, was the reason being the python's GIL?


On Friday, June 5, 2020 at 12:01:14 AM UTC+5:30, Matt Martz wrote:
>
> The only current process model is forking.  There has been some work done 
> to add a threaded process model, but there are some large hurdles to 
> overcome.
>
> In practice, it is not necessarily more performant, and in many cases it 
> was less performant, as it causes more CPU contention on a single core that 
> is already resource constrained.
>
> On Thu, Jun 4, 2020 at 1:18 PM Jagadeeshkumar Dittakavi <
> d.jagad...@gmail.com > wrote:
>
>> I am a newbie to ansible but I got to explore how to run a tasks in 
>> parallel by spawing a thread for each task instead of a process. My 
>> requirement is to run the playbook on my localhost and there is no remote 
>> task execution needed.
>> I also would like to wait for all threads to complete before I move on to 
>> a task that has to be serialised.
>>
>> Can I chose thread vs process when it comes to parallel task execution?
>> If it is possible to spawn threads from ansible, are they equivalent to 
>> python greenthreads or pthreads or something else?
>>
>> Thank you in advance!
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Ansible Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to ansible...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/ansible-project/0d59cf96-b053-4390-8dbb-663c70403104o%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Matt Martz
> @sivel
> sivel.net
>

-- 
You received this message because you are subscribed to the Google Groups 
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/ansible-project/48f0ed5f-83c7-4bf4-a991-94ec78ce832eo%40googlegroups.com.

Re: [ansible-project] Multi threading support in Ansible

[Matt Martz]

The only current process model is forking.  There has been some work done
to add a threaded process model, but there are some large hurdles to
overcome.

In practice, it is not necessarily more performant, and in many cases it
was less performant, as it causes more CPU contention on a single core that
is already resource constrained.

On Thu, Jun 4, 2020 at 1:18 PM Jagadeeshkumar Dittakavi <
d.jagadeeshku...@gmail.com> wrote:

> I am a newbie to ansible but I got to explore how to run a tasks in
> parallel by spawing a thread for each task instead of a process. My
> requirement is to run the playbook on my localhost and there is no remote
> task execution needed.
> I also would like to wait for all threads to complete before I move on to
> a task that has to be serialised.
>
> Can I chose thread vs process when it comes to parallel task execution?
> If it is possible to spawn threads from ansible, are they equivalent to
> python greenthreads or pthreads or something else?
>
> Thank you in advance!
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-project+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/0d59cf96-b053-4390-8dbb-663c70403104o%40googlegroups.com
> 
> .
>


-- 
Matt Martz
@sivel
sivel.net

-- 
You received this message because you are subscribed to the Google Groups 
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/ansible-project/CAD8N0v9K-t60p4J%2B6jBWjs%3DQRXFPP0PYFh1NrMeOfZ8aJ_QcXw%40mail.gmail.com.