On 6/12/2020 2:17 PM, Chris Angelico wrote:
> On Sat, Jun 13, 2020 at 3:50 AM Edwin Zimmerman <ed...@211mainstreet.net>
> wrote:
>> My previous timings were slightly inaccurate, as they compared spawning
>> processes on Windows to forking on Linux. Also, I changed my timing code to
>> run all process synchronously, to avoid hitting resource limits.
>>
>> Updated Windows (Windows 7 this time, on a four core processor):
>>
>>>>> timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()',
>>>>> number=1000,globals = globals())
>> 84.7111053659259
>>
> Thanks, I was actually going to ask about joining the processes, since
> you don't really get a good indication of timings from asynchronous
> operations like that. Another interesting data point is that starting
> and joining in batches makes a fairly huge difference to performance,
> at least on my Linux system. Starting with your example and rescaling
> the number by ten to compensate for performance differences:
>
>>>> timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()',
>>>> number=10000,globals = globals())
> 14.261007152497768
>
> Just for completeness and consistency, confirmed that adding a list
> comp around it doesn't change the timings:
>
>>>> timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in
>>>> range(1)];[x.start() for x in xx];[x.join() for x in xx]',
>>>> number=10000,globals = globals())
> 14.030426062643528
>
> But doing a hundred at a time and then joining them all cuts the time in half!
>
>>>> timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in
>>>> range(100)];[x.start() for x in xx];[x.join() for x in xx]',
>>>> number=100,globals = globals())
> 5.470761131495237
>
> The difference is even more drastic with spawn, although since it's
> slower, I also lowered the number of iterations.
>
>>>> ctx = multiprocessing.get_context('spawn')
>>>> timeit.timeit('x=ctx.Process(target=exit);x.start();x.join()',
>>>> number=1000,globals = globals())
> 40.82687543518841
>>>> timeit.timeit('xx=[ctx.Process(target=exit) for _ in
>>>> range(100)];[x.start() for x in xx];[x.join() for x in xx]',
>>>> number=10,globals = globals())8.566341979429126
> 8.566341979429126
>
> Would be curious to know if that's the same on Windows.
Yea, it's the same. Watch your cpu utilization, and you will realize that your
list comprehension is parallelizing the process startups.
> ChrisA
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YFRM3LNO37B5JXNYPO2T7CAVYQRGAYES/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/CICKBHTOAUOW3ARZ2Z4AYAKOWVGKWKVU/
Code of Conduct: http://python.org/psf/codeofconduct/
