Mouse <[email protected]> added the comment:
The fix applied for this problem actually broke multiprocessing on MacOS. The
change to the new default 'spawn' from 'fork' causes program to crash in
spawn.py with `FileNotFoundError: [Errno 2] No such file or directory`.
I've tested this on MacOS Catalina 10.15.3 and 10.15.4, with Python-3.8.2 and
Python-3.7.7.
With Python-3.7.7 everything works as expected.
Here's the output:
{{{
$ python3.8 multi1.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py",
line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py",
line 126, in _main
self = reduction.pickle.load(from_parent)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py",
line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
}}}
Here's the program:
{{{
#!/usr/bin/env python3
#
# Test "multiprocessing" package included with Python-3.6+
#
# Usage:
# ./mylti1.py [nElements [nProcesses [tSleep]]]
#
# nElements - total number of integers to put in the queue
# default: 100
# nProcesses - total number of parallel processes/threads
# default: number of physical cores available
# tSleep - number of milliseconds for a thread to sleep
# after it retrieved an element from the queue
# default: 17
#
# Algorithm:
# 1. Creates a queue and adds nElements integers to it,
# 2. Creates nProcesses threads
# 3. Each thread extracts an element from the queue and sleeps for tSleep
milliseconds
#
import sys, queue, time
import multiprocessing as mp
def getElements(q, tSleep, idx):
l = [] # list of pulled numbers
while True:
try:
l.append(q.get(True, .001))
time.sleep(tSleep)
except queue.Empty:
if q.empty():
print(f'worker {idx} done, got {len(l)} numbers')
return
if __name__ == '__main__':
nElements = int(sys.argv[1]) if len(sys.argv) > 1 else 100
nProcesses = int(sys.argv[2]) if len(sys.argv) > 2 else mp.cpu_count()
tSleep = float(sys.argv[3]) if len(sys.argv) > 3 else 17
# Uncomment the following line to make it working with Python-3.8+
#mp.set_start_method('fork')
# Fill the queue with numbers from 0 to nElements
q = mp.Queue()
for k in range(nElements):
q.put(k)
# Start worker processes
for m in range(nProcesses):
p = mp.Process(target=getElements, args=(q, tSleep / 1000, m))
p.start()
}}}
----------
nosy: +mouse07410
type: -> crash
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue33725>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
