[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

Ronald Oussoren added the comment: Could someone that runs into this issue with python 3.7 please test if the issue is still present in 3.8 or 3.9? BTW. I'm not convinced this is a macOS specific problem, see issue40379 which claims that fork-without-exec strategy is inherently broken.

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

DanilZ added the comment: Dear All, Thanks for the great input. As described above it appears to be a MacOS problem. -- ___ Python tracker ___

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

Ronald Oussoren added the comment: The script as-is doesn't work with 3.8 because it uses the "spawn" strategy. I haven't tried to tweak the script to get it to work on 3.8 because the scripts works fine for me with 3.7. The smaller script in msg380225 works for me on both python 3.7.4 and

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

DanilZ added the comment: Hi Ken, Thanks for your comment. Unfortunately at the time I can not upgrade to 3.8 to run this test. My whole system depends on 3.7 and some peculiarities of 3.8 need to be dealt with. It would be great if someone with OSX and 3.8 could test this out, otherwise

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

Ken Jin added the comment: Danil, thanks for finding the cause behind this. Could you check if the new behavior in Python 3.8 and higher has the same problem on your machine (without your fix)? multiprocessing on MacOS started using spawn in 3.8, and I was wondering if it that fixed it.

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

DanilZ added the comment: I have managed to solve the problem by inserting in the beginning of my program: import multiprocessing multiprocessing.set_start_method('forkserver') as this is explained here:

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

DanilZ added the comment: Thank you so much for the input! I will study all the links you have sent: Here is a screen recording of some additional experiments: https://vimeo.com/user50681456/review/474733642/b712c12c2c --

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

Ken Jin added the comment: Hmm apologies I'm stumped then. The only things I managed to surmise from xgboost's and scikit-learn's GitHub issues is that this is a recurring issue specifically when using GridSearchCV : Threads with discussions on workarounds:

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

DanilZ added the comment: FYI: I’ve tried all the three of the possible backends: ‘loky’ (default) / ’threading’ / ‘multiprocessing’. None of them solved the problem. > On 2 Nov 2020, at 17:34, Ken Jin wrote: > > A temporary workaround might be to reduce n_jobs OR even better: use >

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

DanilZ added the comment: Here is a gif of what’s going on in my ActivityMonitor on a Mac while this code is executed: https://gfycat.com/unselfishthatgraysquirrel -- ___ Python tracker

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

DanilZ added the comment: Hi Ken, thanks for a quick reply. Here are the requested specs. System: Python 3.7.6 OS X 10.15.7 Packages: XGBoost 1.2.0 sklearn 0.22.2 pandas 1.0.5 numpy 1.18.1 I can see that you have reduced the code, which now excludes the RandomizedSearchCV part. This

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

Ken Jin added the comment: Hello, it would be great if you can you provide more details. Like your Operating System and version, how many logical CPU cores there are on your machine, and lastly the exact Python version with major and minor versions included (eg. Python 3.8.2).

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

New submission from DanilZ : Note: problem occurs only after performing the RandomizedSearchCV... When applying a function in a multiprocess using concurrent.futures if the function includes anything else other than print(), it is not executed and the process freezes. Here is the code to