[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Kyle Stanley added the comment: > Agreed, and that is precisely what I suggested in my previous comment. The > attached PR already deals with the executor-specific part of the wait, but it > would be straightforward to have it also affect the executor to create daemon > threads, and the flag

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Hrvoje Nikšić added the comment: > The only way I could see this to work as intended without making any changes > to threading would be to optionally use daemon threads and avoid joining the > threads in `executor.shutdown()` if `wait_at_exit` is set to False in the > constructor for the exec

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Kyle Stanley added the comment: > ThreadPoolExecutor introduces additional waiting of its own, and it is this > wait the PR adds an option to disable. [next post] > Thanks for the clarification, I didn't know about the change to non-daemon > threads. > I still think this patch is useful, a

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Kyle Stanley added the comment: > I don't think there's much ThreadPoolExecutor can do. If you drop the > references to the threads, they still exist and they still be waited upon at > interpreter exit. > > The solution is for you to avoid having hanging threads. In the particular > case

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Hrvoje Nikšić added the comment: Thanks for the clarification, I didn't know about the change to non-daemon threads. I still think this patch is useful, and won't harm general use because it is opt-in, just like daemon threads themselves. I suggest to update the PR to specify non-waiting poo

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Antoine Pitrou added the comment: I think there's a misunderstanding: "wait_at_exit" will make the *executor* forget about the threads, but Python itself still knows about them, and it waits for them to end at interpreter shutdown. These threads were daemon threads in 3.8, so your patch inde

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Hrvoje Nikšić added the comment: > I don't think there's much ThreadPoolExecutor can do. If you drop the > references to the threads, they still exist and they still be waited upon at > interpreter exit. ThreadPoolExecutor introduces additional waiting of its own, and it is this wait the PR

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Antoine Pitrou added the comment: I don't think there's much ThreadPoolExecutor can do. If you drop the references to the threads, they still exist and they still be waited upon at interpreter exit. The solution is for you to avoid having hanging threads. In the particular case of TCP con

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Change by Antoine Pitrou : -- nosy: +aeros ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Torsten Landschoff added the comment: I intended to file a bug because of the shutdown problem but found out there is this ticket already. In our application we sometimes run into this problem that you can't exit because of a hanging TCP connection used inside a ThreadPoolExecutor task. ex

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Andrew Svetlov added the comment: As documentation reader I have a mental problem to figure out when use non-default `wait_at_exit=False` flag and when don't. The rule like: "if you tried to call `executor.shutdown(wait=False)` and it still hangs try `executor.shutdown(wait=False, wait_at_ex

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Hrvoje Nikšić added the comment: @asvetlov The idea of the new flag is to disable any subsequent waiting for futures after ThreadPoolExecutor.shutdown(wait=False) returns. Currently the additional waiting is implemented using "atexit", so I assumed it referred to process exit. (The documentat

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Andrew Svetlov added the comment: Should wait_on_exit be True only on application exit? Do application exit means interpreter shutdown (sys.is_finalizing() == True)? -- nosy: +asvetlov ___ Python tracker ___

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Change by Hrvoje Nikšić : -- keywords: +patch pull_requests: +13161 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-li

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Brian Quinlan added the comment: We can bike shed over the name in the PR ;-) -- ___ Python tracker ___ ___ Python-bugs-list mailin

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Hrvoje Nikšić added the comment: > Are you interested in writing a patch? Yes - I wanted to check if there is interest in the feature before I commit time to write the patch, documentation, tests, etc. (And also I wanted to check if there's a better way to do it.) In any case, thanks for pic

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Brian Quinlan added the comment: Hey Hrvoje, I agree that #1 is the correct approach. `disown` might not be the best name - maybe `allow_shutdown` or something. But we can bike shed about that later. Are you interested in writing a patch? -- ___

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Hrvoje Nikšić added the comment: I agree with the last comment. The disowning functionality is only used in specific circumstances, so it's perfectly fine to keep the functionality as a shutdown flag. I also agree that the change cannot be *unconditional*, for backward compatibility if not fo

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Brian Quinlan added the comment: OK, I completely disagree with my statement: """If you added this as an argument to shutdown() then you'd probably also have to add it as an option to the constructors (for people using Executors as context managers). But, if you have to add it to the constru

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Brian Quinlan added the comment: >> The current behavior is explicitly documented, so presumably >> it can't be (easily) changed And it isn't clear that it would be desirable to change this even if it were possible - doing structured resource clean-up seems consistent with the rest of Python

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

Change by SilentGhost : -- nosy: +bquinlan, pitrou type: -> behavior versions: -Python 3.9 ___ Python tracker ___ ___ Python-bugs-

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

New submission from Hrvoje Nikšić : At interpreter shutdown, Python waits for all pending futures of all executors to finish. There seems to be no way to disable the wait for pools that have been explicitly shut down with pool.shutdown(wait=False). The attached script demonstrates the issue.