Eric Snow <[email protected]> added the comment:
> This means that ThreadPoolExecutor's atexit runs before mine,
> and since I never get a chance to cancel my tasks, it deadlocks.
(assuming we want to support long-running tasks here)
With all the above in mind, there are a few things that may help.
The first that comes to mind is to have the atexit handler call
ThreadPoolExecutor.shutdown() for each instance.
So something like this:
def _python_exit():
global _shutdown
with _global_shutdown_lock:
_shutdown = True
for executor in list(_executors):
executor.shutdown()
That would require a little refactoring to make it work. However, the change
is simpler if each executor has its own atexit handler:
class ThreadPoolExecutor(_base.Executor):
def __init__(self, ...):
...
threading._register_atexit(self._atexit())
def _atexit(self):
global _shutdown
_shutdown = True
self.shutdown()
The value of either approach is that you can then subclass ThreadPoolExecutor
to get what you want:
class MyExecutor(ThreadPoolExecutor):
def shutdown(self, *args, **kwargs):
stop_my_tasks()
super().shutdown(*args, **kwwargs)
----
One thing I thought about was supporting a per-task finalizer instead, since
that aligns more closely with the way ThreadPoolExecutor works. It would only
apply So something like one of the following:
* ThreadPoolExecutor(finalize_task=<callable>)
* ThreadPoolExecutor.submit(finalize=<callable)
----
Other things that could be helpful:
* always cancel all the pending tasks during shutdown (and maybe let users opt
out)
* use a timeout during shutdown
----
FWIW, adding support for some sort of sub-atexit handler isn't so appealing.
I'm talking about something like one of the following:
* ThreadPoolExecutor(onshutdown=<callable>)
* ThreadPoolExecutor.register_atexit(<callable>)
* (classmethod) ThreadPoolExecutor.register_atexit(<callable>)
* concurrent.futures.register_atexit(<callable>)
(It would probably make sense to pass the list of currently running tasks to
the callable.)
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue41962>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
