[issue42647] Unable to use concurrent.futures in atexit hook

[Dong-hee Na]

Change by Dong-hee Na :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42647] Unable to use concurrent.futures in atexit hook

[Julien Danjou]

Julien Danjou  added the comment:

> Is there a real-world situation where it's specifically necessary or even 
> beneficial to utilize ThreadPoolExecutor at this point after thread 
> finalization rather than earlier in the program? Not that it doesn't exist, 
> but to me it intuitively seems very odd to utilize an executor within an 
> atexit hook, which are intended to just be resource finalization/cleanup 
> functions called at interpreter shutdown. Assuming there is a genuine use 
> case I'm not seeing, it may be worth weighing against the decision to convert 
> the executors to not use daemon threads, as I presently don't think there's a 
> way to (safely) allow that behavior without reverting back to using daemon 
> threads.

To put that in perspective, here is the original issue that trigged this bug 
for me:


Traceback (most recent call last):
  File "/root/project/ddtrace/profiling/scheduler.py", line 50, in flush
exp.export(events, start, self._last_export)
  File "/root/project/ddtrace/profiling/exporter/http.py", line 186, in export
self._upload(client, self.endpoint_path, body, headers)
  File "/root/project/ddtrace/profiling/exporter/http.py", line 189, in _upload
self._retry_upload(self._upload_once, client, path, body, headers)
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py",
 line 423, in __call__
do = self.iter(retry_state=retry_state)
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py",
 line 360, in iter
return fut.result()
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", 
line 433, in result
return self.__get_result()
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", 
line 389, in __get_result
raise self._exception
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py",
 line 426, in __call__
result = fn(*args, **kwargs)
  File "/root/project/ddtrace/profiling/exporter/http.py", line 193, in 
_upload_once
client.request("POST", path, body=body, headers=headers)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255, 
in request
self._send_request(method, url, body, headers, encode_chunked)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301, 
in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250, 
in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010, 
in _send_output
self.send(msg)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in 
send
self.connect()
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 921, in 
connect
self.sock = self._create_connection(
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/socket.py",
 line 88, in create_connection
addrs = list(getaddrinfo(host, port, 0, SOCK_STREAM))
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_socketcommon.py",
 line 247, in getaddrinfo
addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, 
flags)
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py",
 line 841, in _get_resolver
self._resolver = self.resolver_class(hub=self) # pylint:disable=not-callable
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/resolver/thread.py",
 line 39, in __init__
self.pool = hub.threadpool
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py",
 line 865, in _get_threadpool
self._threadpool = self.threadpool_class(self.threadpool_size, hub=self)
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py",
 line 860, in threadpool_class
return GEVENT_CONFIG.threadpool
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
 line 50, in getter
return self.settings[setting_name].get()
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
 line 146, in get
self.value = self.validate(self._default())
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
 line 248, in validate
return self._import_one_of([self.shortname_map.get(x, x) for x in value])
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
 line 223, in _import_one_of
return self._import_one(candidates[-1])
  File 
"/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py",
 line 237, in _import_one
module = importlib.import_module(module)
  File 

[issue42647] Unable to use concurrent.futures in atexit hook

[Kyle Stanley]

Kyle Stanley  added the comment:

Thanks for bringing attention to this, Julien. 

While the regression is definitely unfortunate, I'm uncertain about whether the 
behavior is *correct* to allow in the first place. Specifically, allowing the 
registration of an atexit hook which uses a ThreadPoolExecutor within it means 
that the finalization of the executor will be done *after* thread finalization 
occurs, leaving dangling threads which will have to be abruptly killed upon 
interpreter exit instead of being safely joined. From my perspective at least, 
this doesn't seem like something to be encouraged.

Is there a real-world situation where it's specifically necessary or even 
beneficial to utilize ThreadPoolExecutor at this point after thread 
finalization rather than earlier in the program? Not that it doesn't exist, but 
to me it intuitively seems very odd to utilize an executor within an atexit 
hook, which are intended to just be resource finalization/cleanup functions 
called at interpreter shutdown. Assuming there is a genuine use case I'm not 
seeing, it may be worth weighing against the decision to convert the executors 
to not use daemon threads, as I presently don't think there's a way to (safely) 
allow that behavior without reverting back to using daemon threads.

That said, I'll admit that I'm more than a bit biased as the author of the 
commit which introduced the regression, so I'll CC Antoine Pitrou (active 
expert for threading and concurrent.futures) to help make the final decision.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42647] Unable to use concurrent.futures in atexit hook

[Irit Katriel]

Change by Irit Katriel :


--
nosy: +aeros

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42647] Unable to use concurrent.futures in atexit hook

[Irit Katriel]

Irit Katriel  added the comment:

b61b818d916942aad1f8f3e33181801c4a1ed14b is the first bad commit
commit b61b818d916942aad1f8f3e33181801c4a1ed14b
Author: Kyle Stanley 
Date:   Fri Mar 27 15:31:22 2020 -0400

bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)

Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42647] Unable to use concurrent.futures in atexit hook

[Julien Danjou]

New submission from Julien Danjou :

Python 3.9 introduced a regression with concurrent.futures. The following 
program works fine on Python < 3.8 but raises an error on 3.9:

```
import atexit
import concurrent.futures

def spawn():
with concurrent.futures.ThreadPoolExecutor() as t:
pass

atexit.register(spawn)
```

```
$ python3.9 rep.py
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 680, in _load_unlocked
  File "", line 790, in exec_module
  File "", line 228, in _call_with_frames_removed
  File 
"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py",
 line 37, in 
threading._register_atexit(_python_exit)
  File 
"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py",
 line 1370, in _register_atexit
raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown
```

--
messages: 383058
nosy: jd
priority: normal
severity: normal
status: open
title: Unable to use concurrent.futures in atexit hook
type: crash
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com