pitrou commented on issue #44071: URL: https://github.com/apache/arrow/issues/44071#issuecomment-2346643834
Ok, I think I understand what happens. uwsgi finalizes its plugins thanks to a C-level `atexit` hook: https://github.com/unbit/uwsgi/blob/b9618b619a73f2fc9680094eba7c202f5422fa42/core/uwsgi.c#L3202 The atexit handler for the Python handler then does call `Py_Finalize` at the end: https://github.com/unbit/uwsgi/blob/b9618b619a73f2fc9680094eba7c202f5422fa42/plugins/python/python_plugin.c#L388-L446 So we have this: 1. uwsgi receives a Ctrl-C (or decides to shutdown a worker for another reason) 2. for this, it calls the libc's `exit` function 3. the libc's `exit` function shuts down the process 4. at some point the process shutdown triggers C-level `atexit` handlers 5. this calls back into uwsgi, especially `uwsgi_plugins_atexit` 6. `uwsgi_plugins_atexit` calls the Python plugin's `uwsgi_python_atexit` 7. `uwsgi_python_atexit` calls `Py_Finalize` which calls the Python [atexit](https://docs.python.org/fr/3/library/atexit.html) handlers 8. our atexit handler is then executed to finalize S3: https://github.com/apache/arrow/blob/6a38205d69ed97b21ad8dcfba11069ebb28c05d9/python/pyarrow/fs.py#L69-L70 However, at this point, it's already too late, because calling the libc's `exit` function has already deallocated critical resources in the AWS SDK. S3 finalization fails and crashes the process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
