westonpace commented on issue #15054: URL: https://github.com/apache/arrow/issues/15054#issuecomment-1400794524
@pitrou @lidavidm I prototyped a shutdown mechanism this morning (https://github.com/apache/arrow/commit/9d62b2c9f349c4f896d88367b8a0145baed74ba3) and I think it could work but I realized there might be another way we can solve this problem as we ran into a similar problem with open telemetry. We added the concept of `Executor::KeepAlive` which is a method that registers a resource with our thread pool instance and keeps that resource alive until all threads have exited. Our OT code then uses this to ensure that OT is not shutdown until all threads have exited. We can extend this mechanism to S3. The main drawback will be that a call to `FinalizeS3` will not actually call `Aws::ShutdownAPI` until process exit (after the thread pool has shut down). Would this be an acceptable workaround? It would allow us to safely use a python atexit hook to call `FinalizeS3`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
