mikamikasuki opened a new pull request, #54467: URL: https://github.com/apache/arrow/pull/54467
### Rationale for this change Fixes #50482. A Python-backed dataset scan can return `ArrowInvalid` while CPU-pool workers still retain Python-backed resources. During interpreter finalization, the main thread waits for pool shutdown as a worker attempts to attach to Python, so the process can hang or abort. ### What changes are included in this PR? Register an `atexit` callback that releases the GIL while waiting for the CPU thread pool to become idle before interpreter finalization. Add a subprocess regression test for an FSSpec-backed dataset containing one malformed Parquet fragment. ### Are these changes tested? - Built Arrow and editable PyArrow from this branch on macOS. - `pyarrow/tests/test_misc.py`: 123 passed, 1 skipped on CPython 3.12.15 and 3.14.8. - The targeted exit regression passed 12/12 runs on each Python version. - On PyArrow 26.0.0, the baseline timed out in 8/8 runs on CPython 3.12.15 and 7/8 runs on 3.14.8. ### Are there any user-facing changes? The malformed dataset continues to raise `ArrowInvalid`; after the error, interpreter shutdown now waits for Python-backed cleanup to finish. **This PR contains a "Critical Fix."** It prevents the reported process-exit hang or abort when worker cleanup attaches to Python during interpreter finalization. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [ ] Human - [x] AI - [ ] Not reviewed AI assistance was used to trace the thread-pool shutdown path, implement the wait callback and regression test, and draft this description. The generated changes were reviewed and built on CPython 3.12.15 and 3.14.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
