jasonhimmelstein commented on issue #50482:
URL: https://github.com/apache/arrow/issues/50482#issuecomment-7047260609
Here's a Linux data point for the `pa.BufferReader(bytes)` case: on CPython
3.12 it aborts rather than hangs, even with the `Py_IsInitialized()` guard from
#49943 in place.
Environment: pyarrow 26.0.1 (PyPI wheel), CPython 3.12.16 and 3.13.18, Linux
7.0 x86_64, glibc 2.43 (GitHub Actions `ubuntu-latest`).
```python
# repro.py
import io
import threading
import time
import pyarrow as pa
import pyarrow.parquet as pq
def wake():
while True:
time.sleep(0.0005)
for _ in range(8):
threading.Thread(target=wake, daemon=True).start()
sink = io.BytesIO()
pq.write_table(pa.table({"x": list(range(1000)), "s": ["a"] * 1000}), sink)
raw = sink.getvalue()
for _ in range(4):
pq.read_table(pa.BufferReader(raw), use_threads=False)
```
Fresh processes, 5 at a time:
| Run | Exited non-zero |
|---|---|
| 3.12.16 | 9 of 3000, all `terminate called without an active exception` |
| 3.12.16, `raw` copied into `pa.allocate_buffer(len(raw))` first | 0 of
3000 |
| 3.12.16, without the 8 threads | 0 of 600 |
| 3.13.18 | 13 of 3000, with no output and no core dump (probably hung until
our 60 s timeout; we didn't record exit codes) |
The other threads are what make it happen. Without them we couldn't trigger
it. In our application, the native threads of another extension module
(deltalake's tokio runtime) had the same effect, at about 2% of runs.
Aborting thread (3.12, from a core dump):
```
#7 std::terminate()
#10 _Unwind_ForcedUnwind
#13 __GI___pthread_exit
#14 PyThread_exit_thread Python/thread_pthread.h:370
#15 take_gil Python/ceval_gil.c:353
#16 PyEval_RestoreThread Python/ceval_gil.c:708
#17 PyGILState_Ensure Python/pystate.c:2240
#18 arrow::py::PyBuffer::~PyBuffer()
#19 std::_Sp_counted_ptr<arrow::py::PyBuffer*, ...>::_M_dispose()
#21 arrow::io::BufferReader::~BufferReader()
#23 parquet::SerializedFile::~SerializedFile()
#24 parquet::arrow::(anonymous namespace)::FileReaderImpl::~FileReaderImpl()
#26 FnOnce<void (FutureImpl
const&)>::FnImpl<Future<Empty>::WrapResultOnComplete::Callback<...>>
#29 ConcreteFutureImpl::DoMarkFinishedOrFailed(FutureState)
...
#34 FnOnce<void ()>::FnImpl<ThreadPool::SpawnReal(...)>
#35 std::thread::_State_impl<...
ThreadPool::LaunchWorkersUnlocked(int)::{lambda()#1} ...>::_M_run()
```
The main thread is in `Py_FinalizeEx` (`pylifecycle.c:1915`), in
`PyGC_Collect`. So the last reference to the `FileReaderImpl` is dropped by a
future callback on a CPU-pool worker, even with `use_threads=False`, after
`read_table` has returned. `Py_IsInitialized()` is still true that early in
finalization, so the guard doesn't stop `~PyBuffer` from calling
`PyGILState_Ensure`. CPython 3.12 then ends the worker with `pthread_exit`, and
the forced unwind through the `noexcept` destructor calls `std::terminate`. A
check on `Py_IsFinalizing()` (3.13+) or `_Py_IsFinalizing()` (3.12) would catch
it. Releasing the reader on the calling thread before `read_table` returns
would avoid it altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]