masatakashiwagi commented on issue #50482:
URL: https://github.com/apache/arrow/issues/50482#issuecomment-6997273395
Another data point from macOS: the hang is not limited to Python file
objects. Wrapping a Python `bytes` object in `pa.BufferReader` triggers it too,
while the same data in an Arrow-owned buffer does not. It still reproduces on
26.0.1.
Environment: macOS 15.7 (arm64, Apple M4, 10 cores), CPython 3.12.9
(python-build-standalone via uv), fresh venv with only the `pyarrow` wheel from
PyPI.
```python
# repro.py SOURCE THREADS
import io
import sys
import pyarrow as pa
import pyarrow.parquet as pq
source, threads = sys.argv[1], sys.argv[2] == "threads"
stream = pa.BufferOutputStream()
pq.write_table(pa.table({"a": pa.array([f"k{n}" for n in range(200_000)])}),
stream)
arrow_buffer = stream.getvalue()
if source == "arrow_buffer":
pq.read_table(pa.BufferReader(arrow_buffer), use_threads=threads)
elif source == "python_bytes":
pq.read_table(pa.BufferReader(arrow_buffer.to_pybytes()),
use_threads=threads)
elif source == "bytesio":
pq.read_table(io.BytesIO(arrow_buffer.to_pybytes()), use_threads=threads)
print("ok", flush=True)
```
Fresh process per run; a run counts as hung when it printed `ok` but was
still alive after 6 s.
pyarrow 26.0.1, 30 runs each:
| source | `use_threads=True` | `use_threads=False` |
| --- | --- | --- |
| `pa.BufferReader(pa.Buffer)` (Arrow-owned) | 0/30 | 0/30 |
| `pa.BufferReader(bytes)` (Python-owned) | 21/30 | 0/30 |
| `io.BytesIO(bytes)` | 16/30 | 0/30 |
`python_bytes` with `use_threads=True` on older wheels, 30 runs each: 25.0.0
12/30, 24.0.0 15/30, 23.0.1 25/30, 22.0.0 22/30.
In every hung process I sampled (`sample`), the main thread is in `exit` →
`__cxa_finalize_ranges` → `arrow::internal::ThreadPool::~ThreadPool()` →
`ThreadPool::Shutdown(bool)`, waiting on a condition variable. No thread is in
`PyGILState_Ensure`. For `python_bytes`, the only other thread is an idle
libdispatch thread, so no CPU-pool worker is left at all. For `bytesio`, one
worker remains, idle in the worker loop. My guess is that a worker that touched
Python during finalization was terminated by CPython without leaving the pool's
worker list, so `Shutdown` waits for it forever. I have not verified this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]