masatakashiwagi commented on issue #50482:
URL: https://github.com/apache/arrow/issues/50482#issuecomment-6997273395

   Another data point from macOS: the hang is not limited to Python file 
objects. Wrapping a Python `bytes` object in `pa.BufferReader` triggers it too, 
while the same data in an Arrow-owned buffer does not. It still reproduces on 
26.0.1.
   
   Environment: macOS 15.7 (arm64, Apple M4, 10 cores), CPython 3.12.9 
(python-build-standalone via uv), fresh venv with only the `pyarrow` wheel from 
PyPI.
   
   ```python
   # repro.py SOURCE THREADS
   import io
   import sys
   
   import pyarrow as pa
   import pyarrow.parquet as pq
   
   source, threads = sys.argv[1], sys.argv[2] == "threads"
   stream = pa.BufferOutputStream()
   pq.write_table(pa.table({"a": pa.array([f"k{n}" for n in range(200_000)])}), 
stream)
   arrow_buffer = stream.getvalue()
   if source == "arrow_buffer":
       pq.read_table(pa.BufferReader(arrow_buffer), use_threads=threads)
   elif source == "python_bytes":
       pq.read_table(pa.BufferReader(arrow_buffer.to_pybytes()), 
use_threads=threads)
   elif source == "bytesio":
       pq.read_table(io.BytesIO(arrow_buffer.to_pybytes()), use_threads=threads)
   print("ok", flush=True)
   ```
   
   Fresh process per run; a run counts as hung when it printed `ok` but was 
still alive after 6 s.
   
   pyarrow 26.0.1, 30 runs each:
   
   | source | `use_threads=True` | `use_threads=False` |
   | --- | --- | --- |
   | `pa.BufferReader(pa.Buffer)` (Arrow-owned) | 0/30 | 0/30 |
   | `pa.BufferReader(bytes)` (Python-owned) | 21/30 | 0/30 |
   | `io.BytesIO(bytes)` | 16/30 | 0/30 |
   
   `python_bytes` with `use_threads=True` on older wheels, 30 runs each: 25.0.0 
12/30, 24.0.0 15/30, 23.0.1 25/30, 22.0.0 22/30.
   
   In every hung process I sampled (`sample`), the main thread is in `exit` → 
`__cxa_finalize_ranges` → `arrow::internal::ThreadPool::~ThreadPool()` → 
`ThreadPool::Shutdown(bool)`, waiting on a condition variable. No thread is in 
`PyGILState_Ensure`. For `python_bytes`, the only other thread is an idle 
libdispatch thread, so no CPU-pool worker is left at all. For `bytesio`, one 
worker remains, idle in the worker loop. My guess is that a worker that touched 
Python during finalization was terminated by CPython without leaving the pool's 
worker list, so `Shutdown` waits for it forever. I have not verified this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to