imanhabib opened a new issue, #50482:
URL: https://github.com/apache/arrow/issues/50482
### Describe the bug, including details regarding any error messages,
version, and platform.
### Describe the bug, including details regarding any error messages,
version, and platform.
Reading a parquet file by passing a **Python file object** to
`pyarrow.parquet.read_table`
makes the interpreter hang forever at shutdown on my and other similar
machines:
```python
# repro.py - hangs at exit, every run
import pyarrow.parquet as pq
with open("any.parquet", "rb") as f:
pq.read_table(f)
print("done, exiting...") # prints - then the process never exits
```
```
$ python repro.py
done, exiting...
# hangs forever; SIGKILL required
```
Passing a **path** instead of a file object does not hang:
```python
pq.read_table("any.parquet") # exits cleanly, 8/8 runs
```
`use_threads=False` also avoids the hang even with a file object (8/8
clean), as does
`pa.set_cpu_count(1)`, so this looks like a CPU thread-pool teardown
deadlock that is
only triggered when the read went through a Python file handle (GIL
interaction during
pool shutdown?).
#### How we found it
`pandas.read_parquet` (which internally opens the file and passes a handle
to pyarrow)
was hanging **intermittently** (~50% of runs when reading 3 files, rarer
with 1 file).
Bisecting pandas out of the equation led to the deterministic file-object
reproducer
above. So this same bug likely explains "flaky CI hang" reports from pandas
users.
#### Details
- The script body completes and **`atexit` handlers run**; the hang is after
that,
during interpreter/C-level finalization. `faulthandler` (SIGABRT) produces
no dump.
- The file content doesn't matter - reproduces with a 5-row single-column
file.
- Closing the file object (or not) before exit makes no difference.
- A hung process shows the main thread plus ~24 CPU-pool threads all in
`futex_do_wait` (from `/proc/<pid>/task/*/wchan`), and a `jemalloc_bg_thd`:
```
python(futex_do_wait) jemalloc_bg_thd(futex_do_wait) python(futex_do_wait)
x24 ...
```
#### Version matrix (same reproducer, 4 runs each; `124` = hung/timeout, `0`
= clean exit)
| pyarrow | result |
|---|---|
| 25.0.0 | 124 124 124 124 |
| 24.0.0 | 124 124 124 124 |
| 23.0.1 | 124 124 124 124 |
| 22.0.0 | 124 0 124 124 |
| 25.0.0 + `use_threads=False` | 0 0 0 0 |
| 25.0.0 + path instead of file object | 0 0 0 0 |
(22.0.0 being intermittent rather than deterministic suggests a
timing-dependent race
that became reliably lost at some point.)
#### Second interpreter build
Also reproduces on the distro CPython (Python 3.14.4, GCC 15.2.0 build,
`/usr/bin/python3.14`, fresh venv, pyarrow 25.0.0 wheel) - intermittent there
(3/6 runs hang) instead of deterministic, path-based reads clean 6/6. So
this is
not specific to one interpreter build; build/timing differences only shift
how
often the race is lost.
#### Environment
- pyarrow 25.0.0 (manylinux wheel from PyPI, installed via `uv`)
- Python 3.13.13 (python-build-standalone, Clang 22.1.3) - deterministic hang
- Python 3.14.4 (distro build, GCC 15.2.0) - intermittent hang (~50%)
- Linux 7.0.0-22-generic x86_64, glibc 2.43
- 24 CPUs
- pandas 3.0.3 (only for the original discovery path; not needed for the
reproducer)
- Persists after OS upgrade + reboot (kernel 7.0.0-22 → 7.0.0-27, glibc
2.43-2ubuntu2)
### Component(s)
Python
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
