keyboardas opened a new issue, #54957:
URL: https://github.com/apache/arrow/issues/54957
### Describe the bug, including details regarding any error messages,
version, and platform.
When a Parquet footer is damaged,
`pq.ParquetFile(path).metadata.row_group(i).column(j)` (and, for a different
kind of damage, `.statistics` on the returned `ColumnChunkMetaData`) can kill
the whole Python process with `std::terminate`. A C++
`parquet::ParquetException` escapes through the Cython wrapper, so Python never
gets a chance to raise or catch anything: the process dies with SIGABRT (exit
status 134 from a shell, `-6` from `subprocess`), and `try/except` around the
call is useless.
I found this by flipping or zeroing a few bytes inside the Thrift
`FileMetaData` of ordinary files written by `pyarrow.parquet.write_table`: 178
of 45,000 mutants (0.4%) aborted the interpreter, and every one of them changed
at most three bytes of the footer. The footer is not checksummed, so a torn
write or a flipped bit on disk produces exactly this input, and a reader that
only wants the footer statistics of many files (and expects
`OSError`/`ArrowInvalid` for a bad one) cannot survive it.
Minimal reproduction (14 lines; one byte of the footer is changed):
```python
import pyarrow as pa
import pyarrow.parquet as pq
pq.write_table(pa.table({"x": pa.array([1, 2, 3], pa.int64())}),
"repro_clean.parquet")
data = bytearray(open("repro_clean.parquet", "rb").read())
# Footer, SchemaElement for "x": repetition_type (Thrift zigzag 2 =
OPTIONAL), name "x".
i = data.index(b"\x25\x02\x18\x01x")
data[i + 1] = 0 # one byte: repetition_type OPTIONAL -> REQUIRED, now at
odds with the column's size_statistics
open("repro_crash.parquet", "wb").write(data)
md = pq.ParquetFile("repro_crash.parquet").metadata # fine: the footer
parses
print("footer opened, calling row_group(0).column(0)", flush=True)
md.row_group(0).column(0) # aborts the process: std::terminate, not a
Python exception
print("not reached")
```
Output (macOS arm64, pyarrow 25.0.1; identical on 26.0.0):
```
footer opened, calling row_group(0).column(0)
libc++abi: terminating due to uncaught exception of type
parquet::ParquetException: Definition level histogram size mismatch, size: 2,
expected: 1
```
The shell reports exit status 134 (SIGABRT). I ran it three times and got
the same result each time.
A second trigger with the same effect, on `.statistics` rather than
`.column()`: change the physical type in the schema so that it disagrees with
the `ColumnMetaData` of the chunk.
```python
import pyarrow as pa
import pyarrow.parquet as pq
pq.write_table(pa.table({"x": pa.array([1.0, 2.0, 3.0], pa.float64())}),
"repro_stats_clean.parquet")
data = bytearray(open("repro_stats_clean.parquet", "rb").read())
# Footer, SchemaElement for "x": type (Thrift zigzag 10 = DOUBLE),
repetition_type, name "x".
i = data.index(b"\x15\x0a\x25\x02\x18\x01x")
data[i + 1] = 0 # one byte: physical type DOUBLE -> BOOLEAN, but the
ColumnMetaData still says DOUBLE
open("repro_stats_crash.parquet", "wb").write(data)
col =
pq.ParquetFile("repro_stats_crash.parquet").metadata.row_group(0).column(0) #
fine
print("column(0) ok, reading .statistics", flush=True)
col.statistics # aborts the process
```
```
column(0) ok, reading .statistics
libc++abi: terminating due to uncaught exception of type
parquet::ParquetException: ColumnMetaData type does not match ColumnDescriptor
physical type: DOUBLE vs. BOOLEAN
```
In my fuzzing run, 65 of the 178 aborts were in `.column()`
(definition/repetition level histogram size mismatches) and 113 were in
`.statistics` (column type mismatches). Walking `num_row_groups`,
`row_group(i)`, `num_columns`, the file-level `num_rows`, and `path_in_schema`
/ `has_min_max` / `min` / `max` of a chunk that did construct never aborted.
The fuzz harness is deterministic and I can attach it if useful.
**Expected behaviour.** A Python exception, not a process abort. The same
validation already surfaces as a regular exception on the read path: for the
file produced by the first snippet,
`pq.ParquetFile("repro_crash.parquet").read_row_group(0)` (and `.read()`) raises
```
OSError: Definition level histogram size mismatch, size: 2, expected: 1
```
so `OSError` (or `ArrowInvalid`) from `column()` / `.statistics` would be
consistent with the rest of the Python API.
**What I think is going on** (from the pyarrow wheel; I have not built from
source). `pyarrow/includes/libparquet.pxd` declares
`CRowGroupMetaData.ColumnChunk(int i)` and `CColumnChunkMetaData.statistics()`
inside `cdef extern from "parquet/api/reader.h" ... nogil:` without `except +`.
`RowGroupMetaData.column()` reaches `parent.metadata.ColumnChunk(i)` through
`ColumnChunkMetaData.init` (`_parquet.pxd`), and
`ColumnChunkMetaData.statistics` calls `self.metadata.statistics()`. Both
appear to run C++ that validates the chunk metadata against the schema
descriptor and throws `ParquetException` when they disagree, and with no
translation at the Cython boundary the exception unwinds into the interpreter
and ends in `std::terminate`. `RowGroupInfo.statistics` in
`_dataset_parquet.pyx` calls `self.metadata.column(i)` and `col.statistics` and
aborts the same way, so `ds.dataset(path, format="parquet").get_fragments()` ->
`fragment.row_groups[i].statistics` is another way to reach it. The
other metadata accessors I tried do not abort, but I have not audited the
rest of the file's `extern` declarations for the same missing `except +`; other
accessors that build per-column metadata may be affected as well.
**Versions.**
- The `.column()` abort (first snippet): aborts on pyarrow 20.0.0, 21.0.0,
22.0.0, 23.0.1, 24.0.0, 25.0.1 and 26.0.0 (26.0.0 is the latest on PyPI today).
15.0.2, 17.0.0, 18.1.0 and 19.0.1 do not abort (on 19.0.1 `column()` returns
normally for this file; `read_row_group(0)` there fails with `OSError:
Unexpected end of stream`).
- The `.statistics` abort (second snippet): aborts on 21.0.0, 22.0.0,
23.0.1, 24.0.0, 25.0.1 and 26.0.0. 15.0.2, 17.0.0, 18.1.0, 19.0.1 and 20.0.0 do
not abort.
- Platform: macOS 27.0.1 (Darwin 27.0.0), Apple M4 (arm64), Python 3.14.8
for 25.0.1 and 26.0.0 and Python 3.12.13 for the older versions; wheels from
PyPI. I have not tried Linux. I would expect the message there to read
`terminate called after throwing an instance of 'parquet::ParquetException'`.
I searched the tracker for this and found nothing that matches. The closest
is GH-45283 ("OSError: Repetition level histogram size mismatch" when reading
parquet file in pyarrow since 19.0.0), which is the same validation reported as
a regular `OSError` on the read path.
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
