Klavinco commented on issue #50503:
URL: https://github.com/apache/arrow/issues/50503#issuecomment-5024137362
**Additional data points + a public 6-line reproducer** (independent hit of
what looks like the same bug)
Environment: AWS c7g.large (Graviton3 / Neoverse V1; `sve svebf16 svei8mm`
in cpuinfo), Ubuntu 24.04 aarch64, kernel 6.17.0-1019-aws, CPython 3.13,
manylinux aarch64 wheel `pyarrow==25.0.0`. The same versions on Apple Silicon
macOS (NEON-only arm64) are **not** affected — consistent with the SVE-dispatch
suspicion.
**1. Public minimal repro — no proprietary data needed.** A single
dictionary-encoded int64 column with 300 distinct values comes back with
silently duplicated values (no crash, no error):
```python
import pyarrow as pa, pyarrow.parquet as pq
vals = list(range(100_000, 100_300))
pq.write_table(pa.table({"x": pa.array(vals, pa.int64())}),
"/tmp/min.parquet")
back = pq.read_table("/tmp/min.parquet")["x"].to_pylist()
print("match:", back == vals, "| unique:", len(set(back)), "of", len(back))
# Graviton3 + pyarrow 25.0.0: match: False | unique: 84 of 300
# same + ARROW_USER_SIMD_LEVEL=NONE: match: True | unique: 300 of 300
# same box + pyarrow 24.0.0: match: True | unique: 300 of 300
```
Unlike the SIGSEGV / "Index not in dictionary bounds" reported above, on our
data the flat-column failure mode is **silent wrong data** (values repeated
from earlier in the page), fully deterministic across runs, identical with
`use_threads=False` — so the manifestation seems data-shape-dependent, which
makes the silent case the scarier one.
**2. `ARROW_USER_SIMD_LEVEL=NONE` fully fixes every read path on this box**
(the sweep offered above) — pointing at a runtime-dispatched SIMD kernel.
**3. Not codec-related.** Re-encoding the same data as zstd, snappy, and
uncompressed gives identical corruption; writing with `use_dictionary=False`
avoids it for flat columns → the corrupt path is the RLE/bit-packed
dictionary-index decode.
**4. Repetition/definition levels are also affected** (same RLE/bit-packed
decoder): reading tables that contain a `list<struct<decimal128, decimal128>>`
column on this box raises e.g. `OSError: Malformed levels. min: 0 max: 21827
out of range. Max Level: 1` or `ArrowInvalid: Column 7 named asks expected
length 218 but got length 239` (the message varies between layouts), **even for
files written with `use_dictionary=False`**. So the blast radius is wider than
dictionary decode.
**5. Write path appears unaffected:** parquet written by 25.0.0 on the same
box is read back correctly by polars 1.42.1 and by pyarrow 24.0.0 (update-id
chains over millions of rows verify clean).
Happy to run candidate wheels or further env sweeps on this box.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
