jaideeppyne commented on PR #11136: URL: https://github.com/apache/arrow-rs/pull/11136#issuecomment-6646816689
@alamb, the panic is at `sub_src[i + j * stride]` in [`join_streams_const`](https://github.com/apache/arrow-rs/blob/32902415b841ac824470d8ee9ffd4b785d8b2a9a/parquet/src/encodings/decoding/byte_stream_split_decoder.rs#L68), reached through the public `ParquetRecordBatchReaderBuilder` when a page's value count exceeds its encoded payload. Here are actual files and a runnable public-API program: - [Malformed Parquet, 487 bytes](https://raw.githubusercontent.com/jaideeppyne/arrow-rs/3f70bb88cb0c1029e84fcd297fe6a742dc13500b/malformed-bss.parquet) - [Valid control](https://raw.githubusercontent.com/jaideeppyne/arrow-rs/3f70bb88cb0c1029e84fcd297fe6a742dc13500b/valid-bss.parquet) - [Reproducer source](https://raw.githubusercontent.com/jaideeppyne/arrow-rs/3f70bb88cb0c1029e84fcd297fe6a742dc13500b/reproduce.rs) and [run instructions](https://github.com/jaideeppyne/arrow-rs/blob/3f70bb88cb0c1029e84fcd297fe6a742dc13500b/README.md) The writer creates eight doubles and the control reads all eight back. Only zero-based byte 14 changes, from 16 to 126, declaring 63 values while the payload and footer stay in place. [Verification](https://github.com/jaideeppyne/arrow-rs/actions/runs/41945965606) observed `index out of bounds: the len is 56 but the index is 56` on unchanged upstream. With b280e76, the identical file returns an insufficient-values error. The update also follows @jhorstmann's suggestion: both decoder variants use the encoded count, and the column-reader workaround is removed. The new integration test exercises this file-construction path directly. Focused tests, Clippy, and formatting passed; full fork CI is running. This follow-up and its reproducer were prepared with OpenAI Codex assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
