jaideeppyne commented on code in PR #11136:
URL: https://github.com/apache/arrow-rs/pull/11136#discussion_r4475395308
##########
parquet/src/encodings/decoding.rs:
##########
@@ -2190,6 +2190,35 @@ mod tests {
test_byte_stream_split_decode::<Int64Type>(data, -1);
}
+ #[test]
+ fn test_byte_stream_split_uses_encoded_value_count() {
+ let mut decoder = ByteStreamSplitDecoder::<DoubleType>::new();
+
+ decoder.set_data(Bytes::from(vec![0; 64]), 63).unwrap();
Review Comment:
Both BYTE_STREAM_SPLIT decoder variants now derive their count from the
encoded bytes and retain the partial-value check in `set_data`, so level-count
estimates can overstate the physical values. The column-reader workaround is
removed. Tests cover oversized estimates, partial reads, skipping, and both
variable-width join paths. The real malformed-Parquet regression also returns
an ordinary insufficient-values error instead of panicking. Published in
b280e76; [focused tests and Clippy
passed](https://github.com/jaideeppyne/arrow-rs/actions/runs/41945965606).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
