HippoBaro commented on code in PR #11262:
URL: https://github.com/apache/arrow-rs/pull/11262#discussion_r4210610627
##########
parquet/src/column/reader.rs:
##########
@@ -484,15 +496,46 @@ where
)?;
offset += bytes_read;
+ if normalize_fixed {
+ // V1 has no non-null count. Both strict
PLAIN validation
+ // and legacy repair need the physical
count: nulls and
+ // nested placeholders have no values in
the data section.
Review Comment:
Thanks @etseidl! I tried a prototype along these lines; it’s effectively on
par with baseline for the original round-trip cases. The issue I have with it
is deferred validation. At least for what I came up with, malformed pages can
yield rows before an error, and partial reads may never detect it. Matching
PLAIN prefixes also aren’t conclusive (valid payloads can look identical) so
those cases would still need a (slow) independent prescan.
I'm still on it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
