zhuqi-lucas commented on code in PR #11532:
URL: https://github.com/apache/arrow-rs/pull/11532#discussion_r4255387779
##########
parquet/src/column/reader.rs:
##########
@@ -310,6 +314,11 @@ where
///
/// Returns the number of records skipped
pub fn skip_records(&mut self, num_records: usize) -> Result<usize> {
+ // A page skipped whole costs nothing, but a page walked through to
+ // reach a selected row inside it is decompressed and then thrown away.
+ // Attribute that to skipping rather than to reading.
+ self.page_reader
+ .set_decompression_phase(DecompressionPhase::Skip);
Review Comment:
Good catch, fixed in fb8ca9977: a page decompressed while skipping, and the
chunk's dictionary, are now held until it is known whether any of their values
is read. Read from, they count as Read; walked past or dropped, as Skip. Added
a test for skip(1) then reading the rest (all Read) and a list-column selective
read (pages with a kept row Read, pages between Skip).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
