tustvold commented on code in PR #4376:
URL: https://github.com/apache/arrow-rs/pull/4376#discussion_r1223323728
##########
parquet/src/arrow/record_reader/mod.rs:
##########
@@ -201,31 +149,10 @@ where
///
/// Number of records skipped
pub fn skip_records(&mut self, num_records: usize) -> Result<usize> {
- // First need to clear the buffer
- let end_of_column = match self.column_reader.as_mut() {
- Some(reader) => !reader.peek_next()?,
- None => return Ok(0),
- };
-
- let (buffered_records, buffered_values) =
- self.count_records(num_records, end_of_column);
-
- self.num_records += buffered_records;
- self.num_values += buffered_values;
-
- let remaining = num_records - buffered_records;
-
- if remaining == 0 {
- return Ok(buffered_records);
+ match self.column_reader.as_mut() {
+ Some(reader) => reader.skip_records(num_records),
+ None => Ok(0),
Review Comment:
Yes, RecordReader no longer performs this intermediate buffering, it only
buffers requested rows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
