Copilot commented on code in PR #27079:
URL: https://github.com/apache/datafusion/pull/27079#discussion_r4476828930
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1426,6 +1436,29 @@ impl RowGroupsPrunedParquetOpen {
return Ok(true);
}
}
+ // A row-reversing scan under a limit reads only the tail of the first
+ // row group it reaches when that row group's output is known up
+ // front (see `PreparedAccessPlan::select_tail_for_limit`). The
+ // offset index lets the decoder skip the pages ahead of that tail
+ // instead of fetching the whole column chunk.
+ let prepared = &self.prepared.loaded.prepared;
+ if prepared.reverse_rows
+ && let Some(limit) = prepared.limit
+ && let Some(first_read) = row_groups.row_group_indexes().last()
Review Comment:
`first_read` is derived using `.last()`, but the comment says this should be
the *first* row group reached in scan order. For a reversed scan, `.last()`
will typically point at the final row group in the prepared plan rather than
the first one read, which can prevent offset-index loading when it would be
beneficial (or enable it for the wrong row group). Use the first row group in
scan order (e.g., `.first()` on the scan-ordered list) or otherwise explicitly
choose the row group that will actually be read first under the current
row-group order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
