HippoBaro commented on code in PR #11235:
URL: https://github.com/apache/arrow-rs/pull/11235#discussion_r4140879056
##########
parquet/src/arrow/push_decoder/remaining.rs:
##########
@@ -122,6 +122,16 @@ impl QueuedRowGroups {
}
}
+ /// Returns `true` if `row_group_idx` is in the queue.
+ fn contains(&self, row_group_idx: usize) -> bool {
+ match self {
+ Self::Global { row_groups, .. } =>
row_groups.contains(&row_group_idx),
+ Self::PerRowGroup(row_groups) => row_groups
+ .iter()
+ .any(|row_group| row_group.row_group_index == row_group_idx),
+ }
+ }
Review Comment:
This searches the remaining queue after every completed row group. For a
normal scan of N distinct groups, this mean quadratic aggregate work. Could we
maintain remaining occurrence counts as the queue advances, rather than
repeatedly scanning it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
