jayzhan211 commented on code in PR #25811:
URL: https://github.com/apache/datafusion/pull/25811#discussion_r4585821006
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1595,53 +1551,28 @@ impl RowGroupsPrunedParquetOpen {
continue;
}
};
- row_group_filters.insert(
- column_name,
- bf,
- *physical_type,
- *type_length,
- );
+ any_filter_loaded = true;
+
+ if self.row_groups.prune_row_group_with_bloom_filter(
Review Comment:
@kosiew the bot run on this PR already covers the wide surviving case:
`slow/survive_all/columns/32` (32 predicate columns, none rejects, 64 row
groups, 200 µs added to every request) takes 2.7 s on both main and the PR.
That run was on 2af7118ef6, whose load loop matches the current head apart from
moving code into helpers. Main already awaits each filter read in turn, so the
only new gap between reads is the check itself. The whole in-memory
`survive_all/columns/32` query, all 2,048 checks included, takes 14.7 ms on the
PR (27.6 ms on main), and that gap does not grow with store latency.
Does this settle the scheduling concern for you, or is there a workload you
want measured before approving?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]