brgr-s commented on code in PR #2961:
URL: https://github.com/apache/iceberg-rust/pull/2961#discussion_r3728609923
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -400,13 +395,12 @@ impl FileScanTaskReader {
use_position_fallback,
)?;
- let row_filter = ArrowReader::get_row_filter(
+ row_filter_predicates.push(ArrowReader::build_scan_predicate(
Review Comment:
TBH, I'd keep it that way at the moment.
The machinery does exist in DataFusion, but it is opt-in (`reorder_filters`
defaults to `false`).
The heuristic is compressed column byte size, as you state, so a cpu-heavy
predicate over column A cannot be distinguished from a cheaper hash probe over
the same column A. Across different columns, a CPU heavy predicate over a
narrow column sorts before a one lookup probe over a narrow one, regardless
which one is cheaper or more selective.
It is a good proxy for decoder effort, which is frequently the right thing
to optimize, but it is not a proxy for evaluation cost, which is what this PR
aims to improve. Getting the order right would need a cost model for
selectivity or per-predicate cost.
I think this is more a follow-up situation if a workload is discovered where
the order measurably matters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
