comphead commented on code in PR #3540:
URL: https://github.com/apache/iceberg-rust/pull/3540#discussion_r4407249737
##########
crates/iceberg/src/arrow/reader/row_filter.rs:
##########
@@ -62,8 +70,18 @@ impl ArrowReader {
// After collecting required leaf column indices used in the predicate,
// creates the projection mask for the Arrow predicates.
let projection_mask = ProjectionMask::leaves(parquet_schema,
column_indices.clone());
- let predicate_func = visit(&mut converter, predicates)?;
- let arrow_predicate = ArrowPredicateFn::new(projection_mask,
predicate_func);
+ let mut predicate_func = visit(&mut converter, predicates)?;
+ // Predicates are bound to the table schema, so evaluate them on the
filter columns
+ // converted to it, as the reader's output is (e.g. a file written
before an `int` ->
+ // `long` promotion).
+ let mut transformer =
+ RecordBatchTransformerBuilder::new(snapshot_schema,
&field_ids).build();
Review Comment:
Done in 03639f1bc, thanks for measuring. I used `as_struct().field_by_id`,
the top-level lookup, so a nested field can't become a top-level column that
the transformer fills with nulls. The converter rejects nested predicates
before this today, but #2906 would lift that. Opened #3572 for the output
transformer and linked it from the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
