gene-bordegaray commented on code in PR #20331:
URL: https://github.com/apache/datafusion/pull/20331#discussion_r2801905807
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -417,6 +419,14 @@ impl FileOpener for ParquetOpener {
predicate = predicate
.map(|p| simplifier.simplify(rewriter.rewrite(p)?))
.transpose()?;
+
+ // Snapshot per-partition dynamic filters if available.
+ // When both sides of a hash join preserve their file
partitioning, each partition gets
+ // its own filter.
+ predicate = predicate
+ .map(|p| snapshot_physical_expr_for_partition(p,
partition_index))
+ .transpose()?;
Review Comment:
decided to only do this in the parquet opener, if we did for all files (by
default) just do nothing since predicates aren't passed to other opneers. This
does mean that users will have to implement this for their on data sources.
Given this is a large PR, didn't want to include logic for a fallback and
doing nothing seemed out of place, could still reconsider if others have an
opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
