gene-bordegaray commented on code in PR #20331:
URL: https://github.com/apache/datafusion/pull/20331#discussion_r2801905807
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -417,6 +419,14 @@ impl FileOpener for ParquetOpener {
predicate = predicate
.map(|p| simplifier.simplify(rewriter.rewrite(p)?))
.transpose()?;
+
+ // Snapshot per-partition dynamic filters if available.
+ // When both sides of a hash join preserve their file
partitioning, each partition gets
+ // its own filter.
+ predicate = predicate
+ .map(|p| snapshot_physical_expr_for_partition(p,
partition_index))
+ .transpose()?;
Review Comment:
decided to only do this in the parquet opener, if we did for all files (by
default) we would need some type of fallback or just do nothing. Given this is
a large PR, didn't want to include logic for a fallback and doing nothing
seemed out of place, could still reconsider if others have an opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
