adriangb commented on code in PR #15301:
URL: https://github.com/apache/datafusion/pull/15301#discussion_r2020152758
##########
datafusion/physical-plan/src/topk/mod.rs:
##########
@@ -186,6 +235,90 @@ impl TopK {
Ok(())
}
+ fn calculate_dynamic_filters(
+ thresholds: Vec<ColumnThreshold>,
+ ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+ // Create filter expressions for each threshold
+ let mut filters: Vec<Arc<dyn PhysicalExpr>> =
Review Comment:
I wonder: is there any way we could make only the right hand side of `col >
123` "dynamic"? That is: could we return `col > <dynamic thing>` instead of
`<dynamic thing>`? I guess we'd need to some sort of `PhysicalExpr` that
`BinaryExpr` and others can work with? I don't know how `BinaryExpr` is
implemented internally.
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -1224,6 +1245,28 @@ impl ExecutionPlan for SortExec {
.with_preserve_partitioning(self.preserve_partitioning()),
)))
}
+
+ // Pass though filter pushdown.
+ // This often happens in partitioned plans with a TopK because we end up
with 1 TopK per partition + a final TopK at the end.
+ // Implementing this pass-through allows global/top/final TopK to push
down filters to the partitions.
+ fn push_down_filter(
+ &self,
+ expr: Arc<dyn PhysicalExpr>,
+ ) -> Result<Option<Arc<dyn ExecutionPlan>>> {
Review Comment:
This won't do anything for the multi-partition case _until_ we implement
filter pushdown for `SortingMergeExec`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
