alamb commented on PR #15301: URL: https://github.com/apache/datafusion/pull/15301#issuecomment-2759867166
> Hmm interesting about the join conditions. I have not thought about how this will interact with a join in the same query. > > Another concern with a dynamic physicalexpr: more lock contention. Presumably every time it's evaluated (for each row?) we need to acquire a lock to read from the TopK heap. I agree we would have to be careful -- maybe like updating on each update to the Heap (aka once per batch processed or something) > And for using an optimizer rule: would that be compatible with custom ExecutionPlans? As this is currently implemented it's possible for a custom execution plan to accept and/or forward the dynamic filters downstream. Could I implement eg a CachedParquetExec that could have the dynamic filters pushed down via an optimizer rule? I am not sure how the filters can be pushed down. I actually doubled checked and it seems like we don't have filter pushdown for ExecutionPlans yet (only LogicalPlans). https://github.com/apache/datafusion/tree/main/datafusion/core/tests/physical_optimizer InfluxDB has an implementation for physical plans 🤔 I suppose we could make a physical filter pushdown plan starting with the TopK Though now this is starting to sound crazy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
