adriangb commented on PR #15301: URL: https://github.com/apache/datafusion/pull/15301#issuecomment-2762301099
From discussion with Andrew here are a couple notes: - The most granular update frequency of the filters is when the TopK itself updates, so we should switch from polling to pushing: `TopKDynamicFilterSource` should just hold a reference to it's current filter and the TopK should update `TopKDynamicFilterSource` when it gets updated and not the other way around. - When you have say 10 partitions you end up with 11 TopK operators: one per partition + a global one on top. Can we push down filters from the global TopK? My suggestion was that yes we can if we have each TopK accept filters from above it and forward them on. - Can we update the dynamic filters for each batch that gets read inside of the openers? I think we might be able to, might require some trickery though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
