Dandandan commented on PR #15697: URL: https://github.com/apache/datafusion/pull/15697#issuecomment-2807089262
> I think this would be nicer (and tie in better with future work 😉) if we essentially followed the structure of #15301 but do the filtering in `TopK` or `SortExec`: > > 1. Keep track of a `thresholds: Arc<RwLock<Vec<Option<ScalarValue>>>>` and `filter: `Option<Arc>`on`TopK`. > 2. For each batch check pass it through the existing filter, if any, and exit early if no rows remain. > 3. If we updated our heap propagate the update to `thresholds` and `filter`. Yeah this sounds like a good idea, let me have a look to make some changes in this direction within the scope of this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
