comphead commented on PR #103: URL: https://github.com/apache/datafusion-site/pull/103#issuecomment-3275523866
> > @adriangb @alamb what is still unclear for me, how the dynamic filter gets the min value? is it under the hood by scanning lets say stats, like parquet file footers or histograms? > > @comphead -- the min value is the minimum value of what has _already been seen_ during query execution. > > * So when the query starts, there is no min value. > * As data flows through and the top K operator starts updating the heap, the min value is set > * As more data flows through and the heap is refined (new values potentially replace existing values) then the min value can be updated as well > > The TopK operator heap here: https://github.com/apache/datafusion/blob/ab108a50d75e4e12fb6ebbfac0d0bffa24c265ea/datafusion/physical-plan/src/topk/mod.rs#L119 Thanks @alamb I was referring to statement ``` Figure 3 is better, but it still reads and decodes all 100M rows of the hits table, which is often unnecessary once we have found the top 10 rows ``` which implies the optimization doesn't have to read and decode 100M rows to get top10, and I cannot see how this is exactly happening 🤔 This is makes sense with the filter, but to get the min value for the filter we still to full scan, that is something i'm still missing, lets go ahead, yes, thanks for explanations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
