unikdahal opened a new issue, #6588: URL: https://github.com/apache/datafusion-comet/issues/6588
I'm exploring pushing Comet's execution-time predicates into the native Iceberg reader. This overlaps with #4808, so the intent is not to introduce another independent join dynamic-filter implementation. For hash joins, the current direction is to reuse the dynamic-filter machinery already landed in #5699 and make the Iceberg scan another consumer of those runtime bounds. The additional part I'm experimenting with is using the same Iceberg runtime-predicate path for multiple producers: - hash-join build-side bounds - improving MIN/MAX bounds - improving local TopK thresholds For join filtering, a useful predicate may already be available before the probe scan starts. For MIN/MAX and TopK, the interesting case is different: the scan can start without a useful bound, then execution produces a better bound after reading some data. If the Iceberg reader can observe that updated predicate, it can prune row groups that have not yet been read. The reader-side API for live predicate refresh is being explored separately in iceberg-rust: https://github.com/apache/iceberg-rust/issues/3336 ## Early prototype results Initial before/after measurements compare the baseline native library against the experimental runtime-pruning build using the same persisted Iceberg data file. Setup: - Spark 4.1 - 4M rows - ~1.06 GB Parquet file - 126 row groups - sorted integer key - five alternating before/after pairs - two warm-up executions before measurement - warm local filesystem | Query | Reader bytes before | Reader bytes after | Byte reduction | Median time before | Median time after | Time reduction | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Selective hash join | ~1.056 GB | ~1.63 MB | ~99.85% | ~468 ms | ~202 ms | ~56.7% | | MIN | ~16.5 MB | ~0.65 MB | ~96.1% | ~93 ms | ~82 ms | ~12.0% | | TopK LIMIT 10 | ~1.056 GB | ~8.9 MB | ~99.16% | ~345 ms | ~82 ms | ~76.1% | | Non-selective join control | ~1.056 GB | ~1.056 GB | 0% | ~420 ms | ~418 ms | ~0.6% | For the selective cases, the experimental reader prunes 125 of 126 row groups in this ordered fixture. The non-selective join is included as a control: the runtime-predicate path is active, but the predicate cannot eliminate any row groups. Reader bytes remain unchanged and the timing difference is essentially noise. These are early warm-local measurements on intentionally pruning-friendly data, so I'm treating them as validation of the mechanism rather than general performance claims. I'm mainly opening this issue to avoid duplicated work and get feedback on how this should build on #4808 and #5699 before splitting the experiment into upstream PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
