pingzh opened a new pull request, #5699: URL: https://github.com/apache/datafusion-comet/pull/5699
## Which issue does this PR close? Closes #5695. ## Rationale for this change Selective joins can reject most probe rows after those rows have already been read and decoded. Connect completed native hash-join build domains to eligible Parquet readers so they can skip row groups, while retaining the original join and residual predicates. ## What changes are included in this PR? - Add the default-off `spark.comet.exec.join.dynamicFilter.enabled` option for inner joins with one direct signed integer key and one native partition per input. - Create fresh producer/consumer state for each execution, attach the live predicate to eligible native Parquet readers, and filter decoded probe batches. Preserve build-side swaps, Spark operator structure, and join/scan/filter metric ownership. - Support scan projection remapping and direct-column null-check conjunctions. Stop reader attachment at standalone projections, other predicates, limits, and unsupported readers; do not transport filters across Spark exchanges or JVM/Arrow boundaries. - Add dedicated runtime-filter metrics and user documentation. The original join remains responsible for exact matching, and existing defaults and dependency versions remain unchanged. This includes native Parquet reader integration beyond the probe-batch filtering proposed in #4807 and #4810. Iceberg and Delta kernel reader integration remain separate work. ## How are these changes tested? - Native runtime-filter and planner tests: 14 passed, covering completed and delayed builds, swaps, null/empty/duplicate keys, unsupported shapes, execution isolation and cleanup, reader/filter limits, schema remapping, metric attribution, and clustered versus unclustered Parquet pruning with byte-read assertions. - Native build and clean Spark 4.0 JVM package build passed. - Full Spark 4.0 `CometJoinSuite`: 44 passed, including 13 new runtime-filter regressions with native-plan assertions, both Spark build sides, broadcast/shuffle strategies, AQE, reader pruning, and seeded-randomness boundaries. - Rustfmt, Scalafix, Spotless, Scalastyle, and workspace Clippy with warnings denied passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
