pingzh opened a new pull request, #7657: URL: https://github.com/apache/datafusion-comet/pull/7657
## Which issue does this PR close? Part 2 of #5774, following the merged native reader-filter foundation in #7598. This PR does not close the issue. ## Rationale for this change For `SELECT MIN(id) FROM parquet_table`, each Spark task can tighten its bound as it reads input and skip later row groups that cannot improve its partial result. The final aggregate combines those partial results as usual. ## What changes are included in this PR? Add `spark.comet.exec.aggregate.dynamicFilter.enabled`, disabled by default. Capture the flag in the aggregate protobuf and include it in Spark plan equality, so execution and plan reuse preserve the setting chosen during planning. When enabled, the Rust planner uses `MinMaxReaderFilterExec` for the narrow subset supported by #7598: one unfiltered MIN or MAX over a direct signed-integer column, without grouping, in raw-input Partial mode over an eligible native Parquet scan. Bounds remain local to each execution. Unsupported plans retain ordinary aggregation; Final/PartialMerge stages and JVM boundaries remain ineligible. Only the enabled path removes a matching integer column's identity cast, allowing DataFusion to recognize the bound producer. The disabled path retains that cast and avoids DataFusion's otherwise unconsumed bound updates. Spark displays reader-filter attachment counters on aggregates; pruning and byte metrics remain on scans. Register the new Spark suite in Linux and macOS CI. Benchmarks and detailed tuning documentation remain for PR 3; the configuration description documents the current limits. ## How are these changes tested? - `CometMinMaxSuite`: 10 tests passed on Spark 4.1.3. Filtering off/on matches Spark for all four signed integer types, empty/null/duplicate input, multiple Spark partitions with AQE off/on, and unsupported plans. Tests verify actual row-group/row/byte reductions, config capture and canonicalization, and a Comet Partial above a JVM-to-Arrow boundary with no reader attachment. - Native planner tests: 128 passed, including three new tests for serialized flag handling, pruning and metric ownership, ordinary stage/JVM fallback, retained real casts, and no dynamic expressions when disabled. - Existing dynamic-filter tests: 75 passed. - Workspace Clippy with warnings denied, Scala style checks, Spotless, Rust formatting, and whitespace checks passed. - Two independent source reviews completed. - Source-archive license check fails identically on this branch and upstream base `36ab96ad8`: the unchanged `spark/src/test/resources/test-data/text-test-1.txt` fixture from #6784 is not approved by RAT. There are no license findings in the files added by this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
