pingzh opened a new pull request, #7657:
URL: https://github.com/apache/datafusion-comet/pull/7657

   ## Which issue does this PR close?
   
   Part 2 of #5774, following the merged native reader-filter foundation in 
#7598. This PR does not close the issue.
   
   ## Rationale for this change
   
   For `SELECT MIN(id) FROM parquet_table`, each Spark task can tighten its 
bound as it reads input and skip later row groups that cannot improve its 
partial result. The final aggregate combines those partial results as usual.
   
   ## What changes are included in this PR?
   
   Add `spark.comet.exec.aggregate.dynamicFilter.enabled`, disabled by default. 
Capture the flag in the aggregate protobuf and include it in Spark plan 
equality, so execution and plan reuse preserve the setting chosen during 
planning.
   
   When enabled, the Rust planner uses `MinMaxReaderFilterExec` for the narrow 
subset supported by #7598: one unfiltered MIN or MAX over a direct 
signed-integer column, without grouping, in raw-input Partial mode over an 
eligible native Parquet scan. Bounds remain local to each execution. 
Unsupported plans retain ordinary aggregation; Final/PartialMerge stages and 
JVM boundaries remain ineligible.
   
   Only the enabled path removes a matching integer column's identity cast, 
allowing DataFusion to recognize the bound producer. The disabled path retains 
that cast and avoids DataFusion's otherwise unconsumed bound updates. Spark 
displays reader-filter attachment counters on aggregates; pruning and byte 
metrics remain on scans.
   
   Register the new Spark suite in Linux and macOS CI. Benchmarks and detailed 
tuning documentation remain for PR 3; the configuration description documents 
the current limits.
   
   ## How are these changes tested?
   
   - `CometMinMaxSuite`: 10 tests passed on Spark 4.1.3. Filtering off/on 
matches Spark for all four signed integer types, empty/null/duplicate input, 
multiple Spark partitions with AQE off/on, and unsupported plans. Tests verify 
actual row-group/row/byte reductions, config capture and canonicalization, and 
a Comet Partial above a JVM-to-Arrow boundary with no reader attachment.
   - Native planner tests: 128 passed, including three new tests for serialized 
flag handling, pruning and metric ownership, ordinary stage/JVM fallback, 
retained real casts, and no dynamic expressions when disabled.
   - Existing dynamic-filter tests: 75 passed.
   - Workspace Clippy with warnings denied, Scala style checks, Spotless, Rust 
formatting, and whitespace checks passed.
   - Two independent source reviews completed.
   - Source-archive license check fails identically on this branch and upstream 
base `36ab96ad8`: the unchanged 
`spark/src/test/resources/test-data/text-test-1.txt` fixture from #6784 is not 
approved by RAT. There are no license findings in the files added by this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to