pingzh opened a new pull request, #7683:
URL: https://github.com/apache/datafusion-comet/pull/7683

   ## Which issue does this PR close?
   
   Part of #5774. Follows the merged reader support (#7598) and Spark 
integration (#7657).
   
   ## Rationale for this change
   
   MIN/MAX reader filtering can skip later Parquet row groups after a task 
observes a useful bound. The benefit depends on data order and task boundaries, 
while footer inspection and bound updates add overhead. This adds an off/on 
benchmark and documents eligibility, fallback limits, and metrics before 
considering broader enablement.
   
   ## What changes are included in this PR?
   
   - `CometMinMaxBenchmark`: separate MIN and MAX queries over ascending, 
descending, deterministic random, and constant INT data, with one or four scan 
tasks by default. It checks Spark result parity, the direct scan/partial 
aggregate plan, actual task count, and per-task filter attachment before timing.
   - Scan and attachment metrics from untimed executions, plus elapsed query 
time in both configurable mode orders. Uncompressed Parquet v1 pages, disabled 
dictionaries, and disabled page-index/decoder filtering provide a controlled 
row-group comparison, including constant values.
   - User-guide configuration, supported plans, footer limits, fallback 
behavior, and metric interpretation; contributor instructions for reproducing 
measurements.
   
   The existing configuration remains disabled by default. Runtime behavior and 
supported scope are unchanged. The two commits separate the benchmark from 
documentation.
   
   ## How are these changes tested?
   
   - Native release build passed, with `-Ctarget-cpu=native 
-Clink-arg=-fuse-ld=bfd` and the standard release profile/default features.
   - JVM reactor `test-compile`, including formatting and style checks, passed. 
Packaged binary/source jar-content checks passed (the final check was rerun 
with JDK tools on PATH).
   - Both complete benchmark runs passed: 16 query/layout/task-count 
combinations × two modes × two mode orders. Results matched Spark, enabled 
filters attached once per scan task, and all pruning metrics agreed across mode 
orders.
   - `git diff --check` passed.
   
   ### Local release measurements
   
   Environment: AMD EPYC-Genoa, 32 logical CPUs, about 98 GiB RAM, Ubuntu 
24.04.4/Linux 6.8; Spark 4.1.3, OpenJDK 21.0.13.0.3, Rust 1.99.1. Spark used 
`local[1]` and an 8 GiB benchmark JVM heap. Production code is from 
`d187e6823`; the benchmark is in this PR's `d2f9fdfc6` commit. Each fixture 
contained 8,388,608 rows and nine row groups per file. Four scan tasks ran 
serially under `local[1]`.
   
   Equivalent standard invocations after the same release build:
   
   ```sh
   COMET_BENCHMARK_MASTER='local[1]' make BENCH_HEAP=8g 
benchmark-org.apache.spark.sql.benchmark.CometMinMaxBenchmark --   8388608 1,4 
ascending,descending,random,constant 5 off,on
   COMET_BENCHMARK_MASTER='local[1]' make BENCH_HEAP=8g 
benchmark-org.apache.spark.sql.benchmark.CometMinMaxBenchmark --   8388608 1,4 
ascending,descending,random,constant 5 on,off
   ```
   
   Favorable ordering and constant values reduced emitted scan rows and 
requested data bytes by about 87.5%. The unfavorable ordered cases pruned no 
groups and expose filtering overhead. Random layouts pruned some groups but did 
not consistently improve elapsed time. These short local runs are sensitive to 
warmup and long-tail outliers; the first ascending-MIN timing even changes 
direction when case order reverses. This is evidence of scan-work reduction and 
overhead cases, not a general speedup claim or a basis to enable the feature by 
default.
   
   <details>
   <summary>All elapsed times and scan-work counters</summary>
   
   Times are mean ± standard deviation in milliseconds, always shown as **off → 
on**, regardless of execution order. Every off execution emitted 8,388,608 rows 
and pruned zero groups. The byte column is `bytes_scanned` in bytes, excluding 
metadata. All on attachment counts equaled the scan-task count; skipped counts 
were zero. Statistics, page-index, and decoder pruning counters were zero.
   
   | Tasks | Layout | Aggregate | On scan rows | Groups pruned | Data bytes off 
→ on | Mean ± SD, off-first | Mean ± SD, on-first |
   | ---: | --- | --- | ---: | ---: | --- | --- | --- |
   | 1 | ascending | MIN | 1,048,287 | 8 | 33,566,729 → 4,194,680 | 176 ± 21 → 
101 ± 22 | 77 ± 10 → 135 ± 96 |
   | 1 | ascending | MAX | 8,388,608 | 0 | 33,566,729 → 33,566,729 | 105 ± 4 → 
98 ± 5 | 81 ± 3 → 96 ± 7 |
   | 1 | descending | MIN | 8,388,608 | 0 | 33,566,728 → 33,566,728 | 73 ± 4 → 
84 ± 6 | 69 ± 3 → 68 ± 6 |
   | 1 | descending | MAX | 1,048,287 | 8 | 33,566,728 → 4,194,682 | 74 ± 5 → 
48 ± 5 | 87 ± 12 → 44 ± 3 |
   | 1 | random | MIN | 3,144,861 | 6 | 33,570,132 → 12,585,317 | 63 ± 4 → 67 ± 
10 | 62 ± 4 → 54 ± 6 |
   | 1 | random | MAX | 5,241,435 | 4 | 33,570,132 → 20,975,530 | 62 ± 2 → 62 ± 
1 | 65 ± 3 → 63 ± 3 |
   | 1 | constant | MIN | 1,048,287 | 8 | 33,566,756 → 4,194,685 | 53 ± 9 → 102 
± 152 | 60 ± 4 → 42 ± 8 |
   | 1 | constant | MAX | 1,048,287 | 8 | 33,566,756 → 4,194,685 | 56 ± 4 → 59 
± 14 | 64 ± 9 → 48 ± 1 |
   | 4 | ascending | MIN | 1,048,196 | 32 | 33,567,471 → 4,194,401 | 78 ± 3 → 
73 ± 15 | 76 ± 1 → 66 ± 3 |
   | 4 | ascending | MAX | 8,388,608 | 0 | 33,567,471 → 33,567,471 | 70 ± 3 → 
87 ± 18 | 79 ± 5 → 107 ± 21 |
   | 4 | descending | MIN | 8,388,608 | 0 | 33,567,469 → 33,567,469 | 73 ± 2 → 
88 ± 15 | 79 ± 2 → 94 ± 5 |
   | 4 | descending | MAX | 1,048,196 | 32 | 33,567,469 → 4,194,400 | 59 ± 2 → 
69 ± 17 | 74 ± 4 → 68 ± 15 |
   | 4 | random | MIN | 3,144,588 | 24 | 33,571,051 → 12,584,536 | 61 ± 2 → 72 
± 17 | 61 ± 3 → 64 ± 2 |
   | 4 | random | MAX | 3,406,637 | 23 | 33,571,051 → 13,633,247 | 63 ± 4 → 179 
± 196 | 67 ± 2 → 125 ± 46 |
   | 4 | constant | MIN | 1,048,196 | 32 | 33,567,496 → 4,194,404 | 75 ± 3 → 64 
± 14 | 95 ± 9 → 50 ± 14 |
   | 4 | constant | MAX | 1,048,196 | 32 | 33,567,496 → 4,194,404 | 68 ± 2 → 58 
± 1 | 81 ± 11 → 68 ± 13 |
   
   Reader `scan_io_data_bytes` equaled `bytes_scanned`. 
`scan_io_metadata_bytes` was 524,288 bytes per task in each mode (524,288 for 
one task and 2,097,152 for four); attachment does not eliminate footer 
inspection. Timings include planning and result collection, while fixture 
generation, Spark parity checks, plan validation, and metric reporting are 
outside the timed cases.
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to