comphead opened a new issue, #27279: URL: https://github.com/apache/datafusion/issues/27279
### Is your feature request related to a problem or challenge? Part of #26169, following [this plan](https://github.com/apache/datafusion/issues/26169#issuecomment-6744990263): compare the plans first, then the execution. At SF1000, DataFusion was 1.74x slower than Polars on Q01 ([results](https://github.com/apache/datafusion/issues/26169#issuecomment-6744781906)). **Setup:** DataFusion `main` at 14b3edd9a (`dfbench`, fat LTO) and Polars `main` at pola-rs/polars@09251856d4 (streaming engine, release build features). Both run the same `benchmarks/queries/q1.sql` on the same TPC-H SF10 Parquet files (16 files, 64 row groups), on an Apple M5 Max (6 + 12 cores) with 18 threads. The runs are warm, with 5 interleaved rounds. **The plans are the same.** Q01 has no joins. Both engines read 7 of 16 columns and compute `l_extendedprice * (1 - l_discount)` once. Statistics prune nothing. Polars applies the filter inside the scan, and `pushdown_filters = true` changes nothing for DataFusion. | SF10, median ms | DataFusion | Polars | |---|---:|---:| | default | 237 | 211 | | `target_partitions = 64` | 202 | | | 1 thread | about 2500 | about 2200 | At 18 threads DataFusion reports less CPU than Polars (2.7 s vs 3.3 s, as each engine measures it). So most of the wall-time gap comes from how the work is spread. **Findings** 1. **Scan work granularity.** `SharedWorkSource` (#21351) shares the 33 byte-range file splits, not row groups. The per-partition `BaselineMetrics` show partitions reading between 2.69M and 4.79M rows. The partial aggregates finish between 125 and 211 ms of a 211 ms query. With `target_partitions = 64` each split holds about one row group, and DataFusion is about 5% faster than Polars, which hands out morsels of about 105K rows. At SF1 (one row group per file) at most 16 of the 18 partitions get rows. 2. **Group accumulators stall on store-to-load forwarding.** Q01 has 4 groups, and 64% of consecutive rows fall in the same group as the previous row. So `values[g] += v` in `PrimitiveGroupsAccumulator`, `AvgGroupsAccumulator` and `CountGroupsAccumulator` waits for the previous store. The 8 accumulators take about 760 ms of the single-threaded run (Polars about 330 ms). Polars spreads a small number of groups over 64 slots (pola-rs/polars#29613). An `i128` sum over Q01's group sequence from one SF10 file takes 1.55 ns/row with one slot per group and 0.43 ns/row with 16 lanes. 3. **Decimal `+`, `-` and `*`.** `1 - l_discount`, `1 + l_tax` and the two multiplies take about 480 ms single-threaded (Polars about 215 ms). The profile shows `try_unary` calling the `decimal_op` closure out of line for every value, as in apache/arrow-rs#11529. apache/arrow-rs#12062 replaces these kernels for `add`, `sub` and `mul`. Group keys (#18411) and ZSTD cost about the same in both engines. Polars spends about 2.7x more CPU than DataFusion on Parquet decoding and filtering (ZSTD excluded), which hides the gaps above in the single-threaded totals. ### Describe the solution you'd like - Share row groups (or row ranges) between sibling `FileStream`s, as prototyped in #21766 and #23285, or through #21719. - Spread the state of single-value `GroupsAccumulator`s (sum, avg, count) over lanes when there are few groups. - Pick up apache/arrow-rs#12062. ### Describe alternatives you've considered Setting `target_partitions` above the core count fixes this query. It also raises the number of hash partitions and the per-partition state of every other operator. ### Additional context Measured on a local machine at SF1 and SF10, not on the SF1000 S3 setup. The per-partition times come from the `start_timestamp`, `end_timestamp` and `output_rows` of `BaselineMetrics`, which `EXPLAIN ANALYZE` sums over partitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
