comphead commented on issue #7660:
URL:
https://github.com/apache/datafusion-comet/issues/7660#issuecomment-6932039753
Plan, in PR order:
**Invariant.** Spark latches a group when its running sum leaves `10^p - 1`
at any row. Each accumulator keeps `B`, an upper bound on `|sum|` over the
groups that have not latched. Each batch gets `R`, an upper bound on `Σ|v|`
over its rows. If `B + R <= 10^p - 1`, no running sum can cross the limit at
any row of the batch. A plain `sums[g] += v` loop then gives the same values,
nulls, and ANSI errors, and it cannot overflow `i128`.
**`R` comes from the values, not the declared input precision.** Parquet
values are not validated against the declared precision. One branchless pass
ORs the `|v|` bit patterns into a power-of-two bound on `max|v|`, null and
filtered slots included, which only loosens it. If the batch does not fit, `B`
is recomputed from the sums, at most once per `num_groups` rows so the cost
stays amortized. If it still does not fit, the batch runs today's per-row loop,
with the precision bounds loaded once instead of calling
`Decimal128Type::is_valid_decimal_precision`.
1. **Grouped AVG.** A shared helper for `B` and `R`, the `update_batch` fast
path, and a `benches/decimal_groups.rs` bench. AVG goes first because its
layout (`Vec<i128>` plus a bitmap) is already right. `merge_batch` invalidates
`B`.
2. **Grouped SUM.** The same helper. The fast loop skips latched slots.
3. **SUM state layout.** `Vec<i128>` plus bitmaps instead of
`Vec<Option<i128>>` and `Vec<bool>`, measured separately.
4. **Later, measured first.** `merge_batch` fast paths. For low-cardinality
groups (Q1), reordering adds into independent accumulators when the fast path
holds. An arrow-rs `#[inline]` on `is_valid_decimal_precision`.
Out of scope: the ungrouped AVG, which belongs to #7116.
**Validation for each PR:**
- A seeded test comparing the fast path with the forced per-row path. It
mixes small and near-limit values, nulls, filters, merges, and `EmitTo::First`,
in both eval modes.
- The existing Rust tests, `CometAggregateSuite`, and
`sql-tests/expressions/aggregate/{sum,avg,avg_ansi}.sql`.
- Bench shapes: 4, 1024, and 64K groups, a `(25,2)` and a capped `(38,6)`
buffer, and nulls. A near-limit shape forces the slow path, and it is the
regression gate.
Criterion bench builds inline the arrow check and release `libcomet.dylib`
does not, so the bench understates the release win.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]