comphead commented on issue #7660:
URL: 
https://github.com/apache/datafusion-comet/issues/7660#issuecomment-6932039753

   Plan, in PR order:
   
   **Invariant.** Spark latches a group when its running sum leaves `10^p - 1` 
at any row. Each accumulator keeps `B`, an upper bound on `|sum|` over the 
groups that have not latched. Each batch gets `R`, an upper bound on `Σ|v|` 
over its rows. If `B + R <= 10^p - 1`, no running sum can cross the limit at 
any row of the batch. A plain `sums[g] += v` loop then gives the same values, 
nulls, and ANSI errors, and it cannot overflow `i128`.
   
   **`R` comes from the values, not the declared input precision.** Parquet 
values are not validated against the declared precision. One branchless pass 
ORs the `|v|` bit patterns into a power-of-two bound on `max|v|`, null and 
filtered slots included, which only loosens it. If the batch does not fit, `B` 
is recomputed from the sums, at most once per `num_groups` rows so the cost 
stays amortized. If it still does not fit, the batch runs today's per-row loop, 
with the precision bounds loaded once instead of calling 
`Decimal128Type::is_valid_decimal_precision`.
   
   1. **Grouped AVG.** A shared helper for `B` and `R`, the `update_batch` fast 
path, and a `benches/decimal_groups.rs` bench. AVG goes first because its 
layout (`Vec<i128>` plus a bitmap) is already right. `merge_batch` invalidates 
`B`.
   2. **Grouped SUM.** The same helper. The fast loop skips latched slots.
   3. **SUM state layout.** `Vec<i128>` plus bitmaps instead of 
`Vec<Option<i128>>` and `Vec<bool>`, measured separately.
   4. **Later, measured first.** `merge_batch` fast paths. For low-cardinality 
groups (Q1), reordering adds into independent accumulators when the fast path 
holds. An arrow-rs `#[inline]` on `is_valid_decimal_precision`.
   
   Out of scope: the ungrouped AVG, which belongs to #7116.
   
   **Validation for each PR:**
   - A seeded test comparing the fast path with the forced per-row path. It 
mixes small and near-limit values, nulls, filters, merges, and `EmitTo::First`, 
in both eval modes.
   - The existing Rust tests, `CometAggregateSuite`, and 
`sql-tests/expressions/aggregate/{sum,avg,avg_ansi}.sql`.
   - Bench shapes: 4, 1024, and 64K groups, a `(25,2)` and a capped `(38,6)` 
buffer, and nulls. A near-limit shape forces the slow path, and it is the 
regression gate.
   
   Criterion bench builds inline the arrow check and release `libcomet.dylib` 
does not, so the bench understates the release win.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to