comphead opened a new issue, #27591:
URL: https://github.com/apache/datafusion/issues/27591

   ### Is your feature request related to a problem or challenge?
   
   Once `SkipAggregationProbe` decides to skip partial aggregation, the 
decision is final. `update_state` sets `is_locked = should_skip`, and the 
stream converts every remaining batch to state rows without aggregating. The 
probe only sees the rows just past 
`skip_partial_aggregation_probe_rows_threshold`, so two kinds of input lose:
   
   - **Keys whose pattern changes after the probe.** 400K distinct keys 
followed by 1.6M rows over 1000 keys:
   
     ```sql
     SET datafusion.execution.target_partitions = 2;
     EXPLAIN ANALYZE
     SELECT CASE WHEN value <= 400000 THEN value ELSE value % 1000 END AS k, 
SUM(value)
     FROM generate_series(1, 2000000)
     GROUP BY k;
     ```
   
     The Partial skips 1.79M rows and sends 2.00M rows downstream. With 
`skip_partial_aggregation_probe_ratio_threshold = 1.0` it sends 401K. This 
happens with or without a memory limit. Query time is about the same on one 
node (0.012 s and 0.013 s), so the cost is the 5x larger Partial output, which 
matters most where that output is shuffled over the network.
   
   - **A burst of memory pressure.** With #27351 the probe also counts groups 
emitted early under memory pressure. If the pool is tight while the probe is 
still deciding, for example because another operator or a concurrent query 
holds memory, the bounded table reduces little and the stream starts skipping. 
It keeps skipping after the memory is released. The same data shows how much 
the decision depends on the memory at that moment: 20M rows over 50K shuffled 
keys skip under a 16M fair pool but reduce 50x without a limit.
   
   ### Describe the solution you'd like
   
   Re-probe after skipping. Trino re-enables partial aggregation after 
processing 200 times the window it used to disable it, and resets its counters 
(`PartialAggregationController`). DuckDB resumes hash table lookups after 
skipping 128K to 1M rows (`RadixHTAdaptivity::MaybeResumeLookups`). In 
DataFusion this could be: after skipping some multiple of 
`skip_partial_aggregation_probe_rows_threshold` rows, aggregate the next window 
into an empty table, check its ratio, and then keep aggregating or skip again.
   
   This is point 3 of #22405. Its prototype deferred re-probing because 
re-entering aggregation after a skip panicked in `multi_group_by/primitive.rs`. 
That looks like the stale hash map after `emit(EmitTo::All)` that #27501 fixed.
   
   ### Describe alternatives you've considered
   
   - Re-probe only when the memory pool frees up. That covers the burst case 
but not a key pattern that changes.
   - Estimate repetition in the skipped batches with a sketch instead of 
aggregating them, as Polars does with an F2 sketch before growing its bounded 
table. That adds hashing to every skipped row.
   
   ### Additional context
   
   Raised in the review of #27351: 
https://github.com/apache/datafusion/pull/27351#discussion_r4551339139. 
Related: #22405, #27350, #26124.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to