avantgardnerio commented on code in PR #2211:
URL:
https://github.com/apache/datafusion-ballista/pull/2211#discussion_r3825122083
##########
ballista/core/proto/ballista.proto:
##########
@@ -656,6 +723,48 @@ message SuccessfulTask {
// executed `RuntimeStatsExec`; the scheduler groups by `order_by` tag
// to combine reports across tasks/executors.
repeated RuntimeStatsReport runtime_stats = 3;
+ // Finalized window-aggregate state captured during this task, one entry
+ // per (output partition, window expression, PARTITION BY group) that
+ // closed. Empty unless the plan contains an ever-expanding-frame window
+ // (`UNBOUNDED PRECEDING`), which is the only shape DataFusion will
+ // publish accumulator state for. The scheduler prefix-merges these across
+ // tasks and bakes the result into a downstream `PrefixMergeExec`.
+ //
+ // TODO: watch the size of this. Task completion is a hot, frequent
+ // message, and sketch-backed aggregates make the payload unbounded in a
+ // way row counts and quantile sketches are not — an HLL or KLL state is
+ // kilobytes per window expression per partition, and a task covering a
+ // wide partition slice carries one of each. If it stops being small,
+ // write the state as a sidecar next to the shuffle files instead, the way
+ // sort-shuffle already writes `<data>.arrow.index` beside its data
+ // (`sort_shuffle::get_index_path`), and send only a reference here. That
+ // keeps the completion message fixed-size regardless of aggregate.
Review Comment:
I measured it and the above statement is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
