andygrove commented on PR #2308: URL: https://github.com/apache/datafusion-ballista/pull/2308#issuecomment-5308451871
Pushed another round of changes based on the latest feedback. Doc changes: - **Flink link** (@milenkovicm). Linked in the intro, it was the only engine named without one. - **Shuffle affinity is prototypable today** (@milenkovicm). The bullet now says so: task distribution is already pluggable, so a `TaskDistributionPolicy::Custom` implementation can bind consumer tasks to the executors holding their `PartitionLocation`s and be measured against bias and round-robin without touching the core. - **Exact vs sampled statistics is not a clean split.** While checking the above I noticed the page overstated the case. Ballista's range repartition operators already sample in flight: `RuntimeStatsExec` keeps a T-Digest over the routing expression and the repartition operator snapshots it on the first batch to pick quantile cuts. Added a paragraph saying so, and noted that the same mechanism is what would spread a hot key across downstream tasks, which is @avantgardnerio's point about dynamic range repartitioning preventing skew. - **In-memory shuffle** (@Dandandan). Rewrote the bullet around the bounded-buffer-with-spill shape and your RAM disk result, and called it out as the largest single win available, since it says the cost is the filesystem round trip rather than the barrier. - **Config gating** (@villebro). The directions section now states that where one of these changes behavior it would be config-gated, with today's model as the default. - Each open direction now links to its tracking issue. Filed issues for the ideas that were not already tracked, so they do not get lost: - #2318 — buffer shuffle output in memory and spill to disk only when a configured budget is exceeded - #2319 — shuffle affinity: prototype a task distribution policy that places consumer tasks where their input already lives - #2320 — allow a stage to start on partially complete inputs instead of waiting for the whole upstream stage Already tracked, so no new issue: cheaper shuffle format and fewer small files (#660, epic #1319), streaming/push-based exchange and all-at-once scheduling (#1151, discussion in #2003), bubble execution (#408), remote shuffle service (#1539). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
