Hi all,
I'd like to introduce CIP-24, Adaptive partition write parallelism, and
would love to hear the community's thoughts.

Today, a reduce partition is written through a single active location at
any given time: every map task writes to that one file until it fills up.
For a hot partition this concentrates the entire aggregate write pressure
on a single point — push latency climbs to seconds, and the window between
the soft threshold and the hard limit collapses, so HARD_SPLIT ends up
blocking every map task writing that partition. We hit this in production
with a heavily skewed job where one reduce partition received all 26,090
mappers: 90% of task time was spent just waiting on shuffle writes.

The idea is simple: let a hot partition write to multiple active locations
in parallel. There is nothing to tune by hand — the LifecycleManager
measures how fast each location fills and adjusts the parallelism in a
closed loop, and executors learn the full active set through the existing
revive RPC. The change surface is deliberately small: one additive proto
field and three client-side configs, with the feature off by default.

With the feature enabled, that same production job went from 6.33 MB/s to
649 MB/s write throughput.

Design doc:
https://docs.google.com/document/d/1Kzn6_sEkzR4aDu2XofKBLUYxHyArSYdcYrlfZJuDBIw/edit?usp=sharing

PR: https://github.com/apache/celeborn/pull/3843

Any feedback is welcome — here, on the PR, or in the design doc. Thanks for
reading!

Best regards,

Hai Zhou

Reply via email to