Hi all, I'd like to introduce CIP-24, Adaptive partition write parallelism, and would love to hear the community's thoughts.
Today, a reduce partition is written through a single active location at any given time: every map task writes to that one file until it fills up. For a hot partition this concentrates the entire aggregate write pressure on a single point — push latency climbs to seconds, and the window between the soft threshold and the hard limit collapses, so HARD_SPLIT ends up blocking every map task writing that partition. We hit this in production with a heavily skewed job where one reduce partition received all 26,090 mappers: 90% of task time was spent just waiting on shuffle writes. The idea is simple: let a hot partition write to multiple active locations in parallel. There is nothing to tune by hand — the LifecycleManager measures how fast each location fills and adjusts the parallelism in a closed loop, and executors learn the full active set through the existing revive RPC. The change surface is deliberately small: one additive proto field and three client-side configs, with the feature off by default. With the feature enabled, that same production job went from 6.33 MB/s to 649 MB/s write throughput. Design doc: https://docs.google.com/document/d/1Kzn6_sEkzR4aDu2XofKBLUYxHyArSYdcYrlfZJuDBIw/edit?usp=sharing PR: https://github.com/apache/celeborn/pull/3843 Any feedback is welcome — here, on the PR, or in the design doc. Thanks for reading! Best regards, Hai Zhou
