yew1eb opened a new pull request, #3843:
URL: https://github.com/apache/celeborn/pull/3843
### What changes were proposed in this pull request?
Adaptive partition write parallelism (design doc:
<[link](https://docs.google.com/document/d/1Kzn6_sEkzR4aDu2XofKBLUYxHyArSYdcYrlfZJuDBIw/edit?usp=sharing)>):
- A hot reduce partition writes to multiple active locations in
parallel; mappers are routed by mapId % activeCount.
- The LifecycleManager measures per-location fill time from retire
reports and ramps parallelism toward ceil(targetSplitInterval / fillTime) —
closed-loop, no manual tuning. Every revive reply carries the full active
set.
- One additive proto field
(PbChangeLocationPartitionInfo.additionalPartitions); three client-side
configs, all off by default.
### Why are the changes needed?
A reduce partition is written through a single active location at a time,
concentrating all mappers' write pressure on one point. In one of
our production jobs (one reduce partition, 26,090 mappers), 90% of task
time was spent waiting on shuffle writes; with this feature, write
throughput improved from 6.33 MB/s to 649 MB/s.
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed
automatically. -->
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [x] Yes
Three new client configs (`enabled` / `maxLocations` /
`targetSplitInterval`), disabled by default — no behavior change unless
explicitly enabled. Protocol change is additive and safe for mixed old/new
client and LifecycleManager versions.
### How was this patch tested?
- New unit suites: `PartitionLocationGroupSuiteJ`, `ReviveManagerSuiteJ`,
`ShuffleClientSuiteJ`, `PartitionHotnessTrackerSuite`,
`ChangePartitionManagerAdaptiveParallelismSuite`,
`RequestLocationCallContextSuite`.
- Existing client/LM suites pass unchanged with the feature off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
