sunchao opened a new pull request, #3685: URL: https://github.com/apache/celeborn/pull/3685
## Why are the changes needed? Celeborn load-aware slot placement currently orders candidate disks using flush and fetch timing only. That can still keep assigning new partitions onto disks that already carry a large amount of reserved active-slot pressure, which makes placement skew worse under overlapping shuffle-heavy workloads. CELEBORN-2327 tracks this gap. ## What changes were proposed in this PR? - Add an optional, default-off `celeborn.master.slot.assign.loadAware.activeSlotsWeight` config. - Include `activeSlots * activeSlotsWeight` in load-aware disk ordering. - Thread the new config through the master allocation path. - Document the new tuning knob and update the slot-allocation developer docs. - Add a regression test showing that, when configured, lower-active-slot disks are preferred over otherwise equivalent disks. ## How was this PR tested? - `UPDATE=1 build/mvn clean test -pl common -am -Dtest=none -DwildcardSuites=org.apache.celeborn.ConfigurationSuite` - `./build/mvn -pl master -am -Dtest=SlotsAllocatorSuiteJ -DwildcardSuites=org.apache.celeborn.NoSuchSuite -DfailIfNoTests=false test` - `./build/mvn -pl master -am -DskipTests test-compile` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
