venkata91 opened a new pull request, #3859: URL: https://github.com/apache/celeborn/pull/3859
### What changes were proposed in this pull request? Add a worker config `celeborn.worker.memoryFileStorage.replica.enabled` (default `false`). When it is disabled, `StoragePolicy` skips `MEMORY` for partitions whose `PartitionLocation` mode is `REPLICA`, so replica files fall through to the next storage type in `celeborn.worker.storage.storagePolicy.createFilePolicy` (e.g. SSD/HDD/DFS). Primary partitions keep using memory file storage as before. ### Why are the changes needed? When memory file storage is enabled (`celeborn.worker.directMemoryRatioForMemoryFileStorage` > 0), `StoragePolicy#createFileWriter` always tries `MEMORY` first and does not distinguish primary from replica partitions (`Controller#createPartitionDataWriter` receives `isPrimary` but only uses it to look up an existing working partition). With replication enabled, roughly half of the memory file storage budget, which also counts toward the pause-push thresholds, is spent on replica data. Replica data is only read when fetching from the primary fails, and in that case reading from disk is acceptable. Keeping replicas out of memory leaves memory for data that reducers actually read and delays eviction and push pauses. ### Does this PR resolve a correctness bug? - [ ] Yes ### Does this PR introduce _any_ user-facing change? - [x] Yes Adds the new worker config `celeborn.worker.memoryFileStorage.replica.enabled` (default `false`). With the default, replica partitions are no longer stored in memory file storage. Set it to `true` to restore the previous behavior. ### How was this patch tested? - Added `StoragePolicyReplicaSuite`, which covers: primary goes to memory, replica skips memory by default, and replica goes to memory when the config is enabled. - `build/mvn test -pl worker -DwildcardSuites=org.apache.celeborn.service.deploy.worker.storage.storagePolicy -Dtest=none` (JDK 17): passed, 8 tests. - `build/mvn test -pl common -DwildcardSuites=org.apache.celeborn.ConfigurationSuite -Dtest=none` (JDK 17): passed, docs in sync. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
