Kalvin2077 opened a new pull request, #3775: URL: https://github.com/apache/celeborn/pull/3775
### What changes were proposed in this pull request? This PR introduces a Protobuf-based `RequestWorkers`/`RequestWorkersResponse` RPC for read-only worker discovery. On the Master side, the new handler selects currently available workers after applying client exclusions and tag filters. It limits the response using the smaller of `celeborn.client.slot.assign.maxWorkers` and `celeborn.master.splitSlot.assign.maxWorkers`, while preserving the minimum worker count required for replication. When authentication is enabled, application metadata is pushed to the selected workers. On the client side, LifecycleManager periodically requests workers from the Master, creates endpoints for newly selected workers, records connection failures, and maintains an endpoint-ready worker pool. ChangePartitionManager uses that pool for change-partition requests and falls back to the shuffle's existing worker snapshots when no refreshed candidates are available. The PR also makes the following configuration changes: - removes `celeborn.client.shuffle.dynamicResourceFactor`; - adds `celeborn.client.shuffle.dynamicResource.updateTime`, defaulting to `30s`; - adds `celeborn.master.splitSlot.assign.maxWorkers`, defaulting to `500`; - documents the breaking client configuration change in the migration guide. ### Why are the changes needed? The existing factor-based logic refreshes candidates only after enough workers from the shuffle's original allocation become unavailable. It therefore cannot use newly added workers during normal workers scale-out while the original workers remain healthy. Requesting slots merely to discover workers also mixes worker discovery with slot allocation and mutates Master shuffle state. A dedicated read-only RPC allows ChangePartitionManager to use the current cluster membership without those side effects, while rate limiting and worker-count caps prevent excessive RPC connections and oversized responses. ### Does this PR resolve a correctness bug? <!-- Check if yes. The `correctness` label will be added/removed automatically. --> - [ ] Yes ### Does this PR introduce _any_ user-facing change? <!-- Check if yes. --> - [x] Yes Users of `celeborn.client.shuffle.dynamicResourceFactor` must migrate to `celeborn.client.shuffle.dynamicResource.updateTime`. ### How was this patch tested? 6 tests: - 2 in `MasterSuite`; - 2 in `CelebornConfSuite`; - 1 in `WorkerStatusTrackerSuite`; - 1 in `UtilsSuite`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
