leixm commented on PR #2819: URL: https://github.com/apache/celeborn/pull/2819#issuecomment-4234488252
> @leixm have we seen any issues because of this in production setup or is this precaution? > > I am asking this because this change cause split brain till new Master receives heartbeats from all workers (around 30-60s) and quota might be inconsistent during this time. > > * It will allow new applications to admit even though quota is breached. > * Interrupt of high shuffle applications will be delayed. @s0nskar This is a design trade-off rather than a response to a specific production issue. The resourceConsumption data is updated on every worker heartbeat and the payload can be large (per-user per-worker). Replicating it through Ratis consensus would significantly increase log volume and write latency. You're right that during leader transition, quota checks may be temporarily relaxed (~30-60s). We consider this acceptable because: - Leader transitions are rare events. - Quota enforcement is inherently "soft" — even under normal operation, the updateResourceConsumption task runs on a 30s interval (celeborn.master.userResourceConsumption.update.interval), so there is already up to ~30s delay in quota detection. The additional ~30s from rebuilding resource consumption after leader change is the same order of magnitude. - The data recovers quickly once the new leader receives worker heartbeats. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
