avantgardnerio opened a new issue, #2071:
URL: https://github.com/apache/datafusion-ballista/issues/2071
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
Ballista users are now running clusters 24/7, and can't afford downtime
during upgrades. However, Ballista is still young & flexible, and we want to
maintain velocity even as we make architectural changes. It would be good to
have an official upgrade strategy that protects both.
**Describe the solution you'd like**
Add a ballista_protocol_version: u32 field to ExecutorRegistration /
HeartbeatFromExecutor. The scheduler holds its own compiled-in
BALLISTA_PROTOCOL_VERSION constant. On every
heartbeat or registration:
- If executor.protocol_version == scheduler.protocol_version → proceed
normally.
- Otherwise → drop the heartbeat with an info! log and never dispatch
tasks to that executor.
We bump BALLISTA_PROTOCOL_VERSION only on releases that break wire compat,
not once per major version. Most releases won't bump it; some will.
Rolling upgrade flow
1. New scheduler comes up with bumped BALLISTA_PROTOCOL_VERSION.
2. Old executors heartbeat in; scheduler drops them; they drain naturally
as their existing tasks finish (or the operator kills them).
3. New executors with the bumped version come up and register normally.
4. No dual-implementation window, no code carrying old + new proto types.
Both scheduler versions are still up during the transition, so operators can
roll executors and schedulers in either order — mismatched pairs just don't
talk to each other, they don't
corrupt state.
Prior art
DQE (Coralogix' Ballista fork) has been doing this for a while with
reasonable success.
**Describe alternatives you've considered**
3 stage upgades.
**Additional context**
The downsides:
1. cluster capacity is at worst halved during upgrades - mitigation: upgrade
off hours, autoscale, k8s operator
2. more operational complexity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
