avantgardnerio commented on PR #2204: URL: https://github.com/apache/datafusion-ballista/pull/2204#issuecomment-5185526199
> I don't remember if that got implemented already. The field is hooked up, including the [scheduler side rejection](https://github.com/apache/datafusion-ballista/blob/a3161a301ece1c3d2a25bc574869accf8bcbc0da/ballista/scheduler/src/scheduler_server/grpc.rs#L67-L83), and the [health check](https://github.com/apache/datafusion-ballista/blob/a3161a301ece1c3d2a25bc574869accf8bcbc0da/ballista/scheduler/src/api/health.rs#L62-L83). But that only covers the normal protocol path - it's just for schedulers & executors speaking to each other. For the hard part: > This is obviously a breaking change - how does this impact rolling upgrades? 1. Currently, we don't currently ship shared scheduler state. I'm going to assume folks run in k8s. The scheduler now has the readiness endpoint. So new scheduler fires up, new protocol version. New executors register with it. When it has enough to do work, the readiness endpoint turns green, k8s starts routing queries to it and kills the old one. During the transition, in-flight queries probably die. Solution: k8s operator, bring back KV stores. 2. There is no persistent shuffles. If there were, this would be a breaking change. But since shuffles die with their host executor, this PR effectively falls under # 1 above. 3. Ballista is a library. @phillipleblanc likely has better answers for how he uses it. I'm pretty sure he persists shuffles to s3 I saw in his video? In which case this PR could be a deeper breaking change if new executors try to read old shuffles. I think we could create a fallback path for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
