andygrove commented on PR #2514: URL: https://github.com/apache/datafusion-ballista/pull/2514#issuecomment-5950894256
@avantgardnerio 1. I'd keep Substrait under the same rule, at least for now. The plan bytes are portable, but a Substrait client still speaks the rest of Ballista's protocol to get its results. It submits the job, follows the job status, and fetches the final partitions from the executors. That part isn't stable across majors. 55 already changed the shuffle fetch action that clients use to read results (`is_sort_shuffle` became `layout`). So exempting Substrait at the scheduler would still leave a 54 Substrait client speaking an older fetch protocol to a 55 executor. Loosening the rule later is also backward compatible, while tightening it later would break people. If someone needs Substrait to work across majors, I think that means committing to a stable job and fetch protocol across majors, which probably belongs in #2261. 2. Yes, that's the contract, and I agree it should be explicit. Any client works with any scheduler of the same major version, so a change that would break an older client of that major has to wait for the next major release. `BALLISTA_PROTOCOL_VERSION` can still change in any release, because schedulers and executors are upgraded together. One consequence is that a minor release that changes the shuffle fetch has to keep the old fetch working, since clients use it too. I've written this down in 0fab1d1, in the `ballista_core::version` docs, next to `BALLISTA_PROTOCOL_VERSION`, and in the upgrade guide. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
