andygrove commented on issue #2370:
URL:
https://github.com/apache/datafusion-ballista/issues/2370#issuecomment-5415218873
Two notes for whoever picks this up.
**We now have a concrete case.** #2367 turned out to be a `ballista-core`
52.0.0 Python extension talking to a current cluster. 52.0.0 predates `file_id`
and `is_sort_shuffle` on `FetchPartition` (added in 8c6c864f8, #1527), so the
client sent a fetch ticket with no `file_id`, the executor took the `(None,
false)` branch of `create_shuffle_path` and looked for `data.arrow` while the
writer had produced `data-{task_id}.arrow`. The job completed successfully and
then the result fetch failed with `NotFound`. Four releases of skew, no error
until the very last step, and the symptom pointed at the shuffle machinery
rather than at the client. That is the failure mode this issue is about.
**There is a policy question to settle first, and it conflicts with #2374.**
The Python client cannot track the workspace DataFusion version. `pyballista`
re-exports datafusion-python types (`python/src/lib.rs`), so
`datafusion-python`, `datafusion` and `ballista-core` all have to link the same
`datafusion`, and crates.io `datafusion-python` is still at 54.0.0 while the
workspace is on the `55.0.0-rc3` tag. The practical consequence is that the
Python client normally lags the cluster by a DataFusion release cycle, so a
*supported* Python user is routinely running a client that is a release or two
behind.
That means a strict equality check on `BALLISTA_PROTOCOL_VERSION`, like the
one executors get today, would reject the normal Python setup. #2374 adds a CI
job that runs the pinned 54.0.0 client against a cluster built from the branch,
precisely because that is what users run, and a strict check would turn that
job red by design.
So this probably wants something more nuanced than the executor check, for
example:
* a minimum supported client version rather than exact equality, with the
version bumped only when something client-visible actually breaks, or
* a capability or feature negotiation at connect time, so the scheduler can
tell whether the client understands `file_id` rather than inferring it from a
version number.
Either way the win is the same and worth having: fail at connect with a
message naming both versions, instead of at the last fetch with a missing file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
