andygrove opened a new pull request, #2385:
URL: https://github.com/apache/datafusion-ballista/pull/2385
# Which issue does this PR close?
N/A - follow-up to the DataFusion 55 upgrade of the Rust workspace.
# Rationale for this change
`python/` is a separate Cargo workspace from the rest of the repo, so the
DataFusion 55 upgrade did not reach it. The Python client is still built
against the DataFusion 54 crates and against the crates.io `ballista` 54.0.0
release, which means the wheel we ship no longer matches the engine in `main`.
datafusion-python has not published a DataFusion 55 release yet, so this PR
tracks the bump branch at https://github.com/andygrove/datafusion-python.git
until 55.0.0 is on crates.io.
That choice forces the `ballista` crates back onto path dependencies. The
published 54.0.0 crates are built against DataFusion 54, and `pyballista`
passes a DataFusion `DataFrame` produced by a Ballista `SessionContext`
straight into `datafusion_python::dataframe::PyDataFrame::new`, so the two
sides have to agree on the DataFusion version. This restores the arrangement
that was in place before #2277.
# What changes are included in this PR?
- Bump the `pyballista` crate version to 55.0.0.
- Point `datafusion-python` at the `bump-datafusion-55.0.0` branch of
https://github.com/andygrove/datafusion-python.git, and pin `datafusion` /
`datafusion-proto` to `=55.0.0`.
- Switch `ballista`, `ballista-core`, `ballista-executor` and
`ballista-scheduler` back to path dependencies on the workspace crates.
- Bump `pyo3` from 0.28 to 0.29 so it matches the version datafusion-python
is built with.
- Bump the `datafusion` Python dependency in `pyproject.toml` from `==54` to
`==55`.
- Drop the two `concurrent_tasks` hacks in `python/src/cluster.rs`. They
carried a `TODO` waiting for the rename, and `ExecutorProcessConfig::vcores` is
now available.
- Refresh `python/Cargo.lock`.
Verified locally in the `python/` workspace: `cargo check --locked`, `cargo
clippy --locked --all-targets -- -D warnings` and `cargo fmt --check` all pass.
Two things are deliberately left for a follow-up, both blocked on the
datafusion-python release:
- `python/uv.lock` is unchanged, because `uv lock` cannot resolve
`datafusion==55` while it is absent from PyPI. Every job that runs `uv sync`
will fail until datafusion-python 55.0.0 is published, at which point the
lockfile can be regenerated.
- The `datafusion-python` git dependency should become a `=55.0.0` crates.io
pin once that release exists.
This is why the PR is a draft.
# Are there any user-facing changes?
The Python client will require `datafusion==55` instead of `datafusion==54`.
There are no API changes to the client itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
