andygrove opened a new issue, #2511: URL: https://github.com/apache/datafusion-ballista/issues/2511
## Summary DataFusion 55.0.0 was released on August 18. Six weeks later we still can't release Ballista 55.0.0, because our release includes the Python client, and the Python client can't ship until datafusion-python 55.0.0 is published. That release is being prepared in apache/datafusion-python#1696, but the vote hasn't started yet. This isn't a complaint about datafusion-python, which has its own priorities, including the FFI query planner support that #2252 builds on. But now that our Rust side is ready soon after each DataFusion release, waiting for datafusion-python has become the main thing holding up our releases, and there's a lot on `main` that I'd like to ship. So I'd like to propose that we release the Python client separately from the Rust crates. ## How far datafusion-python trails DataFusion | DataFusion | Released | datafusion-python released | Lag | | --- | --- | --- | ---: | | 44.0.0 | 2024-12-31 | 2025-02-12 | 43 days | | 45.0.0 | 2025-02-07 | 2025-02-24 (45.2.0) | 17 days | | 46.0.0 | 2025-03-07 | 2025-03-30 | 23 days | | 47.0.0 | 2025-04-20 | 2025-05-28 | 38 days | | 48.0.0 | 2025-06-11 | 2025-07-12 | 31 days | | 49.0.0 | 2025-07-27 | 2025-08-29 | 33 days | | 50.0.0 | 2025-09-16 | 2025-09-23 | 7 days | | 51.0.0 | 2025-11-19 | 2026-01-09 | 51 days | | 52.0.0 | 2026-01-12 | 2026-02-23 | 42 days | | 53.0.0 | 2026-03-23 | 2026-04-13 | 21 days | | 54.0.0 | 2026-06-08 | 2026-06-29 | 21 days | | 55.0.0 | 2026-08-18 | not released yet | 42 days so far | Dates are from crates.io and PyPI. We need both, so where they differ I used the later one. Over the last eleven releases, the median lag was 31 days. The shortest was one week and the longest was over seven weeks. This didn't hurt us much before, because our own DataFusion upgrades took about as long. For 54, our upgrade (#1906) landed on June 30, the day after datafusion-python 54.0.0 was published. Since #1920 we've tracked DataFusion `main` between releases, and this time we were on the released 55.0.0 crates eight days after DataFusion published them (#2375). So datafusion-python is now what we wait on, and with the current process our releases will usually trail DataFusion by at least a month. ## What's waiting on `main` 310 commits have landed on `main` since `branch-54` was cut on July 12, and 54.1.0 only picked up a handful of backported fixes. Some highlights: - **Correctness.** Distributed `NOT IN` could return wrong results (#2188), and uncorrelated `NOT IN` subqueries now get a plan that runs in parallel (#2199). A set of AQE fixes means TPC-DS now verifies clean across all 99 queries with AQE on (#2315). - **Stability.** Executors default to a bounded, auto-sized memory pool, so heavy queries spill instead of getting the executor OOM-killed (#2160). Busy executors keep heartbeating instead of being declared dead (#2159), and jobs fail with a clear error instead of hanging when all executors are lost (#2212). - **Performance.** Adaptive query planning is on by default (#2315), and it can now measure a join's build side before choosing the join (#2434). Tasks can cover several partitions (#2038), some window functions with no `PARTITION BY` now run in parallel instead of on a single core (#2211, #2223), and sharing the file statistics cache cuts total planning time for the 22 TPC-H queries at SF100 from 1.5 s to 0.13 s (#2498). - **Operations.** A history server for browsing completed jobs (#2265), and an OpenAPI description of the REST API (#2397). ## Previous discussion We discussed this in #2370 in August: - I listed the costs of splitting. We'd need separate tarball scripts and separate votes, it would be harder for Python users to help test and vote on the Rust release, and a Python release vote could turn up bugs in crates we'd already released, which would mean patch releases. - @avantgardnerio asked whether Python users would have to wait to upgrade until the Python client caught up, and showed how a 54 client could get silently different results from a 55 cluster (the `preserve_nulls` example). - @milenkovicm noted that the Python bindings are "best effort" and a "technical preview" today and that we don't have many Python users yet, and suggested reviving the separate `datafusion-ballista-python` repo once the FFI integration is solid, so we could release Ballista without waiting for datafusion-python. - The consensus was to keep the single release "until proven otherwise". I think the 55.0.0 release is a good reason to look at it again. Some older history is relevant too: - In 2023 we moved the Python bindings to their own repo (#635), partly because datafusion-python releases on a different schedule. They went unmaintained for about a year, and #970 rebuilt them in this repo in 2024. That PR said the new bindings "will be versioned and released independently from the main project". - In #1142 (2024), @milenkovicm proposed shipping the Rust release without the Python bindings until the Python integration was sorted out. - #1120 then set the goal of keeping the Python client in step with the Rust releases, and #1610 documented the current process. We only started publishing the Python client to PyPI with 54.0.0 in July, so the coupling is fairly new. ## Proposal Release the Rust crates and the Python client separately, and keep both in this repo. - **Rust release.** The same process as today, minus the Python wheels. We cut it as soon as we're ready after a DataFusion release. - **Python release.** Its own RC tag (something like `python-55.0.0-rc1`), source tarball and vote, once datafusion-python has published the matching version. It would build against the `ballista` crates we've already published to crates.io, so the wheels contain exactly the Rust code we voted on. The Python client keeps the version of the Ballista release it wraps, so Ballista 55.0.0 would be followed by `ballista` 55.0.0 on PyPI. - **Same repo.** `python/` stays where it is. Keeping it next to the Rust code makes it much easier to test against `main` (#2372), and the separate repo didn't work out last time. For 55.0.0, this would mean starting the Rust release now and following up with the Python release once datafusion-python 55.0.0 is out. What this means for users: - **Rust users** get each release when it's ready, instead of waiting for datafusion-python. - **Python users** get the Python client at the same time they would today, since today's release also waits for datafusion-python. - **Mixed versions.** In the gap between the two releases, Python users should keep their cluster on the previous release. Since #2378 the client logs a warning when the scheduler's version doesn't match. Given #2376 and the `preserve_nulls` example, I think a major version mismatch should be an error rather than a warning. ## The concerns from #2370 - **Separate scripts and votes.** This is real, but it's mostly one-time work in `dev/release` and the wheel build workflow, and I'm happy to do it. The PMC already runs separate votes for DataFusion, datafusion-python, Comet and Ballista, so one more vote isn't unusual. - **Python users voting on the Rust release.** They would vote on the Python release instead, which is the artifact they actually use. - **Bugs found late.** If the Python release turns up a bug in the Rust crates, we'd do a patch release, the same as for any bug found after a release. I'd rather do that occasionally than hold 300 commits for six weeks. - **FFI.** #2252 would remove the Rust-level dependency on datafusion-python, which is great. As @milenkovicm pointed out, though, we'd probably still depend on a matching `datafusion` Python release, so I see it as complementary rather than a fix for the release timing. ## Alternatives - **Keep the single release.** Every Ballista release waits for datafusion-python, a median of 31 days after DataFusion. - **Only split when we're blocked.** Hold one vote when datafusion-python is already out, and split otherwise. Going by the table we'd end up splitting almost every time, so I'd rather have one process. - **Move the Python client back to its own repo.** This was @milenkovicm's suggestion in #2370. It gives us the same release independence, but it makes testing the client against `main` harder, and it didn't go well last time. ## Questions - Is anyone opposed to releasing the Python client separately? - Should a major version mismatch between the client and the scheduler be an error? - Should we start the 55.0.0 Rust release now, or wait for datafusion-python 55.0.0 this time and switch to the new process for 56? If there's support, I'm happy to update the release scripts, the wheel build workflow and the release docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
