andygrove opened a new issue, #2371:
URL: https://github.com/apache/datafusion-ballista/issues/2371
**Describe the bug**
#2315 flipped `ballista.planner.adaptive.enabled` to default `true`, so a
client built from `main` gets the adaptive (AQE) planner. The published Python
client does not, and it silently forces the static planner instead.
`python/Cargo.toml` pins the Rust side to the released crates:
```toml
ballista = { version = "=54.0.0" }
ballista-core = { version = "=54.0.0" }
```
In 54.0.0 the default for that key is `false`. `BallistaConfig` materialises
every setting including defaults and ships it to the scheduler with the session
config, so the scheduler sees an explicit
`ballista.planner.adaptive.enabled=false` and honours it. The user never asked
for that, and there is nothing in the logs at INFO level to tell them which
planner ran.
Confirmed against a cluster built at `apache/main` with scheduler debug
logging on:
```
DEBUG ballista_scheduler::state::task_manager: Using adaptive query planner
(AQE) for job planning <- Rust client from main
DEBUG ballista_scheduler::state::task_manager: Using static query planner
for job planning <- ballista 54.0.0 from PyPI
```
Same query, same cluster, different planner purely because of which client
submitted it.
**To Reproduce**
1. Build and start a scheduler and executor from `main`.
2. Run any TPC-H query with the Rust client, and again with `pip install
ballista` (54.0.0 or 54.1.0).
3. With `RUST_LOG=ballista_scheduler::state::task_manager=debug` on the
scheduler, compare the "query planner" lines.
**Expected behavior**
Python users should get the same planner as everyone else, and if a client
does pin a non-default value it should be because the user asked for it.
Two things worth separating here:
1. Any config key whose default changes on the Rust side is invisible to a
Python client pinned to an older release, and the mismatch is silent. It would
be better if a client only sent keys the user actually set, so the scheduler's
own defaults apply to everything else.
2. The version pin itself. `python/Cargo.toml` has the path dependencies
commented out and depends on crates.io, so the Python bindings on `main` are
not built against the code on `main`. That is fine for release builds but means
the bindings lag every default and behaviour change until the next release.
**Additional context**
Found while investigating #2367. Related: #2315, and the CI gap in <ISSUE3>.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
