andygrove commented on issue #2367:
URL:
https://github.com/apache/datafusion-ballista/issues/2367#issuecomment-5414091061
Thanks for the detailed report @martin-g. I spent some time on this and I
can't reproduce it yet, so I could use a few details about your setup.
Here is what I tried. Cluster built from `apache/main` (`ee25f53`, three
commits ahead of your `50d7322`, none of them relevant here), TPC-H SF1 parquet
from `tpchgen-cli`, one scheduler and one executor with the default temp work
dir like yours. I ran your `tpch.py` with the PyPI `ballista` 54.0.0 wheel and
the 54.1.0 wheel, all 22 queries plus q1 twenty five times in a row, and also
tried two executors, 8 vcores vs 28, and an explicit `--work-dir`. Everything
passed.
The interesting part is the filename in your error. You are asking for
`.../{job}/3/0/data.arrow`, but on current `main` the final stage writes
`.../{job}/3/0/data-{task_id}.arrow`. That naming changed after 54.0.0. The
passthrough `ShuffleWriterExec` used to report `file_id: None` and write
`data.arrow`, and now it reports `file_id: Some(task_id)` and writes
`data-{task_id}.arrow`. I checked every `file_id: None` construction site on
`main` and they are all inside `#[cfg(test)]`, so no current code path should
ever produce a request for the un-suffixed name. That points at something in
your setup running the older layout.
So, could you check a few things?
1. `pip show ballista` in the environment you ran `tpch.py` from. Also
whether that is the PyPI wheel or a local `maturin` build. If you built it
locally with the path dependencies in `python/Cargo.toml` uncommented, that
puts you on a different code path than the wheel and I would like to try that
combination.
2. How you built and launched the scheduler and executor, and whether
**both** binaries were rebuilt at `50d7322`. A leftover 54.0.0 executor binary
is my leading suspicion right now.
3. The scheduler and executor logs for job `0381QN35XP000` if you still have
them.
4. Whether it fails every time, or only sometimes.
One thing I noticed along the way that may or may not be related: the PyPI
Python client pins `ballista* = "=54.0.0"` from crates.io, and 54.0.0 defaults
`ballista.planner.adaptive.enabled` to `false`. Since `BallistaConfig` ships
every setting including defaults, the scheduler receives an explicit `false`
and uses the static planner, even though `main` now defaults that to `true`. So
the Python client and the Rust client are exercising different planners against
the same cluster. I filed #2371 for that, plus #2370 for adding a client side
protocol version check so a mismatch like this fails with a clear message
instead of a confusing missing file, and #2372 for the CI gap that let both
slip through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
