andygrove commented on issue #2367:
URL:
https://github.com/apache/datafusion-ballista/issues/2367#issuecomment-5415183253
Found it, and it turned out to be sitting in my own working tree the whole
time. Ignore my earlier request for logs, though the version of the Python
package you have would still be useful to confirm.
While wiring up CI to run the Python tests against a cluster built from the
branch, every single test failed with your exact error, including `SELECT 1`:
```
Failed to open partition file at ".../{job}/1/0/data.arrow": Os { code: 2,
kind: NotFound, ... }
```
The cause is a stale compiled extension. `maturin develop` and `pip install
-e python/` leave an `_internal_ballista.abi3.so` inside
`python/python/ballista/` and put that directory on `sys.path`, so it shadows
whatever wheel you have installed. It is never rebuilt unless you re-run
maturin, so it can drift arbitrarily far behind. Mine was from April:
```
$ ls -la python/python/ballista/_internal_ballista.abi3.so
-rwxr-xr-x 285706480 Apr 29 07:43 _internal_ballista.abi3.so
$ strings _internal_ballista.abi3.so | grep -oE 'ballista-core-[0-9.]+' |
sort -u
ballista-core-52.0.0
$ strings _internal_ballista.abi3.so | grep -cE 'file_id|is_sort_shuffle'
0
```
That last number is the whole story. `file_id` and `is_sort_shuffle` were
added to `FetchPartition` and `PartitionLocation` in 8c6c864f8 (#1527), which
landed after 52.0.0. A 52.0.0 client sends a fetch ticket with no `file_id` at
all, so the executor takes the `(None, false)` branch of `create_shuffle_path`
and looks for
```
{work_dir}/{job_id}/{stage_id}/{partition_id}/data.arrow
```
while a current executor actually wrote `data-{task_id}.arrow`. The final
stage writer changed from `file_id: None` to `file_id: Some(task_id)` after
54.0.0, which is why the name is one suffix off. The job really did complete,
the data really was written, the client was just asking for a filename from a
previous era.
Removing the `.so` fixes it. With the plain PyPI wheel against the same
branch-built cluster, `SELECT 1` and all 22 TPC-H queries pass.
Worth checking on your side:
```
ls -la python/python/ballista/*.so
strings python/python/ballista/*.so | grep -oE 'ballista-core-[0-9.]+' |
sort -u
```
If that prints anything older than your cluster, that is your bug too. `rm`
it, and `pip show ballista` will tell you what you actually fall back to.
A couple of follow-ups came out of this:
* #2372 to fix the CI gap that let this go unnoticed. The existing Python
test job cannot catch Rust regressions at all, because `setup_test_cluster()`
starts the scheduler and executor in-process from the crates.io release that
`python/Cargo.toml` pins, so the code under review is never loaded. The new job
builds the cluster from the branch and points the client at it, which is what
surfaced this.
* #2370 to add a client side protocol version check. A four-release-old
client should be refused at connect time with a clear message rather than
failing later on a missing file. That is the piece that would have turned this
into a five second diagnosis.
Leaving this open until you confirm it is the same thing on your machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
