andygrove opened a new pull request, #5108:
URL: https://github.com/apache/datafusion-comet/pull/5108
## Which issue does this PR close?
N/A
## Rationale for this change
The PyArrow UDF test harness locates the built Comet jar with a glob that
hardcodes the `-SNAPSHOT` qualifier:
```
spark/target/comet-spark-{spark_tag}{scala_tag}-*-SNAPSHOT.jar
```
On a release branch the Maven version is the final release version with no
`-SNAPSHOT` qualifier, so the glob matches nothing and every PyArrow UDF test
errors out with `FileNotFoundError: Comet jar not found`. This is what happened
on the 1.0.0 release branch in #5105, where the PyArrow UDF jobs were the only
failing checks. The release branch runs the same workflows as `main`, so this
class of breakage only surfaces after the release branch is cut and the version
is bumped.
Two benchmark helpers have the same problem: the TPC docker build copies
`comet-spark-spark3.5_2.12-*-SNAPSHOT.jar` out of the container, and
`run_all_benchmarks.sh` defaults `COMET_JAR` to a hardcoded `0.14.0-SNAPSHOT`
path that is already stale.
## What changes are included in this PR?
- `spark/src/test/resources/pyspark/conftest.py`: match any version suffix
rather than only `-SNAPSHOT`, and widen the artifact filter so
sources/tests/javadoc/shaded jars are still excluded.
- `benchmarks/tpc/infra/docker/Dockerfile.build-comet`: drop `-SNAPSHOT`
from the jar copy glob.
- `benchmarks/pyspark/run_all_benchmarks.sh`: resolve the default
`COMET_JAR` by glob instead of a hardcoded version, and fail with a clear
message when no jar is found.
- `docs/source/contributor-guide/release_process.md`: enumerate the files
that carry the Maven version (including the Gradle version catalog entry in the
Iceberg diffs, which uses a different syntax than the Maven property in the
Spark diffs), add a `git grep` verification step after the bump, and note that
jar globs must not assume a `-SNAPSHOT` suffix.
## How are these changes tested?
Existing CI. The PyArrow UDF workflow triggers on `conftest.py`, so it
exercises the new resolution logic on `main`, where the version still carries
`-SNAPSHOT`. The non-SNAPSHOT path is verified on the 1.0.0 release branch,
where the same `conftest.py` change turns the previously failing PyArrow UDF
jobs green in #5105.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
