andygrove opened a new pull request, #6888:
URL: https://github.com/apache/datafusion-comet/pull/6888
## Which issue does this PR close?
No tracking issue. This follows up the nightly snapshot workflow added in
#5902.
## Rationale for this change
Each Spark and Scala variant currently publishes up to four artifacts, but a
consumer needs only `comet-spark-spark<X>_<scala>`:
- The published `comet-spark` pom is flattened and shade-reduced, so it has
no `<parent>` and no `comet-common` dependency. `comet-common` is six exception
classes, all bundled into the shaded jar.
- The parent pom is installed as the raw root `pom.xml`, with
`${spark.version.short}_${scala.binary.version}` unresolved in its own
`artifactId`, so it could not serve as a parent. `comet-spark-integration` only
runs `dev/ensure-jars-have-correct-contents.sh`.
The nightly snapshot workflow passes `-Dmaven.deploy.skip=false`, which
overrides every module's setting, so it uploads all four modules for each of
the five variants (20 directories in the 2026-10-12 run). Release staging
uploads whatever `install` leaves in its temporary repository, which includes
`comet-common` and the parent pom. `comet-spark-integration` has not been
uploaded by a release since #1091 skipped its install.
## What changes are included in this PR?
- `pom.xml`, `common/pom.xml`, `spark-integration/pom.xml`, `spark/pom.xml`:
`maven.deploy.skip` stays `true` at the root and is no longer set to `false` in
`common` or `spark-integration`, so only `comet-spark` deploys.
- `.github/workflows/publish_snapshot.yml`: drop
`-Dmaven.deploy.skip=false`, and replace the comment that said consumers need
the parent pom.
- `dev/release/publish-to-maven.sh`: upload only `comet-spark-spark*`
directories. The release uploads everything under `org/apache/datafusion` in
its local repository, so the Maven property does not apply to it.
- `docs/source/contributor-guide/release_process.md`: say what the staging
repository should contain.
Installing is deliberately unchanged. Builds run from inside `spark/` (the
`benchmark-*` Makefile targets, and the TPC-H and TPC-DS data generation in
`pr_build_linux.yml`) resolve `comet-common` from the local repository, so
skipping its install would break them.
Versions already released keep their `comet-common` and parent artifacts,
and old snapshot directories stay until the snapshot repository's retention
removes them.
## How are these changes tested?
Pull request CI does not exercise these paths: the snapshot workflow only
deploys on `apache/datafusion-comet` (a `dry_run` dispatch runs `install`
instead), and the release scripts need Docker, GPG and Nexus credentials. They
were checked locally against a copy-on-write clone of `~/.m2`, deploying to a
`file://` repository:
- Building the Spark 4.1 variant with the previous poms and
`-Dmaven.deploy.skip=false` deploys `comet-common`, the parent pom,
`comet-spark-integration` and `comet-spark`. With this change and no override
it deploys only `comet-spark`: the jar, pom, sources jar and test-sources jar.
Install still writes the parent, `comet-common` and `comet-spark`, and still
skips `comet-spark-integration`.
- The cleanup lines from `publish-to-maven.sh`, run against the installed
tree plus a stray non-Comet directory, leave only the four `comet-spark` files
to sign and upload.
- A project depending only on `comet-spark` passes `dependency:resolve` when
the repository holds no parent pom or `comet-common`. The same check for the
`comet-common` jar fails, so the result is meaningful.
- `cd spark && ../mvnw dependency:resolve` still succeeds, because
`comet-common` is installed.
- `actionlint`, `prettier --check`, `dev/ci/check-ci-config.py` and `bash
-n` pass, and `shellcheck` reports the same findings as before.
Only the Spark 4.1 variant was built. The skip properties do not depend on
the Spark or Scala profile.
After this merges, the `Uploading to apache.snapshots.https` lines in the
next nightly run should name only `comet-spark-*` directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
