jprinet opened a new pull request, #59431: URL: https://github.com/apache/spark/pull/59431
### What changes were proposed in this pull request? Adds [Develocity](https://develocity.ai/) integration to the Maven and sbt builds. CI builds publish Build Scans to [develocity.apache.org](https://develocity.apache.org). Both builds follow the same rules: - **Build Scans are published only when `DEVELOCITY_ACCESS_KEY` is set.** Local builds and contributors' fork CI publish nothing, never contact the server and never wait on an upload. A developer who wants a scan sets the variable. - **Builds from forks:** Spark runs pull request CI in the contributor's fork, which doesn't have the secret, so those builds publish nothing. A maintainer who wants Build Scans from their own fork has to add a `SPARK_DEVELOCITY_ACCESS_KEY` secret there, holding their own develocity.apache.org key (`develocity.apache.org=<access key>`). - **Upload timing:** on CI the scan uploads before the build ends, since runners are torn down right after the job. Locally it uploads in the background. - **CI detection:** `GITHUB_ACTIONS`, the only CI Spark builds on. It only decides the upload timing and whether to store to the cache, and both also require the key, which only Spark's GitHub secrets provide. - **Project-level access control:** every Build Scan is associated with the Develocity project `spark`, through `<projectId>` in `.mvn/develocity.xml` and `withProjectId` in `build.sbt`. The project has to exist on develocity.apache.org, and the key's owner needs access to it. Maven: - `.mvn/extensions.xml`: `develocity-maven-extension` 2.6.0 and `common-custom-user-data-maven-extension` 2.4.0. - `.mvn/develocity.xml`: the remote build cache is used only with the key, and only CI stores to it. - The root `pom.xml` marks `maven-surefire-plugin` test runs as not cacheable (see below). Scala tests run through `scalatest-maven-plugin`, which the extension doesn't cache. - `.mvn/.develocity/` is added to `.gitignore`. sbt: - `project/plugins.sbt`: `sbt-develocity` 1.4.5 and `sbt-develocity-common-custom-user-data` 1.5.0. - `build.sbt` (new, at the root): `ThisBuild / develocityConfiguration ~= ...`, which composes with `SparkBuild.scala` instead of replacing anything. Build Scans only: the build cache is off (see below). **Why tests are not cacheable.** With a build cache, a test run whose inputs haven't changed is skipped and its previous result reused. Spark's tests read inputs the cache key doesn't track: environment variables (`SPARK_HOME`, `SPARK_LOCAL_IP`, `HIVE_PROFILE`), files outside the module, and the Python and R interpreters on the `PATH`. A cached pass could hide a real failure. Their settings also contain machine-specific absolute paths (`java.io.tmpdir`, `spark.test.home`, a classpath under `~/.m2`), so container and host jobs would compute different keys and rarely share entries anyway. So: - **Maven** keeps caching compilation, but surefire always runs. - **sbt** can't exclude only test tasks from its build cache ("It's not possible to disable caching only for a specific task" in the sbt plugin manual), so caching is off and sbt only publishes Build Scans. Workflows: - The key is set on the 22 steps that run sbt or Maven (`build/sbt`, `build/mvn`, `dev/run-tests`, the JVM linters, MiMa checks and the docs build), not on the job or workflow. The exceptions are the six jobs whose test step already sets `env: ${{ fromJSON(inputs.envs) }}`, an expression that can't take extra keys: `build`, `pyspark`, `sparkr` and `docker-integration-tests` in `build_and_test.yml`, and `build` in `maven_test.yml` and `python_hosted_runner_test.yml`. There, the key is set on the job. Checkout, caching, artifact uploads, codecov and other actions in the remaining jobs never see it. The steps that run sbt or Maven still pass it to their test processes, because the tools need it there to publish. - Also covered: `python_hosted_runner_test.yml`, `build_python_connect.yml`, `build_python_connect40.yml`, `benchmark.yml` and `build_sparkr_window.yml`. - Callers of the reusable workflows (`build_and_test.yml`, `maven_test.yml`, `python_hosted_runner_test.yml`) pass the key with an explicit `secrets:` mapping rather than `secrets: inherit`, so each reusable workflow only receives the secrets it is given (least privilege). ### Why are the changes needed? Build Scans give Spark's CI per-test results with history (flaky and slow tests), console logs, failure grouping and performance data on develocity.apache.org, plus a shared build cache for Maven compilation. ### Does this PR introduce _any_ user-facing change? No. Without `DEVELOCITY_ACCESS_KEY`, local builds behave as before and publish nothing. ### How was this patch tested? - **Maven:** `build/mvn validate -pl common/tags`. Without the key, nothing was published. With a key set, the extension tried to publish to develocity.apache.org. - **sbt:** `build/sbt "show ThisBuild/develocityConfiguration"` resolves the server, background upload locally, and the local and remote caches off. `build/sbt about` publishes nothing without the key. - **License headers:** `dev/check-license` passes. Before merging: add `SPARK_DEVELOCITY_ACCESS_KEY` as a repository secret (exposed to the builds as `DEVELOCITY_ACCESS_KEY`) with the value `develocity.apache.org=<access key>`, requested from ASF Infra for a user with access to the `spark` project. The key's owner needs permission to publish Build Scans and to write to the build cache. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5). It adapted the configuration from the official Develocity samples, edited the workflows, ran the checks above and wrote this description. cc @holdenk 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
