andygrove opened a new pull request, #6947: URL: https://github.com/apache/datafusion-comet/pull/6947
## Which issue does this PR close? No issue. Found while triaging the CI of #6877, whose Spark 3.5 Spark SQL build failed this way. ## Rationale for this change `setup-spark-builder` restores the Maven repository from `Linux-spark-sql-<hash>` and falls back to the restore keys `Linux-spark-sql-`, then `Linux-java-maven-`. Since #5973, nothing writes `spark-sql-` entries, so once the last ones are evicted every Spark SQL build starts from the newest `java-maven-` entry. Since this morning that has been main's `Linux-java-maven-<hash>-delta-3.5`, saved by the nightly's Delta contrib (Spark 3.5) job. That job downloads the Arrow 12.0.1 POMs for `arrow-memory-core` and `arrow-memory-netty`, the Arrow version Spark 3.5 uses, but not their JARs. The "Purge partial Maven cache entries" step is there to remove such entries before sbt runs. It only removed a JAR-less POM that declares `<packaging>jar</packaging>` or `bundle`, and Arrow's POMs have no `<packaging>` element, so they survived. Coursier took them as found in mavenLocal, and sbt compiled Spark 3.5's `sql/api` without `org.apache.arrow.memory`: ```text [error] .../sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:24:25: object memory is not a member of package org.apache.arrow ``` A re-run restores the same cache and fails the same way. Today's nightly Spark 3.5 build passed only because a `spark-sql-` entry still existed when it ran, so the next nightly is exposed too. Spark 4.0 and 4.1 use a newer Arrow, which Comet's own build downloads in full, so their builds pass from the same cache. ## What changes are included in this PR? The purge step now treats a POM with no `<packaging>` element as a jar, which is Maven's default. POMs that declare any other packaging, such as parent POMs, are still kept. ## How are these changes tested? I ran the step's script, extracted from the action, against a scratch `~/.m2/repository` that held the real Arrow 12.0.1 POMs. Before the change it removed neither JAR-less Arrow POM. Now it removes the `arrow-memory-core` and `arrow-memory-netty` POMs and still removes a JAR-less POM that declares `jar`. It keeps the `pom`-packaging parents (`arrow-memory`, `arrow-java-root`) and an entry that has its JAR. `actionlint --shellcheck=off` and `dev/ci/check-ci-config.py` pass. The pull request tier on main does not run the Spark SQL suites. The same commit is on #6877, whose `branch-1.1` CI runs the Spark 3.5 Spark SQL build against the same fallback cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
