andygrove opened a new pull request, #6079: URL: https://github.com/apache/datafusion-comet/pull/6079
## Which issue does this PR close? Closes #6078. ## Rationale for this change The nightly run on 2026-09-20 ([run 35494125457](https://github.com/apache/datafusion-comet/actions/runs/35494125457), reported as #6057) failed in `Spark SQL Tests (Spark 4.0) / Build Native + JVM Test Classes`. sbt was still loading Spark's project definition when coursier took a connection reset from Maven Central fetching a POM for `com.typesafe:mima-core_2.12`, a dependency of Spark's own MiMa plugin: ``` [error] sbt.librarymanagement.ResolveException: Error downloading com.typesafe:mima-core_2.12:1.1.4 [error] download error: Caught java.net.SocketException (Connection reset) while downloading https://repo1.maven.org/maven2/com/typesafe/mima-core_2.12/1.1.4/mima-core_2.12-1.1.4.pom ``` Nothing had been compiled yet and no Comet code was involved, but the whole Spark SQL job for that version died and `Required Checks` went red behind it. Downloads driven by `./mvnw` are already protected against this: `.github/actions/maven-bootstrap` retries the wrapper download with exponential backoff so that, in its own words, "a Maven Central hiccup does not fail a job before it has built or tested anything." The sbt path had no equivalent, even though the pre-compile step is the one that resolves Spark's whole plugin and dependency graph on a cold runner. ## What changes are included in this PR? The `Pre-compile Spark Test classes` step in `spark_sql_test_reusable.yml` now retries, up to three attempts with exponential backoff and jitter, but only when the failure output matches a dependency resolution signature. Any other failure, a real compile error above all, still exits on the first attempt rather than burning three more runs of a step that takes roughly twelve minutes. Zinc's incremental analysis means a retry resumes rather than recompiling from scratch. The step also moves from the container default `sh` to `bash`, which the retry needs for `pipefail`, `RANDOM`, and shell arithmetic. Scope is deliberately limited to this one step. The `Run Spark tests` step and the writer tests run after the dependency cache is warm, and retrying test execution is what the `maven-bootstrap` precedent explicitly warns against. ## How are these changes tested? `dev/ci/check-ci-config.py` passes, and the edited workflow parses as YAML with the step resolving to `shell: bash`. The retry logic itself was exercised by extracting the step's `run` block from the workflow, stubbing `build/sbt`, and running it against four scenarios: | Scenario | Exit | sbt invocations | Result | |---|---|---|---| | compiles first try | 0 | 1 | no retry | | resolution failure, then success | 0 | 2 | retried and recovered | | resolution failure every time | 1 | 3 | gives up with an error annotation | | compile error | 1 | 1 | fails immediately, no retry | The path that matters in production, a transient Central failure followed by success, is the second row; the fourth confirms a real compile error is not retried. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
