This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6079-39c686dba59d542200b2a17df0ca3bc6f5f8b6fb in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit d0da09e9b491145bb54f7a5fe6a13e8d9b50f625 Author: Andy Grove <[email protected]> AuthorDate: Tue Sep 22 13:36:27 2026 +0000 ci: retry the Spark test pre-compile step on resolution failures (#6079) The Pre-compile Spark Test classes step resolves Spark's entire plugin and dependency graph from Maven Central on a cold runner, which makes it the largest transient-failure surface in the Spark SQL tier. A connection reset there kills the whole job for that Spark version before a single test runs. Wrap only that invocation in a retry that fires solely on a dependency resolution failure, so a genuine compile error still fails on the first attempt instead of burning three more lengthy runs. The step now runs under bash rather than the container default sh, which the retry needs for pipefail, RANDOM, and shell arithmetic. Closes #6078 --- .github/workflows/spark_sql_test_reusable.yml | 36 ++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/.github/workflows/spark_sql_test_reusable.yml b/.github/workflows/spark_sql_test_reusable.yml index 9fa254f9b6..ffcc36d7eb 100644 --- a/.github/workflows/spark_sql_test_reusable.yml +++ b/.github/workflows/spark_sql_test_reusable.yml @@ -149,7 +149,11 @@ jobs: skip-native-build: true - name: Pre-compile Spark Test classes + # Bash rather than the container default `sh`: the retry below needs + # pipefail, $RANDOM, and shell arithmetic. + shell: bash run: | + set -o pipefail cd apache-spark # Mirror the workaround from `Run Spark tests` below: Comet's mvn # install populates partial Parquet entries (main JAR + POM but no @@ -163,10 +167,34 @@ jobs: # plus any Test/compile pulled in by `dependsOn(... % "test->test")`, # which is enough to satisfy `sql/testOnly`, `hive/testOnly`, and # `catalyst/test` in the downstream test jobs. - NOLINT_ON_COMPILE=true build/sbt -Dsbt.log.noformat=true -mem 3072 \ - 'catalyst/Test/compile' \ - 'sql/Test/compile' \ - 'hive/Test/compile' + # + # This is also the step that resolves Spark's entire plugin and + # dependency graph from Maven Central on a cold runner, so it is + # where a Central hiccup is most likely to kill the job before + # anything has been built. Retry that, and only that: a genuine + # compile error still fails on the first attempt instead of burning + # three more ~12 minute runs. Zinc's incremental analysis means a + # retry resumes rather than recompiling from scratch. + log=$(mktemp) + for attempt in 1 2 3; do + if NOLINT_ON_COMPILE=true build/sbt -Dsbt.log.noformat=true -mem 3072 \ + 'catalyst/Test/compile' \ + 'sql/Test/compile' \ + 'hive/Test/compile' 2>&1 | tee "$log"; then + exit 0 + fi + if ! grep -qE 'ResolveException|download error|Connection reset' "$log"; then + echo "::error::sbt failed for a non-resolution reason; not retrying." + exit 1 + fi + if [ "$attempt" -eq 3 ]; then + echo "::error::sbt could not resolve dependencies after $attempt attempts." + exit 1 + fi + delay=$((10 * (1 << (attempt - 1)) + RANDOM % 5)) + echo "::warning::Dependency resolution failed; retrying in ${delay}s." + sleep "$delay" + done - name: Pack apache-spark/ (sources + compiled output) run: | --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
