comphead commented on code in PR #6113:
URL: https://github.com/apache/datafusion-comet/pull/6113#discussion_r4637039471


##########
.github/workflows/iceberg_spark_test_reusable.yml:
##########
@@ -238,21 +241,52 @@ jobs:
           ENABLE_COMET=true ENABLE_COMET_ONHEAP=true 
COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT=1 
COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \
             ./gradlew -DsparkVersions=${{ inputs.spark-short }} 
-DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
             :iceberg-spark:iceberg-spark-extensions-${{ inputs.spark-short 
}}_${{ inputs.scala }}:test \
+            --init-script ../dev/ci/iceberg-test-shards.gradle \
+            -PcometShardTask=:iceberg-spark:iceberg-spark-extensions-${{ 
inputs.spark-short }}_${{ inputs.scala }}:test \
+            -PcometShardIndex=${{ matrix.shard }} -PcometShardCount=${{ 
needs.build-native.outputs.shard-count }} \
             -Pquick=true -x javadoc
       - name: Summarize Iceberg writes
         if: ${{ !cancelled() }}
         run: |
-          python3 dev/ci/summarize-iceberg-writes.py --title 
"iceberg-spark-extensions" \
+          python3 dev/ci/summarize-iceberg-writes.py --title 
"iceberg-spark-extensions shard ${{ matrix.shard }}" \
             apache-iceberg/build/comet-iceberg-writes
-      - name: Upload Iceberg write report
+      - name: Upload Iceberg shard inventory and test reports
         if: ${{ !cancelled() }}
+        # iceberg-spark-extensions-shard-coverage downloads the inventory, so a
+        # flaky upload here fails that job too. Retry rather than fail on 403.
         uses: ./.github/actions/upload-artifact-retry
         with:
-          name: iceberg-spark-extensions-writes-${{ inputs.iceberg-full 
}}-spark-${{ inputs.spark-full }}-scala-${{ inputs.scala }}-jdk${{ inputs.java 
}}-attempt-${{ github.run_attempt }}
-          path: apache-iceberg/build/comet-iceberg-writes/*.jsonl
-          if-no-files-found: ignore
+          name: iceberg-spark-extensions-${{ inputs.iceberg-full }}-spark-${{ 
inputs.spark-full }}-scala-${{ inputs.scala }}-jdk${{ inputs.java }}-shard-${{ 
matrix.shard }}-attempt-${{ github.run_attempt }}
+          path: |
+            apache-iceberg/**/build/comet-shards/*.json
+            apache-iceberg/**/build/test-results/test/*.xml
+            apache-iceberg/build/comet-iceberg-writes/*.jsonl
           retention-days: 7
 
+  iceberg-spark-extensions-shard-coverage:
+    needs: iceberg-spark-extensions
+    if: ${{ !cancelled() }}
+    name: iceberg-spark-extensions-shard-coverage/iceberg-${{ 
inputs.iceberg-full }}/spark-${{ inputs.spark-full }}/scala-${{ inputs.scala 
}}/java-${{ inputs.java }}
+    runs-on: ubuntu-slim

Review Comment:
   After the rebase this new job also needs a `timeout-minutes`. 
`.github/workflows/README.md` now says every job that runs on a runner sets 
one. The existing `iceberg-spark-shard-coverage` job uses 15, which I expect 
fits here too since it only downloads the inventories and runs two Python 
scripts.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to