abhinav-phi commented on code in PR #19986:
URL: https://github.com/apache/hudi/pull/19986#discussion_r4057606220
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
+ if: needs.changes.outputs.relevant == 'true'
+ id: spark-archive
+ uses: actions/cache/restore@v4
+ with:
+ path: ~/spark-archives/${{ matrix.sparkArchive }}
+ key: spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}
- name: 'IT'
if: needs.changes.outputs.relevant == 'true'
env:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ SPARK_ARCHIVE_CACHE_HIT: ${{ steps.spark-archive.outputs.cache-hit }}
run: |
# dlcdn only carries the current release of each line; fall back to
# the archive for older pins (#19883). Plain --retry, not
# --retry-all-errors, so a 404 on the CDN falls through immediately.
# --speed-limit is a dead-connection detector, not a slowness one:
# --retry truncates the output back to byte 0, so a floor set near
# the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ DEST="$HOME/spark-archives/$SPARK_ARCHIVE"
downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
+ if [ "$SPARK_ARCHIVE_CACHE_HIT" != 'true' ]; then
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
+ else
+ echo "Restored $SPARK_ARCHIVE from the actions cache"
fi
tar -xf "$DEST" -C $GITHUB_WORKSPACE/
mkdir /tmp/spark-events/
SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
- rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+ - name: Save Spark archive
+ if: needs.changes.outputs.relevant == 'true' && github.event_name ==
'push' && github.ref == 'refs/heads/master' &&
steps.spark-archive.outputs.cache-hit != 'true'
Review Comment:
Done - validated before master as suggested. Tracking issue #20011;
temporary fork branch run is fully green: restore-miss, download, sha512sum,
extract, save, and the complete IT (mvn verify) with the new SPARK_HOME wiring.
The fork cache now holds the 382.72 MiB entry, proving the save gate end to
end. Temp scaffolding deleted; none of it is in this PR.
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
+ if: needs.changes.outputs.relevant == 'true'
+ id: spark-archive
+ uses: actions/cache/restore@v4
+ with:
+ path: ~/spark-archives/${{ matrix.sparkArchive }}
+ key: spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}
- name: 'IT'
if: needs.changes.outputs.relevant == 'true'
env:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ SPARK_ARCHIVE_CACHE_HIT: ${{ steps.spark-archive.outputs.cache-hit }}
run: |
# dlcdn only carries the current release of each line; fall back to
# the archive for older pins (#19883). Plain --retry, not
# --retry-all-errors, so a 404 on the CDN falls through immediately.
# --speed-limit is a dead-connection detector, not a slowness one:
# --retry truncates the output back to byte 0, so a floor set near
# the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ DEST="$HOME/spark-archives/$SPARK_ARCHIVE"
downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
+ if [ "$SPARK_ARCHIVE_CACHE_HIT" != 'true' ]; then
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
+ else
+ echo "Restored $SPARK_ARCHIVE from the actions cache"
fi
tar -xf "$DEST" -C $GITHUB_WORKSPACE/
mkdir /tmp/spark-events/
SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
- rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+ - name: Save Spark archive
Review Comment:
Done as suggested (non-blocking, implemented anyway): the download/extract
now lives in its own Fetch step, checksum plus tar run in Verify-and-extract,
and Save sits immediately after a successful tar -xf and before mvn verify. The
extract itself is the integrity gate on seeding now, decoupled from master IT
health.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
