abhinav-phi commented on code in PR #19986:
URL: https://github.com/apache/hudi/pull/19986#discussion_r4057600883
##########
.github/workflows/scheduled_workflow.yml:
##########
@@ -34,11 +34,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Delete Cache
+ # Spark archives must survive the purge so integration-tests can reuse
them.
Review Comment:
Done - fixed the purge once instead of per-prefix: the filter now also
excludes ^trino-m2-v2-, so the dead trino-m2 cache in hudi_trino_ci.yml gets
the same protection as the spark archive. Aapne jo bola tha, wahi kiya.
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
+ if: needs.changes.outputs.relevant == 'true'
+ id: spark-archive
+ uses: actions/cache/restore@v4
+ with:
+ path: ~/spark-archives/${{ matrix.sparkArchive }}
+ key: spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}
- name: 'IT'
if: needs.changes.outputs.relevant == 'true'
env:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ SPARK_ARCHIVE_CACHE_HIT: ${{ steps.spark-archive.outputs.cache-hit }}
run: |
# dlcdn only carries the current release of each line; fall back to
# the archive for older pins (#19883). Plain --retry, not
# --retry-all-errors, so a 404 on the CDN falls through immediately.
# --speed-limit is a dead-connection detector, not a slowness one:
# --retry truncates the output back to byte 0, so a floor set near
# the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ DEST="$HOME/spark-archives/$SPARK_ARCHIVE"
downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
+ if [ "$SPARK_ARCHIVE_CACHE_HIT" != 'true' ]; then
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
+ else
+ echo "Restored $SPARK_ARCHIVE from the actions cache"
fi
tar -xf "$DEST" -C $GITHUB_WORKSPACE/
mkdir /tmp/spark-events/
SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
- rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+ - name: Save Spark archive
+ if: needs.changes.outputs.relevant == 'true' && github.event_name ==
'push' && github.ref == 'refs/heads/master' &&
steps.spark-archive.outputs.cache-hit != 'true'
Review Comment:
Done - validated before master as suggested. Tracking issue #20011;
temporary fork branch run is fully green: restore-miss, download, sha512sum,
extract, save, and the complete IT (mvn verify) with the new SPARK_HOME wiring.
The fork cache now holds the 382.72 MiB entry, proving the save gate end to
end. Temp scaffolding deleted; none of it is in this PR.
##########
.github/workflows/scheduled_workflow.yml:
##########
@@ -34,11 +34,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Delete Cache
+ # Spark archives must survive the purge so integration-tests can reuse
them.
run: |
gh extension install actions/gh-actions-cache
echo "Fetching list of cache key"
- cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 )
+ cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 |
grep -v '^spark-archive-v1-' || true)
Review Comment:
done, the purge filter is anchored on the version independent prefix now and
the key in bot.yml points back at it
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
+ if: needs.changes.outputs.relevant == 'true'
+ id: spark-archive
+ uses: actions/cache/restore@v4
+ with:
+ path: ~/spark-archives/${{ matrix.sparkArchive }}
+ key: spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}
- name: 'IT'
if: needs.changes.outputs.relevant == 'true'
env:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ SPARK_ARCHIVE_CACHE_HIT: ${{ steps.spark-archive.outputs.cache-hit }}
run: |
# dlcdn only carries the current release of each line; fall back to
# the archive for older pins (#19883). Plain --retry, not
# --retry-all-errors, so a 404 on the CDN falls through immediately.
# --speed-limit is a dead-connection detector, not a slowness one:
# --retry truncates the output back to byte 0, so a floor set near
# the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ DEST="$HOME/spark-archives/$SPARK_ARCHIVE"
downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
+ if [ "$SPARK_ARCHIVE_CACHE_HIT" != 'true' ]; then
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
+ else
+ echo "Restored $SPARK_ARCHIVE from the actions cache"
fi
tar -xf "$DEST" -C $GITHUB_WORKSPACE/
mkdir /tmp/spark-events/
SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
- rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+ - name: Save Spark archive
Review Comment:
done, download and extract are split now and the save sits right after a
successful tar -xf, before mvn verify
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
Review Comment:
done, Impact states the measured numbers and the claim is framed as removing
the 382MB egress plus the stale pin tail risk
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
+ if: needs.changes.outputs.relevant == 'true'
+ id: spark-archive
+ uses: actions/cache/restore@v4
+ with:
+ path: ~/spark-archives/${{ matrix.sparkArchive }}
+ key: spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}
- name: 'IT'
if: needs.changes.outputs.relevant == 'true'
env:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ SPARK_ARCHIVE_CACHE_HIT: ${{ steps.spark-archive.outputs.cache-hit }}
run: |
# dlcdn only carries the current release of each line; fall back to
# the archive for older pins (#19883). Plain --retry, not
# --retry-all-errors, so a 404 on the CDN falls through immediately.
# --speed-limit is a dead-connection detector, not a slowness one:
# --retry truncates the output back to byte 0, so a floor set near
# the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ DEST="$HOME/spark-archives/$SPARK_ARCHIVE"
downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
+ if [ "$SPARK_ARCHIVE_CACHE_HIT" != 'true' ]; then
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
+ else
+ echo "Restored $SPARK_ARCHIVE from the actions cache"
fi
tar -xf "$DEST" -C $GITHUB_WORKSPACE/
Review Comment:
done, sha512sum -c runs on both the restored and the downloaded path against
Apache's published .sha512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
