abhinav-phi commented on code in PR #19986:
URL: https://github.com/apache/hudi/pull/19986#discussion_r4190019745
##########
.github/workflows/scheduled_workflow.yml:
##########
@@ -34,11 +34,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Delete Cache
+ # Spark archives must survive the purge so integration-tests can reuse
them.
run: |
gh extension install actions/gh-actions-cache
echo "Fetching list of cache key"
- cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 )
+ cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 |
grep -v '^spark-archive-v1-' || true)
Review Comment:
done, the purge filter is anchored on the version independent prefix now and
the key in bot.yml points back at it
##########
.github/workflows/scheduled_workflow.yml:
##########
@@ -34,11 +34,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Delete Cache
+ # Spark archives must survive the purge so integration-tests can reuse
them.
Review Comment:
fixed properly in 7bb7a14a5.. your point was to cover trino once instead of
one prefix per PR, and the filter did add trino but with the literal version in
it, so a bump to trino-m2-v3 would have re-armed the purge and killed the cache
again.. it is ^trino-m2- now, which is version independent like the spark side,
and both trino keys (hudi_trino_ci.yml and hudi_trino_compat.yml) carry a
pointer comment back to scheduled_workflow.yml so the next person editing a key
sees the exception.. still open and your call: the setup-java maven caches from
the 13 cache: maven jobs are still purged every 5 minutes, I did not touch that
because it is a bigger question than this PR, tell me if you want it in scope
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
+ if: needs.changes.outputs.relevant == 'true'
+ id: spark-archive
+ uses: actions/cache/restore@v4
+ with:
+ path: ~/spark-archives/${{ matrix.sparkArchive }}
+ key: spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}
- name: 'IT'
if: needs.changes.outputs.relevant == 'true'
env:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ SPARK_ARCHIVE_CACHE_HIT: ${{ steps.spark-archive.outputs.cache-hit }}
run: |
# dlcdn only carries the current release of each line; fall back to
# the archive for older pins (#19883). Plain --retry, not
# --retry-all-errors, so a 404 on the CDN falls through immediately.
# --speed-limit is a dead-connection detector, not a slowness one:
# --retry truncates the output back to byte 0, so a floor set near
# the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ DEST="$HOME/spark-archives/$SPARK_ARCHIVE"
downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
+ if [ "$SPARK_ARCHIVE_CACHE_HIT" != 'true' ]; then
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
+ else
+ echo "Restored $SPARK_ARCHIVE from the actions cache"
fi
tar -xf "$DEST" -C $GITHUB_WORKSPACE/
mkdir /tmp/spark-events/
SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
- rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+ - name: Save Spark archive
+ if: needs.changes.outputs.relevant == 'true' && github.event_name ==
'push' && github.ref == 'refs/heads/master' &&
steps.spark-archive.outputs.cache-hit != 'true'
Review Comment:
validated on a temporary fork branch before it can run on master, run
35523562633 job 106111723466, gate widened and a non yml file touched so the
steps actually executed: restore miss, download, sha512sum, extract, save with
key spark-archive-v1-Linux-spark-3.5.9 (401321029 bytes) and the full mvn
verify.. tracking issue is #20011 as you suggested. three honest gaps I should
not paper over: the overall run shows cancelled on test-utilities, only the
cache miss path was exercised and never a hit, and the purge exception itself
cannot be fork validated because delete-all-caches is gated on
github.repository == apache/hudi.. I will post the log excerpts on #20011 so
the evidence is where the issue says it is
##########
.github/workflows/bot.yml:
##########
@@ -1509,43 +1509,60 @@ jobs:
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run:
mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: Restore Spark archive
+ if: needs.changes.outputs.relevant == 'true'
+ id: spark-archive
+ uses: actions/cache/restore@v4
+ with:
+ path: ~/spark-archives/${{ matrix.sparkArchive }}
+ key: spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}
- name: 'IT'
if: needs.changes.outputs.relevant == 'true'
env:
SPARK_PROFILE: ${{ matrix.sparkProfile }}
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ SPARK_ARCHIVE_CACHE_HIT: ${{ steps.spark-archive.outputs.cache-hit }}
run: |
# dlcdn only carries the current release of each line; fall back to
# the archive for older pins (#19883). Plain --retry, not
# --retry-all-errors, so a 404 on the CDN falls through immediately.
# --speed-limit is a dead-connection detector, not a slowness one:
# --retry truncates the output back to byte 0, so a floor set near
# the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ DEST="$HOME/spark-archives/$SPARK_ARCHIVE"
downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
+ if [ "$SPARK_ARCHIVE_CACHE_HIT" != 'true' ]; then
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
+ else
+ echo "Restored $SPARK_ARCHIVE from the actions cache"
fi
tar -xf "$DEST" -C $GITHUB_WORKSPACE/
mkdir /tmp/spark-events/
SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
- rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+ - name: Save Spark archive
Review Comment:
done, download and extract are split now and the save sits right after a
successful tar -xf, before mvn verify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
