gaogaotiantian commented on code in PR #56798:
URL: https://github.com/apache/spark/pull/56798#discussion_r3494507295
##########
.github/workflows/python_hosted_runner_test.yml:
##########
@@ -56,8 +56,83 @@ on:
type: string
default: '{}'
jobs:
+ # Precompile Spark with SBT once and publish target/ as an artifact for the
+ # matrix entries below to consume. Optional: any failure here degrades the
+ # matrix to its original local SBT build path.
+ precompile:
+ name: "Precompile Spark"
+ runs-on: ${{ inputs.os }}
+ # If this job fails or is cancelled, the matrix entries fall back to
+ # running the SBT build locally as before.
+ continue-on-error: true
+ env:
+ HADOOP_PROFILE: ${{ inputs.hadoop }}
+ HIVE_PROFILE: hive2.3
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ - name: Cache SBT and Maven
+ # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
+ if: ${{ runner.os != 'macOS' }}
+ uses: actions/cache@v4
+ with:
+ path: |
+ build/apache-maven-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-${{ runner.os }}-
+ - name: Cache Coursier local repository
+ # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
+ if: ${{ runner.os != 'macOS' }}
+ uses: actions/cache@v4
+ with:
+ path: ~/.cache/coursier
+ key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
Review Comment:
Yeah this yml file is never used, either in branch-4.1 or branch-4.x. I
backported the changes only to avoid future conflicts for backporting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
