LuciferYang commented on code in PR #56201:
URL: https://github.com/apache/spark/pull/56201#discussion_r3349605034
##########
.github/workflows/build_and_test.yml:
##########
@@ -382,11 +382,10 @@ jobs:
uses: actions/cache@v5
with:
path: ~/.cache/coursier
- key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{
hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop
}}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
- precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
- precompile-coursier-
+ ${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ coursier-
Review Comment:
Why was the `coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}`
cache key no longer used for matching?
##########
.github/workflows/build_and_test.yml:
##########
@@ -1433,15 +1432,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
Review Comment:
The `tpcds-1g` Coursier step is also restore-only, but the job only depends
on `precondition`, not `precompile`. Normal SQL changes and explicit full
workflows can trigger both jobs, yet `tpcds-1g` may start before `precompile`
saves the shared cache; the old `tpcds-coursier-*` writer was also removed.
TPC-DS can still cold-download dependencies in that run.
##########
.github/workflows/build_and_test.yml:
##########
@@ -1553,15 +1550,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
Review Comment:
These downstream jobs depend on `precompile`, but `precompile` is
`continue-on-error`. If `precompile` times out or fails, the downstream jobs
intentionally continue with local builds. Their Coursier steps are now
restore-only, so the dependencies downloaded in that fallback path are not
saved. The precompile fallback path can repeatedly cold-download dependencies.
Maybe we should keep a writable cache/save path for these fallback jobs when
`precompile` did not produce the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
