This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 27c0ccc [SPARK-33464][INFRA][2.4] Add/remove (un)necessary cache and
restructure GitHub Actions yaml
27c0ccc is described below
commit 27c0ccc24bf66a5419dac1f0a4887dd40cd16774
Author: HyukjinKwon <gurwls...@apache.org>
AuthorDate: Thu Nov 19 14:22:44 2020 +0900
[SPARK-33464][INFRA][2.4] Add/remove (un)necessary cache and restructure
GitHub Actions yaml
### What changes were proposed in this pull request?
This PR backports https://github.com/apache/spark/pull/30391. Note that
it's a partial backport.
This PR proposes:
- Add `~/.sbt` directory into the build cache, see also
https://github.com/sbt/sbt/issues/3681
- ~Move `hadoop-2` below to put up together with `java-11` and `scala-213`,
see https://github.com/apache/spark/pull/30391#discussion_r524881430~
- Remove unnecessary `.m2` cache if you run SBT tests only.
- Remove `rm ~/.m2/repository/org/apache/spark`. If you don't `sbt
publishLocal` or `mvn install`, we don't need to care about it.
- ~Use Java 8 in Scala 2.13 build. We can switch the Java version to 11
used for release later.~
- Add caches into linters. The linter scripts uses `sbt` in, for example,
`./dev/lint-scala`, and uses `mvn` in, for example, `./dev/lint-java`. Also, it
requires to `sbt package` in Jekyll build, see:
https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L160-L161.
We need full caches here for SBT, Maven and build tools.
- Use the same syntax of Java version, 1.8 -> 8.
### Why are the changes needed?
- Remove unnecessary stuff
- Cache what we can in the build
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
It will be tested in GitHub Actions build at the current PR
Closes #30417 from HyukjinKwon/SPARK-33464-2.4.
Authored-by: HyukjinKwon <gurwls...@apache.org>
Signed-off-by: HyukjinKwon <gurwls...@apache.org>
---
.github/workflows/build_and_test.yml | 44 +++++++++++++++++++++++-------------
1 file changed, 28 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index b69932e..991313b 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -18,7 +18,7 @@ jobs:
fail-fast: false
matrix:
java:
- - 1.8
+ - 8
hadoop:
- hadoop2.6
# TODO(SPARK-32246): We don't test 'streaming-kinesis-asl' for now.
@@ -51,12 +51,12 @@ jobs:
include:
# Hive tests
- modules: hive
- java: 1.8
+ java: 8
hadoop: hadoop2.6
included-tags: org.apache.spark.tags.SlowHiveTest
comment: "- slow tests"
- modules: hive
- java: 1.8
+ java: 8
hadoop: hadoop2.6
excluded-tags: org.apache.spark.tags.SlowHiveTest
comment: "- other tests"
@@ -84,16 +84,10 @@ jobs:
build/zinc-*
build/scala-*
build/*.jar
+ ~/.sbt
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Maven local repository
- uses: actions/cache@v2
- with:
- path: ~/.m2/repository
- key: ${{ matrix.java }}-${{ matrix.hadoop }}-maven-${{
hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ matrix.java }}-${{ matrix.hadoop }}-maven-
- name: Cache Ivy local repository
uses: actions/cache@v2
with:
@@ -101,7 +95,7 @@ jobs:
key: ${{ matrix.java }}-${{ matrix.hadoop }}-ivy-${{
hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
${{ matrix.java }}-${{ matrix.hadoop }}-ivy-
- - name: Install JDK ${{ matrix.java }}
+ - name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}
@@ -160,9 +154,7 @@ jobs:
run: |
# Hive tests become flaky when running in parallel as it's too
intensive.
if [[ "$MODULES_TO_TEST" == "hive" ]]; then export SERIAL_SBT_TESTS=1;
fi
- mkdir -p ~/.m2
./dev/run-tests --parallelism 2 --modules "$MODULES_TO_TEST"
--included-tags "$INCLUDED_TAGS" --excluded-tags "$EXCLUDED_TAGS"
- rm -rf ~/.m2/repository/org/apache/spark
- name: Upload test results to report
if: always()
uses: actions/upload-artifact@v2
@@ -183,17 +175,37 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT, Maven and Zinc
+ uses: actions/cache@v2
+ with:
+ path: |
+ build/apache-maven-*
+ build/zinc-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Ivy local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.ivy2/cache
+ key: docs-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ docs-ivy-
- name: Cache Maven local repository
uses: actions/cache@v2
with:
path: ~/.m2/repository
- key: docs-maven-repo-${{ hashFiles('**/pom.xml') }}
+ key: docs-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
docs-maven-
- - name: Install JDK 1.8
+ - name: Install Java 8
uses: actions/setup-java@v1
with:
- java-version: 1.8
+ java-version: 8
- name: Install Python 3.6
uses: actions/setup-python@v2
with:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
