HyukjinKwon commented on a change in pull request #29312:
URL: https://github.com/apache/spark/pull/29312#discussion_r463402736
##########
File path: .github/workflows/master.yml
##########
@@ -271,3 +271,29 @@ jobs:
mkdir -p ~/.m2
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes
-Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install
rm -rf ~/.m2/repository/org/apache/spark
+
+ scala213:
+ name: Scala 2.13
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v2
+ - name: Cache Maven local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.m2/repository
+ key: scala213-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ scala213-maven-
+ - name: Install Java 11
+ uses: actions/setup-java@v1
+ with:
+ java-version: 11
+ - name: Test with Maven
+ run: |
+ export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g
-Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ mkdir -p ~/.m2
+ dev/change-scala-version.sh 2.13
+ build/mvn test -pl core --am -Pscala-2.13
Review comment:
My only concern is that it becomes difficult to port these to periodical
GitHub Actions jobs or merge it to the main GitHub Actions job (`Build modules:
...` above) later.
Eventually I would like to:
- put those different profiles build into the matrix in the main job `Build
modules: ...` above so we can leverage the optimization in `run-tests.py`
(running only relevant jobs, etc.). We could decide which profiles to run by
default or optionally.
- provide a way to set other optional profiles, for example, via GitHub PR
title or tags, which we currently does via Jenkins.
- set periodical jobs for other profiles for Maven, Scala, JDK, Hive,
Hadoop, etc. that we run in Jenkins, or possibly more. I think we need to
resolve SPARK-32264 first to do this.
Currently `run-tests.py` runs the build for all components always if Maven
is used. So, I was also thinking about fixing `run-tests.py` script to support
for Maven to run tests for only relevant components just like we do for SBT,
and then we only have one main job (`Build modules: ...`) that adds tests by
matrix or environment control in this file rather than adding separate jobs for
different profiles.
I guess I am okay with this as a temporary status for the time being if we
all agree with this plan. I would like to encourage us to stick to this plan in
long run.
Maybe we could alternatively just think about setting a Jenkins that that
runs all tests too, which will ideally be converted into periodic Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
