LuciferYang commented on code in PR #41529:
URL: https://github.com/apache/spark/pull/41529#discussion_r1223872778
##########
.github/workflows/build_and_test.yml:
##########
@@ -728,6 +729,83 @@ jobs:
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud
-Djava.version=${JAVA_VERSION/-ea} install
rm -rf ~/.m2/repository/org/apache/spark
+ connect-maven:
+ needs: precondition
+ if: fromJson(needs.precondition.outputs.required).connect-maven == 'true'
+ name: Test connect modules with Maven using Java ${{ matrix.java }}
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ runs-on: ubuntu-22.04
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Maven local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.m2/repository
+ key: java${{ matrix.java }}-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ java${{ matrix.java }}-maven-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Build and Test with Maven
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in
tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm
typescript
+ # `set -e` to make the exit status as expected due to use `script -q
-e -c` to run the commands
Review Comment:
another way is add a new script, maybe named `dev/run-connect-maven-tests`
as follows:
```shell
#!/usr/bin/env bash
set -e
# Go to the Spark project root directory
FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
cd "$FWDIR"
export SPARK_HOME=$FWDIR
echo "$SPARK_HOME"
if [[ -z "$JAVA_VERSION" ]]; then
JAVA_VERSION=8
fi
export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g
-Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
export MAVEN_CLI_OPTS="--no-transfer-progress"
# 1. Test with -Phive
# It uses Maven's 'install' intentionally, see
https://github.com/apache/spark/pull/26414.
build/mvn $MAVEN_CLI_OPTS -DskipTests -Djava.version=${JAVA_VERSION/-ea}
install -Phive
build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} test -pl
connector/connect/client/jvm -Phive
# 2. Test without -Phive
build/mvn $MAVEN_CLI_OPTS -DskipTests -Djava.version=${JAVA_VERSION/-ea}
install -pl assembly
build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} test -pl
connector/connect/client/jvm
```
and `run:` can change to:
```
# Fix for TTY related issues when launching the Ammonite REPL in
tests.
export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm
typescript
export JAVA_VERSION=${{ matrix.java }}
./dev/run-connect-maven-tests
TEST_RETCODE=$?
rm -rf ~/.m2/repository/org/apache/spark
exit $TEST_RETCODE
```
I tested it and it worked, but due to lack of this script in branch-3.4, we
need to add `"connect-maven" : "false"` to `build_branch34.yml` when using
script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
