akshaytayal commented on code in PR #13520:
URL: https://github.com/apache/gluten/pull/13520#discussion_r4648673251
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1669,3 +1686,150 @@ jobs:
**/target/*.log
**/gluten-ut/**/hs_err_*.log
**/gluten-ut/**/core.*
+
+ spark-test-spark43:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims43 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ # Split tests into 3 groups to run in parallel and cut the ~2h
wall-clock time.
+ # group1 – streaming tests
+ # group2 – execution / catalyst / errors / extension tests
+ # group3 – top-level sql, connector, sources, hive and remaining
tests
+ group: [1, 2, 3]
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v7
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v8
+ with:
+ name: velox-native-lib-centos-8-${{github.sha}}
+ path: ./cpp/build/releases/
+ - name: Prepare
+ run: |
+ dnf install -y python3.11 python3.11-pip python3.11-devel && \
+ ls -la /usr/bin/python3.11 && \
+ alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11
1 && \
+ alternatives --set python3 /usr/bin/python3.11 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 cython && \
+ pip3 install pandas==2.2.3 pyarrow==20.0.0
+ - name: Prepare Spark 4.3 resources
+ run: |
+ # TODO: Remove once the CI images provide /opt/shims/spark43.
+ if [ ! -d /opt/shims/spark43/spark_home ]; then
+ bash .github/workflows/util/install-spark-resources.sh 4.3
+ fi
+ - name: Build and Run unit test for Spark 4.3.0 with scala-2.13 (other
tests, group ${{ matrix.group }})
+ run: |
+ cd $GITHUB_WORKSPACE/
+ export SPARK_SCALA_VERSION=2.13
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+
TAGS_EXCLUDE="org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest"
+ # Install all modules to the local repo first so gluten-ut resolves
its
+ # sibling Gluten modules (gluten-substrait etc.) before testing.
+ $MVN_CMD clean install -Pspark-4.3 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -DskipTests
+ if [ "${{ matrix.group }}" = "1" ]; then
+ # Group 1: streaming + gluten utils + top-level spark tests (~55
classes)
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.3 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark43/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
+
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
Review Comment:
Added in ef3d8f0. `org.apache.spark.util.SparkResourceUtilSuite` is now
selected in the first Spark 4.3 test group, along with the new
`GlutenShuffleBlockFetcherIteratorSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]