Copilot commented on code in PR #12529:
URL: https://github.com/apache/gluten/pull/12529#discussion_r3593547448
##########
.github/workflows/velox_nightly.yml:
##########
@@ -179,6 +180,103 @@ jobs:
path: package/target/gluten-velox-bundle-*.jar
retention-days: 7
+ spark-test-spark35-smj:
+ if: ${{ startsWith(github.repository, 'apache/') }}
+ needs: build-native-lib-x86
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare
Review Comment:
The native Velox build artifact uploaded by `build-native-lib-x86` is
`./cpp/build/releases/`, but this job downloads it into `./cpp/build/`. Because
the CMake build places `libgluten.so` under `cpp/build/releases` (see
`cpp/core/CMakeLists.txt`), downloading into `./cpp/build/` will extract the
libraries into the wrong location and can cause Maven tests to fail to find the
native libs.
##########
.github/workflows/velox_nightly.yml:
##########
@@ -179,6 +180,103 @@ jobs:
path: package/target/gluten-velox-bundle-*.jar
retention-days: 7
+ spark-test-spark35-smj:
+ if: ${{ startsWith(github.repository, 'apache/') }}
+ needs: build-native-lib-x86
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare
+ run: |
+ dnf module -y install python39 && \
+ alternatives --set python3 /usr/bin/python3.9 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 cython && \
+ pip3 install pandas==2.2.3 pyarrow==20.0.0
+ - name: Build and Run unit test for Spark 3.5.5 (other tests)
+ run: |
+ cd $GITHUB_WORKSPACE/
+ export SPARK_SCALA_VERSION=2.12
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Ppaimon -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/
-Dspark.gluten.sql.columnar.forceShuffledHashJoin=false" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.SkipTest
+ - name: Upload test report
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-report
+ path: '**/surefire-reports/TEST-*.xml'
+ - name: Upload unit tests log files
+ if: ${{ !success() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-test-log
+ path: |
+ **/target/*.log
+ **/gluten-ut/**/hs_err_*.log
+ **/gluten-ut/**/core.*
+
+ spark-test-spark35-slow-smj:
+ if: ${{ startsWith(github.repository, 'apache/') }}
+ needs: build-native-lib-x86
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ LANG: C.UTF-8 #TODO remove after image update
+ LC_ALL: C.UTF-8
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare Spark Resources for Spark 3.5.5 #TODO remove after image
update
Review Comment:
Same artifact-path issue as the other Spark 3.5 test job:
`build-native-lib-x86` uploads from `./cpp/build/releases/`, so downloading
into `./cpp/build/` will place shared libraries outside the expected
`cpp/build/releases` directory and can break test execution.
##########
.github/workflows/velox_nightly.yml:
##########
@@ -179,6 +180,103 @@ jobs:
path: package/target/gluten-velox-bundle-*.jar
retention-days: 7
+ spark-test-spark35-smj:
+ if: ${{ startsWith(github.repository, 'apache/') }}
+ needs: build-native-lib-x86
+ runs-on: ubuntu-22.04
Review Comment:
The PR title says these Spark 3.5 SMJ-enabled full-test jobs are being moved
into the **weekly** workflow, but the change adds them to `velox_nightly.yml`
(nightly release workflow). This looks like a mismatch between the intended
scheduling and the actual workflow being modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
