baibaichen commented on code in PR #13126:
URL: https://github.com/apache/gluten/pull/13126#discussion_r4118183307
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1244,6 +1246,21 @@ jobs:
cd $GITHUB_WORKSPACE/
$MVN_CMD clean test-compile -Pspark-4.0 -Pscala-2.13
-Pbackends-velox -Pspark-ut -Piceberg,iceberg-test,delta,paimon -Pfast-build
+ build-test-spark42:
+ needs: detect-changes
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims42 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-8-jdk17
+ steps:
+ - uses: actions/checkout@v7
+ - name: Compile Spark 4.2 shim (Java 17, Scala 2.13)
+ run: |
+ cd $GITHUB_WORKSPACE/
+ $MVN_CMD clean test-compile -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -pl shims/spark42 -am -DskipTests
Review Comment:
Could you clarify why the Spark 4.2 lane only selects `shims/spark42` here,
instead of building the full Velox reactor as the Spark 4.1 lane does?
With `-pl shims/spark42 -am`, Maven builds the shim and its upstream
dependencies, but it does not build downstream modules such as
`gluten-substrait` or `backends-velox`. As a result, Spark 4.2 compatibility
changes in the Velox integration—for example
`ColumnarArrowEvalPythonExec.scala`—are not compiled under the Spark 4.2
profile.
Is this an intentional temporary workaround because the full Spark 4.2 Velox
build is not ready yet? If not, should this select `backends-velox -am`, or
follow the Spark 4.1 lane without `-pl`, so the integration code is also
compiled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
