Copilot commented on code in PR #13421:
URL: https://github.com/apache/gluten/pull/13421#discussion_r4364193666
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1652,3 +1657,94 @@ jobs:
**/target/*.log
**/gluten-ut/**/hs_err_*.log
**/gluten-ut/**/core.*
+
+ # Runs upstream Iceberg tests on the static native libraries built above for
the head
+ # revision; see docs/backends/velox/developers/velox-backend-CI.md.
+ iceberg-spark-ut:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ permissions:
+ contents: read
+ actions: read
+ pull-requests: read
+ uses: ./.github/workflows/iceberg_spark_ut.yml
+ with:
+ native_artifact: velox-native-lib-centos-8-${{github.sha}}
+
+ # The Kafka native tests run in cpp-test-udf-test, whose build enables Kafka.
+ spark-test-kafka-spark35:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ runs-on: ubuntu-22.04
+ timeout-minutes: 60
+ env:
+ KAFKA_BOOTSTRAP_SERVERS: localhost:9092
+ SPARK_LOCAL_IP: 127.0.0.1
+ services:
+ kafka:
+ image: apache/kafka:3.9.1
+ ports:
+ - 9092:9092
+ env:
+ CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk
+ KAFKA_NODE_ID: 1
+ KAFKA_PROCESS_ROLES: broker,controller
+ KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
+ KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
+ KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
+ KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+ KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
+ KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+ KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+ KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+ KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
+ options: >-
+ --health-cmd "/opt/kafka/bin/kafka-topics.sh --bootstrap-server
localhost:9092 --list"
+ --health-interval 10s
+ --health-timeout 10s
+ --health-retries 12
+ steps:
+ - uses: actions/checkout@v7
+ - name: Setup java
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y openjdk-17-jdk
+ echo "JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64" >> $GITHUB_ENV
+ echo "/usr/lib/jvm/java-17-openjdk-amd64/bin" >> $GITHUB_PATH
+ - name: Download All Native Artifacts
+ uses: actions/download-artifact@v8
+ with:
+ name: velox-native-lib-centos-8-${{github.sha}}
+ path: ./cpp/build/releases/
+ - name: Install Spark 3.5 modules with Kafka support
+ run: |
+ ./build/mvn -B -ntp install -Pbackends-velox -Pspark-3.5
-Pscala-2.12 \
+ -Pjava-17 -Pkafka -Pspark-ut -Pfast-build -DskipTests
+ # TODO: The suites below read Kafka natively, so re-enabling them needs
a static build
+ # with --enable_kafka=ON (dev/ci-velox-buildstatic-centos-8.sh does not
enable it).
Review Comment:
The job name `spark-test-kafka-spark35` is misleading because it currently
only performs a Maven install with `-DskipTests` and explicitly has Kafka test
suites TODO/disabled. Consider renaming the job to reflect its current behavior
(e.g., `spark-build-kafka-spark35`), or re-enable at least one Kafka test step
so the name matches what it actually does.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1652,3 +1657,94 @@ jobs:
**/target/*.log
**/gluten-ut/**/hs_err_*.log
**/gluten-ut/**/core.*
+
+ # Runs upstream Iceberg tests on the static native libraries built above for
the head
+ # revision; see docs/backends/velox/developers/velox-backend-CI.md.
+ iceberg-spark-ut:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ permissions:
+ contents: read
+ actions: read
+ pull-requests: read
+ uses: ./.github/workflows/iceberg_spark_ut.yml
+ with:
+ native_artifact: velox-native-lib-centos-8-${{github.sha}}
+
+ # The Kafka native tests run in cpp-test-udf-test, whose build enables Kafka.
+ spark-test-kafka-spark35:
Review Comment:
The job name `spark-test-kafka-spark35` is misleading because it currently
only performs a Maven install with `-DskipTests` and explicitly has Kafka test
suites TODO/disabled. Consider renaming the job to reflect its current behavior
(e.g., `spark-build-kafka-spark35`), or re-enable at least one Kafka test step
so the name matches what it actually does.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1652,3 +1657,94 @@ jobs:
**/target/*.log
**/gluten-ut/**/hs_err_*.log
**/gluten-ut/**/core.*
+
+ # Runs upstream Iceberg tests on the static native libraries built above for
the head
+ # revision; see docs/backends/velox/developers/velox-backend-CI.md.
+ iceberg-spark-ut:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ permissions:
+ contents: read
+ actions: read
+ pull-requests: read
+ uses: ./.github/workflows/iceberg_spark_ut.yml
+ with:
+ native_artifact: velox-native-lib-centos-8-${{github.sha}}
+
+ # The Kafka native tests run in cpp-test-udf-test, whose build enables Kafka.
+ spark-test-kafka-spark35:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
Review Comment:
The identical multi-line `if` condition is duplicated across multiple jobs.
To reduce repetition and make future edits less error-prone, consider emitting
a single aggregated flag from `detect-changes` (e.g.,
`iceberg_or_kafka_needed=true`) and key job conditions off that one output.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1652,3 +1657,94 @@ jobs:
**/target/*.log
**/gluten-ut/**/hs_err_*.log
**/gluten-ut/**/core.*
+
+ # Runs upstream Iceberg tests on the static native libraries built above for
the head
+ # revision; see docs/backends/velox/developers/velox-backend-CI.md.
+ iceberg-spark-ut:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ permissions:
+ contents: read
+ actions: read
+ pull-requests: read
+ uses: ./.github/workflows/iceberg_spark_ut.yml
+ with:
+ native_artifact: velox-native-lib-centos-8-${{github.sha}}
+
+ # The Kafka native tests run in cpp-test-udf-test, whose build enables Kafka.
+ spark-test-kafka-spark35:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ runs-on: ubuntu-22.04
Review Comment:
The job name `spark-test-kafka-spark35` is misleading because it currently
only performs a Maven install with `-DskipTests` and explicitly has Kafka test
suites TODO/disabled. Consider renaming the job to reflect its current behavior
(e.g., `spark-build-kafka-spark35`), or re-enable at least one Kafka test step
so the name matches what it actually does.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1652,3 +1657,94 @@ jobs:
**/target/*.log
**/gluten-ut/**/hs_err_*.log
**/gluten-ut/**/core.*
+
+ # Runs upstream Iceberg tests on the static native libraries built above for
the head
+ # revision; see docs/backends/velox/developers/velox-backend-CI.md.
+ iceberg-spark-ut:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
Review Comment:
The identical multi-line `if` condition is duplicated across multiple jobs.
To reduce repetition and make future edits less error-prone, consider emitting
a single aggregated flag from `detect-changes` (e.g.,
`iceberg_or_kafka_needed=true`) and key job conditions off that one output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
