mbutrovich commented on code in PR #2528:
URL: https://github.com/apache/datafusion-comet/pull/2528#discussion_r2547444993
##########
.github/workflows/iceberg_spark_test.yml:
##########
@@ -156,3 +156,120 @@ jobs:
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew
-DsparkVersions=${{ matrix.spark-version.short }} -DscalaVersion=${{
matrix.scala-version }} -DflinkVersions= -DkafkaVersions= \
:iceberg-spark:iceberg-spark-runtime-${{
matrix.spark-version.short }}_${{ matrix.scala-version }}:integrationTest \
-Pquick=true -x javadoc
+
+ iceberg-spark-rust:
+ if: contains(github.event.pull_request.title, '[iceberg]')
+ strategy:
+ matrix:
+ os: [ubuntu-24.04]
+ java-version: [11, 17]
+ iceberg-version: [{short: '1.8', full: '1.8.1'}, {short: '1.9', full:
'1.9.1'}, {short: '1.10', full: '1.10.0'}]
+ spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full:
'3.5.7'}]
Review Comment:
Here's what we currently test with this PR:
| | 3.4
| 3.5
| 4.0
|
|-------|-------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
| 1.5.2 | CometIcebergNativeSuite CometFuzzIcebergSuite
IcebergReadFromS3Suite (not run in CI due to MinIO container) |
|
|
| 1.8.1 | Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark
Runtime Tests | Iceberg Spark Tests Iceberg Spark
Extensions Tests Iceberg Spark Runtime Tests CometIcebergNativeSuite
CometFuzzIcebergSuite IcebergReadFromS3Suite (not run in CI due to MinIO
container) |
|
| 1.9.1 | Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark
Runtime Tests | Iceberg Spark Tests Iceberg Spark
Extensions Tests Iceberg Spark Runtime Tests
|
|
| 1.10 | Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark
Runtime Tests | Iceberg Spark Tests Iceberg Spark
Extensions Tests Iceberg Spark Runtime Tests
|
CometIcebergNativeSuite CometFuzzIcebergSuite IcebergReadFromS3Suite (not run
in CI due to MinIO container) |
I leaned on newer versions for the Iceberg tests because as best as I could
tell, never versions are a superset of the older versions. For the Comet-native
tests we are running 1.5.2.
We should have a discussion of what we want to run long term, because right
now tagging a PR `[iceberg]` makes CI take hours and causes so many parallel
Iceberg suites that we start getting network timeouts (likely due to
throttling).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
