This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 7de270b [SPARK-57059] Add `integration-test-mac-spark41-iceberg`
GitHub Actions job
7de270b is described below
commit 7de270bc00d220d515456fec1a80cf1dc5d36599
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon May 25 17:46:51 2026 -0700
[SPARK-57059] Add `integration-test-mac-spark41-iceberg` GitHub Actions job
### What changes were proposed in this pull request?
This PR aims to add a new GitHub Actions job,
`integration-test-mac-spark41-iceberg`, which runs Apache Iceberg integration
tests against Apache Spark 4.1.2 on macOS.
The new job mirrors the existing `integration-test-mac-spark4-iceberg` job
with the following differences:
- Spark distribution: `spark-4.1.2-bin-hadoop3`
- Spark Connect package: `org.apache.spark:spark-connect_2.13:4.1.2`
- Iceberg runtime:
`org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0`
- Java version: 21 (Spark 4.1 supports JDK 21)
### Why are the changes needed?
Apache Spark 4.1.2 is the latest 4.1.x release, and we already have
`integration-test-mac-spark41` covering core integration tests against 4.1.2.
However, there is no CI coverage that verifies the Iceberg integration
(`DataFrameWriterV2Tests`, `IcebergTest`) against Spark 4.1. This PR closes
that gap so that regressions affecting the Spark 4.1 + Iceberg combination are
caught in CI.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs with the new `integration-test-mac-spark41-iceberg` job.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7
Closes #387 from dongjoon-hyun/SPARK-57059.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 83daabc..d8b4551 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -235,3 +235,31 @@ jobs:
cd -
swift test --filter DataFrameWriterV2Tests -c release
swift test --filter IcebergTest -c release
+
+ integration-test-mac-spark41-iceberg:
+ runs-on: macos-26
+ timeout-minutes: 20
+ env:
+ SPARK_LOCAL_IP: localhost
+ SPARK_ICEBERG_TEST_ENABLED: "true"
+ steps:
+ - uses: actions/checkout@v6
+ - name: Select Xcode 26.5
+ run: sudo xcode-select -s /Applications/Xcode_26.5.app/Contents/Developer
+ - name: Install Java
+ uses: actions/setup-java@v5
+ with:
+ distribution: zulu
+ java-version: 21
+ - name: Build
+ run: swift test --filter NOTHING -c release
+ - name: Test
+ run: |
+ curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.1.2/spark-4.1.2-bin-hadoop3.tgz?action=download
+ tar xvfz spark-4.1.2-bin-hadoop3.tgz && rm spark-4.1.2-bin-hadoop3.tgz
+ mv spark-4.1.2-bin-hadoop3 /tmp/spark
+ cd /tmp/spark/sbin
+ ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.1.2,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
+ cd -
+ swift test --filter DataFrameWriterV2Tests -c release
+ swift test --filter IcebergTest -c release
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
