This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 7ac060e [SPARK-60619] Use Apache Spark 4.0.5, 4.1.4, and 4.2.1
instead of RCs
7ac060e is described below
commit 7ac060eb652c47aff9e14e2567c73e01f5ba72bb
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 28 10:27:22 2026 -0700
[SPARK-60619] Use Apache Spark 4.0.5, 4.1.4, and 4.2.1 instead of RCs
### What changes were proposed in this pull request?
This PR updates the `integration-test-mac`, `integration-test-mac-spark41`,
and `integration-test-mac-spark42` CI jobs to use the official Apache Spark
releases instead of RC1, with the updated SHA-512 checksums.
| Job | Version | Before | After |
|---|---|---|---|
| `integration-test-mac` | 4.0.5 | RC1 (`dist.apache.org/repos/dist/dev`) |
official release |
| `integration-test-mac-spark41` | 4.1.4 | RC1
(`dist.apache.org/repos/dist/dev`) | official release |
| `integration-test-mac-spark42` | 4.2.1 | RC1
(`dist.apache.org/repos/dist/dev`) | official release |
- The RC1 `url:` input is removed so that the `setup-spark` action falls
back to its default ASF mirror redirector (`closer.lua`).
- The `sha512:` input is updated to the checksum of the official tarball
(verified against both `downloads.apache.org` and `dlcdn.apache.org`).
- `AGENTS.md` is updated to list `4.0.5 / 4.1.4 / 4.2.1` as the tested
Spark versions.
The 4.3.0 job still uses RC1 because it is not released yet.
### Why are the changes needed?
To test against the official Apache Spark releases.
- https://dlcdn.apache.org/spark/spark-4.0.5/
- https://dlcdn.apache.org/spark/spark-4.1.4/
- https://dlcdn.apache.org/spark/spark-4.2.1/
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #623 from dongjoon-hyun/SPARK-60619.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 25 +++++++++++--------------
AGENTS.md | 2 +-
2 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index a95000f..30f6de6 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -131,8 +131,7 @@ jobs:
- uses: ./.github/actions/setup-spark
with:
version: 4.1.4
- url:
https://dist.apache.org/repos/dist/dev/spark/v4.1.4-rc1-bin/spark-4.1.4-bin-hadoop3.tgz
- sha512:
4a97fcddb71621607033431988af2bbb37b1c5d20c7baca6927029a1ac5713ffc6d49c05976123fe2ae57bc18acb1aab357d1633461bc30046ae7a105032e624
+ sha512:
1461b6c425d290913e7be97bfec738abc843c663d177dc57e32b82874d20144ed67dcca692a57efc4e5b40d2f001ee0e9e0d2d7b088ae58f5378ecac92739076
- name: Test
run: |
cd /tmp/spark/sbin
@@ -152,8 +151,7 @@ jobs:
- uses: ./.github/actions/setup-spark
with:
version: 4.2.1
- url:
https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/spark-4.2.1-bin-hadoop3.tgz
- sha512:
4f2e5368987ab6d5bba33859c67c32f910829c1ce536c2d81e9e3d0c37dcb9d0ae659dbc5872c9e568d03094c9ed3f551c585a08b0e44e0cefde9b29cfc15371
+ sha512:
2f76b113ebb2f3677766e38b6dc78f27756c5ae5a7c103f1a76dac523dba8e5f82077b7f480360391c0cceee79ad712d7843f23389fbfeb18bc018fb85e27866
- name: Test
run: |
cd /tmp/spark/sbin
@@ -194,8 +192,7 @@ jobs:
- uses: ./.github/actions/setup-spark
with:
version: 4.0.5
- url:
https://dist.apache.org/repos/dist/dev/spark/v4.0.5-rc1-bin/spark-4.0.5-bin-hadoop3.tgz
- sha512:
6a276aa40e5ff150295428a7d762669b2b106ac1a9de3290bd4b3ff73fc9669849f35089553b5ddd55a0e0dbc1dd19f9a6bc4ef31970e1e3133837c694132107
+ sha512:
c36f0e8445513d3d746d20aa051787f417477a2550b002bb4e4c3797e6152a6a04cce7121344cd11511e18642c852ca5893e872564a207ee9d8d3befe6309b20
- name: Test
run: |
cd /tmp/spark/sbin
@@ -215,8 +212,8 @@ jobs:
run: swift test --filter NOTHING -c release
- uses: ./.github/actions/setup-spark
with:
- version: 4.1.3
- sha512:
391d04f40da0f6e961513d1ad72a6e3d51aef99f751c835e058a329b01369d83d17c2a9c68fad6be9496eea6687107d0d3d6893f7a53f53ddbfe669022220523
+ version: 4.1.4
+ sha512:
1461b6c425d290913e7be97bfec738abc843c663d177dc57e32b82874d20144ed67dcca692a57efc4e5b40d2f001ee0e9e0d2d7b088ae58f5378ecac92739076
- name: Test
run: |
cd /tmp/spark/sbin
@@ -241,12 +238,12 @@ jobs:
run: swift test --filter NOTHING -c release
- uses: ./.github/actions/setup-spark
with:
- version: 4.0.4
- sha512:
1b91bf197df884368293cb91e7eda866751570b0c98f38fda712030dd8256962460d68d9f2c9075bc83061428b18108faa89510d3a20e1436a41fc73cadc24dd
+ version: 4.0.5
+ sha512:
c36f0e8445513d3d746d20aa051787f417477a2550b002bb4e4c3797e6152a6a04cce7121344cd11511e18642c852ca5893e872564a207ee9d8d3befe6309b20
- name: Test
run: |
cd /tmp/spark/sbin
- ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.0.4,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
+ ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.0.5,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
# Iceberg jars are resolved at startup, so wait until the server is
ready.
for i in $(seq 1 12); do
grep -q "Spark Connect server started at" /tmp/spark/logs/*.out
2>/dev/null && break
@@ -273,12 +270,12 @@ jobs:
run: swift test --filter NOTHING -c release
- uses: ./.github/actions/setup-spark
with:
- version: 4.1.3
- sha512:
391d04f40da0f6e961513d1ad72a6e3d51aef99f751c835e058a329b01369d83d17c2a9c68fad6be9496eea6687107d0d3d6893f7a53f53ddbfe669022220523
+ version: 4.1.4
+ sha512:
1461b6c425d290913e7be97bfec738abc843c663d177dc57e32b82874d20144ed67dcca692a57efc4e5b40d2f001ee0e9e0d2d7b088ae58f5378ecac92739076
- name: Test
run: |
cd /tmp/spark/sbin
- ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.1.3,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
+ ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.1.4,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
# Iceberg jars are resolved at startup, so wait until the server is
ready.
for i in $(seq 1 12); do
grep -q "Spark Connect server started at" /tmp/spark/logs/*.out
2>/dev/null && break
diff --git a/AGENTS.md b/AGENTS.md
index 49c29ae..1eefbf9 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -21,7 +21,7 @@ catalog, ML) over gRPC, exchanging results in Apache Arrow
format.
- **Swift 6.4.0+** with Swift Package Manager (`swift-tools-version: 6.4.0`).
- **Platforms**: macOS 15+, iOS 18+, watchOS 11+, tvOS 18+, and Linux
(built and tested on Ubuntu x86_64 & arm64 with the Swift 6.4.0 toolchain).
-- **Server**: Apache Spark 4.x Connect server (tested against 4.0.4 / 4.1.3 /
4.2.0).
+- **Server**: Apache Spark 4.x Connect server (tested against 4.0.5 / 4.1.4 /
4.2.1).
- **Dependencies** (all pinned with `exact` in [Package.swift](Package.swift)):
`grpc-swift-2`, `grpc-swift-protobuf`, `grpc-swift-nio-transport`,
`flatbuffers`, `swift-system`. Keep version bumps as `exact` pins, never
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
