This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new d4b38b4 [SPARK-58560] Use Apache Spark `4.0.4` and `4.1.3` in CI
d4b38b4 is described below
commit d4b38b463f13bbbe6da9a0fd71e2260e0da7e506
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Aug 4 07:56:35 2026 -0700
[SPARK-58560] Use Apache Spark `4.0.4` and `4.1.3` in CI
### What changes were proposed in this pull request?
This PR aims to use Apache Spark `4.0.4` and `4.1.3` in CI instead of
`4.0.3` and `4.1.2`.
In addition, this PR updates the tested server versions in `AGENTS.md`
accordingly.
### Why are the changes needed?
To test with the latest Apache Spark maintenance releases:
- [v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11)
- [v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12)
### Does this PR introduce _any_ user-facing change?
No, this is a test infra update.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #478 from dongjoon-hyun/SPARK-58560.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 38 ++++++++++++++++++------------------
AGENTS.md | 2 +-
2 files changed, 20 insertions(+), 20 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 91fce9f..7a7d129 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -83,7 +83,7 @@ jobs:
SPARK_REMOTE: "sc://localhost:15003"
services:
spark:
- image: apache/spark:4.0.3-scala
+ image: apache/spark:4.0.4-scala
env:
SPARK_NO_DAEMONIZE: 1
ports:
@@ -102,7 +102,7 @@ jobs:
SPARK_REMOTE: "sc://localhost:15003"
services:
spark:
- image: apache/spark:4.1.2-scala
+ image: apache/spark:4.1.3-scala
env:
SPARK_NO_DAEMONIZE: 1
ports:
@@ -144,9 +144,9 @@ jobs:
run: swift test --filter NOTHING -c release
- name: Test
run: |
- curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.1.2/spark-4.1.2-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.1.2-bin-hadoop3.tgz && rm spark-4.1.2-bin-hadoop3.tgz
- mv spark-4.1.2-bin-hadoop3 /tmp/spark
+ curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz?action=download
+ tar xvfz spark-4.1.3-bin-hadoop3.tgz && rm spark-4.1.3-bin-hadoop3.tgz
+ mv spark-4.1.3-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh
cd -
@@ -187,9 +187,9 @@ jobs:
run: swift test --filter NOTHING -c release
- name: Test
run: |
- curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.0.3/spark-4.0.3-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.0.3-bin-hadoop3.tgz && rm spark-4.0.3-bin-hadoop3.tgz
- mv spark-4.0.3-bin-hadoop3 /tmp/spark
+ curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz?action=download
+ tar xvfz spark-4.0.4-bin-hadoop3.tgz && rm spark-4.0.4-bin-hadoop3.tgz
+ mv spark-4.0.4-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh
cd -
@@ -209,9 +209,9 @@ jobs:
run: swift test --filter NOTHING -c release
- name: Test
run: |
- curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.1.2/spark-4.1.2-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.1.2-bin-hadoop3.tgz && rm spark-4.1.2-bin-hadoop3.tgz
- mv spark-4.1.2-bin-hadoop3 /tmp/spark
+ curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz?action=download
+ tar xvfz spark-4.1.3-bin-hadoop3.tgz && rm spark-4.1.3-bin-hadoop3.tgz
+ mv spark-4.1.3-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh
cd -
@@ -236,11 +236,11 @@ jobs:
run: swift test --filter NOTHING -c release
- name: Test
run: |
- curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.0.3/spark-4.0.3-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.0.3-bin-hadoop3.tgz && rm spark-4.0.3-bin-hadoop3.tgz
- mv spark-4.0.3-bin-hadoop3 /tmp/spark
+ curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz?action=download
+ tar xvfz spark-4.0.4-bin-hadoop3.tgz && rm spark-4.0.4-bin-hadoop3.tgz
+ mv spark-4.0.4-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
- ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.0.3,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
+ ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.0.4,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
# Iceberg jars are resolved at startup, so wait until the server is
ready.
for i in $(seq 1 12); do
grep -q "Spark Connect server started at" /tmp/spark/logs/*.out
2>/dev/null && break
@@ -269,11 +269,11 @@ jobs:
run: swift test --filter NOTHING -c release
- name: Test
run: |
- curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.1.2/spark-4.1.2-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.1.2-bin-hadoop3.tgz && rm spark-4.1.2-bin-hadoop3.tgz
- mv spark-4.1.2-bin-hadoop3 /tmp/spark
+ curl -LO
https://www.apache.org/dyn/closer.lua/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz?action=download
+ tar xvfz spark-4.1.3-bin-hadoop3.tgz && rm spark-4.1.3-bin-hadoop3.tgz
+ mv spark-4.1.3-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
- ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.1.2,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
+ ./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.1.3,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
# Iceberg jars are resolved at startup, so wait until the server is
ready.
for i in $(seq 1 12); do
grep -q "Spark Connect server started at" /tmp/spark/logs/*.out
2>/dev/null && break
diff --git a/AGENTS.md b/AGENTS.md
index 939262c..f7a77cd 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -21,7 +21,7 @@ catalog, ML) over gRPC, exchanging results in Apache Arrow
format.
- **Swift 6.3.2+** with Swift Package Manager (`swift-tools-version: 6.3.2`).
- **Platforms**: macOS 15+, iOS 18+, watchOS 11+, tvOS 18+, and Linux
(built and tested on Ubuntu x86_64 & arm64 with the Swift 6.3.2 toolchain).
-- **Server**: Apache Spark 4.x Connect server (tested against 4.0.3 / 4.1.2 /
4.2.0).
+- **Server**: Apache Spark 4.x Connect server (tested against 4.0.4 / 4.1.3 /
4.2.0).
- **Dependencies** (all pinned with `exact` in [Package.swift](Package.swift)):
`grpc-swift-2`, `grpc-swift-protobuf`, `grpc-swift-nio-transport`,
`flatbuffers`, `swift-system`. Keep version bumps as `exact` pins, never
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
