This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 29e41fd [SPARK-58196] Use `apache/spark:4.2.0-scala-java25` image in
Spark 4.2 integration test
29e41fd is described below
commit 29e41fd5b6bfc0484e00d526b29964a5eb731adb
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 17 09:26:15 2026 -0700
[SPARK-58196] Use `apache/spark:4.2.0-scala-java25` image in Spark 4.2
integration test
### What changes were proposed in this pull request?
This PR aims to use the `apache/spark:4.2.0-scala-java25` GA image instead
of `apache/spark:4.2.0-preview5-scala` for Spark 4.2 integration tests. In
addition, this PR removes the remaining `preview` image reference from
`AGENTS.md` and the `preview`-version runtime guard from `DataFrameTests`.
### Why are the changes needed?
Apache Spark 4.2.0 GA Docker images are published on Docker Hub. We should
test against the GA image (with Java 25) instead of the outdated `preview5`
image.
- https://hub.docker.com/r/apache/spark/tags?name=4.2.0-scala-java25
Since we no longer test against `preview` versions, the `preview` runtime
guard in `nearestByJoin` test is not needed anymore.
### Does this PR introduce _any_ user-facing change?
No. This is a test infrastructure and documentation change.
### How was this patch tested?
Pass the CIs, especially `integration-test-ubuntu-spark42` job.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #438 from dongjoon-hyun/SPARK-58196.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 2 +-
AGENTS.md | 2 +-
Tests/SparkConnectTests/DataFrameTests.swift | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index b253fd4..91fce9f 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -123,7 +123,7 @@ jobs:
run: |
docker run -d --name spark -p 15003:15002 -e SPARK_NO_DAEMONIZE=1 \
--entrypoint /opt/spark/sbin/start-connect-server.sh \
- apache/spark:4.2.0-preview5-scala \
+ apache/spark:4.2.0-scala-java25 \
-c spark.sql.geospatial.enabled=true \
-c spark.sql.timeType.enabled=true
- name: Test
diff --git a/AGENTS.md b/AGENTS.md
index 3915582..939262c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -78,7 +78,7 @@ Start a local Spark Connect server first (default endpoint
`sc://localhost:15002
```bash
docker run -it --rm -p 15002:15002 -e SPARK_NO_DAEMONIZE=1 \
- apache/spark:4.2.0-preview5 bash -c /opt/spark/sbin/start-connect-server.sh
+ apache/spark:4.2.0 bash -c /opt/spark/sbin/start-connect-server.sh
```
Environment variables that gate behavior:
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index b54e459..e495b13 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -634,7 +634,7 @@ struct DataFrameTests {
func nearestByJoin() async throws {
let spark = try await SparkSession.builder.getOrCreate()
let version = await spark.version
- if version >= "4.2", !version.contains("preview") {
+ if version >= "4.2" {
let users = try await spark.sql(
"SELECT * FROM VALUES (1, 10.0), (2, 20.0), (3, 30.0) AS T(user_id,
score)")
let products = try await spark.sql(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
