This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 2abb238 [SPARK-57311] Check the startup logs to make Iceberg
integration tests robust
2abb238 is described below
commit 2abb238ddeac63396b293a591e29a60d03070d8a
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Jun 7 18:54:31 2026 -0700
[SPARK-57311] Check the startup logs to make Iceberg integration tests
robust
### What changes were proposed in this pull request?
This PR adds a readiness wait loop to the
`integration-test-mac-spark4-iceberg` and
`integration-test-mac-spark41-iceberg` jobs. After
`./start-connect-server.sh`, the job now
waits (up to 60 seconds) until `Spark Connect server started at` appears in
`/tmp/spark/logs/*.out` before running the tests.
### Why are the changes needed?
These two jobs pass `--packages org.apache.iceberg:...`, which resolves and
downloads the
Iceberg jars at server startup. This creates a large window where the Spark
Connect server is
not yet listening. Since the Swift client has no connection retry, `swift
test` can start before
the server is ready and fail immediately, making the jobs flaky. Waiting
for the ready log line
removes this race.
-
https://github.com/apache/spark-connect-swift/actions/runs/26650934447/job/78548793141
### Does this PR introduce _any_ user-facing change?
No. This is a CI-only change.
### How was this patch tested?
Pass the CI (`integration-test-mac-spark4-iceberg` and
`integration-test-mac-spark41-iceberg`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #413 from dongjoon-hyun/SPARK-57311.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 5f4506f..25fe0a0 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -230,6 +230,11 @@ jobs:
mv spark-4.0.2-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.0.2,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
+ # Iceberg jars are resolved at startup, so wait until the server is
ready.
+ for i in $(seq 1 12); do
+ grep -q "Spark Connect server started at" /tmp/spark/logs/*.out
2>/dev/null && break
+ sleep 5
+ done
cd -
swift test --filter DataFrameWriterV2Tests -c release
swift test --filter IcebergTest -c release
@@ -258,6 +263,11 @@ jobs:
mv spark-4.1.2-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh --packages
org.apache.spark:spark-connect_2.13:4.1.2,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0
-c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c
spark.sql.catalog.local.type=hadoop -c
spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c
spark.sql.defaultCatalog=local
+ # Iceberg jars are resolved at startup, so wait until the server is
ready.
+ for i in $(seq 1 12); do
+ grep -q "Spark Connect server started at" /tmp/spark/logs/*.out
2>/dev/null && break
+ sleep 5
+ done
cd -
swift test --filter DataFrameWriterV2Tests -c release
swift test --filter IcebergTest -c release
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
