This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 9afbb7c24872 [SPARK-58369][CONNECT][TEST] Introduce a way to
`SparkConnectServiceKeepAliveSuite` to wait until unbinding a port is complete
9afbb7c24872 is described below
commit 9afbb7c248721c88134891ed3a7128fd1ea3ec3b
Author: Kousuke Saruta <[email protected]>
AuthorDate: Wed Jul 29 09:02:58 2026 -0700
[SPARK-58369][CONNECT][TEST] Introduce a way to
`SparkConnectServiceKeepAliveSuite` to wait until unbinding a port is complete
### What changes were proposed in this pull request?
This PR introduce `stopSparkConnectServiceAndUnbind()` to
`SparkConnectServiceKeepAliveSuite` as a way to wait until unbinding a port is
complete.
### Why are the changes needed?
SparkConnectServiceKeepAliveSuite sometimes flakily fails. #57342 tried to
fix it but it but it still happens.
https://github.com/apache/spark/actions/runs/30041141074/job/89325300649
```
[info] - SPARK-58094: disabling spark.connect.grpc.keepAlive.enabled
reverts to the pre-fix hang *** FAILED *** (6 milliseconds)
[info] java.net.BindException: Failed to bind to address
0.0.0.0/0.0.0.0:15788: Service
'org.apache.spark.sql.connect.service.SparkConnectService' failed after 0
retries (starting from 15788)! Consider explicitly setting the appropriate port
for the service 'org.apache.spark.sql.connect.service.SparkConnectService' (for
example spark.ui.port for SparkUI) to an available port or increasing
spark.port.maxRetries.
[info] at io.grpc.netty.NettyServer.start(NettyServer.java:341)
[info] at io.grpc.internal.ServerImpl.start(ServerImpl.java:185)
[info] at io.grpc.internal.ServerImpl.start(ServerImpl.java:94)
[info] at
org.apache.spark.sql.connect.service.SparkConnectService$.$anonfun$startGRPCService$1(SparkConnectService.scala:451)
[info] at
org.apache.spark.sql.connect.service.SparkConnectService$.$anonfun$startGRPCService$1$adapted(SparkConnectService.scala:411)
[info] at
org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2276)
[info] at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:256)
[info] at
org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2268)
[info] at
org.apache.spark.sql.connect.service.SparkConnectService$.startGRPCService(SparkConnectService.scala:471)
[info] at
org.apache.spark.sql.connect.service.SparkConnectService$.start(SparkConnectService.scala:483)
[info] at
org.apache.spark.sql.connect.service.SparkConnectServiceKeepAliveSuite.$anonfun$new$17(SparkConnectServiceKeepAliveSuite.scala:262)
[info] at
org.apache.spark.SparkTestSuite.withSparkEnvConfs(SparkTestSuite.scala:268)
[info] at
org.apache.spark.SparkTestSuite.withSparkEnvConfs$(SparkTestSuite.scala:257)
[info] at
org.apache.spark.SparkFunSuite.withSparkEnvConfs(SparkFunSuite.scala:33)
[info] at
org.apache.spark.sql.connect.service.SparkConnectServiceKeepAliveSuite.$anonfun$new$16(SparkConnectServiceKeepAliveSuite.scala:262)
...
```
The root cause of the issue is that `SparkConnectService.stop()` may return
before unbinding the port is complete in Linux environment even though
`io.grpc.Server#awaitTermination()` is called within
`SparkConnectService.stop()`.
We can see this behavior with the following code. With this code,
`SparkConnectService.start()` may throw `BindException`.
```
for (_ <- 1 to 3000) {
SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -> "1s",
Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -> "1s") {
SparkConnectService.start(spark.sparkContext)
}
}
```
We use Netty in `SparkConnectService` so the socket is closed in
`AbstractChannel#doClose()`, more specifically in
[AbstractEpollChannel#doClose()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport-classes-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollChannel.java#L214)
in Linux.
`doClose()` is also called from
[AbstractChannel#doClose0()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport/src/main/java/io/netty/channel/AbstractChannel.java#L617)
In Linux environment, `doClose0()` is called asynchronously
[here](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport/src/main/java/io/netty/channel/AbstractChannel.java#L574)
because `EpollSocketChannelUnsafe` overrides
[prepareToClose()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport-classes-epoll/src/main/java/io/netty/channel/epoll/EpollSocketChannel.java#L159)
and [closeExecutor](https://github.com/netty/ [...]
On the other hand, I don't see `prepareToClose()` overridden in any class
under `io.netty.channel.kqueue`. So the flakiness should not affect in Mac
environment.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Confirmed that the following code doesn't throw `BindException`.
```
for (_ <- 1 to 3000) {
stopSparkConnectServiceAndUnbind()
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -> "1s",
Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -> "1s") {
SparkConnectService.start(spark.sparkContext)
}
}
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57560 from sarutak/fix-flaky-SparkConnectServiceKeepAliveSuite.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit fd4c5869a0d5633eeb4853debe929282f268d573)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../SparkConnectServiceKeepAliveSuite.scala | 23 +++++++++++++++-------
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
index de8d6b84e30d..a0b249a002ab 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
@@ -16,6 +16,7 @@
*/
package org.apache.spark.sql.connect.service
+import java.net.ServerSocket
import java.util.UUID
import java.util.concurrent.TimeUnit
@@ -28,6 +29,7 @@ import org.apache.spark.SparkException
import org.apache.spark.sql.connect.SparkConnectServerTest
import org.apache.spark.sql.connect.client.{RetryPolicy, SparkConnectClient}
import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.util.Utils.tryWithResource
/**
* End-to-end test that the *real* `SparkConnectService.startGRPCService()`
wiring actually
@@ -53,6 +55,13 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -> "1s",
Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -> "1s")
+ private def stopSparkConnectServiceAndUnbind(): Unit = {
+ SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+ Eventually.eventually(timeout(eventuallyTimeout)) {
+ tryWithResource(new ServerSocket(serverPort))(identity)
+ }
+ }
+
test("SPARK-58094: real SparkConnectService applies configured keepalive
end-to-end") {
val serverSession =
SparkConnectService
@@ -121,7 +130,7 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
// behavior), this client's cadence would violate that coupled invariant;
with today's fixed,
// decoupled GRPC_KEEPALIVE_PERMIT_TIME_SECONDS floor (10s), it's
comfortably tolerated.
val clientKeepAliveMs =
(SparkConnectService.GRPC_KEEPALIVE_PERMIT_TIME_SECONDS + 1) * 1000
- SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+ stopSparkConnectServiceAndUnbind()
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -156,7 +165,7 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
client.shutdown()
}
} finally {
- SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+ stopSparkConnectServiceAndUnbind()
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -190,7 +199,7 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
// truly idle connection -- hence one call to establish the transport,
then real idle time
// (no further calls) spanning several ping intervals, then a
connection-count check.
val clientKeepAliveMs =
(SparkConnectService.GRPC_KEEPALIVE_PERMIT_TIME_SECONDS + 1) * 1000
- SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+ stopSparkConnectServiceAndUnbind()
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
@@ -238,7 +247,7 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
}
} finally {
relay.close()
- SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+ stopSparkConnectServiceAndUnbind()
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -253,7 +262,7 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
"SPARK-58094: disabling spark.connect.grpc.keepAlive.enabled reverts to
the pre-fix hang") {
// Restart the real service with keepalive fully disabled (not just given
short/aggressive
// timing) to prove the flag genuinely gates the fix rather than only
tuning its timing.
- SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+ stopSparkConnectServiceAndUnbind()
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
@@ -321,7 +330,7 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
}
} finally {
// Restore the enabled server for afterAll()/subsequent tests in this
suite.
- SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+ stopSparkConnectServiceAndUnbind()
withSparkEnvConfs(
Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -341,7 +350,7 @@ class SparkConnectServiceKeepAliveSuite extends
SparkConnectServerTest {
* shared across modules since it's a small, self-contained test utility.
*/
private class FreezableTcpRelay(targetPort: Int) {
- private val listener = new java.net.ServerSocket(0)
+ private val listener = new ServerSocket(0)
@volatile private var frozen = false
private val sockets = mutable.ListBuffer[java.net.Socket]()
private val acceptedConnections = new
java.util.concurrent.atomic.AtomicInteger(0)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]