This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 9afbb7c24872 [SPARK-58369][CONNECT][TEST] Introduce a way to 
`SparkConnectServiceKeepAliveSuite` to wait until unbinding a port is complete
9afbb7c24872 is described below

commit 9afbb7c248721c88134891ed3a7128fd1ea3ec3b
Author: Kousuke Saruta <[email protected]>
AuthorDate: Wed Jul 29 09:02:58 2026 -0700

    [SPARK-58369][CONNECT][TEST] Introduce a way to 
`SparkConnectServiceKeepAliveSuite` to wait until unbinding a port is complete
    
    ### What changes were proposed in this pull request?
    This PR introduce `stopSparkConnectServiceAndUnbind()` to 
`SparkConnectServiceKeepAliveSuite` as a way to wait until unbinding a port is 
complete.
    
    ### Why are the changes needed?
    SparkConnectServiceKeepAliveSuite sometimes flakily fails. #57342 tried to 
fix it but it but it still happens.
    https://github.com/apache/spark/actions/runs/30041141074/job/89325300649
    ```
    [info] - SPARK-58094: disabling spark.connect.grpc.keepAlive.enabled 
reverts to the pre-fix hang *** FAILED *** (6 milliseconds)
    [info]   java.net.BindException: Failed to bind to address 
0.0.0.0/0.0.0.0:15788: Service 
'org.apache.spark.sql.connect.service.SparkConnectService' failed after 0 
retries (starting from 15788)! Consider explicitly setting the appropriate port 
for the service 'org.apache.spark.sql.connect.service.SparkConnectService' (for 
example spark.ui.port for SparkUI) to an available port or increasing 
spark.port.maxRetries.
    [info]   at io.grpc.netty.NettyServer.start(NettyServer.java:341)
    [info]   at io.grpc.internal.ServerImpl.start(ServerImpl.java:185)
    [info]   at io.grpc.internal.ServerImpl.start(ServerImpl.java:94)
    [info]   at 
org.apache.spark.sql.connect.service.SparkConnectService$.$anonfun$startGRPCService$1(SparkConnectService.scala:451)
    [info]   at 
org.apache.spark.sql.connect.service.SparkConnectService$.$anonfun$startGRPCService$1$adapted(SparkConnectService.scala:411)
    [info]   at 
org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2276)
    [info]   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:256)
    [info]   at 
org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2268)
    [info]   at 
org.apache.spark.sql.connect.service.SparkConnectService$.startGRPCService(SparkConnectService.scala:471)
    [info]   at 
org.apache.spark.sql.connect.service.SparkConnectService$.start(SparkConnectService.scala:483)
    [info]   at 
org.apache.spark.sql.connect.service.SparkConnectServiceKeepAliveSuite.$anonfun$new$17(SparkConnectServiceKeepAliveSuite.scala:262)
    [info]   at 
org.apache.spark.SparkTestSuite.withSparkEnvConfs(SparkTestSuite.scala:268)
    [info]   at 
org.apache.spark.SparkTestSuite.withSparkEnvConfs$(SparkTestSuite.scala:257)
    [info]   at 
org.apache.spark.SparkFunSuite.withSparkEnvConfs(SparkFunSuite.scala:33)
    [info]   at 
org.apache.spark.sql.connect.service.SparkConnectServiceKeepAliveSuite.$anonfun$new$16(SparkConnectServiceKeepAliveSuite.scala:262)
    ...
    ```
    
    The root cause of the issue is that `SparkConnectService.stop()` may return 
before unbinding the port is complete in Linux environment even though 
`io.grpc.Server#awaitTermination()` is called within 
`SparkConnectService.stop()`.
    We can see this behavior with the following code. With this code, 
`SparkConnectService.start()` may throw `BindException`.
    
    ```
    for (_ <- 1 to 3000) {
      SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
      withSparkEnvConfs(
        Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
        Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
        Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -> "1s",
        Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -> "1s") {
        SparkConnectService.start(spark.sparkContext)
      }
    }
    ```
    
    We use Netty in `SparkConnectService` so the socket is closed in 
`AbstractChannel#doClose()`, more specifically in 
[AbstractEpollChannel#doClose()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport-classes-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollChannel.java#L214)
 in Linux.
    `doClose()` is also called from 
[AbstractChannel#doClose0()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport/src/main/java/io/netty/channel/AbstractChannel.java#L617)
    In Linux environment, `doClose0()` is called asynchronously 
[here](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport/src/main/java/io/netty/channel/AbstractChannel.java#L574)
 because `EpollSocketChannelUnsafe` overrides 
[prepareToClose()](https://github.com/netty/netty/blob/3703d79669ee024f6483c9d8697ac58ba546df33/transport-classes-epoll/src/main/java/io/netty/channel/epoll/EpollSocketChannel.java#L159)
 and [closeExecutor](https://github.com/netty/ [...]
    On the other hand, I don't see `prepareToClose()` overridden in any class 
under `io.netty.channel.kqueue`. So the flakiness should not affect in Mac 
environment.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Confirmed that the following code doesn't throw `BindException`.
    ```
    for (_ <- 1 to 3000) {
      stopSparkConnectServiceAndUnbind()
      withSparkEnvConfs(
        Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
        Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
        Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -> "1s",
        Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -> "1s") {
        SparkConnectService.start(spark.sparkContext)
      }
    }
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No.
    
    Closes #57560 from sarutak/fix-flaky-SparkConnectServiceKeepAliveSuite.
    
    Authored-by: Kousuke Saruta <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit fd4c5869a0d5633eeb4853debe929282f268d573)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../SparkConnectServiceKeepAliveSuite.scala        | 23 +++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
index de8d6b84e30d..a0b249a002ab 100644
--- 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
+++ 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectServiceKeepAliveSuite.scala
@@ -16,6 +16,7 @@
  */
 package org.apache.spark.sql.connect.service
 
+import java.net.ServerSocket
 import java.util.UUID
 import java.util.concurrent.TimeUnit
 
@@ -28,6 +29,7 @@ import org.apache.spark.SparkException
 import org.apache.spark.sql.connect.SparkConnectServerTest
 import org.apache.spark.sql.connect.client.{RetryPolicy, SparkConnectClient}
 import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.util.Utils.tryWithResource
 
 /**
  * End-to-end test that the *real* `SparkConnectService.startGRPCService()` 
wiring actually
@@ -53,6 +55,13 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
     Connect.CONNECT_GRPC_KEEPALIVE_TIME.key -> "1s",
     Connect.CONNECT_GRPC_KEEPALIVE_TIMEOUT.key -> "1s")
 
+  private def stopSparkConnectServiceAndUnbind(): Unit = {
+    SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+    Eventually.eventually(timeout(eventuallyTimeout)) {
+      tryWithResource(new ServerSocket(serverPort))(identity)
+    }
+  }
+
   test("SPARK-58094: real SparkConnectService applies configured keepalive 
end-to-end") {
     val serverSession =
       SparkConnectService
@@ -121,7 +130,7 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
     // behavior), this client's cadence would violate that coupled invariant; 
with today's fixed,
     // decoupled GRPC_KEEPALIVE_PERMIT_TIME_SECONDS floor (10s), it's 
comfortably tolerated.
     val clientKeepAliveMs = 
(SparkConnectService.GRPC_KEEPALIVE_PERMIT_TIME_SECONDS + 1) * 1000
-    SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+    stopSparkConnectServiceAndUnbind()
     withSparkEnvConfs(
       Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
       Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -156,7 +165,7 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
         client.shutdown()
       }
     } finally {
-      SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+      stopSparkConnectServiceAndUnbind()
       withSparkEnvConfs(
         Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
         Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -190,7 +199,7 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
     // truly idle connection -- hence one call to establish the transport, 
then real idle time
     // (no further calls) spanning several ping intervals, then a 
connection-count check.
     val clientKeepAliveMs = 
(SparkConnectService.GRPC_KEEPALIVE_PERMIT_TIME_SECONDS + 1) * 1000
-    SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+    stopSparkConnectServiceAndUnbind()
     withSparkEnvConfs(
       Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
       Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
@@ -238,7 +247,7 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
       }
     } finally {
       relay.close()
-      SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+      stopSparkConnectServiceAndUnbind()
       withSparkEnvConfs(
         Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
         Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -253,7 +262,7 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
     "SPARK-58094: disabling spark.connect.grpc.keepAlive.enabled reverts to 
the pre-fix hang") {
     // Restart the real service with keepalive fully disabled (not just given 
short/aggressive
     // timing) to prove the flag genuinely gates the fix rather than only 
tuning its timing.
-    SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+    stopSparkConnectServiceAndUnbind()
     withSparkEnvConfs(
       Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
       Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "false",
@@ -321,7 +330,7 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
       }
     } finally {
       // Restore the enabled server for afterAll()/subsequent tests in this 
suite.
-      SparkConnectService.stop(Some(30), Some(TimeUnit.SECONDS))
+      stopSparkConnectServiceAndUnbind()
       withSparkEnvConfs(
         Connect.CONNECT_GRPC_BINDING_PORT.key -> serverPort.toString,
         Connect.CONNECT_GRPC_KEEPALIVE_ENABLED.key -> "true",
@@ -341,7 +350,7 @@ class SparkConnectServiceKeepAliveSuite extends 
SparkConnectServerTest {
  * shared across modules since it's a small, self-contained test utility.
  */
 private class FreezableTcpRelay(targetPort: Int) {
-  private val listener = new java.net.ServerSocket(0)
+  private val listener = new ServerSocket(0)
   @volatile private var frozen = false
   private val sockets = mutable.ListBuffer[java.net.Socket]()
   private val acceptedConnections = new 
java.util.concurrent.atomic.AtomicInteger(0)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to