Shekhar Prasad Rajak created SPARK-57584:
--------------------------------------------
Summary: Add standard gRPC health checking support to Spark
Connect server
Key: SPARK-57584
URL: https://issues.apache.org/jira/browse/SPARK-57584
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: kubernetes-operator-0.9.0, 4.1.2
Reporter: Shekhar Prasad Rajak
Fix For: 4.2.0, kubernetes-operator-1.0.0
Spark Connect currently starts a Netty gRPC server and registers
SparkConnectService, plus optional ProtoReflectionService in debug mode. It
does not appear to register the standard grpc.health.v1.Health service.
SparkConnectService.startGRPCService() registers Spark Connect service:
sql/connect/server/.../SparkConnectService.scala
Kubernetes support exposes the Spark Connect port with appProtocol: grpc, but
that does not imply standard gRPC health checking is available.
Motivation
Spark Connect is often deployed as a long-running service on Kubernetes.
Operators and SDKs need a reliable readiness signal before routing clients or
returning a SparkSession.
TCP readiness only proves the port is open. It does not prove that the Spark
Connect gRPC service is ready to accept requests. Standard gRPC health checking
would provide a portable
readiness contract for:
- Kubernetes grpc readiness/startup probes
- Spark Kubernetes Operator implementations
- Kubeflow Spark Operator SparkConnect readiness
- SDKs and gateways that wait for Spark Connect before connecting clients
- service meshes and load balancers that understand gRPC health
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
