dongjoon-hyun opened a new pull request, #509:
URL: https://github.com/apache/spark-connect-swift/pull/509
### What changes were proposed in this pull request?
This PR makes `SparkConnectClient` create its gRPC channel once and reuse it
for
every RPC of a session, instead of creating and tearing down a `GRPCClient`
per RPC.
- A new `getGRPCClient()` lazily creates a `GRPCClient` and runs its
connections in
a detached task. `stop()` shuts it down with `beginGracefulShutdown()`.
- `DataFrame.withGPRC` no longer builds its own transport; it uses the
shared client.
This removes a duplicated transport configuration that had to be kept in
sync in
two places.
Keepalive, compression, proxy support, and custom TLS are left as follow-ups.
### Why are the changes needed?
Every RPC paid for a full connection setup: DNS resolution, TCP handshake,
TLS
handshake, and the HTTP/2 preface, and retries paid it again on every
attempt.
Reusing the channel removes that cost and matches how the Scala and Python
Spark
Connect clients work.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Reconnection was checked by killing the server container while a session was
idle
and starting a new one on the same port; the next query on the same
`SparkSession`
succeeded, so the shared channel reconnects on its own.
Performance, timed on loopback with plaintext transport over three runs each
on two
freshly started servers. `spark.conf.get(...)` is a single unary RPC per
iteration,
so it isolates the connection cost:
| benchmark | before | after |
| --- | --- | --- |
| `spark.conf.get(...)` x 100 | 2.58-4.00 ms/RPC | 0.95-1.52 ms/RPC |
| `spark.catalog.currentDatabase()` x 50 | 9.18-12.88 ms | 6.74-9.57 ms |
| `spark.range(1).count()` x 50 | 16.34-31.19 ms | 12.85-22.34 ms |
That is roughly 1.5-2.5 ms saved per RPC. Loopback with plaintext is the most
favorable case for the old code; against a remote server over TLS, each RPC
previously paid about three extra round trips that are now paid once per
session.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
