kkondamadugula opened a new issue, #57123:
URL: https://github.com/apache/spark/issues/57123
### Summary
We are running Spark Connect as a long-lived Kubernetes service. After Spark
Connect client jobs complete and executor pods terminate, the Connect
server/driver pod retains high JVM RSS. Some RSS retention may be normal JVM
heap commitment behavior, but heap inspection also shows a large number of
Spark SQL session/state/catalog objects still live after jobs complete.
I am trying to determine whether this is expected Spark Connect behavior,
missing server-side session cleanup configuration, Spark UI/status-store
retention, or a potential Spark Connect memory-retention issue.
Important distinction: I understand that JVM RSS may not immediately return
to the OS after GC because the heap can remain committed. The concern here is
not only RSS. The concern is that the live heap histogram after completed jobs
still shows many Spark SQL session/state/catalog objects and large retained map
structures.
### Operational impact
The main operational issue is not only that the JVM RSS remains high. The
bigger issue is that the only reliable way we have found to return the Spark
Connect server to baseline memory is restarting the Connect server pod.
Restarting the Spark Connect server is disruptive because it loses
server-side Spark Connect session context. Any active or long-running client
workload connected to that server can fail with errors such as lost/stopped
Spark Connect context, unavailable session, or internal server errors. In our
orchestration layer, this can cause the parent workflow to fail or cancel
dependent child workflows.
For long-running workloads, this means we cannot safely use “restart the
Connect server” as a cleanup strategy unless we accept losing in-flight job
progress and forcing the workflow to restart from its own external
checkpointing/retry logic.
### Component
Spark Connect / Spark SQL / Spark UI status store
### Spark version
Apache Spark `4.0.0`
### Environment
- Spark Connect server running as a long-lived Kubernetes pod
- Kubernetes namespace: internal stage environment
- Spark Connect server acts as the driver
- Driver config:
- `spark.driver.memory=20g`
- `spark.driver.memoryOverhead=6g`
- Kubernetes driver memory limit: `26g`
- JVM has `-Xmx20g`
- Dynamic executor pods are created per workload and terminate after
completion
- Event logging enabled to S3:
- `spark.eventLog.enabled=true`
- `spark.eventLog.dir=s3a://.../spark-event-logs/`
- `spark.eventLog.rolling.enabled=true`
### Summary
We are seeing high JVM/process memory retention in a long-lived Spark
Connect server after client workloads complete.
The workload completes successfully, executor pods terminate, and the Spark
Connect server remains running, but the driver pod RSS does not return near its
initial baseline. Over several completed Spark Connect workloads, the server
retained progressively higher memory.
The retained memory appears to be JVM/process memory in the Spark Connect
server, not executor memory, local disk files, event-log files, or `/tmp`
growth.
### Reproduction pattern
1. Start a fresh Spark Connect server pod.
2. Run Spark Connect client workloads that create Spark SQL/DataFrame jobs.
3. Let the client workload complete.
4. Ensure executor pods have terminated.
5. Observe Spark Connect server pod memory.
6. Repeat with additional workloads.
The server pod memory remains high after workloads complete.
### Observed memory progression
Fresh server baseline:
| State | Kubernetes pod memory | Java RSS |
|---|---:|---:|
| Fresh Spark Connect server | ~`1.6Gi` | ~`676Mi` |
After completed workloads:
| Workload | Workload status | Executor pods after completion | Retained
driver pod memory |
|---|---|---:|---:|
| Small job | Completed | 0 | ~`3.9Gi` |
| Medium job | Completed | 0 | ~`10.8Gi` |
| Large job | Completed | 0 | ~`18.4Gi` |
| Large job | Completed | 0 | ~`23.7Gi` |
The highest retained state observed was around:
| Metric | Value |
|---|---:|
| Kubernetes pod memory | ~`23.7Gi` |
| Java RSS | ~`22.5Gi` |
| cgroup `memory.current` | ~`23.5Gi` |
| cgroup `memory.peak` | ~`23.56Gi` |
### What was ruled out
This does not appear to be local filesystem growth.
`/tmp` remained stable around `2.8Gi` throughout the tests, mostly Ivy cache
and Spark user files.
`/proc/1/smaps_rollup` showed the memory is almost entirely
anonymous/private process memory:
```text
VmRSS: 23636848 kB
RssAnon: 23613348 kB
RssFile: 23500 kB
RssShmem: 0 kB
Private_Dirty: 23613356 kB
Anonymous: 23613348 kB
This suggests retained JVM/process memory, not file-backed Spark UI files,
event logs, local disk cache, or page cache.
Event logs are configured to S3, not local disk.
Server-side cleanup attempted
From a separate Spark Connect client session:
**spark.catalog.clearCache()
spark.range(1).count()
spark.stop()**
This completed successfully, but the Spark Connect server memory did not
reduce materially.
JVM / heap inspection
The Spark Connect server JVM command line includes:
**-Xmx20g
spark.driver.memory=20g
spark.driver.memoryOverhead=6g
spark.kubernetes.driver.limit.memory=26g
spark.eventLog.enabled=true
spark.eventLog.dir=s3a://.../spark-event-logs/
spark.eventLog.rolling.enabled=true**
jcmd 1 GC.heap_info showed the G1 heap committed at the full 20Gi heap size.
Before explicit GC:
garbage-first heap total 20971520K, used 10989919K
After explicit GC:
garbage-first heap total 20971520K, used 10810926K
The used heap only reduced slightly, and RSS/cgroup memory did not return
near baseline.
During a later active workload, heap used moved up and down, for example:
used 17405456K # during materialization / executor churn
used 11819465K # later after GC during active job execution
This shows live heap can reduce, but the Spark Connect server process still
keeps high committed/RSS memory.
Heap histogram evidence
jcmd 1 GC.class_histogram showed about 10.9GB of live Java heap objects
after completed workloads.
Top retained object groups included:
java.util.concurrent.ConcurrentHashMap$Node ~4.07 GB
[Ljava.util.concurrent.ConcurrentHashMap$Node; ~1.76 GB
scala.collection.mutable.HashMap$Node ~1.43 GB
scala.Tuple2 ~1.05 GB
[Lscala.collection.mutable.HashMap$Node; ~460 MB
byte[] ~447 MB
boolean[] ~309 MB
Spark SQL / Connect / session-state-related objects were also present in
high counts:
org.apache.spark.sql.classic.SparkSession ~51,673 instances
org.apache.spark.sql.internal.SessionState ~51,672 instances
org.apache.spark.sql.hive.HiveSessionCatalog ~51,672 instances
org.apache.spark.sql.artifact.ArtifactManager ~51,672 instances
org.apache.spark.sql.hive.HiveSessionStateBuilder ~51,672 instances
Spark UI/status objects were present too:
org.apache.spark.status.TaskDataWrapper
org.apache.spark.sql.execution.ui.SQLPlanMetric
But UI/status objects alone were not the dominant retained byte count. The
dominant retained heap was large Java/Scala map structures, along with very
high counts of Spark SQL session/state/catalog/artifact objects.
Concern
The high count of SparkSession, SessionState, HiveSessionCatalog, and
ArtifactManager instances looks suspicious for a long-lived Spark Connect
server after client jobs complete.
It suggests that Spark Connect may be retaining server-side
session/session-state/catalog/listener/status structures after client sessions
finish, or that there is no effective cleanup/expiration happening for closed
client sessions.
Expected behavior
After a Spark Connect client session completes and calls spark.stop(), I
would expect the Spark Connect server to eventually release server-side state
associated with that client session, including:
Spark SQL session state
catalog/cache state
artifact/session manager state
query execution/status/listener state where safe
any per-session structures no longer needed
Alternatively, if this retention is expected, it would be helpful to
document the recommended Spark Connect server configs and operational lifecycle
for long-lived servers.
Actual behavior
The Spark Connect server remains at high JVM RSS after:
client workload completes
client calls spark.stop()
executor pods terminate
Spark SQL cache clear is called
explicit JVM GC is triggered
The only reliable way observed to return memory to the original baseline is
restarting the Spark Connect server pod.
Questions
Is it expected for a Spark Connect server to retain many SparkSession /
SessionState / HiveSessionCatalog / ArtifactManager objects after client
sessions complete?
Are there Spark Connect server-side session expiration or cleanup configs
that should be enabled for long-lived Connect servers?
Are Spark UI/status retention configs involved in retaining SQL/session
state in Spark Connect?
Is there a known memory-retention issue in Spark Connect 4.0.0 related to
session cleanup?
Is this behavior improved in Spark 4.1.x or later?
What additional diagnostics would be useful to confirm whether this is a
Connect session leak, SQL status-store retention, catalog cache retention, or
expected JVM heap behavior?
### Bottomline:
Is there a supported way to clean up or expire completed Spark Connect
sessions and their server-side state without restarting the Spark Connect
server and losing active sessions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
