Bobby Wang created SPARK-55326:
----------------------------------
Summary: Add an optional connect session release in pyspark
Key: SPARK-55326
URL: https://issues.apache.org/jira/browse/SPARK-55326
Project: Spark
Issue Type: Improvement
Components: Connect, PySpark
Affects Versions: 4.2
Reporter: Bobby Wang
In Spark Connect, if a user’s code does not include {{{}spark.stop(){}}}—an
omission that is quite common—the session will persist on the Spark Connect
Server until the idle timeout elapses, which can lead to unnecessary resource
waste. For this reason, I propose adding an environment variable in PySpark to
implement session release logic within the PySpark process’s shutdown hook.
When this environment variable is enabled, PySpark will attempt to release the
session as best it can before the Python process exits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
