Juliusz Sompolski created SPARK-44730:
-----------------------------------------
Summary: Spark Connect: Cleaner thread not stopped when
SparkSession stops
Key: SPARK-44730
URL: https://issues.apache.org/jira/browse/SPARK-44730
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 3.5.0, 4.0.0
Reporter: Juliusz Sompolski
Spark Connect scala client SparkSession has a cleaner, which starts a daemon
thread to clean up Closeable objects after GC. This daemon thread is never
stopped, and every SparkSession creates a new one.
Cleaner implements a stop() function, but no-one ever calls it. Possibly
because even after SparkSession.stop(), the cleaner may still be needed when
remaining references are GCed... For this reason it seems that the Cleaner
should rather be a global singleton than within a session.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
