Jungtaek Lim created SPARK-61998:
------------------------------------

             Summary: Python worker factories are never released for ended 
sessions
                 Key: SPARK-61998
                 URL: https://issues.apache.org/jira/browse/SPARK-61998
             Project: Spark
          Issue Type: Bug
          Components: Spark Core, PySpark, Connect
    Affects Versions: 4.0.5, 4.1.4, 4.2.1, 4.3.0, 4.4.0, 5.0.0
            Reporter: Jungtaek Lim


{{SparkEnv}} keys {{PythonWorkerFactory}} by the worker environment, which 
includes {{SPARK_JOB_ARTIFACT_UUID}}, and stops the factories only in 
{{SparkEnv.stop}} ({{SparkEnv.scala}}). Each factory keeps a {{pyspark.daemon}} 
process and an idle-worker monitor thread whose loop never ends 
({{PythonWorkerFactory.scala}}). Neither the executor's cleanup of an evicted 
isolated session nor the driver's cleanup of a session's artifacts releases 
them.

So these leave a Python daemon and a thread on each executor that ran their 
Python tasks, until the JVM exits:

* every Spark Connect session that runs Python UDFs;
* every run of a Python {{foreachBatch}} that runs in the stream's cloned 
session ({{spark.sql.connect.streaming.foreachBatch.useClonedSession}}, default 
true since 4.3.0);
* after SPARK-61959, every run of a streaming query with Python UDFs in an 
isolated session, because each run's cloned session has a new UUID.

On a long-lived Spark Connect cluster, this can grow until it hits memory, 
process or thread limits.

Proposal:

* Add a {{SparkEnv}} operation that removes and stops the Python worker 
factories of a given {{SPARK_JOB_ARTIFACT_UUID}}, without stopping a factory 
while one of its workers is in use.
* Make {{PythonWorkerFactory.stop}} also end its idle-worker monitor thread.
* Call it where a UUID's lifetime ends: in the executor's cleanup of an 
isolated session, and in the driver's cleanup of a session's artifacts.
* Keep the factories of the default UUID as they are.

Found in the review of SPARK-61959 (https://github.com/apache/spark/pull/60657).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to