[ 
https://issues.apache.org/jira/browse/SPARK-61983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61983:
------------------------------------

    Assignee: Hyukjin Kwon

> Report the DataFrame of toLocalIterator, the session of 
> DataSourceRegistration and the channel builder and hook factories of the 
> session builder to Python's garbage collector
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61983
>                 URL: https://issues.apache.org/jira/browse/SPARK-61983
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> SPARK-61971 made the native objects that hold the UDF functions or the 
> DataFrame of a plan report
> them to Python's garbage collector, as PySpark's plain Python objects are 
> seen by it. Three more
> native objects hold Python objects that the collector does not see, so a 
> reference cycle through
> them is never collected, where PySpark's is:
> 1. The rows of DataFrame.toLocalIterator() (PyLocalRowIterator, 
> crates/pyspark-rs/src/dataframe.rs)
>    hold a clone of the plan of the DataFrame, and not the DataFrame. 
> PySpark's toLocalIterator is
>    a generator, whose frame holds self until it ends 
> (python/pyspark/sql/connect/dataframe.py:
>    2223-2241). So:
>    - a cycle through the generator and a function of a UDF of the plan, such 
> as a bound method of
>      an object that keeps the generator, is never freed;
>    - while the generator lives, the DataFrame does not report its own plan 
> either, as the clone
>      is not counted by its holders (crate::gc::Holders);
>    - the DataFrame is freed while its generator, not started or not ended, 
> lives, where PySpark
>      keeps it until the generator ends.
> 2. spark.dataSource (PyDataSourceRegistration, 
> crates/pyspark-rs/src/datasource.rs) holds its
>    session, as PySpark's DataSourceRegistration(self) does 
> (connect/session.py:1089-1092), and does
>    not report it: a session that keeps its dataSource is never freed, nor is 
> its client.
> 3. The builder of pyspark.sql.connect.session.SparkSession 
> (PySparkSessionBuilder,
>    crates/pyspark-rs/src/session.rs) holds its channel builder and its hook 
> factories, as PySpark's
>    builder holds _channel_builder and _hook_factories 
> (connect/session.py:131-132, 189, 196), and
>    does not report them.
> Repro (master 4f0d0fa9, whose code of these objects 9cf73197 keeps, against a 
> Spark 4.3.0 server)
>   import gc, weakref
>   from pyspark.sql import functions as F
>   from pyspark.sql.connect.client import DefaultChannelBuilder
>   from pyspark.sql.connect.session import SparkSession as ConnectSession
>   class Job:
>       def f(self, x):
>           return x
>   def collected(make):
>       job = Job(); job.held = make(job); freed = weakref.ref(job); del job; 
> gc.collect()
>       return freed() is None
>   def channel_builder(job):
>       channel = DefaultChannelBuilder("sc://localhost:15002"); channel.job = 
> job
>       return ConnectSession.builder.channelBuilder(channel)
>   collected(lambda job: spark.range(1).select(F.udf(job.f, 
> "long")("id")).toLocalIterator())
>   collected(channel_builder)
>   collected(lambda job: ConnectSession.builder._registerHook(lambda session: 
> job))
>   s2 = spark.newSession(); s2.kept = s2.dataSource; freed = weakref.ref(s2); 
> del s2; gc.collect()
>   freed() is None
>   df = spark.range(3); rows = df.toLocalIterator(); freed = weakref.ref(df); 
> del df; gc.collect()
>   freed() is not None
> The drop-in gives False for each; pyspark-client 4.3.0 gives True for each.
> Fix
> - The rows of toLocalIterator hold the DataFrame and a share of the holders 
> of its plan
>   (PyDataFrame::plan_share), report the DataFrame in __traverse__, and drop 
> their clone of the plan
>   in __clear__, as grouped data and the writers do since SPARK-61971.
> - DataSourceRegistration reports its session, and the session builder its 
> channel builder and its
>   hook factories, those that a copy of it does not share, each guarded by 
> crate::gc::GcReady.
> - docs/compatibility.md and dev/parity/known-differences.md name them with 
> the other objects that
>   report to the collector.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to