Hyukjin Kwon created SPARK-61983:
------------------------------------

             Summary: Report the DataFrame of toLocalIterator, the session of 
DataSourceRegistration and the channel builder and hook factories of the 
session builder to Python's garbage collector
                 Key: SPARK-61983
                 URL: https://issues.apache.org/jira/browse/SPARK-61983
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
SPARK-61971 made the native objects that hold the UDF functions or the 
DataFrame of a plan report
them to Python's garbage collector, as PySpark's plain Python objects are seen 
by it. Three more
native objects hold Python objects that the collector does not see, so a 
reference cycle through
them is never collected, where PySpark's is:

1. The rows of DataFrame.toLocalIterator() (PyLocalRowIterator, 
crates/pyspark-rs/src/dataframe.rs)
   hold a clone of the plan of the DataFrame, and not the DataFrame. PySpark's 
toLocalIterator is
   a generator, whose frame holds self until it ends 
(python/pyspark/sql/connect/dataframe.py:
   2223-2241). So:
   - a cycle through the generator and a function of a UDF of the plan, such as 
a bound method of
     an object that keeps the generator, is never freed;
   - while the generator lives, the DataFrame does not report its own plan 
either, as the clone
     is not counted by its holders (crate::gc::Holders);
   - the DataFrame is freed while its generator, not started or not ended, 
lives, where PySpark
     keeps it until the generator ends.
2. spark.dataSource (PyDataSourceRegistration, 
crates/pyspark-rs/src/datasource.rs) holds its
   session, as PySpark's DataSourceRegistration(self) does 
(connect/session.py:1089-1092), and does
   not report it: a session that keeps its dataSource is never freed, nor is 
its client.
3. The builder of pyspark.sql.connect.session.SparkSession 
(PySparkSessionBuilder,
   crates/pyspark-rs/src/session.rs) holds its channel builder and its hook 
factories, as PySpark's
   builder holds _channel_builder and _hook_factories 
(connect/session.py:131-132, 189, 196), and
   does not report them.

Repro (master 4f0d0fa9, whose code of these objects 9cf73197 keeps, against a 
Spark 4.3.0 server)
  import gc, weakref
  from pyspark.sql import functions as F
  from pyspark.sql.connect.client import DefaultChannelBuilder
  from pyspark.sql.connect.session import SparkSession as ConnectSession

  class Job:
      def f(self, x):
          return x

  def collected(make):
      job = Job(); job.held = make(job); freed = weakref.ref(job); del job; 
gc.collect()
      return freed() is None

  def channel_builder(job):
      channel = DefaultChannelBuilder("sc://localhost:15002"); channel.job = job
      return ConnectSession.builder.channelBuilder(channel)

  collected(lambda job: spark.range(1).select(F.udf(job.f, 
"long")("id")).toLocalIterator())
  collected(channel_builder)
  collected(lambda job: ConnectSession.builder._registerHook(lambda session: 
job))
  s2 = spark.newSession(); s2.kept = s2.dataSource; freed = weakref.ref(s2); 
del s2; gc.collect()
  freed() is None
  df = spark.range(3); rows = df.toLocalIterator(); freed = weakref.ref(df); 
del df; gc.collect()
  freed() is not None

The drop-in gives False for each; pyspark-client 4.3.0 gives True for each.

Fix
- The rows of toLocalIterator hold the DataFrame and a share of the holders of 
its plan
  (PyDataFrame::plan_share), report the DataFrame in __traverse__, and drop 
their clone of the plan
  in __clear__, as grouped data and the writers do since SPARK-61971.
- DataSourceRegistration reports its session, and the session builder its 
channel builder and its
  hook factories, those that a copy of it does not share, each guarded by 
crate::gc::GcReady.
- docs/compatibility.md and dev/parity/known-differences.md name them with the 
other objects that
  report to the collector.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to