[
https://issues.apache.org/jira/browse/SPARK-61983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61983:
------------------------------------
Assignee: Hyukjin Kwon
> Report the DataFrame of toLocalIterator, the session of
> DataSourceRegistration and the channel builder and hook factories of the
> session builder to Python's garbage collector
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61983
> URL: https://issues.apache.org/jira/browse/SPARK-61983
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Problem
> SPARK-61971 made the native objects that hold the UDF functions or the
> DataFrame of a plan report
> them to Python's garbage collector, as PySpark's plain Python objects are
> seen by it. Three more
> native objects hold Python objects that the collector does not see, so a
> reference cycle through
> them is never collected, where PySpark's is:
> 1. The rows of DataFrame.toLocalIterator() (PyLocalRowIterator,
> crates/pyspark-rs/src/dataframe.rs)
> hold a clone of the plan of the DataFrame, and not the DataFrame.
> PySpark's toLocalIterator is
> a generator, whose frame holds self until it ends
> (python/pyspark/sql/connect/dataframe.py:
> 2223-2241). So:
> - a cycle through the generator and a function of a UDF of the plan, such
> as a bound method of
> an object that keeps the generator, is never freed;
> - while the generator lives, the DataFrame does not report its own plan
> either, as the clone
> is not counted by its holders (crate::gc::Holders);
> - the DataFrame is freed while its generator, not started or not ended,
> lives, where PySpark
> keeps it until the generator ends.
> 2. spark.dataSource (PyDataSourceRegistration,
> crates/pyspark-rs/src/datasource.rs) holds its
> session, as PySpark's DataSourceRegistration(self) does
> (connect/session.py:1089-1092), and does
> not report it: a session that keeps its dataSource is never freed, nor is
> its client.
> 3. The builder of pyspark.sql.connect.session.SparkSession
> (PySparkSessionBuilder,
> crates/pyspark-rs/src/session.rs) holds its channel builder and its hook
> factories, as PySpark's
> builder holds _channel_builder and _hook_factories
> (connect/session.py:131-132, 189, 196), and
> does not report them.
> Repro (master 4f0d0fa9, whose code of these objects 9cf73197 keeps, against a
> Spark 4.3.0 server)
> import gc, weakref
> from pyspark.sql import functions as F
> from pyspark.sql.connect.client import DefaultChannelBuilder
> from pyspark.sql.connect.session import SparkSession as ConnectSession
> class Job:
> def f(self, x):
> return x
> def collected(make):
> job = Job(); job.held = make(job); freed = weakref.ref(job); del job;
> gc.collect()
> return freed() is None
> def channel_builder(job):
> channel = DefaultChannelBuilder("sc://localhost:15002"); channel.job =
> job
> return ConnectSession.builder.channelBuilder(channel)
> collected(lambda job: spark.range(1).select(F.udf(job.f,
> "long")("id")).toLocalIterator())
> collected(channel_builder)
> collected(lambda job: ConnectSession.builder._registerHook(lambda session:
> job))
> s2 = spark.newSession(); s2.kept = s2.dataSource; freed = weakref.ref(s2);
> del s2; gc.collect()
> freed() is None
> df = spark.range(3); rows = df.toLocalIterator(); freed = weakref.ref(df);
> del df; gc.collect()
> freed() is not None
> The drop-in gives False for each; pyspark-client 4.3.0 gives True for each.
> Fix
> - The rows of toLocalIterator hold the DataFrame and a share of the holders
> of its plan
> (PyDataFrame::plan_share), report the DataFrame in __traverse__, and drop
> their clone of the plan
> in __clear__, as grouped data and the writers do since SPARK-61971.
> - DataSourceRegistration reports its session, and the session builder its
> channel builder and its
> hook factories, those that a copy of it does not share, each guarded by
> crate::gc::GcReady.
> - docs/compatibility.md and dev/parity/known-differences.md name them with
> the other objects that
> report to the collector.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]