Hyukjin Kwon created SPARK-61983:
------------------------------------
Summary: Report the DataFrame of toLocalIterator, the session of
DataSourceRegistration and the channel builder and hook factories of the
session builder to Python's garbage collector
Key: SPARK-61983
URL: https://issues.apache.org/jira/browse/SPARK-61983
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
SPARK-61971 made the native objects that hold the UDF functions or the
DataFrame of a plan report
them to Python's garbage collector, as PySpark's plain Python objects are seen
by it. Three more
native objects hold Python objects that the collector does not see, so a
reference cycle through
them is never collected, where PySpark's is:
1. The rows of DataFrame.toLocalIterator() (PyLocalRowIterator,
crates/pyspark-rs/src/dataframe.rs)
hold a clone of the plan of the DataFrame, and not the DataFrame. PySpark's
toLocalIterator is
a generator, whose frame holds self until it ends
(python/pyspark/sql/connect/dataframe.py:
2223-2241). So:
- a cycle through the generator and a function of a UDF of the plan, such as
a bound method of
an object that keeps the generator, is never freed;
- while the generator lives, the DataFrame does not report its own plan
either, as the clone
is not counted by its holders (crate::gc::Holders);
- the DataFrame is freed while its generator, not started or not ended,
lives, where PySpark
keeps it until the generator ends.
2. spark.dataSource (PyDataSourceRegistration,
crates/pyspark-rs/src/datasource.rs) holds its
session, as PySpark's DataSourceRegistration(self) does
(connect/session.py:1089-1092), and does
not report it: a session that keeps its dataSource is never freed, nor is
its client.
3. The builder of pyspark.sql.connect.session.SparkSession
(PySparkSessionBuilder,
crates/pyspark-rs/src/session.rs) holds its channel builder and its hook
factories, as PySpark's
builder holds _channel_builder and _hook_factories
(connect/session.py:131-132, 189, 196), and
does not report them.
Repro (master 4f0d0fa9, whose code of these objects 9cf73197 keeps, against a
Spark 4.3.0 server)
import gc, weakref
from pyspark.sql import functions as F
from pyspark.sql.connect.client import DefaultChannelBuilder
from pyspark.sql.connect.session import SparkSession as ConnectSession
class Job:
def f(self, x):
return x
def collected(make):
job = Job(); job.held = make(job); freed = weakref.ref(job); del job;
gc.collect()
return freed() is None
def channel_builder(job):
channel = DefaultChannelBuilder("sc://localhost:15002"); channel.job = job
return ConnectSession.builder.channelBuilder(channel)
collected(lambda job: spark.range(1).select(F.udf(job.f,
"long")("id")).toLocalIterator())
collected(channel_builder)
collected(lambda job: ConnectSession.builder._registerHook(lambda session:
job))
s2 = spark.newSession(); s2.kept = s2.dataSource; freed = weakref.ref(s2);
del s2; gc.collect()
freed() is None
df = spark.range(3); rows = df.toLocalIterator(); freed = weakref.ref(df);
del df; gc.collect()
freed() is not None
The drop-in gives False for each; pyspark-client 4.3.0 gives True for each.
Fix
- The rows of toLocalIterator hold the DataFrame and a share of the holders of
its plan
(PyDataFrame::plan_share), report the DataFrame in __traverse__, and drop
their clone of the plan
in __clear__, as grouped data and the writers do since SPARK-61971.
- DataSourceRegistration reports its session, and the session builder its
channel builder and its
hook factories, those that a copy of it does not share, each guarded by
crate::gc::GcReady.
- docs/compatibility.md and dev/parity/known-differences.md name them with the
other objects that
report to the collector.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]