[
https://issues.apache.org/jira/browse/SPARK-62063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62063:
------------------------------------
Assignee: Hyukjin Kwon
> Report the option keys and values that are objects, and the arguments that
> the writers keep, to Python's garbage collector in the readers, writers and
> session builder, also of their shallow copies
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62063
> URL: https://issues.apache.org/jira/browse/SPARK-62063
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Problem
> PySpark's readers, writers and Connect session builder are plain Python
> objects. Their option
> dicts keep a key that is not a str as it is (DataFrameReader.option,
> DataStreamReader.option,
> DataFrameWriter.option, DataFrameWriterV2.option and tableProperty,
> Builder.config;
> python/pyspark/sql/connect/readwriter.py, streaming/readwriter.py,
> session.py), and so do a table
> property's value. Their writers keep a format, a save mode, partitioning and
> clustering columns
> and a provider that are not strs as they are until the write. Python's
> garbage collector sees all
> of these through the objects' __dict__, also when a shallow copy (copy.copy)
> shares the dict.
> The drop-in keeps them in native state that the shallow copies share
> (crate::options::
> UnencodableOptions, the writers' UnsentArgs and kept columns, the builder's
> BuilderOptions), which
> the collector did not see:
> - a cycle through an option key, such as a key that keeps its reader, was
> never collected;
> - the V1 and V2 writers reported a few kept arguments only when no copy
> shared them, so a cycle
> through a writer and a copy of it was never collected either.
> Repro (master 80046f2f, against a Spark 4.3.0 server; these holders are the
> same at 4332de49)
> import copy, gc, weakref
> class Key:
> def __hash__(self): return 1
> def __eq__(self, other): return self is other
> def collected(make):
> key = Key(); key.held = make(key); freed = weakref.ref(key); del key;
> gc.collect()
> return freed() is None
> collected(lambda k: spark.read.option(k, "v"))
> collected(lambda k: spark.readStream.option(k, "v"))
> collected(lambda k: df.write.option(k, "v"))
> collected(lambda k: df.writeTo("t").tableProperty("p", k))
> from pyspark.sql.connect.session import SparkSession as ConnectSession
> collected(lambda k: ConnectSession.builder.config(k, "v"))
> collected(lambda k: (lambda w: (w, copy.copy(w)))(df.write.format(k)))
> collected(lambda k: (lambda w: (w, copy.copy(w)))(df.writeTo("t").using(k)))
> Actual (drop-in)
> False for each.
> Expected (pyspark-client 4.3.0)
> True for each. df.writeStream.option(key, "v") raises the TypeError of
> protobuf's map in both
> clients, and bucketBy and sortBy refuse such a column in both.
> Fix
> - A native object holds each state that the copies share
> (crate::options::SharedObjects,
> PySharedObjects). Each copy holds the object and reports it in its
> __traverse__, and the object
> reports the Python objects of the state, each once, and clears them in its
> __clear__.
> - So each object is freed with the last copy that holds the state.
> - The V1 and V2 writers report all of their kept arguments through it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]