Hyukjin Kwon created SPARK-62063:
------------------------------------

             Summary: Report the option keys and values that are objects, and 
the arguments that the writers keep, to Python's garbage collector in the 
readers, writers and session builder, also of their shallow copies
                 Key: SPARK-62063
                 URL: https://issues.apache.org/jira/browse/SPARK-62063
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
PySpark's readers, writers and Connect session builder are plain Python 
objects. Their option
dicts keep a key that is not a str as it is (DataFrameReader.option, 
DataStreamReader.option,
DataFrameWriter.option, DataFrameWriterV2.option and tableProperty, 
Builder.config;
python/pyspark/sql/connect/readwriter.py, streaming/readwriter.py, session.py), 
and so do a table
property's value. Their writers keep a format, a save mode, partitioning and 
clustering columns
and a provider that are not strs as they are until the write. Python's garbage 
collector sees all
of these through the objects' __dict__, also when a shallow copy (copy.copy) 
shares the dict.

The drop-in keeps them in native state that the shallow copies share 
(crate::options::
UnencodableOptions, the writers' UnsentArgs and kept columns, the builder's 
BuilderOptions), which
the collector did not see:
- a cycle through an option key, such as a key that keeps its reader, was never 
collected;
- the V1 and V2 writers reported a few kept arguments only when no copy shared 
them, so a cycle
  through a writer and a copy of it was never collected either.

Repro (master 80046f2f, against a Spark 4.3.0 server; these holders are the 
same at 4332de49)
  import copy, gc, weakref
  class Key:
      def __hash__(self): return 1
      def __eq__(self, other): return self is other
  def collected(make):
      key = Key(); key.held = make(key); freed = weakref.ref(key); del key; 
gc.collect()
      return freed() is None
  collected(lambda k: spark.read.option(k, "v"))
  collected(lambda k: spark.readStream.option(k, "v"))
  collected(lambda k: df.write.option(k, "v"))
  collected(lambda k: df.writeTo("t").tableProperty("p", k))
  from pyspark.sql.connect.session import SparkSession as ConnectSession
  collected(lambda k: ConnectSession.builder.config(k, "v"))
  collected(lambda k: (lambda w: (w, copy.copy(w)))(df.write.format(k)))
  collected(lambda k: (lambda w: (w, copy.copy(w)))(df.writeTo("t").using(k)))

Actual (drop-in)
  False for each.

Expected (pyspark-client 4.3.0)
  True for each. df.writeStream.option(key, "v") raises the TypeError of 
protobuf's map in both
  clients, and bucketBy and sortBy refuse such a column in both.

Fix
- A native object holds each state that the copies share 
(crate::options::SharedObjects,
  PySharedObjects). Each copy holds the object and reports it in its 
__traverse__, and the object
  reports the Python objects of the state, each once, and clears them in its 
__clear__.
- So each object is freed with the last copy that holds the state.
- The V1 and V2 writers report all of their kept arguments through it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to