Hyukjin Kwon created SPARK-62017:
------------------------------------

             Summary: Report the types, flags, metadata and fields that the 
array, map, field and struct types hold to Python's garbage collector
                 Key: SPARK-62017
                 URL: https://issues.apache.org/jira/browse/SPARK-62017
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
PySpark's ArrayType, MapType, StructField and StructType are plain Python 
classes. Their __dict__
holds the type objects that they are made of, their flags, the metadata dict of 
a field, and the
fields and names lists of a struct, so the garbage collector collects a 
reference cycle through
them. The drop-in holds them in Rust, and the four classes have no 
__traverse__. So such a cycle is
never collected, as SPARK-61971 found of the other holders of Python objects:
- an array of a DecimalType that has an attribute that keeps the array;
- a map of such a key or value;
- a field of such a type, or of a metadata dict that keeps the field;
- a struct of such a field, also of StructType.add.

Repro (master 50594b2e)
  >>> class Job: pass
  >>> job = Job(); t = DecimalType(10, 2); t.job = job
  >>> job.held = ArrayType(t)
  >>> ref = weakref.ref(job); del job, t; gc.collect(); ref() is None
  False

Expected (as with pyspark-client 4.3.0)
  True, and so for the map, the field, the metadata dict and the struct. A 
child that is a
  singleton, such as LongType(), keeps the cycle in both clients, as PySpark's 
singleton holds it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to