Hyukjin Kwon created SPARK-62017:
------------------------------------
Summary: Report the types, flags, metadata and fields that the
array, map, field and struct types hold to Python's garbage collector
Key: SPARK-62017
URL: https://issues.apache.org/jira/browse/SPARK-62017
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
PySpark's ArrayType, MapType, StructField and StructType are plain Python
classes. Their __dict__
holds the type objects that they are made of, their flags, the metadata dict of
a field, and the
fields and names lists of a struct, so the garbage collector collects a
reference cycle through
them. The drop-in holds them in Rust, and the four classes have no
__traverse__. So such a cycle is
never collected, as SPARK-61971 found of the other holders of Python objects:
- an array of a DecimalType that has an attribute that keeps the array;
- a map of such a key or value;
- a field of such a type, or of a metadata dict that keeps the field;
- a struct of such a field, also of StructType.add.
Repro (master 50594b2e)
>>> class Job: pass
>>> job = Job(); t = DecimalType(10, 2); t.job = job
>>> job.held = ArrayType(t)
>>> ref = weakref.ref(job); del job, t; gc.collect(); ref() is None
False
Expected (as with pyspark-client 4.3.0)
True, and so for the map, the field, the metadata dict and the struct. A
child that is a
singleton, such as LongType(), keeps the cycle in both clients, as PySpark's
singleton holds it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]