[
https://issues.apache.org/jira/browse/SPARK-62017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62017:
-----------------------------------
Labels: pull-request-available (was: )
> Report the types, flags, metadata and fields that the array, map, field and
> struct types hold to Python's garbage collector
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62017
> URL: https://issues.apache.org/jira/browse/SPARK-62017
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Problem
> PySpark's ArrayType, MapType, StructField and StructType are plain Python
> classes. Their __dict__
> holds the type objects that they are made of, their flags, the metadata dict
> of a field, and the
> fields and names lists of a struct, so the garbage collector collects a
> reference cycle through
> them. The drop-in holds them in Rust, and the four classes have no
> __traverse__. So such a cycle is
> never collected, as SPARK-61971 found of the other holders of Python objects:
> - an array of a DecimalType that has an attribute that keeps the array;
> - a map of such a key or value;
> - a field of such a type, or of a metadata dict that keeps the field;
> - a struct of such a field, also of StructType.add.
> Repro (master 50594b2e)
> >>> class Job: pass
> >>> job = Job(); t = DecimalType(10, 2); t.job = job
> >>> job.held = ArrayType(t)
> >>> ref = weakref.ref(job); del job, t; gc.collect(); ref() is None
> False
> Expected (as with pyspark-client 4.3.0)
> True, and so for the map, the field, the metadata dict and the struct. A
> child that is a
> singleton, such as LongType(), keeps the cycle in both clients, as
> PySpark's singleton holds it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]