Hyukjin Kwon created SPARK-61492:
------------------------------------
Summary: Upgrade PyO3 to 0.29 in the Python client, so that its
PyO3 classes do not leak the instance dict that CPython 3.11 and 3.12 create
Key: SPARK-61492
URL: https://issues.apache.org/jira/browse/SPARK-61492
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
On CPython 3.11 and 3.12, object.__new__ creates the instance dict of every
instance of a type with a dict offset (_PyObject_InitializeDict), and PyO3 0.28
then overwrites that slot with null when it writes the contents of a
#[pyclass(dict)] instance (pyclass_init.rs), so every instance of the Python
client's dict classes (Column, DataFrame, SparkSession, DataType and its
subclasses, GroupedData, readers and writers, Observation, ...) leaks an empty
dict, 64 bytes, at creation, whether or not it ever gets an attribute: 20k
Columns keep 1.28 MB. CPython 3.9 and 3.13 do not pre-create the dict. PyO3
0.29 carries that dict over (pyclass_init.rs:161-185); it also releases the
dict at dealloc (release_dict) and traverses it for the garbage collector
(traverse_dict), which SPARK-61490 works around for the dealloc leak only.
The upgrade is blocked today: arrow-pyarrow 58.3.0 pins pyo3 0.28, and
pyo3-ffi's links = "python" forbids two versions in one build. Upgrade PyO3 to
0.29 together with an arrow/arrow-pyarrow release that supports it, then remove
the tp_dealloc workaround of SPARK-61490
(crates/pyspark-rs/src/instance_dict.rs), which stays harmless under 0.29 until
removed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]