[
https://issues.apache.org/jira/browse/SPARK-61492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61492:
------------------------------------
Assignee: Hyukjin Kwon
> Upgrade PyO3 to 0.29 in the Python client, so that its PyO3 classes do not
> leak the instance dict that CPython 3.11 and 3.12 create
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61492
> URL: https://issues.apache.org/jira/browse/SPARK-61492
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> On CPython 3.11 and 3.12, object.__new__ creates the instance dict of every
> instance of a type with a dict offset (_PyObject_InitializeDict), and PyO3
> 0.28 then overwrites that slot with null when it writes the contents of a
> #[pyclass(dict)] instance (pyclass_init.rs), so every instance of the Python
> client's dict classes (Column, DataFrame, SparkSession, DataType and its
> subclasses, GroupedData, readers and writers, Observation, ...) leaks an
> empty dict, 64 bytes, at creation, whether or not it ever gets an attribute:
> 20k Columns keep 1.28 MB. CPython 3.9 and 3.13 do not pre-create the dict.
> PyO3 0.29 carries that dict over (pyclass_init.rs:161-185); it also releases
> the dict at dealloc (release_dict) and traverses it for the garbage collector
> (traverse_dict), which SPARK-61490 works around for the dealloc leak only.
> The upgrade is blocked today: arrow-pyarrow 58.3.0 pins pyo3 0.28, and
> pyo3-ffi's links = "python" forbids two versions in one build. Upgrade PyO3
> to 0.29 together with an arrow/arrow-pyarrow release that supports it, then
> remove the tp_dealloc workaround of SPARK-61490
> (crates/pyspark-rs/src/instance_dict.rs), which stays harmless under 0.29
> until removed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]