Hyukjin Kwon created SPARK-61418:
------------------------------------

             Summary: Make VariantVal, Geometry and Geography PySpark's plain 
classes in the Python client, whose attributes are instance attributes as given 
and which are freed with their values
                 Key: SPARK-61418
                 URL: https://issues.apache.org/jira/browse/SPARK-61418
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The Python client's VariantVal, Geometry and Geography are PyO3 classes whose 
value/metadata and wkb/srid are Rust fields with read-only getters. PySpark's 
are plain Python classes whose __init__ stores its arguments, as given, as 
instance attributes. So, unlike PySpark's:
- the attributes cannot be written or deleted, and a subclass whose __init__ 
sets self.value (or self.wkb) fails: "attribute 'value' of 
'pyspark.sql.types.VariantVal' objects is not writable";
- vars(v) is {} where PySpark's is {'value': ..., 'metadata': ...};
- VariantVal("x", None) or Geometry(bytearray(...), 4326) raise or convert 
where PySpark's store the objects as given.

They are also #[pyclass(dict)] types, and PyO3 0.28 clears but never releases 
the __dict__ of such an instance when it is freed (PyClassDictSlot::clear_dict 
calls PyDict_Clear only; PyO3 0.29 adds release_dict). Since SPARK-61402, 
pickling a value creates its __dict__, so each pickled value, and each value 
with an attribute set, keeps 128 bytes after it is freed: 100k pickled 
VariantVals keep 12.8 MB (official PySpark: 88 bytes).

Fix: embed PySpark 4.3.0's own three classes as Python source, as the client 
embeds Row (row.py), and build and read their attributes from Rust.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to