Hyukjin Kwon created SPARK-61165:
------------------------------------
Summary: Keep the metadata of a StructField as its own dict, so
that edits of it show in the field and its struct, in the Python client as
PySpark does
Key: SPARK-61165
URL: https://issues.apache.org/jira/browse/SPARK-61165
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In-place edits of a StructField's metadata are silently lost in the Python
client: `f.metadata["comment"] = "x"` changes a temporary dict, so `f.metadata`
is `{}` again, and the edit is not in `f.jsonValue()`, the struct holding the
field, or the schema of a DataFrame created with it. PySpark's StructField
keeps its metadata dict, so the edit persists.
Repro: `f = StructField("a", IntegerType()); f.metadata["comment"] = "x";
f.metadata` gives `{}` in the client and `{'comment': 'x'}` in PySpark 4.3.0.
The fix keeps the dict in the field and reads the metadata from it when the
field is converted to a Rust type.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
