This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new c96d2156 [SPARK-61165][CONNECT][PYTHON] Keep the metadata of a 
StructField as its own dict, so that edits of it show in the field and its 
struct, in the Python client as PySpark does
c96d2156 is described below

commit c96d21567ea3fc2963ed2af88cc534c0636ea109
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 14:51:02 2026 +0900

    [SPARK-61165][CONNECT][PYTHON] Keep the metadata of a StructField as its 
own dict, so that edits of it show in the field and its struct, in the Python 
client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61165](https://issues.apache.org/jira/browse/SPARK-61165): Keep the 
metadata of a `StructField` as its own dict, so that edits of it show in the 
field and its struct, in the Python client as PySpark does.
    
    A `StructField` of the Python client now keeps its metadata dict, as 
PySpark's `StructField` does (`self.metadata = metadata or {}`):
    
    - `f.metadata` is the same dict on each access, so `f.metadata["comment"] = 
"x"` changes the field. The edit shows in `f.jsonValue()`, `f.json()`, the 
equality of fields, pickling, and the `StructType` that holds the field 
(`st.jsonValue()`, `st.json()`, `st["a"].metadata[...]`), and in the schema of 
a DataFrame created with the struct.
    - A non-empty dict passed to the constructor is stored as is, so an edit of 
it shows in the field. An empty or no dict gives a new one.
    - The metadata is read from the dict when the field is converted to the 
core type, so the field and the struct use the metadata as it is now. A 
`StructType` takes the metadata of each field that has its `StructField` object 
from that object (SPARK-61137 keeps those objects); the others are as before.
    
    Official PySpark 4.3.0 gives the same results for each of these cases.
    
    ### Why are the changes needed?
    
    The client's `StructField` built a new metadata dict from the Rust field on 
each access, so an in-place edit went to a temporary and was lost without an 
error:
    
    ```python
    f = StructField("a", IntegerType())
    f.metadata["comment"] = "x"
    f.metadata                      # {} in the client, {'comment': 'x'} in 
PySpark 4.3.0
    ```
    
    The same happened through a struct: `st["a"].metadata["comment"] = "x"` did 
not change `st.jsonValue()`, `StructType.fromJson(st.jsonValue()) == st` was 
false, and `createDataFrame(rows, st)` did not keep the metadata in `df.schema`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. An in-place edit of the metadata of a `StructField` now persists, as 
in PySpark 4.3.0. Otherwise the metadata of a field is the same.
    
    ### How was this patch tested?
    
    - `test_struct_field_metadata_is_the_stored_dict` (offline): an edit in 
place of `f.metadata` (identity of the dict, `jsonValue`, `json`, equality, 
pickling), the identity of a passed non-empty dict and the new dict of an empty 
or no one, and the edit through a struct made of the field, of an added field 
and of a field taken from the struct (`jsonValue`, `json`, `fromJson` equality, 
pickling). It fails without this change and passes with it, and the same 
results come from official PyS [...]
    - `test_create_dataframe_schema_keeps_metadata_edited_in_place` (live): 
`createDataFrame(rows, st)` after an in-place metadata edit keeps the metadata 
in `df.schema`, `df.schema.jsonValue()`, and the rows are collected. It fails 
without this change (`{} == {'comment': 'the a'}`) and passes with it; official 
PySpark 4.3.0 gives the same schema.
    - Apache Spark's `test_parity_types.py` (connect, against a local Spark 
4.3.0 server, drop-in run as `SPARK_CONNECT_TESTING_REMOTE`): 69 passed, 21 
skipped, 1 failed, the same as before the change. The failure is 
`test_infer_schema_specification`, which fails on the server with 
`UNSUPPORTED_TIME_TYPE` in both runs. The struct, metadata, field and schema 
cases: 20 passed, 10 skipped, 1 failed (the same test).
    - Offline suites (`test_dropin_offline.py`, `test_arrow_handoff.py`, 
`test_ml_offline.py`): 2212 passed, 8 skipped.
    - Live `-k "struct or schema or metadata or types"` and the new live test: 
46 passed.
    - `cargo fmt --all --check` passes. `cargo clippy` reports no diagnostics 
in the changed lines. `cargo test -p apache-spark-connect --release --no-run` 
compiles all tests.
    - The tests of both crates, e2e included, against a local Spark 4.3.0 
server (884 passed), the build of all tests, also with `wasm-udf`, the offline 
Python suites (2212 passed, 8 skipped), and the live drop-in, Arrow and ML 
suites (193 passed) and live cases (10 passed). rustfmt passes, and clippy 
reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #354 from HyukjinKwon/SPARK-61165-struct-field-metadata-dict.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/catalog.rs    |   2 +-
 crates/pyspark-rs/src/readwriter.rs |   2 +-
 crates/pyspark-rs/src/types.rs      | 108 ++++++++++++++++++++++++++++--------
 python/tests/test_dropin_e2e.py     |  13 +++++
 python/tests/test_dropin_offline.py |  41 ++++++++++++++
 5 files changed, 140 insertions(+), 26 deletions(-)

diff --git a/crates/pyspark-rs/src/catalog.rs b/crates/pyspark-rs/src/catalog.rs
index d587ee6d..754fa327 100644
--- a/crates/pyspark-rs/src/catalog.rs
+++ b/crates/pyspark-rs/src/catalog.rs
@@ -37,7 +37,7 @@ fn table_schema(
     };
     if let Ok(st) = schema.extract::<PyRef<'_, crate::types::PyStructType>>() {
         return Ok(Some(spark_connect::types::DataType::Struct {
-            fields: st.fields.clone(),
+            fields: st.current_fields(schema.py())?,
         }));
     }
     let type_name = schema.get_type().name()?.to_string();
diff --git a/crates/pyspark-rs/src/readwriter.rs 
b/crates/pyspark-rs/src/readwriter.rs
index 2c22f04d..ae37bec0 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -202,7 +202,7 @@ pub(crate) fn reader_schema(schema: &Bound<'_, PyAny>) -> 
PyResult<String> {
     }
     if let Ok(st) = schema.extract::<PyRef<crate::types::PyStructType>>() {
         return Ok(spark_connect::types::DataType::Struct {
-            fields: st.fields.clone(),
+            fields: st.current_fields(schema.py())?,
         }
         .json());
     }
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 88d01c45..1d479945 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -1862,6 +1862,30 @@ impl PyMapType {
 #[pyclass(name = "StructField", module = "pyspark.sql.types")]
 pub struct PyStructField {
     pub(crate) field: StructField,
+    /// The metadata dict, which PySpark's `StructField` keeps as it is: an 
edit of it in place
+    /// is the metadata of the field. `field.metadata` is only its snapshot at 
the time it was
+    /// made, so the metadata is read through `current_field`.
+    metadata: Py<pyo3::types::PyDict>,
+}
+
+impl PyStructField {
+    /// A field of a core field, with a metadata dict of its metadata.
+    pub(crate) fn from_field(py: Python<'_>, field: StructField) -> 
PyResult<Self> {
+        let metadata = metadata_map_to_py(py, &field.metadata)?.unbind();
+        Ok(PyStructField { field, metadata })
+    }
+
+    /// The core field, with the metadata of the dict as it is now.
+    pub(crate) fn current_field(&self, py: Python<'_>) -> 
PyResult<StructField> {
+        let mut field = self.field.clone();
+        let metadata = self.metadata.bind(py);
+        field.metadata = if metadata.is_empty() {
+            BTreeMap::new()
+        } else {
+            py_metadata_to_map(py, metadata)?
+        };
+        Ok(field)
+    }
 }
 
 #[pymethods]
@@ -1889,12 +1913,12 @@ impl PyStructField {
         ))
     }
     #[pyo3(name = "json")]
-    fn __sf_json(&self) -> String {
-        self.field.json_value().to_string()
+    fn __sf_json(&self, py: Python<'_>) -> PyResult<String> {
+        Ok(self.current_field(py)?.json_value().to_string())
     }
     #[pyo3(name = "jsonValue")]
     fn __sf_json_value<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, 
PyAny>> {
-        let v = self.field.json_value();
+        let v = self.current_field(py)?.json_value();
         py_json_value(py, &v, true)
     }
     #[pyo3(name = "needConversion")]
@@ -1972,7 +1996,11 @@ impl PyStructField {
     /// Mirrors `StructField.fromDDL`.
     #[classmethod]
     #[pyo3(name = "fromDDL")]
-    fn __sf_from_ddl(_cls: &Bound<'_, pyo3::types::PyType>, ddl: &str) -> 
PyResult<PyStructField> {
+    fn __sf_from_ddl(
+        _cls: &Bound<'_, pyo3::types::PyType>,
+        py: Python<'_>,
+        ddl: &str,
+    ) -> PyResult<PyStructField> {
         let trimmed = ddl.trim();
         // Split "name type" at the first run of whitespace; tolerate a 
"name:" colon.
         let split = trimmed.find(char::is_whitespace).ok_or_else(|| {
@@ -1984,14 +2012,15 @@ impl PyStructField {
         let type_str = trimmed[split..].trim();
         let data_type = DataType::from_ddl(type_str)
             .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError, 
_>(e.to_string()))?;
-        Ok(PyStructField {
-            field: StructField {
+        PyStructField::from_field(
+            py,
+            StructField {
                 name,
                 data_type,
                 nullable: true,
                 metadata: BTreeMap::new(),
             },
-        })
+        )
     }
 
     #[classmethod]
@@ -2009,15 +2038,22 @@ impl PyStructField {
     #[pyo3(signature = (name, dataType, nullable=true, metadata=None))]
     #[allow(non_snake_case)]
     fn new(
+        py: Python<'_>,
         name: String,
         dataType: &Bound<'_, PyAny>,
         nullable: bool,
         metadata: Option<&Bound<'_, pyo3::types::PyDict>>,
     ) -> PyResult<Self> {
         let md = match metadata {
-            Some(d) => py_metadata_to_map(d.py(), d)?,
+            Some(d) => py_metadata_to_map(py, d)?,
             None => BTreeMap::new(),
         };
+        // As PySpark's `metadata or {}`: a non-empty dict is kept as it is, 
so that an edit of
+        // it shows in the field; an empty or no dict gives a new one.
+        let metadata = match metadata {
+            Some(d) if !d.is_empty() => d.clone().unbind(),
+            _ => pyo3::types::PyDict::new(py).unbind(),
+        };
         Ok(PyStructField {
             field: StructField {
                 name,
@@ -2025,6 +2061,7 @@ impl PyStructField {
                 nullable,
                 metadata: md,
             },
+            metadata,
         })
     }
     #[getter]
@@ -2043,8 +2080,8 @@ impl PyStructField {
     }
     /// The field's metadata dict (mirrors `StructField.metadata`; values may 
be any type).
     #[getter]
-    fn metadata<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, 
pyo3::types::PyDict>> {
-        metadata_map_to_py(py, &self.field.metadata)
+    fn metadata<'py>(&self, py: Python<'py>) -> Bound<'py, 
pyo3::types::PyDict> {
+        self.metadata.bind(py).clone()
     }
     fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
         // Mirror pyspark: StructField('name', <DataType repr>, nullable).
@@ -2057,11 +2094,11 @@ impl PyStructField {
             if self.field.nullable { "True" } else { "False" }
         ))
     }
-    fn __eq__(&self, other: &Bound<'_, PyAny>) -> bool {
+    fn __eq__(&self, py: Python<'_>, other: &Bound<'_, PyAny>) -> 
PyResult<bool> {
         if let Ok(o) = other.extract::<PyRef<PyStructField>>() {
-            self.field == o.field
+            Ok(self.current_field(py)? == o.current_field(py)?)
         } else {
-            false
+            Ok(false)
         }
     }
     /// pyspark: `hash(str(self))` (inherited from `DataType`). Defining 
`__eq__` alone
@@ -2083,7 +2120,7 @@ impl PyStructField {
             this.field.name.clone(),
             data_type_to_py(py, &this.field.data_type)?,
             this.field.nullable,
-            metadata_map_to_py(py, &this.field.metadata)?,
+            this.metadata.clone_ref(py),
         )
             .into_pyobject(py)?;
         Ok((slf.get_type(), args))
@@ -2092,11 +2129,13 @@ impl PyStructField {
 
 #[pyclass(name = "StructType", module = "pyspark.sql.types", extends = 
PyDataType)]
 pub struct PyStructType {
+    /// The core fields. Where a field has its `StructField` object, its 
metadata may have been
+    /// edited since, so the metadata that counts is `current_fields`.
     pub(crate) fields: Vec<StructField>,
     /// The `StructField` objects of `fields`, by their positions: those that 
the struct was
     /// made of, and the others once they are made. So a field is the same 
object each time,
-    /// as PySpark's `StructType`, which holds its fields, gives them. A 
`StructField` cannot
-    /// change, so its object is always that of its field.
+    /// as PySpark's `StructType`, which holds its fields, gives them. The 
name and the type of
+    /// a field cannot change, so only its metadata can differ from `fields`.
     objects: std::sync::Mutex<Vec<Option<Py<PyStructField>>>>,
 }
 
@@ -2112,9 +2151,7 @@ impl PyStructType {
         }
         let object = Py::new(
             py,
-            PyStructField {
-                field: self.fields[index].clone(),
-            },
+            PyStructField::from_field(py, self.fields[index].clone())?,
         )?;
         objects[index] = Some(object.clone_ref(py));
         Ok(object)
@@ -2126,6 +2163,29 @@ impl PyStructType {
             .map(|index| self.field_object(py, index))
             .collect()
     }
+
+    /// The core fields with their metadata as it is now: a field that has its 
object takes its
+    /// metadata from the object, where an edit in place is kept, as PySpark's 
fields do. Those
+    /// without an object are as they are in `fields`.
+    pub(crate) fn current_fields(&self, py: Python<'_>) -> 
PyResult<Vec<StructField>> {
+        let objects: Vec<Option<Py<PyStructField>>> = {
+            let objects = self.objects.lock().unwrap_or_else(|e| 
e.into_inner());
+            objects
+                .iter()
+                .map(|object| object.as_ref().map(|object| 
object.clone_ref(py)))
+                .collect()
+        };
+        self.fields
+            .iter()
+            .enumerate()
+            .map(
+                |(index, field)| match 
objects.get(index).and_then(Option::as_ref) {
+                    Some(object) => object.bind(py).borrow().current_field(py),
+                    None => Ok(field.clone()),
+                },
+            )
+            .collect()
+    }
 }
 
 #[pymethods]
@@ -2165,7 +2225,7 @@ impl PyStructType {
         let py = slf.py();
         let this = slf.borrow();
         let dt = DataType::Struct {
-            fields: this.fields.clone(),
+            fields: this.current_fields(py)?,
         };
         reduce_with_udts(slf.get_type(), &dt, || {
             (this.field_objects(py)?,).into_pyobject(py)
@@ -2181,7 +2241,7 @@ impl PyStructType {
         if let Some(fs) = fields {
             for f in fs {
                 if let Ok(sf) = f.cast::<PyStructField>() {
-                    out.push(sf.borrow().field.clone());
+                    out.push(sf.borrow().current_field(f.py())?);
                     objects.push(Some(sf.clone().unbind()));
                 } else if let Ok(s) = f.extract::<String>() {
                     // "name type" DDL fragment.
@@ -2227,7 +2287,7 @@ impl PyStructType {
         let mut object = None;
         let new_field = if let Ok(sf) = field.cast::<PyStructField>() {
             object = Some(sf.clone().unbind());
-            sf.borrow().field.clone()
+            sf.borrow().current_field(field.py())?
         } else {
             // As PySpark's: a name takes a data type, of which a str is the 
JSON value of one.
             let Some(data_type) = data_type else {
@@ -2422,7 +2482,7 @@ impl PyStructType {
     #[pyo3(name = "toNullable")]
     fn __st_to_nullable(&self, py: Python<'_>) -> PyResult<Py<PyStructType>> {
         let fields = match (DataType::Struct {
-            fields: self.fields.clone(),
+            fields: self.current_fields(py)?,
         })
         .to_nullable()
         {
@@ -3716,7 +3776,7 @@ pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>) -> 
PyResult<DataType> {
     }
     if let Ok(st) = obj.extract::<PyRef<PyStructType>>() {
         return Ok(DataType::Struct {
-            fields: st.fields.clone(),
+            fields: st.current_fields(obj.py())?,
         });
     }
     if let Ok(c) = obj.extract::<PyRef<PyCharType>>() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index af4ab5a1..f3f383e8 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2873,6 +2873,19 @@ assert 
df.select(F.to_json(F.struct("p"))).collect()[1][0] == '{"p":[10.0,20.0]}
 """, remote], check=True)
 
 
+def test_create_dataframe_schema_keeps_metadata_edited_in_place(session):
+    # As PySpark: the schema of a DataFrame is the StructType it is created 
with, so a metadata
+    # edited in place of one of its fields is in df.schema.
+    from pyspark.sql import types as T
+
+    st = T.StructType([T.StructField("a", T.IntegerType()), T.StructField("b", 
T.StringType())])
+    st["a"].metadata["comment"] = "the a"
+    df = session.createDataFrame([(1, "x")], st)
+    assert df.schema["a"].metadata == {"comment": "the a"}
+    assert df.schema.jsonValue()["fields"][0]["metadata"] == {"comment": "the 
a"}
+    assert df.collect() == [Row(a=1, b="x")]
+
+
 def 
test_collected_values_of_user_defined_types_are_deserialized_as_pyspark(session):
     # As official PySpark 4.3.0: a collected value of a UDT is its 
deserialized object, and
     # dtypes names the UDT by its simpleString(), in every way that rows are 
collected.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 030892f7..c83fdb00 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6044,6 +6044,47 @@ def test_udt_not_implemented_defaults():
         udt.deserialize(1)
 
 
+def test_struct_field_metadata_is_the_stored_dict():
+    # As PySpark's StructField, which keeps its metadata dict: an edit in 
place shows in the
+    # field, its JSON, its equality and its pickle, and in the StructType that 
holds the field.
+    f = _SF("a", _IT())
+    f.metadata["comment"] = "x"
+    assert f.metadata == {"comment": "x"} and f.metadata is f.metadata
+    assert f.jsonValue()["metadata"] == {"comment": "x"}
+    assert json.loads(f.json())["metadata"] == {"comment": "x"}
+    assert f == _SF("a", _IT(), True, {"comment": "x"})
+    assert pickle.loads(pickle.dumps(f)).metadata == {"comment": "x"}
+    # A non-empty dict passed in is stored as is; an empty or no dict gives a 
new one.
+    d = {"k": 1}
+    g = _SF("a", _IT(), True, d)
+    assert g.metadata is d
+    d["k"] = 2
+    assert g.jsonValue()["metadata"] == {"k": 2}
+    empty = {}
+    h = _SF("a", _IT(), True, empty)
+    assert h.metadata == {} and h.metadata is not empty
+    a, b = _SF("a", _IT()), _SF("a", _IT())
+    assert a.metadata is not b.metadata
+    # The edits show in the StructType that holds the field, whether the 
struct is made of
+    # the field, the field is added, or the field is taken from the struct.
+    f2 = _SF("b", _IT())
+    st = _StT([f2])
+    f2.metadata["k"] = "v"
+    assert st.jsonValue()["fields"][0]["metadata"] == {"k": "v"}
+    st2 = _StT()
+    f3 = _SF("c", _IT())
+    st2.add(f3)
+    f3.metadata["k"] = "w"
+    assert st2.jsonValue()["fields"][0]["metadata"] == {"k": "w"}
+    st3 = _StT([_SF("a", _IT()), _SF("b", _AT(_IT()))])
+    st3["a"].metadata["comment"] = "x"
+    assert st3.jsonValue()["fields"][0]["metadata"] == {"comment": "x"}
+    assert st3["a"].metadata is st3.fields[0].metadata
+    assert json.loads(st3.json())["fields"][0]["metadata"] == {"comment": "x"}
+    assert _StT.fromJson(st3.jsonValue()) == st3
+    assert pickle.loads(pickle.dumps(st3)).jsonValue() == st3.jsonValue()
+
+
 def test_struct_field_metadata_any_values():
     # StructField metadata accepts arbitrary JSON values (Dict[str, Any]), not 
just str,
     # and round-trips through json.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to