This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new c96d2156 [SPARK-61165][CONNECT][PYTHON] Keep the metadata of a
StructField as its own dict, so that edits of it show in the field and its
struct, in the Python client as PySpark does
c96d2156 is described below
commit c96d21567ea3fc2963ed2af88cc534c0636ea109
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 14:51:02 2026 +0900
[SPARK-61165][CONNECT][PYTHON] Keep the metadata of a StructField as its
own dict, so that edits of it show in the field and its struct, in the Python
client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61165](https://issues.apache.org/jira/browse/SPARK-61165): Keep the
metadata of a `StructField` as its own dict, so that edits of it show in the
field and its struct, in the Python client as PySpark does.
A `StructField` of the Python client now keeps its metadata dict, as
PySpark's `StructField` does (`self.metadata = metadata or {}`):
- `f.metadata` is the same dict on each access, so `f.metadata["comment"] =
"x"` changes the field. The edit shows in `f.jsonValue()`, `f.json()`, the
equality of fields, pickling, and the `StructType` that holds the field
(`st.jsonValue()`, `st.json()`, `st["a"].metadata[...]`), and in the schema of
a DataFrame created with the struct.
- A non-empty dict passed to the constructor is stored as is, so an edit of
it shows in the field. An empty or no dict gives a new one.
- The metadata is read from the dict when the field is converted to the
core type, so the field and the struct use the metadata as it is now. A
`StructType` takes the metadata of each field that has its `StructField` object
from that object (SPARK-61137 keeps those objects); the others are as before.
Official PySpark 4.3.0 gives the same results for each of these cases.
### Why are the changes needed?
The client's `StructField` built a new metadata dict from the Rust field on
each access, so an in-place edit went to a temporary and was lost without an
error:
```python
f = StructField("a", IntegerType())
f.metadata["comment"] = "x"
f.metadata # {} in the client, {'comment': 'x'} in
PySpark 4.3.0
```
The same happened through a struct: `st["a"].metadata["comment"] = "x"` did
not change `st.jsonValue()`, `StructType.fromJson(st.jsonValue()) == st` was
false, and `createDataFrame(rows, st)` did not keep the metadata in `df.schema`.
### Does this PR introduce _any_ user-facing change?
Yes. An in-place edit of the metadata of a `StructField` now persists, as
in PySpark 4.3.0. Otherwise the metadata of a field is the same.
### How was this patch tested?
- `test_struct_field_metadata_is_the_stored_dict` (offline): an edit in
place of `f.metadata` (identity of the dict, `jsonValue`, `json`, equality,
pickling), the identity of a passed non-empty dict and the new dict of an empty
or no one, and the edit through a struct made of the field, of an added field
and of a field taken from the struct (`jsonValue`, `json`, `fromJson` equality,
pickling). It fails without this change and passes with it, and the same
results come from official PyS [...]
- `test_create_dataframe_schema_keeps_metadata_edited_in_place` (live):
`createDataFrame(rows, st)` after an in-place metadata edit keeps the metadata
in `df.schema`, `df.schema.jsonValue()`, and the rows are collected. It fails
without this change (`{} == {'comment': 'the a'}`) and passes with it; official
PySpark 4.3.0 gives the same schema.
- Apache Spark's `test_parity_types.py` (connect, against a local Spark
4.3.0 server, drop-in run as `SPARK_CONNECT_TESTING_REMOTE`): 69 passed, 21
skipped, 1 failed, the same as before the change. The failure is
`test_infer_schema_specification`, which fails on the server with
`UNSUPPORTED_TIME_TYPE` in both runs. The struct, metadata, field and schema
cases: 20 passed, 10 skipped, 1 failed (the same test).
- Offline suites (`test_dropin_offline.py`, `test_arrow_handoff.py`,
`test_ml_offline.py`): 2212 passed, 8 skipped.
- Live `-k "struct or schema or metadata or types"` and the new live test:
46 passed.
- `cargo fmt --all --check` passes. `cargo clippy` reports no diagnostics
in the changed lines. `cargo test -p apache-spark-connect --release --no-run`
compiles all tests.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (884 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2212 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (193 passed) and live cases (10 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #354 from HyukjinKwon/SPARK-61165-struct-field-metadata-dict.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/catalog.rs | 2 +-
crates/pyspark-rs/src/readwriter.rs | 2 +-
crates/pyspark-rs/src/types.rs | 108 ++++++++++++++++++++++++++++--------
python/tests/test_dropin_e2e.py | 13 +++++
python/tests/test_dropin_offline.py | 41 ++++++++++++++
5 files changed, 140 insertions(+), 26 deletions(-)
diff --git a/crates/pyspark-rs/src/catalog.rs b/crates/pyspark-rs/src/catalog.rs
index d587ee6d..754fa327 100644
--- a/crates/pyspark-rs/src/catalog.rs
+++ b/crates/pyspark-rs/src/catalog.rs
@@ -37,7 +37,7 @@ fn table_schema(
};
if let Ok(st) = schema.extract::<PyRef<'_, crate::types::PyStructType>>() {
return Ok(Some(spark_connect::types::DataType::Struct {
- fields: st.fields.clone(),
+ fields: st.current_fields(schema.py())?,
}));
}
let type_name = schema.get_type().name()?.to_string();
diff --git a/crates/pyspark-rs/src/readwriter.rs
b/crates/pyspark-rs/src/readwriter.rs
index 2c22f04d..ae37bec0 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -202,7 +202,7 @@ pub(crate) fn reader_schema(schema: &Bound<'_, PyAny>) ->
PyResult<String> {
}
if let Ok(st) = schema.extract::<PyRef<crate::types::PyStructType>>() {
return Ok(spark_connect::types::DataType::Struct {
- fields: st.fields.clone(),
+ fields: st.current_fields(schema.py())?,
}
.json());
}
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 88d01c45..1d479945 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -1862,6 +1862,30 @@ impl PyMapType {
#[pyclass(name = "StructField", module = "pyspark.sql.types")]
pub struct PyStructField {
pub(crate) field: StructField,
+ /// The metadata dict, which PySpark's `StructField` keeps as it is: an
edit of it in place
+ /// is the metadata of the field. `field.metadata` is only its snapshot at
the time it was
+ /// made, so the metadata is read through `current_field`.
+ metadata: Py<pyo3::types::PyDict>,
+}
+
+impl PyStructField {
+ /// A field of a core field, with a metadata dict of its metadata.
+ pub(crate) fn from_field(py: Python<'_>, field: StructField) ->
PyResult<Self> {
+ let metadata = metadata_map_to_py(py, &field.metadata)?.unbind();
+ Ok(PyStructField { field, metadata })
+ }
+
+ /// The core field, with the metadata of the dict as it is now.
+ pub(crate) fn current_field(&self, py: Python<'_>) ->
PyResult<StructField> {
+ let mut field = self.field.clone();
+ let metadata = self.metadata.bind(py);
+ field.metadata = if metadata.is_empty() {
+ BTreeMap::new()
+ } else {
+ py_metadata_to_map(py, metadata)?
+ };
+ Ok(field)
+ }
}
#[pymethods]
@@ -1889,12 +1913,12 @@ impl PyStructField {
))
}
#[pyo3(name = "json")]
- fn __sf_json(&self) -> String {
- self.field.json_value().to_string()
+ fn __sf_json(&self, py: Python<'_>) -> PyResult<String> {
+ Ok(self.current_field(py)?.json_value().to_string())
}
#[pyo3(name = "jsonValue")]
fn __sf_json_value<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py,
PyAny>> {
- let v = self.field.json_value();
+ let v = self.current_field(py)?.json_value();
py_json_value(py, &v, true)
}
#[pyo3(name = "needConversion")]
@@ -1972,7 +1996,11 @@ impl PyStructField {
/// Mirrors `StructField.fromDDL`.
#[classmethod]
#[pyo3(name = "fromDDL")]
- fn __sf_from_ddl(_cls: &Bound<'_, pyo3::types::PyType>, ddl: &str) ->
PyResult<PyStructField> {
+ fn __sf_from_ddl(
+ _cls: &Bound<'_, pyo3::types::PyType>,
+ py: Python<'_>,
+ ddl: &str,
+ ) -> PyResult<PyStructField> {
let trimmed = ddl.trim();
// Split "name type" at the first run of whitespace; tolerate a
"name:" colon.
let split = trimmed.find(char::is_whitespace).ok_or_else(|| {
@@ -1984,14 +2012,15 @@ impl PyStructField {
let type_str = trimmed[split..].trim();
let data_type = DataType::from_ddl(type_str)
.map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))?;
- Ok(PyStructField {
- field: StructField {
+ PyStructField::from_field(
+ py,
+ StructField {
name,
data_type,
nullable: true,
metadata: BTreeMap::new(),
},
- })
+ )
}
#[classmethod]
@@ -2009,15 +2038,22 @@ impl PyStructField {
#[pyo3(signature = (name, dataType, nullable=true, metadata=None))]
#[allow(non_snake_case)]
fn new(
+ py: Python<'_>,
name: String,
dataType: &Bound<'_, PyAny>,
nullable: bool,
metadata: Option<&Bound<'_, pyo3::types::PyDict>>,
) -> PyResult<Self> {
let md = match metadata {
- Some(d) => py_metadata_to_map(d.py(), d)?,
+ Some(d) => py_metadata_to_map(py, d)?,
None => BTreeMap::new(),
};
+ // As PySpark's `metadata or {}`: a non-empty dict is kept as it is,
so that an edit of
+ // it shows in the field; an empty or no dict gives a new one.
+ let metadata = match metadata {
+ Some(d) if !d.is_empty() => d.clone().unbind(),
+ _ => pyo3::types::PyDict::new(py).unbind(),
+ };
Ok(PyStructField {
field: StructField {
name,
@@ -2025,6 +2061,7 @@ impl PyStructField {
nullable,
metadata: md,
},
+ metadata,
})
}
#[getter]
@@ -2043,8 +2080,8 @@ impl PyStructField {
}
/// The field's metadata dict (mirrors `StructField.metadata`; values may
be any type).
#[getter]
- fn metadata<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py,
pyo3::types::PyDict>> {
- metadata_map_to_py(py, &self.field.metadata)
+ fn metadata<'py>(&self, py: Python<'py>) -> Bound<'py,
pyo3::types::PyDict> {
+ self.metadata.bind(py).clone()
}
fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
// Mirror pyspark: StructField('name', <DataType repr>, nullable).
@@ -2057,11 +2094,11 @@ impl PyStructField {
if self.field.nullable { "True" } else { "False" }
))
}
- fn __eq__(&self, other: &Bound<'_, PyAny>) -> bool {
+ fn __eq__(&self, py: Python<'_>, other: &Bound<'_, PyAny>) ->
PyResult<bool> {
if let Ok(o) = other.extract::<PyRef<PyStructField>>() {
- self.field == o.field
+ Ok(self.current_field(py)? == o.current_field(py)?)
} else {
- false
+ Ok(false)
}
}
/// pyspark: `hash(str(self))` (inherited from `DataType`). Defining
`__eq__` alone
@@ -2083,7 +2120,7 @@ impl PyStructField {
this.field.name.clone(),
data_type_to_py(py, &this.field.data_type)?,
this.field.nullable,
- metadata_map_to_py(py, &this.field.metadata)?,
+ this.metadata.clone_ref(py),
)
.into_pyobject(py)?;
Ok((slf.get_type(), args))
@@ -2092,11 +2129,13 @@ impl PyStructField {
#[pyclass(name = "StructType", module = "pyspark.sql.types", extends =
PyDataType)]
pub struct PyStructType {
+ /// The core fields. Where a field has its `StructField` object, its
metadata may have been
+ /// edited since, so the metadata that counts is `current_fields`.
pub(crate) fields: Vec<StructField>,
/// The `StructField` objects of `fields`, by their positions: those that
the struct was
/// made of, and the others once they are made. So a field is the same
object each time,
- /// as PySpark's `StructType`, which holds its fields, gives them. A
`StructField` cannot
- /// change, so its object is always that of its field.
+ /// as PySpark's `StructType`, which holds its fields, gives them. The
name and the type of
+ /// a field cannot change, so only its metadata can differ from `fields`.
objects: std::sync::Mutex<Vec<Option<Py<PyStructField>>>>,
}
@@ -2112,9 +2151,7 @@ impl PyStructType {
}
let object = Py::new(
py,
- PyStructField {
- field: self.fields[index].clone(),
- },
+ PyStructField::from_field(py, self.fields[index].clone())?,
)?;
objects[index] = Some(object.clone_ref(py));
Ok(object)
@@ -2126,6 +2163,29 @@ impl PyStructType {
.map(|index| self.field_object(py, index))
.collect()
}
+
+ /// The core fields with their metadata as it is now: a field that has its
object takes its
+ /// metadata from the object, where an edit in place is kept, as PySpark's
fields do. Those
+ /// without an object are as they are in `fields`.
+ pub(crate) fn current_fields(&self, py: Python<'_>) ->
PyResult<Vec<StructField>> {
+ let objects: Vec<Option<Py<PyStructField>>> = {
+ let objects = self.objects.lock().unwrap_or_else(|e|
e.into_inner());
+ objects
+ .iter()
+ .map(|object| object.as_ref().map(|object|
object.clone_ref(py)))
+ .collect()
+ };
+ self.fields
+ .iter()
+ .enumerate()
+ .map(
+ |(index, field)| match
objects.get(index).and_then(Option::as_ref) {
+ Some(object) => object.bind(py).borrow().current_field(py),
+ None => Ok(field.clone()),
+ },
+ )
+ .collect()
+ }
}
#[pymethods]
@@ -2165,7 +2225,7 @@ impl PyStructType {
let py = slf.py();
let this = slf.borrow();
let dt = DataType::Struct {
- fields: this.fields.clone(),
+ fields: this.current_fields(py)?,
};
reduce_with_udts(slf.get_type(), &dt, || {
(this.field_objects(py)?,).into_pyobject(py)
@@ -2181,7 +2241,7 @@ impl PyStructType {
if let Some(fs) = fields {
for f in fs {
if let Ok(sf) = f.cast::<PyStructField>() {
- out.push(sf.borrow().field.clone());
+ out.push(sf.borrow().current_field(f.py())?);
objects.push(Some(sf.clone().unbind()));
} else if let Ok(s) = f.extract::<String>() {
// "name type" DDL fragment.
@@ -2227,7 +2287,7 @@ impl PyStructType {
let mut object = None;
let new_field = if let Ok(sf) = field.cast::<PyStructField>() {
object = Some(sf.clone().unbind());
- sf.borrow().field.clone()
+ sf.borrow().current_field(field.py())?
} else {
// As PySpark's: a name takes a data type, of which a str is the
JSON value of one.
let Some(data_type) = data_type else {
@@ -2422,7 +2482,7 @@ impl PyStructType {
#[pyo3(name = "toNullable")]
fn __st_to_nullable(&self, py: Python<'_>) -> PyResult<Py<PyStructType>> {
let fields = match (DataType::Struct {
- fields: self.fields.clone(),
+ fields: self.current_fields(py)?,
})
.to_nullable()
{
@@ -3716,7 +3776,7 @@ pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>) ->
PyResult<DataType> {
}
if let Ok(st) = obj.extract::<PyRef<PyStructType>>() {
return Ok(DataType::Struct {
- fields: st.fields.clone(),
+ fields: st.current_fields(obj.py())?,
});
}
if let Ok(c) = obj.extract::<PyRef<PyCharType>>() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index af4ab5a1..f3f383e8 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2873,6 +2873,19 @@ assert
df.select(F.to_json(F.struct("p"))).collect()[1][0] == '{"p":[10.0,20.0]}
""", remote], check=True)
+def test_create_dataframe_schema_keeps_metadata_edited_in_place(session):
+ # As PySpark: the schema of a DataFrame is the StructType it is created
with, so a metadata
+ # edited in place of one of its fields is in df.schema.
+ from pyspark.sql import types as T
+
+ st = T.StructType([T.StructField("a", T.IntegerType()), T.StructField("b",
T.StringType())])
+ st["a"].metadata["comment"] = "the a"
+ df = session.createDataFrame([(1, "x")], st)
+ assert df.schema["a"].metadata == {"comment": "the a"}
+ assert df.schema.jsonValue()["fields"][0]["metadata"] == {"comment": "the
a"}
+ assert df.collect() == [Row(a=1, b="x")]
+
+
def
test_collected_values_of_user_defined_types_are_deserialized_as_pyspark(session):
# As official PySpark 4.3.0: a collected value of a UDT is its
deserialized object, and
# dtypes names the UDT by its simpleString(), in every way that rows are
collected.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 030892f7..c83fdb00 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6044,6 +6044,47 @@ def test_udt_not_implemented_defaults():
udt.deserialize(1)
+def test_struct_field_metadata_is_the_stored_dict():
+ # As PySpark's StructField, which keeps its metadata dict: an edit in
place shows in the
+ # field, its JSON, its equality and its pickle, and in the StructType that
holds the field.
+ f = _SF("a", _IT())
+ f.metadata["comment"] = "x"
+ assert f.metadata == {"comment": "x"} and f.metadata is f.metadata
+ assert f.jsonValue()["metadata"] == {"comment": "x"}
+ assert json.loads(f.json())["metadata"] == {"comment": "x"}
+ assert f == _SF("a", _IT(), True, {"comment": "x"})
+ assert pickle.loads(pickle.dumps(f)).metadata == {"comment": "x"}
+ # A non-empty dict passed in is stored as is; an empty or no dict gives a
new one.
+ d = {"k": 1}
+ g = _SF("a", _IT(), True, d)
+ assert g.metadata is d
+ d["k"] = 2
+ assert g.jsonValue()["metadata"] == {"k": 2}
+ empty = {}
+ h = _SF("a", _IT(), True, empty)
+ assert h.metadata == {} and h.metadata is not empty
+ a, b = _SF("a", _IT()), _SF("a", _IT())
+ assert a.metadata is not b.metadata
+ # The edits show in the StructType that holds the field, whether the
struct is made of
+ # the field, the field is added, or the field is taken from the struct.
+ f2 = _SF("b", _IT())
+ st = _StT([f2])
+ f2.metadata["k"] = "v"
+ assert st.jsonValue()["fields"][0]["metadata"] == {"k": "v"}
+ st2 = _StT()
+ f3 = _SF("c", _IT())
+ st2.add(f3)
+ f3.metadata["k"] = "w"
+ assert st2.jsonValue()["fields"][0]["metadata"] == {"k": "w"}
+ st3 = _StT([_SF("a", _IT()), _SF("b", _AT(_IT()))])
+ st3["a"].metadata["comment"] = "x"
+ assert st3.jsonValue()["fields"][0]["metadata"] == {"comment": "x"}
+ assert st3["a"].metadata is st3.fields[0].metadata
+ assert json.loads(st3.json())["fields"][0]["metadata"] == {"comment": "x"}
+ assert _StT.fromJson(st3.jsonValue()) == st3
+ assert pickle.loads(pickle.dumps(st3)).jsonValue() == st3.jsonValue()
+
+
def test_struct_field_metadata_any_values():
# StructField metadata accepts arbitrary JSON values (Dict[str, Any]), not
just str,
# and round-trips through json.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]