This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 2b95591  [SPARK-61074][CONNECT][PYTHON] Write and read the collations 
of the strings of a struct field in its metadata, and order the keys of 
jsonValue, in the Python client as PySpark does
2b95591 is described below

commit 2b9559113aa073e9204e11cfb0b10e2f904c8e13
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 14:21:31 2026 +0900

    [SPARK-61074][CONNECT][PYTHON] Write and read the collations of the strings 
of a struct field in its metadata, and order the keys of jsonValue, in the 
Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61074](https://issues.apache.org/jira/browse/SPARK-61074): the JSON 
of a struct field now holds the collations of its strings in its metadata, and 
the `fromJson` methods read them back, as PySpark's do.
    
    As Spark and PySpark do:
    - `StructType.json()` and `jsonValue()` write the collations of the strings 
of a field, also those in its arrays and maps, in the `__COLLATIONS` of its 
metadata. Each is `<provider>.<collation>`, keyed by its path: `s`, 
`a.element`, `m.key`, `m.value.element`. The type of the field is written 
without them, for example `"string"`.
    - `StructField.fromJson`, `StructType.fromJson` and 
`_parse_datatype_json_value` read them back. So do `ArrayType.fromJson` and 
`MapType.fromJson`, which use their `fieldPath` and `collationsMap`. They raise 
`COLLATION_INVALID_PROVIDER` for a provider that is neither `spark` nor `icu`. 
They raise `INVALID_JSON_DATA_TYPE_FOR_COLLATIONS` for a collation of a field 
that is not a string.
    - The dicts of `jsonValue()` have their keys in PySpark's order: `name`, 
`type`, `nullable`, `metadata` for a field, and `type` first for a nested type.
    
    The `fromJson` methods now call the helpers of the shim 
`pyspark.sql.types`, which follow PySpark's code. So they also read the struct 
of a user-defined type. The core reads the collations of the JSON of a struct 
field, and the JSON of a UDT, which it can write but could not read back.
    
    ### Why are the changes needed?
    
    The client wrote `"string collate UTF8_LCASE"` as the type of a field, so 
the JSON of a schema differed from PySpark's:
    
    ```
    {"fields":[{"metadata":{},"name":"s","nullable":false,"type":"string 
collate UTF8_LCASE"}],"type":"struct"}
    ```
    
    In addition:
    - `ArrayType.fromJson` and `MapType.fromJson` ignored their `collationsMap`.
    - `StructType.fromJson` and `StructField.fromJson` raised 
`INVALID_DATATYPE_FORMAT` for a field of a user-defined type.
    - `jsonValue()` sorted its keys, so `json.dumps(schema.jsonValue())` 
differed from PySpark's.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The JSON of the types, and its readers, are those of PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test checks the JSON of the schema of collated columns from 
the server, and reads it back with `StructType.fromJson`. It passes with 
official PySpark 4.3.0 and fails without this change.
    - A new offline test checks the JSON and the `jsonValue` of a struct of 
collated strings, also in arrays, maps and nested structs, against PySpark's. 
It reads the struct back with each reader, round-trips a struct of a 
user-defined type, and checks the errors.
    - A new lib test checks the JSON of collated strings of the core, its 
errors, and the JSON of a UDT.
    - On 28 cases of the readers and their errors, and on the writers, the 
client gives the same results as official PySpark 4.3.0.
    - The lib tests of both crates (589 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (1979 passed, 8 skipped), and the 
live drop-in and Arrow suites (135 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #276 from HyukjinKwon/SPARK-61074-collation-json.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/types.rs                | 202 +++++++++-------
 crates/spark-connect/src/types.rs             | 325 ++++++++++++++++++++------
 crates/spark-connect/tests/types_roundtrip.rs |  86 +++++++
 python/pyspark/sql/types.py                   | 151 +++++++++---
 python/tests/test_dropin_e2e.py               |  19 ++
 python/tests/test_dropin_offline.py           |  66 ++++++
 6 files changed, 666 insertions(+), 183 deletions(-)

diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index a333443..eee200a 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -87,32 +87,11 @@ fn py_new_struct(py: Python<'_>, fields: Vec<StructField>) 
-> PyResult<Py<PyStru
     )
 }
 
-/// Parse a type-JSON dict (as passed to `<Type>.fromJson`) into a core 
`DataType`.
-/// The dict may omit the top-level `"type"` key (each `fromJson` is a 
classmethod on a
-/// specific type), so inject `default_type` when it is absent before handing 
the object
-/// to the core JSON parser.
-fn from_json_dict(
-    py: Python<'_>,
-    json: &Bound<'_, PyAny>,
-    default_type: &str,
-) -> PyResult<DataType> {
-    use pyo3::types::PyDict;
-    let value: Bound<'_, PyAny> = if let Ok(d) = json.cast::<PyDict>() {
-        let d = d.copy()?;
-        if !d.contains("type")? {
-            d.set_item("type", default_type)?;
-        }
-        d.into_any()
-    } else {
-        json.clone()
-    };
-    let s: String = py
-        .import("json")?
-        .getattr("dumps")?
-        .call1((value,))?
-        .extract()?;
-    DataType::from_json_str(&s)
-        .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError, 
_>(e.to_string()))
+/// The `fromJson` helper `name` of the shim `pyspark.sql.types`, which parses 
the nested JSON
+/// forms of the types as PySpark's: the collations of a struct field by their 
paths, and the
+/// Python class of a `udt`. The core's grammar parses the strings of the 
atomic types.
+fn shim_from_json<'py>(py: Python<'py>, name: &str) -> PyResult<Bound<'py, 
PyAny>> {
+    py.import("pyspark.sql.types")?.getattr(name)
 }
 
 /// Convert a Python `StructField` metadata dict to the core JSON-value map. 
Values may be any
@@ -131,6 +110,82 @@ fn py_metadata_to_map(
         .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError, 
_>(e.to_string()))
 }
 
+/// The keys of the JSON objects of the nested data types, in the order of 
PySpark's
+/// `jsonValue`: struct, array, map, and the UDTs of a Scala class or of a 
pickled Python class.
+const TYPE_JSON_KEYS: [&str; 11] = [
+    "type",
+    "fields",
+    "elementType",
+    "containsNull",
+    "keyType",
+    "valueType",
+    "valueContainsNull",
+    "class",
+    "pyClass",
+    "serializedClass",
+    "sqlType",
+];
+
+/// The keys of the JSON object of a struct field, in the order of PySpark's 
`jsonValue`.
+const FIELD_JSON_KEYS: [&str; 4] = ["name", "type", "nullable", "metadata"];
+
+/// The JSON text of the JSON value `v` of a data type (of a struct field, if 
`field`), with
+/// the keys of its objects in the order of PySpark's `jsonValue`. The core's 
JSON objects
+/// sort their keys, as `json()` does; the metadata of a field stays in that 
order.
+fn ordered_json_text(v: &serde_json::Value, field: bool) -> String {
+    let Some(object) = v.as_object() else {
+        return v.to_string();
+    };
+    let order: &[&str] = if field {
+        &FIELD_JSON_KEYS
+    } else {
+        &TYPE_JSON_KEYS
+    };
+    let keys = order
+        .iter()
+        .copied()
+        .filter(|key| object.contains_key(*key))
+        .chain(
+            object
+                .keys()
+                .map(String::as_str)
+                .filter(|key| !order.contains(key)),
+        );
+    let entries: Vec<String> = keys
+        .map(|key| {
+            let value = &object[key];
+            let text = match (field, key) {
+                (true, "type") | (false, "elementType" | "keyType" | 
"valueType" | "sqlType") => {
+                    ordered_json_text(value, false)
+                }
+                (false, "fields") => match value.as_array() {
+                    Some(fields) => {
+                        let fields: Vec<String> =
+                            fields.iter().map(|f| ordered_json_text(f, 
true)).collect();
+                        format!("[{}]", fields.join(","))
+                    }
+                    None => value.to_string(),
+                },
+                _ => value.to_string(),
+            };
+            format!("{}:{}", serde_json::Value::from(key), text)
+        })
+        .collect();
+    format!("{{{}}}", entries.join(","))
+}
+
+/// The `jsonValue` of the JSON value `v` of a data type (of a struct field, 
if `field`): its
+/// dicts have the keys in the order of PySpark's, as 
`json.dumps(t.jsonValue())` shows.
+fn py_json_value<'py>(
+    py: Python<'py>,
+    v: &serde_json::Value,
+    field: bool,
+) -> PyResult<Bound<'py, PyAny>> {
+    py.import("json")?
+        .getattr("loads")?
+        .call1((ordered_json_text(v, field),))
+}
+
 /// Convert the core JSON-value metadata map back to a Python dict.
 fn metadata_map_to_py<'py>(
     py: Python<'py>,
@@ -182,7 +237,7 @@ impl PyDataType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -311,7 +366,7 @@ impl PyNullType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -361,7 +416,7 @@ impl PyBooleanType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -411,7 +466,7 @@ impl PyByteType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -465,7 +520,7 @@ impl PyShortType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -519,7 +574,7 @@ impl PyIntegerType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -573,7 +628,7 @@ impl PyLongType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -627,7 +682,7 @@ impl PyFloatType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -681,7 +736,7 @@ impl PyDoubleType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -738,7 +793,7 @@ impl PyDecimalType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -821,7 +876,7 @@ impl PyStringType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -908,7 +963,7 @@ impl PyBinaryType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -961,7 +1016,7 @@ impl PyDateType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1011,7 +1066,7 @@ impl PyTimestampType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1065,7 +1120,7 @@ impl PyTimestampNTZType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1133,7 +1188,7 @@ impl PyArrayType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1220,9 +1275,9 @@ impl PyArrayType {
         fieldPath: String,
         collationsMap: Option<Bound<'_, PyAny>>,
     ) -> PyResult<Py<PyAny>> {
-        let _ = (fieldPath, collationsMap);
-        let dt = from_json_dict(py, json, "array")?;
-        data_type_to_py(py, &dt)
+        Ok(shim_from_json(py, "_array_type_from_json")?
+            .call1((json, fieldPath, collationsMap))?
+            .unbind())
     }
 }
 
@@ -1262,7 +1317,7 @@ impl PyMapType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1367,9 +1422,9 @@ impl PyMapType {
         fieldPath: String,
         collationsMap: Option<Bound<'_, PyAny>>,
     ) -> PyResult<Py<PyAny>> {
-        let _ = (fieldPath, collationsMap);
-        let dt = from_json_dict(py, json, "map")?;
-        data_type_to_py(py, &dt)
+        Ok(shim_from_json(py, "_map_type_from_json")?
+            .call1((json, fieldPath, collationsMap))?
+            .unbind())
     }
 }
 
@@ -1405,7 +1460,7 @@ impl PyStructField {
     #[pyo3(name = "jsonValue")]
     fn __sf_json_value<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, 
PyAny>> {
         let v = self.field.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, true)
     }
     #[pyo3(name = "needConversion")]
     fn __sf_need_conversion(&self) -> bool {
@@ -1504,15 +1559,10 @@ impl PyStructField {
         _cls: &Bound<'_, pyo3::types::PyType>,
         py: Python<'_>,
         json: &Bound<'_, PyAny>,
-    ) -> PyResult<PyStructField> {
-        let s: String = py
-            .import("json")?
-            .getattr("dumps")?
+    ) -> PyResult<Py<PyAny>> {
+        Ok(shim_from_json(py, "_struct_field_from_json")?
             .call1((json,))?
-            .extract()?;
-        let field = StructField::from_json_str(&s)
-            .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError, 
_>(e.to_string()))?;
-        Ok(PyStructField { field })
+            .unbind())
     }
     #[new]
     #[pyo3(signature = (name, dataType, nullable=true, metadata=None))]
@@ -1598,7 +1648,7 @@ impl PyStructType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1868,20 +1918,10 @@ impl PyStructType {
         _cls: &Bound<'_, pyo3::types::PyType>,
         py: Python<'_>,
         json: &Bound<'_, PyAny>,
-    ) -> PyResult<Py<PyStructType>> {
-        let s: String = py
-            .import("json")?
-            .getattr("dumps")?
+    ) -> PyResult<Py<PyAny>> {
+        Ok(shim_from_json(py, "_struct_type_from_json")?
             .call1((json,))?
-            .extract()?;
-        match DataType::from_json_str(&s)
-            .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError, 
_>(e.to_string()))?
-        {
-            DataType::Struct { fields } => py_new_struct(py, fields),
-            _ => Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
-                "fromJson did not produce a StructType",
-            )),
-        }
+            .unbind())
     }
 
     fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
@@ -1921,7 +1961,7 @@ impl PyCharType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1981,7 +2021,7 @@ impl PyVarcharType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2041,7 +2081,7 @@ impl PyTimeType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2105,7 +2145,7 @@ impl PyTimestampNTZNanosType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2178,7 +2218,7 @@ impl PyTimestampLTZNanosType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2247,7 +2287,7 @@ impl PyCalendarIntervalType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2304,7 +2344,7 @@ impl PyYearMonthIntervalType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2384,7 +2424,7 @@ impl PyDayTimeIntervalType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2453,7 +2493,7 @@ impl PyVariantType {
         py: Python<'py>,
     ) -> PyResult<Bound<'py, PyAny>> {
         let v = py_to_data_type(slf.as_any())?.json_value();
-        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+        py_json_value(py, &v, false)
     }
     #[pyo3(name = "needConversion")]
     fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
diff --git a/crates/spark-connect/src/types.rs 
b/crates/spark-connect/src/types.rs
index 043b951..4a8d4f6 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -7,7 +7,7 @@ use std::collections::BTreeMap;
 use std::fmt::{self, Write};
 use std::hash::{Hash, Hasher};
 
-use spark_connect_core::error::{Result, SparkError};
+use spark_connect_core::error::{Result, SparkError, SparkErrorKind};
 
 /// The base DataType representation, mirroring `pyspark.sql.types.DataType`.
 ///
@@ -477,7 +477,7 @@ impl DataType {
 
     /// Parses a JSON value into a DataType, mirroring the reverse of `json()` 
/ `jsonValue()`.
     pub fn from_json(value: &serde_json::Value) -> Result<DataType> {
-        parse_json_value(value, None)
+        parse_json_value(value, "", &CollationsMap::new())
     }
 
     /// Parses a JSON string into a DataType (convenience over [`from_json`]).
@@ -855,13 +855,27 @@ impl StructField {
         format!("{}:{}", self.name, self.data_type.simple_string())
     }
 
-    /// Returns the JSON value representation, mirroring 
`StructField.jsonValue()`.
+    /// Returns the JSON value representation, mirroring 
`StructField.jsonValue()`: the
+    /// collations of the strings of the field, also in its arrays and maps, 
are in the
+    /// `__COLLATIONS` of its metadata, by their paths, and not in its type, 
as Spark writes them.
     pub fn json_value(&self) -> serde_json::Value {
+        let mut collations = serde_json::Map::new();
+        visit_collations(&self.data_type, &self.name, &mut collations);
+        let (data_type, metadata) = if collations.is_empty() {
+            (self.data_type.json_value(), self.metadata.clone())
+        } else {
+            let mut metadata = self.metadata.clone();
+            metadata.insert(
+                COLLATIONS_METADATA_KEY.to_string(),
+                serde_json::Value::Object(collations),
+            );
+            (without_collations(&self.data_type).json_value(), metadata)
+        };
         serde_json::json!({
             "name": &self.name,
-            "type": self.data_type.json_value(),
+            "type": data_type,
             "nullable": self.nullable,
-            "metadata": self.metadata,
+            "metadata": metadata,
         })
     }
 
@@ -869,25 +883,10 @@ impl StructField {
     pub fn from_json_str(s: &str) -> Result<StructField> {
         let v: serde_json::Value = serde_json::from_str(s)
             .map_err(|e| SparkError::value("INVALID_JSON", &[("detail", 
&e.to_string())]))?;
-        let name = v
-            .get("name")
-            .and_then(|x| x.as_str())
-            .ok_or_else(|| SparkError::value("INVALID_JSON", &[("detail", 
"missing field name")]))?
-            .to_string();
-        let nullable = v.get("nullable").and_then(|x| 
x.as_bool()).unwrap_or(true);
-        let data_type = 
DataType::from_json(v.get("type").unwrap_or(&serde_json::Value::Null))?;
-        let mut metadata = BTreeMap::new();
-        if let Some(md) = v.get("metadata").and_then(|x| x.as_object()) {
-            for (k, val) in md {
-                metadata.insert(k.clone(), val.clone());
-            }
-        }
-        Ok(StructField {
-            name,
-            data_type,
-            nullable,
-            metadata,
-        })
+        let field = v.as_object().ok_or_else(|| {
+            SparkError::value("INVALID_JSON", &[("detail", "a field is not an 
object")])
+        })?;
+        parse_json_field(field)
     }
 }
 
@@ -1170,11 +1169,201 @@ fn interval_string(start_field: i32, end_field: i32, 
fields: &[(&str, i32)]) ->
     }
 }
 
-/// Parses a JSON value into a DataType
-fn parse_json_value(value: &serde_json::Value, _field_name: Option<&str>) -> 
Result<DataType> {
+/// The key of the metadata of a struct field of the collations of its 
strings, as Spark's.
+const COLLATIONS_METADATA_KEY: &str = "__COLLATIONS";
+
+/// The collations of the strings of a field of `data_type` at `path`, also in 
its arrays and
+/// maps but not in its structs, whose fields have their own, as PySpark's
+/// `StructField.getCollationMetadata`: `<provider>.<collation>` of each path.
+fn visit_collations(
+    data_type: &DataType,
+    path: &str,
+    collations: &mut serde_json::Map<String, serde_json::Value>,
+) {
+    match data_type {
+        DataType::Array { element_type, .. } => {
+            visit_collations(element_type, &format!("{path}.element"), 
collations)
+        }
+        DataType::Map {
+            key_type,
+            value_type,
+            ..
+        } => {
+            visit_collations(key_type, &format!("{path}.key"), collations);
+            visit_collations(value_type, &format!("{path}.value"), collations);
+        }
+        DataType::String { collation } if collation != "UTF8_BINARY" => {
+            // As PySpark's `StringType.collationProvider`.
+            let provider = if collation.starts_with("UTF8") {
+                "spark"
+            } else {
+                "icu"
+            };
+            collations.insert(path.to_string(), 
format!("{provider}.{collation}").into());
+        }
+        _ => {}
+    }
+}
+
+/// `data_type` without the collations of its strings, also in its arrays and 
maps, which the
+/// JSON of a struct field keeps in its metadata.
+fn without_collations(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Array {
+            element_type,
+            contains_null,
+        } => DataType::Array {
+            element_type: Box::new(without_collations(element_type)),
+            contains_null: *contains_null,
+        },
+        DataType::Map {
+            key_type,
+            value_type,
+            value_contains_null,
+        } => DataType::Map {
+            key_type: Box::new(without_collations(key_type)),
+            value_type: Box::new(without_collations(value_type)),
+            value_contains_null: *value_contains_null,
+        },
+        DataType::String { .. } => DataType::String {
+            collation: "UTF8_BINARY".to_string(),
+        },
+        other => other.clone(),
+    }
+}
+
+/// The collations of the strings of a struct field by their paths, 
`__COLLATIONS` of its
+/// metadata without their providers, as the `collationsMap` of PySpark's 
`fromJson`.
+type CollationsMap = std::collections::HashMap<String, String>;
+
+/// The JSON type names to which a collation of a struct field applies, as 
PySpark's
+/// `_all_mappable_types`: it makes a `string` a collated string, and raises
+/// `INVALID_JSON_DATA_TYPE_FOR_COLLATIONS` for the others.
+const MAPPABLE_TYPE_NAMES: [&str; 16] = [
+    "string",
+    "binary",
+    "boolean",
+    "decimal",
+    "float",
+    "double",
+    "byte",
+    "short",
+    "integer",
+    "long",
+    "date",
+    "timestamp",
+    "timestamp_ntz",
+    "void",
+    "variant",
+    "interval",
+];
+
+/// The path of `child` (`element`, `key` or `value`) of a type at `path`, as 
PySpark's
+/// `fromJson` of arrays and maps names it.
+fn child_path(path: &str, child: &str) -> String {
+    if path.is_empty() {
+        child.to_string()
+    } else {
+        format!("{path}.{child}")
+    }
+}
+
+/// The collations of the `__COLLATIONS` of `metadata`, which this removes, as 
PySpark's
+/// `StructField.fromJson`: each is `<provider>.<collation>`, of the `spark` 
or `icu` provider.
+fn take_field_collations(
+    metadata: &mut BTreeMap<String, serde_json::Value>,
+) -> Result<CollationsMap> {
+    let Some(collations) = metadata.remove(COLLATIONS_METADATA_KEY) else {
+        return Ok(CollationsMap::new());
+    };
+    let mut by_path = CollationsMap::new();
+    for (path, value) in collations.as_object().into_iter().flatten() {
+        let parts: Vec<&str> = 
value.as_str().unwrap_or_default().split('.').collect();
+        let [provider, collation] = parts[..] else {
+            return Err(SparkError::value(
+                "INVALID_DATATYPE_FORMAT",
+                &[("detail", "a collation is not of a provider and a name")],
+            ));
+        };
+        if !["spark", "icu"].contains(&provider.to_lowercase().as_str()) {
+            return Err(SparkError::value(
+                "COLLATION_INVALID_PROVIDER",
+                &[("provider", provider), ("supportedProviders", "spark, 
icu")],
+            ));
+        }
+        by_path.insert(path.clone(), collation.to_string());
+    }
+    Ok(by_path)
+}
+
+/// Parses the JSON object of a struct field, mirroring 
`StructField.fromJson`: the strings at
+/// the paths of the `__COLLATIONS` of its metadata are of their collations.
+fn parse_json_field(field: &serde_json::Map<String, serde_json::Value>) -> 
Result<StructField> {
+    let name = field
+        .get("name")
+        .and_then(|v| v.as_str())
+        .ok_or_else(|| {
+            SparkError::value(
+                "INVALID_DATATYPE_FORMAT",
+                &[("detail", "StructField missing name")],
+            )
+        })?
+        .to_string();
+    let field_type = field.get("type").ok_or_else(|| {
+        SparkError::value(
+            "INVALID_DATATYPE_FORMAT",
+            &[("detail", "StructField missing type")],
+        )
+    })?;
+    let nullable = field
+        .get("nullable")
+        .and_then(|v| v.as_bool())
+        .unwrap_or(true);
+    let mut metadata: BTreeMap<String, serde_json::Value> = field
+        .get("metadata")
+        .and_then(|v| v.as_object())
+        .map(|m| m.iter().map(|(k, v)| (k.clone(), v.clone())).collect())
+        .unwrap_or_default();
+    let collations = take_field_collations(&mut metadata)?;
+    Ok(StructField {
+        data_type: parse_json_value(field_type, &name, &collations)?,
+        name,
+        nullable,
+        metadata,
+    })
+}
+
+/// Raises, as PySpark's `_assert_valid_type_for_collation`, for a collation 
of a field path of
+/// a type that is not a string.
+fn check_collated_type(path: &str, json_type: &str, collations: 
&CollationsMap) -> Result<()> {
+    if collations.contains_key(path) && json_type != "string" {
+        return Err(SparkError::classed(
+            SparkErrorKind::TypeError,
+            "INVALID_JSON_DATA_TYPE_FOR_COLLATIONS",
+            &[("jsonType", json_type)],
+        ));
+    }
+    Ok(())
+}
+
+/// Parses a JSON value into a DataType at `field_path` of a struct field, 
mirroring PySpark's
+/// `_parse_datatype_json_value`: a string of a path of `collations` is of its 
collation.
+fn parse_json_value(
+    value: &serde_json::Value,
+    field_path: &str,
+    collations: &CollationsMap,
+) -> Result<DataType> {
     match value {
-        serde_json::Value::String(s) => parse_json_type_string(s),
-        serde_json::Value::Object(obj) => parse_json_object(obj),
+        serde_json::Value::String(s) => match collations.get(field_path) {
+            Some(collation) if MAPPABLE_TYPE_NAMES.contains(&s.as_str()) => {
+                check_collated_type(field_path, s, collations)?;
+                Ok(DataType::String {
+                    collation: collation.clone(),
+                })
+            }
+            _ => parse_json_type_string(s),
+        },
+        serde_json::Value::Object(obj) => parse_json_object(obj, field_path, 
collations),
         _ => Err(SparkError::value(
             "INVALID_DATATYPE_FORMAT",
             &[("detail", "Expected string or object for DataType")],
@@ -1235,8 +1424,15 @@ fn parse_json_type_string(s: &str) -> Result<DataType> {
     }
 }
 
-fn parse_json_object(obj: &serde_json::Map<String, serde_json::Value>) -> 
Result<DataType> {
+fn parse_json_object(
+    obj: &serde_json::Map<String, serde_json::Value>,
+    field_path: &str,
+    collations: &CollationsMap,
+) -> Result<DataType> {
     let type_field = obj.get("type").and_then(|v| v.as_str());
+    if let Some(json_type @ ("array" | "map" | "struct")) = type_field {
+        check_collated_type(field_path, json_type, collations)?;
+    }
 
     match type_field {
         Some("array") => {
@@ -1251,7 +1447,11 @@ fn parse_json_object(obj: &serde_json::Map<String, 
serde_json::Value>) -> Result
                 .and_then(|v| v.as_bool())
                 .unwrap_or(true);
             Ok(DataType::Array {
-                element_type: Box::new(parse_json_value(element_type, None)?),
+                element_type: Box::new(parse_json_value(
+                    element_type,
+                    &child_path(field_path, "element"),
+                    collations,
+                )?),
                 contains_null,
             })
         }
@@ -1273,8 +1473,16 @@ fn parse_json_object(obj: &serde_json::Map<String, 
serde_json::Value>) -> Result
                 .and_then(|v| v.as_bool())
                 .unwrap_or(true);
             Ok(DataType::Map {
-                key_type: Box::new(parse_json_value(key_type, None)?),
-                value_type: Box::new(parse_json_value(value_type, None)?),
+                key_type: Box::new(parse_json_value(
+                    key_type,
+                    &child_path(field_path, "key"),
+                    collations,
+                )?),
+                value_type: Box::new(parse_json_value(
+                    value_type,
+                    &child_path(field_path, "value"),
+                    collations,
+                )?),
                 value_contains_null,
             })
         }
@@ -1290,45 +1498,26 @@ fn parse_json_object(obj: &serde_json::Map<String, 
serde_json::Value>) -> Result
                 })?;
 
             let mut fields = Vec::new();
-            for field_obj in fields_arr {
-                if let Some(field_map) = field_obj.as_object() {
-                    let name = field_map
-                        .get("name")
-                        .and_then(|v| v.as_str())
-                        .ok_or_else(|| {
-                            SparkError::value(
-                                "INVALID_DATATYPE_FORMAT",
-                                &[("detail", "StructField missing name")],
-                            )
-                        })?
-                        .to_string();
-                    let field_type = field_map.get("type").ok_or_else(|| {
-                        SparkError::value(
-                            "INVALID_DATATYPE_FORMAT",
-                            &[("detail", "StructField missing type")],
-                        )
-                    })?;
-                    let data_type = parse_json_value(field_type, Some(&name))?;
-                    let nullable = field_map
-                        .get("nullable")
-                        .and_then(|v| v.as_bool())
-                        .unwrap_or(true);
-                    let metadata = field_map
-                        .get("metadata")
-                        .and_then(|v| v.as_object())
-                        .map(|m| m.iter().map(|(k, v)| (k.clone(), 
v.clone())).collect())
-                        .unwrap_or_default();
-
-                    fields.push(StructField {
-                        name,
-                        data_type,
-                        nullable,
-                        metadata,
-                    });
-                }
+            for field in fields_arr.iter().filter_map(|field| 
field.as_object()) {
+                fields.push(parse_json_field(field)?);
             }
             Ok(DataType::Struct { fields })
         }
+        // The inverse of `json_value` of a UDT: its classes and the JSON of 
its SQL type.
+        Some("udt") => {
+            let text = |key: &str| obj.get(key).and_then(|v| 
v.as_str()).map(str::to_string);
+            let sql_type = match obj.get("sqlType") {
+                Some(sql_type) => 
Some(Box::new(DataType::from_json(sql_type)?)),
+                None => None,
+            };
+            Ok(DataType::Udt {
+                type_str: "udt".to_string(),
+                jvm_class: text("class"),
+                python_class: text("pyClass"),
+                serialized_python_class: text("serializedClass"),
+                sql_type,
+            })
+        }
         _ => Err(SparkError::value(
             "INVALID_DATATYPE_FORMAT",
             &[("detail", "Unknown type in object")],
diff --git a/crates/spark-connect/tests/types_roundtrip.rs 
b/crates/spark-connect/tests/types_roundtrip.rs
index bffae04..d28a2ef 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -232,6 +232,92 @@ fn special_variant_arms() {
     }
 }
 
+/// A struct field writes the collations of its strings, also in its arrays 
and maps, in the
+/// `__COLLATIONS` of its metadata, as Spark and PySpark do, and reads them 
back; a UDT reads
+/// back its JSON too.
+#[test]
+fn collations_are_in_the_metadata_of_the_json_of_a_field_as_spark() {
+    let string = |collation: &str| DataType::String {
+        collation: collation.to_string(),
+    };
+    let field = |name: &str, data_type: DataType| StructField {
+        name: name.to_string(),
+        data_type,
+        nullable: true,
+        metadata: BTreeMap::new(),
+    };
+    let st = DataType::Struct {
+        fields: vec![
+            field("s", string("UTF8_LCASE")),
+            field(
+                "m",
+                DataType::Map {
+                    key_type: Box::new(string("UTF8_LCASE")),
+                    value_type: Box::new(DataType::Array {
+                        element_type: Box::new(string("UNICODE_CI")),
+                        contains_null: false,
+                    }),
+                    value_contains_null: true,
+                },
+            ),
+            field(
+                "n",
+                DataType::Struct {
+                    fields: vec![field("t", string("UTF8_LCASE"))],
+                },
+            ),
+        ],
+    };
+    assert_eq!(
+        st.json(),
+        concat!(
+            
r#"{"fields":[{"metadata":{"__COLLATIONS":{"s":"spark.UTF8_LCASE"}},"name":"s","#,
+            
r#""nullable":true,"type":"string"},{"metadata":{"__COLLATIONS":{"m.key":"#,
+            
r#""spark.UTF8_LCASE","m.value.element":"icu.UNICODE_CI"}},"name":"m","#,
+            
r#""nullable":true,"type":{"keyType":"string","type":"map","valueContainsNull":true,"#,
+            
r#""valueType":{"containsNull":false,"elementType":"string","type":"array"}}},"#,
+            
r#"{"metadata":{},"name":"n","nullable":true,"type":{"fields":[{"metadata":"#,
+            
r#"{"__COLLATIONS":{"t":"spark.UTF8_LCASE"}},"name":"t","nullable":true,"#,
+            r#""type":"string"}],"type":"struct"}}],"type":"struct"}"#
+        )
+    );
+    assert_eq!(DataType::from_json_str(&st.json()).unwrap(), st);
+    let DataType::Struct { fields } = &st else {
+        unreachable!()
+    };
+    assert_eq!(
+        
StructField::from_json_str(&fields[1].json_value().to_string()).unwrap(),
+        fields[1]
+    );
+    // The errors of PySpark's readers: a provider of no collation, and a 
collation of a field
+    // that is not a string.
+    let of = |type_json: &str, collation: &str| {
+        let json = format!(
+            
r#"{{"name":"f","type":{type_json},"nullable":true,"metadata":{{"__COLLATIONS":{{"f":"{collation}"}}}}}}"#
+        );
+        StructField::from_json_str(&json).unwrap_err()
+    };
+    let error = of(r#""string""#, "other.UTF8_LCASE");
+    assert_eq!(
+        error.get_error_class().as_deref(),
+        Some("COLLATION_INVALID_PROVIDER")
+    );
+    let error = of(r#""integer""#, "spark.UTF8_LCASE");
+    assert_eq!(
+        error.get_error_class().as_deref(),
+        Some("INVALID_JSON_DATA_TYPE_FOR_COLLATIONS")
+    );
+    // A UDT reads back the JSON that it writes.
+    let udt = DataType::Udt {
+        type_str: "udt".to_string(),
+        jvm_class: None,
+        python_class: Some("mymod.MyUDT".to_string()),
+        serialized_python_class: Some("<pickle>".to_string()),
+        sql_type: Some(Box::new(DataType::Integer)),
+    };
+    assert_eq!(DataType::from_json_str(&udt.json()).unwrap(), udt);
+}
+
 /// Exercise the DDL parser across the full grammar it accepts. Calling 
`from_ddl` on
 /// each form pins the large `parse_datatype_string` body; a few are asserted 
to the
 /// expected type.
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 062dfc6..d7800c1 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -306,53 +306,136 @@ def _parse_datatype_json_string(json_string):
     return _parse_datatype_json_value(_json.loads(json_string))
 
 
-def _parse_datatype_json_value(v):
+#: The key of the metadata of a struct field of the collations of its strings, 
as PySpark's.
+_COLLATIONS_METADATA_KEY = "__COLLATIONS"
+
+#: The JSON type names to which a collation of a struct field applies, as 
PySpark's
+#: ``_all_mappable_types``: it makes a ``string`` a collated string, and raises
+#: ``INVALID_JSON_DATA_TYPE_FOR_COLLATIONS`` for the others.
+_MAPPABLE_TYPE_NAMES = frozenset(
+    [
+        "string", "binary", "boolean", "decimal", "float", "double", "byte", 
"short",
+        "integer", "long", "date", "timestamp", "timestamp_ntz", "void", 
"variant", "interval",
+    ]
+)
+
+
+def _parse_datatype_json_value(json_value, fieldPath="", collationsMap=None):
     """Recursively reconstruct a DataType from a parsed Spark-JSON value (a 
string for
     atomic/parameterized types, or a dict for the nested struct/array/map 
forms).
 
     The string forms (atomics, ``decimal(p,s)``, ``string collate X``, 
intervals,
     ``geometry(<crs>)``, ``geography(<crs>, <alg>)``, ...) are parsed by the 
Rust core's
     JSON grammar, so the drop-in keeps one grammar. The nested object forms 
recurse here,
-    because a ``udt`` leaf has to reconstruct its Python class."""
+    because a ``udt`` leaf has to reconstruct its Python class.
+
+    As PySpark's, a string at a ``fieldPath`` of ``collationsMap``, the 
collations of the
+    ``__COLLATIONS`` of the metadata of a struct field by their paths, is of 
its collation."""
     import json as _json
 
-    if isinstance(v, str):
+    if not isinstance(json_value, dict):
+        if (
+            collationsMap is not None
+            and fieldPath in collationsMap
+            and json_value in _MAPPABLE_TYPE_NAMES
+        ):
+            _assert_valid_type_for_collation(fieldPath, json_value, 
collationsMap)
+            return StringType(collationsMap[fieldPath])
         from pyspark._pyspark import parse_datatype_json
 
         try:
-            return parse_datatype_json(_json.dumps(v))
+            return parse_datatype_json(_json.dumps(json_value))
         except ValueError:
             # As PySpark's, of a str that is not the JSON value of a data type.
-            raise _pyspark_value_error("CANNOT_PARSE_DATATYPE", {"msg": 
str(v)}) from None
-    elif isinstance(v, dict):
-        t = v.get("type")
-        if t == "struct":
-            fields = [
-                StructField(
-                    f["name"],
-                    _parse_datatype_json_value(f["type"]),
-                    f.get("nullable", True),
-                    f.get("metadata", {}) or {},
-                )
-                for f in v["fields"]
-            ]
-            return StructType(fields)
-        elif t == "array":
-            return ArrayType(
-                _parse_datatype_json_value(v["elementType"]),
-                v.get("containsNull", True),
-            )
-        elif t == "map":
-            return MapType(
-                _parse_datatype_json_value(v["keyType"]),
-                _parse_datatype_json_value(v["valueType"]),
-                v.get("valueContainsNull", True),
-            )
-        elif t == "udt":
-            return UserDefinedType.fromJson(v)
-        # As PySpark's, of an object of a type that is not one of a data type.
-        raise _pyspark_value_error("UNSUPPORTED_DATA_TYPE", {"data_type": 
str(t)})
-    raise _pyspark_value_error("CANNOT_PARSE_DATATYPE", {"msg": str(v)})
+            raise _pyspark_value_error(
+                "CANNOT_PARSE_DATATYPE", {"msg": str(json_value)}
+            ) from None
+    tpe = json_value["type"]
+    if tpe in ("array", "map", "struct"):
+        if collationsMap is not None and fieldPath in collationsMap:
+            _assert_valid_type_for_collation(fieldPath, tpe, collationsMap)
+        if tpe == "array":
+            return _array_type_from_json(json_value, fieldPath, collationsMap)
+        elif tpe == "map":
+            return _map_type_from_json(json_value, fieldPath, collationsMap)
+        return _struct_type_from_json(json_value)
+    elif tpe == "udt":
+        return UserDefinedType.fromJson(json_value)
+    # As PySpark's, of an object of a type that is not one of a data type.
+    raise _pyspark_value_error("UNSUPPORTED_DATA_TYPE", {"data_type": 
str(tpe)})
+
+
+def _array_type_from_json(json_value, fieldPath="", collationsMap=None):
+    """``ArrayType.fromJson``: its element at ``element`` of ``fieldPath``, as 
PySpark's."""
+    elementType = _parse_datatype_json_value(
+        json_value["elementType"],
+        "element" if fieldPath == "" else fieldPath + ".element",
+        collationsMap,
+    )
+    return ArrayType(elementType, json_value["containsNull"])
+
+
+def _map_type_from_json(json_value, fieldPath="", collationsMap=None):
+    """``MapType.fromJson``: its key and value at ``key`` and ``value`` of 
``fieldPath``, as
+    PySpark's."""
+    keyType = _parse_datatype_json_value(
+        json_value["keyType"], "key" if fieldPath == "" else fieldPath + 
".key", collationsMap
+    )
+    valueType = _parse_datatype_json_value(
+        json_value["valueType"],
+        "value" if fieldPath == "" else fieldPath + ".value",
+        collationsMap,
+    )
+    return MapType(keyType, valueType, json_value["valueContainsNull"])
+
+
+def _struct_field_from_json(json_value):
+    """``StructField.fromJson``, as PySpark's: the ``__COLLATIONS`` of its 
metadata,
+    ``<provider>.<collation>`` by the paths of its strings, are the collations 
of those
+    strings, and not metadata."""
+    metadata = json_value.get("metadata")
+    collationsMap = {}
+    if metadata and _COLLATIONS_METADATA_KEY in metadata:
+        for key, value in metadata[_COLLATIONS_METADATA_KEY].items():
+            nameParts = value.split(".")
+            assert len(nameParts) == 2
+            provider, name = nameParts[0], nameParts[1]
+            _assert_valid_collation_provider(provider)
+            collationsMap[key] = name
+        metadata = {
+            key: value for key, value in metadata.items() if key != 
_COLLATIONS_METADATA_KEY
+        }
+    return StructField(
+        json_value["name"],
+        _parse_datatype_json_value(json_value["type"], json_value["name"], 
collationsMap),
+        json_value.get("nullable", True),
+        metadata,
+    )
+
+
+def _struct_type_from_json(json_value):
+    """``StructType.fromJson``, as PySpark's."""
+    return StructType([_struct_field_from_json(f) for f in 
json_value["fields"]])
+
+
+def _assert_valid_type_for_collation(fieldPath, fieldType, collationMap):
+    """As PySpark's: a collation is of a string."""
+    if fieldPath in collationMap and fieldType != "string":
+        from pyspark.errors import PySparkTypeError
+
+        raise PySparkTypeError(
+            errorClass="INVALID_JSON_DATA_TYPE_FOR_COLLATIONS",
+            messageParameters={"jsonType": fieldType},
+        )
+
+
+def _assert_valid_collation_provider(provider):
+    """As PySpark's: a collation is of a provider of 
``StringType.providers``."""
+    if provider.lower() not in StringType.providers:
+        raise _pyspark_value_error(
+            "COLLATION_INVALID_PROVIDER",
+            {"provider": provider, "supportedProviders": ", 
".join(StringType.providers)},
+        )
 
 
 def _pyspark_value_error(error_class, parameters):
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index d5e8f44..35be445 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1968,6 +1968,25 @@ def 
test_create_dataframe_takes_the_ddl_of_spark_as_pyspark(session):
 
 
 
+def test_schemas_of_collated_strings_are_of_the_json_of_pyspark(session):
+    # As PySpark's: the JSON of a schema has the collations of its strings in 
the metadata of
+    # their fields, and StructType.fromJson reads them back.
+    import json
+
+    from pyspark.sql.types import StructType
+
+    schema = session.sql(
+        "select 'a' collate UTF8_LCASE as s, array('b' collate UNICODE) as 
a").schema
+    assert schema.json() == (
+        
'{"fields":[{"metadata":{"__COLLATIONS":{"s":"spark.UTF8_LCASE"}},"name":"s",'
+        
'"nullable":false,"type":"string"},{"metadata":{"__COLLATIONS":{"a.element":'
+        
'"icu.UNICODE"}},"name":"a","nullable":false,"type":{"containsNull":false,'
+        '"elementType":"string","type":"array"}}],"type":"struct"}')
+    assert StructType.fromJson(json.loads(schema.json())) == schema
+
+
+
+
 def test_ddl_types_are_parsed_by_the_server_as_pyspark(session):
     # As PySpark Connect's DataType.fromDDL and _parse_datatype_string, which 
ask the server
     # of the active session to parse a DDL string, as does a UDF of a DDL 
return type.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index bab829d..6686fe5 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2731,6 +2731,72 @@ def test_map_type_from_json():
     assert reconstructed.simpleString() == map_type.simpleString()
 
 
+def 
test_struct_fields_keep_the_collations_of_their_strings_in_their_metadata_as_pyspark():
+    # As PySpark's: the JSON of a struct field has the collations of its 
strings, also in its
+    # arrays and maps, in the __COLLATIONS of its metadata, and the readers 
take them back;
+    # jsonValue has the keys of PySpark's, in its order.
+    from pyspark.errors import PySparkTypeError, PySparkValueError
+    from pyspark.sql.types import _parse_datatype_json_string
+    from pyspark.testing.objects import PythonOnlyUDT
+
+    st = T.StructType([
+        T.StructField("s", T.StringType("UTF8_LCASE")),
+        T.StructField("a", T.ArrayType(T.StringType("UNICODE"))),
+        T.StructField("m", T.MapType(
+            T.StringType("UTF8_LCASE"), 
T.ArrayType(T.StringType("UNICODE_CI"), False))),
+        T.StructField("n", T.StructType([T.StructField("t", 
T.StringType("UTF8_LCASE"))])),
+        T.StructField("plain", T.StringType(), False, {"k": "v"}),
+    ])
+    assert st.json() == (
+        
'{"fields":[{"metadata":{"__COLLATIONS":{"s":"spark.UTF8_LCASE"}},"name":"s",'
+        
'"nullable":true,"type":"string"},{"metadata":{"__COLLATIONS":{"a.element":'
+        
'"icu.UNICODE"}},"name":"a","nullable":true,"type":{"containsNull":true,'
+        
'"elementType":"string","type":"array"}},{"metadata":{"__COLLATIONS":{"m.key":'
+        
'"spark.UTF8_LCASE","m.value.element":"icu.UNICODE_CI"}},"name":"m","nullable":true,'
+        
'"type":{"keyType":"string","type":"map","valueContainsNull":true,"valueType":'
+        
'{"containsNull":false,"elementType":"string","type":"array"}}},{"metadata":{},'
+        
'"name":"n","nullable":true,"type":{"fields":[{"metadata":{"__COLLATIONS":{"t":'
+        
'"spark.UTF8_LCASE"}},"name":"t","nullable":true,"type":"string"}],"type":"struct"}},'
+        
'{"metadata":{"k":"v"},"name":"plain","nullable":false,"type":"string"}],'
+        '"type":"struct"}')
+    assert json.dumps(st.jsonValue()).startswith(
+        '{"type": "struct", "fields": [{"name": "s", "type": "string", 
"nullable": true, '
+        '"metadata": {"__COLLATIONS": {"s": "spark.UTF8_LCASE"}}}, {"name": 
"a", "type": '
+        '{"type": "array", "elementType": "string", "containsNull": true}')
+    assert _parse_datatype_json_string(st.json()) == st
+    assert T.StructType.fromJson(json.loads(st.json())) == st
+    assert T.StructField.fromJson(st.fields[2].jsonValue()) == st.fields[2]
+    assert pickle.loads(pickle.dumps(st)) == st
+    assert T.ArrayType.fromJson(
+        {"elementType": "string", "containsNull": True}, "f", {"f.element": 
"UTF8_LCASE"}
+    ) == T.ArrayType(T.StringType("UTF8_LCASE"))
+    assert T.MapType.fromJson(
+        {"keyType": "string", "valueType": "string", "valueContainsNull": 
True}, "",
+        {"key": "UNICODE"},
+    ) == T.MapType(T.StringType("UNICODE"), T.StringType())
+    # A struct of a user-defined type reads back too.
+    udt = T.StructType([T.StructField("p", PythonOnlyUDT())])
+    assert T.StructType.fromJson(udt.jsonValue()) == udt
+
+    def field(type_json, collation):
+        return {"name": "f", "type": type_json, "nullable": True,
+                "metadata": {"__COLLATIONS": {"f": collation}}}
+
+    with pytest.raises(PySparkValueError) as e:
+        T.StructField.fromJson(field("string", "other.UTF8_LCASE"))
+    assert e.value.getCondition() == "COLLATION_INVALID_PROVIDER"
+    assert e.value.getMessageParameters() == {
+        "provider": "other", "supportedProviders": "spark, icu"}
+    for type_json, json_type in (("integer", "integer"), ({"type": "array", 
"elementType": (
+            "string"), "containsNull": True}, "array")):
+        with pytest.raises(PySparkTypeError) as e:
+            T.StructField.fromJson(field(type_json, "spark.UTF8_LCASE"))
+        assert e.value.getCondition() == 
"INVALID_JSON_DATA_TYPE_FOR_COLLATIONS"
+        assert e.value.getMessageParameters() == {"jsonType": json_type}
+
+
+
+
 def test_string_type_collation():
     """Test StringType with collation parameter."""
     # StringType with explicit collation


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to