This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 2b95591 [SPARK-61074][CONNECT][PYTHON] Write and read the collations
of the strings of a struct field in its metadata, and order the keys of
jsonValue, in the Python client as PySpark does
2b95591 is described below
commit 2b9559113aa073e9204e11cfb0b10e2f904c8e13
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 14:21:31 2026 +0900
[SPARK-61074][CONNECT][PYTHON] Write and read the collations of the strings
of a struct field in its metadata, and order the keys of jsonValue, in the
Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61074](https://issues.apache.org/jira/browse/SPARK-61074): the JSON
of a struct field now holds the collations of its strings in its metadata, and
the `fromJson` methods read them back, as PySpark's do.
As Spark and PySpark do:
- `StructType.json()` and `jsonValue()` write the collations of the strings
of a field, also those in its arrays and maps, in the `__COLLATIONS` of its
metadata. Each is `<provider>.<collation>`, keyed by its path: `s`,
`a.element`, `m.key`, `m.value.element`. The type of the field is written
without them, for example `"string"`.
- `StructField.fromJson`, `StructType.fromJson` and
`_parse_datatype_json_value` read them back. So do `ArrayType.fromJson` and
`MapType.fromJson`, which use their `fieldPath` and `collationsMap`. They raise
`COLLATION_INVALID_PROVIDER` for a provider that is neither `spark` nor `icu`.
They raise `INVALID_JSON_DATA_TYPE_FOR_COLLATIONS` for a collation of a field
that is not a string.
- The dicts of `jsonValue()` have their keys in PySpark's order: `name`,
`type`, `nullable`, `metadata` for a field, and `type` first for a nested type.
The `fromJson` methods now call the helpers of the shim
`pyspark.sql.types`, which follow PySpark's code. So they also read the struct
of a user-defined type. The core reads the collations of the JSON of a struct
field, and the JSON of a UDT, which it can write but could not read back.
### Why are the changes needed?
The client wrote `"string collate UTF8_LCASE"` as the type of a field, so
the JSON of a schema differed from PySpark's:
```
{"fields":[{"metadata":{},"name":"s","nullable":false,"type":"string
collate UTF8_LCASE"}],"type":"struct"}
```
In addition:
- `ArrayType.fromJson` and `MapType.fromJson` ignored their `collationsMap`.
- `StructType.fromJson` and `StructField.fromJson` raised
`INVALID_DATATYPE_FORMAT` for a field of a user-defined type.
- `jsonValue()` sorted its keys, so `json.dumps(schema.jsonValue())`
differed from PySpark's.
### Does this PR introduce _any_ user-facing change?
Yes. The JSON of the types, and its readers, are those of PySpark 4.3.0.
### How was this patch tested?
- A new live test checks the JSON of the schema of collated columns from
the server, and reads it back with `StructType.fromJson`. It passes with
official PySpark 4.3.0 and fails without this change.
- A new offline test checks the JSON and the `jsonValue` of a struct of
collated strings, also in arrays, maps and nested structs, against PySpark's.
It reads the struct back with each reader, round-trips a struct of a
user-defined type, and checks the errors.
- A new lib test checks the JSON of collated strings of the core, its
errors, and the JSON of a UDT.
- On 28 cases of the readers and their errors, and on the writers, the
client gives the same results as official PySpark 4.3.0.
- The lib tests of both crates (589 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1979 passed, 8 skipped), and the
live drop-in and Arrow suites (135 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #276 from HyukjinKwon/SPARK-61074-collation-json.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/types.rs | 202 +++++++++-------
crates/spark-connect/src/types.rs | 325 ++++++++++++++++++++------
crates/spark-connect/tests/types_roundtrip.rs | 86 +++++++
python/pyspark/sql/types.py | 151 +++++++++---
python/tests/test_dropin_e2e.py | 19 ++
python/tests/test_dropin_offline.py | 66 ++++++
6 files changed, 666 insertions(+), 183 deletions(-)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index a333443..eee200a 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -87,32 +87,11 @@ fn py_new_struct(py: Python<'_>, fields: Vec<StructField>)
-> PyResult<Py<PyStru
)
}
-/// Parse a type-JSON dict (as passed to `<Type>.fromJson`) into a core
`DataType`.
-/// The dict may omit the top-level `"type"` key (each `fromJson` is a
classmethod on a
-/// specific type), so inject `default_type` when it is absent before handing
the object
-/// to the core JSON parser.
-fn from_json_dict(
- py: Python<'_>,
- json: &Bound<'_, PyAny>,
- default_type: &str,
-) -> PyResult<DataType> {
- use pyo3::types::PyDict;
- let value: Bound<'_, PyAny> = if let Ok(d) = json.cast::<PyDict>() {
- let d = d.copy()?;
- if !d.contains("type")? {
- d.set_item("type", default_type)?;
- }
- d.into_any()
- } else {
- json.clone()
- };
- let s: String = py
- .import("json")?
- .getattr("dumps")?
- .call1((value,))?
- .extract()?;
- DataType::from_json_str(&s)
- .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))
+/// The `fromJson` helper `name` of the shim `pyspark.sql.types`, which parses
the nested JSON
+/// forms of the types as PySpark's: the collations of a struct field by their
paths, and the
+/// Python class of a `udt`. The core's grammar parses the strings of the
atomic types.
+fn shim_from_json<'py>(py: Python<'py>, name: &str) -> PyResult<Bound<'py,
PyAny>> {
+ py.import("pyspark.sql.types")?.getattr(name)
}
/// Convert a Python `StructField` metadata dict to the core JSON-value map.
Values may be any
@@ -131,6 +110,82 @@ fn py_metadata_to_map(
.map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))
}
+/// The keys of the JSON objects of the nested data types, in the order of
PySpark's
+/// `jsonValue`: struct, array, map, and the UDTs of a Scala class or of a
pickled Python class.
+const TYPE_JSON_KEYS: [&str; 11] = [
+ "type",
+ "fields",
+ "elementType",
+ "containsNull",
+ "keyType",
+ "valueType",
+ "valueContainsNull",
+ "class",
+ "pyClass",
+ "serializedClass",
+ "sqlType",
+];
+
+/// The keys of the JSON object of a struct field, in the order of PySpark's
`jsonValue`.
+const FIELD_JSON_KEYS: [&str; 4] = ["name", "type", "nullable", "metadata"];
+
+/// The JSON text of the JSON value `v` of a data type (of a struct field, if
`field`), with
+/// the keys of its objects in the order of PySpark's `jsonValue`. The core's
JSON objects
+/// sort their keys, as `json()` does; the metadata of a field stays in that
order.
+fn ordered_json_text(v: &serde_json::Value, field: bool) -> String {
+ let Some(object) = v.as_object() else {
+ return v.to_string();
+ };
+ let order: &[&str] = if field {
+ &FIELD_JSON_KEYS
+ } else {
+ &TYPE_JSON_KEYS
+ };
+ let keys = order
+ .iter()
+ .copied()
+ .filter(|key| object.contains_key(*key))
+ .chain(
+ object
+ .keys()
+ .map(String::as_str)
+ .filter(|key| !order.contains(key)),
+ );
+ let entries: Vec<String> = keys
+ .map(|key| {
+ let value = &object[key];
+ let text = match (field, key) {
+ (true, "type") | (false, "elementType" | "keyType" |
"valueType" | "sqlType") => {
+ ordered_json_text(value, false)
+ }
+ (false, "fields") => match value.as_array() {
+ Some(fields) => {
+ let fields: Vec<String> =
+ fields.iter().map(|f| ordered_json_text(f,
true)).collect();
+ format!("[{}]", fields.join(","))
+ }
+ None => value.to_string(),
+ },
+ _ => value.to_string(),
+ };
+ format!("{}:{}", serde_json::Value::from(key), text)
+ })
+ .collect();
+ format!("{{{}}}", entries.join(","))
+}
+
+/// The `jsonValue` of the JSON value `v` of a data type (of a struct field,
if `field`): its
+/// dicts have the keys in the order of PySpark's, as
`json.dumps(t.jsonValue())` shows.
+fn py_json_value<'py>(
+ py: Python<'py>,
+ v: &serde_json::Value,
+ field: bool,
+) -> PyResult<Bound<'py, PyAny>> {
+ py.import("json")?
+ .getattr("loads")?
+ .call1((ordered_json_text(v, field),))
+}
+
/// Convert the core JSON-value metadata map back to a Python dict.
fn metadata_map_to_py<'py>(
py: Python<'py>,
@@ -182,7 +237,7 @@ impl PyDataType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -311,7 +366,7 @@ impl PyNullType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -361,7 +416,7 @@ impl PyBooleanType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -411,7 +466,7 @@ impl PyByteType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -465,7 +520,7 @@ impl PyShortType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -519,7 +574,7 @@ impl PyIntegerType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -573,7 +628,7 @@ impl PyLongType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -627,7 +682,7 @@ impl PyFloatType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -681,7 +736,7 @@ impl PyDoubleType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -738,7 +793,7 @@ impl PyDecimalType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -821,7 +876,7 @@ impl PyStringType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -908,7 +963,7 @@ impl PyBinaryType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -961,7 +1016,7 @@ impl PyDateType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1011,7 +1066,7 @@ impl PyTimestampType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1065,7 +1120,7 @@ impl PyTimestampNTZType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1133,7 +1188,7 @@ impl PyArrayType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1220,9 +1275,9 @@ impl PyArrayType {
fieldPath: String,
collationsMap: Option<Bound<'_, PyAny>>,
) -> PyResult<Py<PyAny>> {
- let _ = (fieldPath, collationsMap);
- let dt = from_json_dict(py, json, "array")?;
- data_type_to_py(py, &dt)
+ Ok(shim_from_json(py, "_array_type_from_json")?
+ .call1((json, fieldPath, collationsMap))?
+ .unbind())
}
}
@@ -1262,7 +1317,7 @@ impl PyMapType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1367,9 +1422,9 @@ impl PyMapType {
fieldPath: String,
collationsMap: Option<Bound<'_, PyAny>>,
) -> PyResult<Py<PyAny>> {
- let _ = (fieldPath, collationsMap);
- let dt = from_json_dict(py, json, "map")?;
- data_type_to_py(py, &dt)
+ Ok(shim_from_json(py, "_map_type_from_json")?
+ .call1((json, fieldPath, collationsMap))?
+ .unbind())
}
}
@@ -1405,7 +1460,7 @@ impl PyStructField {
#[pyo3(name = "jsonValue")]
fn __sf_json_value<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py,
PyAny>> {
let v = self.field.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, true)
}
#[pyo3(name = "needConversion")]
fn __sf_need_conversion(&self) -> bool {
@@ -1504,15 +1559,10 @@ impl PyStructField {
_cls: &Bound<'_, pyo3::types::PyType>,
py: Python<'_>,
json: &Bound<'_, PyAny>,
- ) -> PyResult<PyStructField> {
- let s: String = py
- .import("json")?
- .getattr("dumps")?
+ ) -> PyResult<Py<PyAny>> {
+ Ok(shim_from_json(py, "_struct_field_from_json")?
.call1((json,))?
- .extract()?;
- let field = StructField::from_json_str(&s)
- .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))?;
- Ok(PyStructField { field })
+ .unbind())
}
#[new]
#[pyo3(signature = (name, dataType, nullable=true, metadata=None))]
@@ -1598,7 +1648,7 @@ impl PyStructType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1868,20 +1918,10 @@ impl PyStructType {
_cls: &Bound<'_, pyo3::types::PyType>,
py: Python<'_>,
json: &Bound<'_, PyAny>,
- ) -> PyResult<Py<PyStructType>> {
- let s: String = py
- .import("json")?
- .getattr("dumps")?
+ ) -> PyResult<Py<PyAny>> {
+ Ok(shim_from_json(py, "_struct_type_from_json")?
.call1((json,))?
- .extract()?;
- match DataType::from_json_str(&s)
- .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError,
_>(e.to_string()))?
- {
- DataType::Struct { fields } => py_new_struct(py, fields),
- _ => Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
- "fromJson did not produce a StructType",
- )),
- }
+ .unbind())
}
fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
@@ -1921,7 +1961,7 @@ impl PyCharType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -1981,7 +2021,7 @@ impl PyVarcharType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2041,7 +2081,7 @@ impl PyTimeType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2105,7 +2145,7 @@ impl PyTimestampNTZNanosType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2178,7 +2218,7 @@ impl PyTimestampLTZNanosType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2247,7 +2287,7 @@ impl PyCalendarIntervalType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2304,7 +2344,7 @@ impl PyYearMonthIntervalType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2384,7 +2424,7 @@ impl PyDayTimeIntervalType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
@@ -2453,7 +2493,7 @@ impl PyVariantType {
py: Python<'py>,
) -> PyResult<Bound<'py, PyAny>> {
let v = py_to_data_type(slf.as_any())?.json_value();
- py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ py_json_value(py, &v, false)
}
#[pyo3(name = "needConversion")]
fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
diff --git a/crates/spark-connect/src/types.rs
b/crates/spark-connect/src/types.rs
index 043b951..4a8d4f6 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -7,7 +7,7 @@ use std::collections::BTreeMap;
use std::fmt::{self, Write};
use std::hash::{Hash, Hasher};
-use spark_connect_core::error::{Result, SparkError};
+use spark_connect_core::error::{Result, SparkError, SparkErrorKind};
/// The base DataType representation, mirroring `pyspark.sql.types.DataType`.
///
@@ -477,7 +477,7 @@ impl DataType {
/// Parses a JSON value into a DataType, mirroring the reverse of `json()`
/ `jsonValue()`.
pub fn from_json(value: &serde_json::Value) -> Result<DataType> {
- parse_json_value(value, None)
+ parse_json_value(value, "", &CollationsMap::new())
}
/// Parses a JSON string into a DataType (convenience over [`from_json`]).
@@ -855,13 +855,27 @@ impl StructField {
format!("{}:{}", self.name, self.data_type.simple_string())
}
- /// Returns the JSON value representation, mirroring
`StructField.jsonValue()`.
+ /// Returns the JSON value representation, mirroring
`StructField.jsonValue()`: the
+ /// collations of the strings of the field, also in its arrays and maps,
are in the
+ /// `__COLLATIONS` of its metadata, by their paths, and not in its type,
as Spark writes them.
pub fn json_value(&self) -> serde_json::Value {
+ let mut collations = serde_json::Map::new();
+ visit_collations(&self.data_type, &self.name, &mut collations);
+ let (data_type, metadata) = if collations.is_empty() {
+ (self.data_type.json_value(), self.metadata.clone())
+ } else {
+ let mut metadata = self.metadata.clone();
+ metadata.insert(
+ COLLATIONS_METADATA_KEY.to_string(),
+ serde_json::Value::Object(collations),
+ );
+ (without_collations(&self.data_type).json_value(), metadata)
+ };
serde_json::json!({
"name": &self.name,
- "type": self.data_type.json_value(),
+ "type": data_type,
"nullable": self.nullable,
- "metadata": self.metadata,
+ "metadata": metadata,
})
}
@@ -869,25 +883,10 @@ impl StructField {
pub fn from_json_str(s: &str) -> Result<StructField> {
let v: serde_json::Value = serde_json::from_str(s)
.map_err(|e| SparkError::value("INVALID_JSON", &[("detail",
&e.to_string())]))?;
- let name = v
- .get("name")
- .and_then(|x| x.as_str())
- .ok_or_else(|| SparkError::value("INVALID_JSON", &[("detail",
"missing field name")]))?
- .to_string();
- let nullable = v.get("nullable").and_then(|x|
x.as_bool()).unwrap_or(true);
- let data_type =
DataType::from_json(v.get("type").unwrap_or(&serde_json::Value::Null))?;
- let mut metadata = BTreeMap::new();
- if let Some(md) = v.get("metadata").and_then(|x| x.as_object()) {
- for (k, val) in md {
- metadata.insert(k.clone(), val.clone());
- }
- }
- Ok(StructField {
- name,
- data_type,
- nullable,
- metadata,
- })
+ let field = v.as_object().ok_or_else(|| {
+ SparkError::value("INVALID_JSON", &[("detail", "a field is not an
object")])
+ })?;
+ parse_json_field(field)
}
}
@@ -1170,11 +1169,201 @@ fn interval_string(start_field: i32, end_field: i32,
fields: &[(&str, i32)]) ->
}
}
-/// Parses a JSON value into a DataType
-fn parse_json_value(value: &serde_json::Value, _field_name: Option<&str>) ->
Result<DataType> {
+/// The key of the metadata of a struct field of the collations of its
strings, as Spark's.
+const COLLATIONS_METADATA_KEY: &str = "__COLLATIONS";
+
+/// The collations of the strings of a field of `data_type` at `path`, also in
its arrays and
+/// maps but not in its structs, whose fields have their own, as PySpark's
+/// `StructField.getCollationMetadata`: `<provider>.<collation>` of each path.
+fn visit_collations(
+ data_type: &DataType,
+ path: &str,
+ collations: &mut serde_json::Map<String, serde_json::Value>,
+) {
+ match data_type {
+ DataType::Array { element_type, .. } => {
+ visit_collations(element_type, &format!("{path}.element"),
collations)
+ }
+ DataType::Map {
+ key_type,
+ value_type,
+ ..
+ } => {
+ visit_collations(key_type, &format!("{path}.key"), collations);
+ visit_collations(value_type, &format!("{path}.value"), collations);
+ }
+ DataType::String { collation } if collation != "UTF8_BINARY" => {
+ // As PySpark's `StringType.collationProvider`.
+ let provider = if collation.starts_with("UTF8") {
+ "spark"
+ } else {
+ "icu"
+ };
+ collations.insert(path.to_string(),
format!("{provider}.{collation}").into());
+ }
+ _ => {}
+ }
+}
+
+/// `data_type` without the collations of its strings, also in its arrays and
maps, which the
+/// JSON of a struct field keeps in its metadata.
+fn without_collations(data_type: &DataType) -> DataType {
+ match data_type {
+ DataType::Array {
+ element_type,
+ contains_null,
+ } => DataType::Array {
+ element_type: Box::new(without_collations(element_type)),
+ contains_null: *contains_null,
+ },
+ DataType::Map {
+ key_type,
+ value_type,
+ value_contains_null,
+ } => DataType::Map {
+ key_type: Box::new(without_collations(key_type)),
+ value_type: Box::new(without_collations(value_type)),
+ value_contains_null: *value_contains_null,
+ },
+ DataType::String { .. } => DataType::String {
+ collation: "UTF8_BINARY".to_string(),
+ },
+ other => other.clone(),
+ }
+}
+
+/// The collations of the strings of a struct field by their paths,
`__COLLATIONS` of its
+/// metadata without their providers, as the `collationsMap` of PySpark's
`fromJson`.
+type CollationsMap = std::collections::HashMap<String, String>;
+
+/// The JSON type names to which a collation of a struct field applies, as
PySpark's
+/// `_all_mappable_types`: it makes a `string` a collated string, and raises
+/// `INVALID_JSON_DATA_TYPE_FOR_COLLATIONS` for the others.
+const MAPPABLE_TYPE_NAMES: [&str; 16] = [
+ "string",
+ "binary",
+ "boolean",
+ "decimal",
+ "float",
+ "double",
+ "byte",
+ "short",
+ "integer",
+ "long",
+ "date",
+ "timestamp",
+ "timestamp_ntz",
+ "void",
+ "variant",
+ "interval",
+];
+
+/// The path of `child` (`element`, `key` or `value`) of a type at `path`, as
PySpark's
+/// `fromJson` of arrays and maps names it.
+fn child_path(path: &str, child: &str) -> String {
+ if path.is_empty() {
+ child.to_string()
+ } else {
+ format!("{path}.{child}")
+ }
+}
+
+/// The collations of the `__COLLATIONS` of `metadata`, which this removes, as
PySpark's
+/// `StructField.fromJson`: each is `<provider>.<collation>`, of the `spark`
or `icu` provider.
+fn take_field_collations(
+ metadata: &mut BTreeMap<String, serde_json::Value>,
+) -> Result<CollationsMap> {
+ let Some(collations) = metadata.remove(COLLATIONS_METADATA_KEY) else {
+ return Ok(CollationsMap::new());
+ };
+ let mut by_path = CollationsMap::new();
+ for (path, value) in collations.as_object().into_iter().flatten() {
+ let parts: Vec<&str> =
value.as_str().unwrap_or_default().split('.').collect();
+ let [provider, collation] = parts[..] else {
+ return Err(SparkError::value(
+ "INVALID_DATATYPE_FORMAT",
+ &[("detail", "a collation is not of a provider and a name")],
+ ));
+ };
+ if !["spark", "icu"].contains(&provider.to_lowercase().as_str()) {
+ return Err(SparkError::value(
+ "COLLATION_INVALID_PROVIDER",
+ &[("provider", provider), ("supportedProviders", "spark,
icu")],
+ ));
+ }
+ by_path.insert(path.clone(), collation.to_string());
+ }
+ Ok(by_path)
+}
+
+/// Parses the JSON object of a struct field, mirroring
`StructField.fromJson`: the strings at
+/// the paths of the `__COLLATIONS` of its metadata are of their collations.
+fn parse_json_field(field: &serde_json::Map<String, serde_json::Value>) ->
Result<StructField> {
+ let name = field
+ .get("name")
+ .and_then(|v| v.as_str())
+ .ok_or_else(|| {
+ SparkError::value(
+ "INVALID_DATATYPE_FORMAT",
+ &[("detail", "StructField missing name")],
+ )
+ })?
+ .to_string();
+ let field_type = field.get("type").ok_or_else(|| {
+ SparkError::value(
+ "INVALID_DATATYPE_FORMAT",
+ &[("detail", "StructField missing type")],
+ )
+ })?;
+ let nullable = field
+ .get("nullable")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(true);
+ let mut metadata: BTreeMap<String, serde_json::Value> = field
+ .get("metadata")
+ .and_then(|v| v.as_object())
+ .map(|m| m.iter().map(|(k, v)| (k.clone(), v.clone())).collect())
+ .unwrap_or_default();
+ let collations = take_field_collations(&mut metadata)?;
+ Ok(StructField {
+ data_type: parse_json_value(field_type, &name, &collations)?,
+ name,
+ nullable,
+ metadata,
+ })
+}
+
+/// Raises, as PySpark's `_assert_valid_type_for_collation`, for a collation
of a field path of
+/// a type that is not a string.
+fn check_collated_type(path: &str, json_type: &str, collations:
&CollationsMap) -> Result<()> {
+ if collations.contains_key(path) && json_type != "string" {
+ return Err(SparkError::classed(
+ SparkErrorKind::TypeError,
+ "INVALID_JSON_DATA_TYPE_FOR_COLLATIONS",
+ &[("jsonType", json_type)],
+ ));
+ }
+ Ok(())
+}
+
+/// Parses a JSON value into a DataType at `field_path` of a struct field,
mirroring PySpark's
+/// `_parse_datatype_json_value`: a string of a path of `collations` is of its
collation.
+fn parse_json_value(
+ value: &serde_json::Value,
+ field_path: &str,
+ collations: &CollationsMap,
+) -> Result<DataType> {
match value {
- serde_json::Value::String(s) => parse_json_type_string(s),
- serde_json::Value::Object(obj) => parse_json_object(obj),
+ serde_json::Value::String(s) => match collations.get(field_path) {
+ Some(collation) if MAPPABLE_TYPE_NAMES.contains(&s.as_str()) => {
+ check_collated_type(field_path, s, collations)?;
+ Ok(DataType::String {
+ collation: collation.clone(),
+ })
+ }
+ _ => parse_json_type_string(s),
+ },
+ serde_json::Value::Object(obj) => parse_json_object(obj, field_path,
collations),
_ => Err(SparkError::value(
"INVALID_DATATYPE_FORMAT",
&[("detail", "Expected string or object for DataType")],
@@ -1235,8 +1424,15 @@ fn parse_json_type_string(s: &str) -> Result<DataType> {
}
}
-fn parse_json_object(obj: &serde_json::Map<String, serde_json::Value>) ->
Result<DataType> {
+fn parse_json_object(
+ obj: &serde_json::Map<String, serde_json::Value>,
+ field_path: &str,
+ collations: &CollationsMap,
+) -> Result<DataType> {
let type_field = obj.get("type").and_then(|v| v.as_str());
+ if let Some(json_type @ ("array" | "map" | "struct")) = type_field {
+ check_collated_type(field_path, json_type, collations)?;
+ }
match type_field {
Some("array") => {
@@ -1251,7 +1447,11 @@ fn parse_json_object(obj: &serde_json::Map<String,
serde_json::Value>) -> Result
.and_then(|v| v.as_bool())
.unwrap_or(true);
Ok(DataType::Array {
- element_type: Box::new(parse_json_value(element_type, None)?),
+ element_type: Box::new(parse_json_value(
+ element_type,
+ &child_path(field_path, "element"),
+ collations,
+ )?),
contains_null,
})
}
@@ -1273,8 +1473,16 @@ fn parse_json_object(obj: &serde_json::Map<String,
serde_json::Value>) -> Result
.and_then(|v| v.as_bool())
.unwrap_or(true);
Ok(DataType::Map {
- key_type: Box::new(parse_json_value(key_type, None)?),
- value_type: Box::new(parse_json_value(value_type, None)?),
+ key_type: Box::new(parse_json_value(
+ key_type,
+ &child_path(field_path, "key"),
+ collations,
+ )?),
+ value_type: Box::new(parse_json_value(
+ value_type,
+ &child_path(field_path, "value"),
+ collations,
+ )?),
value_contains_null,
})
}
@@ -1290,45 +1498,26 @@ fn parse_json_object(obj: &serde_json::Map<String,
serde_json::Value>) -> Result
})?;
let mut fields = Vec::new();
- for field_obj in fields_arr {
- if let Some(field_map) = field_obj.as_object() {
- let name = field_map
- .get("name")
- .and_then(|v| v.as_str())
- .ok_or_else(|| {
- SparkError::value(
- "INVALID_DATATYPE_FORMAT",
- &[("detail", "StructField missing name")],
- )
- })?
- .to_string();
- let field_type = field_map.get("type").ok_or_else(|| {
- SparkError::value(
- "INVALID_DATATYPE_FORMAT",
- &[("detail", "StructField missing type")],
- )
- })?;
- let data_type = parse_json_value(field_type, Some(&name))?;
- let nullable = field_map
- .get("nullable")
- .and_then(|v| v.as_bool())
- .unwrap_or(true);
- let metadata = field_map
- .get("metadata")
- .and_then(|v| v.as_object())
- .map(|m| m.iter().map(|(k, v)| (k.clone(),
v.clone())).collect())
- .unwrap_or_default();
-
- fields.push(StructField {
- name,
- data_type,
- nullable,
- metadata,
- });
- }
+ for field in fields_arr.iter().filter_map(|field|
field.as_object()) {
+ fields.push(parse_json_field(field)?);
}
Ok(DataType::Struct { fields })
}
+ // The inverse of `json_value` of a UDT: its classes and the JSON of
its SQL type.
+ Some("udt") => {
+ let text = |key: &str| obj.get(key).and_then(|v|
v.as_str()).map(str::to_string);
+ let sql_type = match obj.get("sqlType") {
+ Some(sql_type) =>
Some(Box::new(DataType::from_json(sql_type)?)),
+ None => None,
+ };
+ Ok(DataType::Udt {
+ type_str: "udt".to_string(),
+ jvm_class: text("class"),
+ python_class: text("pyClass"),
+ serialized_python_class: text("serializedClass"),
+ sql_type,
+ })
+ }
_ => Err(SparkError::value(
"INVALID_DATATYPE_FORMAT",
&[("detail", "Unknown type in object")],
diff --git a/crates/spark-connect/tests/types_roundtrip.rs
b/crates/spark-connect/tests/types_roundtrip.rs
index bffae04..d28a2ef 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -232,6 +232,92 @@ fn special_variant_arms() {
}
}
+/// A struct field writes the collations of its strings, also in its arrays
and maps, in the
+/// `__COLLATIONS` of its metadata, as Spark and PySpark do, and reads them
back; a UDT reads
+/// back its JSON too.
+#[test]
+fn collations_are_in_the_metadata_of_the_json_of_a_field_as_spark() {
+ let string = |collation: &str| DataType::String {
+ collation: collation.to_string(),
+ };
+ let field = |name: &str, data_type: DataType| StructField {
+ name: name.to_string(),
+ data_type,
+ nullable: true,
+ metadata: BTreeMap::new(),
+ };
+ let st = DataType::Struct {
+ fields: vec![
+ field("s", string("UTF8_LCASE")),
+ field(
+ "m",
+ DataType::Map {
+ key_type: Box::new(string("UTF8_LCASE")),
+ value_type: Box::new(DataType::Array {
+ element_type: Box::new(string("UNICODE_CI")),
+ contains_null: false,
+ }),
+ value_contains_null: true,
+ },
+ ),
+ field(
+ "n",
+ DataType::Struct {
+ fields: vec![field("t", string("UTF8_LCASE"))],
+ },
+ ),
+ ],
+ };
+ assert_eq!(
+ st.json(),
+ concat!(
+
r#"{"fields":[{"metadata":{"__COLLATIONS":{"s":"spark.UTF8_LCASE"}},"name":"s","#,
+
r#""nullable":true,"type":"string"},{"metadata":{"__COLLATIONS":{"m.key":"#,
+
r#""spark.UTF8_LCASE","m.value.element":"icu.UNICODE_CI"}},"name":"m","#,
+
r#""nullable":true,"type":{"keyType":"string","type":"map","valueContainsNull":true,"#,
+
r#""valueType":{"containsNull":false,"elementType":"string","type":"array"}}},"#,
+
r#"{"metadata":{},"name":"n","nullable":true,"type":{"fields":[{"metadata":"#,
+
r#"{"__COLLATIONS":{"t":"spark.UTF8_LCASE"}},"name":"t","nullable":true,"#,
+ r#""type":"string"}],"type":"struct"}}],"type":"struct"}"#
+ )
+ );
+ assert_eq!(DataType::from_json_str(&st.json()).unwrap(), st);
+ let DataType::Struct { fields } = &st else {
+ unreachable!()
+ };
+ assert_eq!(
+
StructField::from_json_str(&fields[1].json_value().to_string()).unwrap(),
+ fields[1]
+ );
+ // The errors of PySpark's readers: a provider of no collation, and a
collation of a field
+ // that is not a string.
+ let of = |type_json: &str, collation: &str| {
+ let json = format!(
+
r#"{{"name":"f","type":{type_json},"nullable":true,"metadata":{{"__COLLATIONS":{{"f":"{collation}"}}}}}}"#
+ );
+ StructField::from_json_str(&json).unwrap_err()
+ };
+ let error = of(r#""string""#, "other.UTF8_LCASE");
+ assert_eq!(
+ error.get_error_class().as_deref(),
+ Some("COLLATION_INVALID_PROVIDER")
+ );
+ let error = of(r#""integer""#, "spark.UTF8_LCASE");
+ assert_eq!(
+ error.get_error_class().as_deref(),
+ Some("INVALID_JSON_DATA_TYPE_FOR_COLLATIONS")
+ );
+ // A UDT reads back the JSON that it writes.
+ let udt = DataType::Udt {
+ type_str: "udt".to_string(),
+ jvm_class: None,
+ python_class: Some("mymod.MyUDT".to_string()),
+ serialized_python_class: Some("<pickle>".to_string()),
+ sql_type: Some(Box::new(DataType::Integer)),
+ };
+ assert_eq!(DataType::from_json_str(&udt.json()).unwrap(), udt);
+}
+
/// Exercise the DDL parser across the full grammar it accepts. Calling
`from_ddl` on
/// each form pins the large `parse_datatype_string` body; a few are asserted
to the
/// expected type.
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 062dfc6..d7800c1 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -306,53 +306,136 @@ def _parse_datatype_json_string(json_string):
return _parse_datatype_json_value(_json.loads(json_string))
-def _parse_datatype_json_value(v):
+#: The key of the metadata of a struct field of the collations of its strings,
as PySpark's.
+_COLLATIONS_METADATA_KEY = "__COLLATIONS"
+
+#: The JSON type names to which a collation of a struct field applies, as
PySpark's
+#: ``_all_mappable_types``: it makes a ``string`` a collated string, and raises
+#: ``INVALID_JSON_DATA_TYPE_FOR_COLLATIONS`` for the others.
+_MAPPABLE_TYPE_NAMES = frozenset(
+ [
+ "string", "binary", "boolean", "decimal", "float", "double", "byte",
"short",
+ "integer", "long", "date", "timestamp", "timestamp_ntz", "void",
"variant", "interval",
+ ]
+)
+
+
+def _parse_datatype_json_value(json_value, fieldPath="", collationsMap=None):
"""Recursively reconstruct a DataType from a parsed Spark-JSON value (a
string for
atomic/parameterized types, or a dict for the nested struct/array/map
forms).
The string forms (atomics, ``decimal(p,s)``, ``string collate X``,
intervals,
``geometry(<crs>)``, ``geography(<crs>, <alg>)``, ...) are parsed by the
Rust core's
JSON grammar, so the drop-in keeps one grammar. The nested object forms
recurse here,
- because a ``udt`` leaf has to reconstruct its Python class."""
+ because a ``udt`` leaf has to reconstruct its Python class.
+
+ As PySpark's, a string at a ``fieldPath`` of ``collationsMap``, the
collations of the
+ ``__COLLATIONS`` of the metadata of a struct field by their paths, is of
its collation."""
import json as _json
- if isinstance(v, str):
+ if not isinstance(json_value, dict):
+ if (
+ collationsMap is not None
+ and fieldPath in collationsMap
+ and json_value in _MAPPABLE_TYPE_NAMES
+ ):
+ _assert_valid_type_for_collation(fieldPath, json_value,
collationsMap)
+ return StringType(collationsMap[fieldPath])
from pyspark._pyspark import parse_datatype_json
try:
- return parse_datatype_json(_json.dumps(v))
+ return parse_datatype_json(_json.dumps(json_value))
except ValueError:
# As PySpark's, of a str that is not the JSON value of a data type.
- raise _pyspark_value_error("CANNOT_PARSE_DATATYPE", {"msg":
str(v)}) from None
- elif isinstance(v, dict):
- t = v.get("type")
- if t == "struct":
- fields = [
- StructField(
- f["name"],
- _parse_datatype_json_value(f["type"]),
- f.get("nullable", True),
- f.get("metadata", {}) or {},
- )
- for f in v["fields"]
- ]
- return StructType(fields)
- elif t == "array":
- return ArrayType(
- _parse_datatype_json_value(v["elementType"]),
- v.get("containsNull", True),
- )
- elif t == "map":
- return MapType(
- _parse_datatype_json_value(v["keyType"]),
- _parse_datatype_json_value(v["valueType"]),
- v.get("valueContainsNull", True),
- )
- elif t == "udt":
- return UserDefinedType.fromJson(v)
- # As PySpark's, of an object of a type that is not one of a data type.
- raise _pyspark_value_error("UNSUPPORTED_DATA_TYPE", {"data_type":
str(t)})
- raise _pyspark_value_error("CANNOT_PARSE_DATATYPE", {"msg": str(v)})
+ raise _pyspark_value_error(
+ "CANNOT_PARSE_DATATYPE", {"msg": str(json_value)}
+ ) from None
+ tpe = json_value["type"]
+ if tpe in ("array", "map", "struct"):
+ if collationsMap is not None and fieldPath in collationsMap:
+ _assert_valid_type_for_collation(fieldPath, tpe, collationsMap)
+ if tpe == "array":
+ return _array_type_from_json(json_value, fieldPath, collationsMap)
+ elif tpe == "map":
+ return _map_type_from_json(json_value, fieldPath, collationsMap)
+ return _struct_type_from_json(json_value)
+ elif tpe == "udt":
+ return UserDefinedType.fromJson(json_value)
+ # As PySpark's, of an object of a type that is not one of a data type.
+ raise _pyspark_value_error("UNSUPPORTED_DATA_TYPE", {"data_type":
str(tpe)})
+
+
+def _array_type_from_json(json_value, fieldPath="", collationsMap=None):
+ """``ArrayType.fromJson``: its element at ``element`` of ``fieldPath``, as
PySpark's."""
+ elementType = _parse_datatype_json_value(
+ json_value["elementType"],
+ "element" if fieldPath == "" else fieldPath + ".element",
+ collationsMap,
+ )
+ return ArrayType(elementType, json_value["containsNull"])
+
+
+def _map_type_from_json(json_value, fieldPath="", collationsMap=None):
+ """``MapType.fromJson``: its key and value at ``key`` and ``value`` of
``fieldPath``, as
+ PySpark's."""
+ keyType = _parse_datatype_json_value(
+ json_value["keyType"], "key" if fieldPath == "" else fieldPath +
".key", collationsMap
+ )
+ valueType = _parse_datatype_json_value(
+ json_value["valueType"],
+ "value" if fieldPath == "" else fieldPath + ".value",
+ collationsMap,
+ )
+ return MapType(keyType, valueType, json_value["valueContainsNull"])
+
+
+def _struct_field_from_json(json_value):
+ """``StructField.fromJson``, as PySpark's: the ``__COLLATIONS`` of its
metadata,
+ ``<provider>.<collation>`` by the paths of its strings, are the collations
of those
+ strings, and not metadata."""
+ metadata = json_value.get("metadata")
+ collationsMap = {}
+ if metadata and _COLLATIONS_METADATA_KEY in metadata:
+ for key, value in metadata[_COLLATIONS_METADATA_KEY].items():
+ nameParts = value.split(".")
+ assert len(nameParts) == 2
+ provider, name = nameParts[0], nameParts[1]
+ _assert_valid_collation_provider(provider)
+ collationsMap[key] = name
+ metadata = {
+ key: value for key, value in metadata.items() if key !=
_COLLATIONS_METADATA_KEY
+ }
+ return StructField(
+ json_value["name"],
+ _parse_datatype_json_value(json_value["type"], json_value["name"],
collationsMap),
+ json_value.get("nullable", True),
+ metadata,
+ )
+
+
+def _struct_type_from_json(json_value):
+ """``StructType.fromJson``, as PySpark's."""
+ return StructType([_struct_field_from_json(f) for f in
json_value["fields"]])
+
+
+def _assert_valid_type_for_collation(fieldPath, fieldType, collationMap):
+ """As PySpark's: a collation is of a string."""
+ if fieldPath in collationMap and fieldType != "string":
+ from pyspark.errors import PySparkTypeError
+
+ raise PySparkTypeError(
+ errorClass="INVALID_JSON_DATA_TYPE_FOR_COLLATIONS",
+ messageParameters={"jsonType": fieldType},
+ )
+
+
+def _assert_valid_collation_provider(provider):
+ """As PySpark's: a collation is of a provider of
``StringType.providers``."""
+ if provider.lower() not in StringType.providers:
+ raise _pyspark_value_error(
+ "COLLATION_INVALID_PROVIDER",
+ {"provider": provider, "supportedProviders": ",
".join(StringType.providers)},
+ )
def _pyspark_value_error(error_class, parameters):
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index d5e8f44..35be445 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1968,6 +1968,25 @@ def
test_create_dataframe_takes_the_ddl_of_spark_as_pyspark(session):
+def test_schemas_of_collated_strings_are_of_the_json_of_pyspark(session):
+ # As PySpark's: the JSON of a schema has the collations of its strings in
the metadata of
+ # their fields, and StructType.fromJson reads them back.
+ import json
+
+ from pyspark.sql.types import StructType
+
+ schema = session.sql(
+ "select 'a' collate UTF8_LCASE as s, array('b' collate UNICODE) as
a").schema
+ assert schema.json() == (
+
'{"fields":[{"metadata":{"__COLLATIONS":{"s":"spark.UTF8_LCASE"}},"name":"s",'
+
'"nullable":false,"type":"string"},{"metadata":{"__COLLATIONS":{"a.element":'
+
'"icu.UNICODE"}},"name":"a","nullable":false,"type":{"containsNull":false,'
+ '"elementType":"string","type":"array"}}],"type":"struct"}')
+ assert StructType.fromJson(json.loads(schema.json())) == schema
+
+
+
+
def test_ddl_types_are_parsed_by_the_server_as_pyspark(session):
# As PySpark Connect's DataType.fromDDL and _parse_datatype_string, which
ask the server
# of the active session to parse a DDL string, as does a UDF of a DDL
return type.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index bab829d..6686fe5 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2731,6 +2731,72 @@ def test_map_type_from_json():
assert reconstructed.simpleString() == map_type.simpleString()
+def
test_struct_fields_keep_the_collations_of_their_strings_in_their_metadata_as_pyspark():
+ # As PySpark's: the JSON of a struct field has the collations of its
strings, also in its
+ # arrays and maps, in the __COLLATIONS of its metadata, and the readers
take them back;
+ # jsonValue has the keys of PySpark's, in its order.
+ from pyspark.errors import PySparkTypeError, PySparkValueError
+ from pyspark.sql.types import _parse_datatype_json_string
+ from pyspark.testing.objects import PythonOnlyUDT
+
+ st = T.StructType([
+ T.StructField("s", T.StringType("UTF8_LCASE")),
+ T.StructField("a", T.ArrayType(T.StringType("UNICODE"))),
+ T.StructField("m", T.MapType(
+ T.StringType("UTF8_LCASE"),
T.ArrayType(T.StringType("UNICODE_CI"), False))),
+ T.StructField("n", T.StructType([T.StructField("t",
T.StringType("UTF8_LCASE"))])),
+ T.StructField("plain", T.StringType(), False, {"k": "v"}),
+ ])
+ assert st.json() == (
+
'{"fields":[{"metadata":{"__COLLATIONS":{"s":"spark.UTF8_LCASE"}},"name":"s",'
+
'"nullable":true,"type":"string"},{"metadata":{"__COLLATIONS":{"a.element":'
+
'"icu.UNICODE"}},"name":"a","nullable":true,"type":{"containsNull":true,'
+
'"elementType":"string","type":"array"}},{"metadata":{"__COLLATIONS":{"m.key":'
+
'"spark.UTF8_LCASE","m.value.element":"icu.UNICODE_CI"}},"name":"m","nullable":true,'
+
'"type":{"keyType":"string","type":"map","valueContainsNull":true,"valueType":'
+
'{"containsNull":false,"elementType":"string","type":"array"}}},{"metadata":{},'
+
'"name":"n","nullable":true,"type":{"fields":[{"metadata":{"__COLLATIONS":{"t":'
+
'"spark.UTF8_LCASE"}},"name":"t","nullable":true,"type":"string"}],"type":"struct"}},'
+
'{"metadata":{"k":"v"},"name":"plain","nullable":false,"type":"string"}],'
+ '"type":"struct"}')
+ assert json.dumps(st.jsonValue()).startswith(
+ '{"type": "struct", "fields": [{"name": "s", "type": "string",
"nullable": true, '
+ '"metadata": {"__COLLATIONS": {"s": "spark.UTF8_LCASE"}}}, {"name":
"a", "type": '
+ '{"type": "array", "elementType": "string", "containsNull": true}')
+ assert _parse_datatype_json_string(st.json()) == st
+ assert T.StructType.fromJson(json.loads(st.json())) == st
+ assert T.StructField.fromJson(st.fields[2].jsonValue()) == st.fields[2]
+ assert pickle.loads(pickle.dumps(st)) == st
+ assert T.ArrayType.fromJson(
+ {"elementType": "string", "containsNull": True}, "f", {"f.element":
"UTF8_LCASE"}
+ ) == T.ArrayType(T.StringType("UTF8_LCASE"))
+ assert T.MapType.fromJson(
+ {"keyType": "string", "valueType": "string", "valueContainsNull":
True}, "",
+ {"key": "UNICODE"},
+ ) == T.MapType(T.StringType("UNICODE"), T.StringType())
+ # A struct of a user-defined type reads back too.
+ udt = T.StructType([T.StructField("p", PythonOnlyUDT())])
+ assert T.StructType.fromJson(udt.jsonValue()) == udt
+
+ def field(type_json, collation):
+ return {"name": "f", "type": type_json, "nullable": True,
+ "metadata": {"__COLLATIONS": {"f": collation}}}
+
+ with pytest.raises(PySparkValueError) as e:
+ T.StructField.fromJson(field("string", "other.UTF8_LCASE"))
+ assert e.value.getCondition() == "COLLATION_INVALID_PROVIDER"
+ assert e.value.getMessageParameters() == {
+ "provider": "other", "supportedProviders": "spark, icu"}
+ for type_json, json_type in (("integer", "integer"), ({"type": "array",
"elementType": (
+ "string"), "containsNull": True}, "array")):
+ with pytest.raises(PySparkTypeError) as e:
+ T.StructField.fromJson(field(type_json, "spark.UTF8_LCASE"))
+ assert e.value.getCondition() ==
"INVALID_JSON_DATA_TYPE_FOR_COLLATIONS"
+ assert e.value.getMessageParameters() == {"jsonType": json_type}
+
+
+
+
def test_string_type_collation():
"""Test StringType with collation parameter."""
# StringType with explicit collation
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]