This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 01119459 [SPARK-61207][CONNECT][PYTHON] Stop StructType.add from
re-reading every field and its metadata, in the Python client as PySpark does
01119459 is described below
commit 011194591ec38a7f2ca285802b1f7cdb546c6d9b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:06:33 2026 +0900
[SPARK-61207][CONNECT][PYTHON] Stop StructType.add from re-reading every
field and its metadata, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61207](https://issues.apache.org/jira/browse/SPARK-61207): Stop
`StructType.add` from re-reading every field and its metadata, in the Python
client as PySpark does.
Since SPARK-61178, `add` rebuilt the core type of every field on each call.
For each field it JSON-encoded the metadata dict, and for a field of a struct,
array or map type it read every nested field the same way. A loop of `add`
calls was therefore quadratic in that work. PySpark's `add` appends the new
field and computes `_needConversion` from the type of each field
(`[f.needConversion() for f in self]`). This change does the same in
`StructType.add`:
- It reads the metadata of the new field only. The base core type takes the
new field, as the field list does. The struct's other code reads its live
fields; only the fallback for a bare base `DataType` in the type conversion
reads this base type.
- It computes `_needConversion` from the type object of each field, as
PySpark's `add` does. A nested struct needs a conversion without its fields
being read, and an array or a map reads the type objects of its elements. A
field of a nested type therefore costs about as much as one of an atomic type.
- It keeps the value per field for the types that cannot change in place,
which are the atomic types. A year-month interval is the exception: its value
follows `PYSPARK_YM_INTERVAL_LEGACY` at each `add`, as in PySpark. The value is
reset when `StructField.dataType` is set. A nested struct, array, map or UDT is
read again on each `add`, since an edit in place can change it.
The results are the same as before, and the same as in official PySpark
4.3.0. That covers list identity, edits through the list, and the
`_needConversion` values after an edit of a field type or of
`PYSPARK_YM_INTERVAL_LEGACY`.
Timings of 1000 `add` calls, best of three, against official PySpark 4.3.0
on a loaded machine:
| Fields | Before | After | Official |
|---|---|---|---|
| `add(name, StringType(), True, {"comment": ...})` | 1.6 s | 24 ms | 55 ms
|
| a commented struct, array of a struct or map of a struct, in turn, with a
commented nested field | 4.0 s | 0.22 s | 0.10 s |
### Why are the changes needed?
A struct built with a loop of `add` calls took a time quadratic in its
fields and their metadata. Official PySpark 4.3.0 takes 55 ms for the same 1000
commented fields:
```python
st = StructType()
for i in range(1000):
st.add(f"c{i}", StringType(), True, {"comment": f"col {i}"}) # 1.6 s
before, 24 ms after
```
### Does this PR introduce _any_ user-facing change?
No. The results are the same. Only the time of `StructType.add` changes.
### How was this patch tested?
- New offline test
`test_struct_type_add_with_metadata_is_linear_and_matches_the_constructor` in
`python/tests/test_dropin_offline.py`:
- It adds 2000 commented fields under a 3 s bound, and checks that the
struct equals the one built by the constructor (`==`, `json`, `simpleString`,
`_needConversion`, `names`, `fieldNames`). It also checks that a set `dataType`
is seen by the next `add`.
- It adds 1500 commented fields of nested types under a 3 s bound, and
checks the struct and its `_needConversion` against the constructor's.
- It checks that the `_needConversion` of a year-month interval follows
`PYSPARK_YM_INTERVAL_LEGACY` at each `add`.
- Without this change, the first bound fails: 6.7 s on a loaded machine
(17 s on an idle one). With it, the test takes 0.9 s. It passes on official
PySpark 4.3.0 in 0.7 s.
- A probe of `_needConversion` and `_needSerializeAnyField` after adds of
26 types gives the same values as official PySpark 4.3.0. The types are atomic,
nested arrays and maps, structs, a UDT and intervals. The probe also covers
edits of field types, a struct from `fromJson`, and
`PYSPARK_YM_INTERVAL_LEGACY`.
- The live drop-in and Arrow tests of `-k "struct or schema"` against a
local Spark 4.3.0 server pass (36 passed).
- The lib tests of both crates (618 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2457 passed, 8 skipped);
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs
the e2e and live suites.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #393 from
HyukjinKwon/SPARK-61207-r3-struct-type-add-reads-only-the-new-field.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/types.rs | 99 ++++++++++++++++++++++++++++++++++---
python/tests/test_dropin_offline.py | 60 ++++++++++++++++++++++
2 files changed, 152 insertions(+), 7 deletions(-)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 093600a6..53b0cd63 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -279,6 +279,42 @@ fn need_conversion_of(py: Python<'_>, fields:
&[StructField]) -> PyResult<Vec<bo
.collect()
}
+/// The `needConversion` of each of the `StructField` objects of `fields`, as
they are now.
+fn need_conversion_of_fields(py: Python<'_>, fields: &Bound<'_, PyList>) ->
PyResult<Vec<bool>> {
+ fields
+ .iter()
+ .map(|item| as_struct_field(&item)?.borrow().need_conversion_now(py))
+ .collect()
+}
+
+/// The `needConversion` of the data type object `obj`, as PySpark's: a struct
and a user-defined
+/// type always need a conversion, and an array or a map when one of its
element types does. The
+/// fields of a struct are not read, nor their metadata, so this does not grow
with the struct.
+fn need_conversion_of_object(py: Python<'_>, obj: &Bound<'_, PyAny>) ->
PyResult<bool> {
+ fn of_child(py: Python<'_>, kept: &KeptType, snapshot: &DataType) ->
PyResult<bool> {
+ match kept.kept(py) {
+ Some(object) => need_conversion_of_object(py, object.bind(py)),
+ None => need_conversion(py, snapshot),
+ }
+ }
+ if obj.is_instance_of::<PyStructType>() {
+ return Ok(true);
+ }
+ if let Ok(array) = obj.cast::<PyArrayType>() {
+ let array = array.borrow();
+ return of_child(py, &array.element_object, &array.element_type);
+ }
+ if let Ok(map) = obj.cast::<PyMapType>() {
+ let map = map.borrow();
+ return Ok(of_child(py, &map.key_object, &map.key_type)?
+ || of_child(py, &map.value_object, &map.value_type)?);
+ }
+ if is_udt(obj)? {
+ return Ok(true);
+ }
+ need_conversion(py, &py_to_data_type(obj)?)
+}
+
/// The `StructField` object of `item`, which a `StructType` holds in its
`fields`.
fn as_struct_field<'py>(item: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyStructField>> {
item.cast::<PyStructField>().cloned().map_err(|_| {
@@ -1979,6 +2015,10 @@ pub struct PyStructField {
/// is the metadata of the field. `field.metadata` is only its snapshot at
the time it was
/// made, so the metadata is read through `current_field`.
metadata: Py<pyo3::types::PyDict>,
+ /// The `needConversion` of the type, kept while the type cannot change:
an atomic type
+ /// object is not changed in place, so `add` need not read it again for
each field. Reset by
+ /// a set `dataType`.
+ need_conversion_of_type: std::sync::OnceLock<bool>,
}
impl PyStructField {
@@ -1989,6 +2029,7 @@ impl PyStructField {
field,
metadata,
data_type_object: KeptType::default(),
+ need_conversion_of_type: std::sync::OnceLock::new(),
})
}
@@ -2009,9 +2050,47 @@ impl PyStructField {
field,
metadata,
data_type_object,
+ need_conversion_of_type: std::sync::OnceLock::new(),
})
}
+ /// The `needConversion` of the type as it is now (see
`need_conversion_of_type`).
+ fn need_conversion_now(&self, py: Python<'_>) -> PyResult<bool> {
+ if let Some(&need) = self.need_conversion_of_type.get() {
+ return Ok(need);
+ }
+ // The type is read from its object when the field has one, without
reading the fields
+ // of a nested struct, or else from the snapshot. The value is kept
only for a type that
+ // does not change: not a container, which can be edited in place, nor
a year-month
+ // interval, whose value follows `PYSPARK_YM_INTERVAL_LEGACY`.
+ let (need, fixed) = match self.data_type_object.kept(py) {
+ Some(object) => {
+ let object = object.bind(py);
+ let fixed = !(object.is_instance_of::<PyStructType>()
+ || object.is_instance_of::<PyArrayType>()
+ || object.is_instance_of::<PyMapType>()
+ || object.is_instance_of::<PyYearMonthIntervalType>()
+ || is_udt(object)?);
+ (need_conversion_of_object(py, object)?, fixed)
+ }
+ None => {
+ let data_type = &self.field.data_type;
+ let fixed = !(matches!(
+ data_type,
+ DataType::Array { .. }
+ | DataType::Map { .. }
+ | DataType::Struct { .. }
+ | DataType::YearMonthInterval { .. }
+ ) || has_udt(data_type));
+ (need_conversion(py, data_type)?, fixed)
+ }
+ };
+ if fixed {
+ let _ = self.need_conversion_of_type.set(need);
+ }
+ Ok(need)
+ }
+
/// The core field, with the metadata of the dict as it is now.
pub(crate) fn current_field(&self, py: Python<'_>) ->
PyResult<StructField> {
let mut field = self.field.clone();
@@ -2223,6 +2302,7 @@ impl PyStructField {
fn set_data_type(&mut self, dataType: &Bound<'_, PyAny>) -> PyResult<()> {
self.field.data_type = py_to_data_type(dataType)?;
self.data_type_object = KeptType::of(dataType);
+ self.need_conversion_of_type = std::sync::OnceLock::new();
Ok(())
}
/// The field's metadata dict (mirrors `StructField.metadata`; values may
be any type).
@@ -2461,17 +2541,22 @@ impl PyStructType {
)?;
(object.into_bound(py).into_any(), name)
};
- {
+ // Only the types are read here, not the metadata: `add` is then
linear in the fields
+ // (a loop of adds with metadata would otherwise encode every field's
metadata each time).
+ let new_field = as_struct_field(&object)?.borrow().current_field(py)?;
+ let need_conversion = {
let me = slf.borrow();
me.fields.bind(py).append(object)?;
me.names.bind(py).append(name)?;
- }
- // The base `DataType` is what `resolve_datatype` reads for the
schemas of `mapInArrow`
- // and friends, so it must follow the fields too.
- let fields = slf.borrow().current_fields(py)?;
- let need_conversion = need_conversion_of(py, &fields)?;
+ need_conversion_of_fields(py, me.fields.bind(py))?
+ };
+ // The base `DataType` is a snapshot, which only the bare-base
fallback of
+ // `py_to_data_type` reads (a struct is read from its live fields), so
it takes the new
+ // field, as the fields of `add` do.
let mut me = slf.borrow_mut();
- me.as_super().inner = DataType::Struct { fields };
+ if let DataType::Struct { fields } = &mut me.as_super().inner {
+ fields.push(new_field);
+ }
me.need_conversion = need_conversion;
drop(me);
Ok(slf)
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 11e747e1..52eaa5ac 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4604,6 +4604,66 @@ def test_row_fields():
}
+def
test_struct_type_add_with_metadata_is_linear_and_matches_the_constructor(monkeypatch):
+ # As in PySpark, whose add reads no field of the struct but the new one: a
loop of adds with
+ # metadata is quick, also of fields of nested types, and its struct is the
one the
+ # constructor gives. The bounds are generous: the add that read every
field and its metadata
+ # on each call took about 17 s for the 2000 fields, and 10 s for the 1500
nested ones.
+ count = 2000
+ started = time.perf_counter()
+ st = T.StructType()
+ for i in range(count):
+ st.add(f"c{i}", T.StringType(), True, {"comment": f"col {i}"})
+ elapsed = time.perf_counter() - started
+ assert elapsed < 3.0, elapsed
+ fields = [
+ T.StructField(f"c{i}", T.StringType(), True, {"comment": f"col {i}"})
+ for i in range(count)
+ ]
+ expected = T.StructType(fields)
+ assert st == expected and st.json() == expected.json()
+ assert st.simpleString() == expected.simpleString()
+ assert st._needConversion == expected._needConversion
+ assert st.names == expected.names and st.fieldNames() ==
expected.fieldNames()
+ # A set dataType is seen by the next add, for all fields, as in PySpark's
add.
+ st["c0"].dataType = T.DateType()
+ st.add("d", T.IntegerType())
+ assert st._needConversion[0] is True and st._needConversion[-1] is False
+ assert st._needSerializeAnyField is True
+
+ # Fields of a struct, an array and a map of a struct, and an array of a
long, with metadata
+ # in the nested struct too.
+ def nested_type(i):
+ inner = T.StructType([T.StructField("x", T.IntegerType(), True,
{"comment": f"x {i}"})])
+ return [inner, T.ArrayType(inner), T.MapType(T.StringType(), inner)][i
% 3]
+
+ started = time.perf_counter()
+ nested = T.StructType()
+ for i in range(1500):
+ nested.add(f"n{i}", nested_type(i), True, {"comment": f"col {i}"})
+ nested.add("longs", T.ArrayType(T.LongType()))
+ elapsed = time.perf_counter() - started
+ assert elapsed < 3.0, elapsed
+ expected = T.StructType(
+ [T.StructField(f"n{i}", nested_type(i), True, {"comment": f"col {i}"})
for i in range(1500)]
+ + [T.StructField("longs", T.ArrayType(T.LongType()))]
+ )
+ assert nested == expected and nested.json() == expected.json()
+ assert nested._needConversion == expected._needConversion
+ assert nested._needConversion[-2:] == [True, False]
+
+ # The needConversion of a year-month interval follows
PYSPARK_YM_INTERVAL_LEGACY at each add.
+ monkeypatch.delenv("PYSPARK_YM_INTERVAL_LEGACY", raising=False)
+ ym = T.StructType().add("y", T.YearMonthIntervalType())
+ assert ym._needConversion == [True]
+ monkeypatch.setenv("PYSPARK_YM_INTERVAL_LEGACY", "1")
+ ym.add("z", T.IntegerType())
+ assert ym._needConversion == [False, False]
+ monkeypatch.delenv("PYSPARK_YM_INTERVAL_LEGACY")
+ ym.add("w", T.IntegerType())
+ assert ym._needConversion == [True, False, False]
+
+
def test_row_as_dict():
from pyspark.errors import PySparkTypeError
from pyspark.sql import Row
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]