This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 016b342d [SPARK-61221][CONNECT][PYTHON] Raise ValueError for a
non-nullable null type in createDataFrame of local rows, as PySpark does,
instead of a panic or an Arrow error
016b342d is described below
commit 016b342defe90609bbc6513c2ee459ef6bd72f85
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 17:32:00 2026 +0900
[SPARK-61221][CONNECT][PYTHON] Raise ValueError for a non-nullable null
type in createDataFrame of local rows, as PySpark does, instead of a panic or
an Arrow error
### What changes were proposed in this pull request?
[SPARK-61221](https://issues.apache.org/jira/browse/SPARK-61221): raise
`ValueError` for a non-nullable null type in `createDataFrame` of local rows,
as PySpark does, instead of a panic or a raw Arrow error.
- A non-empty `createDataFrame` of local rows now raises `ValueError: A
null type field may not be non-nullable` when the schema has a null type that
is not nullable, at any depth: a struct field, an array element
(`ArrayType(NullType(), False)`), a map value (`MapType(..., NullType(),
False)`) or a map key. PySpark raises it when its Arrow schema conversion
builds the field with `pa.field(name, pa.null(), nullable=False)`, after the
values are converted and before any data is sent, s [...]
- The Rust core builds list arrays with `ListArray::try_new`, so a
non-nullable element that has nulls is an error (`list build: ...`) instead of
a panic from `unwrap()` on `ListArray::new`.
Before this change, the list of a non-nullable NullType element with a
`None` panicked the client (a `PanicException`, which `except Exception` does
not catch). A map value or a struct field of such a type raised
`PySparkRuntimeError` with the raw Arrow text. An empty or `None` value was
accepted silently, including `createDataFrame(spark.sql("SELECT
array()").collect(), schema)`.
Checked against official PySpark 4.3.0 with the same probes (rows, pandas,
Arrow tables, NumPy arrays, with and without data): the exception class, the
message and the cases that raise now match. The pandas and Arrow paths raise
from PyArrow, as before, and the NumPy path raises the same server error as
before.
### Why are the changes needed?
A non-nullable NullType with data is not a valid Arrow field, so the client
must reject it as PySpark does. Repro:
```python
spark.createDataFrame([Row(a=[None])], StructType([StructField("a",
ArrayType(NullType(), False))]))
```
On master this panics the Python process (`pyo3_runtime.PanicException:
called Result::unwrap() on an Err value: InvalidArgumentError("Non-nullable
field of ListArray \"element\" cannot contain nulls")`). Official PySpark 4.3.0
raises `ValueError: A null type field may not be non-nullable`. The same schema
with `Row(s=Row(x=None))` (a non-nullable NullType struct field) or `{"k":
None}` (a map value) raised `PySparkRuntimeError: struct build: Invalid
argument error: Found unmasked nul [...]
### Does this PR introduce _any_ user-facing change?
Yes. `createDataFrame` of non-empty local rows with a non-nullable NullType
raises `ValueError` ("A null type field may not be non-nullable") instead of
panicking or raising `PySparkRuntimeError`, and an empty or `None` value no
longer passes silently for it. Nullable NullType schemas and empty data are
unchanged.
### How was this patch tested?
- `test_non_nullable_null_type_is_rejected_before_the_rpc_as_pyspark`
(offline, mock Connect server): nine rejected schemas (struct field, array
element, array of array, array of struct, map value, map key, struct in a map
value) raise `ValueError` with the message, and send no Execute RPC; accepted
schemas (empty data, nullable element, nullable field) are not rejected. The
test fails on the code before this change (the check was disabled to confirm).
- `a_non_nullable_element_with_nulls_is_an_error_not_a_panic` (Rust,
`apache-spark-connect` lib): a list with a null element of a non-nullable
NullType element returns an error naming `list build`, and an empty list is
accepted. The test fails on the code before this change (the panic from
`ListArray::new`).
- The same probe matrix (28 cases over rows, pandas, Arrow tables, NumPy
arrays, and the server `SELECT array()` schema) gives identical results on
official PySpark 4.3.0 and on the drop-in, for class, message and success.
- The lib tests of both crates (659 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2704 passed, 8 skipped), and the
live drop-in and Arrow suites (224 passed) and live cases (12 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #409 from HyukjinKwon/SPARK-61221-fu.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/infer.rs | 32 +++++++++++++++++++++++++++++
crates/pyspark-rs/src/session.rs | 9 ++++++--
crates/spark-connect/src/session.rs | 27 ++++++++++++++++++++++--
python/tests/test_dropin_offline.py | 41 +++++++++++++++++++++++++++++++++++++
4 files changed, 105 insertions(+), 4 deletions(-)
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index c2e51146..4e4c23d6 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -1406,6 +1406,38 @@ fn date_days(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
Ok(c_int_from_python(obj, 32, Some("Integer too large for date32"))? as
i32)
}
+/// Whether a field of `fields`, or a type in one, is a null type that is not
nullable. PyArrow
+/// rejects such a field when it builds it, `pa.field(name, pa.null(),
nullable=False)`, which
+/// PySpark's `to_arrow_schema` does. A map key is never nullable.
+pub(crate) fn has_non_nullable_null_type(fields: &[StructField]) -> bool {
+ fn in_type(dt: &DataType) -> bool {
+ match dt {
+ DataType::Struct { fields } => has_non_nullable_null_type(fields),
+ DataType::Array {
+ element_type,
+ contains_null,
+ } => {
+ (!contains_null && matches!(**element_type, DataType::Null))
+ || in_type(element_type)
+ }
+ DataType::Map {
+ key_type,
+ value_type,
+ value_contains_null,
+ } => {
+ matches!(**key_type, DataType::Null)
+ || (!value_contains_null && matches!(**value_type,
DataType::Null))
+ || in_type(key_type)
+ || in_type(value_type)
+ }
+ _ => false,
+ }
+ }
+ fields
+ .iter()
+ .any(|f| (!f.nullable && matches!(f.data_type, DataType::Null)) ||
in_type(&f.data_type))
+}
+
/// The `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` of the first char,
varchar, year-month or
/// calendar interval type of `dt`, depth first, which PySpark's
`to_arrow_schema` raises when it
/// converts the schema of local rows to Arrow: it has no Arrow type of these
types.
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 81f120a1..6d51ded1 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -886,8 +886,13 @@ impl PySparkSession {
}
}
}
- // Then, as PySpark, the schema converts to Arrow, which has no char,
varchar, year-month or
- // calendar interval type.
+ // Then, as PySpark, the schema converts to Arrow, which rejects a
null type that is not
+ // nullable, and has no char, varchar, year-month or calendar interval
type.
+ if infer::has_non_nullable_null_type(&fields) {
+ return Err(pyo3::exceptions::PyValueError::new_err(
+ "A null type field may not be non-nullable",
+ ));
+ }
for field in &fields {
if let Some(err) = infer::unsupported_arrow_type(py,
&field.data_type)? {
return Err(err);
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index aafc34c6..8e86a7fd 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -1712,12 +1712,15 @@ fn values_to_arrow(values: &[&Value], dt: &DataType) ->
Result<Arc<dyn arrow::ar
datatype_to_arrow(element_type)?,
*contains_null,
));
- Ok(Arc::new(ListArray::new(
+ // A non-nullable element that has nulls is an error, not a panic,
as for a struct.
+ ListArray::try_new(
field,
OffsetBuffer::new(offsets.into()),
child_arr,
Some(NullBuffer::new(valid.finish())),
- )))
+ )
+ .map(|array| Arc::new(array) as Arc<dyn Array>)
+ .map_err(|e| SparkError::connect_msg(format!("list build: {e}")))
}
// The struct of a VARIANT's value and metadata, and of a GEOMETRY's
or a GEOGRAPHY's
// srid and WKB, as `datatype_to_arrow` declares; their fields are not
nullable, so a
@@ -2559,6 +2562,26 @@ mod tests {
);
}
+ #[test]
+ fn a_non_nullable_element_with_nulls_is_an_error_not_a_panic() {
+ // An array of a non-nullable NullType that has a null element: Arrow
rejects the
+ // element field, which must be an error, as for a struct field with
nulls.
+ let array = DataType::Array {
+ element_type: Box::new(DataType::Null),
+ contains_null: false,
+ };
+ let value = Value::List(vec![Value::Null]);
+ let error = values_to_arrow(&[&value], &array)
+ .map(|_| ())
+ .unwrap_err()
+ .to_string();
+ assert!(error.contains("list build"), "{error}");
+ assert!(error.contains("cannot contain nulls"), "{error}");
+ // An empty list has no null element, which Arrow accepts.
+ let empty = Value::List(vec![]);
+ assert!(values_to_arrow(&[&empty], &array).is_ok());
+ }
+
#[test]
fn struct_arrow_encoding_preserves_lookup_map_and_null_semantics() {
use arrow::array::{Array, Int16Array, Int32Array, Int64Array,
StructArray};
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 43b2723b..23550788 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -10362,6 +10362,47 @@ def test_decimal_rescaling_follows_pyspark(
assert not payload.exists()
+def
test_non_nullable_null_type_is_rejected_before_the_rpc_as_pyspark(mock_connect):
+ # As PySpark's to_arrow_schema, which local rows of a non-empty
createDataFrame go through:
+ # a null type that is not nullable, as a field, an array element, a map
value or a map key,
+ # at any depth, raises ValueError before anything is sent, never a panic
or an Arrow error.
+ # Empty data is not converted, and a nullable null type is accepted.
+ spark, service = mock_connect
+ null = T.NullType
+
+ def struct(*fields):
+ return T.StructType([T.StructField(name, typ, nullable) for name, typ,
nullable in fields])
+
+ rejected = [
+ ([(None,)], struct(("a", null(), False))),
+ ([Row(a=[])], struct(("a", T.ArrayType(null(), False), True))),
+ ([Row(a=[None])], struct(("a", T.ArrayType(null(), False), True))),
+ ([Row(a=[[]])], struct(("a", T.ArrayType(T.ArrayType(null(), False)),
True))),
+ ([Row(a=[Row(x=None)])], struct(("a", T.ArrayType(struct(("x", null(),
False))), True))),
+ ([Row(m={})], struct(("m", T.MapType(T.StringType(), null(), False),
True))),
+ ([Row(m={"k": None})], struct(("m", T.MapType(T.StringType(), null(),
False), True))),
+ ([Row(m={})], struct(("m", T.MapType(null(), T.StringType(), True),
True))),
+ ([Row(s=Row(x=None))], struct(("s", struct(("x", null(), False)),
True))),
+ ]
+ for data, schema in rejected:
+ calls = service.execute_calls
+ with pytest.raises(ValueError) as error:
+ spark.createDataFrame(data, schema)
+ assert type(error.value) is ValueError
+ assert str(error.value) == "A null type field may not be non-nullable"
+ assert service.execute_calls == calls
+ accepted = [
+ ([], struct(("a", null(), False))),
+ ([Row(a=[])], struct(("a", T.ArrayType(null(), True), True))),
+ ([Row(a=[None])], struct(("a", T.ArrayType(null(), True), True))),
+ ([(None,)], struct(("a", null(), True))),
+ ]
+ calls = service.execute_calls
+ for data, schema in accepted:
+ spark.createDataFrame(data, schema)
+ assert service.execute_calls == calls
+
+
@pytest.mark.parametrize("kind", ["tuples", "pandas"])
@pytest.mark.parametrize("schema_mode", ["explicit", "names", "inferred"])
@pytest.mark.parametrize("texts", [
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]