This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 60a669e0 [SPARK-61156][CONNECT][PYTHON] Raise
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION for year-month and calendar
intervals in createDataFrame, in the Python client as PySpark does
60a669e0 is described below
commit 60a669e069f64da721992814a9c2c1ac1607af62
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:18:28 2026 +0900
[SPARK-61156][CONNECT][PYTHON] Raise
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION for year-month and calendar
intervals in createDataFrame, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61156](https://issues.apache.org/jira/browse/SPARK-61156): Raise
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` for year-month and calendar
interval columns of `createDataFrame` on the client, as PySpark does.
PySpark's local-rows conversion has no converter for
`YearMonthIntervalType` and `CalendarIntervalType`, so it passes their values
through to `to_arrow_schema`, which raises a `PySparkTypeError` with
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` and the type's `str`, for example
`YearMonthIntervalType(0, 1)` or `CalendarIntervalType()`. The Python client
already did this for char and varchar types. This change does the same for the
two interval types:
- `convert` passes a value of these types through, as it does for char and
varchar, so the schema check runs for None values and for other values alike.
- `unsupported_arrow_type` also matches these types, so the check raises
the PySpark error before the server core sees the schema, for top-level, nested
in struct, array and map, and atomic schemas.
### Why are the changes needed?
With a year-month interval column, `spark.createDataFrame([(1,)], "a
interval year to month")` raised `TypeError: unsupported value type for
createDataFrame: int`. With `[(None,)]` it raised `PySparkRuntimeError:
Unsupported type for Arrow createDataFrame conversion: YearMonthInterval {
start_field: 0, end_field: 1 }`, which is Rust Debug text. A calendar interval
column gave the same two errors with `CalendarInterval`. PySpark raises
`PySparkTypeError` `UNSUPPORTED_DATA_TYPE_FOR_ARRO [...]
### Does this PR introduce _any_ user-facing change?
Yes. `createDataFrame` of local rows with a year-month or calendar interval
column now raises `PySparkTypeError`
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`, with the same message as PySpark,
instead of a `TypeError` or a `PySparkRuntimeError` with Rust text.
### How was this patch tested?
- `test_year_month_and_calendar_intervals_raise_as_pyspark_for_local_rows`
(offline, mock server): values and None of top-level, struct, nested calendar,
array and map year-month and calendar columns, and atomic year-month schemas,
check the error class, the condition and the `data_type` parameter. Each case
gives the same result with official PySpark 4.3.0.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and
the live tests of intervals, char types and createDataFrame (49 passed) against
a local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics
in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #344 from HyukjinKwon/SPARK-61156-interval-types-arrow-conversion.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/infer.rs | 32 ++++++++++++++++++++------------
crates/pyspark-rs/src/session.rs | 3 ++-
python/tests/test_dropin_offline.py | 30 ++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+), 13 deletions(-)
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 7b0bc7af..0234547d 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -1161,7 +1161,10 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
}
// As PySpark, which has no converter of these types: the value goes
as it is to
// `to_arrow_schema`, which raises for the type
(`unsupported_arrow_type`).
- DataType::Char { .. } | DataType::Varchar { .. } => Ok(Value::Null),
+ DataType::Char { .. }
+ | DataType::Varchar { .. }
+ | DataType::YearMonthInterval { .. }
+ | DataType::CalendarInterval => Ok(Value::Null),
DataType::Boolean => {
if obj.is_instance_of::<PyBool>() || numpy_scalar(obj)? ==
Some(NumpyScalar::Bool) {
Ok(Value::Bool(obj.is_truthy()?))
@@ -1286,29 +1289,34 @@ fn date_days(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
Ok(c_int_from_python(obj, 32, Some("Integer too large for date32"))? as
i32)
}
-/// The `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` of the first char or
varchar type of `dt`,
-/// depth first, which PySpark's `to_arrow_schema` raises when it converts the
schema of local
-/// rows to Arrow: it has no Arrow type of these types.
+/// The `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` of the first char,
varchar, year-month or
+/// calendar interval type of `dt`, depth first, which PySpark's
`to_arrow_schema` raises when it
+/// converts the schema of local rows to Arrow: it has no Arrow type of these
types.
pub(crate) fn unsupported_arrow_type(py: Python<'_>, dt: &DataType) ->
PyResult<Option<PyErr>> {
- fn first_char(dt: &DataType) -> Option<&DataType> {
+ fn first_unsupported(dt: &DataType) -> Option<&DataType> {
match dt {
- DataType::Char { .. } | DataType::Varchar { .. } => Some(dt),
- DataType::Struct { fields } => fields.iter().find_map(|f|
first_char(&f.data_type)),
- DataType::Array { element_type, .. } => first_char(element_type),
+ DataType::Char { .. }
+ | DataType::Varchar { .. }
+ | DataType::YearMonthInterval { .. }
+ | DataType::CalendarInterval => Some(dt),
+ DataType::Struct { fields } => {
+ fields.iter().find_map(|f| first_unsupported(&f.data_type))
+ }
+ DataType::Array { element_type, .. } =>
first_unsupported(element_type),
DataType::Map {
key_type,
value_type,
..
- } => first_char(key_type).or_else(|| first_char(value_type)),
+ } => first_unsupported(key_type).or_else(||
first_unsupported(value_type)),
_ => None,
}
}
- Ok(match first_char(dt) {
- Some(char_type) => Some(raise_pyspark(
+ Ok(match first_unsupported(dt) {
+ Some(unsupported) => Some(raise_pyspark(
py,
"PySparkTypeError",
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION",
- &[("data_type", &data_type_str(py, char_type)?)],
+ &[("data_type", &data_type_str(py, unsupported)?)],
)),
None => None,
})
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index a5ab6d16..9e05146f 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -700,7 +700,8 @@ impl PySparkSession {
out.push(infer::convert(&row[i], &field.data_type,
field.nullable)?);
}
}
- // Then, as PySpark, the schema converts to Arrow, which has no char
or varchar type.
+ // Then, as PySpark, the schema converts to Arrow, which has no char,
varchar, year-month or
+ // calendar interval type.
for field in &fields {
if let Some(err) = infer::unsupported_arrow_type(py,
&field.data_type)? {
return Err(err);
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index c7a299c2..35a45aff 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7586,6 +7586,36 @@ def exact_date_capture_input():
child.close()
+def
test_year_month_and_calendar_intervals_raise_as_pyspark_for_local_rows(mock_connect):
+ # As PySpark's to_arrow_schema, which has no Arrow type of these
intervals: the values
+ # and the None values of such a column raise
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION,
+ # top-level, nested and atomic alike, rather than a TypeError or a Rust
error text.
+ from pyspark.errors import PySparkTypeError
+
+ spark, _ = mock_connect
+ cases = [
+ ([(None,)], "a interval year to month", "YearMonthIntervalType(0, 1)"),
+ ([(1,)], "a interval year to month", "YearMonthIntervalType(0, 1)"),
+ ([(None,)], T.StructType([T.StructField("a",
T.CalendarIntervalType())]),
+ "CalendarIntervalType()"),
+ ([(1,)], T.StructType([T.StructField("a", T.CalendarIntervalType())]),
+ "CalendarIntervalType()"),
+ ([(None,)], T.StructType([T.StructField("a", T.StructType(
+ [T.StructField("x", T.CalendarIntervalType())]))]),
"CalendarIntervalType()"),
+ ([([1],)], T.StructType([T.StructField("a",
T.ArrayType(T.CalendarIntervalType()))]),
+ "CalendarIntervalType()"),
+ ([(None,)], T.StructType([T.StructField("a",
T.YearMonthIntervalType())]),
+ "YearMonthIntervalType(0, 1)"),
+ ([([None],)], "a array<interval year to month>",
"YearMonthIntervalType(0, 1)"),
+ ([1], T.YearMonthIntervalType(), "YearMonthIntervalType(0, 1)"),
+ ([None], T.YearMonthIntervalType(), "YearMonthIntervalType(0, 1)"),
+ ]
+ for data, schema, data_type in cases:
+ with pytest.raises(PySparkTypeError) as e:
+ spark.createDataFrame(data, schema)
+ assert e.value.getCondition() ==
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+ assert e.value.getMessageParameters() == {"data_type": data_type}
+
@pytest.mark.parametrize("source", ["tuples", "pandas"])
@pytest.mark.parametrize("schema_kind", ["inferred", "names", "explicit",
"ddl"])
def test_exact_date_ipc_calendar_range(exact_date_capture_input, source,
schema_kind):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]