This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 60a669e0 [SPARK-61156][CONNECT][PYTHON] Raise 
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION for year-month and calendar 
intervals in createDataFrame, in the Python client as PySpark does
60a669e0 is described below

commit 60a669e069f64da721992814a9c2c1ac1607af62
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:18:28 2026 +0900

    [SPARK-61156][CONNECT][PYTHON] Raise 
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION for year-month and calendar 
intervals in createDataFrame, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61156](https://issues.apache.org/jira/browse/SPARK-61156): Raise 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` for year-month and calendar 
interval columns of `createDataFrame` on the client, as PySpark does.
    
    PySpark's local-rows conversion has no converter for 
`YearMonthIntervalType` and `CalendarIntervalType`, so it passes their values 
through to `to_arrow_schema`, which raises a `PySparkTypeError` with 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` and the type's `str`, for example 
`YearMonthIntervalType(0, 1)` or `CalendarIntervalType()`. The Python client 
already did this for char and varchar types. This change does the same for the 
two interval types:
    
    - `convert` passes a value of these types through, as it does for char and 
varchar, so the schema check runs for None values and for other values alike.
    - `unsupported_arrow_type` also matches these types, so the check raises 
the PySpark error before the server core sees the schema, for top-level, nested 
in struct, array and map, and atomic schemas.
    
    ### Why are the changes needed?
    
    With a year-month interval column, `spark.createDataFrame([(1,)], "a 
interval year to month")` raised `TypeError: unsupported value type for 
createDataFrame: int`. With `[(None,)]` it raised `PySparkRuntimeError: 
Unsupported type for Arrow createDataFrame conversion: YearMonthInterval { 
start_field: 0, end_field: 1 }`, which is Rust Debug text. A calendar interval 
column gave the same two errors with `CalendarInterval`. PySpark raises 
`PySparkTypeError` `UNSUPPORTED_DATA_TYPE_FOR_ARRO [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `createDataFrame` of local rows with a year-month or calendar interval 
column now raises `PySparkTypeError` 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`, with the same message as PySpark, 
instead of a `TypeError` or a `PySparkRuntimeError` with Rust text.
    
    ### How was this patch tested?
    
    - `test_year_month_and_calendar_intervals_raise_as_pyspark_for_local_rows` 
(offline, mock server): values and None of top-level, struct, nested calendar, 
array and map year-month and calendar columns, and atomic year-month schemas, 
check the error class, the condition and the `data_type` parameter. Each case 
gives the same result with official PySpark 4.3.0.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and 
the live tests of intervals, char types and createDataFrame (49 passed) against 
a local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics 
in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #344 from HyukjinKwon/SPARK-61156-interval-types-arrow-conversion.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/infer.rs      | 32 ++++++++++++++++++++------------
 crates/pyspark-rs/src/session.rs    |  3 ++-
 python/tests/test_dropin_offline.py | 30 ++++++++++++++++++++++++++++++
 3 files changed, 52 insertions(+), 13 deletions(-)

diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 7b0bc7af..0234547d 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -1161,7 +1161,10 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
         }
         // As PySpark, which has no converter of these types: the value goes 
as it is to
         // `to_arrow_schema`, which raises for the type 
(`unsupported_arrow_type`).
-        DataType::Char { .. } | DataType::Varchar { .. } => Ok(Value::Null),
+        DataType::Char { .. }
+        | DataType::Varchar { .. }
+        | DataType::YearMonthInterval { .. }
+        | DataType::CalendarInterval => Ok(Value::Null),
         DataType::Boolean => {
             if obj.is_instance_of::<PyBool>() || numpy_scalar(obj)? == 
Some(NumpyScalar::Bool) {
                 Ok(Value::Bool(obj.is_truthy()?))
@@ -1286,29 +1289,34 @@ fn date_days(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
     Ok(c_int_from_python(obj, 32, Some("Integer too large for date32"))? as 
i32)
 }
 
-/// The `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` of the first char or 
varchar type of `dt`,
-/// depth first, which PySpark's `to_arrow_schema` raises when it converts the 
schema of local
-/// rows to Arrow: it has no Arrow type of these types.
+/// The `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` of the first char, 
varchar, year-month or
+/// calendar interval type of `dt`, depth first, which PySpark's 
`to_arrow_schema` raises when it
+/// converts the schema of local rows to Arrow: it has no Arrow type of these 
types.
 pub(crate) fn unsupported_arrow_type(py: Python<'_>, dt: &DataType) -> 
PyResult<Option<PyErr>> {
-    fn first_char(dt: &DataType) -> Option<&DataType> {
+    fn first_unsupported(dt: &DataType) -> Option<&DataType> {
         match dt {
-            DataType::Char { .. } | DataType::Varchar { .. } => Some(dt),
-            DataType::Struct { fields } => fields.iter().find_map(|f| 
first_char(&f.data_type)),
-            DataType::Array { element_type, .. } => first_char(element_type),
+            DataType::Char { .. }
+            | DataType::Varchar { .. }
+            | DataType::YearMonthInterval { .. }
+            | DataType::CalendarInterval => Some(dt),
+            DataType::Struct { fields } => {
+                fields.iter().find_map(|f| first_unsupported(&f.data_type))
+            }
+            DataType::Array { element_type, .. } => 
first_unsupported(element_type),
             DataType::Map {
                 key_type,
                 value_type,
                 ..
-            } => first_char(key_type).or_else(|| first_char(value_type)),
+            } => first_unsupported(key_type).or_else(|| 
first_unsupported(value_type)),
             _ => None,
         }
     }
-    Ok(match first_char(dt) {
-        Some(char_type) => Some(raise_pyspark(
+    Ok(match first_unsupported(dt) {
+        Some(unsupported) => Some(raise_pyspark(
             py,
             "PySparkTypeError",
             "UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION",
-            &[("data_type", &data_type_str(py, char_type)?)],
+            &[("data_type", &data_type_str(py, unsupported)?)],
         )),
         None => None,
     })
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index a5ab6d16..9e05146f 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -700,7 +700,8 @@ impl PySparkSession {
                 out.push(infer::convert(&row[i], &field.data_type, 
field.nullable)?);
             }
         }
-        // Then, as PySpark, the schema converts to Arrow, which has no char 
or varchar type.
+        // Then, as PySpark, the schema converts to Arrow, which has no char, 
varchar, year-month or
+        // calendar interval type.
         for field in &fields {
             if let Some(err) = infer::unsupported_arrow_type(py, 
&field.data_type)? {
                 return Err(err);
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index c7a299c2..35a45aff 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7586,6 +7586,36 @@ def exact_date_capture_input():
         child.close()
 
 
+def 
test_year_month_and_calendar_intervals_raise_as_pyspark_for_local_rows(mock_connect):
+    # As PySpark's to_arrow_schema, which has no Arrow type of these 
intervals: the values
+    # and the None values of such a column raise 
UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION,
+    # top-level, nested and atomic alike, rather than a TypeError or a Rust 
error text.
+    from pyspark.errors import PySparkTypeError
+
+    spark, _ = mock_connect
+    cases = [
+        ([(None,)], "a interval year to month", "YearMonthIntervalType(0, 1)"),
+        ([(1,)], "a interval year to month", "YearMonthIntervalType(0, 1)"),
+        ([(None,)], T.StructType([T.StructField("a", 
T.CalendarIntervalType())]),
+         "CalendarIntervalType()"),
+        ([(1,)], T.StructType([T.StructField("a", T.CalendarIntervalType())]),
+         "CalendarIntervalType()"),
+        ([(None,)], T.StructType([T.StructField("a", T.StructType(
+            [T.StructField("x", T.CalendarIntervalType())]))]), 
"CalendarIntervalType()"),
+        ([([1],)], T.StructType([T.StructField("a", 
T.ArrayType(T.CalendarIntervalType()))]),
+         "CalendarIntervalType()"),
+        ([(None,)], T.StructType([T.StructField("a", 
T.YearMonthIntervalType())]),
+         "YearMonthIntervalType(0, 1)"),
+        ([([None],)], "a array<interval year to month>", 
"YearMonthIntervalType(0, 1)"),
+        ([1], T.YearMonthIntervalType(), "YearMonthIntervalType(0, 1)"),
+        ([None], T.YearMonthIntervalType(), "YearMonthIntervalType(0, 1)"),
+    ]
+    for data, schema, data_type in cases:
+        with pytest.raises(PySparkTypeError) as e:
+            spark.createDataFrame(data, schema)
+        assert e.value.getCondition() == 
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+        assert e.value.getMessageParameters() == {"data_type": data_type}
+
 @pytest.mark.parametrize("source", ["tuples", "pandas"])
 @pytest.mark.parametrize("schema_kind", ["inferred", "names", "explicit", 
"ddl"])
 def test_exact_date_ipc_calendar_range(exact_date_capture_input, source, 
schema_kind):


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to