This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 01dbea1  [SPARK-61106][CONNECT][PYTHON] Convert the values of local 
rows of a schema, and raise their errors, as PyArrow does in createDataFrame in 
the Python client
01dbea1 is described below

commit 01dbea17b943dff96f0e4fa30919ac7392e893f4
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 18:53:00 2026 +0900

    [SPARK-61106][CONNECT][PYTHON] Convert the values of local rows of a 
schema, and raise their errors, as PyArrow does in createDataFrame in the 
Python client
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61106](https://issues.apache.org/jira/browse/SPARK-61106): 
`spark.createDataFrame` of local rows with a schema now converts their values, 
and raises their errors, as PyArrow does in PySpark Connect.
    
    PySpark Connect passes the values of most atomic types to PyArrow as they 
are. The client converts them in Rust (`crates/pyspark-rs/src/infer.rs`), 
emulating PyArrow. This change makes the emulation follow PyArrow's code:
    - An int, date, day-time interval or time field takes a value as PyArrow's 
`CIntFromPython` does:
      - an int, else the `__index__` of the value, else its `__int__` 
(`PyObjectToPyInt`), such as of a `Decimal`, a float or a NumPy scalar
      - a bool raises `ArrowTypeError` ("Expected integer, got bool")
      - a value out of range raises `ArrowInvalid` ("Value X too large to fit 
in C integer type", or "Integer too large for date32")
    - A NumPy scalar in a day-time interval field is taken only as a 
`timedelta64[us]`, as PyArrow's `duration[us]` takes it. Another NumPy scalar 
raises `ArrowNotImplementedError`.
    - The errors are PyArrow's classes, `ArrowInvalid`, `ArrowTypeError` and 
`ArrowNotImplementedError`, with PyArrow's messages:
      - "tried to convert to float32" for a float field
      - "PyLong is too large to fit int64" for an int beyond int64 in a float 
or double field
      - "was not a utf8 string" for bytes of a string field
      - Without PyArrow, they are the built-in classes these derive from.
    - A date field takes the days of any int32, as PyArrow does; it no longer 
stops at the dates of Python. `collect` turns days outside those dates into 
`date(1970, 1, 1) + timedelta(days=d)`, as PyArrow's `Date32Scalar.as_py` does. 
That raises its `OverflowError`: "date value out of range", or 
"days=1000000000; must have magnitude <= 999999999".
    - A `char` or `varchar` field raises 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`, after the values are converted, 
as PySpark's `to_arrow_schema` does: it has no Arrow type for these types.
    
    ### Why are the changes needed?
    
    Of 374 cases of a value of each kind in a field of each atomic type, 119 
differed from PySpark 4.3.0 on a local Spark 4.3.0 server. For example:
    
    | Input | master | This change and PySpark 4.3.0 |
    | --- | --- | --- |
    | `Decimal("1.5")` into `int`, `date` or `interval day to second` | 
`ValueError` or `ArrowTypeError` | 1, `1970-01-02`, 1 microsecond |
    | `"x"` into `int` | `ValueError: Could not convert 'x' ...` | 
`pyarrow.lib.ArrowInvalid: Could not convert 'x' ...` |
    | `True` into `int` | `TypeError` | `ArrowTypeError` |
    | `"x"` into `float` | `... tried to convert to float` | `... tried to 
convert to float32` |
    | `2**70` into `double` | `ValueError: Integer value ... is outside of the 
range ...` | `ArrowInvalid: PyLong is too large to fit int64` |
    | `2**40` into `date` | `OverflowError: days=...` | `ArrowInvalid: Integer 
too large for date32` |
    | `2932897` days into `date` | `OverflowError` in `createDataFrame` | 
created; `OverflowError: date value out of range` when collected |
    | `True` or `NaN` into `time` | taken | `ArrowTypeError`, `ValueError` |
    | bytes, or an int, into `char(3)` | the server's 
`UNSUPPORTED_CHAR_OR_VARCHAR_AS_STRING`, or `TypeError` | 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` |
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The values of local rows convert, or fail, as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test checks:
      - a Decimal in an int and a date field
      - the `ArrowInvalid` of a str in an int field
      - the error of a char field
      - the error of a date out of the dates of Python
      It passes with official PySpark 4.3.0 and fails without this change.
    - A new offline test checks the values sent for a Decimal in each kind of 
field and for a `timedelta64[us]`, the errors and their exact classes, and the 
error of char and varchar fields.
    - The test of ints in date fields expected the errors of days outside the 
dates of Python from `createDataFrame`. It now expects the days to be sent, as 
PySpark does.
    - On a local server, 401 cases give the same results as official PySpark 
4.3.0: 374 of a value of each kind into each atomic type, and 27 of char, 
varchar and nested fields, NumPy scalars and edge values. The only differences 
are the addresses of `object()`s in the messages.
    - The rows of the client convert their values when they are read. So the 
`OverflowError` of a date outside the dates of Python is raised when the value 
is read, where PySpark raises it in `collect`.
    - The lib tests of both crates (588 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2054 passed, 12 skipped), and the 
live drop-in and Arrow suites (158 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #304 from HyukjinKwon/SPARK-61106-local-row-conversion.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/infer.rs      | 343 ++++++++++++++++++------------------
 crates/pyspark-rs/src/row.rs        |  15 +-
 crates/pyspark-rs/src/session.rs    |   6 +
 python/tests/test_dropin_e2e.py     |  23 +++
 python/tests/test_dropin_offline.py |  58 +++++-
 5 files changed, 269 insertions(+), 176 deletions(-)

diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 319c4ce..4825c72 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -7,7 +7,9 @@
 use std::cell::OnceCell;
 use std::collections::{HashMap, HashSet};
 
-use pyo3::exceptions::{PyAssertionError, PyOverflowError, PyTypeError, 
PyValueError};
+use pyo3::exceptions::{
+    PyAssertionError, PyNotImplementedError, PyOverflowError, PyTypeError, 
PyValueError,
+};
 use pyo3::intern;
 use pyo3::prelude::*;
 use pyo3::sync::PyOnceLock;
@@ -841,22 +843,86 @@ fn arrow_type_name(obj: &Bound<'_, PyAny>) -> 
PyResult<String> {
     })
 }
 
-/// An error of PyArrow, of `class` with `message`, as its conversions raise 
it.
+/// An error of PyArrow, of `class` with `message`, as its conversions raise 
it: `ArrowInvalid`,
+/// a `ValueError`, `ArrowTypeError`, a `TypeError`, or 
`ArrowNotImplementedError`, a
+/// `NotImplementedError`. Without PyArrow, the built-in class that it derives 
from.
 fn pyarrow_error(py: Python<'_>, class: &str, message: String) -> 
PyResult<PyErr> {
-    let error = py
-        .import(intern!(py, "pyarrow"))?
-        .getattr(class)?
-        .call1((message,))?;
-    Ok(PyErr::from_value(error))
+    match py
+        .import(intern!(py, "pyarrow"))
+        .and_then(|pyarrow| pyarrow.getattr(class))
+    {
+        Ok(class) => Ok(PyErr::from_value(class.call1((message,))?)),
+        Err(_) => Ok(match class {
+            "ArrowTypeError" => PyTypeError::new_err(message),
+            "ArrowNotImplementedError" => 
PyNotImplementedError::new_err(message),
+            _ => PyValueError::new_err(message),
+        }),
+    }
 }
 
-/// The error of a value that PyArrow cannot convert to the type it is named 
by.
+/// The `ArrowInvalid` of a value that PyArrow cannot convert to the type it 
is named by.
 fn arrow_invalid(obj: &Bound<'_, PyAny>, arrow_type: &str) -> PyResult<PyErr> {
-    Ok(PyValueError::new_err(format!(
-        "Could not convert {} with type {}: tried to convert to {arrow_type}",
-        obj.repr()?,
-        arrow_type_name(obj)?
-    )))
+    pyarrow_error(
+        obj.py(),
+        "ArrowInvalid",
+        format!(
+            "Could not convert {} with type {}: tried to convert to 
{arrow_type}",
+            obj.repr()?,
+            arrow_type_name(obj)?
+        ),
+    )
+}
+
+/// `obj` as an int, as PyArrow's `PyObjectToPyInt` takes it: by `__index__`, 
else by the
+/// `__int__` of its type, which may raise, such as for a NaN; else the 
`ArrowTypeError` "object
+/// of type <class 'T'> cannot be converted to int". A float, a `Decimal` and 
a NumPy number or
+/// bool are taken by their `__int__`.
+fn py_int<'py>(obj: &Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
+    let py = obj.py();
+    // SAFETY: PyNumber_Index returns a new reference, or NULL with an 
exception set.
+    if let Ok(index) =
+        unsafe { Bound::from_owned_ptr_or_err(py, 
pyo3::ffi::PyNumber_Index(obj.as_ptr())) }
+    {
+        return Ok(index);
+    }
+    let ty = obj.get_type();
+    if ty.hasattr(intern!(py, "__int__"))? {
+        return ty.getattr(intern!(py, "__int__"))?.call1((obj,));
+    }
+    Err(pyarrow_error(
+        py,
+        "ArrowTypeError",
+        format!("object of type {} cannot be converted to int", ty.repr()?),
+    )?)
+}
+
+/// `obj` as an integer of `bits` bits, as PyArrow's `CIntFromPython` takes 
it: not a bool (an
+/// `ArrowTypeError`), an int, else by `py_int`. A value out of the range of 
the bits raises an
+/// `ArrowInvalid` of `overflow`, else "Value X too large to fit in C integer 
type", and one out
+/// of the range of a C long Python's `OverflowError`.
+fn c_int_from_python(obj: &Bound<'_, PyAny>, bits: u32, overflow: 
Option<&str>) -> PyResult<i64> {
+    let py = obj.py();
+    if obj.is_instance_of::<PyBool>() {
+        return Err(pyarrow_error(
+            py,
+            "ArrowTypeError",
+            "Expected integer, got bool".into(),
+        )?);
+    }
+    let int = if obj.is_instance_of::<PyInt>() {
+        obj.clone()
+    } else {
+        py_int(obj)?
+    };
+    let value: i64 = int.extract()?;
+    if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits - 
1))).contains(&value) {
+        let message = match overflow {
+            Some(message) => message.to_string(),
+            None => format!("Value {} too large to fit in C integer type", 
int.str()?),
+        };
+        return Err(pyarrow_error(py, "ArrowInvalid", message)?);
+    }
+    Ok(value)
 }
 
 /// The kind of a NumPy scalar of a bool or a number, which PyArrow converts 
as the Python
@@ -891,6 +957,18 @@ fn numpy_scalar(obj: &Bound<'_, PyAny>) -> 
PyResult<Option<NumpyScalar>> {
     Ok(None)
 }
 
+/// Whether `obj` is a NumPy scalar of any kind. NumPy is not imported when it 
is not loaded.
+fn numpy_generic(obj: &Bound<'_, PyAny>) -> PyResult<bool> {
+    let py = obj.py();
+    let modules = py
+        .import(intern!(py, "sys"))?
+        .getattr(intern!(py, "modules"))?;
+    match modules.cast::<PyDict>()?.get_item(intern!(py, "numpy"))? {
+        Some(numpy) if !numpy.is_none() => 
obj.is_instance(&numpy.getattr(intern!(py, "generic"))?),
+        _ => Ok(false),
+    }
+}
+
 /// An integer as a floating-point value, which PyArrow takes only when it is 
exact: within
 /// 2^53 for a double and 2^24 for a float.
 fn exact_float(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<f64> {
@@ -899,15 +977,25 @@ fn exact_float(obj: &Bound<'_, PyAny>, dt: &DataType) -> 
PyResult<f64> {
         _ => (53, "double"),
     };
     let out_of_range = || -> PyResult<PyErr> {
-        Ok(PyValueError::new_err(format!(
-            "Integer value {} is outside of the range exactly representable by 
a IEEE 754 \
-             {precision} precision value",
-            obj.str()?
-        )))
+        pyarrow_error(
+            obj.py(),
+            "ArrowInvalid",
+            format!(
+                "Integer value {} is outside of the range exactly 
representable by a IEEE 754 \
+                 {precision} precision value",
+                obj.str()?
+            ),
+        )
     };
     let value: i64 = match obj.extract() {
         Ok(value) => value,
-        Err(_) => return Err(out_of_range()?),
+        Err(_) => {
+            return Err(pyarrow_error(
+                obj.py(),
+                "ArrowInvalid",
+                "PyLong is too large to fit int64".into(),
+            )?)
+        }
     };
     if value.unsigned_abs() > 1u64 << bits {
         return Err(out_of_range()?);
@@ -1071,21 +1159,9 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
             };
             Ok(Value::String(text))
         }
-        DataType::Char { .. } | DataType::Varchar { .. } => {
-            if let Ok(text) = obj.cast::<PyString>() {
-                Ok(Value::String(text.to_cow()?.into_owned()))
-            } else if let Ok(bytes) = obj.cast::<PyBytes>() {
-                Ok(Value::String(
-                    String::from_utf8(bytes.as_bytes().to_vec())
-                        .map_err(|e| PyValueError::new_err(e.to_string()))?,
-                ))
-            } else {
-                Err(PyTypeError::new_err(format!(
-                    "Expected bytes, got a '{}' object",
-                    type_name(obj)?
-                )))
-            }
-        }
+        // As PySpark, which has no converter of these types: the value goes 
as it is to
+        // `to_arrow_schema`, which raises for the type 
(`unsupported_arrow_type`).
+        DataType::Char { .. } | DataType::Varchar { .. } => Ok(Value::Null),
         DataType::Boolean => {
             if obj.is_instance_of::<PyBool>() || numpy_scalar(obj)? == 
Some(NumpyScalar::Bool) {
                 Ok(Value::Bool(obj.is_truthy()?))
@@ -1109,7 +1185,7 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
                     Some(NumpyScalar::Integer) => exact_float(obj, dt)?,
                     _ => {
                         let arrow_type = if matches!(dt, DataType::Float) {
-                            "float"
+                            "float32"
                         } else {
                             "double"
                         };
@@ -1124,20 +1200,17 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
         }
         DataType::DayTimeInterval { .. } => 
Ok(Value::DayTimeInterval(duration_micros(obj)?)),
         DataType::Date => Ok(Value::Date(date_days(obj)?)),
-        // As PyArrow's `time64[ns]`, to which PySpark passes the value as it 
is: a time, and
-        // the nanoseconds of an int, or of a float, truncated.
+        // As PyArrow's `time64[ns]`, to which PySpark passes the value as it 
is: a time, else
+        // the nanoseconds of the value as `CIntFromPython` takes it.
         DataType::Time { .. } => {
             if obj.is_instance(TIME.import(py, "datetime", "time")?.as_any())? 
{
                 Ok(Value::Time(time_nanos(obj)?))
-            } else if obj.is_instance_of::<PyInt>() {
-                Ok(Value::Time(obj.extract()?))
-            } else if obj.is_instance_of::<PyFloat>() {
-                Ok(Value::Time(obj.extract::<f64>()? as i64))
             } else {
-                Err(PyTypeError::new_err(format!(
-                    "object of type {} cannot be converted to int",
-                    obj.get_type().repr()?
-                )))
+                Ok(Value::Time(c_int_from_python(
+                    obj,
+                    64,
+                    Some("Integer too large for int64"),
+                )?))
             }
         }
         // As PySpark's converters of these types: the value and metadata of a 
VariantVal, and
@@ -1202,71 +1275,43 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
 }
 
 /// The days since the epoch of a value of a date column, as PyArrow converts 
it to date32: a
-/// date, or the date of a datetime; an int, or a float truncated, as days; 
and a NumPy bool or
-/// number as the value it stands for. Another value raises PyArrow's error.
+/// date, or the date of a datetime, else the days of the value as 
`CIntFromPython` takes it, in
+/// the range of an int32, else the `ArrowInvalid` "Integer too large for 
date32".
 fn date_days(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
     let py = obj.py();
     if obj.is_instance(DATE.import(py, "datetime", "date")?.as_any())? {
         let ordinal: i64 = obj.call_method0(intern!(py, 
"toordinal"))?.extract()?;
         return Ok((ordinal - 719_163) as i32);
     }
-    if obj.is_instance_of::<PyBool>() {
-        return Err(pyarrow_error(
-            py,
-            "ArrowTypeError",
-            "Expected integer, got bool".into(),
-        )?);
-    }
-    let kind = if obj.is_instance_of::<PyInt>() {
-        Some(NumpyScalar::Integer)
-    } else if obj.is_instance_of::<PyFloat>() {
-        Some(NumpyScalar::Floating)
-    } else {
-        numpy_scalar(obj)?
-    };
-    let days: i64 = match kind {
-        Some(NumpyScalar::Bool) => i64::from(obj.is_truthy()?),
-        Some(NumpyScalar::Integer) => obj.extract()?,
-        Some(NumpyScalar::Floating) => {
-            let value: f64 = obj.extract()?;
-            if value.is_nan() {
-                return Err(PyValueError::new_err("cannot convert float NaN to 
integer"));
-            }
-            if value.is_infinite() {
-                return Err(PyOverflowError::new_err(
-                    "cannot convert float infinity to integer",
-                ));
-            }
-            let truncated = value.trunc();
-            if 
!(-9_223_372_036_854_775_808.0..9_223_372_036_854_775_808.0).contains(&truncated)
 {
-                return Err(PyOverflowError::new_err(
-                    "Python int too large to convert to C long",
-                ));
-            }
-            truncated as i64
-        }
-        None => {
-            return Err(pyarrow_error(
-                py,
-                "ArrowTypeError",
-                format!(
-                    "object of type {} cannot be converted to int",
-                    obj.get_type().repr()?
-                ),
-            )?)
+    Ok(c_int_from_python(obj, 32, Some("Integer too large for date32"))? as 
i32)
+}
+
+/// The `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` of the first char or 
varchar type of `dt`,
+/// depth first, which PySpark's `to_arrow_schema` raises when it converts the 
schema of local
+/// rows to Arrow: it has no Arrow type of these types.
+pub(crate) fn unsupported_arrow_type(py: Python<'_>, dt: &DataType) -> 
PyResult<Option<PyErr>> {
+    fn first_char(dt: &DataType) -> Option<&DataType> {
+        match dt {
+            DataType::Char { .. } | DataType::Varchar { .. } => Some(dt),
+            DataType::Struct { fields } => fields.iter().find_map(|f| 
first_char(&f.data_type)),
+            DataType::Array { element_type, .. } => first_char(element_type),
+            DataType::Map {
+                key_type,
+                value_type,
+                ..
+            } => first_char(key_type).or_else(|| first_char(value_type)),
+            _ => None,
         }
-    };
-    // The days of PyArrow's dates: from 0001-01-01 to 9999-12-31, and a 
larger magnitude is
-    // out of the range of a Python timedelta.
-    if days.unsigned_abs() > 999_999_999 {
-        return Err(PyOverflowError::new_err(format!(
-            "days={days}; must have magnitude <= 999999999"
-        )));
-    }
-    if !(-719_162..=2_932_896).contains(&days) {
-        return Err(PyOverflowError::new_err("date value out of range"));
     }
-    Ok(days as i32)
+    Ok(match first_char(dt) {
+        Some(char_type) => Some(raise_pyspark(
+            py,
+            "PySparkTypeError",
+            "UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION",
+            &[("data_type", &data_type_str(py, char_type)?)],
+        )),
+        None => None,
+    })
 }
 
 /// Whether PySpark's `LocalDataToArrowConversion._need_converter` gives a 
value of `dt` a
@@ -1368,60 +1413,33 @@ fn udt_of<'py>(
     class.call0()
 }
 
-/// A value of a day-time interval column, as PyArrow converts one to 
`duration[us]`: a
-/// timedelta to its microseconds, as `DayTimeIntervalType.toInternal`, and an 
int, or a
-/// float truncated, as microseconds. Another value raises PyArrow's error.
+/// The microseconds of a value of a day-time interval column, as PyArrow 
converts it to
+/// `duration[us]`: a timedelta; a NumPy scalar of `timedelta64[us]`, as 
another NumPy scalar
+/// raises `ArrowNotImplementedError`; else the value as `CIntFromPython` 
takes it.
 fn duration_micros(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
     let py = obj.py();
     if obj.is_instance(TIMEDELTA.import(py, "datetime", 
"timedelta")?.as_any())? {
         return timedelta_micros(obj);
     }
-    if obj.is_instance_of::<PyBool>() {
-        return Err(pyarrow_error(
-            py,
-            "ArrowTypeError",
-            "Expected integer, got bool".into(),
-        )?);
-    }
-    if numpy_scalar(obj)?.is_some() {
+    if numpy_generic(obj)? {
         let arrow_type = py.import(intern!(py, "pyarrow"))?.call_method1(
             intern!(py, "from_numpy_dtype"),
             (obj.getattr(intern!(py, "dtype"))?,),
         )?;
+        let arrow_type = arrow_type.str()?.to_cow()?.into_owned();
+        if arrow_type == "duration[us]" {
+            // The microseconds that the scalar holds, as PyArrow reads them.
+            return obj
+                .call_method1(intern!(py, "astype"), (intern!(py, "int64"),))?
+                .extract();
+        }
         return Err(pyarrow_error(
             py,
             "ArrowNotImplementedError",
-            format!("Expected np.timedelta64 but got: {}", arrow_type.str()?),
+            format!("Expected np.timedelta64 but got: {arrow_type}"),
         )?);
     }
-    let too_large = || PyOverflowError::new_err("Python int too large to 
convert to C long");
-    if obj.is_instance_of::<PyInt>() {
-        return obj.extract::<i64>().map_err(|_| too_large());
-    }
-    if let Ok(value) = obj.cast::<PyFloat>() {
-        let value = value.value();
-        if value.is_nan() {
-            return Err(PyValueError::new_err("cannot convert float NaN to 
integer"));
-        }
-        if value.is_infinite() {
-            return Err(PyOverflowError::new_err(
-                "cannot convert float infinity to integer",
-            ));
-        }
-        let truncated = value.trunc();
-        if 
!(-9_223_372_036_854_775_808.0..9_223_372_036_854_775_808.0).contains(&truncated)
 {
-            return Err(too_large());
-        }
-        return Ok(truncated as i64);
-    }
-    Err(pyarrow_error(
-        py,
-        "ArrowTypeError",
-        format!(
-            "object of type {} cannot be converted to int",
-            obj.get_type().repr()?
-        ),
-    )?)
+    c_int_from_python(obj, 64, None)
 }
 
 /// The microseconds of a `datetime.timedelta`, as 
`DayTimeIntervalType.toInternal`:/// The microseconds of a 
`datetime.timedelta`, as `DayTimeIntervalType.toInternal`:
@@ -1438,7 +1456,9 @@ pub(crate) fn timedelta_micros(obj: &Bound<'_, PyAny>) -> 
PyResult<i64> {
         .ok_or_else(|| PyOverflowError::new_err("Python int too large to 
convert to C long"))
 }
 
-/// An integral value, as PyArrow converts one: an int in range, or a float 
truncated.
+/// An integral value, as PyArrow converts one: the value as `CIntFromPython` 
takes it, such as an
+/// int in range, a float truncated or a `Decimal`. The error of a value that 
is not an int, or a
+/// NumPy integer, is the `ArrowInvalid` "Could not convert".
 fn convert_integral(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<Value> {
     let (bits, arrow_type) = match dt {
         DataType::Byte => (8, "int8"),
@@ -1446,35 +1466,18 @@ fn convert_integral(obj: &Bound<'_, PyAny>, dt: 
&DataType) -> PyResult<Value> {
         DataType::Integer => (32, "int32"),
         _ => (64, "int64"),
     };
-    if obj.is_instance_of::<PyBool>() {
-        return Err(PyTypeError::new_err("Expected integer, got bool"));
-    }
-    // A float, also of NumPy, is truncated, as PyArrow does.
-    let truncated = |obj: &Bound<'_, PyAny>| -> PyResult<i64> {
-        let value: f64 = obj.extract()?;
-        // The range of i64 is [-2^63, 2^63), both bounds exact as f64.
-        if !(i64::MIN as f64..i64::MAX as f64).contains(&value.trunc()) {
-            return Err(arrow_invalid(obj, arrow_type)?);
-        }
-        Ok(value.trunc() as i64)
-    };
-    let value: i64 = if obj.is_instance_of::<PyInt>() {
-        obj.extract()?
-    } else if obj.is_instance_of::<PyFloat>() {
-        truncated(obj)?
-    } else {
-        match numpy_scalar(obj)? {
-            Some(NumpyScalar::Integer) => obj.extract()?,
-            Some(NumpyScalar::Floating) => truncated(obj)?,
-            Some(NumpyScalar::Bool) => i64::from(obj.is_truthy()?),
-            None => return Err(arrow_invalid(obj, arrow_type)?),
+    let value = match c_int_from_python(obj, bits, None) {
+        Ok(value) => value,
+        Err(err) => {
+            let int_scalar =
+                obj.is_instance_of::<PyInt>() || numpy_scalar(obj)? == 
Some(NumpyScalar::Integer);
+            return Err(if int_scalar {
+                err
+            } else {
+                arrow_invalid(obj, arrow_type)?
+            });
         }
     };
-    if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits - 
1))).contains(&value) {
-        return Err(PyValueError::new_err(format!(
-            "Value {value} too large to fit in C integer type"
-        )));
-    }
     Ok(match dt {
         DataType::Byte => Value::Byte(value as i8),
         DataType::Short => Value::Short(value as i16),
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index c39773f..4524877 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -227,9 +227,18 @@ pub fn value_to_py<'py>(py: Python<'py>, v: &Value) -> 
PyResult<Bound<'py, PyAny
             // Convert days-since-epoch (i32) to datetime.date
             let date_mod = py.import("datetime")?;
             let date_cls = date_mod.getattr("date")?;
-            // 719163 is the ordinal of 1970-01-01
-            let ordinal = (*d as i64) + 719163i64;
-            date_cls.call_method1("fromordinal", (ordinal,))?.into_any()
+            if (-719_162..=2_932_896).contains(d) {
+                // 719163 is the ordinal of 1970-01-01
+                let ordinal = (*d as i64) + 719163i64;
+                date_cls.call_method1("fromordinal", (ordinal,))?.into_any()
+            } else {
+                // Out of the dates of Python, as PyArrow's 
`Date32Scalar.as_py`, of
+                // `date(1970, 1, 1) + timedelta(days=d)`, which raises its 
OverflowError.
+                let kwargs = PyDict::new(py);
+                kwargs.set_item("days", *d)?;
+                let delta = date_mod.getattr("timedelta")?.call((), 
Some(&kwargs))?;
+                date_cls.call1((1970, 1, 1))?.add(delta)?.into_any()
+            }
         }
         // TIMESTAMP (with a time zone): the instant, shown in the local time 
zone of the client,
         // as PySpark's Arrow conversion does (`astimezone()` of the UTC 
datetime, naive).
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 659ec20..88fe6f3 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -607,6 +607,12 @@ impl PySparkSession {
                 out.push(infer::convert(&row[i], &field.data_type, 
field.nullable)?);
             }
         }
+        // Then, as PySpark, the schema converts to Arrow, which has no char 
or varchar type.
+        for field in &fields {
+            if let Some(err) = infer::unsupported_arrow_type(py, 
&field.data_type)? {
+                return Err(err);
+            }
+        }
         // The names, with those that inference added, must name every column.
         if let Some(cols) = &cols {
             if cols.len() != fields.len() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 65db098..f3be9c7 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2674,6 +2674,29 @@ def 
test_persist_cache_and_unpersist_return_the_dataframe_itself_as_pyspark(sess
         df.unpersist()
 
 
+def test_values_of_local_rows_convert_and_raise_as_pyarrow(session):
+    # As PySpark Connect's: PyArrow takes the __int__ of a Decimal in an int 
or a date field,
+    # raises its own errors, and takes the days of a date out of the dates of 
Python, which
+    # then raise the OverflowError of PyArrow's `date(1970, 1, 1) + 
timedelta(days=d)`.
+    import decimal
+
+    import pyarrow as pa
+    from pyspark.errors import PySparkTypeError
+
+    D = decimal.Decimal
+    df = session.createDataFrame([(D("2.7"), D("1.5"))], "i int, d date")
+    assert df.collect() == [Row(i=2, d=date(1970, 1, 2))]
+    with pytest.raises(pa.ArrowInvalid, match="tried to convert to int32"):
+        session.createDataFrame([("x",)], "v int")
+    with pytest.raises(PySparkTypeError) as e:
+        session.createDataFrame([("x",)], "v char(3)")
+    assert e.value.getCondition() == 
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+    df = session.createDataFrame([(2932897,)], "v date")
+    assert df.count() == 1
+    with pytest.raises(OverflowError, match="^date value out of range$"):
+        # PySpark raises in collect; the client's rows convert their values 
when read.
+        df.collect()[0].v
+
 def test_create_dataframe_converts_numpy_scalars_as_pyspark(session):
     np = pytest.importorskip("numpy")
     df = session.createDataFrame(
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index a1aa149..f1488a7 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8033,20 +8033,72 @@ def 
test_ints_of_date_fields_convert_as_pyarrow(mock_connect):
     table, _ = _local_rows_relation_of(service, spark.createDataFrame(
         [(np.int64(3),), (1.5,), (-1.5,), (np.bool_(True),), (2932896,), 
(-719162,)], "a date"))
     assert table.column(0).cast(pa.int32()).to_pylist() == [3, 1, -1, 1, 
2932896, -719162]
+    # PyArrow takes the days of an int32, also out of the dates of Python, and 
the Decimal
+    # of the `__int__` of its value.
+    table, _ = _local_rows_relation_of(service, spark.createDataFrame(
+        [(2932897,), (-719163,), (10**9,), (decimal.Decimal("2.5"),)], "a 
date"))
+    assert table.column(0).cast(pa.int32()).to_pylist() == [2932897, -719163, 
10**9, 2]
 
     for value, error, message in (
         (True, pa.ArrowTypeError, "Expected integer, got bool"),
         ("x", pa.ArrowTypeError, "object of type <class 'str'> cannot be 
converted to int"),
         (float("nan"), ValueError, "cannot convert float NaN to integer"),
-        (2932897, OverflowError, "date value out of range"),
-        (-719163, OverflowError, "date value out of range"),
-        (10**9, OverflowError, "days=1000000000; must have magnitude <= 
999999999"),
+        (2**40, pa.ArrowInvalid, "Integer too large for date32"),
         (2**70, OverflowError, "Python int too large to convert to C long"),
     ):
         with pytest.raises(error, match="^" + re.escape(message) + "$"):
             spark.createDataFrame([(value,)], "a date")
 
 
+def test_values_of_local_rows_convert_and_raise_as_pyarrow(mock_connect):
+    # As PySpark Connect's, which passes these values to PyArrow as they are: 
an int, date,
+    # day-time interval and time field takes the __index__, else the __int__, 
of a value, such
+    # as a Decimal, and the errors are PyArrow's. A char or varchar field 
raises the error of
+    # PySpark's to_arrow_schema.
+    import re
+
+    import pyarrow as pa
+    from pyspark.errors import PySparkTypeError
+
+    np = pytest.importorskip("numpy")
+    spark, service = mock_connect
+    D = decimal.Decimal
+    table, _ = _local_rows_relation_of(service, spark.createDataFrame(
+        [(D("2.7"), D("1.5"), D("3.9"), D("4.2"), np.timedelta64(5, "us"))],
+        "i int, d date, t interval day to second, n time, u interval day to 
second"))
+    assert [table.column(name).cast(cast).to_pylist()[0]
+            for name, cast in [("i", pa.int32()), ("d", pa.int32()), ("t", 
pa.int64()),
+                               ("n", pa.int64()), ("u", pa.int64())]] == [2, 
1, 3, 4, 5]
+    for value, ddl, error, message in [
+        ("x", "int", pa.ArrowInvalid,
+         "Could not convert 'x' with type str: tried to convert to int32"),
+        ("x", "float", pa.ArrowInvalid,
+         "Could not convert 'x' with type str: tried to convert to float32"),
+        (2**40, "int", pa.ArrowInvalid, "Value 1099511627776 too large to fit 
in C integer type"),
+        (2**70, "double", pa.ArrowInvalid, "PyLong is too large to fit int64"),
+        (2**40, "float", pa.ArrowInvalid,
+         "Integer value 1099511627776 is outside of the range exactly 
representable by a "
+         "IEEE 754 single precision value"),
+        (True, "int", pa.ArrowTypeError, "Expected integer, got bool"),
+        (True, "time", pa.ArrowTypeError, "Expected integer, got bool"),
+        (float("nan"), "time", ValueError, "cannot convert float NaN to 
integer"),
+        ("x", "interval day to second", pa.ArrowTypeError,
+         "object of type <class 'str'> cannot be converted to int"),
+        (np.timedelta64(5, "ns"), "interval day to second", 
pa.ArrowNotImplementedError,
+         "Expected np.timedelta64 but got: duration[ns]"),
+        ("x", "boolean", pa.ArrowInvalid,
+         "Could not convert 'x' with type str: tried to convert to boolean"),
+    ]:
+        with pytest.raises(error, match="^" + re.escape(message) + "$") as e:
+            spark.createDataFrame([(value,)], f"v {ddl}")
+        assert type(e.value) is error
+    for field_type, data_type in [(T.CharType(3), "CharType(3)"),
+                                  (T.ArrayType(T.VarcharType(2)), 
"VarcharType(2)")]:
+        with pytest.raises(PySparkTypeError) as e:
+            spark.createDataFrame([("x",)], T.StructType([T.StructField("v", 
field_type)]))
+        assert e.value.getCondition() == 
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+        assert e.value.getMessageParameters() == {"data_type": data_type}
+
 def 
test_nested_duplicate_field_names_are_deduplicated_in_the_arrow_schema(mock_connect):
     # As PySpark's _deduplicate_field_names: the Arrow schema of nested struct 
fields numbers
     # duplicate names, which the server cannot read, while the schema of the 
relation keeps


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to