This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 01dbea1 [SPARK-61106][CONNECT][PYTHON] Convert the values of local
rows of a schema, and raise their errors, as PyArrow does in createDataFrame in
the Python client
01dbea1 is described below
commit 01dbea17b943dff96f0e4fa30919ac7392e893f4
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 18:53:00 2026 +0900
[SPARK-61106][CONNECT][PYTHON] Convert the values of local rows of a
schema, and raise their errors, as PyArrow does in createDataFrame in the
Python client
### What changes were proposed in this pull request?
[SPARK-61106](https://issues.apache.org/jira/browse/SPARK-61106):
`spark.createDataFrame` of local rows with a schema now converts their values,
and raises their errors, as PyArrow does in PySpark Connect.
PySpark Connect passes the values of most atomic types to PyArrow as they
are. The client converts them in Rust (`crates/pyspark-rs/src/infer.rs`),
emulating PyArrow. This change makes the emulation follow PyArrow's code:
- An int, date, day-time interval or time field takes a value as PyArrow's
`CIntFromPython` does:
- an int, else the `__index__` of the value, else its `__int__`
(`PyObjectToPyInt`), such as of a `Decimal`, a float or a NumPy scalar
- a bool raises `ArrowTypeError` ("Expected integer, got bool")
- a value out of range raises `ArrowInvalid` ("Value X too large to fit
in C integer type", or "Integer too large for date32")
- A NumPy scalar in a day-time interval field is taken only as a
`timedelta64[us]`, as PyArrow's `duration[us]` takes it. Another NumPy scalar
raises `ArrowNotImplementedError`.
- The errors are PyArrow's classes, `ArrowInvalid`, `ArrowTypeError` and
`ArrowNotImplementedError`, with PyArrow's messages:
- "tried to convert to float32" for a float field
- "PyLong is too large to fit int64" for an int beyond int64 in a float
or double field
- "was not a utf8 string" for bytes of a string field
- Without PyArrow, they are the built-in classes these derive from.
- A date field takes the days of any int32, as PyArrow does; it no longer
stops at the dates of Python. `collect` turns days outside those dates into
`date(1970, 1, 1) + timedelta(days=d)`, as PyArrow's `Date32Scalar.as_py` does.
That raises its `OverflowError`: "date value out of range", or
"days=1000000000; must have magnitude <= 999999999".
- A `char` or `varchar` field raises
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`, after the values are converted,
as PySpark's `to_arrow_schema` does: it has no Arrow type for these types.
### Why are the changes needed?
Of 374 cases of a value of each kind in a field of each atomic type, 119
differed from PySpark 4.3.0 on a local Spark 4.3.0 server. For example:
| Input | master | This change and PySpark 4.3.0 |
| --- | --- | --- |
| `Decimal("1.5")` into `int`, `date` or `interval day to second` |
`ValueError` or `ArrowTypeError` | 1, `1970-01-02`, 1 microsecond |
| `"x"` into `int` | `ValueError: Could not convert 'x' ...` |
`pyarrow.lib.ArrowInvalid: Could not convert 'x' ...` |
| `True` into `int` | `TypeError` | `ArrowTypeError` |
| `"x"` into `float` | `... tried to convert to float` | `... tried to
convert to float32` |
| `2**70` into `double` | `ValueError: Integer value ... is outside of the
range ...` | `ArrowInvalid: PyLong is too large to fit int64` |
| `2**40` into `date` | `OverflowError: days=...` | `ArrowInvalid: Integer
too large for date32` |
| `2932897` days into `date` | `OverflowError` in `createDataFrame` |
created; `OverflowError: date value out of range` when collected |
| `True` or `NaN` into `time` | taken | `ArrowTypeError`, `ValueError` |
| bytes, or an int, into `char(3)` | the server's
`UNSUPPORTED_CHAR_OR_VARCHAR_AS_STRING`, or `TypeError` |
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` |
### Does this PR introduce _any_ user-facing change?
Yes. The values of local rows convert, or fail, as in PySpark 4.3.0.
### How was this patch tested?
- A new live test checks:
- a Decimal in an int and a date field
- the `ArrowInvalid` of a str in an int field
- the error of a char field
- the error of a date out of the dates of Python
It passes with official PySpark 4.3.0 and fails without this change.
- A new offline test checks the values sent for a Decimal in each kind of
field and for a `timedelta64[us]`, the errors and their exact classes, and the
error of char and varchar fields.
- The test of ints in date fields expected the errors of days outside the
dates of Python from `createDataFrame`. It now expects the days to be sent, as
PySpark does.
- On a local server, 401 cases give the same results as official PySpark
4.3.0: 374 of a value of each kind into each atomic type, and 27 of char,
varchar and nested fields, NumPy scalars and edge values. The only differences
are the addresses of `object()`s in the messages.
- The rows of the client convert their values when they are read. So the
`OverflowError` of a date outside the dates of Python is raised when the value
is read, where PySpark raises it in `collect`.
- The lib tests of both crates (588 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2054 passed, 12 skipped), and the
live drop-in and Arrow suites (158 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #304 from HyukjinKwon/SPARK-61106-local-row-conversion.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/infer.rs | 343 ++++++++++++++++++------------------
crates/pyspark-rs/src/row.rs | 15 +-
crates/pyspark-rs/src/session.rs | 6 +
python/tests/test_dropin_e2e.py | 23 +++
python/tests/test_dropin_offline.py | 58 +++++-
5 files changed, 269 insertions(+), 176 deletions(-)
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 319c4ce..4825c72 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -7,7 +7,9 @@
use std::cell::OnceCell;
use std::collections::{HashMap, HashSet};
-use pyo3::exceptions::{PyAssertionError, PyOverflowError, PyTypeError,
PyValueError};
+use pyo3::exceptions::{
+ PyAssertionError, PyNotImplementedError, PyOverflowError, PyTypeError,
PyValueError,
+};
use pyo3::intern;
use pyo3::prelude::*;
use pyo3::sync::PyOnceLock;
@@ -841,22 +843,86 @@ fn arrow_type_name(obj: &Bound<'_, PyAny>) ->
PyResult<String> {
})
}
-/// An error of PyArrow, of `class` with `message`, as its conversions raise
it.
+/// An error of PyArrow, of `class` with `message`, as its conversions raise
it: `ArrowInvalid`,
+/// a `ValueError`, `ArrowTypeError`, a `TypeError`, or
`ArrowNotImplementedError`, a
+/// `NotImplementedError`. Without PyArrow, the built-in class that it derives
from.
fn pyarrow_error(py: Python<'_>, class: &str, message: String) ->
PyResult<PyErr> {
- let error = py
- .import(intern!(py, "pyarrow"))?
- .getattr(class)?
- .call1((message,))?;
- Ok(PyErr::from_value(error))
+ match py
+ .import(intern!(py, "pyarrow"))
+ .and_then(|pyarrow| pyarrow.getattr(class))
+ {
+ Ok(class) => Ok(PyErr::from_value(class.call1((message,))?)),
+ Err(_) => Ok(match class {
+ "ArrowTypeError" => PyTypeError::new_err(message),
+ "ArrowNotImplementedError" =>
PyNotImplementedError::new_err(message),
+ _ => PyValueError::new_err(message),
+ }),
+ }
}
-/// The error of a value that PyArrow cannot convert to the type it is named
by.
+/// The `ArrowInvalid` of a value that PyArrow cannot convert to the type it
is named by.
fn arrow_invalid(obj: &Bound<'_, PyAny>, arrow_type: &str) -> PyResult<PyErr> {
- Ok(PyValueError::new_err(format!(
- "Could not convert {} with type {}: tried to convert to {arrow_type}",
- obj.repr()?,
- arrow_type_name(obj)?
- )))
+ pyarrow_error(
+ obj.py(),
+ "ArrowInvalid",
+ format!(
+ "Could not convert {} with type {}: tried to convert to
{arrow_type}",
+ obj.repr()?,
+ arrow_type_name(obj)?
+ ),
+ )
+}
+
+/// `obj` as an int, as PyArrow's `PyObjectToPyInt` takes it: by `__index__`,
else by the
+/// `__int__` of its type, which may raise, such as for a NaN; else the
`ArrowTypeError` "object
+/// of type <class 'T'> cannot be converted to int". A float, a `Decimal` and
a NumPy number or
+/// bool are taken by their `__int__`.
+fn py_int<'py>(obj: &Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
+ let py = obj.py();
+ // SAFETY: PyNumber_Index returns a new reference, or NULL with an
exception set.
+ if let Ok(index) =
+ unsafe { Bound::from_owned_ptr_or_err(py,
pyo3::ffi::PyNumber_Index(obj.as_ptr())) }
+ {
+ return Ok(index);
+ }
+ let ty = obj.get_type();
+ if ty.hasattr(intern!(py, "__int__"))? {
+ return ty.getattr(intern!(py, "__int__"))?.call1((obj,));
+ }
+ Err(pyarrow_error(
+ py,
+ "ArrowTypeError",
+ format!("object of type {} cannot be converted to int", ty.repr()?),
+ )?)
+}
+
+/// `obj` as an integer of `bits` bits, as PyArrow's `CIntFromPython` takes
it: not a bool (an
+/// `ArrowTypeError`), an int, else by `py_int`. A value out of the range of
the bits raises an
+/// `ArrowInvalid` of `overflow`, else "Value X too large to fit in C integer
type", and one out
+/// of the range of a C long Python's `OverflowError`.
+fn c_int_from_python(obj: &Bound<'_, PyAny>, bits: u32, overflow:
Option<&str>) -> PyResult<i64> {
+ let py = obj.py();
+ if obj.is_instance_of::<PyBool>() {
+ return Err(pyarrow_error(
+ py,
+ "ArrowTypeError",
+ "Expected integer, got bool".into(),
+ )?);
+ }
+ let int = if obj.is_instance_of::<PyInt>() {
+ obj.clone()
+ } else {
+ py_int(obj)?
+ };
+ let value: i64 = int.extract()?;
+ if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits -
1))).contains(&value) {
+ let message = match overflow {
+ Some(message) => message.to_string(),
+ None => format!("Value {} too large to fit in C integer type",
int.str()?),
+ };
+ return Err(pyarrow_error(py, "ArrowInvalid", message)?);
+ }
+ Ok(value)
}
/// The kind of a NumPy scalar of a bool or a number, which PyArrow converts
as the Python
@@ -891,6 +957,18 @@ fn numpy_scalar(obj: &Bound<'_, PyAny>) ->
PyResult<Option<NumpyScalar>> {
Ok(None)
}
+/// Whether `obj` is a NumPy scalar of any kind. NumPy is not imported when it
is not loaded.
+fn numpy_generic(obj: &Bound<'_, PyAny>) -> PyResult<bool> {
+ let py = obj.py();
+ let modules = py
+ .import(intern!(py, "sys"))?
+ .getattr(intern!(py, "modules"))?;
+ match modules.cast::<PyDict>()?.get_item(intern!(py, "numpy"))? {
+ Some(numpy) if !numpy.is_none() =>
obj.is_instance(&numpy.getattr(intern!(py, "generic"))?),
+ _ => Ok(false),
+ }
+}
+
/// An integer as a floating-point value, which PyArrow takes only when it is
exact: within
/// 2^53 for a double and 2^24 for a float.
fn exact_float(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<f64> {
@@ -899,15 +977,25 @@ fn exact_float(obj: &Bound<'_, PyAny>, dt: &DataType) ->
PyResult<f64> {
_ => (53, "double"),
};
let out_of_range = || -> PyResult<PyErr> {
- Ok(PyValueError::new_err(format!(
- "Integer value {} is outside of the range exactly representable by
a IEEE 754 \
- {precision} precision value",
- obj.str()?
- )))
+ pyarrow_error(
+ obj.py(),
+ "ArrowInvalid",
+ format!(
+ "Integer value {} is outside of the range exactly
representable by a IEEE 754 \
+ {precision} precision value",
+ obj.str()?
+ ),
+ )
};
let value: i64 = match obj.extract() {
Ok(value) => value,
- Err(_) => return Err(out_of_range()?),
+ Err(_) => {
+ return Err(pyarrow_error(
+ obj.py(),
+ "ArrowInvalid",
+ "PyLong is too large to fit int64".into(),
+ )?)
+ }
};
if value.unsigned_abs() > 1u64 << bits {
return Err(out_of_range()?);
@@ -1071,21 +1159,9 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
};
Ok(Value::String(text))
}
- DataType::Char { .. } | DataType::Varchar { .. } => {
- if let Ok(text) = obj.cast::<PyString>() {
- Ok(Value::String(text.to_cow()?.into_owned()))
- } else if let Ok(bytes) = obj.cast::<PyBytes>() {
- Ok(Value::String(
- String::from_utf8(bytes.as_bytes().to_vec())
- .map_err(|e| PyValueError::new_err(e.to_string()))?,
- ))
- } else {
- Err(PyTypeError::new_err(format!(
- "Expected bytes, got a '{}' object",
- type_name(obj)?
- )))
- }
- }
+ // As PySpark, which has no converter of these types: the value goes
as it is to
+ // `to_arrow_schema`, which raises for the type
(`unsupported_arrow_type`).
+ DataType::Char { .. } | DataType::Varchar { .. } => Ok(Value::Null),
DataType::Boolean => {
if obj.is_instance_of::<PyBool>() || numpy_scalar(obj)? ==
Some(NumpyScalar::Bool) {
Ok(Value::Bool(obj.is_truthy()?))
@@ -1109,7 +1185,7 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
Some(NumpyScalar::Integer) => exact_float(obj, dt)?,
_ => {
let arrow_type = if matches!(dt, DataType::Float) {
- "float"
+ "float32"
} else {
"double"
};
@@ -1124,20 +1200,17 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
}
DataType::DayTimeInterval { .. } =>
Ok(Value::DayTimeInterval(duration_micros(obj)?)),
DataType::Date => Ok(Value::Date(date_days(obj)?)),
- // As PyArrow's `time64[ns]`, to which PySpark passes the value as it
is: a time, and
- // the nanoseconds of an int, or of a float, truncated.
+ // As PyArrow's `time64[ns]`, to which PySpark passes the value as it
is: a time, else
+ // the nanoseconds of the value as `CIntFromPython` takes it.
DataType::Time { .. } => {
if obj.is_instance(TIME.import(py, "datetime", "time")?.as_any())?
{
Ok(Value::Time(time_nanos(obj)?))
- } else if obj.is_instance_of::<PyInt>() {
- Ok(Value::Time(obj.extract()?))
- } else if obj.is_instance_of::<PyFloat>() {
- Ok(Value::Time(obj.extract::<f64>()? as i64))
} else {
- Err(PyTypeError::new_err(format!(
- "object of type {} cannot be converted to int",
- obj.get_type().repr()?
- )))
+ Ok(Value::Time(c_int_from_python(
+ obj,
+ 64,
+ Some("Integer too large for int64"),
+ )?))
}
}
// As PySpark's converters of these types: the value and metadata of a
VariantVal, and
@@ -1202,71 +1275,43 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
}
/// The days since the epoch of a value of a date column, as PyArrow converts
it to date32: a
-/// date, or the date of a datetime; an int, or a float truncated, as days;
and a NumPy bool or
-/// number as the value it stands for. Another value raises PyArrow's error.
+/// date, or the date of a datetime, else the days of the value as
`CIntFromPython` takes it, in
+/// the range of an int32, else the `ArrowInvalid` "Integer too large for
date32".
fn date_days(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
let py = obj.py();
if obj.is_instance(DATE.import(py, "datetime", "date")?.as_any())? {
let ordinal: i64 = obj.call_method0(intern!(py,
"toordinal"))?.extract()?;
return Ok((ordinal - 719_163) as i32);
}
- if obj.is_instance_of::<PyBool>() {
- return Err(pyarrow_error(
- py,
- "ArrowTypeError",
- "Expected integer, got bool".into(),
- )?);
- }
- let kind = if obj.is_instance_of::<PyInt>() {
- Some(NumpyScalar::Integer)
- } else if obj.is_instance_of::<PyFloat>() {
- Some(NumpyScalar::Floating)
- } else {
- numpy_scalar(obj)?
- };
- let days: i64 = match kind {
- Some(NumpyScalar::Bool) => i64::from(obj.is_truthy()?),
- Some(NumpyScalar::Integer) => obj.extract()?,
- Some(NumpyScalar::Floating) => {
- let value: f64 = obj.extract()?;
- if value.is_nan() {
- return Err(PyValueError::new_err("cannot convert float NaN to
integer"));
- }
- if value.is_infinite() {
- return Err(PyOverflowError::new_err(
- "cannot convert float infinity to integer",
- ));
- }
- let truncated = value.trunc();
- if
!(-9_223_372_036_854_775_808.0..9_223_372_036_854_775_808.0).contains(&truncated)
{
- return Err(PyOverflowError::new_err(
- "Python int too large to convert to C long",
- ));
- }
- truncated as i64
- }
- None => {
- return Err(pyarrow_error(
- py,
- "ArrowTypeError",
- format!(
- "object of type {} cannot be converted to int",
- obj.get_type().repr()?
- ),
- )?)
+ Ok(c_int_from_python(obj, 32, Some("Integer too large for date32"))? as
i32)
+}
+
+/// The `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` of the first char or
varchar type of `dt`,
+/// depth first, which PySpark's `to_arrow_schema` raises when it converts the
schema of local
+/// rows to Arrow: it has no Arrow type of these types.
+pub(crate) fn unsupported_arrow_type(py: Python<'_>, dt: &DataType) ->
PyResult<Option<PyErr>> {
+ fn first_char(dt: &DataType) -> Option<&DataType> {
+ match dt {
+ DataType::Char { .. } | DataType::Varchar { .. } => Some(dt),
+ DataType::Struct { fields } => fields.iter().find_map(|f|
first_char(&f.data_type)),
+ DataType::Array { element_type, .. } => first_char(element_type),
+ DataType::Map {
+ key_type,
+ value_type,
+ ..
+ } => first_char(key_type).or_else(|| first_char(value_type)),
+ _ => None,
}
- };
- // The days of PyArrow's dates: from 0001-01-01 to 9999-12-31, and a
larger magnitude is
- // out of the range of a Python timedelta.
- if days.unsigned_abs() > 999_999_999 {
- return Err(PyOverflowError::new_err(format!(
- "days={days}; must have magnitude <= 999999999"
- )));
- }
- if !(-719_162..=2_932_896).contains(&days) {
- return Err(PyOverflowError::new_err("date value out of range"));
}
- Ok(days as i32)
+ Ok(match first_char(dt) {
+ Some(char_type) => Some(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION",
+ &[("data_type", &data_type_str(py, char_type)?)],
+ )),
+ None => None,
+ })
}
/// Whether PySpark's `LocalDataToArrowConversion._need_converter` gives a
value of `dt` a
@@ -1368,60 +1413,33 @@ fn udt_of<'py>(
class.call0()
}
-/// A value of a day-time interval column, as PyArrow converts one to
`duration[us]`: a
-/// timedelta to its microseconds, as `DayTimeIntervalType.toInternal`, and an
int, or a
-/// float truncated, as microseconds. Another value raises PyArrow's error.
+/// The microseconds of a value of a day-time interval column, as PyArrow
converts it to
+/// `duration[us]`: a timedelta; a NumPy scalar of `timedelta64[us]`, as
another NumPy scalar
+/// raises `ArrowNotImplementedError`; else the value as `CIntFromPython`
takes it.
fn duration_micros(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
let py = obj.py();
if obj.is_instance(TIMEDELTA.import(py, "datetime",
"timedelta")?.as_any())? {
return timedelta_micros(obj);
}
- if obj.is_instance_of::<PyBool>() {
- return Err(pyarrow_error(
- py,
- "ArrowTypeError",
- "Expected integer, got bool".into(),
- )?);
- }
- if numpy_scalar(obj)?.is_some() {
+ if numpy_generic(obj)? {
let arrow_type = py.import(intern!(py, "pyarrow"))?.call_method1(
intern!(py, "from_numpy_dtype"),
(obj.getattr(intern!(py, "dtype"))?,),
)?;
+ let arrow_type = arrow_type.str()?.to_cow()?.into_owned();
+ if arrow_type == "duration[us]" {
+ // The microseconds that the scalar holds, as PyArrow reads them.
+ return obj
+ .call_method1(intern!(py, "astype"), (intern!(py, "int64"),))?
+ .extract();
+ }
return Err(pyarrow_error(
py,
"ArrowNotImplementedError",
- format!("Expected np.timedelta64 but got: {}", arrow_type.str()?),
+ format!("Expected np.timedelta64 but got: {arrow_type}"),
)?);
}
- let too_large = || PyOverflowError::new_err("Python int too large to
convert to C long");
- if obj.is_instance_of::<PyInt>() {
- return obj.extract::<i64>().map_err(|_| too_large());
- }
- if let Ok(value) = obj.cast::<PyFloat>() {
- let value = value.value();
- if value.is_nan() {
- return Err(PyValueError::new_err("cannot convert float NaN to
integer"));
- }
- if value.is_infinite() {
- return Err(PyOverflowError::new_err(
- "cannot convert float infinity to integer",
- ));
- }
- let truncated = value.trunc();
- if
!(-9_223_372_036_854_775_808.0..9_223_372_036_854_775_808.0).contains(&truncated)
{
- return Err(too_large());
- }
- return Ok(truncated as i64);
- }
- Err(pyarrow_error(
- py,
- "ArrowTypeError",
- format!(
- "object of type {} cannot be converted to int",
- obj.get_type().repr()?
- ),
- )?)
+ c_int_from_python(obj, 64, None)
}
/// The microseconds of a `datetime.timedelta`, as
`DayTimeIntervalType.toInternal`:/// The microseconds of a
`datetime.timedelta`, as `DayTimeIntervalType.toInternal`:
@@ -1438,7 +1456,9 @@ pub(crate) fn timedelta_micros(obj: &Bound<'_, PyAny>) ->
PyResult<i64> {
.ok_or_else(|| PyOverflowError::new_err("Python int too large to
convert to C long"))
}
-/// An integral value, as PyArrow converts one: an int in range, or a float
truncated.
+/// An integral value, as PyArrow converts one: the value as `CIntFromPython`
takes it, such as an
+/// int in range, a float truncated or a `Decimal`. The error of a value that
is not an int, or a
+/// NumPy integer, is the `ArrowInvalid` "Could not convert".
fn convert_integral(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<Value> {
let (bits, arrow_type) = match dt {
DataType::Byte => (8, "int8"),
@@ -1446,35 +1466,18 @@ fn convert_integral(obj: &Bound<'_, PyAny>, dt:
&DataType) -> PyResult<Value> {
DataType::Integer => (32, "int32"),
_ => (64, "int64"),
};
- if obj.is_instance_of::<PyBool>() {
- return Err(PyTypeError::new_err("Expected integer, got bool"));
- }
- // A float, also of NumPy, is truncated, as PyArrow does.
- let truncated = |obj: &Bound<'_, PyAny>| -> PyResult<i64> {
- let value: f64 = obj.extract()?;
- // The range of i64 is [-2^63, 2^63), both bounds exact as f64.
- if !(i64::MIN as f64..i64::MAX as f64).contains(&value.trunc()) {
- return Err(arrow_invalid(obj, arrow_type)?);
- }
- Ok(value.trunc() as i64)
- };
- let value: i64 = if obj.is_instance_of::<PyInt>() {
- obj.extract()?
- } else if obj.is_instance_of::<PyFloat>() {
- truncated(obj)?
- } else {
- match numpy_scalar(obj)? {
- Some(NumpyScalar::Integer) => obj.extract()?,
- Some(NumpyScalar::Floating) => truncated(obj)?,
- Some(NumpyScalar::Bool) => i64::from(obj.is_truthy()?),
- None => return Err(arrow_invalid(obj, arrow_type)?),
+ let value = match c_int_from_python(obj, bits, None) {
+ Ok(value) => value,
+ Err(err) => {
+ let int_scalar =
+ obj.is_instance_of::<PyInt>() || numpy_scalar(obj)? ==
Some(NumpyScalar::Integer);
+ return Err(if int_scalar {
+ err
+ } else {
+ arrow_invalid(obj, arrow_type)?
+ });
}
};
- if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits -
1))).contains(&value) {
- return Err(PyValueError::new_err(format!(
- "Value {value} too large to fit in C integer type"
- )));
- }
Ok(match dt {
DataType::Byte => Value::Byte(value as i8),
DataType::Short => Value::Short(value as i16),
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index c39773f..4524877 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -227,9 +227,18 @@ pub fn value_to_py<'py>(py: Python<'py>, v: &Value) ->
PyResult<Bound<'py, PyAny
// Convert days-since-epoch (i32) to datetime.date
let date_mod = py.import("datetime")?;
let date_cls = date_mod.getattr("date")?;
- // 719163 is the ordinal of 1970-01-01
- let ordinal = (*d as i64) + 719163i64;
- date_cls.call_method1("fromordinal", (ordinal,))?.into_any()
+ if (-719_162..=2_932_896).contains(d) {
+ // 719163 is the ordinal of 1970-01-01
+ let ordinal = (*d as i64) + 719163i64;
+ date_cls.call_method1("fromordinal", (ordinal,))?.into_any()
+ } else {
+ // Out of the dates of Python, as PyArrow's
`Date32Scalar.as_py`, of
+ // `date(1970, 1, 1) + timedelta(days=d)`, which raises its
OverflowError.
+ let kwargs = PyDict::new(py);
+ kwargs.set_item("days", *d)?;
+ let delta = date_mod.getattr("timedelta")?.call((),
Some(&kwargs))?;
+ date_cls.call1((1970, 1, 1))?.add(delta)?.into_any()
+ }
}
// TIMESTAMP (with a time zone): the instant, shown in the local time
zone of the client,
// as PySpark's Arrow conversion does (`astimezone()` of the UTC
datetime, naive).
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 659ec20..88fe6f3 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -607,6 +607,12 @@ impl PySparkSession {
out.push(infer::convert(&row[i], &field.data_type,
field.nullable)?);
}
}
+ // Then, as PySpark, the schema converts to Arrow, which has no char
or varchar type.
+ for field in &fields {
+ if let Some(err) = infer::unsupported_arrow_type(py,
&field.data_type)? {
+ return Err(err);
+ }
+ }
// The names, with those that inference added, must name every column.
if let Some(cols) = &cols {
if cols.len() != fields.len() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 65db098..f3be9c7 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2674,6 +2674,29 @@ def
test_persist_cache_and_unpersist_return_the_dataframe_itself_as_pyspark(sess
df.unpersist()
+def test_values_of_local_rows_convert_and_raise_as_pyarrow(session):
+ # As PySpark Connect's: PyArrow takes the __int__ of a Decimal in an int
or a date field,
+ # raises its own errors, and takes the days of a date out of the dates of
Python, which
+ # then raise the OverflowError of PyArrow's `date(1970, 1, 1) +
timedelta(days=d)`.
+ import decimal
+
+ import pyarrow as pa
+ from pyspark.errors import PySparkTypeError
+
+ D = decimal.Decimal
+ df = session.createDataFrame([(D("2.7"), D("1.5"))], "i int, d date")
+ assert df.collect() == [Row(i=2, d=date(1970, 1, 2))]
+ with pytest.raises(pa.ArrowInvalid, match="tried to convert to int32"):
+ session.createDataFrame([("x",)], "v int")
+ with pytest.raises(PySparkTypeError) as e:
+ session.createDataFrame([("x",)], "v char(3)")
+ assert e.value.getCondition() ==
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+ df = session.createDataFrame([(2932897,)], "v date")
+ assert df.count() == 1
+ with pytest.raises(OverflowError, match="^date value out of range$"):
+ # PySpark raises in collect; the client's rows convert their values
when read.
+ df.collect()[0].v
+
def test_create_dataframe_converts_numpy_scalars_as_pyspark(session):
np = pytest.importorskip("numpy")
df = session.createDataFrame(
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index a1aa149..f1488a7 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8033,20 +8033,72 @@ def
test_ints_of_date_fields_convert_as_pyarrow(mock_connect):
table, _ = _local_rows_relation_of(service, spark.createDataFrame(
[(np.int64(3),), (1.5,), (-1.5,), (np.bool_(True),), (2932896,),
(-719162,)], "a date"))
assert table.column(0).cast(pa.int32()).to_pylist() == [3, 1, -1, 1,
2932896, -719162]
+ # PyArrow takes the days of an int32, also out of the dates of Python, and
the Decimal
+ # of the `__int__` of its value.
+ table, _ = _local_rows_relation_of(service, spark.createDataFrame(
+ [(2932897,), (-719163,), (10**9,), (decimal.Decimal("2.5"),)], "a
date"))
+ assert table.column(0).cast(pa.int32()).to_pylist() == [2932897, -719163,
10**9, 2]
for value, error, message in (
(True, pa.ArrowTypeError, "Expected integer, got bool"),
("x", pa.ArrowTypeError, "object of type <class 'str'> cannot be
converted to int"),
(float("nan"), ValueError, "cannot convert float NaN to integer"),
- (2932897, OverflowError, "date value out of range"),
- (-719163, OverflowError, "date value out of range"),
- (10**9, OverflowError, "days=1000000000; must have magnitude <=
999999999"),
+ (2**40, pa.ArrowInvalid, "Integer too large for date32"),
(2**70, OverflowError, "Python int too large to convert to C long"),
):
with pytest.raises(error, match="^" + re.escape(message) + "$"):
spark.createDataFrame([(value,)], "a date")
+def test_values_of_local_rows_convert_and_raise_as_pyarrow(mock_connect):
+ # As PySpark Connect's, which passes these values to PyArrow as they are:
an int, date,
+ # day-time interval and time field takes the __index__, else the __int__,
of a value, such
+ # as a Decimal, and the errors are PyArrow's. A char or varchar field
raises the error of
+ # PySpark's to_arrow_schema.
+ import re
+
+ import pyarrow as pa
+ from pyspark.errors import PySparkTypeError
+
+ np = pytest.importorskip("numpy")
+ spark, service = mock_connect
+ D = decimal.Decimal
+ table, _ = _local_rows_relation_of(service, spark.createDataFrame(
+ [(D("2.7"), D("1.5"), D("3.9"), D("4.2"), np.timedelta64(5, "us"))],
+ "i int, d date, t interval day to second, n time, u interval day to
second"))
+ assert [table.column(name).cast(cast).to_pylist()[0]
+ for name, cast in [("i", pa.int32()), ("d", pa.int32()), ("t",
pa.int64()),
+ ("n", pa.int64()), ("u", pa.int64())]] == [2,
1, 3, 4, 5]
+ for value, ddl, error, message in [
+ ("x", "int", pa.ArrowInvalid,
+ "Could not convert 'x' with type str: tried to convert to int32"),
+ ("x", "float", pa.ArrowInvalid,
+ "Could not convert 'x' with type str: tried to convert to float32"),
+ (2**40, "int", pa.ArrowInvalid, "Value 1099511627776 too large to fit
in C integer type"),
+ (2**70, "double", pa.ArrowInvalid, "PyLong is too large to fit int64"),
+ (2**40, "float", pa.ArrowInvalid,
+ "Integer value 1099511627776 is outside of the range exactly
representable by a "
+ "IEEE 754 single precision value"),
+ (True, "int", pa.ArrowTypeError, "Expected integer, got bool"),
+ (True, "time", pa.ArrowTypeError, "Expected integer, got bool"),
+ (float("nan"), "time", ValueError, "cannot convert float NaN to
integer"),
+ ("x", "interval day to second", pa.ArrowTypeError,
+ "object of type <class 'str'> cannot be converted to int"),
+ (np.timedelta64(5, "ns"), "interval day to second",
pa.ArrowNotImplementedError,
+ "Expected np.timedelta64 but got: duration[ns]"),
+ ("x", "boolean", pa.ArrowInvalid,
+ "Could not convert 'x' with type str: tried to convert to boolean"),
+ ]:
+ with pytest.raises(error, match="^" + re.escape(message) + "$") as e:
+ spark.createDataFrame([(value,)], f"v {ddl}")
+ assert type(e.value) is error
+ for field_type, data_type in [(T.CharType(3), "CharType(3)"),
+ (T.ArrayType(T.VarcharType(2)),
"VarcharType(2)")]:
+ with pytest.raises(PySparkTypeError) as e:
+ spark.createDataFrame([("x",)], T.StructType([T.StructField("v",
field_type)]))
+ assert e.value.getCondition() ==
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+ assert e.value.getMessageParameters() == {"data_type": data_type}
+
def
test_nested_duplicate_field_names_are_deduplicated_in_the_arrow_schema(mock_connect):
# As PySpark's _deduplicate_field_names: the Arrow schema of nested struct
fields numbers
# duplicate names, which the server cannot read, while the schema of the
relation keeps
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]