This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 13afbb4 [SPARK-60908][CONNECT][FOLLOWUP] Convert NumPy scalars,
reject inexact integers for floating types, and stop inference at deep nesting
as PySpark does
13afbb4 is described below
commit 13afbb4d00e641ad4a23fc9a157e2e10ae57e6cb
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:09:45 2026 +0900
[SPARK-60908][CONNECT][FOLLOWUP] Convert NumPy scalars, reject inexact
integers for floating types, and stop inference at deep nesting as PySpark does
### What changes were proposed in this pull request?
A follow-up of #167
([SPARK-60908](https://issues.apache.org/jira/browse/SPARK-60908)), which a
review of the last commits found two problems in:
- **NumPy scalars:** with a schema, `createDataFrame` now takes NumPy
scalars as PyArrow does, as the Python value that each stands for:
- an integer or a float for an integer type, the float truncated;
- a NumPy bool for a boolean or an integer type;
- an integer or a float for a floating type.
- **Integers for floating types:** as PyArrow does, an integer that a
floating type cannot hold exactly now raises: beyond 2^53 for `double`, and
2^24 for `float`.
- **Error messages:** they name the type with its module, as PyArrow's do,
such as `numpy.int64` or `decimal.Decimal`.
- **Nesting:** inference stops at 64 levels with `RecursionError`, as
PySpark's recursion does.
### Why are the changes needed?
- `createDataFrame([(np.int64(1), np.float32(0.5), np.bool_(True))], "a
bigint, b double, c boolean")` raised `ValueError: Could not convert
np.int64(1) ...`, where PySpark returns the row. The conversion before #167
took them, so this was a regression. Rows from NumPy arrays, such as
`list(zip(a, b))`, are of these values.
- `createDataFrame([(2**53 + 1,)], "a double")` silently stored
`9007199254740992.0`.
- A value that refers to itself, such as an object whose attribute is the
object or a list that contains itself, recursed in Rust until the native stack
overflowed and the interpreter crashed, where PySpark raises `RecursionError`.
Inference cannot rely on Python's recursion limit for that: its frames are
Rust frames, which the limit does not count. A level takes about 5 KiB of
stack, so the 64 levels fit a thread of 512 KiB. That is also deeper than the
server can analyze nested types in practice: its analysis of nested map types
grows about threefold per level, for PySpark too.
### Does this PR introduce _any_ user-facing change?
Yes. NumPy scalars convert with a schema, an inexact integer for a floating
type raises, and a value that refers to itself raises `RecursionError`, all as
in PySpark 4.3.0.
### How was this patch tested?
- Against a local Spark 4.3.0 server, a probe of fourteen cases gives the
same results and messages as official PySpark 4.3.0. The cases are:
- NumPy scalars for integer, floating and boolean types, out of range,
truncated and NaN;
- integers at and beyond 2^53 and 2^24;
- an object and a list that contain themselves.
The error class is `ValueError`, of which PyArrow's `ArrowInvalid` is a
subclass.
- Inference of 63 nested levels succeeds on a thread with a 512 KiB stack,
and from 64 levels it raises `RecursionError` there; it crashed at 100 levels
before.
- New offline tests check the NumPy conversions, the errors and the
`RecursionError`. A new live test checks the values, and it also passes with
official PySpark 4.3.0.
- The lib tests and every integration test, e2e included, against the local
server (643 passed), the build of all tests with `wasm-udf`, the offline Python
suites (615 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #171 from HyukjinKwon/SPARK-60908-followup.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/infer.rs | 164 +++++++++++++++++++++++++++++++-----
python/tests/test_dropin_e2e.py | 8 ++
python/tests/test_dropin_offline.py | 40 +++++++++
3 files changed, 191 insertions(+), 21 deletions(-)
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 4d3386b..c820800 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -179,8 +179,44 @@ fn field_of(name: &Bound<'_, PyAny>, data_type: DataType)
-> PyResult<StructFiel
}
}
+/// How deep values may nest in inference. PySpark recurses in Python, so a
value nested
+/// deeper than its recursion limit allows, as an object that refers to
itself, raises
+/// `RecursionError`. Inference here recurses in Rust frames, which that limit
does not count
+/// and which could overflow the thread's stack, so it keeps a depth of its
own. A level takes
+/// about 5 KiB of stack, so 64 levels fit a thread of 512 KiB, and they are
deeper than the
+/// server can analyze nested types in practice.
+const MAX_DEPTH: usize = 64;
+
+thread_local! {
+ static DEPTH: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+}
+
+/// One more level of nesting for as long as it lives.
+struct Nesting;
+
+impl Nesting {
+ fn enter() -> PyResult<Nesting> {
+ DEPTH.with(|depth| {
+ if depth.get() >= MAX_DEPTH {
+ return Err(pyo3::exceptions::PyRecursionError::new_err(
+ "maximum recursion depth exceeded while inferring the type
of a value",
+ ));
+ }
+ depth.set(depth.get() + 1);
+ Ok(Nesting)
+ })
+ }
+}
+
+impl Drop for Nesting {
+ fn drop(&mut self) {
+ DEPTH.with(|depth| depth.set(depth.get() - 1));
+ }
+}
+
/// The type of `obj`, as PySpark's `_infer_type`.
pub(crate) fn infer_type(obj: &Bound<'_, PyAny>, o: Options<'_>) ->
PyResult<DataType> {
+ let _nesting = Nesting::enter()?;
let py = obj.py();
if obj.is_none() {
return Ok(DataType::Null);
@@ -721,15 +757,83 @@ fn unsupported(obj: &Bound<'_, PyAny>) -> PyResult<PyErr>
{
)))
}
+/// The name of the type of `obj` as PyArrow writes it in its errors: with its
module, unless
+/// it is a builtin type, such as `int` or `numpy.int64`.
+fn arrow_type_name(obj: &Bound<'_, PyAny>) -> PyResult<String> {
+ let ty = obj.get_type();
+ let (module, name) = (ty.module()?.to_string(),
ty.qualname()?.to_string());
+ Ok(if module == "builtins" {
+ name
+ } else {
+ format!("{module}.{name}")
+ })
+}
+
/// The error of a value that PyArrow cannot convert to the type it is named
by.
fn arrow_invalid(obj: &Bound<'_, PyAny>, arrow_type: &str) -> PyResult<PyErr> {
Ok(PyValueError::new_err(format!(
"Could not convert {} with type {}: tried to convert to {arrow_type}",
obj.repr()?,
- type_name(obj)?
+ arrow_type_name(obj)?
)))
}
+/// The kind of a NumPy scalar of a bool or a number, which PyArrow converts
as the Python
+/// value that it stands for.
+#[derive(Clone, Copy, PartialEq)]
+enum NumpyScalar {
+ Bool,
+ Integer,
+ Floating,
+}
+
+/// The kind of `obj` if it is a NumPy scalar of a bool or a number. NumPy is
not imported
+/// when it is not loaded, as such a value can only exist once it was.
+fn numpy_scalar(obj: &Bound<'_, PyAny>) -> PyResult<Option<NumpyScalar>> {
+ let py = obj.py();
+ let modules = py
+ .import(intern!(py, "sys"))?
+ .getattr(intern!(py, "modules"))?;
+ let numpy = match modules.cast::<PyDict>()?.get_item(intern!(py,
"numpy"))? {
+ Some(numpy) if !numpy.is_none() => numpy,
+ _ => return Ok(None),
+ };
+ for (class, kind) in [
+ ("bool_", NumpyScalar::Bool),
+ ("integer", NumpyScalar::Integer),
+ ("floating", NumpyScalar::Floating),
+ ] {
+ if obj.is_instance(&numpy.getattr(class)?)? {
+ return Ok(Some(kind));
+ }
+ }
+ Ok(None)
+}
+
+/// An integer as a floating-point value, which PyArrow takes only when it is
exact: within
+/// 2^53 for a double and 2^24 for a float.
+fn exact_float(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<f64> {
+ let (bits, precision) = match dt {
+ DataType::Float => (24, "single"),
+ _ => (53, "double"),
+ };
+ let out_of_range = || -> PyResult<PyErr> {
+ Ok(PyValueError::new_err(format!(
+ "Integer value {} is outside of the range exactly representable by
a IEEE 754 \
+ {precision} precision value",
+ obj.str()?
+ )))
+ };
+ let value: i64 = match obj.extract() {
+ Ok(value) => value,
+ Err(_) => return Err(out_of_range()?),
+ };
+ if value.unsigned_abs() > 1u64 << bits {
+ return Err(out_of_range()?);
+ }
+ Ok(value as f64)
+}
+
/// Microseconds of a `timedelta`, exactly.
fn delta_micros(delta: &Bound<'_, PyAny>) -> PyResult<i64> {
let py = delta.py();
@@ -903,7 +1007,7 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
}
}
DataType::Boolean => {
- if obj.is_instance_of::<PyBool>() {
+ if obj.is_instance_of::<PyBool>() || numpy_scalar(obj)? ==
Some(NumpyScalar::Bool) {
Ok(Value::Bool(obj.is_truthy()?))
} else {
Err(arrow_invalid(obj, "boolean")?)
@@ -913,21 +1017,30 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
convert_integral(obj, dt)
}
DataType::Float | DataType::Double => {
- // PyArrow takes a float, an int or a bool.
- if obj.is_instance_of::<PyFloat>() ||
obj.is_instance_of::<PyInt>() {
- let value: f64 = obj.extract()?;
- Ok(match dt {
- DataType::Float => Value::Float(value as f32),
- _ => Value::Double(value),
- })
+ // PyArrow takes a float, a bool, an int that it represents
exactly, and a NumPy
+ // float or integer, but not a NumPy bool.
+ let value = if obj.is_instance_of::<PyFloat>() ||
obj.is_instance_of::<PyBool>() {
+ obj.extract::<f64>()?
+ } else if obj.is_instance_of::<PyInt>() {
+ exact_float(obj, dt)?
} else {
- let arrow_type = if matches!(dt, DataType::Float) {
- "float"
- } else {
- "double"
- };
- Err(arrow_invalid(obj, arrow_type)?)
- }
+ match numpy_scalar(obj)? {
+ Some(NumpyScalar::Floating) => obj.extract::<f64>()?,
+ Some(NumpyScalar::Integer) => exact_float(obj, dt)?,
+ _ => {
+ let arrow_type = if matches!(dt, DataType::Float) {
+ "float"
+ } else {
+ "double"
+ };
+ return Err(arrow_invalid(obj, arrow_type)?);
+ }
+ }
+ };
+ Ok(match dt {
+ DataType::Float => Value::Float(value as f32),
+ _ => Value::Double(value),
+ })
}
DataType::Date => {
// PyArrow takes a date, or the date of a datetime.
@@ -952,17 +1065,26 @@ fn convert_integral(obj: &Bound<'_, PyAny>, dt:
&DataType) -> PyResult<Value> {
if obj.is_instance_of::<PyBool>() {
return Err(PyTypeError::new_err("Expected integer, got bool"));
}
- let value: i64 = if obj.is_instance_of::<PyInt>() {
- obj.extract()?
- } else if obj.is_instance_of::<PyFloat>() {
+ // A float, also of NumPy, is truncated, as PyArrow does.
+ let truncated = |obj: &Bound<'_, PyAny>| -> PyResult<i64> {
let value: f64 = obj.extract()?;
// The range of i64 is [-2^63, 2^63), both bounds exact as f64.
if !(i64::MIN as f64..i64::MAX as f64).contains(&value.trunc()) {
return Err(arrow_invalid(obj, arrow_type)?);
}
- value.trunc() as i64
+ Ok(value.trunc() as i64)
+ };
+ let value: i64 = if obj.is_instance_of::<PyInt>() {
+ obj.extract()?
+ } else if obj.is_instance_of::<PyFloat>() {
+ truncated(obj)?
} else {
- return Err(arrow_invalid(obj, arrow_type)?);
+ match numpy_scalar(obj)? {
+ Some(NumpyScalar::Integer) => obj.extract()?,
+ Some(NumpyScalar::Floating) => truncated(obj)?,
+ Some(NumpyScalar::Bool) => i64::from(obj.is_truthy()?),
+ None => return Err(arrow_invalid(obj, arrow_type)?),
+ }
};
if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits -
1))).contains(&value) {
return Err(PyValueError::new_err(format!(
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 932c2ab..c72dbdf 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -594,6 +594,14 @@ def
test_create_dataframe_raises_merge_errors_as_pyspark(session):
assert error.value.getMessageParameters() == parameters, data
+def test_create_dataframe_converts_numpy_scalars_as_pyspark(session):
+ np = pytest.importorskip("numpy")
+ df = session.createDataFrame(
+ [(np.int64(1), np.float32(0.5), np.bool_(True), np.float32(1.5))],
+ "a bigint, b double, c boolean, d bigint")
+ assert [tuple(r) for r in df.collect()] == [(1, 0.5, True, 1)]
+
+
def
test_create_dataframe_infers_by_the_session_configurations_as_pyspark(session):
naive = datetime(2020, 1, 2, 3, 4, 5)
cases = [
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index da64eb2..c0ec733 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -3577,6 +3577,46 @@ def
test_create_dataframe_warns_of_the_arguments_that_connect_ignores():
_local_create_dataframe([(1,)], None, None, True)
+def test_create_dataframe_converts_numpy_scalars_as_pyarrow():
+ import re
+
+ np = pytest.importorskip("numpy")
+ # As PySpark, where PyArrow takes a NumPy scalar as the Python value that
it stands for:
+ # an integer, a float (truncated for an integer type) and a bool (also for
an integer).
+ assert _local_create_dataframe(
+ [(np.int64(1), np.float32(0.5), np.bool_(True), np.float32(1.5),
np.bool_(True))],
+ "a bigint, b double, c boolean, d bigint, e bigint").isLocal()
+ for value, schema, message in [
+ (np.bool_(True), "a double",
+ r"Could not convert \S+ with type numpy\.bool_?: tried to convert to
double"),
+ (np.int64(1), "a boolean",
+ r"Could not convert \S+ with type numpy\.int64: tried to convert to
boolean"),
+ (np.int32(300), "a tinyint", re.escape("Value 300 too large to fit in
C integer type")),
+ # An integer that a float type cannot hold exactly.
+ (2**53 + 1, "a double", re.escape(
+ "Integer value 9007199254740993 is outside of the range exactly
representable by "
+ "a IEEE 754 double precision value")),
+ (2**24 + 1, "a float", re.escape(
+ "Integer value 16777217 is outside of the range exactly
representable by "
+ "a IEEE 754 single precision value")),
+ ]:
+ with pytest.raises(ValueError, match=message):
+ _local_create_dataframe([(value,)], schema)
+
+
+def
test_create_dataframe_raises_recursion_error_for_a_value_that_contains_itself():
+ class Node:
+ def __init__(self):
+ self.parent = self
+
+ cyclic = []
+ cyclic.append(cyclic)
+ # As PySpark, rather than overflowing the stack.
+ for value in [Node(), cyclic]:
+ with pytest.raises(RecursionError):
+ _local_create_dataframe([(value,)])
+
+
# ---------------------------------------------------------------------------
schema cache
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]