This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 13afbb4  [SPARK-60908][CONNECT][FOLLOWUP] Convert NumPy scalars, 
reject inexact integers for floating types, and stop inference at deep nesting 
as PySpark does
13afbb4 is described below

commit 13afbb4d00e641ad4a23fc9a157e2e10ae57e6cb
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:09:45 2026 +0900

    [SPARK-60908][CONNECT][FOLLOWUP] Convert NumPy scalars, reject inexact 
integers for floating types, and stop inference at deep nesting as PySpark does
    
    ### What changes were proposed in this pull request?
    
    A follow-up of #167 
([SPARK-60908](https://issues.apache.org/jira/browse/SPARK-60908)), which a 
review of the last commits found two problems in:
    
    - **NumPy scalars:** with a schema, `createDataFrame` now takes NumPy 
scalars as PyArrow does, as the Python value that each stands for:
      - an integer or a float for an integer type, the float truncated;
      - a NumPy bool for a boolean or an integer type;
      - an integer or a float for a floating type.
    - **Integers for floating types:** as PyArrow does, an integer that a 
floating type cannot hold exactly now raises: beyond 2^53 for `double`, and 
2^24 for `float`.
    - **Error messages:** they name the type with its module, as PyArrow's do, 
such as `numpy.int64` or `decimal.Decimal`.
    - **Nesting:** inference stops at 64 levels with `RecursionError`, as 
PySpark's recursion does.
    
    ### Why are the changes needed?
    
    - `createDataFrame([(np.int64(1), np.float32(0.5), np.bool_(True))], "a 
bigint, b double, c boolean")` raised `ValueError: Could not convert 
np.int64(1) ...`, where PySpark returns the row. The conversion before #167 
took them, so this was a regression. Rows from NumPy arrays, such as 
`list(zip(a, b))`, are of these values.
    - `createDataFrame([(2**53 + 1,)], "a double")` silently stored 
`9007199254740992.0`.
    - A value that refers to itself, such as an object whose attribute is the 
object or a list that contains itself, recursed in Rust until the native stack 
overflowed and the interpreter crashed, where PySpark raises `RecursionError`.
    
    Inference cannot rely on Python's recursion limit for that: its frames are 
Rust frames, which the limit does not count. A level takes about 5 KiB of 
stack, so the 64 levels fit a thread of 512 KiB. That is also deeper than the 
server can analyze nested types in practice: its analysis of nested map types 
grows about threefold per level, for PySpark too.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. NumPy scalars convert with a schema, an inexact integer for a floating 
type raises, and a value that refers to itself raises `RecursionError`, all as 
in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of fourteen cases gives the 
same results and messages as official PySpark 4.3.0. The cases are:
      - NumPy scalars for integer, floating and boolean types, out of range, 
truncated and NaN;
      - integers at and beyond 2^53 and 2^24;
      - an object and a list that contain themselves.
    
      The error class is `ValueError`, of which PyArrow's `ArrowInvalid` is a 
subclass.
    - Inference of 63 nested levels succeeds on a thread with a 512 KiB stack, 
and from 64 levels it raises `RecursionError` there; it crashed at 100 levels 
before.
    - New offline tests check the NumPy conversions, the errors and the 
`RecursionError`. A new live test checks the values, and it also passes with 
official PySpark 4.3.0.
    - The lib tests and every integration test, e2e included, against the local 
server (643 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (615 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #171 from HyukjinKwon/SPARK-60908-followup.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/infer.rs      | 164 +++++++++++++++++++++++++++++++-----
 python/tests/test_dropin_e2e.py     |   8 ++
 python/tests/test_dropin_offline.py |  40 +++++++++
 3 files changed, 191 insertions(+), 21 deletions(-)

diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 4d3386b..c820800 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -179,8 +179,44 @@ fn field_of(name: &Bound<'_, PyAny>, data_type: DataType) 
-> PyResult<StructFiel
     }
 }
 
+/// How deep values may nest in inference. PySpark recurses in Python, so a 
value nested
+/// deeper than its recursion limit allows, as an object that refers to 
itself, raises
+/// `RecursionError`. Inference here recurses in Rust frames, which that limit 
does not count
+/// and which could overflow the thread's stack, so it keeps a depth of its 
own. A level takes
+/// about 5 KiB of stack, so 64 levels fit a thread of 512 KiB, and they are 
deeper than the
+/// server can analyze nested types in practice.
+const MAX_DEPTH: usize = 64;
+
+thread_local! {
+    static DEPTH: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+}
+
+/// One more level of nesting for as long as it lives.
+struct Nesting;
+
+impl Nesting {
+    fn enter() -> PyResult<Nesting> {
+        DEPTH.with(|depth| {
+            if depth.get() >= MAX_DEPTH {
+                return Err(pyo3::exceptions::PyRecursionError::new_err(
+                    "maximum recursion depth exceeded while inferring the type 
of a value",
+                ));
+            }
+            depth.set(depth.get() + 1);
+            Ok(Nesting)
+        })
+    }
+}
+
+impl Drop for Nesting {
+    fn drop(&mut self) {
+        DEPTH.with(|depth| depth.set(depth.get() - 1));
+    }
+}
+
 /// The type of `obj`, as PySpark's `_infer_type`.
 pub(crate) fn infer_type(obj: &Bound<'_, PyAny>, o: Options<'_>) -> 
PyResult<DataType> {
+    let _nesting = Nesting::enter()?;
     let py = obj.py();
     if obj.is_none() {
         return Ok(DataType::Null);
@@ -721,15 +757,83 @@ fn unsupported(obj: &Bound<'_, PyAny>) -> PyResult<PyErr> 
{
     )))
 }
 
+/// The name of the type of `obj` as PyArrow writes it in its errors: with its 
module, unless
+/// it is a builtin type, such as `int` or `numpy.int64`.
+fn arrow_type_name(obj: &Bound<'_, PyAny>) -> PyResult<String> {
+    let ty = obj.get_type();
+    let (module, name) = (ty.module()?.to_string(), 
ty.qualname()?.to_string());
+    Ok(if module == "builtins" {
+        name
+    } else {
+        format!("{module}.{name}")
+    })
+}
+
 /// The error of a value that PyArrow cannot convert to the type it is named 
by.
 fn arrow_invalid(obj: &Bound<'_, PyAny>, arrow_type: &str) -> PyResult<PyErr> {
     Ok(PyValueError::new_err(format!(
         "Could not convert {} with type {}: tried to convert to {arrow_type}",
         obj.repr()?,
-        type_name(obj)?
+        arrow_type_name(obj)?
     )))
 }
 
+/// The kind of a NumPy scalar of a bool or a number, which PyArrow converts 
as the Python
+/// value that it stands for.
+#[derive(Clone, Copy, PartialEq)]
+enum NumpyScalar {
+    Bool,
+    Integer,
+    Floating,
+}
+
+/// The kind of `obj` if it is a NumPy scalar of a bool or a number. NumPy is 
not imported
+/// when it is not loaded, as such a value can only exist once it was.
+fn numpy_scalar(obj: &Bound<'_, PyAny>) -> PyResult<Option<NumpyScalar>> {
+    let py = obj.py();
+    let modules = py
+        .import(intern!(py, "sys"))?
+        .getattr(intern!(py, "modules"))?;
+    let numpy = match modules.cast::<PyDict>()?.get_item(intern!(py, 
"numpy"))? {
+        Some(numpy) if !numpy.is_none() => numpy,
+        _ => return Ok(None),
+    };
+    for (class, kind) in [
+        ("bool_", NumpyScalar::Bool),
+        ("integer", NumpyScalar::Integer),
+        ("floating", NumpyScalar::Floating),
+    ] {
+        if obj.is_instance(&numpy.getattr(class)?)? {
+            return Ok(Some(kind));
+        }
+    }
+    Ok(None)
+}
+
+/// An integer as a floating-point value, which PyArrow takes only when it is 
exact: within
+/// 2^53 for a double and 2^24 for a float.
+fn exact_float(obj: &Bound<'_, PyAny>, dt: &DataType) -> PyResult<f64> {
+    let (bits, precision) = match dt {
+        DataType::Float => (24, "single"),
+        _ => (53, "double"),
+    };
+    let out_of_range = || -> PyResult<PyErr> {
+        Ok(PyValueError::new_err(format!(
+            "Integer value {} is outside of the range exactly representable by 
a IEEE 754 \
+             {precision} precision value",
+            obj.str()?
+        )))
+    };
+    let value: i64 = match obj.extract() {
+        Ok(value) => value,
+        Err(_) => return Err(out_of_range()?),
+    };
+    if value.unsigned_abs() > 1u64 << bits {
+        return Err(out_of_range()?);
+    }
+    Ok(value as f64)
+}
+
 /// Microseconds of a `timedelta`, exactly.
 fn delta_micros(delta: &Bound<'_, PyAny>) -> PyResult<i64> {
     let py = delta.py();
@@ -903,7 +1007,7 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
             }
         }
         DataType::Boolean => {
-            if obj.is_instance_of::<PyBool>() {
+            if obj.is_instance_of::<PyBool>() || numpy_scalar(obj)? == 
Some(NumpyScalar::Bool) {
                 Ok(Value::Bool(obj.is_truthy()?))
             } else {
                 Err(arrow_invalid(obj, "boolean")?)
@@ -913,21 +1017,30 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
             convert_integral(obj, dt)
         }
         DataType::Float | DataType::Double => {
-            // PyArrow takes a float, an int or a bool.
-            if obj.is_instance_of::<PyFloat>() || 
obj.is_instance_of::<PyInt>() {
-                let value: f64 = obj.extract()?;
-                Ok(match dt {
-                    DataType::Float => Value::Float(value as f32),
-                    _ => Value::Double(value),
-                })
+            // PyArrow takes a float, a bool, an int that it represents 
exactly, and a NumPy
+            // float or integer, but not a NumPy bool.
+            let value = if obj.is_instance_of::<PyFloat>() || 
obj.is_instance_of::<PyBool>() {
+                obj.extract::<f64>()?
+            } else if obj.is_instance_of::<PyInt>() {
+                exact_float(obj, dt)?
             } else {
-                let arrow_type = if matches!(dt, DataType::Float) {
-                    "float"
-                } else {
-                    "double"
-                };
-                Err(arrow_invalid(obj, arrow_type)?)
-            }
+                match numpy_scalar(obj)? {
+                    Some(NumpyScalar::Floating) => obj.extract::<f64>()?,
+                    Some(NumpyScalar::Integer) => exact_float(obj, dt)?,
+                    _ => {
+                        let arrow_type = if matches!(dt, DataType::Float) {
+                            "float"
+                        } else {
+                            "double"
+                        };
+                        return Err(arrow_invalid(obj, arrow_type)?);
+                    }
+                }
+            };
+            Ok(match dt {
+                DataType::Float => Value::Float(value as f32),
+                _ => Value::Double(value),
+            })
         }
         DataType::Date => {
             // PyArrow takes a date, or the date of a datetime.
@@ -952,17 +1065,26 @@ fn convert_integral(obj: &Bound<'_, PyAny>, dt: 
&DataType) -> PyResult<Value> {
     if obj.is_instance_of::<PyBool>() {
         return Err(PyTypeError::new_err("Expected integer, got bool"));
     }
-    let value: i64 = if obj.is_instance_of::<PyInt>() {
-        obj.extract()?
-    } else if obj.is_instance_of::<PyFloat>() {
+    // A float, also of NumPy, is truncated, as PyArrow does.
+    let truncated = |obj: &Bound<'_, PyAny>| -> PyResult<i64> {
         let value: f64 = obj.extract()?;
         // The range of i64 is [-2^63, 2^63), both bounds exact as f64.
         if !(i64::MIN as f64..i64::MAX as f64).contains(&value.trunc()) {
             return Err(arrow_invalid(obj, arrow_type)?);
         }
-        value.trunc() as i64
+        Ok(value.trunc() as i64)
+    };
+    let value: i64 = if obj.is_instance_of::<PyInt>() {
+        obj.extract()?
+    } else if obj.is_instance_of::<PyFloat>() {
+        truncated(obj)?
     } else {
-        return Err(arrow_invalid(obj, arrow_type)?);
+        match numpy_scalar(obj)? {
+            Some(NumpyScalar::Integer) => obj.extract()?,
+            Some(NumpyScalar::Floating) => truncated(obj)?,
+            Some(NumpyScalar::Bool) => i64::from(obj.is_truthy()?),
+            None => return Err(arrow_invalid(obj, arrow_type)?),
+        }
     };
     if bits < 64 && !(-(1i64 << (bits - 1))..(1i64 << (bits - 
1))).contains(&value) {
         return Err(PyValueError::new_err(format!(
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 932c2ab..c72dbdf 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -594,6 +594,14 @@ def 
test_create_dataframe_raises_merge_errors_as_pyspark(session):
         assert error.value.getMessageParameters() == parameters, data
 
 
+def test_create_dataframe_converts_numpy_scalars_as_pyspark(session):
+    np = pytest.importorskip("numpy")
+    df = session.createDataFrame(
+        [(np.int64(1), np.float32(0.5), np.bool_(True), np.float32(1.5))],
+        "a bigint, b double, c boolean, d bigint")
+    assert [tuple(r) for r in df.collect()] == [(1, 0.5, True, 1)]
+
+
 def 
test_create_dataframe_infers_by_the_session_configurations_as_pyspark(session):
     naive = datetime(2020, 1, 2, 3, 4, 5)
     cases = [
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index da64eb2..c0ec733 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -3577,6 +3577,46 @@ def 
test_create_dataframe_warns_of_the_arguments_that_connect_ignores():
         _local_create_dataframe([(1,)], None, None, True)
 
 
+def test_create_dataframe_converts_numpy_scalars_as_pyarrow():
+    import re
+
+    np = pytest.importorskip("numpy")
+    # As PySpark, where PyArrow takes a NumPy scalar as the Python value that 
it stands for:
+    # an integer, a float (truncated for an integer type) and a bool (also for 
an integer).
+    assert _local_create_dataframe(
+        [(np.int64(1), np.float32(0.5), np.bool_(True), np.float32(1.5), 
np.bool_(True))],
+        "a bigint, b double, c boolean, d bigint, e bigint").isLocal()
+    for value, schema, message in [
+        (np.bool_(True), "a double",
+         r"Could not convert \S+ with type numpy\.bool_?: tried to convert to 
double"),
+        (np.int64(1), "a boolean",
+         r"Could not convert \S+ with type numpy\.int64: tried to convert to 
boolean"),
+        (np.int32(300), "a tinyint", re.escape("Value 300 too large to fit in 
C integer type")),
+        # An integer that a float type cannot hold exactly.
+        (2**53 + 1, "a double", re.escape(
+            "Integer value 9007199254740993 is outside of the range exactly 
representable by "
+            "a IEEE 754 double precision value")),
+        (2**24 + 1, "a float", re.escape(
+            "Integer value 16777217 is outside of the range exactly 
representable by "
+            "a IEEE 754 single precision value")),
+    ]:
+        with pytest.raises(ValueError, match=message):
+            _local_create_dataframe([(value,)], schema)
+
+
+def 
test_create_dataframe_raises_recursion_error_for_a_value_that_contains_itself():
+    class Node:
+        def __init__(self):
+            self.parent = self
+
+    cyclic = []
+    cyclic.append(cyclic)
+    # As PySpark, rather than overflowing the stack.
+    for value in [Node(), cyclic]:
+        with pytest.raises(RecursionError):
+            _local_create_dataframe([(value,)])
+
+
 # --------------------------------------------------------------------------- 
schema cache
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to