This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 24b6bc2  [SPARK-60661][CONNECT][PYTHON] Avoid generic detection for 
exact built-in numbers
24b6bc2 is described below

commit 24b6bc2bbf861d07dffcc3f061f85fb6333a8dbf
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 16:33:09 2026 +0900

    [SPARK-60661][CONNECT][PYTHON] Avoid generic detection for exact built-in 
numbers
    
    ### What changes were proposed in this pull request?
    
    `py_to_value`, the generic Python-to-`Value` conversion, now converts an 
exact built-in `int` that fits in 64 bits, and an exact built-in `float`, 
directly, before its generic checks (bool extraction, `Decimal` detection, and 
the numeric extraction attempts). Everything else keeps the generic path: 
`bool` (a subclass of `int`), other subclasses of `int` and `float`, objects 
with `__index__` or `__float__`, `Decimal`, and integers outside the 64-bit 
range.
    
    `py_to_value` converts the cells of pandas input 
(`createDataFrame(pandas_df)` turns every column into Python objects first), 
and the values of `fillna`, `pivot` and `Row` conversion. Local rows go through 
`infer.convert`, which already dispatches built-in numbers directly.
    
    ### Why are the changes needed?
    
    [SPARK-60661](https://issues.apache.org/jira/browse/SPARK-60661): every 
numeric cell first went through a failing `bool` extraction, which builds and 
drops a Python error, and the `Decimal` check. An earlier version of this 
change, measured on macOS arm64 with Python 3.12 when local rows still went 
through `py_to_value`, used 38-74% less CPU to convert numeric tuples and 
arrays. It has not been re-measured on the current master, where it applies to 
the inputs above.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. The converted values and types are the same.
    
    ### How was this patch tested?
    
    - New offline tests send pandas input through a mock Connect server and 
check the Arrow values and types: `None`, `bool`, the int64 bounds, signed 
zeros, infinities, a NaN payload (bit for bit), nested lists and maps, 
subclasses of `int` and `float`, objects with `__index__` and `__float__`, 
`Decimal` and its subclasses, and integers outside the int64 range (which keep 
the generic conversion to a double; PySpark raises `OverflowError` for them, a 
known difference that this does not change).
    - Against the current master, merged into this branch, and a local Spark 
4.3.0 server: the lib and every integration test of `apache-spark-connect` and 
`apache-spark-connect-core`, e2e included (771 passed), the build of all tests 
with `wasm-udf`, the offline Python suites (736 passed, 9 skipped), the live 
drop-in and Arrow suites (76 passed) and the live cases (10 passed). `cargo fmt 
--all --check` passes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the merge with master and the review 
follow-ups by Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #111 from dbtsai/codex/rsi-native-python-numbers.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    |  11 ++++
 python/tests/test_dropin_offline.py | 110 ++++++++++++++++++++++++++++++++++++
 2 files changed, 121 insertions(+)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 1f7af82..6d2ba32 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1449,6 +1449,17 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Value> {
         }
     }
 
+    // Exact built-in numbers need neither bool coercion nor Decimal detection.
+    // Keep subclasses and overflowing ints on the existing generic path.
+    if let Ok(i) = obj.cast_exact::<pyo3::types::PyInt>() {
+        if let Ok(i) = i.extract::<i64>() {
+            return Ok(Value::Long(i));
+        }
+    }
+    if let Ok(f) = obj.cast_exact::<pyo3::types::PyFloat>() {
+        return Ok(Value::Double(f.value()));
+    }
+
     // Check for bool (before int because bool is a subclass of int in Python)
     if let Ok(b) = obj.extract::<bool>() {
         return Ok(Value::Bool(b));
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index f42f87c..70eb45d 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4041,6 +4041,116 @@ def 
test_create_dataframe_raises_recursion_error_for_a_value_that_contains_itsel
             _local_create_dataframe([(value,)])
 
 
+# py_to_value converts the cells of object-dtype pandas input, as well as the 
values of
+# fillna, pivot and Row conversion; local rows go through infer.convert 
instead, so the
+# "local" source is a control for values that the exact-number fast path must 
not change.
+def _native_number_input_table(mock_connect, data, schema=None, *, 
source="local"):
+    import pyarrow as pa
+
+    if source == "pandas":
+        pd = pytest.importorskip("pandas")
+        # Local rows now use infer.convert. Object-backed pandas input retains
+        # py_to_value, which is the generic converter optimized by this patch.
+        data = pd.DataFrame({"value": [row[0] for row in data]}, dtype=object)
+    spark, service = mock_connect
+    spark.createDataFrame(data, schema).toArrow()
+    request = parse_message(service.execute_requests[-1])
+    relation = parse_message(parse_message(request[3][0])[1][0])
+    while 18 in relation:  # Relation.to_df -> ToDF.input.
+        relation = parse_message(parse_message(relation[18][0])[1][0])
+    local = parse_message(relation[11][0])
+    return pa.ipc.open_stream(local[1][0]).read_all()
+
+
[email protected]("value", [
+    None, False, True, 0, -1, -(2**63), 2**63 - 1,
+    0.0, -0.0, 1.25, float("inf"), float("-inf"),
+])
+def test_local_input_native_number_values_and_types(mock_connect, value):
+    import struct
+
+    dtype = T.BooleanType() if type(value) is bool else (
+        T.LongType() if type(value) is int else T.DoubleType())
+    schema = T.StructType([T.StructField("value", dtype)])
+    actual = _native_number_input_table(
+        mock_connect, [Row(value=value)], schema, source="pandas"
+    ).column(0)[0].as_py()
+    assert type(actual) is type(value)
+    if type(value) is float:
+        assert struct.pack(">d", actual) == struct.pack(">d", value)
+    else:
+        assert actual == value
+
+
+def 
test_pandas_input_overflow_preserves_generic_conversion_behavior(mock_connect):
+    # This checks only that the exact-int fast path leaves integers outside 
the int64 range
+    # to the generic path. That path converts them to a double, which is a 
known difference
+    # from PySpark: it raises OverflowError ("Python int too large to convert 
to C long").
+    pd = pytest.importorskip("pandas")
+    for value in (2**63, -(2**63) - 1):
+        actual = _native_number_input_table(
+            mock_connect, [Row(value=value)], source="pandas"
+        ).column(0)[0].as_py()
+        assert type(actual) is float
+        assert actual == float(value)
+    with pytest.raises(TypeError, match="unsupported value type.*int"):
+        mock_connect[0].createDataFrame(pd.DataFrame({"value": [2**20000]}, 
dtype=object))
+
+
+def test_local_input_nan_payload(mock_connect):
+    import struct
+
+    bits = bytes.fromhex("7ff8000000000042")
+    value = struct.unpack(">d", bits)[0]
+    actual = _native_number_input_table(mock_connect, 
[Row(value=value)]).column(0)[0].as_py()
+    assert struct.pack(">d", actual) == bits
+
+
+def 
test_pandas_input_number_subclasses_decimal_and_conversion_protocols(mock_connect):
+    from decimal import Decimal
+
+    trace = []
+
+    class IntSubclass(int):
+        def __float__(self):
+            raise AssertionError("int subclass must retain integer conversion")
+
+    class FloatSubclass(float):
+        def __float__(self):
+            raise AssertionError("float subclass must retain float extraction")
+
+    class Indexable:
+        def __index__(self):
+            trace.append("index")
+            return 42
+
+    class Floatable:
+        def __float__(self):
+            trace.append("float")
+            return -1.25
+
+    class DecimalSubclass(Decimal):
+        pass
+
+    for value, expected in [(IntSubclass(13), 13), (FloatSubclass(2.25), 2.25),
+                            (Indexable(), 42), (Floatable(), -1.25)]:
+        actual = _native_number_input_table(mock_connect, [(value,)], 
source="pandas").column(0)[0].as_py()
+        assert type(actual) is type(expected)
+        assert actual == expected
+    assert trace == ["index", "float"]
+    for value in (Decimal("-12.340"), DecimalSubclass("1.25")):
+        actual = _native_number_input_table(mock_connect, [(value,)], 
source="pandas").column(0)[0].as_py()
+        assert type(actual) is Decimal
+        assert actual == value
+
+
+def test_local_input_native_numbers_inside_nested_values(mock_connect):
+    value = Row(items=[1, 2, None], mapping={"x": 2**63 - 1})
+    actual = _native_number_input_table(mock_connect, [Row(value=value)], 
source="pandas").column(0)[0].as_py()
+    assert actual["items"] == [1, 2, None]
+    assert dict(actual["mapping"]) == {"x": 2**63 - 1}
+
+
 # --------------------------------------------------------------------------- 
schema cache
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to