This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 24b6bc2 [SPARK-60661][CONNECT][PYTHON] Avoid generic detection for
exact built-in numbers
24b6bc2 is described below
commit 24b6bc2bbf861d07dffcc3f061f85fb6333a8dbf
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 16:33:09 2026 +0900
[SPARK-60661][CONNECT][PYTHON] Avoid generic detection for exact built-in
numbers
### What changes were proposed in this pull request?
`py_to_value`, the generic Python-to-`Value` conversion, now converts an
exact built-in `int` that fits in 64 bits, and an exact built-in `float`,
directly, before its generic checks (bool extraction, `Decimal` detection, and
the numeric extraction attempts). Everything else keeps the generic path:
`bool` (a subclass of `int`), other subclasses of `int` and `float`, objects
with `__index__` or `__float__`, `Decimal`, and integers outside the 64-bit
range.
`py_to_value` converts the cells of pandas input
(`createDataFrame(pandas_df)` turns every column into Python objects first),
and the values of `fillna`, `pivot` and `Row` conversion. Local rows go through
`infer.convert`, which already dispatches built-in numbers directly.
### Why are the changes needed?
[SPARK-60661](https://issues.apache.org/jira/browse/SPARK-60661): every
numeric cell first went through a failing `bool` extraction, which builds and
drops a Python error, and the `Decimal` check. An earlier version of this
change, measured on macOS arm64 with Python 3.12 when local rows still went
through `py_to_value`, used 38-74% less CPU to convert numeric tuples and
arrays. It has not been re-measured on the current master, where it applies to
the inputs above.
### Does this PR introduce _any_ user-facing change?
No. The converted values and types are the same.
### How was this patch tested?
- New offline tests send pandas input through a mock Connect server and
check the Arrow values and types: `None`, `bool`, the int64 bounds, signed
zeros, infinities, a NaN payload (bit for bit), nested lists and maps,
subclasses of `int` and `float`, objects with `__index__` and `__float__`,
`Decimal` and its subclasses, and integers outside the int64 range (which keep
the generic conversion to a double; PySpark raises `OverflowError` for them, a
known difference that this does not change).
- Against the current master, merged into this branch, and a local Spark
4.3.0 server: the lib and every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (771 passed), the build of all tests
with `wasm-udf`, the offline Python suites (736 passed, 9 skipped), the live
drop-in and Arrow suites (76 passed) and the live cases (10 passed). `cargo fmt
--all --check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master and the review
follow-ups by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #111 from dbtsai/codex/rsi-native-python-numbers.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 11 ++++
python/tests/test_dropin_offline.py | 110 ++++++++++++++++++++++++++++++++++++
2 files changed, 121 insertions(+)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 1f7af82..6d2ba32 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1449,6 +1449,17 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Value> {
}
}
+ // Exact built-in numbers need neither bool coercion nor Decimal detection.
+ // Keep subclasses and overflowing ints on the existing generic path.
+ if let Ok(i) = obj.cast_exact::<pyo3::types::PyInt>() {
+ if let Ok(i) = i.extract::<i64>() {
+ return Ok(Value::Long(i));
+ }
+ }
+ if let Ok(f) = obj.cast_exact::<pyo3::types::PyFloat>() {
+ return Ok(Value::Double(f.value()));
+ }
+
// Check for bool (before int because bool is a subclass of int in Python)
if let Ok(b) = obj.extract::<bool>() {
return Ok(Value::Bool(b));
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index f42f87c..70eb45d 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4041,6 +4041,116 @@ def
test_create_dataframe_raises_recursion_error_for_a_value_that_contains_itsel
_local_create_dataframe([(value,)])
+# py_to_value converts the cells of object-dtype pandas input, as well as the
values of
+# fillna, pivot and Row conversion; local rows go through infer.convert
instead, so the
+# "local" source is a control for values that the exact-number fast path must
not change.
+def _native_number_input_table(mock_connect, data, schema=None, *,
source="local"):
+ import pyarrow as pa
+
+ if source == "pandas":
+ pd = pytest.importorskip("pandas")
+ # Local rows now use infer.convert. Object-backed pandas input retains
+ # py_to_value, which is the generic converter optimized by this patch.
+ data = pd.DataFrame({"value": [row[0] for row in data]}, dtype=object)
+ spark, service = mock_connect
+ spark.createDataFrame(data, schema).toArrow()
+ request = parse_message(service.execute_requests[-1])
+ relation = parse_message(parse_message(request[3][0])[1][0])
+ while 18 in relation: # Relation.to_df -> ToDF.input.
+ relation = parse_message(parse_message(relation[18][0])[1][0])
+ local = parse_message(relation[11][0])
+ return pa.ipc.open_stream(local[1][0]).read_all()
+
+
[email protected]("value", [
+ None, False, True, 0, -1, -(2**63), 2**63 - 1,
+ 0.0, -0.0, 1.25, float("inf"), float("-inf"),
+])
+def test_local_input_native_number_values_and_types(mock_connect, value):
+ import struct
+
+ dtype = T.BooleanType() if type(value) is bool else (
+ T.LongType() if type(value) is int else T.DoubleType())
+ schema = T.StructType([T.StructField("value", dtype)])
+ actual = _native_number_input_table(
+ mock_connect, [Row(value=value)], schema, source="pandas"
+ ).column(0)[0].as_py()
+ assert type(actual) is type(value)
+ if type(value) is float:
+ assert struct.pack(">d", actual) == struct.pack(">d", value)
+ else:
+ assert actual == value
+
+
+def
test_pandas_input_overflow_preserves_generic_conversion_behavior(mock_connect):
+ # This checks only that the exact-int fast path leaves integers outside
the int64 range
+ # to the generic path. That path converts them to a double, which is a
known difference
+ # from PySpark: it raises OverflowError ("Python int too large to convert
to C long").
+ pd = pytest.importorskip("pandas")
+ for value in (2**63, -(2**63) - 1):
+ actual = _native_number_input_table(
+ mock_connect, [Row(value=value)], source="pandas"
+ ).column(0)[0].as_py()
+ assert type(actual) is float
+ assert actual == float(value)
+ with pytest.raises(TypeError, match="unsupported value type.*int"):
+ mock_connect[0].createDataFrame(pd.DataFrame({"value": [2**20000]},
dtype=object))
+
+
+def test_local_input_nan_payload(mock_connect):
+ import struct
+
+ bits = bytes.fromhex("7ff8000000000042")
+ value = struct.unpack(">d", bits)[0]
+ actual = _native_number_input_table(mock_connect,
[Row(value=value)]).column(0)[0].as_py()
+ assert struct.pack(">d", actual) == bits
+
+
+def
test_pandas_input_number_subclasses_decimal_and_conversion_protocols(mock_connect):
+ from decimal import Decimal
+
+ trace = []
+
+ class IntSubclass(int):
+ def __float__(self):
+ raise AssertionError("int subclass must retain integer conversion")
+
+ class FloatSubclass(float):
+ def __float__(self):
+ raise AssertionError("float subclass must retain float extraction")
+
+ class Indexable:
+ def __index__(self):
+ trace.append("index")
+ return 42
+
+ class Floatable:
+ def __float__(self):
+ trace.append("float")
+ return -1.25
+
+ class DecimalSubclass(Decimal):
+ pass
+
+ for value, expected in [(IntSubclass(13), 13), (FloatSubclass(2.25), 2.25),
+ (Indexable(), 42), (Floatable(), -1.25)]:
+ actual = _native_number_input_table(mock_connect, [(value,)],
source="pandas").column(0)[0].as_py()
+ assert type(actual) is type(expected)
+ assert actual == expected
+ assert trace == ["index", "float"]
+ for value in (Decimal("-12.340"), DecimalSubclass("1.25")):
+ actual = _native_number_input_table(mock_connect, [(value,)],
source="pandas").column(0)[0].as_py()
+ assert type(actual) is Decimal
+ assert actual == value
+
+
+def test_local_input_native_numbers_inside_nested_values(mock_connect):
+ value = Row(items=[1, 2, None], mapping={"x": 2**63 - 1})
+ actual = _native_number_input_table(mock_connect, [Row(value=value)],
source="pandas").column(0)[0].as_py()
+ assert actual["items"] == [1, 2, None]
+ assert dict(actual["mapping"]) == {"x": 2**63 - 1}
+
+
# ---------------------------------------------------------------------------
schema cache
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]