This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new ac40ac1  [SPARK-60684][CONNECT] Fast-path exact built-in string input
ac40ac1 is described below

commit ac40ac1c619664cd8de71a067267d20238991f13
Author: DB Tsai <[email protected]>
AuthorDate: Thu Nov 5 22:08:08 2026 +0900

    [SPARK-60684][CONNECT] Fast-path exact built-in string input
    
    ### What changes were proposed in this pull request?
    
    Fast-path exact built-in Python `str` values in `py_to_value` before bool, 
numeric, and Decimal probing. String subclasses and failed UTF-8 extraction 
retain the existing generic fallback, including numeric conversion protocols 
and surrogate errors. The numeric conversion code is unchanged.
    
    The focused offline tests cover exact string values (including Unicode, 
NUL, and empty text), the existing surrogate error fallback, and one `str` 
subclass with `__index__` to guard the exact-type check. Arrow encoding remains 
covered by the existing core tests.
    
    ### Why are the changes needed?
    
    Local string input currently pays generic scalar detection for every value. 
The original benchmark used five fresh-process samples per client/case on Apple 
M3 Max with arm64 Python 3.12.14 and release builds. Medians for 10,000 × 3 
string inputs were:
    
    | Source | Baseline | Candidate | Speedup |
    |---|---:|---:|---:|
    | tuples | 20.493 ms | 3.930 ms | 5.21× |
    | pandas | 22.310 ms | 5.313 ms | 4.20× |
    
    The Python reference measured 2.843 ms and 1.784 ms respectively. All 195 
samples across 13 cases and three clients passed the unchanged validators. 
Numeric, Decimal, date, timestamp, array, nested-result, and RPC guards showed 
no material regression; small differences remained noisy with five samples.
    
    These measurements predate the rebase and test cleanup: baseline 
`d884e347452c6446c7bc54af151755c62801d8cd`, Python reference 
`ca1514a2b4e72f9529871f34d09881cd4921df82`, frozen benchmark harness 
`1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4`. The correctness tests exercise 
Python value conversion directly through `Row`; the benchmark validates 
construction results separately.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Faster construction for exact built-in string values. Subclasses retain 
their existing numeric conversion behavior, and failed UTF-8 extraction follows 
the existing error path.
    
    ### How was this patch tested?
    
    Validation on master base `617fa1481b3e6fc7b825928130f280506328c9b1`:
    
    - Prior validation of the unchanged Rust implementation: 398 Rust library 
tests passed. Release integration-test compilation, WASM-feature 
integration-test compilation, `cargo fmt --all --check`, release PyO3 extension 
build, and `git diff --check` passed.
    - All 330 tests in `python/tests/test_dropin_offline.py` passed with normal 
imports and with `--import-mode=importlib`. Importlib mode includes 
`python/tests` on `PYTHONPATH` so the shared helper is available to spawned 
interpreters.
    - Advisory workspace Clippy still fails on existing Rust diagnostics. Every 
reported primary error location maps to unchanged code in master and the prior 
PR head; the review cleanup removes Python test/helper additions and leaves the 
Rust implementation unchanged.
    
    The offline suite uses scripted localhost fixtures. Live Spark/JVM behavior 
was not exercised by this validation.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    This pull request and its description were written with OpenAI Codex 
(GPT-6).
    
    Closes #127 from dbtsai/codex/spark-60684-exact-str.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    |  8 ++++++++
 python/tests/test_dropin_offline.py | 40 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index cd40ee8..aae3795 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1206,6 +1206,14 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Value> {
         return Ok(Value::Null);
     }
 
+    // Exact built-in strings cannot implement numeric conversion protocols.
+    // Keep subclasses and strings that fail UTF-8 extraction on the generic 
path.
+    if let Ok(s) = obj.cast_exact::<pyo3::types::PyString>() {
+        if let Ok(s) = s.extract::<String>() {
+            return Ok(Value::String(s));
+        }
+    }
+
     // Check for bool (before int because bool is a subclass of int in Python)
     if let Ok(b) = obj.extract::<bool>() {
         return Ok(Value::Bool(b));
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 68513c4..65e1365 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -24,7 +24,7 @@ from _mock_connect_server import (
     struct_schema,
     varint_field,
 )
-from pyspark.sql import SparkSession, types as T
+from pyspark.sql import Row, SparkSession, types as T
 
 # Collected here so the existing offline CI suite exercises real Arrow results.
 from day_time_interval_cases import DayTimeIntervalTests  # noqa: F401
@@ -4649,3 +4649,41 @@ def 
test_reader_schema_is_sent_as_pyspark_sends_it(mock_connect, read):
     assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
     assert error.value.getMessageParameters() == {
         "arg_name": "schema", "expected_type": "str or struct type", 
"arg_type": "LongType"}
+
+
+# Exact built-in string conversion guards. A Row made in Python keeps its 
values as they
+# are, as PySpark's does, so these check the conversion that createDataFrame 
makes.
+
+
[email protected]("text", [
+    "", "1.25", "-42", "NaN", "a\x00b", "é雪😀", "e\u0301", "\uffff",
+])
+def test_exact_string_converts_as_text(text):
+    from pyspark.errors import PySparkRuntimeError
+
+    # A string, even when it reads as a number: it shares a column with a 
string, not with
+    # an integer.
+    assert _local_create_dataframe([Row(value=text), Row(value="y")]).isLocal()
+    with pytest.raises(PySparkRuntimeError):
+        _local_create_dataframe([Row(value=text), Row(value=5)])
+
+
[email protected]("text", ["\ud800", "\udfff", "a\ud800b"])
+def test_exact_string_surrogate_error_preserves_generic_fallback(text):
+    with pytest.raises(TypeError):
+        _local_create_dataframe([Row(value=text)])
+
+
+def test_exact_string_subclass_uses_generic_conversion():
+    # Pin existing subclass behavior; treating subclasses as text is a separate
+    # behavior change.
+    from pyspark.errors import PySparkRuntimeError
+
+    class IndexString(str):
+        def __index__(self):
+            return 17
+
+    # Converted through __index__, it shares a column with an integer, not 
with a string.
+    assert _local_create_dataframe([Row(value=IndexString("x")), 
Row(value=5)]).isLocal()
+    with pytest.raises(PySparkRuntimeError):
+        _local_create_dataframe([Row(value=IndexString("x")), Row(value="y")])


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to