This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 9f057df  [SPARK-60727][CONNECT] Preserve customized pandas double 
conversion
9f057df is described below

commit 9f057df23f823d617c49441cebfceef0bd018590
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 19:03:03 2026 +0900

    [SPARK-60727][CONNECT] Preserve customized pandas double conversion
    
    ### What changes were proposed in this pull request?
    
    Preserve the existing row conversion of customized pandas float64 input in 
the fast path added by #207 
([SPARK-60727](https://issues.apache.org/jira/browse/SPARK-60727)).
    
    - A DataFrame constructed with `copy=False` can retain an ndarray subclass 
inside pandas' concrete NumPy wrapper. The column path skips that subclass's 
`astype` hook, which can change values or suppress an existing conversion 
error. Require exact ndarray storage before using this path.
    - Inspect public schema aliases through their namespace dictionary. A 
missing alias should decline the optimization without invoking a module's 
custom `__getattr__` before row conversion.
    
    Add three mock-service regression cases for transformed values, conversion 
errors, and a missing alias with a raising lazy hook. Ordinary ndarray input 
keeps the fast path; subclasses such as `np.memmap` take the row path.
    
    ### Why are the changes needed?
    
    These inputs are customized cases that #207 intended to keep on the 
existing row path. Both guard issues remain on current master. Official Spark 
4.3 also bypasses the ndarray hook; this change preserves the Rust client's 
prior custom-input behavior rather than claiming an official parity fix.
    
    ### Does this PR introduce any user-facing change?
    
    It restores the prior values and errors for the affected customized input. 
Public signatures and inline transport policy are unchanged.
    
    ### How was this patch tested?
    
    - Reproduced both issues with pandas 2.3.3 and 3.0.6, NumPy 2.5.4 and 
PyArrow 26.0.0.
    - Supplemental helper checks pass on both pandas versions, including 
ordinary ndarray float bits and null masks. These use private copies of native 
4.2 schema classes and do not prove current-master native dispatch.
    - Python syntax and `git diff --check` pass.
    - Genuine local Claude Code reviewed the landed patch and follow-up; the 
final full review and all findings are recorded in the local review evidence.
    
    **Draft: required validation remains pending.** The current environment has 
no usable Cargo/rustc toolchain, matching native extension, or protoc. The new 
public-API test attempt is blocked during collection; Rust compile/fmt/Clippy, 
complete native suites, API/official parity, and live Spark 4.3 checks have not 
passed for this candidate. Historical maintainer results are not claimed as 
validation of this follow-up.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex; reviewed with installed local Claude Code.
    
    This pull request and its description were written by Isaac.
    
    Closes #280 from dbtsai/codex/postmerge-pr-207-ndarray-storage.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/_pandas_input.py |  7 ++++-
 python/tests/test_arrow_handoff.py  | 51 +++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/sql/_pandas_input.py 
b/python/pyspark/sql/_pandas_input.py
index cf82827..e2aa9e2 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -474,10 +474,15 @@ def prepare_float64_table(frame, schema):
     if any(type(column) is not Series or type(column.array) is not 
NumpyExtensionArray
            for column in columns):
         return None
+    # NumPy subclasses can customize the row path's astype conversion even
+    # inside pandas' concrete NumPy wrapper. Keep their hooks, accepting the
+    # row path's cost for subclasses such as np.memmap.
+    if any(type(column.array._ndarray) is not np.ndarray for column in 
columns):
+        return None
     # Native row conversion reads Rust fields, bypassing mutable Python 
accessors.
     for name in ("DoubleType", "StructField", "StructType"):
         cls = getattr(native, name)
-        if (getattr(sql_types, name, None) is not cls
+        if (vars(sql_types).get(name) is not cls
                 or cls.__getattribute__ is not object.__getattribute__
                 or any("__class__" in vars(base) for base in cls.__mro__ if 
base is not object)):
             return None
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index ce0ec28..286613c 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -3321,6 +3321,57 @@ def 
test_float64_literal_numpy_dtype_custom_backing_keeps_row_error(protocol):
 
 
 
[email protected]("raises", [False, True])
+def test_float64_ndarray_subclass_converts_as_pyspark(protocol, raises):
+    # The fast path declines a NumPy subclass behind pandas' concrete wrapper; 
the frame then
+    # converts as PySpark's Arrow conversion does, which reads the raw storage 
and does not
+    # call the subclass's astype, whatever it would do.
+    from pyspark.sql._pandas_input import prepare_float64_table
+
+    calls = []
+
+    class Values(np.ndarray):
+        def astype(self, *args, **kwargs):
+            calls.append("astype")
+            if raises:
+                raise ValueError("custom ndarray astype")
+            return np.ndarray.astype(self, *args, **kwargs) + 7.0
+
+    values = np.arange(100000, dtype=np.float64)
+    # Avoid pandas consolidating the subclass storage into a plain ndarray.
+    frame = pd.DataFrame({"x": values.view(Values)}, copy=False)
+    schema = T.StructType([T.StructField("x", T.DoubleType())])
+    assert type(frame["x"].array._ndarray) is Values
+    assert prepare_float64_table(frame, schema) is None
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema).toArrow()
+        _assert_float64_ipc(state["input_tables"][0], [values], schema)
+        assert calls == []
+        assert state["uploads"] == 0
+
+
+def test_float64_missing_schema_alias_keeps_native_row_path(protocol, 
monkeypatch):
+    values = np.arange(100000, dtype=np.float64)
+    frame = pd.DataFrame({"x": values})
+    schema = T.StructType([T.StructField("x", T.DoubleType())])
+    calls = []
+
+    def lazy_attribute(name):
+        if name == "DoubleType":
+            calls.append(name)
+            raise RuntimeError("schema alias hook must not run before row 
conversion")
+        raise AttributeError(name)
+
+    monkeypatch.delitem(vars(T), "DoubleType")
+    monkeypatch.setattr(T, "__getattr__", lazy_attribute, raising=False)
+    with _arrow_input_service(protocol) as (spark, state):
+        spark.createDataFrame(frame, schema).toArrow()
+        _assert_float64_ipc(state["input_tables"][0], [values], schema)
+        assert calls == []
+        assert state["uploads"] == 0
+
+
+
 def 
test_float64_missing_concrete_numpy_backing_class_keeps_fallback(monkeypatch):
     import pandas.core.arrays.numpy_ as numpy_arrays
     from pyspark.sql._pandas_input import prepare_float64_table


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to