This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 9f057df [SPARK-60727][CONNECT] Preserve customized pandas double
conversion
9f057df is described below
commit 9f057df23f823d617c49441cebfceef0bd018590
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 19:03:03 2026 +0900
[SPARK-60727][CONNECT] Preserve customized pandas double conversion
### What changes were proposed in this pull request?
Preserve the existing row conversion of customized pandas float64 input in
the fast path added by #207
([SPARK-60727](https://issues.apache.org/jira/browse/SPARK-60727)).
- A DataFrame constructed with `copy=False` can retain an ndarray subclass
inside pandas' concrete NumPy wrapper. The column path skips that subclass's
`astype` hook, which can change values or suppress an existing conversion
error. Require exact ndarray storage before using this path.
- Inspect public schema aliases through their namespace dictionary. A
missing alias should decline the optimization without invoking a module's
custom `__getattr__` before row conversion.
Add three mock-service regression cases for transformed values, conversion
errors, and a missing alias with a raising lazy hook. Ordinary ndarray input
keeps the fast path; subclasses such as `np.memmap` take the row path.
### Why are the changes needed?
These inputs are customized cases that #207 intended to keep on the
existing row path. Both guard issues remain on current master. Official Spark
4.3 also bypasses the ndarray hook; this change preserves the Rust client's
prior custom-input behavior rather than claiming an official parity fix.
### Does this PR introduce any user-facing change?
It restores the prior values and errors for the affected customized input.
Public signatures and inline transport policy are unchanged.
### How was this patch tested?
- Reproduced both issues with pandas 2.3.3 and 3.0.6, NumPy 2.5.4 and
PyArrow 26.0.0.
- Supplemental helper checks pass on both pandas versions, including
ordinary ndarray float bits and null masks. These use private copies of native
4.2 schema classes and do not prove current-master native dispatch.
- Python syntax and `git diff --check` pass.
- Genuine local Claude Code reviewed the landed patch and follow-up; the
final full review and all findings are recorded in the local review evidence.
**Draft: required validation remains pending.** The current environment has
no usable Cargo/rustc toolchain, matching native extension, or protoc. The new
public-API test attempt is blocked during collection; Rust compile/fmt/Clippy,
complete native suites, API/official parity, and live Spark 4.3 checks have not
passed for this candidate. Historical maintainer results are not claimed as
validation of this follow-up.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex; reviewed with installed local Claude Code.
This pull request and its description were written by Isaac.
Closes #280 from dbtsai/codex/postmerge-pr-207-ndarray-storage.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/_pandas_input.py | 7 ++++-
python/tests/test_arrow_handoff.py | 51 +++++++++++++++++++++++++++++++++++++
2 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/_pandas_input.py
b/python/pyspark/sql/_pandas_input.py
index cf82827..e2aa9e2 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -474,10 +474,15 @@ def prepare_float64_table(frame, schema):
if any(type(column) is not Series or type(column.array) is not
NumpyExtensionArray
for column in columns):
return None
+ # NumPy subclasses can customize the row path's astype conversion even
+ # inside pandas' concrete NumPy wrapper. Keep their hooks, accepting the
+ # row path's cost for subclasses such as np.memmap.
+ if any(type(column.array._ndarray) is not np.ndarray for column in
columns):
+ return None
# Native row conversion reads Rust fields, bypassing mutable Python
accessors.
for name in ("DoubleType", "StructField", "StructType"):
cls = getattr(native, name)
- if (getattr(sql_types, name, None) is not cls
+ if (vars(sql_types).get(name) is not cls
or cls.__getattribute__ is not object.__getattribute__
or any("__class__" in vars(base) for base in cls.__mro__ if
base is not object)):
return None
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index ce0ec28..286613c 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -3321,6 +3321,57 @@ def
test_float64_literal_numpy_dtype_custom_backing_keeps_row_error(protocol):
[email protected]("raises", [False, True])
+def test_float64_ndarray_subclass_converts_as_pyspark(protocol, raises):
+ # The fast path declines a NumPy subclass behind pandas' concrete wrapper;
the frame then
+ # converts as PySpark's Arrow conversion does, which reads the raw storage
and does not
+ # call the subclass's astype, whatever it would do.
+ from pyspark.sql._pandas_input import prepare_float64_table
+
+ calls = []
+
+ class Values(np.ndarray):
+ def astype(self, *args, **kwargs):
+ calls.append("astype")
+ if raises:
+ raise ValueError("custom ndarray astype")
+ return np.ndarray.astype(self, *args, **kwargs) + 7.0
+
+ values = np.arange(100000, dtype=np.float64)
+ # Avoid pandas consolidating the subclass storage into a plain ndarray.
+ frame = pd.DataFrame({"x": values.view(Values)}, copy=False)
+ schema = T.StructType([T.StructField("x", T.DoubleType())])
+ assert type(frame["x"].array._ndarray) is Values
+ assert prepare_float64_table(frame, schema) is None
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ _assert_float64_ipc(state["input_tables"][0], [values], schema)
+ assert calls == []
+ assert state["uploads"] == 0
+
+
+def test_float64_missing_schema_alias_keeps_native_row_path(protocol,
monkeypatch):
+ values = np.arange(100000, dtype=np.float64)
+ frame = pd.DataFrame({"x": values})
+ schema = T.StructType([T.StructField("x", T.DoubleType())])
+ calls = []
+
+ def lazy_attribute(name):
+ if name == "DoubleType":
+ calls.append(name)
+ raise RuntimeError("schema alias hook must not run before row
conversion")
+ raise AttributeError(name)
+
+ monkeypatch.delitem(vars(T), "DoubleType")
+ monkeypatch.setattr(T, "__getattr__", lazy_attribute, raising=False)
+ with _arrow_input_service(protocol) as (spark, state):
+ spark.createDataFrame(frame, schema).toArrow()
+ _assert_float64_ipc(state["input_tables"][0], [values], schema)
+ assert calls == []
+ assert state["uploads"] == 0
+
+
+
def
test_float64_missing_concrete_numpy_backing_class_keeps_fallback(monkeypatch):
import pandas.core.arrays.numpy_ as numpy_arrays
from pyspark.sql._pandas_input import prepare_float64_table
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]