This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new c7c85f6  [SPARK-60716][CONNECT][PYTHON] Use validated backing arrays 
for pandas dates
c7c85f6 is described below

commit c7c85f6e2f9d725355c510f934ea56f70bda758d
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 19:02:34 2026 +0900

    [SPARK-60716][CONNECT][PYTHON] Use validated backing arrays for pandas dates
    
    ### What changes were proposed in this pull request?
    
    Use each explicit pandas date column's exact NumPy backing array for both 
eligibility validation and Arrow conversion. Add regressions for customized 
`Series.values` and `Series.to_numpy`, and document the existing date importer 
path.
    
    ### Why are the changes needed?
    
    Follow-up to #138 / SPARK-60716. For 10,000 stored dates of `2020-01-01`, a 
customized `Series.values` returning `2021-02-03` made the fast path validate 
the stored cells but emit the replacement dates. This client's previous 
DataFrame row conversion retained `2020-01-01`. Validation and serialization 
now consume the same concrete storage.
    
    Canonical pandas column extraction through `iloc` and `Series.array` 
remains trusted, consistent with the existing string path; this change does not 
authenticate every mutable pandas descriptor.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Customized Series accessors no longer replace the validated date cells 
or run during this optimization. Ordinary date values, null handling, names, 
thresholds and transport routing are preserved.
    
    ### How was this patch tested?
    
    - Both new accessor regressions fail against the landed helper; the updated 
helper passes all 26 selected helper cases (865 deselected).
    - Independent expected offsets and null masks pass for 30,000 cells 
spanning Python's full date calendar range; the regression checks Arrow IPC 
roundtrips and unchanged row normalization.
    - Matching official PySpark 4.3.0 conversion passes 10,000 ordinary 
full-range date/null values. The custom-accessor regression targets this 
client's previous row-path contract.
    - `git diff --check` passes. Genuine local Claude Code reviewed the full 
landed change and the complete follow-up patch; the final source review 
approved with no blockers and documented nonblocking limitations/nits.
    
    **Draft validation limitation:** helper checks use installed native schema 
types, not a fresh candidate extension. Cargo/Rust/protoc are unavailable on 
this host, so the required Cargo lib/all-integration/wasm compile/fmt/Clippy 
checks, fresh extension, native mock/live tests and full official parity/API 
gates remain pending. No overall validation convergence is claimed.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); reviewed with Claude Code 2.1.314.
    
    This pull request and its description were written by Isaac.
    
    Closes #278 from dbtsai/codex/postmerge-pr-138-date-backing.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 docs/architecture.md                |  3 +++
 python/pyspark/sql/_pandas_input.py | 18 +++++++++++++++---
 python/tests/test_arrow_handoff.py  | 32 ++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 3 deletions(-)

diff --git a/docs/architecture.md b/docs/architecture.md
index 0c7b7be..abbc4f7 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -159,6 +159,9 @@ Pandas frames of at least 100,000 rows whose columns are 
all NumPy `float64`,
 with an explicit `StructType` of `DoubleType` fields, are encoded to Arrow IPC
 from their columns, with pandas' missing values as nulls, and sent inline as 
the
 row path sends them, without the importer's configuration RPC.
+Pandas object columns containing exact `datetime.date` values or `None` use the
+Arrow importer at 10,000 rows or more with an explicit `StructType` of 
`DateType`
+fields. Unsupported values and customized inputs retain the row conversion 
path.
 Pandas frames of at least 10,000 rows whose columns are all kept in pandas' 
Arrow
 string storage (`string[pyarrow]`, and the `str` dtype of pandas 3), with an
 explicit `StructType` of `StringType` fields, pass that storage to this 
importer
diff --git a/python/pyspark/sql/_pandas_input.py 
b/python/pyspark/sql/_pandas_input.py
index f4891c2..cf82827 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -796,19 +796,31 @@ def prepare_date_table(frame, schema):
         except ImportError:
             return None
     columns = [frame.iloc[:, i] for i in range(width)]
+    arrays = []
     for column, field in zip(columns, schema.fields):
-        if type(column) is not Series or type(column.array) is not 
NumpyExtensionArray:
+        if type(column) is not Series:
+            return None
+        backing = column.array
+        if type(backing) is not NumpyExtensionArray:
+            return None
+        array = backing._ndarray
+        if (type(array) is not np.ndarray or array.ndim != 1
+                or type(array.dtype) is not type(np.dtype("object"))
+                or array.dtype != np.dtype("object") or len(array) != 
len(frame)):
             return None
         # Arrow rejects nulls in non-nullable fields; let the row path retain
         # its existing behavior for that input instead.
         if not all(type(value) is date or (value is None and field.nullable)
-                   for value in column.to_numpy(copy=False)):
+                   for value in array):
             return None
+        arrays.append(array)
 
     import pyarrow as pa
 
     return pa.Table.from_arrays(
-        [pa.array(column, type=pa.date32(), from_pandas=False) for column in 
columns],
+        # Series accessors can be customized independently of the DataFrame row
+        # path. Emit the same concrete storage whose cells were checked above.
+        [pa.array(array, type=pa.date32(), from_pandas=False) for array in 
arrays],
         names=schema.names,
     )
 
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index af993c4..ce0ec28 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -4763,6 +4763,38 @@ def 
test_pandas_inferred_int_beyond_int64_raises_as_pyspark(protocol, schema):
 
 # Exact built-in date columns reuse Arrow without materializing pandas rows.
 
[email protected]("accessor", ["values", "to_numpy"])
+def test_date_series_accessors_do_not_replace_validated_cells(monkeypatch, 
accessor):
+    from pyspark.sql._pandas_input import prepare_date_table
+
+    dates = [date.min, date.max, date(1969, 12, 31), date(2000, 2, 29)] * 2500
+    nullable = [None if i % 7 == 0 else value for i, value in enumerate(dates)]
+    expected = {"renamed": dates, "nullable": nullable}
+    frame = pd.DataFrame({"original": dates, "nullable": nullable}, 
dtype=object)
+    schema = T.StructType([
+        T.StructField("renamed", T.DateType(), False),
+        T.StructField("nullable", T.DateType()),
+    ])
+    row_values = frame.astype(object).where(frame.notna(), 
None).values.tolist()
+    calls = []
+
+    def replacement(series, *args, **kwargs):
+        calls.append(accessor)
+        return np.full(len(series), date(2021, 2, 3), dtype=object)
+
+    monkeypatch.setattr(pd.Series, accessor,
+                        property(replacement) if accessor == "values" else 
replacement)
+    table = prepare_date_table(frame, schema)
+    assert table is not None
+    sink = pa.BufferOutputStream()
+    with pa.ipc.new_stream(sink, table.schema) as writer:
+        writer.write_table(table)
+    actual = pa.ipc.open_stream(sink.getvalue()).read_all()
+    assert actual.to_pydict() == expected
+    assert frame.astype(object).where(frame.notna(), None).values.tolist() == 
row_values
+    assert calls == []
+
+
 @pytest.mark.parametrize("cached", [False, True])
 @pytest.mark.parametrize("verify", [False, True])
 def test_date_columns_preserve_every_ipc_cell(protocol, monkeypatch, cached, 
verify):


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to