This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new c7c85f6 [SPARK-60716][CONNECT][PYTHON] Use validated backing arrays
for pandas dates
c7c85f6 is described below
commit c7c85f6e2f9d725355c510f934ea56f70bda758d
Author: DB Tsai <[email protected]>
AuthorDate: Sat Nov 7 19:02:34 2026 +0900
[SPARK-60716][CONNECT][PYTHON] Use validated backing arrays for pandas dates
### What changes were proposed in this pull request?
Use each explicit pandas date column's exact NumPy backing array for both
eligibility validation and Arrow conversion. Add regressions for customized
`Series.values` and `Series.to_numpy`, and document the existing date importer
path.
### Why are the changes needed?
Follow-up to #138 / SPARK-60716. For 10,000 stored dates of `2020-01-01`, a
customized `Series.values` returning `2021-02-03` made the fast path validate
the stored cells but emit the replacement dates. This client's previous
DataFrame row conversion retained `2020-01-01`. Validation and serialization
now consume the same concrete storage.
Canonical pandas column extraction through `iloc` and `Series.array`
remains trusted, consistent with the existing string path; this change does not
authenticate every mutable pandas descriptor.
### Does this PR introduce _any_ user-facing change?
Yes. Customized Series accessors no longer replace the validated date cells
or run during this optimization. Ordinary date values, null handling, names,
thresholds and transport routing are preserved.
### How was this patch tested?
- Both new accessor regressions fail against the landed helper; the updated
helper passes all 26 selected helper cases (865 deselected).
- Independent expected offsets and null masks pass for 30,000 cells
spanning Python's full date calendar range; the regression checks Arrow IPC
roundtrips and unchanged row normalization.
- Matching official PySpark 4.3.0 conversion passes 10,000 ordinary
full-range date/null values. The custom-accessor regression targets this
client's previous row-path contract.
- `git diff --check` passes. Genuine local Claude Code reviewed the full
landed change and the complete follow-up patch; the final source review
approved with no blockers and documented nonblocking limitations/nits.
**Draft validation limitation:** helper checks use installed native schema
types, not a fresh candidate extension. Cargo/Rust/protoc are unavailable on
this host, so the required Cargo lib/all-integration/wasm compile/fmt/Clippy
checks, fresh extension, native mock/live tests and full official parity/API
gates remain pending. No overall validation convergence is claimed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); reviewed with Claude Code 2.1.314.
This pull request and its description were written by Isaac.
Closes #278 from dbtsai/codex/postmerge-pr-138-date-backing.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
docs/architecture.md | 3 +++
python/pyspark/sql/_pandas_input.py | 18 +++++++++++++++---
python/tests/test_arrow_handoff.py | 32 ++++++++++++++++++++++++++++++++
3 files changed, 50 insertions(+), 3 deletions(-)
diff --git a/docs/architecture.md b/docs/architecture.md
index 0c7b7be..abbc4f7 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -159,6 +159,9 @@ Pandas frames of at least 100,000 rows whose columns are
all NumPy `float64`,
with an explicit `StructType` of `DoubleType` fields, are encoded to Arrow IPC
from their columns, with pandas' missing values as nulls, and sent inline as
the
row path sends them, without the importer's configuration RPC.
+Pandas object columns containing exact `datetime.date` values or `None` use the
+Arrow importer at 10,000 rows or more with an explicit `StructType` of
`DateType`
+fields. Unsupported values and customized inputs retain the row conversion
path.
Pandas frames of at least 10,000 rows whose columns are all kept in pandas'
Arrow
string storage (`string[pyarrow]`, and the `str` dtype of pandas 3), with an
explicit `StructType` of `StringType` fields, pass that storage to this
importer
diff --git a/python/pyspark/sql/_pandas_input.py
b/python/pyspark/sql/_pandas_input.py
index f4891c2..cf82827 100644
--- a/python/pyspark/sql/_pandas_input.py
+++ b/python/pyspark/sql/_pandas_input.py
@@ -796,19 +796,31 @@ def prepare_date_table(frame, schema):
except ImportError:
return None
columns = [frame.iloc[:, i] for i in range(width)]
+ arrays = []
for column, field in zip(columns, schema.fields):
- if type(column) is not Series or type(column.array) is not
NumpyExtensionArray:
+ if type(column) is not Series:
+ return None
+ backing = column.array
+ if type(backing) is not NumpyExtensionArray:
+ return None
+ array = backing._ndarray
+ if (type(array) is not np.ndarray or array.ndim != 1
+ or type(array.dtype) is not type(np.dtype("object"))
+ or array.dtype != np.dtype("object") or len(array) !=
len(frame)):
return None
# Arrow rejects nulls in non-nullable fields; let the row path retain
# its existing behavior for that input instead.
if not all(type(value) is date or (value is None and field.nullable)
- for value in column.to_numpy(copy=False)):
+ for value in array):
return None
+ arrays.append(array)
import pyarrow as pa
return pa.Table.from_arrays(
- [pa.array(column, type=pa.date32(), from_pandas=False) for column in
columns],
+ # Series accessors can be customized independently of the DataFrame row
+ # path. Emit the same concrete storage whose cells were checked above.
+ [pa.array(array, type=pa.date32(), from_pandas=False) for array in
arrays],
names=schema.names,
)
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index af993c4..ce0ec28 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -4763,6 +4763,38 @@ def
test_pandas_inferred_int_beyond_int64_raises_as_pyspark(protocol, schema):
# Exact built-in date columns reuse Arrow without materializing pandas rows.
[email protected]("accessor", ["values", "to_numpy"])
+def test_date_series_accessors_do_not_replace_validated_cells(monkeypatch,
accessor):
+ from pyspark.sql._pandas_input import prepare_date_table
+
+ dates = [date.min, date.max, date(1969, 12, 31), date(2000, 2, 29)] * 2500
+ nullable = [None if i % 7 == 0 else value for i, value in enumerate(dates)]
+ expected = {"renamed": dates, "nullable": nullable}
+ frame = pd.DataFrame({"original": dates, "nullable": nullable},
dtype=object)
+ schema = T.StructType([
+ T.StructField("renamed", T.DateType(), False),
+ T.StructField("nullable", T.DateType()),
+ ])
+ row_values = frame.astype(object).where(frame.notna(),
None).values.tolist()
+ calls = []
+
+ def replacement(series, *args, **kwargs):
+ calls.append(accessor)
+ return np.full(len(series), date(2021, 2, 3), dtype=object)
+
+ monkeypatch.setattr(pd.Series, accessor,
+ property(replacement) if accessor == "values" else
replacement)
+ table = prepare_date_table(frame, schema)
+ assert table is not None
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, table.schema) as writer:
+ writer.write_table(table)
+ actual = pa.ipc.open_stream(sink.getvalue()).read_all()
+ assert actual.to_pydict() == expected
+ assert frame.astype(object).where(frame.notna(), None).values.tolist() ==
row_values
+ assert calls == []
+
+
@pytest.mark.parametrize("cached", [False, True])
@pytest.mark.parametrize("verify", [False, True])
def test_date_columns_preserve_every_ipc_cell(protocol, monkeypatch, cached,
verify):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]