This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 441e45b [SPARK-60685][CONNECT] Fast-path exact builtin date inputs
441e45b is described below
commit 441e45bbc597e3a4252f8871dac5447475124e85
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 20:44:04 2026 +0900
[SPARK-60685][CONNECT] Fast-path exact builtin date inputs
### What changes were proposed in this pull request?
`py_to_value`, the generic Python-to-`Value` conversion, now converts an
exact built-in `datetime.date` directly (`cast_exact::<PyDate>()`, with the
existing `toordinal() - 719163` date32 encoding), before its generic checks:
the `bool` extraction, `Decimal` detection and numeric extraction attempts that
a date never passes, and the lookups of the `datetime` module. Subclasses of
`date` keep the generic path, so their numeric protocols and `toordinal`
overrides take effect as before; [...]
This follows #111 and #129, which did the same for exact numbers, lists and
tuples. `py_to_value` converts the cells of pandas input
(`createDataFrame(pandas_df)` turns every column into Python objects first),
and the values of `fillna`, `pivot` and `Row` conversion; local rows go through
`infer.convert`.
### Why are the changes needed?
[SPARK-60685](https://issues.apache.org/jira/browse/SPARK-60685): each date
went through failing extractions, each building and dropping a Python error,
before it was recognized. Measured on macOS against a local Spark 4.3.0 server,
as the client CPU time of `createDataFrame` of a pandas DataFrame of 200,000
rows of two date columns (median of 5, two rounds): 912-1,155 ms on the current
master, 187-267 ms with this change, with an inferred or an explicit `DateType`
schema.
An earlier review of this change held it because it dispatches differently
when user code rebinds `datetime.date` or removes the `datetime` module, as the
fast path uses the C type of `date` rather than the module attribute; such
rebinding is not supported.
### Does this PR introduce _any_ user-facing change?
No, other than speed. The converted values are the same.
### How was this patch tested?
- New offline tests check the converted dates: the range bounds (years 1
and 9999), epoch neighbors, leap and century dates, subclasses with numeric
protocols and `toordinal` overrides, `datetime` and pandas `Timestamp`, nested
values, nulls and unsupported objects. Separate-process mock servers decode the
Arrow IPC of pandas and tuple input with inferred, named, explicit and DDL
schemas.
- Against the current master, merged into this branch, and a local Spark
4.3.0 server: the lib and every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (789 passed), the build of all tests
with `wasm-udf`, the offline Python suites (1,180 passed, 9 skipped), the live
drop-in and Arrow suites (83 passed) and the live cases (10 passed). `cargo fmt
--all --check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master and the
measurement by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #128 from dbtsai/codex/spark-60685-exact-date.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 8 ++
python/tests/test_dropin_offline.py | 249 ++++++++++++++++++++++++++++++++++++
2 files changed, 257 insertions(+)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 0533018..b2d72d8 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1618,6 +1618,14 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Value> {
return Ok(Value::List(items));
}
+ // Exact dates cannot implement numeric protocols or override toordinal.
Skip
+ // the generic Decimal/numeric probes; subclasses (including datetime) must
+ // retain the fallback ordering below. PyDate also supports abi3 wheels.
+ if let Ok(date) = obj.cast_exact::<pyo3::types::PyDate>() {
+ let ordinal: i64 = date.call_method0("toordinal")?.extract()?;
+ return Ok(Value::Date((ordinal - 719163i64) as i32));
+ }
+
// Check for bool (before int because bool is a subclass of int in Python)
if let Ok(b) = obj.extract::<bool>() {
return Ok(Value::Bool(b));
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 565bd39..4228baa 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4691,6 +4691,255 @@ def
test_container_input_nested_array_struct_binary_ipc(container_input_capture_
"record": {"x": 7, "y": "row"}, "blob":
b"abc"}]
+# ---------------------------------------------------------------------------
exact date input
+
+from datetime import (
+ date as _exact_date_date, datetime as _exact_date_datetime,
+ timezone as _exact_date_timezone,
+)
+from decimal import Decimal as _exact_date_Decimal
+from pyspark.sql import Row as _exact_date_Row
+
+
+def _exact_date_converted_value(mock_connect, value):
+ import pyarrow as pa
+
+ pd = pytest.importorskip("pandas")
+ spark, service = mock_connect
+ # py_to_value remains the generic pandas converter. Local rows now use
+ # infer.convert, which intentionally has different subclass/type semantics.
+ data = pd.DataFrame({"value": [value]}, dtype=object)
+ before = len(service.execute_requests)
+ spark.createDataFrame(data).toArrow()
+ requests = service.execute_requests
+ assert len(requests) == before + 1
+ request = parse_message(requests[-1])
+ relation = parse_message(parse_message(request[3][0])[1][0])
+ renames = []
+ while 18 in relation: # Relation.to_df -> ToDF.input; column_names=2.
+ rename = parse_message(relation[18][0])
+ renames.append([name.decode() for name in rename.get(2, [])])
+ relation = parse_message(rename[1][0])
+ local = parse_message(relation[11][0])
+ table = pa.ipc.open_stream(local[1][0]).read_all()
+ table.validate(full=True)
+ assert (table.num_rows, table.num_columns) == (1, 1)
+ assert table.column_names == ["value"]
+ assert [field["name"] for field in json.loads(local[2][0])["fields"]] ==
table.column_names
+ for names in renames:
+ assert len(names) == table.num_columns
+ # The outermost toDF supplies observable names without rewriting raw IPC.
+ assert (renames[0] if renames else table.column_names) == ["value"]
+ return table.column(0)[0].as_py()
+
+
[email protected]("value", [
+ _exact_date_date.min, _exact_date_date.max, _exact_date_date(1, 3, 1),
_exact_date_date(4, 2, 29),
+ _exact_date_date(1600, 2, 29), _exact_date_date(1900, 2, 28),
_exact_date_date(1900, 3, 1),
+ _exact_date_date(1969, 12, 31), _exact_date_date(1970, 1, 1),
_exact_date_date(1970, 1, 2),
+ _exact_date_date(2000, 2, 29), _exact_date_date(2100, 3, 1),
+])
+def test_exact_date_builtin_roundtrip(mock_connect, value):
+ actual = _exact_date_converted_value(mock_connect, value)
+ assert type(actual) is _exact_date_date
+ assert actual == value
+
+
+def test_exact_date_subclass_protocol_precedence(mock_connect):
+ trace = []
+
+ class Indexed(_exact_date_date):
+ def __index__(self):
+ trace.append("index")
+ return 42
+
+ def toordinal(self):
+ raise AssertionError("numeric extraction must precede date
conversion")
+
+ class Floating(_exact_date_date):
+ def __float__(self):
+ trace.append("float")
+ return 1.25
+
+ def toordinal(self):
+ raise AssertionError("numeric extraction must precede date
conversion")
+
+ class Ordinal(_exact_date_date):
+ def toordinal(self):
+ trace.append("ordinal")
+ return _exact_date_date(2000, 2, 29).toordinal()
+
+ class Plain(_exact_date_date):
+ pass
+
+ assert _exact_date_converted_value(mock_connect, Indexed(2020, 1, 1)) == 42
+ assert _exact_date_converted_value(mock_connect, Floating(2020, 1, 1)) ==
1.25
+ assert _exact_date_converted_value(mock_connect, Ordinal(2020, 1, 1)) ==
_exact_date_date(2000, 2, 29)
+ assert type(_exact_date_converted_value(mock_connect, Plain(2020, 1, 1)))
is _exact_date_date
+ assert trace == ["index", "float", "ordinal"]
+
+
+def test_exact_date_datetime_timestamp_encoding(mock_connect):
+ pd = pytest.importorskip("pandas")
+ for value in [_exact_date_datetime(2020, 2, 29, 12, 34, 56, 123456),
+ _exact_date_datetime(2020, 2, 29,
tzinfo=_exact_date_timezone.utc),
+ pd.Timestamp("2020-02-29 12:34:56.123456"),
+ pd.Timestamp("2020-02-29 12:34:56.123456Z")]:
+ actual = _exact_date_converted_value(mock_connect, value)
+ assert type(actual) is _exact_date_datetime
+ expected = value.to_pydatetime() if isinstance(value, pd.Timestamp)
else value
+ assert actual ==
_exact_date_datetime.fromtimestamp(expected.timestamp(),
_exact_date_timezone.utc)
+
+
+def test_exact_date_nested_unrelated_values(mock_connect):
+ value = _exact_date_Row(items=[_exact_date_date.min, None,
_exact_date_date.max],
+ mapping={"epoch": _exact_date_date(1970, 1, 1)},
+ nested=_exact_date_Row(day=_exact_date_date(2000, 2, 29)),
+ amount=_exact_date_Decimal("-12.340"), number=42, flag=True,
label="date")
+ actual = _exact_date_converted_value(mock_connect, value)
+ assert actual["items"] == [_exact_date_date.min, None,
_exact_date_date.max]
+ assert dict(actual["mapping"]) == {"epoch": _exact_date_date(1970, 1, 1)}
+ assert actual["nested"]["day"] == _exact_date_date(2000, 2, 29)
+ assert actual["amount"] == _exact_date_Decimal("-12.340")
+ assert (actual["number"], actual["flag"], actual["label"]) == (42, True,
"date")
+
+
+def test_exact_date_named_impostor_rejected():
+ pd = pytest.importorskip("pandas")
+ impostor = type("date", (), {"toordinal": lambda self: 719163})
+ with pytest.raises(TypeError, match="unsupported value type"):
+ _local_create_dataframe(pd.DataFrame({"value": [impostor()]},
dtype=object))
+
+
+def _exact_date_fields(data):
+ """Decode only the small protobuf envelope around a local Arrow
relation."""
+ def _exact_date_varint(pos):
+ value, shift = 0, 0
+ while True:
+ byte = data[pos]
+ pos += 1
+ value |= (byte & 127) << shift
+ if byte < 128:
+ return value, pos
+ shift += 7
+
+ result, pos = {}, 0
+ while pos < len(data):
+ tag, pos = _exact_date_varint(pos)
+ wire = tag & 7
+ if wire == 2:
+ length, pos = _exact_date_varint(pos)
+ result[tag >> 3] = data[pos:pos + length]
+ pos += length
+ elif wire == 0:
+ _, pos = _exact_date_varint(pos)
+ elif wire in (1, 5):
+ pos += 8 if wire == 1 else 4
+ else:
+ raise AssertionError(f"unexpected protobuf wire type: {wire}")
+ return result
+
+
+def _exact_date_capture_server(connection):
+ from concurrent.futures import ThreadPoolExecutor
+ import grpc
+
+ captured = []
+
+ def _exact_date_execute(request, context):
+ # ExecutePlanRequest.plan=3 -> Plan.root=1 ->
+ # Relation.local_relation=11 -> LocalRelation.data=1.
+ relation =
_exact_date_fields(_exact_date_fields(_exact_date_fields(request)[3])[1])
+ while 18 in relation: # Relation.to_df -> ToDF.input.
+ relation = _exact_date_fields(_exact_date_fields(relation[18])[1])
+ captured.append(_exact_date_fields(relation[11])[1])
+ context.abort(grpc.StatusCode.INVALID_ARGUMENT, "captured date input")
+ yield b"" # Make this a streaming handler.
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ server = grpc.server(executor)
+ server.add_generic_rpc_handlers((grpc.method_handlers_generic_handler(
+ "spark.connect.SparkConnectService", {
+ "ExecutePlan":
grpc.unary_stream_rpc_method_handler(_exact_date_execute),
+ }),))
+ port = server.add_insecure_port("127.0.0.1:0")
+ server.start()
+ connection.send(port)
+ try:
+ while True:
+ command = connection.recv()
+ if command == "stop":
+ break
+ assert command == "get"
+ connection.send(captured.pop(0))
+ finally:
+ server.stop(0).wait()
+
+
[email protected]
+def exact_date_capture_input():
+ import multiprocessing
+ pytest.importorskip("grpc")
+ from pyspark.sql import SparkSession
+
+ context = multiprocessing.get_context("spawn")
+ parent, child = context.Pipe()
+ process = context.Process(target=_exact_date_capture_server, args=(child,))
+ process.start()
+ try:
+ assert parent.poll(20), "capture service failed to start"
+ spark =
SparkSession.builder.remote(f"sc://127.0.0.1:{parent.recv()}").getOrCreate()
+
+ def _exact_date_capture(data, schema):
+ import pyarrow as pa
+ df = spark.createDataFrame(data, schema=schema)
+ with pytest.raises(Exception, match="captured date input"):
+ df.collect()
+ parent.send("get")
+ assert parent.poll(10), "capture service failed to return the IPC
payload"
+ return pa.ipc.open_stream(parent.recv()).read_all()
+
+ yield _exact_date_capture
+ finally:
+ if process.is_alive():
+ parent.send("stop")
+ process.join(10)
+ if process.is_alive():
+ process.terminate()
+ process.join(10)
+ parent.close()
+ child.close()
+
+
[email protected]("source", ["tuples", "pandas"])
[email protected]("schema_kind", ["inferred", "names", "explicit",
"ddl"])
+def test_exact_date_ipc_calendar_range(exact_date_capture_input, source,
schema_kind):
+ import calendar
+ import pyarrow as pa
+ from pyspark.sql import types as T
+
+ # Every year's endpoints and leap day cover the complete supported calendar
+ # range, including non-leap centuries, without relying on the Rust decoder.
+ dates = [_exact_date_date.min, _exact_date_date.max,
_exact_date_date(1969, 12, 31), _exact_date_date(1970, 1, 1),
+ _exact_date_date(1970, 1, 2), None]
+ for year in range(1, 10000):
+ dates.extend([_exact_date_date(year, 1, 1), _exact_date_date(year, 12,
31)])
+ if calendar.isleap(year):
+ dates.append(_exact_date_date(year, 2, 29))
+ schema = {"inferred": None, "names": ["day"], "ddl": "day date",
+ "explicit": T.StructType([T.StructField("day",
T.DateType())])}[schema_kind]
+ data = [(value,) for value in dates]
+ if source == "pandas":
+ pd = pytest.importorskip("pandas")
+ data = pd.DataFrame(data, columns=["day"])
+ table = exact_date_capture_input(data, schema)
+ assert table.column(0).type == pa.date32()
+ assert table.column(0).to_pylist() == dates
+ epoch = _exact_date_date(1970, 1, 1)
+ expected_days = [None if value is None else (value - epoch).days for value
in dates]
+ assert table.column(0).cast(pa.int32()).to_pylist() == expected_days
+
+
# ---------------------------------------------------------------------------
schema cache
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]