This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 441e45b  [SPARK-60685][CONNECT] Fast-path exact builtin date inputs
441e45b is described below

commit 441e45bbc597e3a4252f8871dac5447475124e85
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 20:44:04 2026 +0900

    [SPARK-60685][CONNECT] Fast-path exact builtin date inputs
    
    ### What changes were proposed in this pull request?
    
    `py_to_value`, the generic Python-to-`Value` conversion, now converts an 
exact built-in `datetime.date` directly (`cast_exact::<PyDate>()`, with the 
existing `toordinal() - 719163` date32 encoding), before its generic checks: 
the `bool` extraction, `Decimal` detection and numeric extraction attempts that 
a date never passes, and the lookups of the `datetime` module. Subclasses of 
`date` keep the generic path, so their numeric protocols and `toordinal` 
overrides take effect as before;  [...]
    
    This follows #111 and #129, which did the same for exact numbers, lists and 
tuples. `py_to_value` converts the cells of pandas input 
(`createDataFrame(pandas_df)` turns every column into Python objects first), 
and the values of `fillna`, `pivot` and `Row` conversion; local rows go through 
`infer.convert`.
    
    ### Why are the changes needed?
    
    [SPARK-60685](https://issues.apache.org/jira/browse/SPARK-60685): each date 
went through failing extractions, each building and dropping a Python error, 
before it was recognized. Measured on macOS against a local Spark 4.3.0 server, 
as the client CPU time of `createDataFrame` of a pandas DataFrame of 200,000 
rows of two date columns (median of 5, two rounds): 912-1,155 ms on the current 
master, 187-267 ms with this change, with an inferred or an explicit `DateType` 
schema.
    
    An earlier review of this change held it because it dispatches differently 
when user code rebinds `datetime.date` or removes the `datetime` module, as the 
fast path uses the C type of `date` rather than the module attribute; such 
rebinding is not supported.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, other than speed. The converted values are the same.
    
    ### How was this patch tested?
    
    - New offline tests check the converted dates: the range bounds (years 1 
and 9999), epoch neighbors, leap and century dates, subclasses with numeric 
protocols and `toordinal` overrides, `datetime` and pandas `Timestamp`, nested 
values, nulls and unsupported objects. Separate-process mock servers decode the 
Arrow IPC of pandas and tuple input with inferred, named, explicit and DDL 
schemas.
    - Against the current master, merged into this branch, and a local Spark 
4.3.0 server: the lib and every integration test of `apache-spark-connect` and 
`apache-spark-connect-core`, e2e included (789 passed), the build of all tests 
with `wasm-udf`, the offline Python suites (1,180 passed, 9 skipped), the live 
drop-in and Arrow suites (83 passed) and the live cases (10 passed). `cargo fmt 
--all --check` passes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the merge with master and the 
measurement by Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #128 from dbtsai/codex/spark-60685-exact-date.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    |   8 ++
 python/tests/test_dropin_offline.py | 249 ++++++++++++++++++++++++++++++++++++
 2 files changed, 257 insertions(+)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 0533018..b2d72d8 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1618,6 +1618,14 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Value> {
         return Ok(Value::List(items));
     }
 
+    // Exact dates cannot implement numeric protocols or override toordinal. 
Skip
+    // the generic Decimal/numeric probes; subclasses (including datetime) must
+    // retain the fallback ordering below. PyDate also supports abi3 wheels.
+    if let Ok(date) = obj.cast_exact::<pyo3::types::PyDate>() {
+        let ordinal: i64 = date.call_method0("toordinal")?.extract()?;
+        return Ok(Value::Date((ordinal - 719163i64) as i32));
+    }
+
     // Check for bool (before int because bool is a subclass of int in Python)
     if let Ok(b) = obj.extract::<bool>() {
         return Ok(Value::Bool(b));
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 565bd39..4228baa 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4691,6 +4691,255 @@ def 
test_container_input_nested_array_struct_binary_ipc(container_input_capture_
                                   "record": {"x": 7, "y": "row"}, "blob": 
b"abc"}]
 
 
+# --------------------------------------------------------------------------- 
exact date input
+
+from datetime import (
+    date as _exact_date_date, datetime as _exact_date_datetime,
+    timezone as _exact_date_timezone,
+)
+from decimal import Decimal as _exact_date_Decimal
+from pyspark.sql import Row as _exact_date_Row
+
+
+def _exact_date_converted_value(mock_connect, value):
+    import pyarrow as pa
+
+    pd = pytest.importorskip("pandas")
+    spark, service = mock_connect
+    # py_to_value remains the generic pandas converter. Local rows now use
+    # infer.convert, which intentionally has different subclass/type semantics.
+    data = pd.DataFrame({"value": [value]}, dtype=object)
+    before = len(service.execute_requests)
+    spark.createDataFrame(data).toArrow()
+    requests = service.execute_requests
+    assert len(requests) == before + 1
+    request = parse_message(requests[-1])
+    relation = parse_message(parse_message(request[3][0])[1][0])
+    renames = []
+    while 18 in relation:  # Relation.to_df -> ToDF.input; column_names=2.
+        rename = parse_message(relation[18][0])
+        renames.append([name.decode() for name in rename.get(2, [])])
+        relation = parse_message(rename[1][0])
+    local = parse_message(relation[11][0])
+    table = pa.ipc.open_stream(local[1][0]).read_all()
+    table.validate(full=True)
+    assert (table.num_rows, table.num_columns) == (1, 1)
+    assert table.column_names == ["value"]
+    assert [field["name"] for field in json.loads(local[2][0])["fields"]] == 
table.column_names
+    for names in renames:
+        assert len(names) == table.num_columns
+    # The outermost toDF supplies observable names without rewriting raw IPC.
+    assert (renames[0] if renames else table.column_names) == ["value"]
+    return table.column(0)[0].as_py()
+
+
[email protected]("value", [
+    _exact_date_date.min, _exact_date_date.max, _exact_date_date(1, 3, 1), 
_exact_date_date(4, 2, 29),
+    _exact_date_date(1600, 2, 29), _exact_date_date(1900, 2, 28), 
_exact_date_date(1900, 3, 1),
+    _exact_date_date(1969, 12, 31), _exact_date_date(1970, 1, 1), 
_exact_date_date(1970, 1, 2),
+    _exact_date_date(2000, 2, 29), _exact_date_date(2100, 3, 1),
+])
+def test_exact_date_builtin_roundtrip(mock_connect, value):
+    actual = _exact_date_converted_value(mock_connect, value)
+    assert type(actual) is _exact_date_date
+    assert actual == value
+
+
+def test_exact_date_subclass_protocol_precedence(mock_connect):
+    trace = []
+
+    class Indexed(_exact_date_date):
+        def __index__(self):
+            trace.append("index")
+            return 42
+
+        def toordinal(self):
+            raise AssertionError("numeric extraction must precede date 
conversion")
+
+    class Floating(_exact_date_date):
+        def __float__(self):
+            trace.append("float")
+            return 1.25
+
+        def toordinal(self):
+            raise AssertionError("numeric extraction must precede date 
conversion")
+
+    class Ordinal(_exact_date_date):
+        def toordinal(self):
+            trace.append("ordinal")
+            return _exact_date_date(2000, 2, 29).toordinal()
+
+    class Plain(_exact_date_date):
+        pass
+
+    assert _exact_date_converted_value(mock_connect, Indexed(2020, 1, 1)) == 42
+    assert _exact_date_converted_value(mock_connect, Floating(2020, 1, 1)) == 
1.25
+    assert _exact_date_converted_value(mock_connect, Ordinal(2020, 1, 1)) == 
_exact_date_date(2000, 2, 29)
+    assert type(_exact_date_converted_value(mock_connect, Plain(2020, 1, 1))) 
is _exact_date_date
+    assert trace == ["index", "float", "ordinal"]
+
+
+def test_exact_date_datetime_timestamp_encoding(mock_connect):
+    pd = pytest.importorskip("pandas")
+    for value in [_exact_date_datetime(2020, 2, 29, 12, 34, 56, 123456),
+                  _exact_date_datetime(2020, 2, 29, 
tzinfo=_exact_date_timezone.utc),
+                  pd.Timestamp("2020-02-29 12:34:56.123456"),
+                  pd.Timestamp("2020-02-29 12:34:56.123456Z")]:
+        actual = _exact_date_converted_value(mock_connect, value)
+        assert type(actual) is _exact_date_datetime
+        expected = value.to_pydatetime() if isinstance(value, pd.Timestamp) 
else value
+        assert actual == 
_exact_date_datetime.fromtimestamp(expected.timestamp(), 
_exact_date_timezone.utc)
+
+
+def test_exact_date_nested_unrelated_values(mock_connect):
+    value = _exact_date_Row(items=[_exact_date_date.min, None, 
_exact_date_date.max],
+                mapping={"epoch": _exact_date_date(1970, 1, 1)},
+                nested=_exact_date_Row(day=_exact_date_date(2000, 2, 29)),
+                amount=_exact_date_Decimal("-12.340"), number=42, flag=True, 
label="date")
+    actual = _exact_date_converted_value(mock_connect, value)
+    assert actual["items"] == [_exact_date_date.min, None, 
_exact_date_date.max]
+    assert dict(actual["mapping"]) == {"epoch": _exact_date_date(1970, 1, 1)}
+    assert actual["nested"]["day"] == _exact_date_date(2000, 2, 29)
+    assert actual["amount"] == _exact_date_Decimal("-12.340")
+    assert (actual["number"], actual["flag"], actual["label"]) == (42, True, 
"date")
+
+
+def test_exact_date_named_impostor_rejected():
+    pd = pytest.importorskip("pandas")
+    impostor = type("date", (), {"toordinal": lambda self: 719163})
+    with pytest.raises(TypeError, match="unsupported value type"):
+        _local_create_dataframe(pd.DataFrame({"value": [impostor()]}, 
dtype=object))
+
+
+def _exact_date_fields(data):
+    """Decode only the small protobuf envelope around a local Arrow 
relation."""
+    def _exact_date_varint(pos):
+        value, shift = 0, 0
+        while True:
+            byte = data[pos]
+            pos += 1
+            value |= (byte & 127) << shift
+            if byte < 128:
+                return value, pos
+            shift += 7
+
+    result, pos = {}, 0
+    while pos < len(data):
+        tag, pos = _exact_date_varint(pos)
+        wire = tag & 7
+        if wire == 2:
+            length, pos = _exact_date_varint(pos)
+            result[tag >> 3] = data[pos:pos + length]
+            pos += length
+        elif wire == 0:
+            _, pos = _exact_date_varint(pos)
+        elif wire in (1, 5):
+            pos += 8 if wire == 1 else 4
+        else:
+            raise AssertionError(f"unexpected protobuf wire type: {wire}")
+    return result
+
+
+def _exact_date_capture_server(connection):
+    from concurrent.futures import ThreadPoolExecutor
+    import grpc
+
+    captured = []
+
+    def _exact_date_execute(request, context):
+        # ExecutePlanRequest.plan=3 -> Plan.root=1 ->
+        # Relation.local_relation=11 -> LocalRelation.data=1.
+        relation = 
_exact_date_fields(_exact_date_fields(_exact_date_fields(request)[3])[1])
+        while 18 in relation:  # Relation.to_df -> ToDF.input.
+            relation = _exact_date_fields(_exact_date_fields(relation[18])[1])
+        captured.append(_exact_date_fields(relation[11])[1])
+        context.abort(grpc.StatusCode.INVALID_ARGUMENT, "captured date input")
+        yield b""  # Make this a streaming handler.
+
+    with ThreadPoolExecutor(max_workers=2) as executor:
+        server = grpc.server(executor)
+        server.add_generic_rpc_handlers((grpc.method_handlers_generic_handler(
+            "spark.connect.SparkConnectService", {
+                "ExecutePlan": 
grpc.unary_stream_rpc_method_handler(_exact_date_execute),
+            }),))
+        port = server.add_insecure_port("127.0.0.1:0")
+        server.start()
+        connection.send(port)
+        try:
+            while True:
+                command = connection.recv()
+                if command == "stop":
+                    break
+                assert command == "get"
+                connection.send(captured.pop(0))
+        finally:
+            server.stop(0).wait()
+
+
[email protected]
+def exact_date_capture_input():
+    import multiprocessing
+    pytest.importorskip("grpc")
+    from pyspark.sql import SparkSession
+
+    context = multiprocessing.get_context("spawn")
+    parent, child = context.Pipe()
+    process = context.Process(target=_exact_date_capture_server, args=(child,))
+    process.start()
+    try:
+        assert parent.poll(20), "capture service failed to start"
+        spark = 
SparkSession.builder.remote(f"sc://127.0.0.1:{parent.recv()}").getOrCreate()
+
+        def _exact_date_capture(data, schema):
+            import pyarrow as pa
+            df = spark.createDataFrame(data, schema=schema)
+            with pytest.raises(Exception, match="captured date input"):
+                df.collect()
+            parent.send("get")
+            assert parent.poll(10), "capture service failed to return the IPC 
payload"
+            return pa.ipc.open_stream(parent.recv()).read_all()
+
+        yield _exact_date_capture
+    finally:
+        if process.is_alive():
+            parent.send("stop")
+        process.join(10)
+        if process.is_alive():
+            process.terminate()
+            process.join(10)
+        parent.close()
+        child.close()
+
+
[email protected]("source", ["tuples", "pandas"])
[email protected]("schema_kind", ["inferred", "names", "explicit", 
"ddl"])
+def test_exact_date_ipc_calendar_range(exact_date_capture_input, source, 
schema_kind):
+    import calendar
+    import pyarrow as pa
+    from pyspark.sql import types as T
+
+    # Every year's endpoints and leap day cover the complete supported calendar
+    # range, including non-leap centuries, without relying on the Rust decoder.
+    dates = [_exact_date_date.min, _exact_date_date.max, 
_exact_date_date(1969, 12, 31), _exact_date_date(1970, 1, 1),
+             _exact_date_date(1970, 1, 2), None]
+    for year in range(1, 10000):
+        dates.extend([_exact_date_date(year, 1, 1), _exact_date_date(year, 12, 
31)])
+        if calendar.isleap(year):
+            dates.append(_exact_date_date(year, 2, 29))
+    schema = {"inferred": None, "names": ["day"], "ddl": "day date",
+              "explicit": T.StructType([T.StructField("day", 
T.DateType())])}[schema_kind]
+    data = [(value,) for value in dates]
+    if source == "pandas":
+        pd = pytest.importorskip("pandas")
+        data = pd.DataFrame(data, columns=["day"])
+    table = exact_date_capture_input(data, schema)
+    assert table.column(0).type == pa.date32()
+    assert table.column(0).to_pylist() == dates
+    epoch = _exact_date_date(1970, 1, 1)
+    expected_days = [None if value is None else (value - epoch).days for value 
in dates]
+    assert table.column(0).cast(pa.int32()).to_pylist() == expected_days
+
+
 # --------------------------------------------------------------------------- 
schema cache
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to