dbtsai opened a new pull request, #129: URL: https://github.com/apache/spark-connect-rust/pull/129
Exact builtin list/tuple values currently pass through Decimal imports and numeric extraction probes before being recognized as arrays. Dispatch them before those probes and recursively reuse `py_to_value` for each element. Container subclasses keep the existing generic path, preserving numeric protocol precedence and existing built-in iteration. Nested Rows remain structs; byte values remain binary and small-integer lists remain arrays. This is stacked on #111 (full head `c80006764108c454a2ced46ac888742bc1c7b277`) merged with current master `d884e347`. Both benchmark wheels include the same #111 implementation, so the numbers below measure only the added container dispatch. The container-only change is commit `b0796a4`; the dependency changes are also visible until #111 lands. Five fresh-process samples per case/client using the unchanged frozen harness `1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4`, with rotated client order and serialized CPU work. Release wheels, Apple M3 Max, Python 3.12.14, NumPy 2.5.4, pandas 2.3.3, PyArrow 26.0.0. These are client-only localhost fixture measurements; no JVM or live Spark server. | 10,000×3 array input | PR111 baseline ms (min–max) | Candidate ms (min–max) | Incremental speedup | |---|---:|---:|---:| | Tuples | 25.467 (23.551–28.613) | 6.820 (6.276–7.612) | 3.73× | | pandas | 26.111 (25.218–26.675) | 7.633 (7.453–8.244) | 3.42× | All 225 samples passed across 15 cases, including timestamp, date, decimal, string, mixed numeric input, API construction and unrelated nullable/nested returns. Guard medians ranged from 4.4% faster to 5.1% slower; every slower guard had overlapping sample ranges. Peak RSS for tuple array input rose from 119.58 to 124.23 MiB; pandas from 120.28 to 121.23 MiB. Tuple conversion remains slower than the Python reference (2.79 ms); Python pandas input was 6.64 ms. Validation: - 392 Rust client library tests passed; formatting and diff checks passed. - Both wheels passed 22 new conversion/protocol tests and all 290 existing offline tests. - Both wheels separately passed 12 full-cell IPC checks over all frozen input fixtures: 24 checks, 720,000 decoded cells. The frozen construction validator checks DataFrame type, so this additional validation independently captures outbound IPC in a separate process and checks Arrow types and every value with PyArrow. Timestamp timezone representation is normalized explicitly to UTC. - Tests cover exact list/tuple nesting, null/empty arrays, errors, subclass numeric protocols and overridden iteration, nested Rows, binary distinctions, tuple/pandas sources and schema variants. No numeric conversion, Arrow encoding, or field-name changes. Full live-server parity remains untested. Initial test setup failures (spawn module importability, sandbox localhost binding, unsupported nested DDL fixture) were preserved and corrected before the successful runs. [JIRA SPARK-60687](https://issues.apache.org/jira/browse/SPARK-60687) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
