dbtsai opened a new pull request, #130: URL: https://github.com/apache/spark-connect-rust/pull/130
## What changes `createDataFrame` currently materializes supported pandas timestamps as Python objects and converts each value separately. Hand explicit `TimestampType` columns to the existing Arrow importer when every column is `datetime64[ns]` or `[us]` and all values are microsecond-aligned. Preserve aware instants; enable naive timestamps only with an actual UTC process (`TZ=UTC`) and UTC session. Preserve explicit names, metadata, nullability and `NaT`. Inference, names/DDL schemas, `TimestampNTZ`, mixed/object/Arrow extension dtypes, other units, empty frames, submicrosecond values and unsupported naive timezones retain the row fallback. The signed-int64 helper and generic fallback are unchanged. **Stacked on #121 at `a8774f0bc8289856b98d11ddbb6b0c0aff95eaee`, transitively #118 and #108.** Both measured Rust environments contain that identical dependency commit. [Timestamp-only diff](https://github.com/dbtsai/spark-connect-rust/compare/a8774f0bc8289856b98d11ddbb6b0c0aff95eaee...codex/spark-60686-pandas-timestamps). ## Validation - 394 Rust library tests passed; all Rust workspace tests compiled with `--release --locked --no-run`; formatting and whitespace checks passed. - 366 Python offline tests passed, nine existing skips. Public tests decode every emitted IPC value and schema and reject Python row materialization. The new no-row test fails on the pinned baseline as expected. - Independent official-Python/baseline/candidate client processes verified 17 full-IPC scenarios: UTC/non-UTC host and session zones, aware/naive values, both DST folds and transitions, `NaT`, negative and boundary values, precision and schema fallbacks. - Unchanged frozen harness/validators, separate pinned environments, release locked wheels, serialized builds/tests/measurements, no JVM/live Spark server. Five rotating fresh samples for 13 cases × three clients: **195 passed, zero failed**. Includes timestamp, integer, mixed numeric, date, string, array, decimal, Arrow input and return guards. A further ten-sample integer guard passed all 30 samples. | Frozen timestamp constructor, 10,000 × 3 | Median ms | Range ms | |---|---:|---:| | Python Connect | 1.321 | 1.225–1.524 | | PR121 baseline | 62.533 | 61.747–63.928 | | Candidate | 2.126 | 1.933–2.199 | **29.4× faster than the identical PR121 baseline.** Constructor validation remains unchanged; correctness is established independently by decoded IPC. These are client fixture measurements, not Spark execution timings. The small 1,000-row integer guard initially moved +0.159 ms (+12.2%); its ten-sample follow-up measured 1.495 → 1.354 ms, with overlapping ranges in both runs. Raw runs are retained. ## Known baseline gaps The supported columnar path matches reference microseconds at range boundaries where the baseline's pandas-object/datetime conversion and floating timestamp rounding lose precision (including an ns lower-bound sign/wrap case). This is an explicit correctness improvement. Unsupported fallbacks preserve pre-existing differences: naive timestamps use the host zone instead of the session zone, DST overlap/gap behavior differs, and -1 ns truncation differs. This PR does not extend those fallback semantics. JIRA: https://issues.apache.org/jira/browse/SPARK-60686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
