dbtsai opened a new pull request, #278: URL: https://github.com/apache/spark-connect-rust/pull/278
### What changes were proposed in this pull request? Use each explicit pandas date column's exact NumPy backing array for both eligibility validation and Arrow conversion. Add regressions for customized `Series.values` and `Series.to_numpy`, and document the existing date importer path. ### Why are the changes needed? Follow-up to #138 / SPARK-60716. For 10,000 stored dates of `2020-01-01`, a customized `Series.values` returning `2021-02-03` made the fast path validate the stored cells but emit the replacement dates. This client's previous DataFrame row conversion retained `2020-01-01`. Validation and serialization now consume the same concrete storage. Canonical pandas column extraction through `iloc` and `Series.array` remains trusted, consistent with the existing string path; this change does not authenticate every mutable pandas descriptor. ### Does this PR introduce _any_ user-facing change? Yes. Customized Series accessors no longer replace the validated date cells or run during this optimization. Ordinary date values, null handling, names, thresholds and transport routing are preserved. ### How was this patch tested? - Both new accessor regressions fail against the landed helper; the updated helper passes all 26 selected helper cases (865 deselected). - Independent expected offsets and null masks pass for 30,000 cells spanning Python's full date calendar range; the regression checks Arrow IPC roundtrips and unchanged row normalization. - Matching official PySpark 4.3.0 conversion passes 10,000 ordinary full-range date/null values. The custom-accessor regression targets this client's previous row-path contract. - `git diff --check` passes. Genuine local Claude Code reviewed the full landed change and the complete follow-up patch; the final source review approved with no blockers and documented nonblocking limitations/nits. **Draft validation limitation:** helper checks use installed native schema types, not a fresh candidate extension. Cargo/Rust/protoc are unavailable on this host, so the required Cargo lib/all-integration/wasm compile/fmt/Clippy checks, fresh extension, native mock/live tests and full official parity/API gates remain pending. No overall validation convergence is claimed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6); reviewed with Claude Code 2.1.314. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
