dbtsai opened a new pull request, #134: URL: https://github.com/apache/spark-connect-rust/pull/134
Nonempty Arrow input currently reads `timestampType`, session timezone, and `useLargeVarTypes` in three sequential Config RPCs. Fetch them together through the existing core `get_configs` transport, retaining fresh reads on every construction and the separate upload-options request. Total Config RPCs fall from four to two; empty input still makes zero. Depends on #118 at `ce89b8dd496bbb1529f1b5531b1eaaa6d739bef0` (transitively #108). The follow-up diff is three files against that head; it adds no config cache, pandas routing, upload redesign, or workflow changes. Five fresh samples per client/case against a frozen localhost mock harness: | Case | Rust baseline | Candidate | Ratio | |---|---:|---:|---:| | Arrow 10k × 3 | 1.127ms | 0.739ms | 1.53× | | Arrow 30k × 3 | 1.095ms | 0.751ms | 1.46× | | pandas int64 10k × 3, PR 121 on both sides | 1.304ms | 0.929ms | 1.40× | | pandas int64 30k × 3, PR 121 on both sides | 1.415ms | 0.940ms | 1.50× | The small warmed cases have disjoint observed timing ranges. The 1M-row cases show overlapping ranges. Cold 30k cache-threshold construction has a lower median (4.397→3.509ms, no artifact RPCs), but overlapping ranges; no reliable cold-case speedup is claimed. All 420 samples passed. Unaffected guards retain their RPC counts and overlapping timing ranges. PR 121 is a measurement-only dependency on both pandas sides and is excluded from this change. Validation: 394 Rust library tests pass; integration tests compile with/without `wasm-udf`; rustfmt and diff checks pass. Python selection: 324 pass, 11 live-server skips. Eighteen new cases in the existing CI-selected Arrow suite cover request keys/order, reversed/missing responses, errors/retry, config mutation, empty input, and independent timestamp/timezone/large-offset IPC/schema/cell expectations. The baseline passes the corresponding semantic checks and fails the new batching assertion as expected. Both combined pandas environments pass 72 tests, including independent nullable int64 IPC cell/schema checks. No JVM/live-server parity run. Clippy is unavailable in the configured offline toolchain. Results establish local mock-RPC construction performance, not end-to-end Spark execution speed. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
