dbtsai opened a new pull request, #137: URL: https://github.com/apache/spark-connect-rust/pull/137
Creating a DataFrame from an already canonical nullable int64 Arrow table still imports the general conversion/error hierarchy and runs redundant casts and renaming. Return that table unchanged when its exact native schema, names, nullability and metadata match. Loaded converters, modified native descriptors/type bindings, and all other inputs retain the general path. Config reads remain fresh; native IPC serialization and upload behavior are unchanged. Depends on #134 at `911e23cbe21409fbb530894939b4f0134e69d6af` (including #118/#108). [Incremental three-file diff](https://github.com/dbtsai/spark-connect-rust/compare/911e23cbe21409fbb530894939b4f0134e69d6af...2e1b60445137cdad9234b5d7ee3acc35a5537fb4). No Rust or workflow changes. Five fresh processes per client/case, identical native binaries and dependencies within each comparison. Cold construction milliseconds, median [min–max]: | Environment / rows × 3 | Baseline | Candidate | Median ratio | |---|---:|---:|---:| | PR134 / 30k | 10.509 [5.159–11.693] | 1.840 [1.266–3.393] | 5.71× | | PR134 / 60k | 22.300 [12.307–27.493] | 8.123 [7.492–8.624] | 2.75× | | Integrated v4 / 30k | 9.916 [7.012–12.204] | 1.584 [1.181–2.178] | 6.26× | | Integrated v4 / 60k | 18.120 [10.875–27.316] | 7.318 [7.194–7.904] | 2.48× | All 255 samples pass. Every raw baseline/candidate pair has identical RPC counters, including request and artifact bytes. The unchanged benchmark harness is pinned at `1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4`; an explicit external workload extension supplies metadata-free `pa.table` inputs from three `np.arange(n, dtype=np.int64) + column_index` arrays and nullable LongType fields. Inputs are prepared outside timing, cold cases warm only the channel, and the cache threshold is 1 MiB. Seven original guard workloads are unchanged. Warm and 1M-row ranges overlap; no repeatable benefit is claimed there. Validation: 362 Python tests pass, 11 live-server skips; 38 focused integrated regressions pass; 394 Rust library tests pass; all integration tests compile with and without `wasm-udf`; rustfmt and diff checks pass. Existing CI-selected suites cover exact native IPC/cache hashes and every cell/schema at 10k/30k/60k/1M, nullable extrema, slices/chunks, empty input, metadata/nullability/casts, custom types/hooks, fresh imports, config errors, size limits and upload failures. Final wheel SHA256: `b9e95c4fe2a1e7103b61e45d13473a75a6e897eff8db8b3a362e1a927501adb0`. No JVM/live Spark execution was used; these are local mock-RPC construction measurements. Clippy is unavailable in the configured offline toolchain. The optimization deliberately falls back once the general converter is loaded. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
