dbtsai opened a new pull request, #135: URL: https://github.com/apache/spark-connect-rust/pull/135
### What changes `createDataFrame` currently materializes a Value vector and Row for every tuple before building Arrow columns. For an explicit StructType containing only LongType fields, build Int64 columns directly when every row is an exact built-in tuple and every cell is an exact built-in int fitting i64 or None. This removes intermediate row allocations. Other containers/types, subclasses, overflow, width errors, and empty input use the existing converter. The core inline constructor reuses the row-path IPC writer and preserves schema metadata/nullability and offline LocalRelation transport. The builder stays out of line to avoid a measured mixed-type guard cost. No new Python dependency, workflow change, or PR dependency; production is based on master `d884e34`. ### Validation and performance Both benchmark sides share complete #111, #113, and #114 runtime changes. The unchanged frozen harness (`1bbdec9`) ran five fresh processes per client/case: **270/270 samples passed** across 18 tuple, wide/nullable, pandas, other-type and API cases. | Exact LongType tuples | Optimized baseline | Candidate | Speedup | |---|---:|---:|---:| | long-tuples-1000x3 | 0.141 ms | 0.070 ms | 2.00x | | long-tuples-10000x3 | 1.006 ms | 0.380 ms | 2.65x | | long-tuples-100000x3 | 12.612 ms | 3.684 ms | 3.42x | | long-tuples-1000000x3 | 136.561 ms | 36.496 ms | 3.74x | | long-tuples-10000x32 | 8.616 ms | 2.839 ms | 3.03x | | nullable-tuples-100000x3 | 12.747 ms | 3.870 ms | 3.29x | 1M×3 reference Python: 106.638 ms. Whole-process sampled peak RSS: **612.4 → 284.8 MiB**. Integer construction retains zero RPC/config/artifact deltas. The mixed-type guard is 7.348 → 7.197 ms. The large-pandas guard showed +3.9% in the five-sample matrix; a 10-sample follow-up measured 219.421 → 223.355 ms (+1.8%), with overlapping ranges. This small difference is disclosed; pandas is outside the fast path. - 392 Rust library tests passed; all integration tests compile with and without wasm-udf; fmt/diff checks passed. - Existing CI-selected Python offline suite: 275 passed; five tuple test groups also pass independently on production/baseline/candidate. - Independent IPC/schema oracles compare every cell, including 1M×3 inputs across official Python and all three Rust environments, widths 1/3/32, nulls, metadata and signed boundaries. Baseline/candidate/production exception snapshots match for overflow, bool, subclasses, mixed rows, width errors and nullability errors. - No live Spark/JVM. Clippy is unavailable in the configured offline toolchain. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
