dbtsai opened a new pull request, #109: URL: https://github.com/apache/spark-connect-rust/pull/109
Local input conversion builds a second complete row matrix, then clones each column before Arrow conversion. Coerce one column at a time to remove those intermediate copies while retaining the existing coercion rules. For homogeneous Int64 result batches, downcast each column once and preserve the generic decoder for all other batches. Each optimization was measured **independently** against `74b8e41`; that baseline has exactly the same source tree as master base `c36c846`. Each screen and fresh confirmation used 15 fresh processes per build/case in rotating order. These are observed median changes; the combined PR head has not been benchmarked. | Workload | CPU, screen / confirmation | Wall, screen / confirmation | Held RSS delta, MiB | | --- | ---: | ---: | ---: | | Tuple input, 10,000 × 32 | −10.10% / −13.56% | −10.05% / −13.49% | −28.03 / −28.95 | | String input, 10,000 × 3 | −9.36% / −8.24% | −9.29% / −8.24% | −3.84 / −3.44 | | Int64 collect, 4,096 × 512 | −55.08% / −57.34% | −62.35% / −67.50% | +0.0625 / +0.125 | | Int64 collect + full scan, 4,096 × 512 | −6.76% / −7.09% | −7.03% / −7.27% | +0.0781 / +0.2344 | **Draft; overall CPU and memory acceptance remain rejected.** The retained Duration(Microsecond) case fails during warmup in both Rust builds, and several timing guards remain uncertain. Available RSS guards pass. The repeated wide-input RSS reduction does not override those failures. Full live Spark parity and CI remain outstanding. Fresh combined validation: 423 Rust library/golden/sharing checks, all integration targets compiled with and without wasm-udf, formatting/core Clippy, unchanged 108-class Python API, and 13 independent Python equivalence checks. The new round trip covers homogeneous Long and nested mixed input across the encode/decode boundary. JIRA: https://issues.apache.org/jira/browse/SPARK-60606 Evidence and remaining guards: [benchmark note](https://github.com/dbtsai/spark-connect-rust/blob/8617d994c730dcbcc12e83ec258c163fd68ada77/dev/benchmarks/rsi-input-int64-20261027.md). [Harness](https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae). Generated with OpenAI Codex. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
