dbtsai opened a new pull request, #112: URL: https://github.com/apache/spark-connect-rust/pull/112
### What changes were proposed in this pull request? Build a borrowed typed decoder once per Arrow column and nested child, then reuse it during row materialization. This removes repeated type dispatch/downcasts and list/map slicing across mixed schemas. Preserve row-major evaluation, lazy null/unsupported-type errors and absolute child offsets. Add regression groups for sliced lists/maps, null parents, lazy errors and mixed nested data. ### Why are the changes needed? Repeated dispatch is avoidable work when consuming nested Arrow output. In an isolated macOS arm64/Python 3.12 benchmark, both deeply mixed targets repeated an improvement in a full screen and independent confirmation. Confirmation relative to `c36c846`: | Fully consumed depth-16 mixed output | CPU median (95% CI) | Wall median | Held RSS | | --- | ---: | ---: | ---: | | Collect and scan | −8.97% [−10.93%, −7.99%] | −8.62% | +0.828 MiB | | Iterator | −8.98% [−9.46%, −7.40%] | −8.94% | +0.938 MiB | The production patch is the measured patch. Each phase used 30 fresh processes per build/case in rotating order across 38 cases, with isolated release wheels and a same-binary control. Returned values are fully consumed during these target timings. Harness: [frozen benchmark revision](https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae). **This draft has not passed the complete CPU or memory acceptance policy.** Each phase retained 60 pre-existing `Duration(Microsecond)` warmup failures across the two Rust builds ([SPARK-60607](https://issues.apache.org/jira/browse/SPARK-60607)); no samples were missing. Five screen and six confirmation timing metrics have inconclusive guard confidence bounds. Successful-operation RSS guards pass, but no repeated memory-saving criterion passes. These target observations do not establish overall acceptance. ### Does this PR introduce _any_ user-facing change? No intended behavior or API change. ### How was this patch tested? Revalidated 415 Rust unit/golden/sharing checks, including five new decoder regression groups; all integration-test compilation in default and `wasm-udf` modes; formatting and core Clippy. Full offline Python suite: 273 passed. The measured build also passed 38 numeric expectations, 11 independent nested Python wire comparisons and an unchanged 108-class API snapshot. Workspace Clippy is blocked by the missing `wasm32-unknown-unknown` target in the example build. Full live Spark/official-suite parity remains pending; no local Connect server is listening. RSS is not an allocation counter. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
