dbtsai opened a new pull request, #113: URL: https://github.com/apache/spark-connect-rust/pull/113
### What changes were proposed in this pull request? Share immutable explicit-schema and pandas column names across rows created by one local DataFrame conversion, using the existing `StructValue`/`Row` shared-name representation. Preserve name precedence, input parsing, owned values and error order. Add an isolated local gRPC fixture that checks complete IPC names/values, duplicates, Unicode, nulls, empty input, caller mutations and overridden-input errors. ### Why are the changes needed? Cloning every field name for every input row wastes conversion work and retained memory. In an isolated macOS arm64/Python 3.12 benchmark, these target effects repeated in a full screen and independent confirmation. Confirmation medians relative to `c36c846`: | Input | CPU | Wall | Held RSS | | --- | ---: | ---: | ---: | | Explicit Rows, 10,000 × 3 | −21.50% | −21.50% | −1.203 MiB | | Tuples, 10,000 × 32 | −7.92% | −7.93% | −17.367 MiB | | pandas, 10,000 × 32 | −7.18% | −7.18% | −22.461 MiB | | Tuples, 100,000 × 3 | −9.90% | −9.90% | −21.430 MiB | | pandas, 100,000 × 3 | −8.23% | −8.24% | −25.953 MiB | The production patch is the measured patch. Each phase used 30 fresh processes per build/case in rotating order across 48 cases, with isolated release wheels and a same-binary control. Input timing includes conversion and IPC serialization; complete expected-value checks happen outside timing. Harness: [frozen benchmark revision](https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae). **This draft has not passed the complete CPU or memory acceptance policy.** Each phase retained 60 pre-existing `Duration(Microsecond)` warmup failures across the two Rust builds ([SPARK-60607](https://issues.apache.org/jira/browse/SPARK-60607)); no samples were missing. Five timing metrics in each phase have inconclusive guard confidence bounds. Successful-operation RSS guards pass. Three held-RSS target metrics repeat the numerical saving criterion (≥10% and ≥8 MiB); no peak metric qualifies and the complete memory gate still fails. RSS does not prove allocation counts. Effects from separate patches are not additive. ### Does this PR introduce _any_ user-facing change? No intended behavior or API change. ### How was this patch tested? Revalidated 410 Rust unit/golden/sharing checks, all integration-test compilation in default and `wasm-udf` modes, formatting and core Clippy. Full offline Python suite: 274 passed, including a subprocess that runs four input-name fixture tests. All three PR builds retain the baseline 108-class API snapshot. The measured build also passed 30 input-name expectations, 38 numeric expectations and 11 nested Python wire comparisons. Workspace Clippy is blocked by the missing `wasm32-unknown-unknown` target in the example build. Full live Spark/official-suite parity remains pending; no local Connect server is listening. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
