dbtsai opened a new pull request, #144: URL: https://github.com/apache/spark-connect-rust/pull/144
### What changes were proposed in this pull request? Pandas object columns containing strings currently materialize Python rows during `createDataFrame`, even with an explicit string schema. Encode eligible columns directly into Arrow IPC in Rust while preserving the existing inline `LocalRelation` transport and Spark schema. The path requires an ordinary pandas DataFrame with at least 10,000 rows, concrete NumPy object storage, an explicit native UTF8_BINARY string schema, and exact `str` values or nullable `None`. Customized pandas/schema hooks, string subclasses, unsupported missing values and surrogates retain the existing conversion/error path. The required internal IPC bridge is included in this change. Depends on [PR121](https://github.com/apache/spark-connect-rust/pull/121), including its [PR118](https://github.com/apache/spark-connect-rust/pull/118)/[PR108](https://github.com/apache/spark-connect-rust/pull/108) prerequisites. The narrow delta against PR121 changes five production files and adds 19 test definitions. ### Why are the changes needed? The frozen client-only benchmark compares published PR121 `c4284a3e0b71c8d624a1253c52337419761b2131` without the bridge against candidate `d31870bc4ceb15c3cadc4bc0a44fcb16f7b5e723` with the complete fix, using separate fresh wheels/environments and unchanged workloads/validators. All 535 samples completed. An independent passive raw artifact audit cross-checks original validator receipts, hashes and all raw metrics; it does not replay Arrow decoding. Medians below use five worker samples. Each string frame has 10,000 rows and three columns. Each focused warm sample averages 20 measured constructions after three warmups; focused first-construction samples measure one construction after zero warmups. | Case/profile | Baseline ms | Candidate ms | Change | |---|---:|---:|---:| | Ordinary pandas 2.3.3 object strings, warm | 23.137 | 2.154 | −90.7% | | Ordinary pandas 2.3.3 object strings, first construction | 24.298 | 2.935 | −87.9% | | Focused all-null strings, warm | 2.868 | 0.876 | −69.5% | | Late subclass fallback, first construction | 23.938 | 25.921 | +8.3% | | Early surrogate error, warm | 1.415 | 1.659 | +17.3% | | Middle surrogate error, first construction | 13.453 | 15.252 | +13.4% | The middle-surrogate candidate maximum was 21.036 ms versus 13.605 ms baseline. The ordinary pandas 3.1 StringDtype case retains the fallback path and increased 1.95% warm/6.61% first-construction elapsed time. Ordinary Arrow input increased 15.0% in warm elapsed time while CPU decreased 8.4%. Other guarded workloads increased: the million-row pandas float case by 3.93% elapsed/5.04% CPU and UDF construction by 4.27% elapsed/3.92% CPU. Focused CPU medians increased 18.72% for early-surrogate warm errors, 12.17% for middle-surrogate first-construction errors and 7.75% for late-subclass first construction. These five-sample observations do not establish statistical significance or a general overhead bound. Warm ordinary held-process RSS under pandas 2.3.3 was: | Input | Baseline MiB | Candidate MiB | |---|---:|---:| | Pandas object strings, 10,000 × 3 | 113.672 | 107.000 | | Pandas array values, 10,000 × 3 | 121.703 | 121.047 | | Wide pandas input, 10,000 × 32 | 118.328 | 118.953 | Measured construction starts after interpreter/library/session/input preparation. Ordinary first-construction timing also follows an untimed version call; the focused worker has no explicit version call. First-construction samples measure one constructor call without constructor warmup. Cold-interpreter costs were not measured. Focused sampled-peak/post-GC RSS was not collected; ordinary post-GC RSS includes later correctness work. The client is 4.3.0 on Python 3.12.14 with pandas 2.3.3/3.1.0, NumPy 2.5.4, PyArrow 26.0.0 and grpcio 1.84.1. A separate official PySpark 5.0.0.dev0 reference at Apache Spark `ca1514a2b4e72f9529871f34d09881cd4921df82` is measured under pandas 2.3.3 only. The Rust-client and official PySpark versions differ, so the official result provides separate context. Improvements above compare the two Rust-client revisions; historical minimal/combined-tree timings were not substituted. ### Does this PR introduce _any_ user-facing change? Eligible pandas string input uses the column encoder. Tests preserve values, nulls, field names/metadata, error ordering and inline transport/configuration behavior. Paired ordinary 20-counter RPC maps are equal. Focused intercepted RPC method maps are zero; their separate inferred zero 20-counter maps do not establish a complete end-to-end transport observation. ### How was this patch tested? - Both exact revisions pass formatting and release library tests: 394 client tests and 95 core tests each. Native and host `wasm-udf` feature test compilation each produce 37 test executables; no WASM-target execution is claimed. - Fresh wheels pass complete same-source `test_arrow_handoff.py` and `test_dropin_offline.py` using `pytest --rootdir=python --import-mode=importlib -q -rs`. Baseline: 522 passed/9 skipped for each pandas version. Candidate: 681 passed/9 skipped with pandas 2.3.3; 679 passed/11 skipped with pandas 3.1.0. The two extra skips concern unavailable older infer-string storage behavior. - All 531 baseline collected cases/outcomes are preserved. The 19 added definitions cover 159 cases per pandas version, including complete IPC cells/schema, subclass/surrogate fallback, descriptors/aliases, normalization and backing-array hooks, column storage and large inline input. - An independent passive raw artifact audit cross-checks original validator receipts/hashes and all raw metrics for all 535 samples: 430 IPC, 60 expected-error and 45 guard samples, with raw requests/IPC, distributions and adverse costs retained. It does not replay Arrow decoding. All six supervised process groups are absent before releasing the shared lock; four actual role packages and the separate official reference retain their required identities. Package checks cover existing dependency RECORD-listed non-bytecode files and recorded PySpark import origins; they do not establish continuous isolation, unlisted/shadow dependencies or loaded dependency origins. Probe receipts cover outer role/reference commands; nested internal `check_output` partial stdout is not separately receipted. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
