dbtsai opened a new pull request, #145: URL: https://github.com/apache/spark-connect-rust/pull/145
### What changes were proposed in this pull request? Keep canonical Arrow-backed pandas `StringDtype` columns columnar for an explicit native schema containing only binary-collated `StringType` fields and at least 10,000 rows. Validate Arrow buffers, cast to standard UTF-8, and reuse the inline IPC bridge. Guards authenticate a finite set of directly skipped normalization/storage hooks; customized or unsupported inputs retain row conversion. This does not guarantee arbitrary pandas customization equivalence. Depends on [PR #144](https://github.com/apache/spark-connect-rust/pull/144), including its prerequisite stack. This narrow delta compares baseline `d31870bc4ceb15c3cadc4bc0a44fcb16f7b5e723` with candidate `28f2b65cfcda7460459e67dcbac7e9437b385c4b`: one production helper and two test files. Measurements do not attribute prerequisite-stack changes to this patch. ### Why are the changes needed? Arrow-backed string columns currently become Python objects and rows, discarding their columnar representation. Eligible inputs can retain Arrow storage through explicit-schema IPC. ### Does this PR introduce _any_ user-facing change? Eligible inputs use columnar conversion. Tests preserve values, schemas and checked error types/messages/order against forced row conversion, covering customized direct hooks, manager caches, malformed buffers and public error/RPC ordering. ### How was this patch tested? The selected offline suites passed: pandas 2.3.3, 865 passed/9 skipped; pandas 3.1.0, 863 passed/11 skipped. All 184 added cases pass in each; all 690 baseline outcomes/skip reasons remain. Original formatting, native client/core units (394/95), native/host-WASM test compilation and wheel builds are explicitly reused under identical production/build source bytes. Baseline Python results are reused; full candidate Python suites ran at the committed test head. Four-environment before/after captures verify source, wheels, installed packages, native and recorded dependency identities. Original/recovery-v2 failures remain FAILED; no fresh Clippy/live Spark result. Runs use Python 3.12.14, NumPy 2.5.4, PyArrow 26.0.0, grpcio 1.84.1 and pandas 2.3.3/3.1.0 with frozen harness `1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4`; Rust packages report 4.3.0, while separate official reference `ca1514a2b4e72f9529871f34d09881cd4921df82` reports 5.0.0.dev0 and is measured only with pandas 2.3.3. Three completed, independently reviewed and root-accepted groups contain **1,175 worker samples: explicit640, object160 and ordinary375**. Cohorts remain separate; five samples per role provide descriptive distributions, not significance, confidence intervals or general neutrality. **Explicit640.** All 16 eligible cells (two pandas versions × two profiles × two nominal missing slots × exact/Unicode-null-chunked variants) improve median elapsed 88.2–98.9% and CPU 89.3–98.9%. Candidate/baseline ratios are 0.010873–0.117645 and 0.010874–0.106766 respectively. Exact nominal `pd_na` medians, baseline → candidate: | pandas/profile | elapsed ms | CPU ms | |---|---:|---:| | 2.3.3/warm | 21.526 → 0.257 | 21.552 → 0.257 | | 2.3.3/first | 25.418 → 2.528 | 25.105 → 2.058 | | 3.1.0/warm | 23.213 → 0.252 | 23.234 → 0.253 | | 3.1.0/first | 26.510 → 2.153 | 26.172 → 2.148 | Fallback/error costs are mixed: 26/64 elapsed and 26/64 CPU medians rise. Warm 3.1.0 hook-error/pd_na elapsed 0.0442 → 0.1036 ms (2.345×, +0.0594), CPU 0.0445 → 0.0772 (1.736×, +0.0327); warm 2.3.3 hook-error elapsed reaches 1.596×. First 3.1.0 hook-error/nan elapsed 1.0639 → 1.4113 ms (+347.38 µs, 1.327×), CPU 0.781 → 0.993. Warm 3.1.0 Python-storage/nan elapsed 20.865 → 23.965 (1.149×), CPU 20.889 → 23.431 (1.122×); first 2.3.3 Python-storage/pd_na elapsed 26.093 → 29.031 (+2.938 ms/+11.26%). First 2.3.3 threshold/nan elapsed 25.594 → 28.164 (1.100×), CPU 25.120 → 27.551 (1.097×). Generic-Arrow worst elapsed/CPU ratios are 1.033×/1.050× in different cells. Eligible held-RSS medians fall 3,096,576–11,403,264 bytes. Warm 3.1.0 Python-storage/nan nevertheless has absolute held/post-validation increases +5,095,424/+4,571,136 bytes and signed held-minus-baseline +1,638,400 (4,784,128 → 6,422,528). First 2.3.3 generic-Arrow/nan has worst signed post-validation increase +2,555,904 (13,828,096 → 16,384,000), alongside absolute post-validation +2,883,584 and held +278,528. Eligible warm 2.3.3 chunked/pd_na absolute post-validation falls 5,505,024 while its signed increment rises 655,360. **Object160.** Pandas 2.3.3 only, eight 10,000×3 fixtures/two profiles. Candidate/baseline median and distribution-maximum timing ratios follow; candidate-minus-baseline absolute held-RSS and signed held-minus-baseline median differences in bytes. Lifetime-peak medians equal held medians. | profile/fixture | elapsed × | CPU × | elapsed max × | CPU max × | held Δ B | signed Δ B | |---|---:|---:|---:|---:|---:|---:| | warm/exact | 0.932 | 0.932 | 0.982 | 0.979 | +65,536 | -16,384 | | warm/all_null | 0.983 | 0.973 | 0.971 | 0.970 | +98,304 | 0 | | warm/early_subclass | 1.053 | 1.049 | 1.071 | 1.054 | -311,296 | +49,152 | | warm/middle_subclass | 0.991 | 0.991 | 0.884 | 0.944 | -409,600 | -98,304 | | warm/late_subclass | 0.986 | 0.986 | 1.013 | 1.012 | +262,144 | +376,832 | | warm/early_surrogate | 0.986 | 0.984 | 1.003 | 1.008 | +147,456 | +16,384 | | warm/middle_surrogate | 1.061 | 1.061 | 1.085 | 1.085 | +180,224 | +16,384 | | warm/late_surrogate | 1.005 | 1.006 | 1.004 | 1.004 | -16,384 | 0 | | first/exact | 1.010 | 1.023 | 0.943 | 1.078 | 0 | +32,768 | | first/all_null | 1.010 | 1.039 | 0.897 | 0.881 | +212,992 | -32,768 | | first/early_subclass | 1.043 | 1.025 | 1.107 | 1.074 | -294,912 | -49,152 | | first/middle_subclass | 1.085 | 1.045 | 1.083 | 1.046 | -147,456 | +32,768 | | first/late_subclass | 0.998 | 1.001 | 0.989 | 1.013 | -114,688 | +81,920 | | first/early_surrogate | 0.931 | 0.952 | 0.907 | 0.835 | -98,304 | +32,768 | | first/middle_surrogate | 1.041 | 1.014 | 1.110 | 1.065 | -32,768 | -16,384 | | first/late_surrogate | 1.002 | 1.004 | 0.943 | 0.940 | +49,152 | +65,536 | Nine elapsed/ten CPU medians rise. Warm early-subclass elapsed/CPU +5.30%/+4.86%; middle-surrogate +6.06%/+6.11%. First middle-subclass elapsed 24.847 → 26.954 ms (+8.48%), CPU 24.844 → 25.970 (+4.53%). Worst baseline-RSS median increase: first all-null +524,288; held/lifetime-peak: warm late-subclass +262,144; signed increment: the latter +376,832 (5,603,328 → 5,980,160). Distribution maxima also rise: first early-subclass elapsed +2.970 ms/CPU +2.007 ms; largest maximum ratios first middle-surrogate elapsed 1.110×/warm middle-surrogate CPU 1.085×. Warm early-subclass absolute held/lifetime-peak maxima rise +5,046,272 bytes, signed maxima +5,931,008. These compare cohort maxima, not matched samples or sampled construction peaks. **Ordinary375.** Twenty-seven cohorts: 21 pandas-2.3.3 cohorts include official Python plus both Rust roles; six selected pandas-3.1.0 cohorts compare Rust roles only. Versions have different selected coverage; official Python is a reference, not this patch's baseline. All candidate/baseline median and distribution-maximum timing ratios: | pandas/profile/case | elapsed × | CPU × | elapsed max × | CPU max × | |---|---:|---:|---:|---:| | 233/first/explicit-10k×32 | 0.795 | 0.990 | 0.740 | 0.567 | | 233/first/type-date-10k×3 | 0.998 | 1.011 | 1.022 | 1.020 | | 233/first/type-string-10k×3 | 0.923 | 1.020 | 1.084 | 0.926 | | 233/warm/arrow-explicit-10k×3 | 0.973 | 1.007 | 0.632 | 0.751 | | 233/warm/ddl-10k×3 | 1.013 | 1.014 | 0.986 | 1.007 | | 233/warm/explicit-10k×3 | 1.016 | 1.016 | 1.046 | 1.045 | | 233/warm/explicit-10k×32 | 0.999 | 1.002 | 0.117 | 0.605 | | 233/warm/inferred-10k×3 | 0.987 | 0.987 | 1.005 | 1.006 | | 233/warm/scale-1M×3 | 1.017 | 1.003 | 0.500 | 0.844 | | 233/warm/scale-100k×3 | 0.952 | 0.948 | 0.307 | 1.241 | | 233/warm/scale-1000x3 | 0.834 | 1.037 | 0.179 | 0.710 | | 233/warm/scale-100x3 | 1.025 | 1.026 | 1.136 | 1.138 | | 233/warm/tuples-explicit-10k×3 | 0.996 | 0.997 | 0.980 | 0.980 | | 233/warm/type-array-10k×3 | 0.997 | 0.997 | 0.974 | 0.974 | | 233/warm/type-date-10k×3 | 1.002 | 1.002 | 1.026 | 1.023 | | 233/warm/type-decimal-10k×3 | 1.020 | 1.020 | 1.104 | 1.060 | | 233/warm/type-string-10k×3 | 0.949 | 0.949 | 0.961 | 0.966 | | 233/warm/type-timestamp-10k×3 | 0.983 | 0.983 | 0.939 | 0.953 | | 233/warm/return-100k×8-collect | 0.999 | 1.015 | 0.853 | 0.979 | | 233/warm/return-100k×8-toArrow | 0.967 | 0.873 | 1.426 | 0.942 | | 233/warm/udf-construct-capture-0 | 0.964 | 0.968 | 1.082 | 1.075 | | 310/first/explicit-10k×32 | 0.758 | 0.996 | 0.693 | 0.929 | | 310/first/type-date-10k×3 | 1.003 | 1.008 | 0.999 | 1.007 | | 310/first/type-string-10k×3 | 0.061 | 0.058 | 0.075 | 0.058 | | 310/warm/explicit-10k×32 | 0.985 | 1.016 | 0.963 | 1.301 | | 310/warm/type-date-10k×3 | 0.997 | 0.996 | 0.986 | 0.986 | | 310/warm/type-string-10k×3 | 0.021 | 0.021 | 0.022 | 0.023 | Seven elapsed/14 CPU medians rise. Largest elapsed ratio: 2.3.3/warm/100×3 +2.5325%; CPU: 1000×3 +3.7037%. Decimal has largest absolute median increases: elapsed 24.831 → 25.325 ms (+1.99%), CPU +0.493 ms. Pandas-3.1.0 strings improve elapsed 22.904 → 0.481 ms warm and 22.198 → 1.345 first. Largest elapsed maximum increase: warm 2.3.3 toArrow 27.990 → 39.903 ms (+42.56%); CPU maximum ratio: warm 3.1.0 wide 1.301×; largest absolute CPU-maximum increase: decimal +1.512 ms. Worst higher absolute RSS medians: baseline +442,368 bytes (2.3.3 first wide); held +507,904/lifetime-peak +606,208 bytes (3.1.0 warm wide); post-GC +475,136 (2.3.3 warm Arrow); sampled peak +884,736 (2.3.3 warm 1M). Worst signed median increases: held +229,376 (warm toArrow), post-GC +1,015,808 (3.1.0 warm strings, despite lower absolute post-GC), sampled peak +950,272 (warm 1M). Decimal has largest absolute held/sampled/lifetime-peak maxima increases +2,408,448, post-GC +2,473,984; signed maxima +2,179,072. Maxima compare distributions, not matched endpoints. Every ordinary cohort has some higher descriptive observation. **Scope and limits.** Explicit/object warm uses 20 constructions/three warmups; first uses one/no constructor warmup after interpreter/input/session/server preparation. Ordinary warm uses one operation warmup; first uses zero after untimed `spark.version`; measured loops have one operation except warm 2.3.3 UDF (five). Automatic GC is not explicitly disabled. No cold-interpreter or per-sample helper-dispatch claim. Explicit/object loop/error-checking overhead is included; preparation/validation IPC decoding is outside construction timing. Ordinary setup/warmup/counters/RSS/validation are outside operation timing; process CPU brackets the wall timer. Explicit/object order alternates deterministically; ordinary retains rotated order, without randomization. Generic Arrow uses actual `pd.NA` in both nominal slots. Independent explicit review decodes all 480 retained last-successful DataFrame payloads/14,399,760 cells and verifies 1,680 measured expected-error checks; preceding successful iterations are not individually decoded. Object independently decodes 100 retained UTF-8 IPC/protobuf payloads/3,000,000 cells, only last successful results. Its 630 surrogate attempts/hook totals remain sealed-worker assertions: printed totals/one error fingerprint, no retained per-iteration errors or independent replay. Object has aggregate stdout timing/RSS, without pre-validator construction-measurement files/nanosecond endpoints. Ordinary rehashes/cross-links all 330 typed-validator receipts/requests/payloads covering 72,149,500 cells plus 45 noninput guards; typed value/null-mask/nested-null/schema validation remains a frozen producer assertion, not fresh independent decoding. Construction RPC zeros apply to explicit/object sole-loopback fixtures; frozen20 zeros are inferred separately. Ordinary directly observes all 20 counter keys in all 375 sample maps, including nonzero input Config traffic (four calls/512 bytes); all 135 Rust-pair maps match. Return/config traffic exists; there is no overall-zero or full-wire claim. RSS baseline/held precedes validation; lifetime `ru_maxrss` includes preparation/warmups. Explicit post-validation RSS follows output/mock-holder release/one GC; inputs/expected/schema/session/server and native arenas/internal frames may remain. Ordinary post-GC includes additive validation allocations. Neither proves complete reclamation/construction-only retention. Ordinary external RSS targets 2 ms but records 1–59 observations/largest gap 5.520125 ms; observation starts before go and stops on measured message, not exact timer endpoints; short transients can be missed. Object has no post-GC/sampled-peak/count/gap fields. Signed deltas can be nonpositive; ratios require positive denominators; medians of signed deltas differ from differences of absolute medians. Ordinary harness truncates stderr at 8,000 and exception stdout at 2,000 characters. Object streams/receipts exist before launch; interruption may leave RUNNING receipts/only written bytes. Selected dependency RECORD/PySpark-origin inventories do not establish shadow-import/loaded-dependency/continuous isolation. Inherited-group supervision does not cover arbitrary detach/setuid/host crash/realtime/external-supervisor-SIGKILL guarantees. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
