dbtsai opened a new pull request, #136: URL: https://github.com/apache/spark-connect-rust/pull/136
### What changes were proposed in this pull request? After #133, explicit Decimal local inputs still import decimal and resolve Decimal for every cell. Resolve the immutable C Decimal type once per construction and use a guarded conversion for explicit all-Decimal schemas with exact list/tuple rows and exact Decimal/None cells. Inferred, mixed, subclass, module-hook and free-threaded paths retain the existing converter. Depends on #133 at `cfd38075468b70cf3c4ac03dfcbde495ff0ba120`. [Incremental diff](https://github.com/dbtsai/spark-connect-rust/compare/cfd38075468b70cf3c4ac03dfcbde495ff0ba120...db4a7880b087e5183645a48c440809fc3588d51a). PR132 is used identically on both correctness-only sides and is not a production dependency. ### Why are the changes needed? [SPARK-60704](https://issues.apache.org/jira/browse/SPARK-60704) removes remaining per-cell module/type lookup after DecimalTuple allocation was removed. The unchanged harness uses separate frozen environments, identical full PR133 dependency, and five fresh samples per client/case. | 10,000 × 3 Decimal input | PR133 median ms [range] | Candidate median ms [range] | Reduction | |---|---:|---:|---:| | Tuples | 15.554 [15.391, 15.690] | 6.897 [6.833, 7.195] | 55.7% | | pandas | 17.867 [17.280, 25.373] | 8.734 [8.647, 10.491] | 51.1% | 255/255 samples pass across 17 cases. All 15 guard ranges overlap; native RPC deltas match. Median sampled peak RSS (baseline/candidate): tuples 111.0/111.3 MiB, pandas 112.4/112.5 MiB. ### Does this PR introduce _any_ user-facing change? Eligible explicit Decimal inputs construct faster. Values, scales, schemas, metadata, subclass hooks, errors and upload behavior are preserved. The fast path requires the immutable C Decimal type and execution with the GIL; other runtimes retain generic conversion. ### How was this patch tested? - 426 tests pass in the existing CI-selected test_dropin_offline.py; 426 also pass with the shared scientific parser. New tests check every IPC cell/validity bit and physical Decimal128 coefficient in 10,000 × 3 tuple/pandas inputs, plus inferred paths, nulls, metadata, mixed values/schemas, module rebinding, subclasses, context traps, errors and the no-GIL fallback. - Exact PR133 baseline passes 104 behavior cases and fails only lookup sensitivity (100 Decimal imports versus 0). A 300,000-cell instrumented profile records 300,000 versus 0 Decimal imports and 0 DecimalTuple constructions on both sides. - Two 40-case scientific matrices per side, including the unchanged PR133 matrix, pass with byte-identical IPC under an identical PR132 parser. 36 special outcomes match exactly between production baseline and candidate. - Rust client 392 tests and core 95 tests pass; all integration targets compile with/without wasm-udf; formatting and diff checks pass. 12 public-class API snapshots match. - No JVM/live Spark or official server suite was run. Clippy is unavailable in the configured offline toolchain. The tests are directly in the existing CI-selected file; no workflow change is needed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6). This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
