dbtsai opened a new pull request, #126: URL: https://github.com/apache/spark-connect-rust/pull/126
The first `pyspark.sql.functions` import loads the error catalog solely to support invalid higher-order callables. Import `PySparkValueError` inside the three validation failure branches so ordinary expressions and valid lambdas avoid that dependency. The canonical exception type, conditions, message parameters, and messages remain unchanged. Validation: 281 candidate tests pass (274 existing offline tests and seven fresh-process tests). Baseline passes all 274 existing tests and the six error compatibility tests; the new lazy-import test fails as expected. Five paired fresh-process higher-order guards also pass. `git diff --check` passes. Using unchanged frozen benchmark harness `1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4`, all 420 samples pass: 14 construction cases plus 14 outbound cases, three clients, five fresh processes each. Against master `d884e347452c6446c7bc54af151755c62801d8cd`, construction improves 2.50x and outbound 2.05x by geometric mean, with identical Rust RPC counters. For union-10 construction, median time falls from 4.488 ms (4.380–4.606) to 1.718 ms (1.659–1.751). These cases include first-use imports; this does not establish a warmed-operation speedup. Python-only validation uses a separate candidate environment and a wheel containing the exact frozen release native binary; only `functions.py` and wheel RECORD differ. All measurement work was serialized. No JVM or live Spark server was used. The first actual invalid lambda now pays the deferred import cost (supplemental fresh-process medians: 12.81 ms versus 0.10 ms after baseline's eager import). This is intentional; error content and identity are preserved. Full live-server/upgrade test gates were not run for this import-only change. JIRA: https://issues.apache.org/jira/browse/SPARK-60683 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
