[
https://issues.apache.org/jira/browse/SPARK-61269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61269:
----------------------------
Fix Version/s: connect-rust-4.3.0
> Fast-path exact integer literals in Spark Connect Rust
> ------------------------------------------------------
>
> Key: SPARK-61269
> URL: https://issues.apache.org/jira/browse/SPARK-61269
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Exact built-in integer operands such as `F.col("id") == 5` currently pass
> through generic enum, NumPy and nonprimitive scalar checks in the Rust
> scalar-to-Column conversion. Repeated integer conversion adds work when
> constructing expressions such as nested CASE expressions.
> Add a type-exact built-in integer branch after the existing Column
> conversion. Integers that fit in signed 64 bits directly produce integer or
> long literals using the existing signed 32-bit boundary. Preserve the generic
> path for booleans, integer subclasses, IntEnum, NumPy scalars and integer
> overflow. Add wire-boundary, boolean and probe-sentinel regression tests.
> Two separate local 45-sample fake-server runs against Rust master
> `0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28` measured 12.84%/10.02% lower wall
> time and 16.29%/16.67% lower CPU time on the expression-construction target,
> with all measured guards passing. The original same-binary control remained
> noise-uncertain; these are exploratory measurements and do not establish
> full-catalog or release acceptance.
> Saved validation: 2,589 offline tests passed with eight remote-server skips;
> a separate 72-test ML offline module passed; 495 Rust library tests passed;
> integration/WebAssembly targets compiled; four selected native live tests
> passed on each candidate/reference role. Full API/native/official parity,
> full holdout and required CI remain incomplete. Workspace Clippy remains
> non-green (exit 101); its diagnosed files matched the pinned master source,
> without a separately executed master Clippy run.
> Generated-by: OpenAI Codex
> All validation and measurements above used the unchanged patch on pinned base
> 0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28. The patch applies cleanly to master
> 8768cc4f780c7fb144e0b7a72ed5d5be14005023; merged-base build/test and CI
> remain pending.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
