DB Tsai created SPARK-61269:
-------------------------------
Summary: Fast-path exact integer literals in Spark Connect Rust
Key: SPARK-61269
URL: https://issues.apache.org/jira/browse/SPARK-61269
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Exact built-in integer operands such as `F.col("id") == 5` currently pass
through generic enum, NumPy and nonprimitive scalar checks in the Rust
scalar-to-Column conversion. Repeated integer conversion adds work when
constructing expressions such as nested CASE expressions.
Add a type-exact built-in integer branch after the existing Column conversion.
Integers that fit in signed 64 bits directly produce integer or long literals
using the existing signed 32-bit boundary. Preserve the generic path for
booleans, integer subclasses, IntEnum, NumPy scalars and integer overflow. Add
wire-boundary, boolean and probe-sentinel regression tests.
Two separate local 45-sample fake-server runs against Rust master
`0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28` measured 12.84%/10.02% lower wall
time and 16.29%/16.67% lower CPU time on the expression-construction target,
with all measured guards passing. The original same-binary control remained
noise-uncertain; these are exploratory measurements and do not establish
full-catalog or release acceptance.
Saved validation: 2,589 offline tests passed with eight remote-server skips; a
separate 72-test ML offline module passed; 495 Rust library tests passed;
integration/WebAssembly targets compiled; four selected native live tests
passed on each candidate/reference role. Full API/native/official parity, full
holdout and required CI remain incomplete. Workspace Clippy remains non-green
(exit 101); its diagnosed files matched the pinned master source, without a
separately executed master Clippy run.
Generated-by: OpenAI Codex
All validation and measurements above used the unchanged patch on pinned base
0ac7a0f4e8c98a2d13157175b20c9ebb3c536c28. The patch applies cleanly to master
8768cc4f780c7fb144e0b7a72ed5d5be14005023; merged-base build/test and CI remain
pending.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
