[
https://issues.apache.org/jira/browse/SPARK-61102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140756#comment-18140756
]
DB Tsai commented on SPARK-61102:
---------------------------------
Implementation and validation:
https://github.com/apache/spark-connect-rust/pull/381.
The final patch passed formatting, Rust core/API unit tests, integration-test
compilation with and without the WASM feature, and the offline Python suites
(2,170 passed; eight explicit live-server skips). Twenty no-server expectation
checks against official PySpark 4.3.0 passed for scalar and array literals. Two
selected native live cases passed, with the actual imported extension checked;
their literal coverage verifies int64, float32 and int8 schema mappings. It
does not cover every NumPy dtype or collected scalar value. Inspection found no
missing members across 12 API surfaces; signatures and behavior have narrower
coverage.
The complete raw-transport rerun recorded 2,586 passing outcomes across 87
normal files, with zero failures/errors, 641 skips, one deselection and no
retries. The four-module cluster suite recorded 15 passing invocations with 14
skips and no failures/errors or retries. These suites execute official Python
plans and converters over this build's Rust RPC transport, with the separate
native checks providing only their stated coverage. No per-child import
attestation or execution-time freeze of the entire official reference tree was
collected for those transport suites.
The first full transport attempt's seeded-random assertion failure remains
recorded. The unchanged test failed in official Python and both saved Rust
transports with the two-thread fixture and passed in all three with the default
four-thread fixture used for the completed rerun. A separate metadata-column
drift attempt failed because the test catalog plugin was unavailable; it is
excluded from the normal totals without adding a skip or waiver. The earlier
cluster attempt executed no checks and remains recorded as failed.
Clippy remains an advisory failure (exit 101). Linux and Windows were not run
locally. Scalar-class rejection follows platform-specific NumPy class
identities; it does not imply that every platform rejects np.longlong.
Prepared by OpenAI Codex.
> Match PySpark NumPy numeric scalar-class checks
> -----------------------------------------------
>
> Key: SPARK-61102
> URL: https://issues.apache.org/jira/browse/SPARK-61102
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust Python client inferred NumPy numeric literal types using dtype kind
> and itemsize without applying PySpark 4.3.0's scalar-class checks. Distinct
> scalar aliases can therefore be accepted even when official PySpark rejects
> their class.
> On macOS arm64, np.longdouble has an eight-byte dtype equal to float64 but
> remains a distinct scalar class. Official PySpark rejects that unsupported
> class with a bare AssertionError after DoubleType inference. Distinct
> np.longlong aliases with supported integer dtypes require the corresponding
> class check on platforms where that class is not accepted. Wider unsupported
> longdouble dtypes retain their structured PySparkTypeError behavior.
> The proposed implementation in
> https://github.com/apache/spark-connect-rust/pull/381 applies numeric
> scalar-class checks after dtype inference for scalar and array literals,
> while preserving canonical supported classes, literal types and structured
> unsupported-dtype errors. Regression expectations follow the reference
> behavior for the actual platform, including the eight-byte longdouble case.
> The earlier baseline failure remains recorded. This issue retains
> connect-rust-4.3.0 release tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
