[
https://issues.apache.org/jira/browse/SPARK-61102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61102:
----------------------------
Description:
The Rust Python client inferred NumPy numeric literal types using dtype kind
and itemsize without applying PySpark 4.3.0's scalar-class checks. Distinct
scalar aliases can therefore be accepted even when official PySpark rejects
their class.
On macOS arm64, np.longdouble has an eight-byte dtype equal to float64 but
remains a distinct scalar class. Official PySpark rejects that unsupported
class with a bare AssertionError after DoubleType inference. Distinct
np.longlong aliases with supported integer dtypes require the corresponding
class check on platforms where that class is not accepted. Wider unsupported
longdouble dtypes retain their structured PySparkTypeError behavior.
The proposed implementation in
https://github.com/apache/spark-connect-rust/pull/381 applies numeric
scalar-class checks after dtype inference for scalar and array literals, while
preserving canonical supported classes, literal types and structured
unsupported-dtype errors. Regression expectations follow the reference behavior
for the actual platform, including the eight-byte longdouble case. The earlier
baseline failure remains recorded. This issue retains connect-rust-4.3.0
release tracking.
was:
The Rust Python client accepts a NumPy longdouble scalar as a Double literal on
macOS arm64, where longdouble has an 8-byte dtype equal to float64 but remains
a distinct Python scalar class. Official PySpark 4.3.0 rejects that scalar
class with AssertionError after DoubleType inference. For larger longdouble
dtypes it rejects the unsupported type with PySparkTypeError. The native
dtype-kind/itemsize dispatch currently misses the scalar-class check.
Add the corresponding class guard while preserving supported float32/float64
literals and arrays. Correct the existing offline regression to assert
platform-dependent reference error behavior rather than assuming every
longdouble dtype raises PySparkTypeError. This existing baseline failure was
reproduced during the Python/Rust compatibility fixes.
Summary: Match PySpark NumPy numeric scalar-class checks (was: Reject
unsupported NumPy longdouble scalar classes as PySpark does)
> Match PySpark NumPy numeric scalar-class checks
> -----------------------------------------------
>
> Key: SPARK-61102
> URL: https://issues.apache.org/jira/browse/SPARK-61102
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust Python client inferred NumPy numeric literal types using dtype kind
> and itemsize without applying PySpark 4.3.0's scalar-class checks. Distinct
> scalar aliases can therefore be accepted even when official PySpark rejects
> their class.
> On macOS arm64, np.longdouble has an eight-byte dtype equal to float64 but
> remains a distinct scalar class. Official PySpark rejects that unsupported
> class with a bare AssertionError after DoubleType inference. Distinct
> np.longlong aliases with supported integer dtypes require the corresponding
> class check on platforms where that class is not accepted. Wider unsupported
> longdouble dtypes retain their structured PySparkTypeError behavior.
> The proposed implementation in
> https://github.com/apache/spark-connect-rust/pull/381 applies numeric
> scalar-class checks after dtype inference for scalar and array literals,
> while preserving canonical supported classes, literal types and structured
> unsupported-dtype errors. Regression expectations follow the reference
> behavior for the actual platform, including the eight-byte longdouble case.
> The earlier baseline failure remains recorded. This issue retains
> connect-rust-4.3.0 release tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
