[
https://issues.apache.org/jira/browse/SPARK-61102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61102.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 381
[https://github.com/apache/spark-connect-rust/pull/381]
> Match PySpark NumPy numeric scalar-class checks
> -----------------------------------------------
>
> Key: SPARK-61102
> URL: https://issues.apache.org/jira/browse/SPARK-61102
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust Python client inferred NumPy numeric literal types using dtype kind
> and itemsize without applying PySpark 4.3.0's scalar-class checks. Distinct
> scalar aliases can therefore be accepted even when official PySpark rejects
> their class.
> On macOS arm64, np.longdouble has an eight-byte dtype equal to float64 but
> remains a distinct scalar class. Official PySpark rejects that unsupported
> class with a bare AssertionError after DoubleType inference. Distinct
> np.longlong aliases with supported integer dtypes require the corresponding
> class check on platforms where that class is not accepted. Wider unsupported
> longdouble dtypes retain their structured PySparkTypeError behavior.
> The proposed implementation in
> https://github.com/apache/spark-connect-rust/pull/381 applies numeric
> scalar-class checks after dtype inference for scalar and array literals,
> while preserving canonical supported classes, literal types and structured
> unsupported-dtype errors. Regression expectations follow the reference
> behavior for the actual platform, including the eight-byte longdouble case.
> The earlier baseline failure remains recorded. This issue retains
> connect-rust-4.3.0 release tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
