[
https://issues.apache.org/jira/browse/SPARK-61320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61320:
-----------------------------------
Labels: pull-request-available (was: )
> Rust pivot turns date, timestamp, decimal and null values into text, and
> dropna accepts any how
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-61320
> URL: https://issues.apache.org/jira/browse/SPARK-61320
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust client's GroupedData.pivot sent every pivot value that is not a
> bool, number or string as the text of its Debug form: a date pivot value
> became "Date(18262)", which the server rejects (CAST_INVALID_INPUT), and a
> NULL became the string "Null", which pivots a string column silently.
> DataFrame.dropna accepted any how and treated every value other than "all" as
> "any", where PySpark raises VALUE_NOT_ALLOWED. Repro: pivot("dt",
> [Value::Date(18262)]) and dropna(Some("bogus"), None, None) on the Rust API.
> Official PySpark 4.3.0 rejects a NULL pivot value (NOT_EXPECTED_TYPE) and an
> invalid how (VALUE_NOT_ALLOWED).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
