Hyukjin Kwon created SPARK-61320:
------------------------------------
Summary: Rust pivot turns date, timestamp, decimal and null values
into text, and dropna accepts any how
Key: SPARK-61320
URL: https://issues.apache.org/jira/browse/SPARK-61320
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The Rust client's GroupedData.pivot sent every pivot value that is not a bool,
number or string as the text of its Debug form: a date pivot value became
"Date(18262)", which the server rejects (CAST_INVALID_INPUT), and a NULL became
the string "Null", which pivots a string column silently. DataFrame.dropna
accepted any how and treated every value other than "all" as "any", where
PySpark raises VALUE_NOT_ALLOWED. Repro: pivot("dt", [Value::Date(18262)]) and
dropna(Some("bogus"), None, None) on the Rust API. Official PySpark 4.3.0
rejects a NULL pivot value (NOT_EXPECTED_TYPE) and an invalid how
(VALUE_NOT_ALLOWED).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
