Tim Meehan created SPARK-61286:
----------------------------------
Summary: Send date, timestamp and decimal pivot values as typed
literals in GroupedData::pivot of the Rust API
Key: SPARK-61286
URL: https://issues.apache.org/jira/browse/SPARK-61286
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
GroupedData::pivot of the Rust API makes a literal of each value with
value_to_lit_expr (crates/spark-connect/src/group.rs:17). A boolean, a number,
a string or a day-time interval becomes a typed literal, but any other Value,
such as Value::Date, Value::Timestamp or Value::Decimal, becomes the string of
its Debug format (group.rs:30). Value::Date(19723), 2024-01-01, is sent as the
string "Date(19723)".
{code:rust}
// df has an int column k and a date column d.
let counted = df
.group_by(vec![col("k")])
.pivot(col("d"), Some(vec![Value::Date(19723)]))?
.count();
{code}
The server casts the string to the type of the pivot column. Against an OSS
Spark 4.3.0 Connect server on loopback, with the pivot value "Date(19723)" on a
date column:
* spark.sql.ansi.enabled true, the default: the query fails with
\[CAST_INVALID_INPUT\] The value 'Date(19723)' of the type "STRING" cannot be
cast to "DATE" because it is malformed.
* spark.sql.ansi.enabled false: a column named Date(19723) that is null in
every row, without an error.
Sent as a date literal, the value would match the rows of 2024-01-01. PySpark's
pivot rejects date, timestamp and Decimal values with NOT_EXPECTED_TYPE, and so
has the Python client of spark-connect-rust since SPARK-61046, but the Rust API
takes them. At spark-connect-rust master 8768cc4.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
