[
https://issues.apache.org/jira/browse/SPARK-61286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61286:
-----------------------------------
Labels: pull-request-available (was: )
> Send date, timestamp and decimal pivot values as typed literals in
> GroupedData::pivot of the Rust API
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-61286
> URL: https://issues.apache.org/jira/browse/SPARK-61286
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> GroupedData::pivot of the Rust API makes a literal of each value with
> value_to_lit_expr (crates/spark-connect/src/group.rs:17). A boolean, a
> number, a string or a day-time interval becomes a typed literal, but any
> other Value, such as Value::Date, Value::Timestamp or Value::Decimal, becomes
> the string of its Debug format (group.rs:30). Value::Date(19723), 2024-01-01,
> is sent as the string "Date(19723)".
> {code:rust}
> // df has an int column k and a date column d.
> let counted = df
> .group_by(vec![col("k")])
> .pivot(col("d"), Some(vec![Value::Date(19723)]))?
> .count();
> {code}
> The server casts the string to the type of the pivot column. Against an OSS
> Spark 4.3.0 Connect server on loopback, with the pivot value "Date(19723)" on
> a date column:
> * spark.sql.ansi.enabled true, the default: the query fails with
> \[CAST_INVALID_INPUT\] The value 'Date(19723)' of the type "STRING" cannot be
> cast to "DATE" because it is malformed.
> * spark.sql.ansi.enabled false: a column named Date(19723) that is null in
> every row, without an error.
> Sent as a date literal, the value would match the rows of 2024-01-01.
> PySpark's pivot rejects date, timestamp and Decimal values with
> NOT_EXPECTED_TYPE, and so has the Python client of spark-connect-rust since
> SPARK-61046, but the Rust API takes them. At spark-connect-rust master
> 8768cc4.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
