HyukjinKwon commented on code in PR #110:
URL:
https://github.com/apache/spark-connect-rust/pull/110#discussion_r4405482566
##########
crates/spark-connect/src/row.rs:
##########
@@ -34,6 +34,10 @@ pub enum Value {
Date(i32),
/// Timestamp (microseconds since epoch)
Timestamp(i64),
+ /// Spark day-time interval, in signed microseconds. The full i64 range is
+ /// representable by Python's datetime.timedelta without nanosecond
conversion.
+ /// Display (including Rust DataFrame::show) uses `<value> microseconds`.
+ DayTimeInterval(i64),
Review Comment:
`GroupedData::pivot` in the Rust API doesn't handle this new variant yet.
Pivot values go through `value_to_lit_expr`
(crates/spark-connect/src/group.rs:17-32). Its wildcard `other =>
L::string(format!("{:?}", other))` has no interval case, and
`LiteralExpression` has no day-time interval variant to lower to. So a value
collected from an interval column, e.g.
`Value::DayTimeInterval(86_400_000_000)`, is sent to Spark as the string
literal `'DayTimeInterval(86400000000)'`.
On the server, pivot resolution casts each value to the pivot column's type,
and the string-to-day-time-interval cast rejects that text. In apache/spark
master, `PivotTransformer` evaluates the `Cast`, and
`IntervalUtils.castStringToDTInterval` has no null-on-error path. So
`df.group_by(...).pivot(col("d"), Some(vals))` fails during analysis with an
illegal interval format error that quotes the client's Debug encoding.
This PR already encodes the same value as a typed
`LiteralType::DayTimeInterval` for NA fill. Pivot is the remaining
`Value`-to-literal path that still falls back to Debug text. The PySpark
drop-in isn't affected: its `pivot` converts values with `py_to_value`, which
rejects `datetime.timedelta` with a `TypeError`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
