andygrove opened a new issue, #96:
URL: https://github.com/apache/datafusion-spark/issues/96
`quote` of a TIMESTAMP separates the date and time with `T`. Spark uses a
space:
```sql
SELECT quote(CAST('2024-01-15T12:30:45' AS TIMESTAMP));
-- datafusion-spark: '2024-01-15T12:30:45'
-- Spark 4.2.0: '2024-01-15 12:30:45'
```
The test in `crates/sqllogictest/test_files/string/quote.slt:125`
(`quote(arrow_cast('2024-01-15T12:30:45', 'Timestamp(µs)'))`) expects the `T`.
I found this by running the file on PySpark 4.2.0 with the runner from #54. The
DATE case on the line above it matches Spark.
### Cause
Spark's
[`Quote`](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L3764-L3780)
takes a STRING with `ImplicitCastInputTypes`, so the analyzer casts a
TIMESTAMP with Spark's `CAST(... AS STRING)`, which prints `yyyy-MM-dd
HH:mm:ss[.fraction]`.
`SparkQuote` asks DataFusion's type coercion for a string, so the timestamp
is cast to `Utf8` with Arrow's cast, which prints ISO 8601 with a `T`.
### Scope
This is not specific to `quote`. Any function in this crate whose signature
coerces a TIMESTAMP, or another non-string type, to a string uses Arrow's
formatting rather than Spark's. The cast port (#88) adds Spark's
timestamp-to-string formatting. Once it lands, functions could coerce through
the Spark cast rather than Arrow's. Until then, `quote` could special-case
timestamp input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
