dongjoon-hyun opened a new pull request, #59652:
URL: https://github.com/apache/spark/pull/59652
### What changes were proposed in this pull request?
This PR aims to infer ORC `TIMESTAMP` columns as `TIMESTAMP_LTZ` regardless
of `spark.sql.timestampType`.
`OrcUtils.toCatalystSchema` parses the type name `timestamp` with
`CatalystSqlParser`, which resolves it by `spark.sql.timestampType`.
`timestamp` is both the ORC type name of a `TIMESTAMP` column and the Spark
type attribute of a `TIMESTAMP_LTZ` column. This PR maps them to
`TimestampType` directly.
### Why are the changes needed?
With `spark.sql.timestampType=TIMESTAMP_NTZ`, Spark returns wrong values
from ORC `TIMESTAMP` columns.
```
$ bin/spark-shell -c spark.sql.session.timeZone=UTC
scala> sql("SELECT TIMESTAMP_LTZ'1970-01-01 00:00:01' AS
ts").write.orc("/tmp/orc_ts")
scala> spark.conf.set("spark.sql.timestampType", "TIMESTAMP_NTZ")
scala> spark.read.orc("/tmp/orc_ts").collect()
val res2: Array[org.apache.spark.sql.Row] =
Array([1970-01-01T00:00:00.000001])
```
The column is inferred as `TIMESTAMP_NTZ` and 1 second is read as 1
microsecond. Spark reads `TIMESTAMP_NTZ` from ORC `bigint` (SPARK-37463), and
ORC converts `TIMESTAMP` values to seconds for it.
### Does this PR introduce _any_ user-facing change?
Yes, this fixes the wrong query results since Apache Spark 3.4.0
([v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07)).
With `spark.sql.timestampType=TIMESTAMP_NTZ`, ORC `TIMESTAMP` columns are
inferred as `TIMESTAMP_LTZ`, and reading them as `TIMESTAMP_NTZ` fails with
`UNSUPPORTED_FEATURE.ORC_TYPE_CAST` as with the default
`spark.sql.timestampType`.
### How was this patch tested?
Pass the CIs with the newly added test case in `OrcSourceV1Suite` and
`OrcSourceV2Suite`. The test case fails without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
