MaxGekk commented on code in PR #56557:
URL: https://github.com/apache/spark/pull/56557#discussion_r3431826121
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:
##########
@@ -752,6 +797,73 @@ private[hive] trait HiveInspectors {
}
}
+ /**
+ * Returns an unwrapper that converts a Hive value into a Catalyst value,
using the target
+ * Catalyst `dataType` to preserve nanosecond timestamp precision. The plain
+ * `unwrapperFor(ObjectInspector)` cannot do this because a Hive
`TimestampObjectInspector`
+ * maps to micros by default; here the nanos timestamp types are produced as
`TimestampNanosVal`,
+ * recursing through array/map/struct so nested nanos timestamps round-trip
correctly. Any other
+ * type is delegated to the `ObjectInspector`-only overload.
+ */
+ def unwrapperFor(objectInspector: ObjectInspector, dataType: DataType): Any
=> Any =
+ (objectInspector, dataType) match {
+ case (ti: TimestampObjectInspector, t: TimestampNTZNanosType) =>
+ data: Any => {
+ if (data != null) {
+ DateTimeUtils.localDateTimeToTimestampNanos(
+ ti.getPrimitiveJavaObject(data).toLocalDateTime, t.precision)
+ } else {
+ null
+ }
+ }
+ case (ti: TimestampObjectInspector, t: TimestampLTZNanosType) =>
+ data: Any => {
+ if (data != null) {
+ DateTimeUtils.instantToTimestampNanos(
+ ti.getPrimitiveJavaObject(data).toInstant, t.precision)
+ } else {
+ null
+ }
+ }
+ case (li: ListObjectInspector, ArrayType(elementType, _)) =>
Review Comment:
Done in 8c7a4640242. Dropped the inline top-level `TimestampObjectInspector`
branch; all nanos timestamps (top-level and nested) now go through
`unwrapperFor(oi, dataType)`, while non-nanos fields keep the `HiveStructField`
fast paths (`setInt`/`setLong`/...), so this avoids regressing the primitive
paths while removing the duplication.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:
##########
@@ -291,6 +291,8 @@ object OrcUtils extends Logging {
case m: MapType =>
s"map<${getOrcSchemaString(m.keyType)},${getOrcSchemaString(m.valueType)}>"
case _: DayTimeIntervalType | _: TimestampNTZType | _: TimeType =>
LongType.catalogString
+ case _: TimestampLTZNanosType => "timestamp with local time zone"
Review Comment:
Good catch — addressed in 8c7a4640242. `checkTimestampCompatibility` now
also rejects nanos-involved mismatches (cross-zone, or micros<->nanos) with a
clear `UNSUPPORTED_FEATURE.ORC_TYPE_CAST` error; nanos of the same kind
(NTZ/LTZ) still coerce by precision since they share an ORC physical category.
Added a `QueryExecutionErrorsSuite` case reading a `timestamp_ntz(9)` file
against a micros `TIMESTAMP_NTZ` schema.
##########
sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala:
##########
@@ -1394,6 +1395,31 @@ class FileBasedDataSourceSuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-57166: ORC supports nanosecond timestamp types in v1 and v2") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ // Validate both v1 and v2 ORC paths.
+ Seq(true, false).foreach { useV1 =>
+ val useV1List = if (useV1) "orc" else ""
+ withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> useV1List) {
+ foreachNanosPrecision { precision =>
+ Seq(TimestampNTZNanosType(precision),
TimestampLTZNanosType(precision)).foreach {
+ nanosType =>
+ withTempDir { dir =>
+ val nanosLiteral = Literal.create(new TimestampNanosVal(0L,
0.toShort), nanosType)
Review Comment:
Aligned in 8c7a4640242 — the positive ORC test now builds rows from external
`java.time` values (matching `nanosTimestampDf`) instead of `TimestampNanosVal`
+ `Literal`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
