uros-b commented on code in PR #56557:
URL: https://github.com/apache/spark/pull/56557#discussion_r3431522112
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:
##########
@@ -752,6 +797,73 @@ private[hive] trait HiveInspectors {
}
}
+ /**
+ * Returns an unwrapper that converts a Hive value into a Catalyst value,
using the target
+ * Catalyst `dataType` to preserve nanosecond timestamp precision. The plain
+ * `unwrapperFor(ObjectInspector)` cannot do this because a Hive
`TimestampObjectInspector`
+ * maps to micros by default; here the nanos timestamp types are produced as
`TimestampNanosVal`,
+ * recursing through array/map/struct so nested nanos timestamps round-trip
correctly. Any other
+ * type is delegated to the `ObjectInspector`-only overload.
+ */
+ def unwrapperFor(objectInspector: ObjectInspector, dataType: DataType): Any
=> Any =
+ (objectInspector, dataType) match {
+ case (ti: TimestampObjectInspector, t: TimestampNTZNanosType) =>
+ data: Any => {
+ if (data != null) {
+ DateTimeUtils.localDateTimeToTimestampNanos(
+ ti.getPrimitiveJavaObject(data).toLocalDateTime, t.precision)
+ } else {
+ null
+ }
+ }
+ case (ti: TimestampObjectInspector, t: TimestampLTZNanosType) =>
+ data: Any => {
+ if (data != null) {
+ DateTimeUtils.instantToTimestampNanos(
+ ti.getPrimitiveJavaObject(data).toInstant, t.precision)
+ } else {
+ null
+ }
+ }
+ case (li: ListObjectInspector, ArrayType(elementType, _)) =>
Review Comment:
Small duplication in Hive OrcFileFormat unwrappers: Top-level
TimestampObjectInspector fields get inline nanos handling, while nested fields
use unwrapperFor(oi, dataType). Functionally fine, but you could simplify to
always call unwrapperFor(fieldRef.getFieldObjectInspector, field.dataType) and
drop the duplicated top-level branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
