MaxGekk commented on a change in pull request #27807: [SPARK-31076][SQL]
Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
URL: https://github.com/apache/spark/pull/27807#discussion_r389758150
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
##########
@@ -618,7 +617,7 @@ private[hive] trait HiveInspectors {
case x: DateObjectInspector if x.preferWritable() =>
data: Any => {
if (data != null) {
-
DateTimeUtils.fromJavaDate(x.getPrimitiveWritableObject(data).get())
+ x.getPrimitiveWritableObject(data).getDays
Review comment:
For consistency, we should rebase them but round trip conversions don't work
correctly:
```scala
hiveIo.DateWritable(DateTimeUtils.toJavaDate(value.asInstanceOf[Int]))
```
```scala
DateTimeUtils.fromJavaDate(x.getPrimitiveWritableObject(data).get())
```
The bug is either in Spark's toJavaDate/fromJavaDate or in Hive's
DateWritable. Hive does some heavy/dangerous conversions
https://github.com/apache/hive/blob/beb57c6e43586ec8be770975a009b955c8a9840e/storage-api/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java#L144-L184
. I would prefer to avoid it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
