[spark] branch branch-3.0 updated: [SPARK-32594][SQL][FOLLOWUP][TEST-HADOOP2.7][TEST-HIVE1.2] Override `get()` and use Julian days in `DaysWritable`

[viirya]

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
     new f088c28  [SPARK-32594][SQL][FOLLOWUP][TEST-HADOOP2.7][TEST-HIVE1.2] 
Override `get()` and use Julian days in `DaysWritable`
f088c28 is described below

commit f088c28a53571afe5146100fd2e76c2b5ec92862
Author: Max Gekk <max.g...@gmail.com>
AuthorDate: Sun Aug 23 12:43:30 2020 -0700

    [SPARK-32594][SQL][FOLLOWUP][TEST-HADOOP2.7][TEST-HIVE1.2] Override `get()` 
and use Julian days in `DaysWritable`
    
    ### What changes were proposed in this pull request?
    Override `def get: Date` in `DaysWritable` use the `daysToMillis(int d)` 
from the parent class `DateWritable` instead of `long daysToMillis(int d, 
boolean doesTimeMatter)`.
    
    ### Why are the changes needed?
    It fixes failures of `HiveSerDeReadWriteSuite` with the profile `hive-1.2`. 
In that case, the parent class `DateWritable` has different implementation 
before the commit to Hive 
https://github.com/apache/hive/commit/da3ed68eda10533f3c50aae19731ac6d059cda87. 
In particular, `get()` calls `new Date(daysToMillis(daysSinceEpoch))` instead 
of overrided `def get(doesTimeMatter: Boolean): Date` in the child class. The 
`get()` method returns wrong result `1970-01-01` because it uses not updated 
[...]
    
    ### Does this PR introduce _any_ user-facing change?
    Yes.
    
    ### How was this patch tested?
    By running the test suite `HiveSerDeReadWriteSuite`:
    ```
    $ build/sbt -Phive-1.2 -Phadoop-2.7 "test:testOnly 
org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite"
    ```
    and
    ```
    $ build/sbt -Phive-2.3 -Phadoop-2.7 "test:testOnly 
org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite"
    ```
    
    Closes #29523 from MaxGekk/insert-date-into-hive-table-1.2.
    
    Authored-by: Max Gekk <max.g...@gmail.com>
    Signed-off-by: Liang-Chi Hsieh <vii...@gmail.com>
    (cherry picked from commit 1c798f973fa8307cc1f15eec067886e8e9aecb59)
    Signed-off-by: Liang-Chi Hsieh <vii...@gmail.com>
---
 .../org/apache/spark/sql/execution/datasources/DaysWritable.scala      | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
index 56c176e..a04c2fc 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
@@ -54,6 +54,9 @@ class DaysWritable(
   }
 
   override def getDays: Int = julianDays
+  override def get: Date = {
+    new Date(DateWritable.daysToMillis(julianDays))
+  }
   override def get(doesTimeMatter: Boolean): Date = {
     new Date(DateWritable.daysToMillis(julianDays, doesTimeMatter))
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org