This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f088c28 [SPARK-32594][SQL][FOLLOWUP][TEST-HADOOP2.7][TEST-HIVE1.2]
Override `get()` and use Julian days in `DaysWritable`
f088c28 is described below
commit f088c28a53571afe5146100fd2e76c2b5ec92862
Author: Max Gekk <[email protected]>
AuthorDate: Sun Aug 23 12:43:30 2020 -0700
[SPARK-32594][SQL][FOLLOWUP][TEST-HADOOP2.7][TEST-HIVE1.2] Override `get()`
and use Julian days in `DaysWritable`
### What changes were proposed in this pull request?
Override `def get: Date` in `DaysWritable` use the `daysToMillis(int d)`
from the parent class `DateWritable` instead of `long daysToMillis(int d,
boolean doesTimeMatter)`.
### Why are the changes needed?
It fixes failures of `HiveSerDeReadWriteSuite` with the profile `hive-1.2`.
In that case, the parent class `DateWritable` has different implementation
before the commit to Hive
https://github.com/apache/hive/commit/da3ed68eda10533f3c50aae19731ac6d059cda87.
In particular, `get()` calls `new Date(daysToMillis(daysSinceEpoch))` instead
of overrided `def get(doesTimeMatter: Boolean): Date` in the child class. The
`get()` method returns wrong result `1970-01-01` because it uses not updated
[...]
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
By running the test suite `HiveSerDeReadWriteSuite`:
```
$ build/sbt -Phive-1.2 -Phadoop-2.7 "test:testOnly
org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite"
```
and
```
$ build/sbt -Phive-2.3 -Phadoop-2.7 "test:testOnly
org.apache.spark.sql.hive.execution.HiveSerDeReadWriteSuite"
```
Closes #29523 from MaxGekk/insert-date-into-hive-table-1.2.
Authored-by: Max Gekk <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
(cherry picked from commit 1c798f973fa8307cc1f15eec067886e8e9aecb59)
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
.../org/apache/spark/sql/execution/datasources/DaysWritable.scala | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
index 56c176e..a04c2fc 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
@@ -54,6 +54,9 @@ class DaysWritable(
}
override def getDays: Int = julianDays
+ override def get: Date = {
+ new Date(DateWritable.daysToMillis(julianDays))
+ }
override def get(doesTimeMatter: Boolean): Date = {
new Date(DateWritable.daysToMillis(julianDays, doesTimeMatter))
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
