[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20774 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175335261
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -479,6 +479,15 @@ object SQLConf {
.checkValues(HiveCaseSensitiveInferenceMode.values.map(_.toString))
.createWithDefault(HiveCaseSensitiveInferenceMode.INFER_AND_SAVE.toString)
+ val HIVE_COMPARE_DATE_TIMESTAMP_IN_TIMESTAMP =
+buildConf("spark.sql.hive.compareDateTimestampInTimestamp")
--- End diff --
`spark.sql.typeCoercion.compareDateTimestampInTimestamp`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175335072
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -479,6 +479,15 @@ object SQLConf {
.checkValues(HiveCaseSensitiveInferenceMode.values.map(_.toString))
.createWithDefault(HiveCaseSensitiveInferenceMode.INFER_AND_SAVE.toString)
+ val HIVE_COMPARE_DATE_TIMESTAMP_IN_TIMESTAMP =
+buildConf("spark.sql.hive.compareDateTimestampInTimestamp")
+ .doc("When true (default), compare Date with Timestamp after
converting both sides to " +
+"Timestamp. This behavior is compatible with Hive 2.2 or later.
See HIVE-15236. " +
+"When false, restore the behavior prior to Spark 2.4. Compare Date
with Timestamp after " +
+"converting both sides to string.")
+.booleanConf
--- End diff --
perhaps mention this config will be removed in spark 3.0.
(on a related note we should look at those configs for backward
compatibility and consider removing them in 3.0)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175334948
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/predicate-functions.sql ---
@@ -39,3 +43,4 @@ select 2.0 <= '2.2';
select 0.5 <= '1.5';
select to_date('2009-07-30 04:17:52') <= to_date('2009-07-30 04:17:52');
select to_date('2009-07-30 04:17:52') <= '2009-07-30 04:17:52';
+select to_date('2017-03-01') <= to_timestamp('2017-03-01 00:00:01');
--- End diff --
+1 it is really confusing to look at the diff
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175333760
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -479,6 +479,15 @@ object SQLConf {
.checkValues(HiveCaseSensitiveInferenceMode.values.map(_.toString))
.createWithDefault(HiveCaseSensitiveInferenceMode.INFER_AND_SAVE.toString)
+ val HIVE_COMPARE_DATE_TIMESTAMP_IN_TIMESTAMP =
+buildConf("spark.sql.hive.compareDateTimestampInTimestamp")
+ .doc("When true (default), compare Date with Timestamp after
converting both sides to " +
+"Timestamp. This behavior is compatible with Hive 2.2 or later.
See HIVE-15236. " +
+"When false, restore the behavior prior to Spark 2.4. Compare Date
with Timestamp after " +
+"converting both sides to string.")
+.booleanConf
--- End diff --
internal()?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175333714
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -479,6 +479,15 @@ object SQLConf {
.checkValues(HiveCaseSensitiveInferenceMode.values.map(_.toString))
.createWithDefault(HiveCaseSensitiveInferenceMode.INFER_AND_SAVE.toString)
+ val HIVE_COMPARE_DATE_TIMESTAMP_IN_TIMESTAMP =
+buildConf("spark.sql.hive.compareDateTimestampInTimestamp")
--- End diff --
It should not be under `hive`, because it is not only for hive
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175332253
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -342,8 +349,8 @@ object TypeCoercion {
p.makeCopy(Array(left, Cast(right, TimestampType)))
case p @ BinaryComparison(left, right)
-if findCommonTypeForBinaryComparison(left.dataType,
right.dataType).isDefined =>
-val commonType = findCommonTypeForBinaryComparison(left.dataType,
right.dataType).get
+if findCommonTypeForBinaryComparison(left.dataType,
right.dataType, conf).isDefined =>
--- End diff --
two more spaces before `if`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175332620
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala
---
@@ -1251,18 +1251,40 @@ class TypeCoercionSuite extends AnalysisTest {
}
test("binary comparison with string promotion") {
-ruleTest(PromoteStrings,
- GreaterThan(Literal("123"), Literal(1)),
- GreaterThan(Cast(Literal("123"), IntegerType), Literal(1)))
-ruleTest(PromoteStrings,
- LessThan(Literal(true), Literal("123")),
- LessThan(Literal(true), Cast(Literal("123"), BooleanType)))
-ruleTest(PromoteStrings,
- EqualTo(Literal(Array(1, 2)), Literal("123")),
- EqualTo(Literal(Array(1, 2)), Literal("123")))
-ruleTest(PromoteStrings,
- GreaterThan(Literal("1.5"), Literal(BigDecimal("0.5"))),
- GreaterThan(Cast(Literal("1.5"), DoubleType),
Cast(Literal(BigDecimal("0.5")), DoubleType)))
+val rule = TypeCoercion.PromoteStrings(conf)
+
+Seq("true", "false").foreach { flag =>
+ withSQLConf("spark.sql.hive.compareDateTimestampInTimestamp" ->
flag) {
+ruleTest(rule,
+ GreaterThan(Literal("123"), Literal(1)),
+ GreaterThan(Cast(Literal("123"), IntegerType), Literal(1)))
+ruleTest(rule,
+ LessThan(Literal(true), Literal("123")),
+ LessThan(Literal(true), Cast(Literal("123"), BooleanType)))
+ruleTest(rule,
+ EqualTo(Literal(Array(1, 2)), Literal("123")),
+ EqualTo(Literal(Array(1, 2)), Literal("123")))
+ruleTest(rule,
+ GreaterThan(Literal("1.5"), Literal(BigDecimal("0.5"))),
+ GreaterThan(Cast(Literal("1.5"), DoubleType),
Cast(Literal(BigDecimal("0.5")),
+DoubleType)))
+val date0301 = Literal(java.sql.Date.valueOf("2017-03-01"))
+val timestamp030100 = Literal(Timestamp.valueOf("2017-03-01
00:00:00"))
+val timestamp030101 = Literal(Timestamp.valueOf("2017-03-01
00:00:01"))
+if ("true".equals(flag)) {
+ // `Date` should be treated as timestamp at 00:00:00 See
SPARK-23549
+ ruleTest(rule, EqualTo(date0301, timestamp030100),
+EqualTo(Cast(date0301, TimestampType), timestamp030100))
+ ruleTest(rule, LessThan(date0301, timestamp030101),
+LessThan(Cast(date0301, TimestampType), timestamp030101))
+} else {
+ ruleTest(rule, LessThan(date0301, timestamp030100),
+LessThan(Cast(date0301, StringType), Cast(timestamp030100,
StringType)))
+ ruleTest(rule, LessThan(date0301, timestamp030101),
+LessThan(Cast(date0301, StringType), Cast(timestamp030101,
StringType)))
+}
+ }
+}
--- End diff --
```Scala
test("binary comparison with string promotion") {
val rule = TypeCoercion.PromoteStrings(conf)
ruleTest(rule,
GreaterThan(Literal("123"), Literal(1)),
GreaterThan(Cast(Literal("123"), IntegerType), Literal(1)))
ruleTest(rule,
LessThan(Literal(true), Literal("123")),
LessThan(Literal(true), Cast(Literal("123"), BooleanType)))
ruleTest(rule,
EqualTo(Literal(Array(1, 2)), Literal("123")),
EqualTo(Literal(Array(1, 2)), Literal("123")))
ruleTest(rule,
GreaterThan(Literal("1.5"), Literal(BigDecimal("0.5"))),
GreaterThan(Cast(Literal("1.5"), DoubleType),
Cast(Literal(BigDecimal("0.5")),
DoubleType)))
Seq(true, false).foreach { convertToTS =>
withSQLConf("spark.sql.hive.compareDateTimestampInTimestamp" ->
convertToTS.toString) {
val date0301 = Literal(java.sql.Date.valueOf("2017-03-01"))
val timestamp030100 = Literal(Timestamp.valueOf("2017-03-01
00:00:00"))
val timestamp030101 = Literal(Timestamp.valueOf("2017-03-01
00:00:01"))
if (convertToTS) {
// `Date` should be treated as timestamp at 00:00:00 See
SPARK-23549
ruleTest(rule, EqualTo(date0301, timestamp030100),
EqualTo(Cast(date0301, TimestampType), timestamp030100))
ruleTest(rule, LessThan(date0301, timestamp030101),
LessThan(Cast(date0301, TimestampType), timestamp030101))
} else {
ruleTest(rule, LessThan(date0301, timestamp030100),
LessThan(Cast(date0301, StringType), Cast(timestamp030100,
StringType)))
ruleTest(rule, LessThan(date0301, timestamp030101),
LessThan(Cast(date0301, StringType), Cast(timestamp030101,
StringType)))
}
}
}
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175332237
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -125,29 +125,36 @@ object TypeCoercion {
/**
* This function determines the target type of a comparison operator
when one operand
* is a String and the other is not. It also handles when one op is a
Date and the
- * other is a Timestamp by making the target type to be String.
+ * other is a Timestamp by making the target type to be Timestamp as
default
*/
- val findCommonTypeForBinaryComparison: (DataType, DataType) =>
Option[DataType] = {
+ private def findCommonTypeForBinaryComparison(dt1: DataType, dt2:
DataType, conf: SQLConf):
+ Option[DataType] = (dt1, dt2) match {
--- End diff --
```Scala
private def findCommonTypeForBinaryComparison(
dt1: DataType, dt2: DataType, conf: SQLConf): Option[DataType] =
(dt1, dt2) match {
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r175332723
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/predicate-functions.sql ---
@@ -39,3 +43,4 @@ select 2.0 <= '2.2';
select 0.5 <= '1.5';
select to_date('2009-07-30 04:17:52') <= to_date('2009-07-30 04:17:52');
select to_date('2009-07-30 04:17:52') <= '2009-07-30 04:17:52';
+select to_date('2017-03-01') <= to_timestamp('2017-03-01 00:00:01');
--- End diff --
A little hard to track the exact changes. Let us move all the new tests to
the end.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20774#discussion_r175332743 --- Diff: docs/sql-programming-guide.md --- @@ -2289,6 +2289,13 @@ Spark SQL supports the vast majority of Hive features, such as: * `MAP<>` * `STRUCT<>` +### Compare a DATE type with a TIMESTAMP type --- End diff -- We also need to add it to migration guide. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173730165
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
--- End diff --
I think so, but @dongjoon-hyun 's comment said it is fixed at 2.0.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173718706
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
--- End diff --
Seems it's fixed in Hive 2.2.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173715303
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
--- End diff --
Thanks
@dongjoon-hyun are you pointing out [this HIVE JIRA
entry|https://issues.apache.org/jira/browse/HIVE-15236] in [this
comment](https://issues.apache.org/jira/browse/SPARK-23549?focusedCommentId=16391877=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16391877)?
Or other HIVE JIRA entries?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173710678
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
--- End diff --
Hive 2.1.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173710174
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
--- End diff --
Thank you for your try.
The current implementation follows [this
answer|https://issues.apache.org/jira/browse/SPARK-23549?focusedCommentId=16391351=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16391351].
[Which version of
hive|https://issues.apache.org/jira/browse/SPARK-23549?focusedCommentId=16391877=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16391877]
did you use?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173709979
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
+case (TimestampType, DateType) => Some(TimestampType)
--- End diff --
I see. I will add a SQLConf. Is there any suggestion for a name of this
entry?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173708865
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
--- End diff --
So it means `TimeStamp('2017-03-01 00:00:01') eq Date('2017-03-01') =
false`? Looks a bit weird.
At least this comparison does not get true in Hive:
```sql
hive> select cast('2017-03-01 00:00:00' as timestamp) = cast('2017-03-01'
as date);
OK
false
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20774#discussion_r173701347
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -135,11 +135,14 @@ object TypeCoercion {
case (DateType, StringType) => Some(StringType)
case (StringType, TimestampType) => Some(StringType)
case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+// Cast to TimestampType when we compare DateType with TimestampType
+// i.e. TimeStamp('2017-03-01 00:00:00') eq Date('2017-03-01') = true
+case (TimestampType, DateType) => Some(TimestampType)
--- End diff --
Although this PR makes sense, we still need a SQLConf. Also add this into
the migration guide in the SQL doc
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20774: [SPARK-23549][SQL] Cast to timestamp when compari...
GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/20774
[SPARK-23549][SQL] Cast to timestamp when comparing timestamp with date
## What changes were proposed in this pull request?
This PR fixes an incorrect comparison in SQL between timestamp and date.
This is because both of them are casted to `string` and then are compared
lexicographically. This implementation shows `false` regarding this query
`spark.sql("select cast('2017-03-01 00:00:00' as timestamp) between
cast('2017-02-28' as date) and cast('2017-03-01' as date)").show`.
This PR shows `true` for this query by casting `date("2017-03-01")` to
`timestamp("2017-03-01 00:00:00")`.
(Please fill in changes proposed in this fix)
## How was this patch tested?
Added new UTs to `TypeCoercionSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-23549
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20774.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20774
commit 21368b4c709045c41eb782408e134cea6037689c
Author: Kazuaki Ishizaki
Date: 2018-03-08T18:19:32Z
initial commit
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
