GitHub user DazhuangSu opened a pull request:
https://github.com/apache/spark/pull/19691
[SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITION should support
comparators
## What changes were proposed in this pull request?
This pr is inspired by @dongjoon-hyun.
quote from https://github.com/apache/spark/pull/15704 :
> **What changes were proposed in this pull request?**
This PR aims to support comparators, e.g. '<', '<=', '>', '>=', again
in Apache Spark 2.0 for backward compatibility.
**Spark 1.6**
`scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country
STRING, quarter STRING)")
res0: org.apache.spark.sql.DataFrame = [result: string]`
`scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")
res1: org.apache.spark.sql.DataFrame = [result: string]`
**Spark 2.0**
`scala> sql("CREATE TABLE sales(id INT) PARTITIONED BY (country
STRING, quarter STRING)")
res0: org.apache.spark.sql.DataFrame = []`
`scala> sql("ALTER TABLE sales DROP PARTITION (country < 'KR')")`
`org.apache.spark.sql.catalyst.parser.ParseException:`
`mismatched input '<' expecting {')', ','}(line 1, pos 42)`
After this PR, it's supported.
**How was this patch tested?**
Pass the Jenkins test with a newly added testcase.
https://github.com/apache/spark/pull/16036 points out that if we use int
literal in DROP PARTITION will fail after patching
https://github.com/apache/spark/pull/15704.
The reason of this failing in https://github.com/apache/spark/pull/15704 is
that AlterTableDropPartitionCommand tells BinayComparison and EqualTo with
following code:
`private def isRangeComparison(expr: Expression): Boolean = {⨠`
`expr.find(e => e.isInstanceOf[BinaryComparison] &&
!e.isInstanceOf[EqualTo]).isDefinedâ¨}`
This PR resolve this problem by telling a drop condition when parsing sqls.
## How was this patch tested?
New testcase introduced from https://github.com/apache/spark/pull/15704
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DazhuangSu/spark SPARK-17732
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19691.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19691
----
commit 20f658ad8e14a94dd23bff6a8d795124d1db24e9
Author: Dylan Su <[email protected]>
Date: 2017-11-08T03:44:28Z
[SPARK-14922][SPARK-17732][SQL]ALTER TABLE DROP PARTITION should support
comparators
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
