GitHub user huaxingao opened a pull request:
https://github.com/apache/spark/pull/14535
[SPARK-16946][SQL]throw Exception if saveAsTable[apend] has differentâ¦
## What changes were proposed in this pull request?
In HiveContext, if saveAsTable[append] has different number of columns,
Spark will throw Exception.
e.g.
```
test("saveAsTable[append]: too many columns") {
withTable("saveAsTable_too_many_columns") {
Seq((1, 2)).toDF("i",
"j").write.saveAsTable("saveAsTable_too_many_columns")
val e = intercept[AnalysisException] {
Seq((3, 4, 5)).toDF("i", "j",
"k").write.mode("append").saveAsTable("saveAsTable_too_many_columns")
}
assert(e.getMessage.contains("doesn't match"))
}
}
```
However, in SparkSession or SQLContext, if use the above code example, the
extra column in the append data will be removed silently without any warning or
Exception. The table becomes
```
i j
3 4
1 2
```
I changed the code to throw Exception if saveAsTable[apend] has different
column numbers
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
Add unit tests to test the patch.
⦠number of columns
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/huaxingao/spark spark-16946
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14535.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14535
----
commit 52e7b66e1ff335566af92c795c1679c11e0ab8cb
Author: Huaxin Gao <huax...@us.ibm.com>
Date: 2016-08-08T06:06:58Z
[SPARK-16946][SQL]throw Exception if saveAsTable[apend] has different
number of columns
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
