GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/17223
[SPARK-19881][SQL] Support Dynamic Partition Inserts params with SET command
## What changes were proposed in this pull request?
Since Spark 2.0.0, `SET` commands do not pass the values to HiveClient. In
most case, Spark handles well. However, for the dynamic partition insert, users
meet the following misleading situation.
```scala
scala> spark.range(1001).selectExpr("id as key", "id as
value").registerTempTable("t1001")
scala> sql("create table p (value int) partitioned by (key int)").show
scala> sql("insert into table p partition(key) select key, value from
t1001")
org.apache.spark.SparkException:
Dynamic partition strict mode requires at least one static partition column.
To turn this off set hive.exec.dynamic.partition.mode=nonstrict
scala> sql("set hive.exec.dynamic.partition.mode=nonstrict")
scala> sql("insert into table p partition(key) select key, value from
t1001")
org.apache.hadoop.hive.ql.metadata.HiveException:
Number of dynamic partitions created is 1001, which is more than 1000.
To solve this try to set hive.exec.max.dynamic.partitions to at least 1001.
scala> sql("set hive.exec.dynamic.partition.mode=1001")
scala> sql("insert into table p partition(key) select key, value from
t1001")
org.apache.hadoop.hive.ql.metadata.HiveException:
Number of dynamic partitions created is 1001, which is more than 1000.
To solve this try to set hive.exec.max.dynamic.partitions to at least 1001.
```
The root cause is that `hive` parameters are passed to `HiveClient` on
creating. So, There is a workaround to use `--hiveconf`. However, we had better
handle this case without misleading error messages ending infinite loop.
## How was this patch tested?
Manual.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-19881
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17223.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17223
----
commit a8744551608cc8dd2fd6ada63c28a93d65e865b4
Author: Dongjoon Hyun <[email protected]>
Date: 2017-03-09T06:35:17Z
[SPARK-19881][SQL] Support Dynamic Partition Inserts params with SET command
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
