LantaoJin edited a comment on issue #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table URL: https://github.com/apache/spark/pull/25840#issuecomment-542056456 @cloud-fan The restriction in Hive only used in Hive client side. Although we store partitions in hive metastore, Hive's restrictions doesn't impact the insertion of data source table. Below is my test: ```shell bin/spark-sql --master local --conf spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict --conf spark.sql.sources.partitionOverwriteMode=DYNAMIC --conf spark.hadoop.hive.exec.max.dynamic.partitions=3 --conf spark.sql.hive.convertInsertingPartitionedTable=false spark-sql> create table hive_dynamic_partition(i int) stored as parquet partitioned by (part1 int, part2 int); spark-sql> insert overwrite table hive_dynamic_partition partition(part1, part2) select 1, 2, id from range(5); Error in query: org.apache.hadoop.hive.ql.metadata.HiveException: Number of dynamic partitions created is 5, which is more than 3. To solve this try to set hive.exec.max.dynamic.partitions to at least 5.; spark-sql> create table data_source_dynamic_partition(i int, part1 int, part2 int) using parquet partitioned by (part1, part2); spark-sql> insert overwrite table data_source_dynamic_partition partition(part1, part2) select 1, 2, id from range(5); Time taken: 3.464 seconds 19/10/15 14:20:03 INFO SparkSQLCLIDriver: Time taken: 3.464 seconds ``` Besides, when `--conf spark.sql.hive.convertInsertingPartitionedTable=true` (by default), the first insertion of Hive table will succeed. `--conf spark.hive.exec.dynamic.partition.mode=nonstrict` not work, it must set to `--conf spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
