LantaoJin opened a new pull request #25840: [SPARK-29166][SQL] Add a parameter to limit the number of dynamic partitions for data source table URL: https://github.com/apache/spark/pull/25840 ### What changes were proposed in this pull request? Dynamic partition in Hive table has some restrictions to limit the max number of partitions. See https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-DynamicPartitionInserts It's very useful to prevent to create mistake partitions like ID. Also it can protect the NameNode from mass RPC calls of creating. Data source table also needs similar limitation. ### Why are the changes needed? Add a parameter to limit the number of dynamic partitions for data source table. By default, the max number of partitions is Int.MaxValue. It's nearly no limitation. When the parameter value we set is reached, it will throw SparkException and abort the job. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Add a unit test.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
