haoyanzhang opened a new pull request, #40298:
URL: https://github.com/apache/spark/pull/40298
### What changes were proposed in this pull request?
Introduce spark.hive.exec.dynamic.partition.savePartitions=true (default
false)
spark.hive.exec.dynamic.partition.savePartitions.tableNamePrefix=hive_dynamic_inserted_partitions
when spark.hive.exec.dynamic.partition.savePartitions=true we save the
partitions to the
temporary view
$spark.hive.exec.dynamic.partition.savePartitions.tableNamePrefix_$dbName_$tableName
### Why are the changes needed?
When hive.exec.dynamic.partition=true and
hive.exec.dynamic.partition.mode=nonstrict, we can insert table by sql like
'insert overwrite table aaa partition(dt) select xxxx', of course we can know
the partitions inserted into the table by the sql itself, but if we want do
something for common use, we need some common way to get the inserted
partitions, for example:
spark.sql("insert overwrite table aaa partition(dt) select xxxx")
//insert table
val partitions = getInsertedPartitions() //need some way to get
inserted partitions
monitorInsertedPartitions(partitions) //do something for common use
This pull request will allow user to get inserted partitions in a common way
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
new unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
