Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/15667#discussion_r85807729
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
---
@@ -257,7 +258,31 @@ case class InsertIntoHiveTable(
table.catalogTable.identifier.table,
partitionSpec)
+ var doOverwrite = overwrite
+
if (oldPart.isEmpty || !ifNotExists) {
+ // SPARK-18107: Insert overwrite runs much slower than
hive-client.
+ // Newer Hive largely improves insert overwrite performance. As
Spark uses older Hive
+ // version and we may not want to catch up new Hive version
every time. We delete the
+ // Hive partition first and then load data file into the Hive
partition.
+ if (oldPart.nonEmpty && overwrite) {
+ oldPart.get.storage.locationUri.map { uri =>
+ val partitionPath = new Path(uri)
+ val fs = partitionPath.getFileSystem(hadoopConf)
+ if (fs.exists(partitionPath)) {
+ val pathPermission =
fs.getFileStatus(partitionPath).getPermission()
+ if (!fs.delete(partitionPath, true)) {
+ throw new RuntimeException(
+ "Cannot remove partition directory '" +
partitionPath.toString)
+ } else {
+ fs.mkdirs(partitionPath, pathPermission)
--- End diff --
Is the mkdir necessary?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
