Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15667#discussion_r85861722
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
---
@@ -257,7 +258,31 @@ case class InsertIntoHiveTable(
table.catalogTable.identifier.table,
partitionSpec)
+ var doOverwrite = overwrite
+
if (oldPart.isEmpty || !ifNotExists) {
+ // SPARK-18107: Insert overwrite runs much slower than
hive-client.
+ // Newer Hive largely improves insert overwrite performance. As
Spark uses older Hive
+ // version and we may not want to catch up new Hive version
every time. We delete the
+ // Hive partition first and then load data file into the Hive
partition.
+ if (oldPart.nonEmpty && overwrite) {
+ oldPart.get.storage.locationUri.map { uri =>
+ val partitionPath = new Path(uri)
+ val fs = partitionPath.getFileSystem(hadoopConf)
+ if (fs.exists(partitionPath)) {
+ val pathPermission =
fs.getFileStatus(partitionPath).getPermission()
+ if (!fs.delete(partitionPath, true)) {
+ throw new RuntimeException(
+ "Cannot remove partition directory '" +
partitionPath.toString)
+ } else {
+ fs.mkdirs(partitionPath, pathPermission)
--- End diff --
I was thinking Hive will complain if the dir is not existing. But looks
like it won't. Let me remove this and see if tests can passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
