Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12871#discussion_r62149522
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -245,7 +301,21 @@ class InMemoryCatalog extends ExternalCatalog {
s"'$db' table '$table':\n$dupSpecsStr")
}
}
- parts.foreach { p => existingParts.put(p.spec, p) }
+
+ val tableDir = new Path(catalog(db).db.locationUri, table)
+ // TODO: we should follow hive to roll back if one partition path
failed to create.
+ parts.foreach { p =>
+ if (p.storage.locationUri.isEmpty) {
+ val partitionPath = p.spec.map { case (k, v) => s"$k=$v"
}.mkString("/")
--- End diff --
One potential issue is that the partition spec is a map and is un-ordered.
In one of my PR, we had to first compute the order based on partition column
names. Lian had found this issue while reviewing my PR :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
