Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/9651#discussion_r44629735
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -166,11 +170,37 @@ private[sql] object PartitioningUtils {
return (None, None)
}
+ // Let's say chopped is a path of /table/a=1/, chopped.getName will
give us a=1.
+ // Once we get the string, we try to parse it and find the partition
column and value.
val maybeColumn = parsePartitionColumn(chopped.getName,
defaultPartitionName, typeInference)
- maybeColumn.foreach(columns += _)
+
+ // Now, basePath will be /table/a=1/
basePath = chopped
+ // chopped will be /table/
chopped = chopped.getParent
- finished = (maybeColumn.isEmpty && !columns.isEmpty) ||
chopped.getParent == null
+
+ // Now, we determine if we should continue.
+ // When we hit any of the following three cases, we will not
continue:
+ // - In this iteration, we could not parse the value of partition
column and value,
+ // i.e. maybeColumn is None, and columns is not empty. At here we
check if columns is
+ // empty to handle cases like /table/a=1/_temporary/something (we
need to find a=1 in
+ // this case).
+ // - After we get the new chopped, this new chopped represent the
path of "/table", i.e.
+ // chopped.getParent == null.
+ // - The chopped we used to parse partition column and value (right
now, it is basePath),
+ // is already the root path of a table. For the example of
/table/a=1/, /table/ is the
+ // root path.
+ finished =
+ (maybeColumn.isEmpty && !columns.isEmpty) ||
+ chopped.getParent == null ||
+ rootPaths.contains(basePath)
+
+ if (maybeColumn.isDefined && !rootPaths.contains(basePath)) {
--- End diff --
We can do it in this way:
```scala
if (rootPaths.contains(basePath)) {
finished = true
} else {
val maybeColumn = parsePartitionColumn(chopped.getName,
defaultPartitionName, typeInference)
maybeColumn.foreach(columns += _)
basePath = chopped
chopped = chopped.getParent
finished = (maybeColumn.isEmpty && columns.nonEmpty) || chopped.getParent
== null
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
