bvaradar commented on code in PR #8452:
URL: https://github.com/apache/hudi/pull/8452#discussion_r1201504641
##########
hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java:
##########
@@ -96,11 +109,32 @@ public List<String>
getPartitionPathWithPathPrefixes(List<String> relativePathPr
}
private List<String> getPartitionPathWithPathPrefix(String
relativePathPrefix) throws IOException {
+ return
getPartitionPathWithPathPrefixUsingFilterExpression(relativePathPrefix, null,
null);
+ }
+
+ private List<String>
getPartitionPathWithPathPrefixUsingFilterExpression(String relativePathPrefix,
+
Types.RecordType partitionFields,
+
Expression expression) throws IOException {
List<Path> pathsToList = new CopyOnWriteArrayList<>();
pathsToList.add(StringUtils.isNullOrEmpty(relativePathPrefix)
- ? new Path(datasetBasePath) : new Path(datasetBasePath,
relativePathPrefix));
+ ? dataBasePath.get() : new Path(dataBasePath.get(),
relativePathPrefix));
List<String> partitionPaths = new CopyOnWriteArrayList<>();
+ int partitionLevel = -1;
+ boolean needPushDownExpressions;
+ // Not like `HoodieBackedTableMetadata`, since we don't know the exact
partition levels here,
+ // given it's possible that partition values contains `/`, which could
affect
+ // the result to get right `partitionValue` when listing paths, here we
have
+ // to make it more strict that `urlEncodePartitioningEnabled` must be
enabled.
+ // TODO better enable urlEncodePartitioningEnabled if
hiveStylePartitioningEnabled is enabled?
Review Comment:
Shall we disable partition push down filters by default when
FileSystemBackedTableMetadata is used ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
