vinothchandar commented on a change in pull request #2475:
URL: https://github.com/apache/hudi/pull/2475#discussion_r569761896
##########
File path:
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -84,6 +86,39 @@ public static String getTablePath(FileSystem fs, Path[]
userProvidedPaths) throw
throw new TableNotFoundException("Unable to find a hudi table for the user
provided paths.");
}
+ public static Option<String> getOnePartitionPath(FileSystem fs, Path
tablePath) throws IOException {
+ // When the table is not partitioned
+ if (HoodiePartitionMetadata.hasPartitionMetadata(fs, tablePath)) {
+ return Option.of(tablePath.toString());
+ }
+ FileStatus[] statuses = fs.listStatus(tablePath);
+ for (FileStatus status : statuses) {
+ if (status.isDirectory()) {
+ if (HoodiePartitionMetadata.hasPartitionMetadata(fs,
status.getPath())) {
+ return Option.of(status.getPath().toString());
+ } else {
+ Option<String> partitionPath = getOnePartitionPath(fs,
status.getPath());
+ if (partitionPath.isPresent()) {
+ return partitionPath;
Review comment:
So, I am wondering if we can use the `HoodieTableMetadata` abstraction
to read a partition path, instead of listing alone. We are trying to avoid any
introduction of single point listings. There is a method to get all partition
paths already FSUtils.getAllPartitionPaths(), lets just use that for now? I am
thinking that it will be little bit of an overkill to list all partition paths,
without metadata table
##########
File path:
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -84,6 +86,39 @@ public static String getTablePath(FileSystem fs, Path[]
userProvidedPaths) throw
throw new TableNotFoundException("Unable to find a hudi table for the user
provided paths.");
}
+ public static Option<String> getOnePartitionPath(FileSystem fs, Path
tablePath) throws IOException {
+ // When the table is not partitioned
+ if (HoodiePartitionMetadata.hasPartitionMetadata(fs, tablePath)) {
+ return Option.of(tablePath.toString());
+ }
+ FileStatus[] statuses = fs.listStatus(tablePath);
+ for (FileStatus status : statuses) {
+ if (status.isDirectory()) {
+ if (HoodiePartitionMetadata.hasPartitionMetadata(fs,
status.getPath())) {
+ return Option.of(status.getPath().toString());
+ } else {
+ Option<String> partitionPath = getOnePartitionPath(fs,
status.getPath());
+ if (partitionPath.isPresent()) {
+ return partitionPath;
Review comment:
this short circuits the recursive stack, once we get one partition path
I guess
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
