prashantwason commented on a change in pull request #3762:
URL: https://github.com/apache/hudi/pull/3762#discussion_r728264572
##########
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java
##########
@@ -126,23 +130,21 @@ protected BaseTableMetadata(HoodieEngineContext
engineContext, HoodieMetadataCon
}
@Override
- public Map<String, FileStatus[]> getAllFilesInPartitions(List<String>
partitionPaths)
+ public Map<String, FileStatus[]> getAllFilesInPartitions(List<String>
partitions)
throws IOException {
if (enabled) {
- Map<String, FileStatus[]> partitionsFilesMap = new HashMap<>();
-
try {
- for (String partitionPath : partitionPaths) {
- partitionsFilesMap.put(partitionPath, fetchAllFilesInPartition(new
Path(partitionPath)));
- }
+ // need to understand why we did not make bulk get before
Review comment:
For simplicity of implementation I suppose - performance was not taken
into consideration. Also, given the number of keys being fetched, batch would
be slower as it may need to read the entire hfile.
@umehrot2 Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
