nsivabalan commented on a change in pull request #3762:
URL: https://github.com/apache/hudi/pull/3762#discussion_r728518751
##########
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java
##########
@@ -126,23 +130,21 @@ protected BaseTableMetadata(HoodieEngineContext
engineContext, HoodieMetadataCon
}
@Override
- public Map<String, FileStatus[]> getAllFilesInPartitions(List<String>
partitionPaths)
+ public Map<String, FileStatus[]> getAllFilesInPartitions(List<String>
partitions)
throws IOException {
if (enabled) {
- Map<String, FileStatus[]> partitionsFilesMap = new HashMap<>();
-
try {
- for (String partitionPath : partitionPaths) {
- partitionsFilesMap.put(partitionPath, fetchAllFilesInPartition(new
Path(partitionPath)));
- }
+ // need to understand why we did not make bulk get before
Review comment:
from what I infer, with HoodieMergedLogRecordScanner, we first read all
records from all log blocks and prepare a hash map of records(record key to
HoodieRecord). And we don't do seek based read prior to this patch and so we do
read all log records from all log blocks. so was bit curious.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
