prashantwason commented on code in PR #8758:
URL: https://github.com/apache/hudi/pull/8758#discussion_r1228519465
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -557,32 +558,33 @@ private HoodieData<HoodieRecord>
readRecordKeysFromBaseFiles(HoodieEngineContext
final String fileId = FSUtils.getFileId(filename);
final String instantTime = FSUtils.getCommitTime(filename);
- HoodieFileReader reader =
HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO).getFileReader(hadoopConf.get(),
dataFilePath);
- Iterator<String> recordKeyIterator = reader.getRecordKeyIterator();
+ try (HoodieFileReader reader =
HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO).getFileReader(hadoopConf.get(),
dataFilePath)) {
Review Comment:
If you include this in the try block then the reader will get closed after
the try block before the returned iterator has been completely consumed by the
engineContext.parallelize call.
In other words, this lambda returns an iterator which will be consumed
(iterated over) outside this function call. The returned iterator holds a
reference to the reader which should not be closed until the iterator has
completed.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -430,9 +429,13 @@ private boolean initializeFromFilesystem(String
initializationTime, List<Metadat
Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
try {
+ // Check and then open the metadata table reader to create a file
system view
+ if (this.metadata != null) {
Review Comment:
@suryaprasanna Do you mean this.metadata == null instead of !=
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -734,6 +736,7 @@ private void initializeFileGroups(HoodieTableMetaClient
dataMetaClient, Metadata
}
} catch (FileNotFoundException e) {
// If the partition did not exist yet, it will be created below
+ LOG.warn("Exception seen while removing existing file groups in
partition {} ", partitionPath.getName(), e);
Review Comment:
this catches FileNotFoundException .... so I feel this log is not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
