RussellSpitzer commented on a change in pull request #2564:
URL: https://github.com/apache/iceberg/pull/2564#discussion_r629679189
##########
File path: core/src/main/java/org/apache/iceberg/MetadataLocationUtils.java
##########
@@ -58,17 +53,22 @@ private static void miscMetadataFiles(String
metadataFileLocation, Set<String> m
if (metadataFileLocation == null) {
return;
}
- try {
- TableMetadata metadata = TableMetadataParser.read(io,
metadataFileLocation);
- List<TableMetadata.MetadataLogEntry> metadataLogEntries =
metadata.previousFiles();
- List<String> previousMetadataFiles =
-
metadataLogEntries.stream().map(TableMetadata.MetadataLogEntry::file).collect(Collectors.toList());
+ TableMetadata metadata = TableMetadataParser.read(io,
metadataFileLocation);
+ List<TableMetadata.MetadataLogEntry> metadataLogEntries =
metadata.previousFiles();
+ List<String> previousMetadataFiles =
+ metadataLogEntries.stream().map(TableMetadata.MetadataLogEntry::file)
+ .collect(Collectors.toList());
+ if (previousMetadataFiles.size() > 0) {
metaFiles.addAll(previousMetadataFiles);
- if (isRecursive && previousMetadataFiles.size() > 0) {
- miscMetadataFiles(previousMetadataFiles.get(0), metaFiles, io,
isRecursive);
+ if (isRecursive) {
Review comment:
I think i missed the discussion on this, won't this recursion end up
visiting the same files over and over?
Ie we should be (by default) storing 100 entries in previous files
Then every time we add a new entry it should be deleting the oldest one.
So in our normal recursion we visit file
1 - 100 and queue up a recursion on all 100 of those
On file 2 we then open up
2 - 102 and queue up a recursion on all 100 of those
on file 3
3- 103 and queue up a recursion on 100 of those?
So we end up opening up files 5 thousand or so times with the default
history saving levels and we open up the same files up to 99 times right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
