akashchamp commented on code in PR #20220:
URL: https://github.com/apache/hudi/pull/20220#discussion_r4238390092
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ExportCommand.java:
##########
@@ -93,26 +98,18 @@ public String exportInstants(
}
// The non archived instants can be listed from the Timeline.
- HoodieTimeline timeline =
HoodieCLI.getTableMetaClient().getActiveTimeline().filterCompletedInstants()
+ HoodieTimeline timeline =
metaClient.getActiveTimeline().filterCompletedInstants()
Review Comment:
Good catch. The active timeline is now reloaded
(metaClient.reloadActiveTimeline()) before the instants are listed, in c0eb6cd.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ExportCommand.java:
##########
@@ -121,31 +118,135 @@ public String exportInstants(
return "Exported " + numCopied + " Instants to " + localFolder;
}
- private int copyArchivedInstants(List<StoragePathInfo> pathInfoList,
+ private int copyArchivedInstants(HoodieTableMetaClient metaClient,
Set<String> actionSet,
+ boolean descending,
int limit,
String localFolder) throws Exception {
+ if (!isLegacyArchive(metaClient)) {
+ return copyArchivedTimelineInstants(metaClient, actionSet, descending,
limit, localFolder);
+ }
+ // Archived instants of a table written before table version 8 are in the
commit archive files
+ List<StoragePathInfo> archivedPathInfoList = metaClient.getStorage()
+ .globEntries(new StoragePath(metaClient.getArchivePath(),
".commits_.archive*")).stream()
+ .sorted(Comparator.comparingLong(StoragePathInfo::getModificationTime))
+ .collect(Collectors.toList());
+ if (descending) {
+ Collections.reverse(archivedPathInfoList);
Review Comment:
Fixed in c0eb6cd: with --desc each legacy archive file's completed entries
are now collected and written from the latest one, so the order is newest-first
inside a file as well. I added a legacy --desc case with a limit to the test; I
could not run it locally, so CI has to confirm.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ExportCommand.java:
##########
@@ -121,31 +118,135 @@ public String exportInstants(
return "Exported " + numCopied + " Instants to " + localFolder;
}
- private int copyArchivedInstants(List<StoragePathInfo> pathInfoList,
+ private int copyArchivedInstants(HoodieTableMetaClient metaClient,
Set<String> actionSet,
+ boolean descending,
int limit,
String localFolder) throws Exception {
+ if (!isLegacyArchive(metaClient)) {
+ return copyArchivedTimelineInstants(metaClient, actionSet, descending,
limit, localFolder);
+ }
+ // Archived instants of a table written before table version 8 are in the
commit archive files
+ List<StoragePathInfo> archivedPathInfoList = metaClient.getStorage()
+ .globEntries(new StoragePath(metaClient.getArchivePath(),
".commits_.archive*")).stream()
+ .sorted(Comparator.comparingLong(StoragePathInfo::getModificationTime))
+ .collect(Collectors.toList());
+ if (descending) {
+ Collections.reverse(archivedPathInfoList);
+ }
+ return copyLegacyArchivedInstants(metaClient, archivedPathInfoList,
actionSet, limit, localFolder);
+ }
+
+ /**
+ * Exports the completed archived instants of a table of version 8 or above,
which keeps them in
+ * the LSM timeline under the timeline history folder, read through its
archived timeline.
+ */
+ private int copyArchivedTimelineInstants(HoodieTableMetaClient metaClient,
+ Set<String> actionSet,
+ boolean descending,
+ int limit,
+ String localFolder) throws
Exception {
+ // bypass the meta client cache, so that an archival triggered earlier in
the same CLI session is visible
+ HoodieArchivedTimeline archivedTimeline =
metaClient.getArchivedTimeline(StringUtils.EMPTY_STRING, false);
+ List<HoodieInstant> instants = archivedTimeline.getInstantsAsStream()
+ .filter(HoodieInstant::isCompleted)
+ .filter(instant -> actionSet.contains(instant.getAction()))
+ .collect(Collectors.toList());
+ if (descending) {
+ Collections.reverse(instants);
+ }
+ if (instants.size() > limit) {
+ instants = instants.subList(0, limit);
+ }
+ if (instants.isEmpty()) {
+ return 0;
+ }
+ // load only the payloads of the time range being exported
+ String first = instants.get(0).requestedTime();
+ String last = instants.get(instants.size() - 1).requestedTime();
+ archivedTimeline.loadCompletedInstantDetailsInMemory(descending ? last :
first, descending ? first : last);
Review Comment:
Done in c0eb6cd: the v8+ path now loads, writes and clears the instant
details in windows of 1000 instants instead of loading the whole range at once.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ExportCommand.java:
##########
@@ -121,31 +118,135 @@ public String exportInstants(
return "Exported " + numCopied + " Instants to " + localFolder;
}
- private int copyArchivedInstants(List<StoragePathInfo> pathInfoList,
+ private int copyArchivedInstants(HoodieTableMetaClient metaClient,
Set<String> actionSet,
+ boolean descending,
int limit,
String localFolder) throws Exception {
+ if (!isLegacyArchive(metaClient)) {
Review Comment:
I kept the boolean. It mirrors the existing --desc option and is passed
through a single private helper, so a separate enum would add a type for one
internal call. Happy to change it if a maintainer prefers.
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ExportCommand.java:
##########
@@ -121,31 +118,135 @@ public String exportInstants(
return "Exported " + numCopied + " Instants to " + localFolder;
}
- private int copyArchivedInstants(List<StoragePathInfo> pathInfoList,
+ private int copyArchivedInstants(HoodieTableMetaClient metaClient,
Set<String> actionSet,
+ boolean descending,
int limit,
String localFolder) throws Exception {
+ if (!isLegacyArchive(metaClient)) {
+ return copyArchivedTimelineInstants(metaClient, actionSet, descending,
limit, localFolder);
+ }
+ // Archived instants of a table written before table version 8 are in the
commit archive files
+ List<StoragePathInfo> archivedPathInfoList = metaClient.getStorage()
+ .globEntries(new StoragePath(metaClient.getArchivePath(),
".commits_.archive*")).stream()
+ .sorted(Comparator.comparingLong(StoragePathInfo::getModificationTime))
+ .collect(Collectors.toList());
+ if (descending) {
+ Collections.reverse(archivedPathInfoList);
+ }
+ return copyLegacyArchivedInstants(metaClient, archivedPathInfoList,
actionSet, limit, localFolder);
+ }
+
+ /**
+ * Exports the completed archived instants of a table of version 8 or above,
which keeps them in
+ * the LSM timeline under the timeline history folder, read through its
archived timeline.
+ */
+ private int copyArchivedTimelineInstants(HoodieTableMetaClient metaClient,
+ Set<String> actionSet,
+ boolean descending,
+ int limit,
+ String localFolder) throws
Exception {
+ // bypass the meta client cache, so that an archival triggered earlier in
the same CLI session is visible
+ HoodieArchivedTimeline archivedTimeline =
metaClient.getArchivedTimeline(StringUtils.EMPTY_STRING, false);
+ List<HoodieInstant> instants = archivedTimeline.getInstantsAsStream()
+ .filter(HoodieInstant::isCompleted)
+ .filter(instant -> actionSet.contains(instant.getAction()))
+ .collect(Collectors.toList());
+ if (descending) {
+ Collections.reverse(instants);
Review Comment:
Addressed in c0eb6cd: the window is now taken from the ascending list, so
the start and end times come straight from its first and last instants and the
descending ternary is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
