xushiyan commented on code in PR #6662:
URL: https://github.com/apache/hudi/pull/6662#discussion_r973800037
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/HoodieSyncClient.java:
##########
@@ -158,4 +163,23 @@ public List<PartitionEvent>
getPartitionEvents(List<Partition> tablePartitions,
}
return events;
}
+
+ /**
+ * Get Last commit's Metadata.
+ */
+ private static Option<HoodieCommitMetadata>
getLatestCommitMetadata(HoodieTableMetaClient metaClient) {
Review Comment:
this method still needed?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -57,6 +58,29 @@ public static List<String>
getPartitionsWritten(HoodieTimeline timeline) {
return getAffectedPartitions(timelineToSync);
}
+ /**
+ * Returns partitions that have been deleted or marked for deletion in the
given timeline.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsDropped(HoodieTimeline timeline) {
Review Comment:
/nit name alignment: getPartitionsWritten getPartitionsDropped
getAffectedPartitions -> getXXXPartitions
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/HoodieSyncClient.java:
##########
@@ -83,18 +86,17 @@ public boolean isBootstrap() {
return metaClient.getTableConfig().getBootstrapBasePath().isPresent();
}
- public boolean isDropPartition() {
- try {
- Option<HoodieCommitMetadata> hoodieCommitMetadata =
HoodieTableMetadataUtil.getLatestCommitMetadata(metaClient);
-
- if (hoodieCommitMetadata.isPresent()
- &&
WriteOperationType.DELETE_PARTITION.equals(hoodieCommitMetadata.get().getOperationType()))
{
- return true;
- }
- } catch (Exception e) {
- throw new HoodieSyncException("Failed to get commit metadata", e);
- }
- return false;
+ /**
+ * Get the set of dropped partitions since the last synced commit.
+ * If last sync time is not known then consider only active timeline.
+ * Going through archive timeline is a costly operation, and it should be
avoided unless some start time is given.
+ */
+ public Set<String> getDroppedPartitions(Option<String> lastCommitTimeSynced)
{
Review Comment:
name alignment: getDroppedPartitions -> getDroppedPartitionsSince(...)
getPartitionsWrittenToSince -> getWrittenPartitionsSince(...)
also since it's not perf sensitive, i'd prefer to align on return type
List<String> as well, for API consistency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
