snleee commented on a change in pull request #5828:
URL: https://github.com/apache/incubator-pinot/pull/5828#discussion_r467356447
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
##########
@@ -177,4 +193,87 @@ private boolean shouldDeleteInProgressLLCSegment(String
segmentName, IdealState
return states.size() == 1 &&
states.contains(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE);
}
}
+
+ private void manageSegmentLineageCleanupForTable(String tableNameWithType) {
+ try {
+ DEFAULT_RETRY_POLICY.attempt(() -> {
+ // Fetch segment lineage
+ ZNRecord segmentLineageZNRecord = SegmentLineageAccessHelper
+
.getSegmentLineageZNRecord(_pinotHelixResourceManager.getPropertyStore(),
tableNameWithType);
+ if (segmentLineageZNRecord == null) {
+ LOGGER.info("Segment lineage does not exist for table: {}",
tableNameWithType);
+ return true;
+ }
+ SegmentLineage segmentLineage =
SegmentLineage.fromZNRecord(segmentLineageZNRecord);
+ int expectedVersion = segmentLineageZNRecord.getVersion();
+
+ // Delete segments based on the segment lineage
+ PinotResourceManagerResponse response = _pinotHelixResourceManager
+ .deleteSegments(tableNameWithType,
computeSegmentsToDeleteFromSegmentLineage(segmentLineage));
+
+ Assert.assertTrue(response.isSuccessful());
+
+ // Fetch available segments for the table
+ Set<String> segmentsForTable = new
HashSet<>(_pinotHelixResourceManager.getSegmentsFor(tableNameWithType));
+
+ // Clean up the segment lineage
+ for (String lineageEntryId : segmentLineage.getLineageEntryIds()) {
+ LineageEntry lineageEntry =
segmentLineage.getLineageEntry(lineageEntryId);
+ if (lineageEntry.getState() == LineageEntryState.COMPLETED) {
+ // The lineage entry for 'COMPLETED' state can only be safely
removed when both segmentFrom & segmentTo
+ // are all removed from the table.
+ if (Collections.disjoint(segmentsForTable,
lineageEntry.getSegmentsFrom()) && Collections
+ .disjoint(segmentsForTable, lineageEntry.getSegmentsTo())) {
+ segmentLineage.deleteLineageEntry(lineageEntryId);
+ }
+ } else if (lineageEntry.getState() == LineageEntryState.IN_PROGRESS)
{
+ // Zombie lineage entry is safe to remove. This will allow the
task scheduler to re-schedule the
+ // source segments to be merged again.
+ segmentLineage.deleteLineageEntry(lineageEntryId);
Review comment:
fixed
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
##########
@@ -177,4 +193,87 @@ private boolean shouldDeleteInProgressLLCSegment(String
segmentName, IdealState
return states.size() == 1 &&
states.contains(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE);
}
}
+
+ private void manageSegmentLineageCleanupForTable(String tableNameWithType) {
+ try {
+ DEFAULT_RETRY_POLICY.attempt(() -> {
+ // Fetch segment lineage
+ ZNRecord segmentLineageZNRecord = SegmentLineageAccessHelper
+
.getSegmentLineageZNRecord(_pinotHelixResourceManager.getPropertyStore(),
tableNameWithType);
+ if (segmentLineageZNRecord == null) {
+ LOGGER.info("Segment lineage does not exist for table: {}",
tableNameWithType);
+ return true;
+ }
+ SegmentLineage segmentLineage =
SegmentLineage.fromZNRecord(segmentLineageZNRecord);
+ int expectedVersion = segmentLineageZNRecord.getVersion();
+
+ // Delete segments based on the segment lineage
+ PinotResourceManagerResponse response = _pinotHelixResourceManager
+ .deleteSegments(tableNameWithType,
computeSegmentsToDeleteFromSegmentLineage(segmentLineage));
+
+ Assert.assertTrue(response.isSuccessful());
+
+ // Fetch available segments for the table
+ Set<String> segmentsForTable = new
HashSet<>(_pinotHelixResourceManager.getSegmentsFor(tableNameWithType));
Review comment:
Changed the logic to handle together.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
