anandheritage commented on issue #15011:
URL: https://github.com/apache/pinot/issues/15011#issuecomment-2642609796
Hi Team , For one of our table the older segments are not getting deleted.
The TTL is of 30 days but the data is still there for beyond (100 days).
The table ingestion is going good and able to serve queries correctly too.
Segments status is good too for All.
Although there are few observations :
private void manageRetentionForRealtimeTable(String realtimeTableName,
RetentionStrategy retentionStrategy) {
List<String> segmentsToDelete = new ArrayList<>();
IdealState idealState = _pinotHelixResourceManager.getHelixAdmin()
.getResourceIdealState(_pinotHelixResourceManager.getHelixClusterName(),
realtimeTableName);
for (SegmentZKMetadata segmentZKMetadata :
_pinotHelixResourceManager.getSegmentsZKMetadata(realtimeTableName)) {
String segmentName = segmentZKMetadata.getSegmentName();
if (segmentZKMetadata.getStatus() ==
[Status.IN](http://status.in/)_PROGRESS) {
// Delete old LLC segment that hangs around. Do not delete segment
that are current since there may be a race
// with RealtimeSegmentValidationManager trying to auto-create the LLC
segment
if (shouldDeleteInProgressLLCSegment(segmentName, idealState,
segmentZKMetadata)) {
segmentsToDelete.add(segmentName);
}
} else {
// Sealed segment
if (retentionStrategy.isPurgeable(realtimeTableName,
segmentZKMetadata)) {
segmentsToDelete.add(segmentName);
}
}
}
_pinotHelixResourceManager.getSegmentsZKMetadata(realtimeTableName))
For some reason the segment path is not there under segments.
But I can see the metadata under External View.
For all the other tables I can see ZK metadata is present in both the places.
Controller logs : 2025-02-07T09:56:52.630692061Z stdout F ZKPath
/PinotCluster/PROPERTYSTORE/SEGMENTS/amxordersntt_REALTIME/533/code/code/code/code/code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
