kfaraz commented on code in PR #19921:
URL: https://github.com/apache/druid/pull/19921#discussion_r3755199738
##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataQuery.java:
##########
@@ -997,20 +1016,53 @@ public int markNonOvershadowedSegmentsAsUsed(
return markNonOvershadowedSegmentsAsUsed(unusedSegments, timeline,
updateTime);
}
- private List<DataSegment> retrieveUnusedSegments(
+ /**
+ * Checks that all the segments were last updated within the kill buffer
period.
+ * If any segment was updated earlier than that, an exception is thrown so
that
+ * none of the segments are updated to ensure atomicity.
+ */
+ private void validateSegmentsForMarkingAsUsed(Collection<DataSegmentPlus>
segments)
+ {
+ if (!managerConfig.getKillUnused().isEnabled()) {
+ // Do not verify the buffer period if embedded kill tasks are not enabled
+ return;
+ }
+
+ final Period bufferPeriod =
managerConfig.getKillUnused().getBufferPeriod();
+ final DateTime minAllowedUpdateTime = DateTimes.nowUtc().minus(
+ managerConfig.getKillUnused().getBufferPeriod()
+ );
+
+ final List<SegmentId> expiredSegmentIds = segments.stream().filter(
+ s -> s.getUsedStatusLastUpdatedDate() != null
+ && !s.getUsedStatusLastUpdatedDate().isAfter(minAllowedUpdateTime)
+ ).map(s -> s.getDataSegment().getId()).toList();
+
+ if (!expiredSegmentIds.isEmpty()) {
+ throw DruidException.forPersona(DruidException.Persona.OPERATOR)
+ .ofCategory(DruidException.Category.CONFLICT)
+ .build(
+ "Segment IDs[%s] cannot be marked as used since"
+ + " they were last updated more than [%s] ago
and"
+ + " are now eligible for permanent deletion.",
+ expiredSegmentIds, bufferPeriod
+ );
Review Comment:
Hmm, I think we should document it either way (at least for embedded kill
tasks).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
