loquisgon commented on code in PR #12392:
URL: https://github.com/apache/druid/pull/12392#discussion_r847612982
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java:
##########
@@ -426,6 +426,17 @@ private CoordinatorStats doRun(
if (config.getIoConfig() != null) {
dropExisting = config.getIoConfig().isDropExisting();
}
+ // force dropExisting to true if dropExisting is not set,
+ // dropExistingis false,
+ // or all segments to compact are tombstones.
+ // Otherwise, auto-compaction will cycle indefinitely since segments
won't be compacted and their
+ // compaction state won't be set
Review Comment:
I will fix this in next commit. I have been thinking a little, not a lot,
and it seems to me, at first sight, that auto-compaction should not track
whether a segment was compacted in its payload. Somehow auto-compaction ought
to reason about current state of segments and decide to compact them based on
that state not whether they were compacted recently or not (i.e. this segment
is tiny and there are other sibling segments in this time chunk that could be
compacted together). But since state about compaction is saved in the segment
payload and this state is needed in current auto-compaction logic to make the
decision to compact we have to be able to save this state for all segments
considered for auto-compaction and thus we need to force `dropExisting` to be
able to compact those tombstones in this case. The current semantics though
should not change (that is , drop will not drop any segments with data that was
not already overshadowed by existing tombstones). Previously there wer
e only tombstones in this interval and after this there will be only
tombstones as well (albeit potentially with different segment granularity).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
