loquisgon commented on code in PR #12392:
URL: https://github.com/apache/druid/pull/12392#discussion_r847612982
##########
server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java:
##########
@@ -426,6 +426,17 @@ private CoordinatorStats doRun(
if (config.getIoConfig() != null) {
dropExisting = config.getIoConfig().isDropExisting();
}
+ // force dropExisting to true if dropExisting is not set,
+ // dropExistingis false,
+ // or all segments to compact are tombstones.
+ // Otherwise, auto-compaction will cycle indefinitely since segments
won't be compacted and their
+ // compaction state won't be set
Review Comment:
I will fix this in next commit. I have been thinking a little, not a lot,
and it seems to me, at first sight, that auto-compaction should not track
whether a segment was compacted in its payload. Somehow auto-compaction ought
to reason about current state of segments and decide to compact them based on
that state not whether they were compacted recently or not (i.e. this segment
is tiny and there are other sibling segments in this time chunk that could be
compacted together). But since state about compaction is saved in the segment
payload and this state is needed to make the decision to compact we have to be
able to save this state for all segments considered for auto-compaction and
thus we need to force `dropExisting` to be able to compact those tombstones in
this case. The current semantics though should not change (that is , drop will
not drop any segments with data that was not already overshadowed by existing
tombstones). Previously there were only tombstones in this interva
l and after this there will be only tombstones as well (albeit with different
segment granularity).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
