capistrant commented on code in PR #18844:
URL: https://github.com/apache/druid/pull/18844#discussion_r2687079273
##########
indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionJobQueue.java:
##########
@@ -357,14 +357,15 @@ private String startTaskIfReady(CompactionJob job)
}
/**
- * Persist the compaction state associated with the given job with {@link
CompactionStateManager}.
+ * Persist the compaction state associated with the given job with {@link
CompactionStateStorage}.
*/
private void persistPendingCompactionState(CompactionJob job)
{
- if (job.getCompactionState() != null &&
job.getCompactionStateFingerprint() != null) {
- jobParams.getCompactionStateManager().persistCompactionState(
+ if (job.getTargetCompactionState() != null &&
job.getTargetCompactionStateFingerprint() != null) {
+ jobParams.getCompactionStateStorageImpl().upsertCompactionState(
Review Comment:
One solution I have in mind is a `pending` flag added to
druid_compactionStates
when the job queue inserts states, it inserts them as pending=true. The kill
duty then has a separate, more permissive, config for killing pending states.
say p7d default. There should be very few states that get orphaned in a way
that they ever need to be cleaned up this way. The config could suggest an
operator never change this to a value less than the p999 runtime of their
compaction tasks (and likely to just stick with default). Then the completion
flow of a compaction task can also fire a query to update pending=false for its
fingerprint in druid_compactionStates (most of the time this will be a 0 row
update)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
