capistrant commented on code in PR #18844:
URL: https://github.com/apache/druid/pull/18844#discussion_r2682947269
##########
indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionJobQueue.java:
##########
@@ -357,14 +357,15 @@ private String startTaskIfReady(CompactionJob job)
}
/**
- * Persist the compaction state associated with the given job with {@link
CompactionStateManager}.
+ * Persist the compaction state associated with the given job with {@link
CompactionStateStorage}.
*/
private void persistPendingCompactionState(CompactionJob job)
{
- if (job.getCompactionState() != null &&
job.getCompactionStateFingerprint() != null) {
- jobParams.getCompactionStateManager().persistCompactionState(
+ if (job.getTargetCompactionState() != null &&
job.getTargetCompactionStateFingerprint() != null) {
+ jobParams.getCompactionStateStorageImpl().upsertCompactionState(
Review Comment:
Yes, persisting before the task runs is intentional for multiple reasons:
* The CompactionState object constructed for `lastCompactionState` is not
useable for storage with the fingerprint associated with the config because
they become disconnected as indexing service adds defaults that may not be
reflected in config.
* Once I realized this, I started passing the fingerprint AND state in
ingestion task context. This "works" but is pretty dirty and scalability
becomes a concern as Compaction states can get very big (think 100s of filter
rules to remove specific row patterns from data as it ages)
You do call out a legit issue with the timing in nuking an unreferenced
state when compaction tasks take some time. I need to think about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
