jihoonson commented on pull request #10861: URL: https://github.com/apache/druid/pull/10861#issuecomment-774753381
Hi @liuxiaohui1221, thank you for your contribution. I'm wondering how this change can reduce the compaction task failures due to lock contention. Here are what should happen when two or more tasks try to lock overlapped intervals. - A high priority task is submitted while a low priority task is running. In this case, the high priority task revokes the lock of the low priority task. The low priority task stops with the `FAILED` state. - A low or equal priority task is submitted while a high or equal priority task is running. In this case, the second task waits (in the `WAITING` state) until the first task releases the lock. Compaction tasks can fail in the first case above when there is lock contention. However, the new overlord API (`getNonLockIntervalSnapshots()`) can be useful only for the second case by preventing the coordinator from submitting compaction tasks that can lead to lock contention. The `skipOffsetFromLatest` in the auto compaction config should be enough to avoid both cases unless data can arrive late frequently. Or am I missing something? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
