YenchangChan opened a new pull request, #69920:
URL: https://github.com/apache/doris/pull/69920
### What problem does this PR solve?
Issue Number: close #66299
Related PR: None
Problem Summary:
When a tablet repeatedly fails compaction, the current scheduler retries it
after the fixed `tablet_sched_delay_time_ms` delay. A permanently failing
tablet can therefore be selected every few seconds and continuously consume
compaction threads and storage I/O without making progress.
This implementation follows the analysis and proposed direction in #66299.
Thanks to @furquan39 for the detailed investigation and production evidence.
This PR:
- tracks consecutive failures independently for cumulative and base
compaction;
- applies exponential backoff starting from `tablet_sched_delay_time_ms`;
- caps the delay at `compaction_failure_backoff_max_second`, which defaults
to 300 seconds;
- resets the failure count after a successful compaction;
- avoids counting the same failure more than once;
- exposes the consecutive failure counts in the tablet compaction status.
With the default configuration, retry delays increase as follows:
`5s -> 10s -> 20s -> 40s -> ... -> 300s`
The existing behavior for tablets without compaction failures remains
unchanged.
This PR focuses on bounded retry backoff. It does not introduce a quarantine
state or new metrics, which can be discussed separately based on operational
needs and feedback.
### Release note
Add capped exponential backoff for tablets with consecutive compaction
failures.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `TestTablet.compaction_failure_backoff`
- [x] Manual test
- Injected persistent compaction failures and observed the retry
delay increase from 5 seconds to the configured 300-second cap.
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Tablets with consecutive compaction failures are retried with
capped exponential backoff.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
