furquan39 opened a new issue, #66299: URL: https://github.com/apache/doris/issues/66299
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Apache Doris 4.1.0-rc03. All file:line references below are at tag `4.1.0-rc03`. We operate Doris 4.1.0-rc03 on Kubernetes with Ceph-backed storage. Related: "[Bug] Vertical compaction sizes output segments by compressed bytes while string columns enforce an uncompressed uint32 limit, making E-3113 deterministic for high-compression-ratio data" (#66298). It analyzes the specific permanent failure we hit; this issue is about the retry policy, which applies to any permanently failing compaction. When a compaction attempt fails on a tablet, the failure is recorded via `set_last_failure_time` (tablet.cpp:1888). The scheduling gate in tablet_manager.cpp:770-780 only skips the tablet while `now - last_failure_time < tablet_sched_delay_time_ms` (default 5000 ms, config.cpp:1668), and the compaction producer loop wakes every 100 ms (olap_server.cpp:668). There is: - no per-tablet failure counter, - no backoff escalation, - no quarantine state, - and the tablet's compaction score does not drop on failure, so a high-score failing tablet is re-selected as soon as the 5 s cooldown expires. This is deliberate: PR #8781 lowered the cooldown from 600 s to 5 s to protect against -235 (too many versions) when failures are transient. For transient failures that tradeoff is right. PR #61696 later added observability around compaction failures, but it is observability-only and does not change the retry policy. The problem is failures that are permanent for a given data shape. The class we hit is E-3113 (`STRING_OVERFLOW_IN_VEC_ENGINE`): vertical compaction plans output segments in compressed bytes while string columns enforce a 4 GiB uncompressed limit, so for data that compresses well enough the planned merge overflows deterministically, at the same point, on every attempt. The full mechanism, source anchors, and its own suggested fixes are in the related issue above; what matters for this issue is only that such permanent failure classes exist. The scheduler retries the doomed merge forever at full I/O cost, every 5 s of cooldown plus however long the merge runs before failing. **Reproduction** (on a dev cluster): 1. Create a table with a VARIANT column and load a highly duplicated corpus (ours: about 145 GB, compression ratio around 100x). 2. Let cumulative compaction select the tablet. The merge fails with E-3113 (failure recorded via `set_last_failure_time` at tablet.cpp:1888 and logged at tablet.cpp:1889; the verbatim error and the overflow arithmetic are in the related issue). 3. Observe the retry loop with zero ingest on the table: on a 3.14 GB tablet we measured 74 failed compaction attempts in 67 minutes, sustaining 13.3 MiB/s write and 29.6 MiB/s read, i.e. 51.9 GiB written to storage with zero durable progress. Each attempt re-reads the input rowsets and re-writes merge output until it hits the same overflow. **Impact:** - The retry storm runs indefinitely and became the dominant writer on the Ceph-backed storage, degrading everything else that uses it. In production terms it contributed to node outages and ingest stalls, and it does not self-heal because the failure is deterministic for the data shape. - The only immediate mitigation we found is `disable_auto_compaction=true`, which trades one failure mode for another: the emergency force-compaction escape in `RowsetBuilder::check_tablet_version_count` (rowset_builder.cpp:149-179) is gated on `!disable_auto_compaction`, so version counts climb toward `max_tablet_version_num` (2000, config.cpp:878) and ingest eventually fails with -235. Operators are left choosing between an I/O storm and a slow-motion ingest stall. **Master status:** the master restructure (PR #61107) moved `be/src/olap/*` to `be/src/storage/*`; the flat-delay gate is at `be/src/storage/tablet/tablet_manager.cpp:787` on master as of 2026-07-30, still with no failure counter and no backoff. So while the reproduction above is on 4.1.0-rc03, the relevant scheduling code is verified unchanged on current master. ### Solution We would keep the fast 5 s retry that PR #8781 introduced for transient failures, and change behavior only for tablets that fail repeatedly: - Track a per-tablet consecutive compaction failure counter, reset on success. - Keep the current 5 s delay for the first few failures, then escalate the delay (for example doubling up to a cap in the minutes-to-an-hour range). - After N consecutive failures, move the tablet into a quarantine state that is retried rarely and is clearly visible to operators (SHOW output and/or a BE metric for failure counts and quarantined tablet count), so storms are observable before they saturate storage. The change looks small and local: the gate at tablet_manager.cpp:770-780 plus failure bookkeeping around tablet.cpp:1888 (on master: `be/src/storage/tablet/tablet_manager.cpp:787` and the corresponding tablet failure bookkeeping under `be/src/storage/tablet/`). We are willing to contribute a PR if maintainers agree with the direction, and we are open to alternative designs. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
