furquan39 commented on issue #66299: URL: https://github.com/apache/doris/issues/66299#issuecomment-5133611053
Adding measured evidence for this retry-loop, from a version-bloated reproduction table where a subset of tablets are permanently un-recompactable: a single already-merged rowset materializes over 4 GiB of one column uncompressed, so every compaction attempt fails E-3113 (the sibling issue #66298). That is exactly the permanent-failure class this issue is about. Under the current no-backoff gate (`tablet_sched_delay_time_ms` = 5000 ms, tablet_manager.cpp:770-780) these tablets retried continuously with zero durable progress. Sampling the cumulative E-3113 failure count on the table across successive ~10-minute windows, it climbed 12 -> 14 -> 26 -> 42 -> 68 while our fix attempts made no progress: a self-sustaining failed-compaction retry loop burning continuous wasted I/O. Nothing throttled it; only manually setting `disable_auto_compaction=true` stopped it. At production scale on shared storage this is the amplifier. A single un-compactable tablet becomes the dominant writer and turns into a cluster-wide I/O storm that starved co-located services and took down unrelated workloads. The underlying failure cause is beside the point for this issue: a per-tablet consecutive-failure counter plus exponential backoff plus quarantine of permanently-failing tablets would bound the wasted I/O to negligible regardless of why a given compaction is doomed. We remain willing to submit the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
