matanper opened a new issue, #69289: URL: https://github.com/apache/doris/issues/69289
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master at 61179b65580 (2026-10-19); the code is unchanged since 2.1/3.0. ### What's Wrong? The cloud checker's inverted check (`enable_inverted_check = true`) is the leak detector. It lists the objects of every storage location and reports those that no longer belong to any rowset, i.e. data the recycler failed to delete. It silently does nothing for any instance that has more than one storage vault: ```cpp // cloud/src/recycler/checker.cpp int InstanceChecker::do_inverted_check() { if (accessor_map_.size() > 1) { LOG(INFO) << "currently not support inverted check for multi accessor. instance_id=" << instance_id_; return 0; } ``` The checker creates one accessor per storage vault, plus the built-in vault. So as soon as an instance has a second vault, the check never runs for any of its vaults, the log shows only the line above, and `num_inverted_check_failed` stays at 0. We hit this while end-to-end testing #69278 on a cluster with two S3 vaults: the forward check ran every minute, while the inverted check was skipped on every run. There is also a smaller inconsistency in the same function. In `check_segment_file`, a segment whose rowset no longer exists is reported as a leak (`return 1`) only on a tablet-cache miss. On a cache hit (the previous object belonged to the same tablet), the same situation returns `-1` ("rowset not exists"). That is counted as a check error rather than a leak, so whether a leak is reported depends on listing order. ### What You Expected? The inverted check runs for every storage vault of an instance and reports each leaked object exactly once. ### How to Reproduce? 1. Cloud mode, recycler with `enable_checker = true` and `enable_inverted_check = true`. 2. Create two S3 storage vaults, a table on each, and load some data. 3. Put an orphan object under one vault, e.g. `<prefix>/data/<tablet_id>/<unknown_rowset_id>_0.dat`. 4. The recycler log shows `currently not support inverted check for multi accessor` on every round, and the orphan is never reported. ### Anything Else? The guard was added in #33280 ("Support hdfs accessor in Checker") without a stated reason. The loop below it already iterates over all accessors, and nothing in the check seems to depend on a single accessor: objects are matched to rowset meta by tablet id and rowset id only. @plat1ko, was there a specific concern behind it? For example, resources backed by the same bucket and prefix being listed twice, or the listing cost. Proposed fix (PR to follow): - remove the guard; - list each distinct storage location (accessor `uri()`) once, so resources sharing a bucket and prefix are not scanned or counted twice; - report a leak on the cache-hit path too. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
