wenzhenghu commented on issue #65779: URL: https://github.com/apache/doris/issues/65779#issuecomment-5132177887
### Additional tracked problem: O(n) retained ID-map cleanup A later PR #65126 review identified another incremental-update performance cost in the same external metadata cache design. The retained \ and \ indexes intentionally outlive names/object-cache eviction so cache-only replay, DROP, and rename can still clean up stale navigation state. When targeted invalidation has only a canonical local name and no hot object carrying its ID, the current implementation removes the mapping with \: - table DROP/rename scans every retained table ID in the database; - database DROP/rename scans every retained database ID in the catalog; - the scan runs while the relevant object-cache publication monitor is held, and table events also run under the database write lock. For N retained IDs, one targeted invalidation is O(N). A burst of M DROP/rename events is O(MN) and can approach O(N^2) churn when M scales with N. This is mainly relevant to very large catalogs/databases with substantial metadata churn. This differs from the master hot-object path, which could remove a known deterministic ID directly, but that path could skip cleanup when the object was already cold. PR #65126 favors correctness by preserving cache-only cleanup even after object eviction; the linear scan is the resulting performance trade-off, not a correctness defect. This will not be fixed in PR #65126 because adding a reverse local-name-to-ID index or safely carrying/recomputing the ID must remain ordered with the existing object/ID publication and invalidation protocol. That change needs broader concurrency and case/mapping-mode validation and would increase late-stage PR risk. The follow-up should: - make targeted table/database ID cleanup O(1), including object-cold cases; - preserve the current stale-load fencing and object/ID ordering guarantees; - preserve lower-case and remote/local name-mapping behavior; - avoid weakening cleanup for replayed, duplicate, or missing-object events; - add large retained-index complexity coverage for DROP and rename bursts. Candidate directions include maintaining a consistently published reverse \ index, carrying a verified ID through invalidation when available, or recomputing a deterministic ID only where its canonicalization contract is guaranteed. Review thread: https://github.com/apache/doris/pull/65126#discussion_r3681246479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
