yujun777 opened a new pull request, #70400: URL: https://github.com/apache/doris/pull/70400
### Background The FE loads missing column statistics asynchronously while planning queries. Column, partition column and histogram caches share a statistics fetch executor. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Distinct cache misses can enqueue unbounded statistics loads because incomplete Caffeine entries have zero eviction weight and the executor queue is effectively unbounded. Bound the shared loading queue and degrade rejected lookups to unknown statistics. Separate cache maintenance from SQL loading and handle rejected refresh and preheat submissions. ### Details - **Change map:** 137 added production lines and 28 removed production lines across three files; 174 lines of concurrency tests. - **Frequency:** When distinct statistics cache misses are submitted faster than the fetch workers can complete them. - **Root cause:** Pending futures are excluded from size-based eviction while the fetch executor accepts an effectively unbounded backlog. - **Why existing tests missed it:** Cache tests cover normal loading and eviction but do not saturate the loader queue. - **Fix:** Add a bounded shared executor with immediate rejection, preserve old values on rejected automatic refresh, invalidate the previous value on rejected explicit refresh, and tolerate rejected preheating during FE role transitions. Register a rejection counter and isolate cache maintenance. - **Other paths and branches:** Cover column, partition column and histogram caches. The shared BasicAsyncCacheLoader and external row-count cache retain their existing behavior. This change targets master. - **Behavior change:** Overloaded lookups may continue using unknown statistics until capacity becomes available. Sessions forbidding unknown column statistics can fail under these conditions. ### Testing Locally passed StatisticsCacheOverloadTest (5 tests), CacheTest (8 tests), and StatisticsCacheTest (6 tests) through tools/fast-fe-ut.sh and its run-fe-ut.sh fallback. Independently passed Checkstyle for fe-common and fe-core and git diff --check. No production workload or memory pressure benchmark was run. ### Release note FE statistics cache loading now has a bounded pending queue controlled by stats_fetch_queue_size (default 1000, effective after restarting FE). When saturated, new lookups use unknown statistics instead of accumulating an unbounded backlog. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; document the new FE configuration with the documentation follow-up. - Pre-submission status: Partially verified: targeted unit tests and Checkstyle passed; production load testing remains. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, and this description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
