suxiaogang223 opened a new issue, #60686: URL: https://github.com/apache/doris/issues/60686
## Background The external table metadata caching layer has grown organically across engines (Hive, Iceberg, Paimon, Hudi, MaxCompute, etc.). Each engine has its own cache design with different structures, invalidation strategies, TTL semantics, and configuration formats. This makes it difficult to reason about cache behavior, maintain consistency, and add new engines. ## Goal Build a unified metadata cache framework for all external table engines, providing: - **Common abstractions**: a single set of interfaces for cache modules, engine adapters, and cache specifications that all engines implement - **Per-catalog isolation**: each catalog owns independent cache instances with independent configuration - **Consistent configuration**: unified property format and TTL/capacity semantics across all engines - **Unified invalidation**: catalog / database / table level invalidation through a common path - **Unified monitoring**: cache stats and metrics accessible through a single entry point - **Lazy loading**: defer expensive metadata loading (snapshots, schemas, file lists) until actually needed - **Schema convergence**: embed schema into table/snapshot cache where possible, reducing dependency on standalone schema cache ## Related PRs - #59056 — Add manifest-level cache for Iceberg tables to reduce I/O and parsing overhead - #59716 — Refactor Iceberg metadata cache structure and add table cache test cases - #60478 — Per-catalog Paimon metadata cache with two-level table+snapshot structure - (in progress) Unified meta cache framework with engine adapters -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
