CalvinKirs opened a new pull request, #67726: URL: https://github.com/apache/doris/pull/67726
### What problem does this PR solve? Related PR: https://github.com/apache/doris/pull/66717 (merged into `branch-4.1`) Problem Summary: Port external metadata cache memory governance to `master`, adapting it to the connector SPI and shared cache framework rather than cherry-picking the legacy 4.1 cache implementation unchanged. **Current merge status:** the latest fetched master (`1335022902a`) has conflicts with this tested head (`f684d9fe479`) in `IcebergConnector.java` and `PaimonConnector.java`. Upstream reconciliation and revalidation are pending; the results below apply to the current PR head before conflict resolution. Entry-count limits alone cannot control FE heap when external metadata values vary substantially in size. This change adds estimated retained-memory budgets for managed caches while keeping existing count-based behavior when no applicable memory limit is configured. #### Scope and design - Add a shared estimator/admission interface and FE-global, catalog-total, and logical-entry budgets. Physical caches representing one logical entry share its quota, including connector-owned sibling caches. - Integrate the core schema cache and managed Hive/HMS, Iceberg, and Paimon caches. Include Hudi's sibling HMS caches in aggregate accounting and statistics without adding an unrelated Hudi-specific quota namespace. - Estimate on load/publication, not on cache hits. Use type-specific estimators and bounded object-graph validation where applicable; incomplete validation rejects cache admission instead of reporting a falsely small weight. Shared infrastructure is excluded at ownership boundaries. - Use strong cache values with explicit reservation ownership and coordinated reclamation. Keep replacement/removal accounting generation-safe and release ownership when catalogs/connectors close. This does not promise a global LRU or cross-catalog fairness. - Isolate weighted Iceberg statement metadata from cached generations so lazy query-side metadata does not silently grow an admitted value. - Expose `MAX_WEIGHT`, `ESTIMATED_WEIGHT`, `WEIGHT_REJECT_COUNT`, and `LAST_WEIGHT_REJECT_REASON` through `information_schema.catalog_meta_cache_statistics`, with rate-limited rejection warnings. No optimizer rules, literal representations, or partition-item implementations are changed. #### Configuration and behavior ```properties # Optional FE-wide total, configured in fe.conf; requires restart. # 0 disables only the global quota. Byte units or a JVM max-heap percentage are supported. external_meta_cache_max_weight=0 ``` ```sql -- Catalog/entry limits also work with the global quota disabled. ALTER CATALOG lake SET PROPERTIES ('meta.cache.max-weight' = '1GB'); -- Optional per-entry limit; individual keys do not all need configuration. ALTER CATALOG lake SET PROPERTIES ('meta.cache.iceberg.table.max-weight' = '256MB'); ``` Catalog/entry property changes rebuild the affected cache owners without restarting FE. An oversized value, or one that cannot obtain budget after reclamation, is returned to the current request without being retained in the cache. Normal admission rejection does not fail the query. These are limits on **estimated retained cache memory**, not pre-load heap reservations or a guarantee against OOM while constructing a large value; active query objects and shared infrastructure are outside this budget. ### Release note Add configurable estimated-memory limits, dynamic catalog/entry quota changes, and memory usage/rejection statistics for managed external metadata caches. ### Check List (For Author) - Test - [x] Unit Test: final focused run passed **113 tests across 7 classes**, with 0 failures/errors/skips. Covers configuration, budgets, concurrent cache lifecycle, reflective estimation, Hudi HMS ownership, and Iceberg table isolation. - [x] Regression test: existing `external_table_p0/test_catalog_ddl` passed on a running cluster (1 suite, 0 failures/fatal errors/skips). - [x] Manual test: local integration suite executed with `run-regression-test.sh`, followed by a separate comparison run against automatically generated and inspected output (1 suite, 0 failures/fatal errors/skips). Actual Iceberg REST/MinIO data: 256 partitions / 8,192 rows. Verified catalog-only quotas with global quota disabled, `64MB -> 1B -> 64MB`, entry-local rejection/recovery, rejection observability, and identical query results with/without cache admission (`COUNT(*)=8192`, `SUM(id)=33550336`). - FE built and packaged with `build.sh --fe`; Checkstyle and `git diff --check` passed. A temporary FE-only guard avoided an unrelated missing native-thirdparty rebuild; that environment adjustment is not included in this PR. - **Validation boundary:** the integration cluster reused a 4.1.3 BE. Actual Iceberg scans passed, but that BE cannot scan the new system-table columns (`no match column ... MAX_WEIGHT`). Memory statistics were checked through the running FE's Thrift endpoint. Matching-master BE compilation/system-table end-to-end validation, full external-engine integration, and follower replay remain unverified by this local run. No new end-to-end performance claims are made. - Behavior changed: - [x] Yes. Configured memory quotas govern admission and expose rejection statistics; unconfigured caches preserve count-based behavior. - Does this need documentation? - [x] Yes. Related 4.1 usage documentation: https://github.com/apache/doris-website/pull/4061. Master-specific connector scope and statistics documentation should be aligned with this adaptation. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label if needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
