morningman opened a new pull request, #69558:
URL: https://github.com/apache/doris/pull/69558
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary:
**In short.** With a metadata-cache weight limit set, the cached metadata of
a wide table (thousands of columns), a long name list or a long partition list
was never cached: its size estimate gave up and the value was reloaded on every
statement. This PR weighs such values one element at a time, so they are
admitted and charged for their size, while the safety bound of the estimate
stays in place.
**Background.** Master (#67726) lets the metadata caches of external
catalogs be bounded by weight: the FE-global `external_meta_cache_max_weight`,
a catalog's `meta.cache.max-weight`, or an entry's
`meta.cache.<engine>.<entry>.max-weight`. Once any of them applies, each value
is weighed before it is admitted. Several entries use the complete walk of
`ReflectiveObjectSizeEstimator`, which visits every object of the value and
gives up after 10,000 of them, so that one pathological value cannot cost
unbounded CPU and temporary memory. An estimate that gives up is incomplete,
and an incomplete value is returned to the query but not cached. branch-4.x
(#66717) governs only a few entries and weighs them with per-element formulas,
so it has no such limit.
**The problem, and what it cost.** The 10,000-visit budget is reached by
values that are large only because of one list, long before they are large in
bytes. Measured on master, the first value that never enters a weighted cache:
| Entry | Value | Never cached from |
|---|---|---|
| `hms-table` | Hive table | 995 INT / 382 `struct<a:int,b:string>` columns |
| `paimon-table`, `paimon-schema-at` | Paimon schema | 2,497 INT / 908
`ROW<a INT, b STRING>` columns |
| fe-core `schema` (`default`, `doris` engines) | schema of any other
external catalog | 3,332 INT / 832 struct columns |
| `adbc-table-schema` | Arrow schema | 2,499 INT / 1,110 struct fields |
| `adbc-table-names`, `adbc-namespaces` | name list | 9,999 names |
| `max-compute-partition` | partition list | 1,249 three-level partitions |
For such a table every statement goes back to the source (an HMS call, a
Paimon schema file read, an ODPS partition listing, an ADBC metadata call), and
`weight_reject_count` in `information_schema.catalog_meta_cache_statistics`
keeps growing. In a unit test with a 128 MiB global limit, three reads of a
2,000-column HMS table loaded it three times and cached nothing.
**How this PR fixes it.**
- `ReflectiveObjectSizeEstimator.estimateCompletePerElement(root,
collections...)` walks `root` like the complete estimate, except that each
listed collection it reaches is counted as its own storage plus every element
estimated separately, each with its own 10,000-visit budget. An element that
appears in more than one of the listed collections is counted once.
`MetaCacheSizeEstimators.reflectivePerElement(...)` and
`reflectiveEachElement()` wrap it as cache estimators.
- The affected entries use it: the HMS table (columns, partition keys), the
Paimon table schema and the schema-at snapshot (fields), the fe-core schema
cache (`SchemaCacheValue.getColumnLists()`: schema, and for plugin tables also
partition and display columns), the ADBC table schema (fields) and name lists,
and the MaxCompute partition list.
- The visit budget still bounds every element and the rest of the value, so
a value that reaches a huge shared structure still fails fast, and the
temporary identity set holds one element's graph at a time instead of the whole
value's. The whole-value walk and its budget are unchanged for every other
entry.
- Objects that elements share (for example a scalar type singleton) are
counted once per element, so the weight is somewhat above the retained size
(measured 1.1x to 1.6x) and never below it. A plugin table's display columns
reuse its column objects and are not counted twice.
**Results.**
| Value, under a weight limit | Before | After |
|---|---|---|
| 3,000-column HMS table (INT or struct columns) | estimate incomplete,
reloaded on every read | complete, loaded once, `weight_reject_count = 0` |
| 4,000-column Paimon table, INT or ROW columns; 4,000-field schema-at
snapshot | incomplete | complete, loaded once |
| fe-core schema with 4,000 INT + 1,000 struct columns | incomplete |
complete, loaded once |
| 3,000-field ADBC schema; 20,000 table names | incomplete | complete,
loaded once |
| 10,000 MaxCompute partitions | incomplete | complete, loaded once |
| single element larger than the visit budget | incomplete | incomplete
(unchanged) |
The new tests also check that the estimate is never below an unbounded walk
of the value and at most twice it, and that it doubles when the column count
doubles. With the per-element walk disabled, all nine new wide-value tests
fail; without the element de-duplication, a plugin schema with reused display
columns weighs twice as much (2,052,208 to 4,104,168 bytes) and its test fails.
### Release note
Fix: with a metadata-cache weight limit set, the schema of a table with
thousands of columns (HMS, Paimon, ADBC and other external catalogs), a long
ADBC name list and a long MaxCompute partition list are cached instead of being
loaded again by every statement.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
New and changed tests: `ReflectiveObjectSizeEstimatorTest` (4 new),
`HmsCacheSizeEstimatorTest` (new, 2), `PaimonCacheSizeEstimatorTest`,
`PaimonSchemaAtMemoTest`, `AdbcMetadataCacheTest`,
`MaxComputePartitionCacheTest` (1 new each), `ExternalCatalogMetaCacheTest` (2
new). All tests of fe-connector-cache (164), hms (163), paimon (847), adbc
(191), maxcompute (157), hive (510), hudi (270) and iceberg (1689) pass, as do
14 fe-core metadata and schema cache test classes (204 tests). No regression
test: a weight limit and a table with thousands of columns are needed to see
the difference, and the unit tests drive the real cache admission path.
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
Wide values that a weighted metadata cache used to reject are admitted
and weighed as the sum of their elements.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
