This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c4f15f35eb9 [improvement](fe) Add the branch-4.x weight budget columns
to catalog_meta_cache_statistics (#69557)
c4f15f35eb9 is described below
commit c4f15f35eb9b3d5391b454bd395c67cd10b5ecee
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sun Oct 25 08:04:34 2026 +0000
[improvement](fe) Add the branch-4.x weight budget columns to
catalog_meta_cache_statistics (#69557)
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary:
**In short.** This brings the weight columns of
`information_schema.catalog_meta_cache_statistics` from branch-4.x
#66717 to master. Master shows four of the ten: this PR adds
`WEIGHT_BOUNDED`, `EVICTION_WEIGHT`, `CATALOG_MAX_WEIGHT`,
`CATALOG_ESTIMATED_WEIGHT`, `GLOBAL_MAX_WEIGHT` and
`GLOBAL_ESTIMATED_WEIGHT` with the 4.x names, types and meanings, and
makes a cache entry without a weight bound report `-1` in every weight
column, as 4.x does, instead of `0` in `ESTIMATED_WEIGHT` and
`WEIGHT_REJECT_COUNT`.
**Background.** External catalogs can bound the memory of their metadata
caches: an FE-global limit (`external_meta_cache_max_weight`), a catalog
limit (`meta.cache.max-weight`) and per-entry limits.
`catalog_meta_cache_statistics` is how users watch those caches. On
branch-4.x (#66717) the view tells, per entry, whether it is weight
bounded, what it holds and has evicted, and how full its catalog and the
FE-global budget are. Master (#67726) re-implemented the governance on
the connector cache framework and exposed only `MAX_WEIGHT`,
`ESTIMATED_WEIGHT`, `WEIGHT_REJECT_COUNT` and
`LAST_WEIGHT_REJECT_REASON`.
**The problem, and what it cost.** Monitoring queries written for 4.x
fail on master with an unknown column as soon as they name one of the
other six, and so do the weight scenarios of 4.x's
`test_paimon_table_meta_cache` and `test_iceberg_table_meta_cache`.
There is also no way on master to see how much of a catalog or of the
FE-global budget is used, or how many bytes were evicted to make room.
And an entry without a weight bound reports `ESTIMATED_WEIGHT = 0` and
`WEIGHT_REJECT_COUNT = 0`, which reads like an empty but bounded cache.
**How this PR fixes it.**
- `MetaCacheBudgetManager.EntryBudget` exposes the catalog limit the
entry is admitted under (the smaller of the catalog and FE-global
limits, as the admission check uses it), the bytes charged to its
catalog, the FE-global limit and the bytes charged to it. A limit that
does not exist is `-1`.
- `ScopedMetaCache` adds the bytes each value was admitted with to an
eviction-weight counter when it removes the value by eviction, by
Caffeine (capacity or expiry) or to make room in the budget. The bytes
are recorded when the value is admitted, because an exact invalidation
can release the reservation of an evicted value before the eviction is
accounted. Invalidation and replacement do not count.
- `ScopedMetaCache.CacheMetrics`, `MetaCacheEntryStats`, `SchemaTable`
and `MetadataGenerator` carry the six values. The columns are appended
after `LAST_WEIGHT_REJECT_REASON`. For an entry without a weight bound,
every weight column is `-1`.
- The BE scanner knows the six columns. An FE that does not know them
rejects the request, and the scanner then asks for the leading 28
columns, then 24, as it already did for the four earlier weight columns.
`SELECT *` positions still differ from 4.x, because master's base
columns also have `EVICTION_RATE`; queries that name their columns now
work on both.
**Results.**
| | master | this PR |
|---|---|---|
| `SELECT weight_bounded, catalog_max_weight ... FROM
catalog_meta_cache_statistics` | fails: unknown column | works, as on
4.x |
| Entry of a catalog with `meta.cache.max-weight = 128MB` | `MAX_WEIGHT
= 134217728` only | also `CATALOG_MAX_WEIGHT = 134217728`, the catalog's
and the FE-global charges, the evicted bytes |
| Entry without any weight bound | `ESTIMATED_WEIGHT = 0`,
`WEIGHT_REJECT_COUNT = 0` | `-1` in every weight column |
| New BE, FE without the six columns | n/a | the BE falls back to the
leading 28 columns |
Tests:
- `MetaCacheWeightGovernanceTest`: an entry without a weight bound
reports `-1` everywhere; the catalog and FE-global limits and charges of
bounded entries, with and without an FE-global limit; a budget reclaim
and a capacity eviction add the evicted reservation to the eviction
weight, an invalidation does not. An eviction keeps the bytes it was
admitted with when the key is invalidated before the deferred eviction
is accounted. Restoring master's `0` for unbounded entries, or dropping
the eviction accounting, turns these tests red.
- `FeMetaCacheEntryTest` checks the fe-core view of an unbounded and of
a bounded entry; `HudiConnectorHmsCacheTest` now expects `-1` for its
unbounded entry.
- Regression: `test_paimon_table_meta_cache` gets the weight scenario of
4.x on master's entry names (a catalog with `meta.cache.max-weight =
128MB` next to an unbounded one); `test_iceberg_table_meta_cache` puts
its first catalog under a catalog limit and checks the budget hierarchy
of its bounded entries, as 4.x does; the new
`test_remote_doris_meta_cache_weight` covers the `doris` engine without
any external service.
- Manually: a BE from this PR against an FE without the six columns
returns the rows (the FE rejects the 34 columns, the BE retries with
28).
Co-authored-by: Calvin Kirs <[email protected]>
---
.../schema_catalog_meta_cache_stats_scanner.cpp | 27 ++++-
.../connector/cache/MetaCacheBudgetManager.java | 20 ++++
.../doris/connector/cache/ScopedMetaCache.java | 57 ++++++++++-
.../cache/MetaCacheWeightGovernanceTest.java | 111 +++++++++++++++++++++
.../connector/hudi/HudiConnectorHmsCacheTest.java | 3 +-
.../java/org/apache/doris/catalog/SchemaTable.java | 8 ++
.../datasource/metacache/MetaCacheEntryStats.java | 50 +++++++++-
.../doris/tablefunction/MetadataGenerator.java | 10 ++
.../datasource/metacache/FeMetaCacheEntryTest.java | 16 +++
.../paimon/test_paimon_table_meta_cache.out | 15 +++
.../test_remote_doris_meta_cache_weight.out | 9 ++
.../iceberg/test_iceberg_table_meta_cache.groovy | 24 ++++-
.../paimon/test_paimon_table_meta_cache.groovy | 71 +++++++++++++
.../test_remote_doris_meta_cache_weight.groovy | 96 ++++++++++++++++++
14 files changed, 506 insertions(+), 11 deletions(-)
diff --git
a/be/src/information_schema/schema_catalog_meta_cache_stats_scanner.cpp
b/be/src/information_schema/schema_catalog_meta_cache_stats_scanner.cpp
index 7dde0c92e0d..8abe2b4e4b3 100644
--- a/be/src/information_schema/schema_catalog_meta_cache_stats_scanner.cpp
+++ b/be/src/information_schema/schema_catalog_meta_cache_stats_scanner.cpp
@@ -17,6 +17,9 @@
#include "information_schema/schema_catalog_meta_cache_stats_scanner.h"
+#include <algorithm>
+#include <array>
+
#include "core/block/block.h"
#include "core/data_type/data_type_factory.hpp"
#include "core/string_ref.h"
@@ -57,9 +60,17 @@ std::vector<SchemaScanner::ColumnDesc>
SchemaCatalogMetaCacheStatsScanner::_s_tb
{"ESTIMATED_WEIGHT", TYPE_BIGINT, sizeof(int64_t), true},
{"WEIGHT_REJECT_COUNT", TYPE_BIGINT, sizeof(int64_t), true},
{"LAST_WEIGHT_REJECT_REASON", TYPE_STRING, sizeof(StringRef), true},
+ {"WEIGHT_BOUNDED", TYPE_BOOLEAN, sizeof(bool), true},
+ {"EVICTION_WEIGHT", TYPE_BIGINT, sizeof(int64_t), true},
+ {"CATALOG_MAX_WEIGHT", TYPE_BIGINT, sizeof(int64_t), true},
+ {"CATALOG_ESTIMATED_WEIGHT", TYPE_BIGINT, sizeof(int64_t), true},
+ {"GLOBAL_MAX_WEIGHT", TYPE_BIGINT, sizeof(int64_t), true},
+ {"GLOBAL_ESTIMATED_WEIGHT", TYPE_BIGINT, sizeof(int64_t), true},
};
-static constexpr size_t kLegacyMetaCacheStatsColumnCount = 24;
+// The leading columns that older FE versions know, newest first: the first
weight columns came before the
+// budget hierarchy columns, and the base columns before any weight column.
+static constexpr std::array<size_t, 2> kOlderFeMetaCacheStatsColumnCounts =
{28, 24};
SchemaCatalogMetaCacheStatsScanner::SchemaCatalogMetaCacheStatsScanner()
: SchemaScanner(_s_tbls_columns,
TSchemaTableType::SCH_CATALOG_META_CACHE_STATISTICS) {}
@@ -111,8 +122,13 @@ Status
SchemaCatalogMetaCacheStatsScanner::_get_meta_cache_from_fe() {
return status;
}
Status first_status = status;
- result = TFetchSchemaTableDataResult();
- status = _fetch_from_fe(kLegacyMetaCacheStatsColumnCount, &result,
&fe_rejected);
+ for (size_t column_count : kOlderFeMetaCacheStatsColumnCounts) {
+ result = TFetchSchemaTableDataResult();
+ status = _fetch_from_fe(column_count, &result, &fe_rejected);
+ if (status.ok() || !fe_rejected) {
+ break;
+ }
+ }
if (!status.ok()) {
LOG(WARNING) << "fetch catalog meta cache stats from FE(" <<
_fe_addr.hostname
<< ") failed, errmsg=" << first_status;
@@ -134,7 +150,10 @@ Status
SchemaCatalogMetaCacheStatsScanner::_get_meta_cache_from_fe() {
size_t col_size = _s_tbls_columns.size();
if (!result_data.empty()) {
col_size = result_data[0].column_value.size();
- if (col_size != _s_tbls_columns.size() && col_size !=
kLegacyMetaCacheStatsColumnCount) {
+ if (col_size != _s_tbls_columns.size() &&
+ std::find(kOlderFeMetaCacheStatsColumnCounts.begin(),
+ kOlderFeMetaCacheStatsColumnCounts.end(),
+ col_size) == kOlderFeMetaCacheStatsColumnCounts.end()) {
return Status::InternalError<false>(
"catalog meta cache stats schema is not match for FE and
BE");
}
diff --git
a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheBudgetManager.java
b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheBudgetManager.java
index 40b22ec76c7..ea285a75d8c 100644
---
a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheBudgetManager.java
+++
b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheBudgetManager.java
@@ -530,6 +530,26 @@ public final class MetaCacheBudgetManager {
}
}
+ /** The catalog limit this entry is admitted under: the smaller of the
catalog and FE-global limits, or -1. */
+ public long getCatalogMaxWeight() {
+ return catalogBucket.maxWeight == Long.MAX_VALUE ? -1L :
catalogBucket.maxWeight;
+ }
+
+ public long getCatalogUsedWeight() {
+ synchronized (manager.lock) {
+ return catalogBucket.usedWeight;
+ }
+ }
+
+ /** The FE-global limit; -1 if none. */
+ public long getGlobalMaxWeight() {
+ return manager.globalMaxWeight.orElse(-1L);
+ }
+
+ public long getGlobalUsedWeight() {
+ return manager.getGlobalUsedWeight();
+ }
+
@Override
public void close() {
manager.close(this);
diff --git
a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/ScopedMetaCache.java
b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/ScopedMetaCache.java
index b58ad13bd9d..7dfaf88eda9 100644
---
a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/ScopedMetaCache.java
+++
b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/ScopedMetaCache.java
@@ -91,6 +91,7 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
private final LongAdder loadFailureCount = new LongAdder();
private final LongAdder totalLoadTimeNanos = new LongAdder();
private final LongAdder evictionCount = new LongAdder();
+ private final AtomicLong evictionWeight = new AtomicLong();
private final LongAdder invalidateCount = new LongAdder();
private final AtomicReference<Long> lastLoadSuccessTimeMs = new
AtomicReference<>(-1L);
private final AtomicReference<Long> lastLoadFailureTimeMs = new
AtomicReference<>(-1L);
@@ -576,8 +577,13 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
lastError.get(),
weightBounded,
entryBudget == null ? -1L :
entryBudget.getEffectiveMaxWeight(),
- entryBudget == null ? 0L : entryBudget.getUsedWeight(),
- weightRejectCount.sum(),
+ entryBudget == null ? -1L : entryBudget.getUsedWeight(),
+ entryBudget == null ? -1L : evictionWeight.get(),
+ entryBudget == null ? -1L : weightRejectCount.sum(),
+ entryBudget == null ? -1L :
entryBudget.getCatalogMaxWeight(),
+ entryBudget == null ? -1L :
entryBudget.getCatalogUsedWeight(),
+ entryBudget == null ? -1L :
entryBudget.getGlobalMaxWeight(),
+ entryBudget == null ? -1L :
entryBudget.getGlobalUsedWeight(),
lastWeightRejectReason.get()));
}
@@ -826,7 +832,7 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
for (Map.Entry<K, VersionedValue<K, V>> candidate :
coldest.entrySet()) {
VersionedValue<K, V> current =
data.getIfPresent(candidate.getKey());
if (current == candidate.getValue()) {
- long weight = current.registration.reservation.getBytes();
+ long weight = current.registration.admittedBytes;
budgetEvictionRegistration.set(current.registration);
try {
if (data.asMap().remove(candidate.getKey(), current)) {
@@ -1144,6 +1150,11 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
VersionedValue<K, V> versioned, RemovalCause cause, boolean
budgetEviction) {
if (cause.wasEvicted() || budgetEviction) {
evictionCount.increment();
+ if (weightBounded) {
+ // Releasing a reservation zeroes it, and an exact
invalidation can release the evicted generation
+ // before this deferred removal runs, so count the bytes the
generation was admitted with.
+
evictionWeight.accumulateAndGet(versioned.registration.admittedBytes,
JvmSizeUtils::saturatedAdd);
+ }
} else if (cause == RemovalCause.EXPLICIT) {
invalidateCount.increment();
}
@@ -1298,7 +1309,12 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
private final boolean weightBounded;
private final long maxWeight;
private final long estimatedWeight;
+ private final long evictionWeight;
private final long weightRejectCount;
+ private final long catalogMaxWeight;
+ private final long catalogEstimatedWeight;
+ private final long globalMaxWeight;
+ private final long globalEstimatedWeight;
private final String lastWeightRejectReason;
private CacheMetrics(
@@ -1322,7 +1338,12 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
boolean weightBounded,
long maxWeight,
long estimatedWeight,
+ long evictionWeight,
long weightRejectCount,
+ long catalogMaxWeight,
+ long catalogEstimatedWeight,
+ long globalMaxWeight,
+ long globalEstimatedWeight,
String lastWeightRejectReason) {
this.physicalEntryCount = physicalEntryCount;
this.keyNodeCount = keyNodeCount;
@@ -1344,7 +1365,12 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
this.weightBounded = weightBounded;
this.maxWeight = maxWeight;
this.estimatedWeight = estimatedWeight;
+ this.evictionWeight = evictionWeight;
this.weightRejectCount = weightRejectCount;
+ this.catalogMaxWeight = catalogMaxWeight;
+ this.catalogEstimatedWeight = catalogEstimatedWeight;
+ this.globalMaxWeight = globalMaxWeight;
+ this.globalEstimatedWeight = globalEstimatedWeight;
this.lastWeightRejectReason = lastWeightRejectReason;
}
@@ -1428,10 +1454,31 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
return estimatedWeight;
}
+ /** Bytes of the values removed by eviction, by Caffeine or to make
room in the budget; -1 if unbounded. */
+ public long getEvictionWeight() {
+ return evictionWeight;
+ }
+
public long getWeightRejectCount() {
return weightRejectCount;
}
+ public long getCatalogMaxWeight() {
+ return catalogMaxWeight;
+ }
+
+ public long getCatalogEstimatedWeight() {
+ return catalogEstimatedWeight;
+ }
+
+ public long getGlobalMaxWeight() {
+ return globalMaxWeight;
+ }
+
+ public long getGlobalEstimatedWeight() {
+ return globalEstimatedWeight;
+ }
+
public String getLastWeightRejectReason() {
return lastWeightRejectReason;
}
@@ -1557,6 +1604,9 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
/** Lightweight ownership token: deliberately does not retain the cached
value. */
private static final class Registration {
private final MetaCacheBudgetManager.AdmissionReservation reservation;
+ // Read when the generation is created, right after its reservation is
admitted: releasing the
+ // reservation zeroes its bytes, while eviction accounting still needs
them.
+ private final long admittedBytes;
private final CacheAddress address;
private final ScopeSnapshot scopeSnapshot;
private final AtomicBoolean released = new AtomicBoolean(false);
@@ -1564,6 +1614,7 @@ public final class ScopedMetaCache<K, V> implements
AutoCloseable {
private Registration(MetaCacheBudgetManager.AdmissionReservation
reservation,
CacheAddress address, ScopeSnapshot scopeSnapshot) {
this.reservation = reservation;
+ this.admittedBytes = reservation == null ? 0L :
reservation.getBytes();
this.address = address;
this.scopeSnapshot = scopeSnapshot;
}
diff --git
a/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheWeightGovernanceTest.java
b/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheWeightGovernanceTest.java
index 59acec37a20..5f8daa712cf 100644
---
a/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheWeightGovernanceTest.java
+++
b/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheWeightGovernanceTest.java
@@ -34,6 +34,7 @@ import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
@@ -243,6 +244,105 @@ class MetaCacheWeightGovernanceTest {
Assertions.assertEquals(2048L,
entry.metrics().getEstimatedWeight());
Assertions.assertEquals(1L,
entry.metrics().getWeightRejectCount());
Assertions.assertEquals(1L, entry.metrics().getEvictionCount());
+ // The value reclaimed for the budget counts with the reservation
it held.
+ Assertions.assertEquals(1024L,
entry.metrics().getEvictionWeight());
+ }
+ }
+
+ @Test
+ void capacityEvictionCountsTheEvictedReservationButInvalidationDoesNot() {
+ MetaCacheBudgetManager manager = new
MetaCacheBudgetManager(OptionalLong.of(4096L));
+ try (CatalogMetaCache owner = new CatalogMetaCache(
+ manager, 25L, "iceberg", Collections.emptyMap())) {
+ MetaCache<String, String> entry = owner.create(MetaCacheDefinition
+ .<String, String>builder("table", CacheSpec.of(true, -1L,
1L),
+ ignored -> ScopePath.catalog())
+ .sizeEstimator((key, value) ->
MetaCacheSizeEstimate.complete(256L))
+ .build());
+
+ entry.put("one", "value");
+ entry.put("two", "value");
+ awaitReclaimed(entry, 1L, ENTRY_OVERHEAD + 256L);
+ Assertions.assertEquals(1L, entry.metrics().getEvictionCount());
+ Assertions.assertEquals(ENTRY_OVERHEAD + 256L,
entry.metrics().getEvictionWeight());
+
+ // Either key may be the one that was evicted; invalidating the
survivor is not an eviction.
+ entry.invalidateKey("one");
+ entry.invalidateKey("two");
+ Assertions.assertEquals(0L, entry.metrics().getEstimatedWeight());
+ Assertions.assertEquals(ENTRY_OVERHEAD + 256L,
entry.metrics().getEvictionWeight());
+ }
+ }
+
+ @Test
+ void evictionWeightKeepsTheAdmittedBytesWhenInvalidationReleasesFirst() {
+ MetaCacheBudgetManager manager = new
MetaCacheBudgetManager(OptionalLong.of(4096L));
+ MetaCacheBudgetManager.EntryBudget budget = manager.createEntryBudget(
+ 28L, "iceberg", "table", "table", OptionalLong.empty(),
OptionalLong.empty());
+ try (ScopedMetaCacheRegistry registry = new ScopedMetaCacheRegistry())
{
+ ScopedMetaCache<String, String> cache =
registry.createCacheWithMetaRemovalListener(
+ "table", CacheSpec.of(true, -1L, 1L), null, null, null,
null,
+ (key, value) -> MetaCacheSizeEstimate.complete(256L),
budget);
+ cache.put("one", ScopePath.catalog(), "value");
+ AtomicBoolean evictedInsideTheDeferral = new AtomicBoolean();
+ // forEach defers the removals its callback causes, so the
capacity eviction of either key is drained
+ // only after both exact invalidations have released the
reservations, which zeroes them.
+ cache.forEach((key, value) -> {
+ if (evictedInsideTheDeferral.compareAndSet(false, true)) {
+ cache.put("two", ScopePath.catalog(), "value");
+ cache.invalidateKey("one");
+ cache.invalidateKey("two");
+ }
+ });
+
+ Assertions.assertTrue(evictedInsideTheDeferral.get());
+ Assertions.assertEquals(1L, cache.metrics().getEvictionCount());
+ Assertions.assertEquals(ENTRY_OVERHEAD + 256L,
cache.metrics().getEvictionWeight(),
+ "eviction must count the bytes the evicted generation was
admitted with");
+ Assertions.assertEquals(0L, manager.getGlobalUsedWeight());
+ }
+ }
+
+ @Test
+ void weightMetricsReportTheCatalogAndGlobalBudgetsOfTheEntry() {
+ MetaCacheBudgetManager manager = new
MetaCacheBudgetManager(OptionalLong.of(8192L));
+ try (CatalogMetaCache owner = new CatalogMetaCache(manager, 26L,
"paimon",
+
Collections.singletonMap(MetaCacheBudgetManager.CATALOG_MAX_WEIGHT_PROPERTY,
"4096"));
+ CatalogMetaCache unlimitedCatalog = new CatalogMetaCache(
+ manager, 27L, "paimon", Collections.emptyMap())) {
+ MetaCache<String, String> table = weightedStringEntry(owner,
"table", 512L);
+ MetaCache<String, String> snapshot = weightedStringEntry(owner,
"snapshot", 256L);
+ MetaCache<String, String> other =
weightedStringEntry(unlimitedCatalog, "table", 512L);
+ table.put("t", "value");
+ snapshot.put("s", "value");
+ other.put("o", "value");
+
+ ScopedMetaCache.CacheMetrics metrics = table.metrics();
+ Assertions.assertTrue(metrics.isWeightBounded());
+ Assertions.assertEquals(4096L, metrics.getMaxWeight());
+ Assertions.assertEquals(1024L, metrics.getEstimatedWeight());
+ Assertions.assertEquals(0L, metrics.getEvictionWeight());
+ Assertions.assertEquals(4096L, metrics.getCatalogMaxWeight());
+ Assertions.assertEquals(1024L + 768L,
metrics.getCatalogEstimatedWeight());
+ Assertions.assertEquals(8192L, metrics.getGlobalMaxWeight());
+ Assertions.assertEquals(1024L + 768L + 1024L,
metrics.getGlobalEstimatedWeight());
+ // Without a catalog limit, the catalog level is bounded by the
FE-global one.
+ Assertions.assertEquals(8192L,
other.metrics().getCatalogMaxWeight());
+ Assertions.assertEquals(1024L,
other.metrics().getCatalogEstimatedWeight());
+ }
+ }
+
+ @Test
+ void weightMetricsHaveNoGlobalMaximumWithoutAGlobalLimit() {
+ MetaCacheBudgetManager manager = new
MetaCacheBudgetManager(OptionalLong.empty());
+ try (CatalogMetaCache owner = new CatalogMetaCache(manager, 28L,
"iceberg",
+
Collections.singletonMap(MetaCacheBudgetManager.CATALOG_MAX_WEIGHT_PROPERTY,
"4096"))) {
+ MetaCache<String, String> entry = weightedStringEntry(owner,
"table", 512L);
+ entry.put("key", "value");
+
+ Assertions.assertEquals(4096L,
entry.metrics().getCatalogMaxWeight());
+ Assertions.assertEquals(-1L, entry.metrics().getGlobalMaxWeight());
+ Assertions.assertEquals(1024L,
entry.metrics().getGlobalEstimatedWeight());
}
}
@@ -329,6 +429,17 @@ class MetaCacheWeightGovernanceTest {
Assertions.assertEquals("value", entry.getIfPresent("key"));
Assertions.assertFalse(entry.isWeightBounded());
Assertions.assertEquals(0, estimates.get());
+ // An entry without a weight limit has no weight to report.
+ ScopedMetaCache.CacheMetrics metrics = entry.metrics();
+ Assertions.assertFalse(metrics.isWeightBounded());
+ Assertions.assertEquals(-1L, metrics.getMaxWeight());
+ Assertions.assertEquals(-1L, metrics.getEstimatedWeight());
+ Assertions.assertEquals(-1L, metrics.getEvictionWeight());
+ Assertions.assertEquals(-1L, metrics.getWeightRejectCount());
+ Assertions.assertEquals(-1L, metrics.getCatalogMaxWeight());
+ Assertions.assertEquals(-1L, metrics.getCatalogEstimatedWeight());
+ Assertions.assertEquals(-1L, metrics.getGlobalMaxWeight());
+ Assertions.assertEquals(-1L, metrics.getGlobalEstimatedWeight());
}
}
diff --git
a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorHmsCacheTest.java
b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorHmsCacheTest.java
index cda50755001..26a814711bc 100644
---
a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorHmsCacheTest.java
+++
b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorHmsCacheTest.java
@@ -130,7 +130,8 @@ public class HudiConnectorHmsCacheTest {
cache.listPartitionNames("db", "t", -1);
cache.listPartitionNames("db", "t", -1);
Assertions.assertEquals(1, delegate.cachedCalls);
- Assertions.assertEquals(0L, entry.metrics().getEstimatedWeight());
+ // An entry without a weight limit has no weight to report.
+ Assertions.assertEquals(-1L, entry.metrics().getEstimatedWeight());
}
// ── wrap
─────────────────────────────────────────────────────────────────────────────────────────────
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
index 5b250467ae8..074a877cb1a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
@@ -660,6 +660,14 @@ public class SchemaTable extends Table {
.column("ESTIMATED_WEIGHT",
ScalarType.createType(PrimitiveType.BIGINT))
.column("WEIGHT_REJECT_COUNT",
ScalarType.createType(PrimitiveType.BIGINT))
.column("LAST_WEIGHT_REJECT_REASON",
ScalarType.createStringType())
+ // Appended last: the BE scanner falls
back to asking an older FE for only the
+ // leading columns.
+ .column("WEIGHT_BOUNDED",
ScalarType.createType(PrimitiveType.BOOLEAN))
+ .column("EVICTION_WEIGHT",
ScalarType.createType(PrimitiveType.BIGINT))
+ .column("CATALOG_MAX_WEIGHT",
ScalarType.createType(PrimitiveType.BIGINT))
+ .column("CATALOG_ESTIMATED_WEIGHT",
ScalarType.createType(PrimitiveType.BIGINT))
+ .column("GLOBAL_MAX_WEIGHT",
ScalarType.createType(PrimitiveType.BIGINT))
+ .column("GLOBAL_ESTIMATED_WEIGHT",
ScalarType.createType(PrimitiveType.BIGINT))
.build())
)
.put("backend_kerberos_ticket_cache",
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryStats.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryStats.java
index 05d71700600..c76c16f3192 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryStats.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryStats.java
@@ -34,6 +34,8 @@ import java.util.Objects;
*
* <p>For last-load timestamps, {@code -1} means no corresponding event
happened yet.
* {@code lastError} keeps the latest load failure message; empty string means
no failure recorded.
+ * The weight fields are {@code -1} for an entry that is not weight bounded,
and the catalog and global maximums
+ * are {@code -1} when that level has no limit.
*/
public final class MetaCacheEntryStats {
private final boolean configEnabled;
@@ -55,9 +57,15 @@ public final class MetaCacheEntryStats {
private final long lastLoadSuccessTimeMs;
private final long lastLoadFailureTimeMs;
private final String lastError;
+ private final boolean weightBounded;
private final long maxWeight;
private final long estimatedWeight;
+ private final long evictionWeight;
private final long weightAdmissionRejectedCount;
+ private final long catalogMaxWeight;
+ private final long catalogEstimatedWeight;
+ private final long globalMaxWeight;
+ private final long globalEstimatedWeight;
private final String lastWeightRejectReason;
/**
@@ -83,9 +91,15 @@ public final class MetaCacheEntryStats {
long lastLoadSuccessTimeMs,
long lastLoadFailureTimeMs,
String lastError,
+ boolean weightBounded,
long maxWeight,
long estimatedWeight,
+ long evictionWeight,
long weightAdmissionRejectedCount,
+ long catalogMaxWeight,
+ long catalogEstimatedWeight,
+ long globalMaxWeight,
+ long globalEstimatedWeight,
String lastWeightRejectReason) {
this.configEnabled = configEnabled;
this.effectiveEnabled = effectiveEnabled;
@@ -106,9 +120,15 @@ public final class MetaCacheEntryStats {
this.lastLoadSuccessTimeMs = lastLoadSuccessTimeMs;
this.lastLoadFailureTimeMs = lastLoadFailureTimeMs;
this.lastError = Objects.requireNonNull(lastError, "lastError");
+ this.weightBounded = weightBounded;
this.maxWeight = maxWeight;
this.estimatedWeight = estimatedWeight;
+ this.evictionWeight = evictionWeight;
this.weightAdmissionRejectedCount = weightAdmissionRejectedCount;
+ this.catalogMaxWeight = catalogMaxWeight;
+ this.catalogEstimatedWeight = catalogEstimatedWeight;
+ this.globalMaxWeight = globalMaxWeight;
+ this.globalEstimatedWeight = globalEstimatedWeight;
this.lastWeightRejectReason =
Objects.requireNonNull(lastWeightRejectReason, "lastWeightRejectReason");
}
@@ -125,8 +145,10 @@ public final class MetaCacheEntryStats {
metrics.getLoadSuccessCount(), metrics.getLoadFailureCount(),
metrics.getTotalLoadTimeNanos(),
loads == 0L ? 0D : (double) metrics.getTotalLoadTimeNanos() /
loads,
metrics.getEvictionCount(), metrics.getInvalidateCount(),
metrics.getLastLoadSuccessTimeMs(),
- metrics.getLastLoadFailureTimeMs(), metrics.getLastError(),
metrics.getMaxWeight(),
- metrics.getEstimatedWeight(), metrics.getWeightRejectCount(),
+ metrics.getLastLoadFailureTimeMs(), metrics.getLastError(),
metrics.isWeightBounded(),
+ metrics.getMaxWeight(), metrics.getEstimatedWeight(),
metrics.getEvictionWeight(),
+ metrics.getWeightRejectCount(), metrics.getCatalogMaxWeight(),
metrics.getCatalogEstimatedWeight(),
+ metrics.getGlobalMaxWeight(),
metrics.getGlobalEstimatedWeight(),
metrics.getLastWeightRejectReason());
}
@@ -228,6 +250,10 @@ public final class MetaCacheEntryStats {
return lastError;
}
+ public boolean isWeightBounded() {
+ return weightBounded;
+ }
+
public long getMaxWeight() {
return maxWeight;
}
@@ -236,10 +262,30 @@ public final class MetaCacheEntryStats {
return estimatedWeight;
}
+ public long getEvictionWeight() {
+ return evictionWeight;
+ }
+
public long getWeightAdmissionRejectedCount() {
return weightAdmissionRejectedCount;
}
+ public long getCatalogMaxWeight() {
+ return catalogMaxWeight;
+ }
+
+ public long getCatalogEstimatedWeight() {
+ return catalogEstimatedWeight;
+ }
+
+ public long getGlobalMaxWeight() {
+ return globalMaxWeight;
+ }
+
+ public long getGlobalEstimatedWeight() {
+ return globalEstimatedWeight;
+ }
+
public String getLastWeightRejectReason() {
return lastWeightRejectReason;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
index dc90bc915d0..fb42bde298a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
@@ -1984,6 +1984,16 @@ public class MetadataGenerator {
trow.addToColumnValue(new
TCell().setLongVal(entryStats.getEstimatedWeight()));
trow.addToColumnValue(new
TCell().setLongVal(entryStats.getWeightAdmissionRejectedCount()));
trow.addToColumnValue(new
TCell().setStringVal(entryStats.getLastWeightRejectReason()));
+ trow.addToColumnValue(new
TCell().setBoolVal(entryStats.isWeightBounded())); // WEIGHT_BOUNDED
+ trow.addToColumnValue(new
TCell().setLongVal(entryStats.getEvictionWeight())); // EVICTION_WEIGHT
+ trow.addToColumnValue(
+ new
TCell().setLongVal(entryStats.getCatalogMaxWeight())); // CATALOG_MAX_WEIGHT
+ trow.addToColumnValue(new TCell().setLongVal(
+ entryStats.getCatalogEstimatedWeight())); //
CATALOG_ESTIMATED_WEIGHT
+ trow.addToColumnValue(
+ new
TCell().setLongVal(entryStats.getGlobalMaxWeight())); // GLOBAL_MAX_WEIGHT
+ trow.addToColumnValue(new TCell().setLongVal(
+ entryStats.getGlobalEstimatedWeight())); //
GLOBAL_ESTIMATED_WEIGHT
dataBatch.add(trow);
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/FeMetaCacheEntryTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/FeMetaCacheEntryTest.java
index 22bbc80ad78..ba4673d8065 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/FeMetaCacheEntryTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/FeMetaCacheEntryTest.java
@@ -59,6 +59,16 @@ public class FeMetaCacheEntryTest {
Assertions.assertEquals(1L, stats.getLoadSuccessCount());
Assertions.assertTrue(stats.getRequestCount() >= 4L);
Assertions.assertTrue(stats.getInvalidateCount() >= 1L);
+ // Without a weight limit the entry reports -1 for every weight
figure.
+ Assertions.assertFalse(stats.isWeightBounded());
+ Assertions.assertEquals(-1L, stats.getMaxWeight());
+ Assertions.assertEquals(-1L, stats.getEstimatedWeight());
+ Assertions.assertEquals(-1L, stats.getEvictionWeight());
+ Assertions.assertEquals(-1L,
stats.getWeightAdmissionRejectedCount());
+ Assertions.assertEquals(-1L, stats.getCatalogMaxWeight());
+ Assertions.assertEquals(-1L, stats.getCatalogEstimatedWeight());
+ Assertions.assertEquals(-1L, stats.getGlobalMaxWeight());
+ Assertions.assertEquals(-1L, stats.getGlobalEstimatedWeight());
} finally {
executor.shutdownNow();
}
@@ -81,6 +91,12 @@ public class FeMetaCacheEntryTest {
Assertions.assertSame(oversized, mutation.get(3L,
TimeUnit.SECONDS));
Assertions.assertNull(entry.getIfPresent("key"));
+ MetaCacheEntryStats stats = entry.stats();
+ Assertions.assertTrue(stats.isWeightBounded());
+ Assertions.assertEquals(1_024L, stats.getMaxWeight());
+ Assertions.assertEquals(1L,
stats.getWeightAdmissionRejectedCount());
+ // The rejected mutation invalidated the old value; an
invalidation is not an eviction.
+ Assertions.assertEquals(0L, stats.getEvictionWeight());
} finally {
worker.shutdownNow();
refreshExecutor.shutdownNow();
diff --git
a/regression-test/data/external_table_p0/paimon/test_paimon_table_meta_cache.out
b/regression-test/data/external_table_p0/paimon/test_paimon_table_meta_cache.out
new file mode 100644
index 00000000000..96b6f02f9a5
--- /dev/null
+++
b/regression-test/data/external_table_p0/paimon/test_paimon_table_meta_cache.out
@@ -0,0 +1,15 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !paimon_weighted_entries --
+paimon-database true 134217728 134217728 -1 0
+paimon-latest-snapshot true 134217728 134217728 -1 0
+paimon-schema-at true 134217728 134217728 -1 0
+paimon-table true 134217728 134217728 -1 0
+paimon.partition-view true 134217728 134217728 -1 0
+
+-- !paimon_unbounded_entries --
+paimon-database false -1 -1 -1 -1 -1 -1
-1 -1
+paimon-latest-snapshot false -1 -1 -1 -1 -1 -1
-1 -1
+paimon-schema-at false -1 -1 -1 -1 -1 -1
-1 -1
+paimon-table false -1 -1 -1 -1 -1 -1 -1
-1
+paimon.partition-view false -1 -1 -1 -1 -1 -1
-1 -1
+
diff --git
a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_meta_cache_weight.out
b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_meta_cache_weight.out
new file mode 100644
index 00000000000..637072013f4
--- /dev/null
+++
b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_meta_cache_weight.out
@@ -0,0 +1,9 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !weighted_entries --
+backends true 134217728 134217728 -1 0
+schema true 134217728 134217728 -1 0
+
+-- !unbounded_entries --
+backends false -1 -1 -1 -1 -1 -1 -1
-1
+schema false -1 -1 -1 -1 -1 -1 -1 -1
+
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_table_meta_cache.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_table_meta_cache.groovy
index ed88421f687..4ccda9d31b7 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_table_meta_cache.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_table_meta_cache.groovy
@@ -28,7 +28,8 @@ suite("test_iceberg_table_meta_cache", "p0,external") {
String default_fs = "hdfs://${externalEnvIp}:${hdfs_port}"
String warehouse = "${default_fs}/warehouse"
- // 1. test default catalog
+ // 1. test a catalog-level memory bound without a global bound.
The existing
+ // create/insert/select/refresh flow below is the weighted-cache
happy path.
sql """drop catalog if exists ${catalog_name};"""
sql """
create catalog ${catalog_name} properties (
@@ -37,6 +38,7 @@ suite("test_iceberg_table_meta_cache", "p0,external") {
'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hmsPort}',
'fs.defaultFS' = '${default_fs}',
'warehouse' = '${warehouse}',
+ 'meta.cache.max-weight' = '128MB',
'meta.cache.iceberg.manifest.enable' = 'true'
);
"""
@@ -63,6 +65,26 @@ suite("test_iceberg_table_meta_cache", "p0,external") {
sql """refresh table test_iceberg_meta_cache_db.sales"""
// select 3 rows
sql """select * from test_iceberg_meta_cache_db.sales"""
+ // The weight-bounded entries expose their budget hierarchy in the
statistics view. Only the
+ // catalog is bounded, so it bounds every weighted entry, and the
catalog's charge covers the entry's.
+ def weightStats = sql """
+ select entry_name, max_weight, estimated_weight,
eviction_weight, catalog_max_weight,
+ catalog_estimated_weight, global_max_weight,
global_estimated_weight
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = "${catalog_name}" and engine_name =
"iceberg" and weight_bounded = true
+ order by entry_name;
+ """
+ assertTrue(weightStats.size() > 0)
+ for (row in weightStats) {
+ assertTrue((row[1] as long) > 0L)
+ assertTrue((row[2] as long) >= 0L)
+ assertTrue((row[3] as long) >= 0L)
+ assertEquals(row[1] as long, row[4] as long)
+ assertTrue((row[5] as long) >= (row[2] as long))
+ long globalMax = row[6] as long
+ assertTrue(globalMax == -1L || globalMax >= (row[4] as long))
+ assertTrue((row[7] as long) >= (row[5] as long))
+ }
sql """drop table test_iceberg_meta_cache_db.sales"""
// 2. test catalog with meta.cache.iceberg.table.ttl-second
diff --git
a/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
b/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
index e4c806bf26d..21f149c7d2b 100644
---
a/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
+++
b/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
@@ -27,10 +27,12 @@ suite("test_paimon_table_meta_cache", "p0,external") {
String catalogWithCache = "test_paimon_table_cache_with_cache"
String catalogNoCache = "test_paimon_table_cache_no_cache"
+ String catalogWeighted = "test_paimon_table_cache_weighted"
String testDb = "paimon_cache_test_db"
sql """drop catalog if exists ${catalogWithCache}"""
sql """drop catalog if exists ${catalogNoCache}"""
+ sql """drop catalog if exists ${catalogWeighted}"""
sql """
CREATE CATALOG ${catalogWithCache} PROPERTIES (
@@ -80,6 +82,19 @@ suite("test_paimon_table_meta_cache", "p0,external") {
exception "is wrong"
}
+ // A catalog-level memory bound puts the table handle and snapshot
projections under weight.
+ sql """
+ CREATE CATALOG ${catalogWeighted} PROPERTIES (
+ 'type' = 'paimon',
+ 'warehouse' = 's3://warehouse/wh',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.path.style.access' = 'true',
+ 'meta.cache.max-weight' = '128MB'
+ );
+ """
+
try {
spark_paimon "CREATE DATABASE IF NOT EXISTS paimon.${testDb}"
@@ -113,6 +128,61 @@ suite("test_paimon_table_meta_cache", "p0,external") {
def result3 = sql """select * from ${testDb}.test_insert order by id"""
assertEquals(2, result3.size())
+ // ==================== Test 1b: weight-bounded cache
====================
+ logger.info("========== Test 1b: weight-bounded cache ==========")
+ sql """switch ${catalogWeighted}"""
+ def resultWeighted = sql """select * from ${testDb}.test_insert order
by id"""
+ assertEquals(2, resultWeighted.size())
+ // desc/table-only paths admit the base table handle; the scan above
admits the snapshot.
+ sql """desc ${testDb}.test_insert"""
+ // With only a catalog limit, every weighted entry is bounded by it,
and nothing was rejected.
+ order_qt_paimon_weighted_entries """
+ select entry_name, weight_bounded, max_weight, catalog_max_weight,
global_max_weight,
+ weight_reject_count
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = "${catalogWeighted}" and engine_name =
"paimon" and weight_bounded = true
+ """
+ def weightStats = sql """
+ select entry_name, estimated_weight, eviction_weight,
catalog_estimated_weight,
+ global_estimated_weight
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = "${catalogWeighted}" and engine_name =
"paimon"
+ and entry_name in ("paimon-table", "paimon-latest-snapshot")
+ """
+ assertEquals(2, weightStats.size())
+ for (row in weightStats) {
+ // An admitted value is charged to its entry, its catalog and the
FE-global total.
+ assertTrue((row[1] as long) > 0L)
+ assertEquals(0L, row[2] as long)
+ assertTrue((row[3] as long) >= (row[1] as long))
+ assertTrue((row[4] as long) >= (row[3] as long))
+ }
+ // A catalog without any memory bound reports -1 for every weight
column.
+ order_qt_paimon_unbounded_entries """
+ select entry_name, weight_bounded, max_weight, estimated_weight,
eviction_weight,
+ weight_reject_count, catalog_max_weight,
catalog_estimated_weight, global_max_weight,
+ global_estimated_weight
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = "${catalogWithCache}" and engine_name =
"paimon"
+ """
+ // Refresh releases the table handle reservation with its projections;
the next scan
+ // re-admits both without rejections.
+ sql """refresh table ${testDb}.test_insert"""
+ def resultWeightedRefreshed = sql """select * from
${testDb}.test_insert order by id"""
+ assertEquals(2, resultWeightedRefreshed.size())
+ def weightStatsRefreshed = sql """
+ select entry_name, estimated_weight, weight_reject_count
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = "${catalogWeighted}" and engine_name =
"paimon"
+ and entry_name in ("paimon-table", "paimon-latest-snapshot")
+ """
+ assertEquals(2, weightStatsRefreshed.size())
+ for (row in weightStatsRefreshed) {
+ assertTrue((row[1] as long) > 0L)
+ assertEquals(0L, row[2] as long)
+ }
+ sql """switch ${catalogWithCache}"""
+
// ==================== Test 2: Schema Change (ADD COLUMN)
====================
logger.info("========== Test 2: Schema Change (ADD COLUMN) ==========")
spark_paimon "DROP TABLE IF EXISTS paimon.${testDb}.test_add_column"
@@ -150,5 +220,6 @@ suite("test_paimon_table_meta_cache", "p0,external") {
}
sql """drop catalog if exists ${catalogWithCache}"""
sql """drop catalog if exists ${catalogNoCache}"""
+ sql """drop catalog if exists ${catalogWeighted}"""
}
}
diff --git
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_meta_cache_weight.groovy
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_meta_cache_weight.groovy
new file mode 100644
index 00000000000..5fdf3f4b360
--- /dev/null
+++
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_meta_cache_weight.groovy
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_remote_doris_meta_cache_weight", "p0,external") {
+ String remote_doris_host =
context.config.otherConfigs.get("extArrowFlightSqlHost")
+ String remote_doris_user =
context.config.otherConfigs.get("extArrowFlightSqlUser")
+ String remote_doris_psw =
context.config.otherConfigs.get("extArrowFlightSqlPassword")
+
+ def showres = sql "show frontends"
+ String remote_doris_arrow_port = showres[0][6]
+ String remote_doris_http_port = showres[0][3]
+ String remote_doris_thrift_port = showres[0][5]
+
+ sql """DROP DATABASE IF EXISTS test_remote_doris_meta_cache_weight_db"""
+ sql """CREATE DATABASE IF NOT EXISTS
test_remote_doris_meta_cache_weight_db"""
+ sql """
+ CREATE TABLE
`test_remote_doris_meta_cache_weight_db`.`test_remote_doris_meta_cache_weight_t`
(
+ `id` INT,
+ `name` STRING
+ ) DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES ("replication_num" = "1");
+ """
+
+ // One catalog bounds its metadata caches by memory weight, the other one
has no bound.
+ sql """DROP CATALOG IF EXISTS `test_remote_doris_meta_cache_weighted`"""
+ sql """
+ CREATE CATALOG `test_remote_doris_meta_cache_weighted` PROPERTIES (
+ 'type' = 'doris',
+ 'fe_thrift_hosts' =
'${remote_doris_host}:${remote_doris_thrift_port}',
+ 'fe_http_hosts' =
'http://${remote_doris_host}:${remote_doris_http_port}',
+ 'fe_arrow_hosts' =
'${remote_doris_host}:${remote_doris_arrow_port}',
+ 'user' = '${remote_doris_user}',
+ 'password' = '${remote_doris_psw}',
+ 'use_arrow_flight' = 'true',
+ 'meta.cache.max-weight' = '128MB'
+ );
+ """
+ sql """DROP CATALOG IF EXISTS `test_remote_doris_meta_cache_unbounded`"""
+ sql """
+ CREATE CATALOG `test_remote_doris_meta_cache_unbounded` PROPERTIES (
+ 'type' = 'doris',
+ 'fe_thrift_hosts' =
'${remote_doris_host}:${remote_doris_thrift_port}',
+ 'fe_http_hosts' =
'http://${remote_doris_host}:${remote_doris_http_port}',
+ 'fe_arrow_hosts' =
'${remote_doris_host}:${remote_doris_arrow_port}',
+ 'user' = '${remote_doris_user}',
+ 'password' = '${remote_doris_psw}',
+ 'use_arrow_flight' = 'true'
+ );
+ """
+
+ // DESC loads the remote table's schema into the schema cache of each
catalog.
+ sql """DESC
test_remote_doris_meta_cache_weighted.test_remote_doris_meta_cache_weight_db.test_remote_doris_meta_cache_weight_t"""
+ sql """DESC
test_remote_doris_meta_cache_unbounded.test_remote_doris_meta_cache_weight_db.test_remote_doris_meta_cache_weight_t"""
+
+ // With only a catalog limit, each entry is bounded by it, and nothing was
rejected.
+ order_qt_weighted_entries """
+ select entry_name, weight_bounded, max_weight, catalog_max_weight,
global_max_weight, weight_reject_count
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = 'test_remote_doris_meta_cache_weighted' and
engine_name = 'doris'
+ """
+ // A catalog without any memory bound reports -1 for every weight column.
+ order_qt_unbounded_entries """
+ select entry_name, weight_bounded, max_weight, estimated_weight,
eviction_weight, weight_reject_count,
+ catalog_max_weight, catalog_estimated_weight,
global_max_weight, global_estimated_weight
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = 'test_remote_doris_meta_cache_unbounded' and
engine_name = 'doris'
+ """
+ // The admitted schema is charged to its entry, its catalog and the
FE-global total.
+ def schemaWeights = sql """
+ select estimated_weight, eviction_weight, catalog_estimated_weight,
global_estimated_weight
+ from internal.information_schema.catalog_meta_cache_statistics
+ where catalog_name = 'test_remote_doris_meta_cache_weighted' and
engine_name = 'doris'
+ and entry_name = 'schema'
+ """
+ assertEquals(1, schemaWeights.size())
+ long schemaWeight = schemaWeights[0][0] as long
+ assertTrue(schemaWeight > 0L)
+ assertEquals(0L, schemaWeights[0][1] as long)
+ assertTrue((schemaWeights[0][2] as long) >= schemaWeight)
+ assertTrue((schemaWeights[0][3] as long) >= (schemaWeights[0][2] as long))
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
