This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ce317f49e90 [fix](fe) Read the latest Iceberg snapshot live when the
table cache is off (#68846)
ce317f49e90 is described below
commit ce317f49e9005387e2ad92c85f9abf0300377e1b
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Oct 10 16:45:25 2026 +0800
[fix](fe) Read the latest Iceberg snapshot live when the table cache is off
(#68846)
### What problem does this PR solve?
Issue Number: None
Related PR: #66717
Problem Summary:
**In short.** This brings master in line with branch-4.x (#66717) on
what turning off an Iceberg catalog's table cache means. On master,
`meta.cache.iceberg.table.enable=false` or
`meta.cache.iceberg.table.capacity=0` turns off only the table cache.
The latest-snapshot cache keeps pinning the snapshot id that an earlier
query cached, so queries keep returning old data after new commits for
up to 24 hours. With this PR the latest snapshot, its schema, the
inferred file format and the table comment are read live in that
configuration, as they already are for `ttl-second=0`.
**Background.** An Iceberg catalog's table metadata cache is configured
by `meta.cache.iceberg.table.enable`, `ttl-second` and `capacity`. On
branch-4.x the latest snapshot used to live inside the table cache
value; #66717 moved it into its own `snapshot` entry and mapped the
three `table` keys onto it, so they keep governing both. The schema
cache there is keyed by schema id, so it follows whichever snapshot is
pinned. On master the connector keeps several caches next to the table
cache that have no keys of their own: the latest-snapshot pin (snapshot
id and schema id per table), the file format inferred from the first
data file, and, for REST vended-credentials catalogs, the table comment.
All three are built from `ttl-second` alone, with a fixed capacity of
1000. The latest schema comes from the FE schema cache, which is keyed
by table name. `IcebergConnector.schemaCacheTtlSecondOverride` turns
that schema cache off only for `ttl-second<=0`.
**The problem, and what it cost.** With `table.enable=false` or
`table.capacity=0`, master loads the table live on every query but still
serves the snapshot id cached by an earlier query. That entry expires
only after it has gone unread for 24 hours, so a catalog that is queried
regularly can stay on one snapshot until someone runs REFRESH. The
latest schema comes from the name-keyed schema cache, so a column added
after it was cached stays hidden too. The comment of a
vended-credentials catalog is cached in the same way. An operator who
switched the table cache off to see external writes at once gets neither
the new data nor the new schema. In addition, `table.capacity` does not
limit these caches. This has been the case since the catalog moved to
the connector framework (#64304, #66633).
**How this PR fixes it.**
- `IcebergConnector` derives one spec from the table entry's `enable`,
`ttl-second` and `capacity` and builds the latest-snapshot, format and
comment caches from it. The cache classes take a `CacheSpec`, like
`IcebergTableCache` and `IcebergPartitionCache` already do; their `(ttl,
capacity)` constructors used by tests are kept.
- The table entry's `max-weight` is not passed on. It bounds only the
table cache, as on branch-4.x, where the snapshot entry inherits enable,
ttl and capacity but not the weight. The catalog-level and FE-global
weight limits still apply to these caches.
- `schemaCacheTtlSecondOverride` returns 0 when `enable=false` or
`capacity=0` turns the table entry off, as it already does for
`ttl-second=0`, so the latest schema is read live together with the
snapshot.
- Caches that have their own keys keep using them:
`meta.cache.iceberg.partition.*`, `partition_view.*` and `manifest.*`
are unchanged.
**Results.**
| Catalog setting | Before | After |
|---|---|---|
| `table.enable=false` | table loaded live; snapshot pin, format and
comment cached for 24 h; latest schema cached | snapshot pin, schema,
format and comment all read live |
| `table.capacity=0` | same as above | same as above |
| `table.capacity=N` (N > 0) | snapshot, format and comment caches hold
up to 1000 entries | up to N entries |
| `table.max-weight` | bounds the table cache | unchanged |
| `table.ttl-second` (any value), default settings | unchanged |
unchanged |
**Classes, and how they call each other.**
```
IcebergConnector(properties)
├─ cacheSpec("table") ─────────────► IcebergTableCache
(enable, ttl, capacity, max-weight)
└─ tableFactsCacheSpec = enable/ttl/capacity of the table spec
├─► IcebergLatestSnapshotCache ◄──
IcebergConnectorMetadata.beginQuerySnapshot
├─► IcebergFormatCache ◄── IcebergScanPlanProvider (file
format inference)
├─► IcebergCommentCache ◄──
IcebergConnectorMetadata.getTableComment (vended REST only)
└─► schemaCacheTtlSecondOverride ──►
PluginDrivenExternalCatalog.overlayMetaCacheConfig
PluginDrivenMvccExternalTable.getLatestSchemaCacheValue
```
---
.../connector/iceberg/IcebergCommentCache.java | 32 +++++++------
.../doris/connector/iceberg/IcebergConnector.java | 33 +++++++++-----
.../iceberg/IcebergConnectorMetadata.java | 7 +--
.../connector/iceberg/IcebergFormatCache.java | 21 +++++----
.../iceberg/IcebergLatestSnapshotCache.java | 23 ++++++----
.../iceberg/IcebergConnectorCacheTest.java | 52 ++++++++++++++++++++++
6 files changed, 125 insertions(+), 43 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java
index cd6bb211ed3..fb6cd767732 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java
@@ -45,12 +45,14 @@ import java.util.function.Supplier;
*
* <p><b>No credential gate</b> (unlike {@link IcebergTableCache}, like {@link
IcebergPartitionCache} /
* {@link IcebergFormatCache}): the cached value is a bare comment {@link
String} with no {@code FileIO} /
- * credential. A comment changes only via external DDL, picked up through the
REFRESH invalidate hooks. TTL is
- * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read
live), so a no-cache catalog serves a
- * fresh comment even when this object is built. Backed identically to {@link
IcebergTableCache}: a contextual,
- * access-TTL {@link MetaCache} with manual miss-load, so the remote load runs
OUTSIDE Caffeine's compute
- * lock and its exception (e.g. the view-handle {@code NoSuchTableException})
propagates verbatim and a failed
- * load is not cached. Lives on the long-lived per-catalog {@link
IcebergConnector}; a REFRESH CATALOG rebuilds it.
+ * credential. A comment changes only via external DDL, picked up through the
REFRESH invalidate hooks. The
+ * connector builds it with the table entry's {@code
meta.cache.iceberg.table.(enable|ttl-second|capacity)};
+ * {@code enable=false}, {@code ttl-second <= 0} or {@code capacity=0}
disables it (read live), so a no-cache
+ * catalog serves a fresh comment even when this object is built. Backed
identically to
+ * {@link IcebergTableCache}: a contextual, access-TTL {@link MetaCache} with
manual miss-load, so the remote load
+ * runs OUTSIDE Caffeine's compute lock and its exception (e.g. the
view-handle {@code NoSuchTableException})
+ * propagates verbatim and a failed load is not cached. Lives on the
long-lived per-catalog
+ * {@link IcebergConnector}; a REFRESH CATALOG rebuilds it.
*/
final class IcebergCommentCache {
@@ -58,22 +60,21 @@ final class IcebergCommentCache {
private final MetaCache<TableIdentifier, String> entry;
IcebergCommentCache(long ttlSeconds, int maxSize) {
- this(CatalogMetaCache.unmanaged(), ttlSeconds, maxSize);
+ // "<= 0 disables" connector TTL contract, folded to CacheSpec's
disable sentinel (CacheSpec.ofConnectorTtl).
+ this(CatalogMetaCache.unmanaged(),
CacheSpec.ofConnectorTtl(ttlSeconds, maxSize));
}
- IcebergCommentCache(CatalogMetaCache owner, long ttlSeconds, int maxSize) {
+ IcebergCommentCache(CatalogMetaCache owner, CacheSpec spec) {
this.owner = owner;
- // "<= 0 disables" connector TTL contract, folded to CacheSpec's
disable sentinel (CacheSpec.ofConnectorTtl).
- // Load-bearing here: a vended no-cache catalog builds this object but
must NOT cache comments
- // (operator "no meta cache" intent).
- CacheSpec spec = CacheSpec.ofConnectorTtl(ttlSeconds, maxSize);
+ // A disabled spec is load-bearing here: a vended no-cache catalog
builds this object but must NOT cache
+ // comments (operator "no meta cache" intent).
this.entry = owner.create(MetaCacheDefinition
.<TableIdentifier, String>builder("iceberg-comment", spec,
IcebergCommentCache::scope)
.sizeEstimator(MetaCacheSizeEstimators.reflective())
.build());
}
- /** Caching is on only when the TTL is positive; ttl-second <= 0 means
"always read the comment live". */
+ /** Caching is on unless the spec disables it; a disabled cache means
"always read the comment live". */
boolean isEnabled() {
return entry.isEnabled();
}
@@ -113,6 +114,11 @@ final class IcebergCommentCache {
return entry.loadSuccessCount();
}
+ /** Test-only: the enable / TTL / capacity / weight spec this cache was
built with. */
+ CacheSpec cacheSpecForTest() {
+ return entry.cacheSpec();
+ }
+
private static ScopePath scope(TableIdentifier identifier) {
return ScopePath.table(identifier.namespace().toString(),
identifier.name());
}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
index da37ab20639..d26b359c601 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
@@ -203,6 +203,9 @@ public class IcebergConnector implements Connector {
// per-user resolveTable). If you ADD a cross-query cache here or in
IcebergConnectorMetadata, it MUST be
// null under session=user and covered by IcebergConnectorCacheTest (which
asserts exactly that at runtime).
private final IcebergLatestSnapshotCache latestSnapshotCache; // null
under session=user
+ // The enable / ttl-second / capacity of the table entry, shared by the
latest-snapshot, format and comment
+ // caches (see the constructor). Also turns off the FE schema cache
(schemaCacheTtlSecondOverride).
+ private final CacheSpec tableFactsCacheSpec;
private final IcebergCatalogResourceTracker catalogResourceTracker = new
IcebergCatalogResourceTracker();
// PERF-01: cross-query cache of the RAW iceberg Table (restores the
legacy IcebergExternalMetaCache table
// cache that the SPI cutover dropped). null when the catalog's
credentials are query-dependent
@@ -264,6 +267,14 @@ public class IcebergConnector implements Connector {
this::pluginAuthenticator);
this.metaCache = CatalogMetaCache.managed(context.getCatalogId(),
"iceberg", this.properties);
this.manifestCache = new IcebergManifestCache(metaCache,
this.properties);
+ // The latest-snapshot, format and comment caches have no
meta.cache.iceberg.<entry> keys of their own; they
+ // take the table entry's enable / ttl-second / capacity (the legacy
IcebergExternalMetaCache served the
+ // latest snapshot from its table entry), so table.enable=false or
capacity=0 turns them off with the table
+ // cache, as ttl-second<=0 already did. The table's max-weight bounds
only the table cache; the catalog and
+ // FE-global limits still bound these caches.
+ CacheSpec tableSpec = cacheSpec("table");
+ this.tableFactsCacheSpec = CacheSpec.of(
+ tableSpec.isEnable(), tableSpec.getTtlSecond(),
tableSpec.getCapacity());
// Authorization-sensitive projection (snapshotId/schemaId). Under
iceberg.rest.session=user the value is
// per-user AUTHORIZED metadata that a "can-list-cannot-load"
principal must not see. beginQuerySnapshot
// reads this cache WITHOUT a preceding per-user loadTable, so a
shared (table-keyed, no user dimension)
@@ -273,8 +284,7 @@ public class IcebergConnector implements Connector {
// tableCache/commentCache discipline: session=user => no LIVE
cross-query metadata cache.
this.latestSnapshotCache = isUserSessionEnabled()
? null
- : new IcebergLatestSnapshotCache(
- metaCache,
resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
+ : new IcebergLatestSnapshotCache(metaCache,
tableFactsCacheSpec);
// PERF-01 cross-query RAW-table cache. Disabled (null) when the
catalog's credentials are
// query-dependent, because a cached raw Table carries its FileIO's
credentials:
// - iceberg.rest.session=user: per-user delegated FileIO -> sharing
across users leaks credentials.
@@ -282,12 +292,12 @@ public class IcebergConnector implements Connector {
// iceberg keeps it fresh by reloading the table each query, so a
24h-TTL hit would hand BE an
// expired token (403 mid-scan). Both gates are independent;
either one disables this layer.
// The query-scoped fat handle stays on in all cases (its token is
fresh within the one query). Same
- // TTL/capacity as the snapshot cache (the single
meta.cache.iceberg.table.ttl-second knob).
+ // enable/TTL/capacity as the snapshot cache, plus the table entry's
own max-weight.
this.tableCache = (isUserSessionEnabled()
||
IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties))
? null
: new IcebergTableCache(
- metaCache, cacheSpec("table"),
+ metaCache, tableSpec,
this::cachedTableCleanup, catalogResourceTracker);
// PERF-02: partition-view cache. Authorization-sensitive projection:
a shared (table+snapshot-keyed, no
// user dimension) hit would disclose one user's partition list. Its
readers are all downstream of a
@@ -303,17 +313,15 @@ public class IcebergConnector implements Connector {
// under session=user, kept otherwise); readers already tolerate a
null cache (resolveFileFormatName).
this.formatCache = isUserSessionEnabled()
? null
- : new IcebergFormatCache(
- metaCache,
resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
+ : new IcebergFormatCache(metaCache, tableFactsCacheSpec);
// PERF-05: table-comment cache, built ONLY for a REST
vended-credentials catalog that is NOT session=user.
// Plain catalogs (tableCache on) already serve the comment path from
tableCache; session=user is excluded
// because a shared comment cache would bypass the per-user loadTable
authorization (a metadata disclosure).
// Comment is pure metadata (no credential) so no gate on the VALUE --
the flavor gate is an authorization
- // decision, not a credential-leak one. Same TTL/capacity (ttl<=0
still disables internally).
+ // decision, not a credential-leak one. Same spec as the snapshot
cache (a disabled one still reads live).
this.commentCache =
(IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties)
&& !isUserSessionEnabled())
- ? new IcebergCommentCache(
- metaCache,
resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY)
+ ? new IcebergCommentCache(metaCache, tableFactsCacheSpec)
: null;
// PERF-06: derived partition-view cache A (generic
ConnectorMetadataCache). Same
// authorization-sensitive treatment as partitionCache -- disabled
(null) under iceberg.rest.session=user so
@@ -733,10 +741,15 @@ public class IcebergConnector implements Connector {
* {@code schema.cache.ttl-second}), so a no-cache catalog ({@code
ttl-second=0}) serves FRESH schema after
* external DDL (mirrors {@code
PaimonConnector.schemaCacheTtlSecondOverride}). Absent -> no override (engine
* default TTL). Do NOT reuse {@link #resolveTableCacheTtlSecond}, which
substitutes the 24h default for a
- * blank value and would defeat the engine default.
+ * blank value and would defeat the engine default. A table entry turned
off by {@code enable=false} or
+ * {@code capacity=0} is a no-cache catalog too: its latest snapshot is
read live, so the schema it names must
+ * not come from the name-keyed schema cache either.
*/
@Override
public OptionalLong schemaCacheTtlSecondOverride() {
+ if (!tableFactsCacheSpec.isEnable() ||
tableFactsCacheSpec.getCapacity() == 0) {
+ return OptionalLong.of(0L);
+ }
String raw = properties.get(TABLE_CACHE_TTL_SECOND);
if (raw == null || raw.trim().isEmpty()) {
return OptionalLong.empty();
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
index 088057890c2..ec7cbd5cdda 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
@@ -2137,7 +2137,8 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
* a HIT returns the cached pin without re-loading the table (saves the
load I/O and keeps the snapshot
* STABLE across queries — the legacy with-cache catalog). The cached
value carries BOTH ids atomically so a
* schema-only ALTER between two queries cannot skew snapshotId vs
schemaId. A disabled cache
- * ({@code meta.cache.iceberg.table.ttl-second <= 0}) reads live every
call (the legacy no-cache catalog).
+ * ({@code meta.cache.iceberg.table.enable=false}, {@code ttl-second <= 0}
or {@code capacity=0}) reads live
+ * every call (the legacy no-cache catalog).
*/
@Override
public Optional<ConnectorMvccSnapshot> beginQuerySnapshot(
@@ -2146,8 +2147,8 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
TableIdentifier id = TableIdentifier.of(iceHandle.getDbName(),
iceHandle.getTableName());
// A null latestSnapshotCache (iceberg.rest.session=user, where a
shared table-keyed hit would bypass the
// per-user loadTable authorization) reads live per-user every call,
so authorization runs every time (no
- // stale-authz window); mirrors resolveTableForRead's tableCache
null-fallback. A disabled cache (ttl<=0)
- // is a non-null cache that reads live internally.
+ // stale-authz window); mirrors resolveTableForRead's tableCache
null-fallback. A disabled cache (the table
+ // entry turned off) is a non-null cache that reads live internally.
IcebergLatestSnapshotCache.CachedSnapshot pin = latestSnapshotCache !=
null
? latestSnapshotCache.getOrLoad(id, () ->
loadLatestSnapshotPin(session, iceHandle))
: loadLatestSnapshotPin(session, iceHandle);
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java
index 822c212b793..49f3794b346 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java
@@ -52,12 +52,13 @@ import java.util.function.Supplier;
*
* <p><b>No credential gate</b> (unlike {@link IcebergTableCache}, like {@link
IcebergPartitionCache}): the cached
* value is a bare format-name {@link String} with no {@code FileIO} /
credential, so it is safe to share across
- * users and is built unconditionally (only the TTL knob disables it).
+ * users and is built unconditionally (only the table entry's cache keys
disable it).
*
* <p>Backed identically to {@link IcebergPartitionCache}: a contextual,
access-TTL {@link MetaCache} with
* manual miss-load, so the inference runs OUTSIDE Caffeine's compute lock and
a failed scan's exception
- * propagates verbatim and is NOT cached (the next query retries — legacy
parity). TTL is
- * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read
live). Lives on the long-lived
+ * propagates verbatim and is NOT cached (the next query retries — legacy
parity). The connector builds it with
+ * the table entry's {@code
meta.cache.iceberg.table.(enable|ttl-second|capacity)}; {@code enable=false},
+ * {@code ttl-second <= 0} or {@code capacity=0} disables it (read live).
Lives on the long-lived
* per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds the
connector and thus the cache.
*/
final class IcebergFormatCache {
@@ -94,20 +95,19 @@ final class IcebergFormatCache {
private final MetaCache<Key, String> entry;
IcebergFormatCache(long ttlSeconds, int maxSize) {
- this(CatalogMetaCache.unmanaged(), ttlSeconds, maxSize);
+ // "<= 0 disables" connector TTL contract, folded to CacheSpec's
disable sentinel (CacheSpec.ofConnectorTtl).
+ this(CatalogMetaCache.unmanaged(),
CacheSpec.ofConnectorTtl(ttlSeconds, maxSize));
}
- IcebergFormatCache(CatalogMetaCache owner, long ttlSeconds, int maxSize) {
+ IcebergFormatCache(CatalogMetaCache owner, CacheSpec spec) {
this.owner = owner;
- // "<= 0 disables" connector TTL contract, folded to CacheSpec's
disable sentinel (CacheSpec.ofConnectorTtl).
- CacheSpec spec = CacheSpec.ofConnectorTtl(ttlSeconds, maxSize);
this.entry = owner.create(MetaCacheDefinition
.<Key, String>builder("iceberg-format", spec,
IcebergFormatCache::scope)
.sizeEstimator(MetaCacheSizeEstimators.reflective())
.build());
}
- /** Caching is on only when the TTL is positive; ttl-second <= 0 means
"always infer live". */
+ /** Caching is on unless the spec disables it; a disabled cache means
"always infer live". */
boolean isEnabled() {
return entry.isEnabled();
}
@@ -147,6 +147,11 @@ final class IcebergFormatCache {
return entry.loadSuccessCount();
}
+ /** Test-only: the enable / TTL / capacity / weight spec this cache was
built with. */
+ CacheSpec cacheSpecForTest() {
+ return entry.cacheSpec();
+ }
+
private static ScopePath scope(Key key) {
return ScopePath.table(key.id.namespace().toString(), key.id.name());
}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java
index af3c4a1d49e..43a9a80621e 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java
@@ -47,10 +47,11 @@ import java.util.function.Supplier;
* an empty scan expose live partition metadata.
*
* <p>Backed by the shared {@link MetaCache} framework (independent-copy
meta-cache migration): a
- * contextual, access-TTL entry whose per-query loader is supplied at {@link
#getOrLoad}. TTL is
- * {@code meta.cache.iceberg.table.ttl-second}: {@code <= 0} disables caching
(every read goes live, matching
- * the legacy "no-cache" catalog); a positive value is Caffeine {@code
expireAfterAccess} with a
- * {@code maxSize} capacity (real LRU eviction, replacing the former
clear-on-overflow). Manual miss-load is
+ * contextual, access-TTL entry whose per-query loader is supplied at {@link
#getOrLoad}. The connector builds it
+ * with the table entry's {@code
meta.cache.iceberg.table.(enable|ttl-second|capacity)}: {@code enable=false},
+ * {@code ttl-second <= 0} or {@code capacity=0} disables caching (every read
goes live, matching the legacy
+ * "no-cache" catalog); otherwise the TTL is Caffeine {@code
expireAfterAccess} with the capacity as its
+ * {@code maxSize} (real LRU eviction, replacing the former
clear-on-overflow). Manual miss-load is
* on so the loader runs OUTSIDE Caffeine's compute lock (single-flight per
key). Lives on the long-lived
* per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds the
connector and thus the cache.
*/
@@ -78,13 +79,12 @@ final class IcebergLatestSnapshotCache {
private final MetaCache<TableIdentifier, CachedSnapshot> entry;
IcebergLatestSnapshotCache(long ttlSeconds, int maxSize) {
- this(CatalogMetaCache.unmanaged(), ttlSeconds, maxSize);
+ // "<= 0 disables" connector TTL contract, folded to CacheSpec's
disable sentinel (CacheSpec.ofConnectorTtl).
+ this(CatalogMetaCache.unmanaged(),
CacheSpec.ofConnectorTtl(ttlSeconds, maxSize));
}
- IcebergLatestSnapshotCache(CatalogMetaCache owner, long ttlSeconds, int
maxSize) {
+ IcebergLatestSnapshotCache(CatalogMetaCache owner, CacheSpec spec) {
this.owner = owner;
- // "<= 0 disables" connector TTL contract, folded to CacheSpec's
disable sentinel (CacheSpec.ofConnectorTtl).
- CacheSpec spec = CacheSpec.ofConnectorTtl(ttlSeconds, maxSize);
this.entry = owner.create(MetaCacheDefinition
.<TableIdentifier, CachedSnapshot>builder(
"iceberg-latest-snapshot", spec,
IcebergLatestSnapshotCache::scope)
@@ -92,7 +92,7 @@ final class IcebergLatestSnapshotCache {
.build());
}
- /** Caching is on only when the TTL is positive; ttl-second <= 0 means
"always read live". */
+ /** Caching is on unless the spec disables it; a disabled cache means
"always read live". */
boolean isEnabled() {
return entry.isEnabled();
}
@@ -133,6 +133,11 @@ final class IcebergLatestSnapshotCache {
return Math.toIntExact(entry.size());
}
+ /** Test-only: the enable / TTL / capacity / weight spec this cache was
built with. */
+ CacheSpec cacheSpecForTest() {
+ return entry.cacheSpec();
+ }
+
private static ScopePath scope(TableIdentifier identifier) {
return ScopePath.table(identifier.namespace().toString(),
identifier.name());
}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java
index 88f358a4b87..ec352b02ca4 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java
@@ -329,6 +329,58 @@ public class IcebergConnectorCacheTest {
}
}
+ @Test
+ public void disabledTableEntryReadsSnapshotFormatAndCommentLive() throws
Exception {
+ // table.enable=false and capacity=0 turn the table entry off, like
ttl-second<=0. The latest-snapshot, format
+ // and comment caches take the table entry's keys, so they read live
too, and the latest schema bypasses the
+ // name-keyed schema cache. MUTATION: building them from ttl-second
alone -> the first pin stays cached -> red.
+ for (String[] off : new String[][] {
+ {IcebergConnector.TABLE_CACHE_ENABLE, "false"},
{IcebergConnector.TABLE_CACHE_CAPACITY, "0"}}) {
+ try (IcebergConnector connector =
+ new IcebergConnector(props(off[0], off[1]), new
RecordingConnectorContext())) {
+ IcebergLatestSnapshotCache snapshots =
connector.latestSnapshotCacheForTest();
+ TableIdentifier id = TableIdentifier.of("db1", "t1");
+ AtomicInteger loads = new AtomicInteger();
+ Supplier<IcebergLatestSnapshotCache.CachedSnapshot> loader =
+ () -> new
IcebergLatestSnapshotCache.CachedSnapshot(loads.incrementAndGet(), 0L);
+ Assertions.assertEquals(1L, snapshots.getOrLoad(id,
loader).snapshotId);
+ Assertions.assertEquals(2L, snapshots.getOrLoad(id,
loader).snapshotId, off[0]);
+ Assertions.assertEquals(0, snapshots.size());
+
Assertions.assertFalse(connector.tableCacheForTest().isEnabled(), off[0]);
+
Assertions.assertFalse(connector.formatCacheForTest().isEnabled(), off[0]);
+ Assertions.assertEquals(OptionalLong.of(0L),
connector.schemaCacheTtlSecondOverride(), off[0]);
+ }
+ Map<String, String> vended = restProps(true, false);
+ vended.put(off[0], off[1]);
+ try (IcebergConnector connector = new IcebergConnector(vended, new
RecordingConnectorContext())) {
+
Assertions.assertFalse(connector.latestSnapshotCacheForTest().isEnabled(),
off[0]);
+
Assertions.assertFalse(connector.commentCacheForTest().isEnabled(), off[0]);
+ }
+ }
+ }
+
+ @Test
+ public void
tableEntryKeysExceptMaxWeightSizeSnapshotFormatAndCommentCaches() throws
Exception {
+ // The table entry's ttl-second and capacity also size these caches.
Its max-weight bounds the table cache
+ // only; the catalog and FE-global limits still bound them. MUTATION:
the fixed default capacity -> red;
+ // handing them the whole table spec, max-weight included -> red.
+ Map<String, String> properties = restProps(true, false);
+ properties.put(IcebergConnector.TABLE_CACHE_TTL_SECOND, "600");
+ properties.put(IcebergConnector.TABLE_CACHE_CAPACITY, "7");
+ properties.put("meta.cache.iceberg.table.max-weight", "1MB");
+ try (IcebergConnector connector = new IcebergConnector(properties, new
RecordingConnectorContext())) {
+ List.of(connector.latestSnapshotCacheForTest().cacheSpecForTest(),
+ connector.formatCacheForTest().cacheSpecForTest(),
+
connector.commentCacheForTest().cacheSpecForTest()).forEach(spec -> {
+ Assertions.assertTrue(spec.isCacheEnabled());
+ Assertions.assertEquals(600L, spec.getTtlSecond());
+ Assertions.assertEquals(7L, spec.getCapacity());
+ Assertions.assertFalse(spec.isWeightBounded());
+ });
+ Assertions.assertEquals(OptionalLong.of(600L),
connector.schemaCacheTtlSecondOverride());
+ }
+ }
+
@Test
public void crossQueryTableCacheDisabledForVendedCredentials() {
// REST vended-credentials: the cached raw table's FileIO carries a
server-vended token that expires
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
