This is an automated email from the ASF dual-hosted git repository.
CalvinKirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0edac26d33f [fix](paimon) Stop clearing the Paimon table cache on
every table lookup (#68860)
0edac26d33f is described below
commit 0edac26d33f62bdbe9c6e3671481deec36e9bcd8
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Oct 10 16:32:59 2026 +0800
[fix](paimon) Stop clearing the Paimon table cache on every table lookup
(#68860)
### What problem does this PR solve?
Issue Number: None
Related PR: #67023, #67395, #67996
Problem Summary:
**In short.** Since #67395, every Paimon table lookup on master clears
that table's cached metadata before reloading it, so the Paimon metadata
cache never hits. Every statement reloads the table and its latest
snapshot from the remote catalog, and a catalog with caching enabled no
longer serves its cached snapshot. This PR removes the invalidation and
restores the regression expectation that #67395 changed.
**Background.** #67395 forward-ported branch-4.1 #67023, which keeps the
Paimon write cache coherent. In branch-4.x, the Doris table cache sits
above Paimon's SDK `CachingCatalog`. On a Doris cache miss, #67023
invalidates the SDK entry before reloading, so the reload cannot hand
back a stale SDK table. #67395 placed that invalidation in
`CatalogBackedPaimonCatalogOps.getTable`. On master, however, #67996 had
already replaced the SDK `CachingCatalog` with the Doris-owned
`PaimonMetaCacheCatalog`, and `PaimonCatalogFactory` always disables the
SDK wrapper. The catalog that `getTable` wraps is therefore the Doris
cache itself.
| | branch-4.x | master since #67395 |
|---|---|---|
| Table cache | Doris table cache above Paimon's SDK `CachingCatalog` |
`PaimonMetaCacheCatalog` only (SDK wrapper disabled) |
| When the reload path runs | only on a Doris cache miss | on every
`getTableHandle`, several times per statement |
| What the invalidation clears | the SDK entry about to be reloaded |
the Doris entries for the table: table, latest snapshot, schema-at memo,
partition view |
**The problem, and what it cost.**
- **The cache never hits.** On a local filesystem catalog, 7 SELECT
statements made 86 table-cache requests, with 0 hits and 38
invalidations. The latest-snapshot cache had 0 hits in 24 requests. On
HMS, REST or object-store catalogs, each of those reloads is a remote
call, and it happens in every statement.
- **The cache semantics changed.** A catalog with caching enabled used
to keep reading its cached snapshot until REFRESH or the cache TTL. Now
it reads data committed by other engines at once. #67395 changed
`test_paimon_table_meta_cache` to expect the rows Spark wrote without a
REFRESH, and attributed this to Paimon 1.4. branch-4.2 runs the same
Paimon 1.4.2, and the same test there still expects the cached rows.
**How this PR fixes it.**
- `CatalogBackedPaimonCatalogOps.getTable` no longer invalidates the
table. Nothing is lost by this: REFRESH TABLE / CATALOG, Doris writes
(`refreshTableAfterExternalMutation`) and DDL all invalidate through
`CatalogMetaCache`, which clears the `PaimonMetaCacheCatalog` entries,
and no other layer caches the table.
- `test_paimon_table_meta_cache` expects the cached rows again before
the REFRESH.
- `PaimonTableOptionsTest#catalogOpsInvalidatesPaimonCacheBeforeReload`
pinned the invalidation. It is replaced by
`PaimonMetaCacheCatalogTest#connectorLookupsHitTheTableCacheUntilTheTableIsInvalidated`:
repeated lookups through `CatalogBackedPaimonCatalogOps` load the table
once, and a lookup after the table is invalidated loads it again.
**Results.** Measured on a local FE + BE with a filesystem Paimon
catalog, with a second catalog on the same warehouse playing the
external writer:
| | before | after |
|---|---|---|
| Cached catalog, after the other catalog wrote a row | sees the new row
| sees the cached rows until REFRESH TABLE |
| Cached catalog, after its own INSERT | sees it | sees it |
| Table-cache hits | 0 of 86 requests (7 SELECTs) | 20 of 20 requests
for 5 repeated SELECTs |
Putting the invalidation back makes the new unit test fail. All 675
tests of `fe-connector-paimon` pass.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_paimon_table_meta_cache` needs the Paimon docker fixture and was
not run locally.
- [x] Unit Test
- `PaimonMetaCacheCatalogTest`, `PaimonTableOptionsTest` and the rest of
`fe-connector-paimon` (675 tests) pass locally.
- [x] Manual test (add detailed scripts or steps below)
- Local FE + BE with two filesystem Paimon catalogs on one warehouse
(one with `meta.cache.paimon.table.ttl-second` = 0); results in the
table above.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. A Paimon catalog with metadata caching enabled serves
repeated queries from its cache again. Data committed by other engines
becomes visible after REFRESH or the cache TTL, as in branch-4.x and on
master before #67395.
---
.../doris/connector/paimon/PaimonCatalogOps.java | 8 +++---
.../paimon/PaimonMetaCacheCatalogTest.java | 21 +++++++++++++++
.../connector/paimon/PaimonTableOptionsTest.java | 30 ----------------------
.../paimon/test_paimon_table_meta_cache.groovy | 5 ++--
4 files changed, 26 insertions(+), 38 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogOps.java
b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogOps.java
index a895f711951..06193e96ccf 100644
---
a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogOps.java
+++
b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogOps.java
@@ -308,11 +308,9 @@ public interface PaimonCatalogOps {
*/
@Override
public Table getTable(Identifier identifier) throws
Catalog.TableNotExistException {
- // Doris owns the outer table-metadata cache and only reaches this
method on an outer-cache
- // load. Paimon's CachingCatalog owns a second cache for the Table
and its lower-level readers.
- // Invalidate that inner entry before reloading, otherwise REFRESH
or outer-cache expiry can
- // immediately repopulate Doris with the same stale bucket, path,
or table options.
- catalog.invalidateTable(identifier);
+ // Every metadata, scan and write lookup lands here, several times
per statement. The wrapped
+ // catalog is the Doris table cache (PaimonMetaCacheCatalog; the
SDK CachingCatalog is disabled),
+ // so this must not invalidate: REFRESH, Doris writes and DDL
invalidate through CatalogMetaCache.
Table table = catalog.getTable(identifier);
Map<String, String> optionsForCopy =
PaimonTableOptions.forCopy(tableOptions);
return optionsForCopy.isEmpty() ? table :
table.copy(optionsForCopy);
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonMetaCacheCatalogTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonMetaCacheCatalogTest.java
index 073535fbd0c..67abc59782d 100644
---
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonMetaCacheCatalogTest.java
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonMetaCacheCatalogTest.java
@@ -294,6 +294,27 @@ class PaimonMetaCacheCatalogTest {
}
}
+ @Test
+ void connectorLookupsHitTheTableCacheUntilTheTableIsInvalidated() throws
Exception {
+ // Each statement resolves its table handle several times through
CatalogBackedPaimonCatalogOps.
+ // Those lookups must be served from the Doris table cache; only an
invalidation of the table
+ // (REFRESH TABLE, a Doris write or DDL all end in CatalogMetaCache)
makes the next one reload.
+ RecordingCatalog recording = new RecordingCatalog();
+ try (CatalogMetaCache owner = CatalogMetaCache.unmanaged()) {
+ PaimonCatalogOps ops = new
PaimonCatalogOps.CatalogBackedPaimonCatalogOps(
+ PaimonMetaCacheCatalog.tryToCreate(recording.catalog(),
owner, 100, 100,
+ cacheOptions(Duration.ofDays(1),
Duration.ofDays(1)), true, false));
+
+ Table first = ops.getTable(TABLE);
+ Assertions.assertSame(first, ops.getTable(TABLE));
+ Assertions.assertEquals(1, recording.tableLoads.get(), "a repeated
lookup must not reload the table");
+
+ owner.invalidateTable("db", "t");
+ Assertions.assertNotSame(first, ops.getTable(TABLE));
+ Assertions.assertEquals(2, recording.tableLoads.get());
+ }
+ }
+
@Test
void systemTableIsRebuiltFromTheCachedOriginTable() throws Exception {
AtomicLong clock = new AtomicLong();
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableOptionsTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableOptionsTest.java
index c759df4e41c..f291ea74fc6 100644
---
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableOptionsTest.java
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableOptionsTest.java
@@ -29,12 +29,9 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
-import java.lang.reflect.Proxy;
import java.nio.file.Path;
-import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
-import java.util.List;
import java.util.Map;
/**
@@ -276,33 +273,6 @@ public class PaimonTableOptionsTest {
}
}
- @Test
- public void catalogOpsInvalidatesPaimonCacheBeforeReload() throws
Exception {
- List<String> calls = new ArrayList<>();
- Table loaded = (Table) Proxy.newProxyInstance(
- Table.class.getClassLoader(), new Class<?>[] {Table.class},
- (proxy, method, args) -> null);
- Catalog catalog = (Catalog) Proxy.newProxyInstance(
- Catalog.class.getClassLoader(), new Class<?>[] {Catalog.class},
- (proxy, method, args) -> {
- if ("invalidateTable".equals(method.getName())) {
- calls.add("invalidate");
- return null;
- }
- if ("getTable".equals(method.getName())) {
- calls.add("get");
- return loaded;
- }
- return null;
- });
-
- Table actual = new
PaimonCatalogOps.CatalogBackedPaimonCatalogOps(catalog)
- .getTable(Identifier.create("db", "t"));
-
- Assertions.assertSame(loaded, actual);
- Assertions.assertEquals(java.util.Arrays.asList("invalidate", "get"),
calls);
- }
-
@Test
public void createCatalogRejectsBadTableOption() {
Map<String, String> props = new HashMap<>();
diff --git
a/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
b/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
index 199f4bdce93..e4c806bf26d 100644
---
a/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
+++
b/regression-test/suites/external_table_p0/paimon/test_paimon_table_meta_cache.groovy
@@ -99,11 +99,10 @@ suite("test_paimon_table_meta_cache", "p0,external") {
spark_paimon "INSERT INTO paimon.${testDb}.test_insert VALUES (2,
'external_insert')"
- // The cache retains the Paimon table object and its schema, not a
data snapshot. Paimon 1.4
- // resolves the latest snapshot when a read is planned from that
cached table object.
+ // The cached catalog keeps reading the latest snapshot it cached
until REFRESH (or its TTL).
sql """switch ${catalogWithCache}"""
def result2 = sql """select * from ${testDb}.test_insert order by id"""
- assertEquals(2, result2.size())
+ assertEquals(1, result2.size())
sql """switch ${catalogNoCache}"""
def result2NoCache = sql """select * from ${testDb}.test_insert order
by id"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
