github-actions[bot] commented on code in PR #68196:
URL: https://github.com/apache/doris/pull/68196#discussion_r4129484721
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##########
@@ -954,6 +967,41 @@ public Optional<ExternalDatabase<? extends ExternalTable>>
getDbForReplay(String
return metaCache.tryGetMetaObj(localDbName);
}
+ /** Resolve a replay log's database identity without reloading an evicted
database object. */
+ public Optional<Pair<String, Long>> getDbIdentityForReplay(String dbName,
long dbId) {
+ if (!isInitialized() || metaCache == null) {
+ return Optional.empty();
+ }
+ if (dbName != null && !dbName.isEmpty()) {
+ String localName = getLocalDatabaseName(dbName, true);
+ return localName == null ? Optional.empty()
+ : Optional.of(Pair.of(localName, Util.genIdByName(name,
localName)));
+ }
+ return metaCache.getNameByIdIfPresent(dbId).map(localName ->
Pair.of(localName, dbId));
+ }
+
+ /** A DROP must not follow a mode-2 mapping that has rebound to a
case-only replacement. */
+ private Optional<Pair<String, Long>> getDbIdentityForDrop(String dbName) {
+ if (!isInitialized() || metaCache == null) {
+ return Optional.empty();
+ }
+ String localName = getLocalDatabaseName(dbName, true);
+ if (localName == null) {
+ return Optional.empty();
+ }
+ if (getLowerCaseDatabaseNames() == 2 && !localName.equals(dbName)) {
+ long historicalId = Util.genIdByName(name, dbName);
+ return metaCache.getNameByIdIfPresent(historicalId)
Review Comment:
[P1] Preserve the actual target of a replayed DROP TABLE. In mode 2, an old
`Foo` DB ID can remain cached while `foo` now maps to `FOO`. A live `DROP TABLE
Foo.t` resolves and drops current `FOO.t`, but its `DropInfo` records caller
spelling `Foo`; follower replay now chooses the retained old `Foo` through this
historical-ID branch and leaves dropped `FOO.t` cached. The earlier thread
covers a delayed DROP that really targets old `Foo`, which is the opposite
history. Record/pass the resolved target identity for new logs and keep a
conservative path for older logs; test both histories.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java:
##########
@@ -520,9 +597,44 @@ private void invalidateLanceTableAccess(long catalogId) {
public void invalidatePartitions(long catalogId,
String dbName, String tableName, List<String> partitions) {
- routeCatalogEngines(catalogId, cache -> safeInvalidate(
- cache, catalogId, "invalidatePartitions",
- () -> cache.invalidatePartitions(catalogId, dbName, tableName,
partitions)));
+ Optional<ExternalDatabase<? extends ExternalTable>> db =
Optional.empty();
+ try {
+ db = getCachedDb(catalogId, dbName);
+ invalidateTableRowCount(catalogId, db, tableName);
+ routeCatalogEngines(catalogId, cache -> safeInvalidate(
+ cache, catalogId, "invalidatePartitions",
+ () -> cache.invalidatePartitions(catalogId, dbName,
tableName, partitions)));
+ } finally {
+ invalidateTableRowCount(catalogId, db, tableName);
+ }
+ }
+
+ private void invalidateTableRowCount(long catalogId,
+ Optional<ExternalDatabase<? extends ExternalTable>> db, String
tableName) {
+ if (db.isPresent()) {
+ Optional<? extends ExternalTable> table =
db.get().getTableForReplay(tableName);
+ if (table.isPresent()) {
+ invalidateRowCountCache(table.get());
+ } else {
+ rowCountCache.invalidateDb(catalogId, db.get().getId());
+ }
+ } else {
+ rowCountCache.invalidateCatalog(catalogId);
Review Comment:
[P2] Keep a known cold database's row-count fence at DB scope.
`invalidateTableByEngine` is called by Hive partition-cache recovery with a
held table, but an ordinary database-object eviction makes `getCachedDb` miss
while the catalog still retains its canonical DB ID. This fallback then scans
and evicts every DB's row counts twice for one table; `invalidateTable` and
`invalidatePartitions` share it. Resolve the retained ID before using catalog
scope, and cover a cold target with an unrelated cached count.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java:
##########
@@ -191,25 +191,10 @@ private boolean performDropDb(String dbName, boolean
ifExists, boolean force) th
@Override
public void afterDropDb(String dbName) {
- Optional<ExternalDatabase<? extends ExternalTable>> db =
dorisCatalog.getDbForReplay(dbName);
try {
- if (db.isPresent()) {
- // getDbForReplay normalizes case-insensitive database names
(lower_case_database_names
- // mode 1/2), so an alternate-case DROP DATABASE can resolve
the cached database while
- // an exact-key eviction with the caller's spelling would miss
it. Evict by the resolved
- // canonical local key so the removal listener still performs
the one typed SDK
- // invalidation; do not add a second typed scan under the
catalog write fence.
- dorisCatalog.unregisterDatabase(db.get().getFullName());
- return;
- }
- // The cached database could not be resolved (for example a mode-2
case mapping was removed
- // by a names refresh before replay). Exact-key eviction can miss
the canonical local key,
- // so also retire the remaining legacy database objects; otherwise
a same-name recreation
- // could reuse the stale object and its nested table-name cache.
The catalog-wide engine
- // flush below covers the SDK side, so the per-database engine
callbacks are suppressed.
+ // The DROP owns historical-name resolution. Ordinary replay
lookup also serves CREATE
Review Comment:
[P1] Keep Paimon's live DROP DATABASE cleanup tied to the database just
dropped. In mode 2, old `Foo` can remain cached after `foo` rebinds to current
`FOO`. On a case-insensitive Paimon catalog that retains canonical casing,
`DROP DATABASE Foo` resolves and remotely drops `FOO`; this changed hook now
passes caller `Foo` to historical-ID cleanup, which retires old `Foo` and
leaves dropped `FOO` cached on the leader. Previously this hook resolved the
current canonical DB before unregistering it. Carry that resolved target into
live cleanup and the new drop log; preserve historical handling for delayed old
logs and test both histories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
