924060929 commented on code in PR #68196:
URL: https://github.com/apache/doris/pull/68196#discussion_r4133990337
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##########
@@ -626,12 +618,20 @@ private List<Pair<String, String>>
getFilteredDatabaseNames() {
return remoteToLocalPairs;
}
- protected boolean isDatabaseAllowedByFilter(String dbName) {
- return isDatabaseAllowedByFilter(dbName, getIncludeDatabaseMap(),
getExcludeDatabaseMap(), false);
+ /** Use the same local identity for database discovery and HMS create
events. */
+ protected final String localDatabaseNameFromRemote(String remoteDbName) {
Review Comment:
Fixed in 04fa4afd6b2. The Turkish-locale reproduction failed on the previous
head: mode-1 CREATE cached remote I as local i, but getDbNullable(I) returned
null. Database discovery, mode-1/2 lookup, event IDs, names routes, and Paimon
SDK invalidation now share Locale.ROOT normalization. The CREATE/lookup/DROP
and mode-2 replay tests, including the Paimon locale test, pass.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalRowCountCache.java:
##########
@@ -24,43 +24,77 @@
import org.apache.doris.statistics.BasicAsyncCacheLoader;
import org.apache.doris.statistics.util.StatisticsUtil;
+import com.github.benmanes.caffeine.cache.AsyncCacheLoader;
import com.github.benmanes.caffeine.cache.AsyncLoadingCache;
+import com.github.benmanes.caffeine.cache.Ticker;
import lombok.Getter;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.util.Objects;
import java.util.Optional;
import java.util.OptionalLong;
+import java.util.Set;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
public class ExternalRowCountCache {
private static final Logger LOG =
LogManager.getLogger(ExternalRowCountCache.class);
private final AsyncLoadingCache<RowCountKey, Optional<Long>> rowCountCache;
+ private final ConcurrentHashMap<LoadKey, Set<LoadFence>> inFlightLoads =
new ConcurrentHashMap<>();
+ // A catalog refresh changes this generation in O(1). Old cache entries
remain bounded by
+ // Caffeine's size/expiry policy but are never addressable by a new
metadata generation.
+ private final ConcurrentHashMap<Long, AtomicLong> catalogGenerations = new
ConcurrentHashMap<>();
Review Comment:
Fixed in 04fa4afd6b2. Catalog invalidation no longer allocates a generation
when no row count was loaded. Permanent DROP releases an existing generation
after engine cleanup, while rename preserves it. Completion-time and if-present
checks do not recreate a released entry; a latching old-reader test verifies
UNKNOWN and zero retained generations. The permanent-vs-rename route is also
tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
