github-actions[bot] commented on code in PR #68196:
URL: https://github.com/apache/doris/pull/68196#discussion_r4132632326
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##########
@@ -626,12 +618,20 @@ private List<Pair<String, String>>
getFilteredDatabaseNames() {
return remoteToLocalPairs;
}
- protected boolean isDatabaseAllowedByFilter(String dbName) {
- return isDatabaseAllowedByFilter(dbName, getIncludeDatabaseMap(),
getExcludeDatabaseMap(), false);
+ /** Use the same local identity for database discovery and HMS create
events. */
+ protected final String localDatabaseNameFromRemote(String remoteDbName) {
Review Comment:
[P2] Keep mode-1 database lookup on the same case rule as discovery. On a
Turkish-locale FE, remote I is now listed and cached as local i by Locale.ROOT,
but getLocalDatabaseName(I) still uses the default locale and seeks dotless ı.
getDbNullable(I) then misses; after an HMS CREATE event, a DROP event for I
invalidates ı and leaves the cached i database visible. Use one normalization
in discovery, lookup, event IDs, and connector invalidation, with a
Turkish-locale CREATE/DROP test. This is separate from the existing Paimon
SDK-cache thread.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalRowCountCache.java:
##########
@@ -24,43 +24,77 @@
import org.apache.doris.statistics.BasicAsyncCacheLoader;
import org.apache.doris.statistics.util.StatisticsUtil;
+import com.github.benmanes.caffeine.cache.AsyncCacheLoader;
import com.github.benmanes.caffeine.cache.AsyncLoadingCache;
+import com.github.benmanes.caffeine.cache.Ticker;
import lombok.Getter;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.util.Objects;
import java.util.Optional;
import java.util.OptionalLong;
+import java.util.Set;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
public class ExternalRowCountCache {
private static final Logger LOG =
LogManager.getLogger(ExternalRowCountCache.class);
private final AsyncLoadingCache<RowCountKey, Optional<Long>> rowCountCache;
+ private final ConcurrentHashMap<LoadKey, Set<LoadFence>> inFlightLoads =
new ConcurrentHashMap<>();
+ // A catalog refresh changes this generation in O(1). Old cache entries
remain bounded by
+ // Caffeine's size/expiry policy but are never addressable by a new
metadata generation.
+ private final ConcurrentHashMap<Long, AtomicLong> catalogGenerations = new
ConcurrentHashMap<>();
Review Comment:
[P3] Release generation state after a permanent catalog drop.
CatalogMgr.removeCatalog calls invalidateRowCountCache(catalogId) for every
DROP, and invalidateCatalog inserts an entry here even if no row count was
loaded. removeCatalogPermanently retires engine state but never removes this
map entry, so repeated CREATE/DROP CATALOG grows the registry for the FE
lifetime outside Caffeine's size limit. Reclaim retired IDs while preserving
the fence for outstanding readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
