github-actions[bot] commented on code in PR #66717:
URL: https://github.com/apache/doris/pull/66717#discussion_r3825382823
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java:
##########
@@ -195,12 +208,11 @@ public void refreshCatalog(long catalogId) {
public void invalidateCatalog(long catalogId) {
super.invalidateCatalog(catalogId);
advanceFileCacheInvalidationGeneration(catalogId);
- // The catalog is being removed (drop or property-driven full
rebuild); its id is never
- // reused, so drop the generation counters instead of letting them
accumulate forever.
- // The advance above already invalidated any in-flight statement keyed
on the old
- // generation; a later get() rebuilds them from zero for a catalog
that no longer exists.
- fileCacheInvalidationGenerations.remove(catalogId);
- fileCacheValueGenerations.remove(catalogId);
+ // A cache-policy ALTER rebuilds the catalog's cache group under the
SAME catalog id, so
+ // the generation counters must stay monotonic: statement-scoped
file-task keys embed
+ // these numbers, and restarting them at zero would let a statement
planned before the
+ // rebuild reuse stale file tasks afterwards. The retained state is
two counters per
+ // catalog id ever seen, which is bounded and negligible.
Review Comment:
[P2] Prune these generations on permanent catalog removal
True DROP/replay-DROP also reaches this method, but it now leaves both
per-catalog generation maps populated forever. Each later CREATE gets a fresh
getNextId(), so create/use/drop churn accumulates unreachable
Long/AtomicLong/map nodes outside the new cache budget for the FE lifetime.
Keeping the counters is necessary for same-ID policy rebuilds and renames (the
prior thread), so please add an explicitly permanent-drop-aware hook/reason
that reaches Hive even if its entry group is already retired and fences an
in-flight scan from recreating the record. Add a create-use-drop cardinality
test alongside the rebuild test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
