yuqi1129 opened a new pull request, #13702: URL: https://github.com/apache/gravitino/pull/13702
### What changes were proposed in this pull request? - The `CaffeineEntityCache` removal listener no longer takes an entity segment lock. It hands evicted and expired keys to a dedicated single-thread index cleanup executor (unbounded queue, never shut down), which removes them from the prefix index under the key's segment lock on a thread that never holds a Caffeine lock. - The index cleanup removes a key only if it is still absent from the cache, so a `put` that re-adds the key before the cleanup runs keeps its index entry. - Document the eviction-lock paths and the `invalidateExpiredItem` contract. ### Why are the changes needed? Caffeine 2.9.3 can run the removal listener while holding its eviction lock. The shared cleanup executor's `CallerRunsPolicy` runs a task on the submitting thread, which may be the one performing eviction, and Caffeine runs a notification inline if submission throws. Meanwhile, a writer holding a segment lock blocks on the eviction lock in `cacheData.put` when Caffeine cleans up synchronously because its write buffer is full. The listener then waited for that writer's segment lock, which formed the deadlock in the issue. `clear()` could also deadlock with it, because it holds the global write gate while `invalidateAll()` waits for the eviction lock. The listener also removed the index entry unconditionally. If the key had been re-cached before the listener ran, a later cascading invalidation of an ancestor missed the live entry. Upgrading to Caffeine 3 would not remove this cycle. It comes from the listener taking an application lock, and Caffeine 3 also runs notifications inline when the executor cannot take them. Fix: #13580 ### Does this PR introduce _any_ user-facing change? No. Evicted keys now leave the prefix index asynchronously on a dedicated thread, so `size()` may over-count briefly, as it already could. ### How was this patch tested? New `TestCaffeineEntityCacheEviction`: - Eviction with a direct maintenance executor completes while the evicted keys' segment locks are held. On the old listener it waits on the segment lock and times out. - A stale eviction notification arriving after the key was re-cached keeps the entry indexed, so invalidating its metalake still drops the child catalog. Without the presence check, the index loses it. - Eight writers with the production executors and `maxEntries=64` finish within a bounded time, and the index converges to the live entries. With the old inline listener, this deadlocked (`TimeoutException`). `org.apache.gravitino.cache.*`, `TestEntityCacheChangeLogListener`, `TestEntityCacheCrossNodeInvalidation`, `TestRelationalEntityStoreBatchGetLateFill` and `TestRelationalEntityStoreHierarchicalCache` pass. `:core:spotlessCheck`, `:core:javadoc`, and the jcstress and jmh source sets compile. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
