bharos opened a new issue, #12170: URL: https://github.com/apache/gravitino/issues/12170
### Version main (also affects 1.3.0; present since 1.2.1) ### Describe what's wrong Under a workload that mixes metadata **list** and **create** operations on a large schema (authorization + entity cache enabled), create throughput collapses, while each workload alone is healthy: | Workload (alone) | Result | | --- | --- | | create-only | healthy (p50 ~3s) | | list-only | slow but stable, O(N), 0 failures | | **list + create concurrently** | **creates hang** — in a deterministic 25-lister/25-creator, 5-min run, no create returned within the window (0 completions, 0 failures; the creates were blocked server-side, not erroring) | Only the *interaction* collapses, which points at a shared lock (CPU, DB connection pool, and the authz thread pool were all measured and ruled out — DB pool peaked ~10/100, authz pool idle). ### Root cause `RelationalEntityStore.batchListEntitiesByRelation` wraps the backend batch-list **DB round-trip inside `cache.withMultipleKeyCacheLock(...)`**. `SegmentedLock.withMultipleKeyLockAndThrow` acquires **all** the cache segment locks for the batch's keys and holds them **across the DB call** — a classic I/O-under-lock antipattern. - **List side:** authorization preloads owners for the whole page — `MetadataAuthzHelper.preloadOwner` → `batchListEntitiesByRelation(OWNER_REL, <all idents>)` — so one list acquires a near-global set of segment locks for the full DB latency. - **Create side:** `createTable` sets the owner (`insertRelation` → `CaffeineEntityCache.invalidate` → `SegmentedLock.withLock`), needing a single segment that is almost always inside the set the list holds → every create blocks for the list's entire DB round-trip. Under sustained lists and an unfair `ReentrantLock`, create waiters can be starved indefinitely. A thread dump under load shows the create handler threads BLOCKED, parked on cache **segment `ReentrantLock`s** in `createTable → setOwner → insertRelation → CaffeineEntityCache.invalidate → SegmentedLock.withLock`. Raising `gravitino.cache.lockSegments` does **not** help: the multi-key list simply grabs proportionally more segments, and objects that share an owner concentrate keys onto a few segments. ### Impact Any deployment with authorization + entity cache enabled can see create/alter operations stall (seconds to minutes, or hang under sustained lists) whenever large list requests run concurrently. ### How to reproduce 1. Enable authorization and the entity cache. 2. Create a schema with a large number of tables. 3. Concurrently run `listTables` (which triggers per-object owner preload) and `createTable` against that schema. 4. Observe create latency spike / requests hang while lists are in flight. ### Proposed fix Restructure `batchListEntitiesByRelation` as a plain cache-aside read that never holds a cache lock across backend I/O: (1) read the cache lock-free (`getIfPresent`) to partition cached vs uncached, (2) run the backend batch-list with **no** cache lock held, (3) populate via `cache.put` (which takes its own brief per-key lock). Caffeine is already thread-safe, so no lock needs to span the DB call. In a load test this restored creates from **0 → ~1,450 / 5 min** with no per-request regression to individual lists, and `create_table` p50 back to baseline parity. I'll attach a draft PR with the fix + a regression test for discussion. ### Additional context The single-key relation paths (`get`, single `listEntitiesByRelation`, `getEntityByRelation`) are unaffected and remain atomic. A separate, residual list-latency item (the per-object DB work list authorization now does) is out of scope for this fix. ### Willingness to contribute I can contribute a PR (draft attached). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
