ringfa11 opened a new issue, #12502:
URL: https://github.com/apache/gravitino/issues/12502
### What would you like to be improved?
Under high concurrency, concurrent reads targeting the same Iceberg table
can overwhelm the underlying storage when the cache is cold or expired. N
concurrent requests may trigger N identical upstream loads, causing redundant
I/O, latency spikes, and unnecessary backend load.
The current read path is:
```
IcebergTableOperations.loadTable / planTableScan
└─ IcebergCatalogWrapper.loadTable / CatalogWrapperForREST.planTableScan
├─ TableMetadataCache / ScanPlanCache: getIfPresent + put
(check-then-act)
└─ cache miss → CatalogHandlers.loadTable / scan planning →
underlying catalog → OSS/metadata store
```
Key observations:
1. `TableMetadataCache`
(`iceberg-common/.../cache/LocalTableMetadataCache.java`) uses `getIfPresent`
followed by `put`, rather than Caffeine's atomic `get(key, mappingFunction)`.
Concurrent misses therefore invoke `IcebergCatalogWrapper.loadTable`
independently and read the same metadata files multiple times.
2. `ScanPlanCache` (`iceberg-rest-server/.../cache/LocalScanPlanCache.java`)
uses the same check-then-act pattern and is disabled by default
(`ScanPlanCache.DUMMY`). In addition, `planTableScan`
(`iceberg-rest-server/.../CatalogWrapperForREST.java`) calls
`getCatalog().loadTable()` before consulting the cache.
As a result, identical concurrent requests are not coalesced.
### How should we improve?
Introduce single-flight semantics into the read path:
1. **`TableMetadataCache` and `ScanPlanCache`**: replace `getIfPresent` +
`put` with a single-flight load, such as Caffeine's `get(key, loader)`, so only
one loader runs per key and concurrent callers share the result. Propagate
loader failures to all waiting callers.
2. Optionally extract a reusable single-flight helper, using shared
`Future`/`CompletableFuture` instances or a thin Caffeine wrapper, with a
configurable per-key timeout.
3. Move `getCatalog().loadTable()` after the scan-plan cache lookup, or
reuse `TableMetadataCache`, so scan-plan cache hits do not access the
underlying catalog.
4. Add unit tests for:
- Concurrent misses on the same key: only one upstream call.
- Concurrent misses on different keys: no cross-key blocking.
- Failure propagation to all waiting callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
