The GitHub Actions job "Required Checks" on texera.git/main has succeeded. Run started by GitHub user github-merge-queue[bot] (triggered by github-merge-queue[bot]).
Head commit for run: a09260a46cbfa844537b8865ff0f79754bf3daa5 / Meng Wang <[email protected]> feat(storage): bound the per-warehouse catalog cache and release evicted catalogs (#7539) ### What changes were proposed in this PR? **TL;DR**: Texera keeps one Iceberg catalog client (an HTTP client + its connection pool) per warehouse, in a process-wide registry that never removes entries. That used to be harmless: there was effectively one shared warehouse, so the registry held one entry forever. With per-user warehouses (#6870) users create warehouses freely, and every warehouse a long-lived JVM (web server, computing unit) ever touches adds one more permanently-held client — the registry only grows for the life of the process, accumulating connection pools that are never released. This PR (1) bounds that registry and closes clients that have gone idle, and (2) reworks every reader/writer that used to pin a client reference long-term to re-resolve it per operation — which is what makes releasing clients safe. Everything below is the detail of those two moves. `IcebergCatalogInstance` kept one catalog client per warehouse name for the life of the process; with per-user warehouses (#6870) that set is unbounded, and each REST catalog holds an HTTP client. The map is now a Guava cache (`maximumSize` 64 + `expireAfterAccess` 60 min, mirroring `HuggingFaceModelResource`'s bounded-cache precedent). An entry idle for the expiry window is closed — nothing can be using it, and idle entries are exactly what a long-lived JVM accumulates. An entry evicted by size is only dropped, never closed: size pressure means more simultaneously hot warehouses than the bound, and closing a hot catalog would fail the operations still using it. Load degrades into rebuild churn, not errors — a dropped catalog lives only as long as its in-flight operations (per-operation resolution bounds every borrow), after which GC reclaims it while the server's keepalive timeout severs its idle connections. For eviction to be safe, holders stop pinning a catalog — or anything derived from one — across a logical operation: `IcebergDocument`'s `lazy val` becomes a per-use `def`, its `clear()` resolves one catalog for the whole check-then-drop, the reader re-resolves its table on every seek instead of refreshing a pinned one (which also keeps a polling reader's cache entry live), and `IcebergTableWriter` takes the warehouse rather than a `Catalog` and loads its table per flush. The lookup rides Guava's per-key locking, dropping the previous JVM-wide `synchronized` that held one lock across a cache miss's REST config round trip, and unwraps Guava's `ExecutionException` family so `createCatalog` failures keep the types they had before. Only idle-expired entries are closed. `replaceInstance` stays a plain `put`: a caller that replaces an entry may still hold and later restore the old reference — amber's integration spec wrap-and-restores the shared catalog, and endpoint reconfiguration (#7358) will swap catalogs the same way. Also dedupes `DocumentFactory`'s three copies of the URI→(warehouse, namespace, storage key) decode block into one resolver, as promised in #6944 review. The Python side is untouched: a PVM is spawned per worker and destroyed when the execution ends, so its catalog dict holds the single warehouse that execution used and dies with the process — nothing accumulates there to bound. ### Any related issues, documentation, discussions? Closes #7290. ### How was this PR tested? New `IcebergCatalogInstanceSpec` covers the cache contract: idle expiry closes the catalog while size eviction drops it un-closed — on isolated caches built through a package-private factory with a manual ticker, so the JVM-wide cache that parallel suites share is never touched — a catalog displaced by `replaceInstance` stays open for its owner, and loader failures keep their original exception type. Per-use resolution is pinned at every holder: `IcebergDocument` sees a replacement immediately, `clear()` addresses one catalog across its check-then-drop, the reader re-resolves per seek, and the writer loads through the catalog installed at flush time. Existing iceberg suites (`IcebergDocumentSpec`, `IcebergTableWriterSpec`, `OnIcebergSpec`, `DocumentFactorySpec`) pass locally — 769 tests in `workflow-core` — and amber's integration `IcebergDocumentSpec` is green in CI. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-fable-5) Report URL: https://github.com/apache/texera/actions/runs/31745103230 With regards, GitHub Actions via GitBox
