mengw15 opened a new issue, #6929:
URL: https://github.com/apache/texera/issues/6929
### Task Summary
Foundation step of the per-user BYO-S3 warehouse feature (part of #6870).
Today the storage layer
only ever talks to one warehouse; this teaches it to handle several —
without changing any
behavior yet.
- **Per-warehouse REST catalog cache.** The Iceberg catalog client — which
resolves tables and
reads/writes their metadata against Lakekeeper — is today a single shared
instance
(`IcebergCatalogInstance` on the JVM, `iceberg_catalog_instance.py` for
Python workers). Turn it
into a cache **keyed by warehouse name**, so one worker process can hold
several REST catalogs at
once, one per warehouse it touches. Only the REST (Lakekeeper) catalog
varies by warehouse; the
hadoop/postgres catalogs stay warehouse-agnostic and keep sharing one.
`getInstance(warehouse)`
falls back to the configured default when no warehouse is given.
- **Warehouse-scoped storage URIs.** Every stored result is identified by an
internal VFS URI (a
path). Add the warehouse name as a leading `/wh/<name>` segment so the URI
itself says which
warehouse the data lives in; the reader parses it back out
(`VFSURIFactory.warehouseFromURI`) and
`DocumentFactory` opens the matching catalog. When the segment is absent,
the URI is
byte-for-byte identical to today's — existing data and behavior are
untouched.
- **Per-execution warehouse in the context.** Add a `warehouse:
Option[String]` field to
`WorkflowContext` so the selected warehouse flows from the request down to
the workers.
Backward-compatible no-op: with no warehouse selected (the default), the
catalog and URIs behave
exactly as they do today.
### Task Type
- [x] Other (feature implementation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
