mengw15 opened a new issue, #6870: URL: https://github.com/apache/texera/issues/6870
### Feature Summary ## Background Every execution's outputs — results, runtime statistics, and console logs — are currently written to a single shared Iceberg warehouse, with the platform absorbing the S3 cost. There is no per-tenant storage isolation, and no way to attribute storage cost to the user who owns the data. Design discussion: #5293. ## Goal Let each user register **their own S3 bucket** as a *warehouse*. A workflow execution writes its Iceberg tables (results / runtime statistics / console messages) into the selected warehouse's bucket, so: - storage cost is attributed to the user who owns the data, - per-user warehouses provide per-tenant isolation, - users who bring their own storage keep full control of their data. ## Architecture <img width="1048" height="540" alt="Image" src="https://github.com/user-attachments/assets/e63c4869-832f-46aa-8ea7-446148b369f9"; /> ## Design **Warehouse (Design 2, #5293).** A warehouse is a Lakekeeper (Iceberg REST) catalog entity backed by the user's S3 bucket, holding the execution namespaces (results, runtime statistics, console messages) plus its storage configuration. Each warehouse is a distinct Lakekeeper warehouse with its own key-prefix in the bucket. It is selected **per execution**, decoupled from the computing unit. **Two-layer runtime.** - *Catalog path* — a per-warehouse REST catalog client talks to Lakekeeper for metadata. - *Data path* — Iceberg reads/writes Parquet **directly** to the user's S3 using Lakekeeper-vended short-lived credentials. **Credentials: assume-role, no static keys.** The user grants Texera an IAM role; the **platform principal** (this deployment's AWS identity, provisioned once by the operator) assumes it via AWS STS, gated by a **per-warehouse external ID**; workers receive only short-lived STS credentials, vended per execution. The external ID is **server-minted** by the Texera web service (the client cannot choose it — this prevents registering another user's role ARN + external ID to reach their bucket), **per-warehouse**, and **unguessable (UUID)**; it is reserved **before** create so the user wires their role's trust policy in one step (no reveal round-trip). Texera stores only **non-secret metadata** — the Lakekeeper warehouse id, role ARN, and external ID — never raw S3 keys. Assuming a role requires all three at once (platform credentials + external ID + role ARN), so a leaked role ARN + external ID grant nothing on their own. **Warehouse types (flavor).** *Cloud* (AWS) uses the assume-role flow above. *Local* uses this deployment's built-in MinIO (dev / on-prem), where the platform supplies the storage. ## Warehouse setup (user side, Cloud/AWS) When the user opens the Cloud form, Texera shows the **platform principal ARN** and a reserved **external ID**. The user creates one IAM role with: **Trust policy** — allow the platform principal to assume the role, gated by the external ID: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "AWS": "<platform-principal-arn>" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "<external-id>" } } }] } ``` **Permission policy** — grant the role access to the bucket: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketLocation"], "Resource": ["arn:aws:s3:::<bucket>", "arn:aws:s3:::<bucket>/*"] }] } ``` ## Warehouse lifecycle - **Create** — reserve identity (external ID + platform principal ARN) → user creates the IAM role (policies above) → submit bucket / region / role ARN → Texera (via Lakekeeper) validates by **actually assuming the role and test-writing to S3**. A wrong role ARN / external ID / bucket fails **at create time** (HTTP 502), before any record is written — no orphaned state. - **Delete** — the warehouse is emptied in the catalog first (Lakekeeper refuses to drop a non-empty warehouse), dropping tables without purge so the **S3 data files are kept** — it is the user's bucket; the user cleans it up themselves. ## Implementation surface - **Backend** — `WarehouseResource` (REST) · `LakekeeperClient` (warehouse management + empty-first delete) · `StorageConfig` (the `storage.warehouse.enabled` flag) · `WorkflowService` (resolves the per-execution warehouse, ownership-checked) · per-warehouse `IcebergCatalogInstance` cache · warehouse-scoped storage URIs (`DocumentFactory` / `VFSURIFactory`). - **Schema** — `user_warehouse`, `user_s3_pending_identity`. - **Frontend** — Warehouse tab, on-canvas per-execution picker, AWS IAM-role warehouse form, warehouse service. ## Scope & rollout - **Target: multi-tenant (Kubernetes) REST-catalog deployments.** Single-node / local Docker Compose is unchanged — the Postgres catalog and the single shared Lakekeeper warehouse keep working. - Ships behind a **default-off** flag (`storage.warehouse.enabled`). With the flag off, storage behaves exactly as it does today. Deployments opt in once catalog-side authentication (see follow-ups) is in place. ## Sub-issues (each → one PR) - [ ] Storage foundation: per-warehouse Iceberg catalog + warehouse-scoped storage URIs (backward-compatible no-op) - [ ] Feature flag (storage.warehouse.enabled, default off) - [ ] Schema: `user_warehouse` - [ ] Warehouse REST endpoints + Lakekeeper client + per-execution injection - [ ] Frontend: warehouse tab + on-canvas per-execution picker - [ ] Schema: assume-role fields (role ARN / external ID / pending identity) - [ ] Backend: assume-role warehouse creation + reserve-identity - [ ] Frontend: AWS IAM-role warehouse form ## Follow-ups (out of scope here) - **Catalog-side authentication** — the compute↔Lakekeeper channel is currently unauthenticated; a per-execution scoped token validated by the catalog (Lakekeeper OIDC + per-warehouse authorization) is required before enabling the feature in a multi-tenant deployment. - **Warehouse sharing** — per-warehouse access control (ACL) for collaboration. ### Proposed Solution or Design 1 ### Affected Area Storage / Metadata -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
