mengw15 opened a new issue, #7739:
URL: https://github.com/apache/texera/issues/7739
### What happened?
With `storage.warehouse.enabled = true`, creating any per-user warehouse
fails. Lakekeeper rejects it:
```
Lakekeeper create warehouse 'user-1-test' failed (HTTP 400):
{"error":{"message":"Storage profile overlaps with existing warehouse
texera",
"type":"CreateWarehouseStorageProfileOverlap","code":400}}
```
No per-user warehouse can be created on a standard deployment, so the Phase
0 exit criterion in #6870 — "a deployment can turn the flag on at the end of
Phase 0" — does not hold.
**Root cause.** The deployment scripts create the default warehouse
**without a `key-prefix`**, so it claims the entire bucket:
- `bin/single-node/docker-compose.yml` — the create payload has no
`key-prefix` field at all
- `bin/k8s/values.yaml` — `keyPrefix: ""`, documented as "Empty = bucket
root"; the init job omits the field when it is empty
`WarehouseResource.create` (#6932) places each per-user warehouse in the
**same** bucket under `key-prefix = user-<uid>-<name>`. Any such prefix is
contained in the default warehouse's location, which Lakekeeper treats as an
overlap and refuses.
**Why tests did not catch it.** `LakekeeperClientSpec` drives an in-process
HTTP stub, which performs no overlap validation; the amber integration suite
does not exercise the warehouse management API. The conflict lives between
deployment configuration and a code assumption, so it is invisible in a diff
and only reproduces end to end.
### How to reproduce?
1. Bring up a standard deployment (`bin/single-node/docker-compose.yml`) so
the default warehouse is created.
2. Set `storage.warehouse.enabled = true` and restart the services.
3. Dashboard → Warehouses → Create Warehouse → any valid name.
4. The request fails with `CreateWarehouseStorageProfileOverlap`.
### Proposed fix
Give the default warehouse its own `key-prefix` (e.g. `shared`) in both
deployment paths, so every warehouse — the platform's and the users' — is a
sibling under its own prefix. `WarehouseResource` needs no change.
**Upgrade impact.** Lakekeeper refuses to change `key_prefix` on an existing
warehouse (`Field key_prefix cannot be updated to prevent loss of data`), so an
existing deployment must drop and recreate its default warehouse. The cost is
bounded: `ComputingUnitMaster.recurringCheckExpiredResults` unconditionally
purges results, runtime statistics and logs of completed executions past
`result-cleanup.ttl-in-seconds` (default 86400), so only in-flight and same-day
executions are affected. Texera addresses the warehouse by name, not id, so
nothing else needs updating.
Verified end to end locally: after recreating the default warehouse with
`key-prefix: shared`, per-user warehouses create successfully and an execution
targeting one writes its results, runtime statistics and console messages
entirely under that warehouse's prefix — the shared prefix stays empty.
### Affected Area
Storage / Metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
