kz930 opened a new issue, #7921: URL: https://github.com/apache/texera/issues/7921
### What happened? `lakekeeper-init` refreshes the existing warehouse's stored S3 endpoint on every `up`, so that the endpoint follows the host LAN IP across networks. That refresh now fails for any warehouse created before #7740, and the endpoint is never updated. #7740 added `key-prefix` to both the create and the update payload in `bin/single-node/docker-compose.yml`. A warehouse created before it has `key_prefix: null`, and Lakekeeper refuses to change that field at all, so it rejects the whole request with HTTP 400 and `Field 'key_prefix' cannot be updated to prevent loss of data.` The endpoint is part of the same payload, so it stays stale forever. The consequence is not obvious from the symptom. Once the host's LAN IP changes, every execution fails before any operator runs: `IcebergUtil.createTable` posts to the catalog, waits about thirty seconds on an S3 connect timeout to the old address, and gets an empty body back, which surfaces as `RESTException: Unable to process:` with nothing pointing at the endpoint or the warehouse. A second failure hides behind it, because that first failed execution leaves the workflow's service cached in a state where every later run throws a `NullPointerException` on `executionRuntimeService` before starting, which the UI shows as being stuck on "Submitting" indefinitely. Reloading the page does not clear it. A fresh checkout is unaffected: the warehouse is created with the same prefix the refresh later sends, so the values match. What is affected is any environment whose warehouse predates 2026-08-19, which is every contributor who has been running the local dev stack since before that date. Expected: the refresh updates the endpoint and leaves `key_prefix` as it is, rather than failing because of a field it did not intend to change. ### How to reproduce? 1. Bring the stack up on a checkout from before #7740, so the warehouse is created without a key prefix. 2. Bring it up again on a checkout that includes #7740, on a machine whose LAN IP has changed since step 1. 3. `lakekeeper-init` exits 1 with the 400 above. `bin/local-dev.sh up` does not stop for it, so the run continues and looks successful. 4. Run any workflow. It fails before any operator runs. Every run after that hangs on "Submitting" until the service is bounced. Confirm the cause by reading the stored profile with `curl -s http://localhost:8181/management/v1/warehouse/<id>` and comparing its `endpoint` against the machine's current address. Two things worth noting for whoever picks this up. The init exits non-zero but the stack still reports as up, so the failure is only visible if you read that container's log. And the same request carries both the field being changed and the field that cannot be changed, so sending the stored `key_prefix` back, or omitting it on the update path, is enough to fix it. ### Version/Branch 1.3.0-incubating-SNAPSHOT (main) ### Commit Hash (Optional) 51f38e13d ### Relevant log output Lakekeeper Warehouse 'texera' already exists. Refreshing its storage endpoint to 'http://texera-minio:9000'... Failed to refresh Lakekeeper Warehouse storage. HTTP Code: 400 ERROR RESPONSE: {"error":{"message":"Field `key_prefix` cannot be updated to prevent loss of data.","type":"UpdateError","code":400}} [WARN] [org.apache.iceberg.rest.ErrorHandlers] - Unable to parse error response java.io.UncheckedIOException: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input [ERROR] [org.apache.texera.web.service.WorkflowService] - error during execution org.apache.iceberg.exceptions.RESTException: Unable to process: at org.apache.iceberg.rest.RESTCatalog.createTable(RESTCatalog.java:132) at org.apache.texera.amber.util.IcebergUtil$.createTable(IcebergUtil.scala:225) at org.apache.texera.web.service.WorkflowExecutionService.executeWorkflow(WorkflowExecutionService.scala:132) [ERROR] [org.apache.texera.web.resource.WorkflowWebsocketResource] - error occurred in websocket java.lang.NullPointerException: Cannot invoke "org.apache.texera.web.service.ExecutionRuntimeService.unsubscribeAll()" because the return value of "org.apache.texera.web.service.WorkflowExecutionService.executionRuntimeService()" is null at org.apache.texera.web.service.WorkflowExecutionService.unsubscribeAll(WorkflowExecutionService.scala:182) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
