GitHub user eugenegujing edited a discussion: What should Texera do when a computing unit dies?
I noticed that a computing unit that is no longer alive is indistinguishable, in the UI, from one that is still booting. Both render as `(Connecting)`. > <img width="1305" height="867" alt="Screenshot 2026-08-13 at 5 26 50 PM" src="https://github.com/user-attachments/assets/7d289051-8ec1-49cd-93f8-dc0baa89fc49"; /> > > *Caption: `CU 2` is healthy (green, selected, workflow running). `CU 1` shows > the gold dot and the literal `(Connecting)` suffix. That label is all the UI > will ever say about it — at two seconds old and at two hours dead alike. The > tooltip reads "Computing unit is starting up".* This isn't a rendering bug. **Texera has no representation of an unhealthy computing unit anywhere in the stack**, so there is nothing for the UI to render: - `ComputingUnitState` (`ComputingUnitState.scala:23`) defines exactly `Running` and `Pending`. - `ComputingUnitHelpers` (`:86`) resolves status as `if (phase == "Running") Running else Pending`. - The frontend DTO type (`workflow-computing-unit.ts:49`) is `status: "Running" | "Pending"` — failure isn't expressible. - `local` CUs are hard-coded to `Running` (`:74-75`) with no liveness check at all, so a dead local process shows green. The frontend already has red/error branches (`computeStatus()`, `getComputingUnitBadgeColor()`). They're dead code, because only `Running` and `Pending` ever reach them. Someone anticipated failure states; the backend never grew them. I've filed the mechanical part as #7669. Three questions I'd rather settle here than inside a PR review. ### 1. What should the status vocabulary be? - **Mirror Kubernetes** — `Running / Pending / Failed / Unknown / Terminating`. Trivial to map, but it leaks the deployment substrate into a user-facing API and means nothing for `local` CUs. - **Texera-level abstraction** — something like `Starting / Ready / Unhealthy / Gone`, named for what the user can *do* with the unit rather than for what k8s calls it. More mapping work, but it stays honest across local, k8s, and whatever comes next. I lean toward the second, but it needs someone to define what "unhealthy" means for a `local` CU. ### 2. How much cluster detail should reach the end user? "Your unit was OOM-killed" is actionable — raise the memory limit. "ImagePullBackOff: manifest unknown" is not something a data scientist can act on and arguably shouldn't be in their face. There's also a sharing dimension: CU access can be shared (`ComputingUnitAccessResource`), so a failure reason reaches READ-privilege users who aren't the owner. Are we comfortable showing pod-level error text to everyone a unit is shared with, or should detail be owner-only with a generic "This unit is unavailable" for others? ### 3. Should a failed CU recover by itself, and is the current answer the one we want? We already have a recovery policy: **restart the container forever, never replace the pod.** It falls out of `KubernetesClient.createPod` (`:117-214`), which builds a **bare `Pod`** — no owning Deployment or other controller — and never sets `restartPolicy`, so it defaults to `Always`. That splits by failure level: - **Container-level failure → retried forever, invisibly.** Crash or OOM-kill, the kubelet restarts it in place, indefinitely. Texera isn't involved and doesn't report it. - **Pod-level loss → never recovered.** Evicted, node drained, node dead: a bare pod has no controller to recreate it. Gone permanently. There's no server-side reaper (no cleanup or reconcile job in `computing-unit-managing-service`), so a dead pod's DB row just persists. Keep delegating to k8s defaults, make it explicit with an owning controller and a restart cap, or move the policy into Texera? ### The proposal in #7669, and whether there's a better one 1. **Add terminal/unhealthy values to `ComputingUnitState`** 2. **Resolve status from real pod state** and carry the reason as a `statusReason` string so the UI can eventually say *why*. GitHub link: https://github.com/apache/texera/discussions/7670 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
