GitHub user eugenegujing added a comment to the discussion: What should Texera do when a computing unit dies?
Thanks @kunwp1 and @yrenat. I think this settles all three questions. Here's the direction I'll take in #7669, so the PR review can point back to this thread: 1. **Status vocabulary: mirror Kubernetes.** I'll add `Failed` / `Unknown` (and `Terminating`) to `ComputingUnitState` and drop the idea of a Texera-level abstraction. 2. **Failure detail: owner-only, actionable.** Per @kunwp1's two production scenarios, the `statusReason` will be a human-actionable message rather than raw k8s output — e.g. OOMKilled → "The unit ran out of memory; consider recreating it with a higher memory limit", disk pressure/eviction → "The unit ran out of local storage; consider storing less data on the local file system". Shared (READ) users will only see a generic "This unit is unavailable". 3. **No auto-recovery.** Failed stays failed; the user recreates with a different configuration. I verified the existing terminate flow already handles this: the delete action is available to the owner regardless of status, and `KubernetesClient.deletePod` is a no-op when the pod is already gone — so what's missing is only the signal telling the user the unit is dead, which is exactly what this change adds. @yrenat Thanks for the pointer to #2123, and I really like the idea of inferring liveness with zero new infrastructure. But I'm afraid it might not work as the primary signal for this particular issue, if I'm reading the code right, `last_update_time` is stamped on state transitions rather than as a heartbeat, so a long-running healthy execution could look the same as a dead CU, and a CU stuck at "(Connecting)" usually has no executions yet, so the table may have no signal for the exact case this issue targets. It also doesn't seem able to tell apart the failure causes (OOM vs. out-of-storage) we want to surface, which I believe only exist in pod state. I will start working on this for a PR in our current design. GitHub link: https://github.com/apache/texera/discussions/7670#discussioncomment-18085328 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
