Yicong-Huang opened a new pull request, #6011:
URL: https://github.com/apache/texera/pull/6011
### What changes were proposed in this PR?
A fast **source** operator stayed **orange (RUNNING)** in the editor after
the run finished (issue #6010).
**Why it's a bug.** The controller reconstructs each worker's state from
three unordered channels (source `RUNNING` via the `startWorker` response,
non-source `RUNNING` via `workerStateUpdated`, `COMPLETED` via the
`queryStatistics` response) and reconciled them with
**last-`System.nanoTime()`-wins** in `WorkerExecution`. Worker state, however,
is single-writer and strictly ordered causally. For a tiny source the run
finishes almost instantly, so the `startWorker` response — carrying the stale
`RUNNING` it sampled at launch — can reach the controller *after* `COMPLETED`
was recorded. Its later receipt timestamp wins and clobbers `COMPLETED`.
Results render fine (separate path), so only the border is stuck.
This PR orders worker state **causally** instead of by wall clock:
| Before | After |
| --- | --- |
| `WorkerExecution.update(nanoTime, state)` — last-write-wins by receipt
time | `updateState(version, state)` — newest **logical version** wins |
| stale late RUNNING clobbers COMPLETED | RUNNING (v2) < COMPLETED (v3) ⇒
COMPLETED kept |
| — | terminal states (COMPLETED/TERMINATED) are absorbing |
- `WorkerStateManager` bumps a monotonic `stateVersion` on every `transitTo`
(its state-machine logical clock).
- The version rides on every state report: `WorkerStateResponse`,
`WorkerStateUpdatedRequest`, `WorkerMetrics` (3 new proto fields).
- The controller applies a state only when its version is newer. Stats stay
timestamp-ordered (monotonic snapshots can share a state version).
- A single per-worker counter ⇒ no cross-process clock-sync assumptions (why
not worker timestamps).
### Any related issues, documentation, discussions?
Closes #6010. The timestamp-based `update` was introduced in #3557.
### How was this PR tested?
New/updated unit tests, run on JDK 17:
```
sbt "WorkflowExecutionService/testOnly \
org.apache.texera.amber.engine.architecture.deploysemantics.layer.WorkerExecutionSpec
\
org.apache.texera.amber.engine.common.statetransition.WorkerStateManagerSpec \
org.apache.texera.amber.engine.architecture.controller.execution.OperatorExecutionSpec"
# 47 succeeded, 1 pending (pre-existing)
sbt "WorkflowExecutionService/testOnly *RegionExecutionCoordinatorSpec \
*WorkflowExecutionCoordinatorSpec *RegionExecutionSpec
*WorkflowExecutionSpec"
# 43 succeeded
```
- `WorkerExecutionSpec`: version ordering (positive + stale/equal-version
negatives), terminal-state absorption, `forceTerminate`, independent
stats-vs-state ordering, and a named regression for #6010 (`COMPLETED` survives
a late `RUNNING`). Verified the regression test goes **red** when `updateState`
is reverted to last-write-wins.
- `WorkerStateManagerSpec`: version starts at 0, bumps per successful
transition, and does not bump on a no-op self-transition or a rejected
transition.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8 (1M context), via Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
