GitHub user yrenat created a discussion: Improve Workflow Runtime Observability
with Execution Status and Flow Control Visualization
## Motivation
Texera allows users to design and execute complex workflows, but during
execution it is still difficult to understand where data flow is being
constrained. When a workflow becomes slow or appears stuck, users may want to
know:
- Which part of the workflow is currently under pressure?
- Is an operator slow because it is processing data, waiting for upstream
input, or blocked by downstream backpressure?
- Which workflow edge is accumulating pressure?
- Is the bottleneck caused by data flow between operators rather than by the
operator state alone?
Texera already exposes multiple operator-level runtime information, such as
execution state and operator statistics. However, flow-control/backpressure
information is currently kept mostly inside the execution engine and is not
visible in the workflow graph to the users.
This proposal focuses on exposing workflow-level flow-control status so users
can better understand runtime bottlenecks without changing the underlying
execution behavior.
## Proposed Solution
This proposal focuses on exposing workflow-level flow-control status by
reusing Amber’s existing credit-based flow-control state.
Amber already tracks per-channel flow-control information, such as available
credit, queued credit, and whether a channel is overloaded. The backend should
periodically summarize this channel-level state into workflow-level link
pressure information and publish it through the existing workflow websocket
event mechanism.
The first version is designed to expose a compact runtime event containing,
for each affected logical workflow link:
- Source operator id.
- Target operator id.
- Target input port id.
- Whether the link is currently overloaded.
- Current/max credit usage, or utilization percentage.
- Optional aggregated queued/inflight byte counts.
Because physical execution may use multiple worker-to-worker channels for one
logical workflow edge, the backend should aggregate channel-level signals into
one logical-link status before sending it to the frontend.
The frontend can then render this as a runtime overlay on the workflow graph.
Edges under pressure can be highlighted by color, with compact labels or
tooltips showing utilization details.
## Backend Design
The backend will reuse the existing flow-control implementation instead of
introducing a separate monitoring system.
The expected backend work includes:
1. Collect flow-control snapshots from existing runtime state.
Amber already maintains per-channel flow-control state internally. The
implementation should expose a read-only snapshot of relevant values, such as:
- available credit
- maximum credit
- queued credit
- overloaded/backpressured status
2. Aggregate physical channel state into logical workflow-link state.
Flow control is tracked at the worker-channel level, while the frontend
visualizes logical workflow links. Therefore, the backend should group
channel-level information by logical workflow connection, such as:
```text
worker channel metrics
-> source logical operator + target logical operator + target input port
-> logical link pressure status
3. Publish a compact websocket event.
A possible event shape is:
```text
FlowControlStatusUpdateEvent {
links: {
sourceOperatorId: string;
targetOperatorId: string;
targetPortId: string;
overloaded: boolean;
queuedCreditBytes?: number;
availableCreditBytes?: number;
maxCreditBytes?: number;
utilization?: number;
};
}
## Frontend Design
The frontend will use the new flow-control status event and render it as a
workflow graph overlay.
The first version may have a focus on edge-level visualization:
- Use edge color to indicate pressure severity.
- Show compact edge labels only when useful, such as when pressure is
non-zero or when a runtime/debug overlay is enabled.
- Show detailed values, such as utilization percentage or queued bytes, in a
tooltip or selected-edge details panel.
- Clear the visualization when execution completes, fails, is killed, or the
websocket connection closes.
The visualization should avoid overloading existing operator-state colors.
Operator status is already used to show execution state, so flow-control
pressure should primarily be shown on workflow edges.
## Related Issues
- #4321: Support flow control/backpressure visualization in backend.
- #4322: Support flow control/backpressure visualization in frontend.
## Discussions are welcomed
1. Should the flow-control status be exposed as a new event in the existing
workflow websocket mechanism?
2. What exact fields should be included in the flow-
control status event?
3. Should the frontend always show pressure labels, or should labels only
appear when the runtime/debug overlay is enabled?
4. What update frequency is appropriate to balance responsiveness and
overhead?
GitHub link: https://github.com/apache/texera/discussions/6401
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
