eugenegujing opened a new issue, #8853:
URL: https://github.com/apache/texera/issues/8853
### What happened?
`WorkflowWebsocketService` learns that the connection to the computing unit
is lost only through `retryWhen`, that is, only when the socket **errors**.
```ts
//
frontend/src/app/workspace/service/workflow-websocket/workflow-websocket.service.ts:117-136
this.websocket = webSocket<TexeraWebsocketEvent |
TexeraWebsocketRequest>(websocketUrl);
// setup reconnection logic
const wsWithReconnect = this.websocket.pipe(
retryWhen(errors =>
errors.pipe(
tap(_ => this.updateConnectionStatus(false)), // update connection
status
tap(_ =>
console.log(`websocket connection lost, reconnecting in
${WS_RECONNECT_INTERVAL_MS / 1000} seconds`)
),
delayWhen(_ => timer(WS_RECONNECT_INTERVAL_MS)), // reconnect
after delay
tap(_ => {
this.send("HeartBeatRequest", {}); // try to send heartbeat
immediately after reconnect
})
)
)
);
// set up event listener on re-connectable websocket observable
this.wsWithReconnectSubscription = wsWithReconnect.subscribe(event =>
this.webSocketResponseSubject.next(event as TexeraWebsocketEvent)
);
```
When the server closes the socket with a close frame, rxjs 7.8.2 (pinned in
`frontend/package.json`) does not error the subject. It completes it:
```js
//
frontend/node_modules/rxjs/dist/cjs/internal/observable/dom/WebSocketSubject.js:192-205
socket.onclose = function (e) {
if (socket === _this._socket) {
_this._resetState();
}
var closeObserver = _this._config.closeObserver;
if (closeObserver) {
closeObserver.next(e);
}
if (e.wasClean) {
observer.complete();
}
else {
observer.error(e);
}
```
A completion passes straight through `retryWhen`. So after a clean close:
- `updateConnectionStatus(false)` is never called, and `isConnected` (a
`BehaviorSubject` read) stays `true`.
- Nothing redials, because `wsWithReconnectSubscription` has simply
completed.
- `send()` calls `this.websocket?.next(...)`. With no open socket the frame
lands in the `ReplaySubject` buffer that `_resetState()` just replaced; it is
flushed only when the same subject connects again, and nothing makes it connect
again. Every later request is silently dropped, including the 10 s heartbeat,
so the heartbeat never exposes the dead socket either: no code checks whether a
`HeartBeatResponse` comes back.
- `ComputingUnitStatusService.monitorConnectionStatus()` re-fetches the unit
list only on a connection-status change, so that does not happen either.
Every control that gates on `isConnected` now treats a dead socket as live:
- the "Connecting" branch of `getRunButtonBehavior()` (`menu.component.ts`,
the `!this.workflowWebsocketService.isConnected` check) is skipped, so the run
button stays enabled;
- the kill button's `[disabled]` binding (`menu.component.html`) keeps it
enabled mid-run, and the kill request goes into the dead buffer;
- the connection badge (`menu.component.html`, `nz-badge` coloured by
`isConnected && Running`) stays green with the tooltip "Connected to Computing
Unit";
- the Form View's Stop button from #7944 (`workflow-form.component.ts`,
`isRunning && isConnected`) is enabled only because `isConnected` is wrong.
A server closes the socket cleanly whenever it shuts down gracefully (a
restart or redeploy of the service behind `/wsapi`, a Kubernetes pod being
deleted or evicted); Jetty sends a 1001 close frame for every open session on
shutdown. An unclean close (code 1006, e.g. the process is killed) already
errors the subject and redials after 3 s, so only the graceful case is broken.
### How to reproduce?
Reproduced in Chrome on a locally running stack. The clean close was
produced from the browser by closing the raw workflow socket with a normal
close code, which the browser reports with `wasClean: true`, the same path a
server-side close frame takes.
1. Open a workflow in the workspace with a Running computing unit selected.
The `/wsapi/workflow-websocket` socket is open and the connection badge is
green.
2. Close that socket cleanly.
3. The socket is closed, but no new socket is opened (the reconnect delay is
3 s), the badge stays green, and the "websocket connection lost, reconnecting"
line never appears in the console. Nothing changes however long the page stays
open.
Server side (not yet run): with a workflow open and connected, stop the
service behind `/wsapi` gracefully (for example the IDE's Stop button on the
computing-unit master) and start it again. The workflow socket should close
with code 1001 and never reopen; the badge stays green; clicking Run sends
nothing.
Unit level: in `workflow-websocket.service.spec.ts`, open a socket with the
spec's `FakeWebSocket` and fire `onclose` with `wasClean: true`. `isConnected`
stays `true` and no new socket opens; with `wasClean: false` the existing retry
path runs and a second socket opens after `WS_RECONNECT_INTERVAL_MS`.
### Version/Branch
1.4.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
