Nachum Barcohen created GUACAMOLE-2293:
------------------------------------------
Summary: Active connection records leak in memory when a WebSocket
client disconnects abruptly during socket teardown
Key: GUACAMOLE-2293
URL: https://issues.apache.org/jira/browse/GUACAMOLE-2293
Project: Guacamole
Issue Type: Bug
Components: guacamole-auth-jdbc, guacamole-common
Affects Versions: 1.6.0
Reporter: Nachum Barcohen
h3. Symptoms
* The Active Sessions admin view shows disconnected sessions that never clear.
* "Kill session" (REST PATCH .../activeConnections with op:remove) returns
HTTP 200, but the record reappears on refresh - the in-memory
ActiveConnectionRecord is never removed.
* guacd has no matching child process; the ghost is purely in guacamole-app
memory.
* Ghosts accumulate indefinitely (observed 83 leaked records, oldest ~35 days)
and count against concurrency limits. With a balancing group at
max_connections_per_user=1, a single own-ghost locks the user out of the whole
group. Only restarting guacamole-app flushes them.
* Log signature: java.lang.IllegalStateException: Message will not be sent
because the WebSocket session has been closed, at
GuacamoleWebSocketTunnelEndpoint.sendInstruction.
h3. Environment
Guacamole 1.6.0 (official Docker image), guacamole-auth-jdbc (MySQL) + LDAP +
TOTP + ban, behind a reverse proxy / tunnel that does not always forward a
WebSocket close frame to the origin (observed with a Cloudflare Zero Trust
tunnel). The guacamole-app container is run with podman
--security-opt=no-new-privileges. Predominantly RDP connections.
h3. Root cause
At disconnect there are two teardown paths, and cleanup can be skipped on
either:
# *Socket-close path (the actual leak).* @OnClose fires normally ->
tunnel.close() -> ManagedInetGuacamoleSocket.close() (or the SSL variant). That
method calls super.close() and only afterwards runs socketClosedTask - the
ConnectionCleanupTask that removes the record from the in-memory
ActiveConnectionMultimap and releases the concurrency seat. The cleanup call is
{*}not in a finally{*}. When the underlying guacd socket is being torn down
concurrently by the read thread, JDK 21's NioSocketImpl fabricates
java.io.IOException: No such file or directory on the second close() (the
close() syscall itself succeeds - verified via strace, close(fd)=0; the ENOENT
is a JVM NIO double-close artifact, not a kernel or permission error).
super.close() throws, the cleanup task is skipped, and the
ActiveConnectionRecord leaks.
# *Read-thread path (secondary symptom).* The transfer thread in
GuacamoleWebSocketTunnelEndpoint catches only IOException. If the client WS
closes mid-send, remote.sendText() throws an unchecked IllegalStateException (a
RuntimeException); the thread dies before closeConnection() runs, so @OnClose
never fires either.
h3. Conclusion
It is a timing race between the read side closing the guacd socket and the
@OnClose close. Running guacamole-app under podman no-new-privileges shifts JVM
thread scheduling enough to tip the race toward the throwing order (cap-drop=ALL
alone does not; only the NoNewPrivs bit matters). RDP's heavy frame bursts keep
the writer busy at disconnect and widen the window; light VNC streams stay
clean.
This is scheduling, not a blocked syscall - strace during teardown shows zero
EPERM/EACCES and zero failed socket calls, and Seccomp=2 is identical between
clean and ghosting configs.
h3. Steps to reproduce
# Run stock guacamole/guacamole:1.6.0 + guacd + MySQL, with guacamole-app
started under --security-opt=no-new-privileges.
# Configure at least 2 RDP connections to reachable Windows hosts.
# Open 3 concurrent browser tabs to those RDP connections; let them connect
(~15–30s).
# Close all tabs near-simultaneously (close-storm).
# Wait ~60s, then looks at the active connections list. Connections still
there. You can also query the in-memory tracker as an admin: GET
/api/session/data/mysql/activeConnections.
A single sequential connect/disconnect does not leak; the leak needs concurrent
live RDP sessions plus a near-simultaneous close-storm.
h3. Fix
Make cleanup order-independent so the record is always released:
# ManagedInetGuacamoleSocket.close() and ManagedSSLGuacamoleSocket.close() -
run socketClosedTask in a finally, so cleanup runs even when super.close()
throws on an already-torn-down socket. (ConnectionCleanupTask is already
guarded to run once via an AtomicBoolean, so this is safe.)
# ActiveConnectionService.deleteObject() - drop the tunnel.isOpen() guard
before tunnel.close(). For a connection whose peer already vanished isOpen()
returns false, so "Kill session" was a no-op; always call close() so an admin
can reap an existing host. The guard doesn't make sense, if the tunnel.isOpen
is false, there is no reason to keep the record in memory.
# GuacamoleWebSocketTunnelEndpoint (hardening) - catch RuntimeException/Error
in the transfer thread and close the WS session in a finally, guaranteeing
@OnClose fires regardless of how the thread terminates.
Fixes 1-2 are the necessary fix; fix 3 is defensive hardening for the parallel
read-thread symptom. I have validated on my production setup and the issue is
gone.
{*}Mailing-list context{*}:
https://lists.apache.org/thread/fwppogp2rl6n6hcn2wq33t3fz0obn0wq
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
