[
https://issues.apache.org/jira/browse/GUACAMOLE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092946#comment-18092946
]
Nachum Barcohen commented on GUACAMOLE-2293:
--------------------------------------------
I see that some of my changes were implemented in GUACAMOLE-2181, so I'm
pushing just the left-overs
> Active connection records leak in memory when a WebSocket client disconnects
> abruptly during socket teardown
> ------------------------------------------------------------------------------------------------------------
>
> Key: GUACAMOLE-2293
> URL: https://issues.apache.org/jira/browse/GUACAMOLE-2293
> Project: Guacamole
> Issue Type: Bug
> Components: guacamole-auth-jdbc, guacamole-common
> Affects Versions: 1.6.0
> Reporter: Nachum Barcohen
> Priority: Major
>
> h3. Symptoms
> * The Active Sessions admin view shows disconnected sessions that never
> clear.
> * "Kill session" (REST PATCH .../activeConnections with op:remove) returns
> HTTP 200, but the record reappears on refresh - the in-memory
> ActiveConnectionRecord is never removed.
> * guacd has no matching child process; the ghost is purely in guacamole-app
> memory.
> * Ghosts accumulate indefinitely (observed 83 leaked records, oldest ~35
> days) and count against concurrency limits. With a balancing group at
> max_connections_per_user=1, a single own-ghost locks the user out of the
> whole group. Only restarting guacamole-app flushes them.
> * Log signature: java.lang.IllegalStateException: Message will not be sent
> because the WebSocket session has been closed, at
> GuacamoleWebSocketTunnelEndpoint.sendInstruction.
> h3. Environment
> Guacamole 1.6.0 (official Docker image), guacamole-auth-jdbc (MySQL) + LDAP +
> TOTP + ban, behind a reverse proxy / tunnel that does not always forward a
> WebSocket close frame to the origin (observed with a Cloudflare Zero Trust
> tunnel). The guacamole-app container is run with podman
> --security-opt=no-new-privileges. Predominantly RDP connections.
> h3. Root cause
> At disconnect there are two teardown paths, and cleanup can be skipped on
> either:
> # *Socket-close path (the actual leak).* @OnClose fires normally ->
> tunnel.close() -> ManagedInetGuacamoleSocket.close() (or the SSL variant).
> That method calls super.close() and only afterwards runs socketClosedTask -
> the ConnectionCleanupTask that removes the record from the in-memory
> ActiveConnectionMultimap and releases the concurrency seat. The cleanup call
> is {*}not in a finally{*}. When the underlying guacd socket is being torn
> down concurrently by the read thread, JDK 21's NioSocketImpl fabricates
> java.io.IOException: No such file or directory on the second close() (the
> close() syscall itself succeeds - verified via strace, close(fd)=0; the
> ENOENT is a JVM NIO double-close artifact, not a kernel or permission error).
> super.close() throws, the cleanup task is skipped, and the
> ActiveConnectionRecord leaks.
> # *Read-thread path (secondary symptom).* The transfer thread in
> GuacamoleWebSocketTunnelEndpoint catches only IOException. If the client WS
> closes mid-send, remote.sendText() throws an unchecked IllegalStateException
> (a RuntimeException); the thread dies before closeConnection() runs, so
> @OnClose never fires either.
> h3. Conclusion
> It is a timing race between the read side closing the guacd socket and the
> @OnClose close. Running guacamole-app under podman no-new-privileges shifts
> JVM
> thread scheduling enough to tip the race toward the throwing order
> (cap-drop=ALL
> alone does not; only the NoNewPrivs bit matters). RDP's heavy frame bursts
> keep
> the writer busy at disconnect and widen the window; light VNC streams stay
> clean.
> This is scheduling, not a blocked syscall - strace during teardown shows zero
> EPERM/EACCES and zero failed socket calls, and Seccomp=2 is identical between
> clean and ghosting configs.
> h3. Steps to reproduce
> # Run stock guacamole/guacamole:1.6.0 + guacd + MySQL, with guacamole-app
> started under --security-opt=no-new-privileges.
> # Configure at least 2 RDP connections to reachable Windows hosts.
> # Open 3 concurrent browser tabs to those RDP connections; let them connect
> (~15–30s).
> # Close all tabs near-simultaneously (close-storm).
> # Wait ~60s, then looks at the active connections list. Connections still
> there. You can also query the in-memory tracker as an admin: GET
> /api/session/data/mysql/activeConnections.
> A single sequential connect/disconnect does not leak; the leak needs
> concurrent
> live RDP sessions plus a near-simultaneous close-storm.
> h3. Fix
> Make cleanup order-independent so the record is always released:
> # ManagedInetGuacamoleSocket.close() and ManagedSSLGuacamoleSocket.close() -
> run socketClosedTask in a finally, so cleanup runs even when super.close()
> throws on an already-torn-down socket. (ConnectionCleanupTask is already
> guarded to run once via an AtomicBoolean, so this is safe.)
> # ActiveConnectionService.deleteObject() - drop the tunnel.isOpen() guard
> before tunnel.close(). For a connection whose peer already vanished isOpen()
> returns false, so "Kill session" was a no-op; always call close() so an admin
> can reap an existing host. The guard doesn't make sense, if the
> tunnel.isOpen is false, there is no reason to keep the record in memory.
> # GuacamoleWebSocketTunnelEndpoint (hardening) - catch
> RuntimeException/Error in the transfer thread and close the WS session in a
> finally, guaranteeing @OnClose fires regardless of how the thread terminates.
> Fixes 1-2 are the necessary fix; fix 3 is defensive hardening for the parallel
> read-thread symptom. I have validated on my production setup and the issue is
> gone.
> {*}Mailing-list context{*}:
> https://lists.apache.org/thread/fwppogp2rl6n6hcn2wq33t3fz0obn0wq
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
