[
https://issues.apache.org/jira/browse/GUACAMOLE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nachum Barcohen updated GUACAMOLE-2293:
---------------------------------------
Description:
h3. Symptoms
* The Active Sessions admin view shows disconnected sessions that never clear.
* "Kill session" (REST PATCH .../activeConnections with op:remove) returns
HTTP 200, but the record reappears on refresh - the in-memory
ActiveConnectionRecord is never removed.
* guacd has no matching child process, the ghost is purely in guacamole-app
memory.
* Ghosts accumulate indefinitely (observed 83 leaked records, oldest ~35 days)
and count against concurrency limits. With a balancing group at
max_connections_per_user=1, a single own-ghost locks the user out of the whole
group. Only restarting guacamole-app flushes them.
h3. Environment
Guacamole 1.6.0 (official Docker image), guacamole-auth-jdbc (MySQL) + LDAP +
TOTP + ban, behind a reverse proxy / tunnel that does not always forward a
WebSocket close frame to the origin (observed with a Cloudflare Zero Trust
tunnel). The guacamole-app container is run with podman
--security-opt=no-new-privileges. Predominantly RDP connections.
h3. Root cause
The in-memory ActiveConnectionRecord is removed, and its concurrency seat
released, by the ConnectionCleanupTask (socketClosedTask) that runs inside
ManagedInetGuacamoleSocket.close() / ManagedSSLGuacamoleSocket.close(). That
task is invoked *after* super.close() and is *not* in a finally block.
At disconnect the underlying guacd socket is closed nearly simultaneously by
the read thread and by the @OnClose handler. When close() races, JDK 21's
NioSocketImpl fabricates java.io.IOException: No such file or directory on the
second close of the same fd (the close() syscall itself succeeds - verified via
strace, close(fd)=0, the ENOENT is a JVM NIO double-close artifact, not a
kernel or permission error). super.close() then throws, the cleanup task is
skipped, and the ActiveConnectionRecord leaks.
Separately, ActiveConnectionService.deleteObject() only calls tunnel.close()
when tunnel.isOpen() is true. AbstractGuacamoleTunnel.isOpen() delegates to the
underlying socket, so once the peer has vanished isOpen() returns false and
"Kill session" becomes a no-op - admin cannot reap an already-leaked ghost.
h3. Conclusion
Whether super.close() throws is a timing race. Running guacamole-app under
podman no-new-privileges probably shifts JVM thread scheduling enough to tip
the race toward the throwing order (cap-drop=ALL alone does not, only the
NoNewPrivs bit matters). RDP's heavy frame bursts keep both threads busy at
disconnect and widen the window, while light VNC streams stay clean. As it
seems, it's a scheduling issue, not a blocked syscall - strace during teardown
shows zero EPERM/EACCES and zero failed socket calls, and Seccomp=2 is
identical between clean and ghosting configs.
h3. Steps to reproduce (deterministic harness)
# Run stock guacamole/guacamole:1.6.0 + guacd + MySQL, with guacamole-app
started under --security-opt=no-new-privileges.
# Configure at least 2 RDP connections to reachable Windows hosts.
# Open 3 concurrent browser tabs to those RDP connections; let them connect
(~15–30s).
# Close all tabs near-simultaneously (close-storm).
# Wait ~60s, then query the in-memory tracker as an admin:
GET /api/session/data/mysql/activeConnections — one or more records persist
with no client connected.
A single sequential connect/disconnect does not leak; the leak needs concurrent
live RDP sessions plus a near-simultaneous close-storm.
h3. Fix
Make cleanup order-independent so the record is always released:
# ManagedInetGuacamoleSocket.close() and ManagedSSLGuacamoleSocket.close() -
run socketClosedTask in a finally, so cleanup runs even when super.close()
throws on an already-torn-down socket. (ConnectionCleanupTask is already
guarded to run once via an AtomicBoolean, so this is safe.)
# ActiveConnectionService.deleteObject() - drop the tunnel.isOpen() guard
before tunnel.close(). always call close() so an admin can reap an existing
ghost. It doesn't make sense either way. If tunnel is closed, there is no
reason to keep the record in memory.
After having a stable reproduction I verified the fix, and then deployed to my
production.
Note: the related uncaught-IllegalStateException path in the WebSocket read
thread (sendInstruction -> remote.sendText) is already handled on main by
GUACAMOLE-2181, which converts that IllegalStateException into an IOException
the read thread already catches. This issue is the remaining, independent leak
in the socket-close cleanup path and is not addressed by GUACAMOLE-2181.
was:
h3. Symptoms
* The Active Sessions admin view shows disconnected sessions that never clear.
* "Kill session" (REST PATCH .../activeConnections with op:remove) returns
HTTP 200, but the record reappears on refresh - the in-memory
ActiveConnectionRecord is never removed.
* guacd has no matching child process; the ghost is purely in guacamole-app
memory.
* Ghosts accumulate indefinitely (observed 83 leaked records, oldest ~35 days)
and count against concurrency limits. With a balancing group at
max_connections_per_user=1, a single own-ghost locks the user out of the whole
group. Only restarting guacamole-app flushes them.
* Log signature: java.lang.IllegalStateException: Message will not be sent
because the WebSocket session has been closed, at
GuacamoleWebSocketTunnelEndpoint.sendInstruction.
h3. Environment
Guacamole 1.6.0 (official Docker image), guacamole-auth-jdbc (MySQL) + LDAP +
TOTP + ban, behind a reverse proxy / tunnel that does not always forward a
WebSocket close frame to the origin (observed with a Cloudflare Zero Trust
tunnel). The guacamole-app container is run with podman
--security-opt=no-new-privileges. Predominantly RDP connections.
h3. Root cause
At disconnect there are two teardown paths, and cleanup can be skipped on
either:
# *Socket-close path (the actual leak).* @OnClose fires normally ->
tunnel.close() -> ManagedInetGuacamoleSocket.close() (or the SSL variant). That
method calls super.close() and only afterwards runs socketClosedTask - the
ConnectionCleanupTask that removes the record from the in-memory
ActiveConnectionMultimap and releases the concurrency seat. The cleanup call is
{*}not in a finally{*}. When the underlying guacd socket is being torn down
concurrently by the read thread, JDK 21's NioSocketImpl fabricates
java.io.IOException: No such file or directory on the second close() (the
close() syscall itself succeeds - verified via strace, close(fd)=0; the ENOENT
is a JVM NIO double-close artifact, not a kernel or permission error).
super.close() throws, the cleanup task is skipped, and the
ActiveConnectionRecord leaks.
# *Read-thread path (secondary symptom).* The transfer thread in
GuacamoleWebSocketTunnelEndpoint catches only IOException. If the client WS
closes mid-send, remote.sendText() throws an unchecked IllegalStateException (a
RuntimeException); the thread dies before closeConnection() runs, so @OnClose
never fires either.
h3. Conclusion
It is a timing race between the read side closing the guacd socket and the
@OnClose close. Running guacamole-app under podman no-new-privileges shifts JVM
thread scheduling enough to tip the race toward the throwing order (cap-drop=ALL
alone does not; only the NoNewPrivs bit matters). RDP's heavy frame bursts keep
the writer busy at disconnect and widen the window; light VNC streams stay
clean.
This is scheduling, not a blocked syscall - strace during teardown shows zero
EPERM/EACCES and zero failed socket calls, and Seccomp=2 is identical between
clean and ghosting configs.
h3. Steps to reproduce
# Run stock guacamole/guacamole:1.6.0 + guacd + MySQL, with guacamole-app
started under --security-opt=no-new-privileges.
# Configure at least 2 RDP connections to reachable Windows hosts.
# Open 3 concurrent browser tabs to those RDP connections; let them connect
(~15–30s).
# Close all tabs near-simultaneously (close-storm).
# Wait ~60s, then looks at the active connections list. Connections still
there. You can also query the in-memory tracker as an admin: GET
/api/session/data/mysql/activeConnections.
A single sequential connect/disconnect does not leak; the leak needs concurrent
live RDP sessions plus a near-simultaneous close-storm.
h3. Fix
Make cleanup order-independent so the record is always released:
# ManagedInetGuacamoleSocket.close() and ManagedSSLGuacamoleSocket.close() -
run socketClosedTask in a finally, so cleanup runs even when super.close()
throws on an already-torn-down socket. (ConnectionCleanupTask is already
guarded to run once via an AtomicBoolean, so this is safe.)
# ActiveConnectionService.deleteObject() - drop the tunnel.isOpen() guard
before tunnel.close(). For a connection whose peer already vanished isOpen()
returns false, so "Kill session" was a no-op; always call close() so an admin
can reap an existing host. The guard doesn't make sense, if the tunnel.isOpen
is false, there is no reason to keep the record in memory.
# GuacamoleWebSocketTunnelEndpoint (hardening) - catch RuntimeException/Error
in the transfer thread and close the WS session in a finally, guaranteeing
@OnClose fires regardless of how the thread terminates.
Fixes 1-2 are the necessary fix; fix 3 is defensive hardening for the parallel
read-thread symptom. I have validated on my production setup and the issue is
gone.
{*}Mailing-list context{*}:
https://lists.apache.org/thread/fwppogp2rl6n6hcn2wq33t3fz0obn0wq
> Active connection record leaks in memory when the guacd socket close throws
> during disconnect
> ---------------------------------------------------------------------------------------------
>
> Key: GUACAMOLE-2293
> URL: https://issues.apache.org/jira/browse/GUACAMOLE-2293
> Project: Guacamole
> Issue Type: Bug
> Components: guacamole-auth-jdbc, guacamole-common
> Affects Versions: 1.6.0
> Reporter: Nachum Barcohen
> Priority: Major
>
> h3. Symptoms
> * The Active Sessions admin view shows disconnected sessions that never
> clear.
> * "Kill session" (REST PATCH .../activeConnections with op:remove) returns
> HTTP 200, but the record reappears on refresh - the in-memory
> ActiveConnectionRecord is never removed.
> * guacd has no matching child process, the ghost is purely in guacamole-app
> memory.
> * Ghosts accumulate indefinitely (observed 83 leaked records, oldest ~35
> days) and count against concurrency limits. With a balancing group at
> max_connections_per_user=1, a single own-ghost locks the user out of the
> whole group. Only restarting guacamole-app flushes them.
> h3. Environment
> Guacamole 1.6.0 (official Docker image), guacamole-auth-jdbc (MySQL) + LDAP +
> TOTP + ban, behind a reverse proxy / tunnel that does not always forward a
> WebSocket close frame to the origin (observed with a Cloudflare Zero Trust
> tunnel). The guacamole-app container is run with podman
> --security-opt=no-new-privileges. Predominantly RDP connections.
> h3. Root cause
> The in-memory ActiveConnectionRecord is removed, and its concurrency seat
> released, by the ConnectionCleanupTask (socketClosedTask) that runs inside
> ManagedInetGuacamoleSocket.close() / ManagedSSLGuacamoleSocket.close(). That
> task is invoked *after* super.close() and is *not* in a finally block.
> At disconnect the underlying guacd socket is closed nearly simultaneously by
> the read thread and by the @OnClose handler. When close() races, JDK 21's
> NioSocketImpl fabricates java.io.IOException: No such file or directory on
> the second close of the same fd (the close() syscall itself succeeds -
> verified via strace, close(fd)=0, the ENOENT is a JVM NIO double-close
> artifact, not a kernel or permission error). super.close() then throws, the
> cleanup task is skipped, and the ActiveConnectionRecord leaks.
> Separately, ActiveConnectionService.deleteObject() only calls tunnel.close()
> when tunnel.isOpen() is true. AbstractGuacamoleTunnel.isOpen() delegates to
> the underlying socket, so once the peer has vanished isOpen() returns false
> and "Kill session" becomes a no-op - admin cannot reap an already-leaked
> ghost.
> h3. Conclusion
> Whether super.close() throws is a timing race. Running guacamole-app under
> podman no-new-privileges probably shifts JVM thread scheduling enough to tip
> the race toward the throwing order (cap-drop=ALL alone does not, only the
> NoNewPrivs bit matters). RDP's heavy frame bursts keep both threads busy at
> disconnect and widen the window, while light VNC streams stay clean. As it
> seems, it's a scheduling issue, not a blocked syscall - strace during
> teardown shows zero EPERM/EACCES and zero failed socket calls, and Seccomp=2
> is identical between clean and ghosting configs.
> h3. Steps to reproduce (deterministic harness)
> # Run stock guacamole/guacamole:1.6.0 + guacd + MySQL, with guacamole-app
> started under --security-opt=no-new-privileges.
> # Configure at least 2 RDP connections to reachable Windows hosts.
> # Open 3 concurrent browser tabs to those RDP connections; let them connect
> (~15–30s).
> # Close all tabs near-simultaneously (close-storm).
> # Wait ~60s, then query the in-memory tracker as an admin:
> GET /api/session/data/mysql/activeConnections — one or more records persist
> with no client connected.
> A single sequential connect/disconnect does not leak; the leak needs
> concurrent live RDP sessions plus a near-simultaneous close-storm.
> h3. Fix
> Make cleanup order-independent so the record is always released:
> # ManagedInetGuacamoleSocket.close() and ManagedSSLGuacamoleSocket.close() -
> run socketClosedTask in a finally, so cleanup runs even when super.close()
> throws on an already-torn-down socket. (ConnectionCleanupTask is already
> guarded to run once via an AtomicBoolean, so this is safe.)
> # ActiveConnectionService.deleteObject() - drop the tunnel.isOpen() guard
> before tunnel.close(). always call close() so an admin can reap an existing
> ghost. It doesn't make sense either way. If tunnel is closed, there is no
> reason to keep the record in memory.
> After having a stable reproduction I verified the fix, and then deployed to
> my production.
> Note: the related uncaught-IllegalStateException path in the WebSocket read
> thread (sendInstruction -> remote.sendText) is already handled on main by
> GUACAMOLE-2181, which converts that IllegalStateException into an IOException
> the read thread already catches. This issue is the remaining, independent
> leak in the socket-close cleanup path and is not addressed by GUACAMOLE-2181.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
