[
https://issues.apache.org/jira/browse/GUACAMOLE-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101840#comment-18101840
]
Alexandr commented on GUACAMOLE-2118:
-------------------------------------
Still reproducible on 1.6.1: 469 connection processes accumulated over ~4 hours
until memory was exhausted (graphs attached). 426 of the 469 had one
display-wrk blocked in sock_alloc_send_pskb.
Root cause is GUACAMOLE-1625, not guac_display. Connection processes inherit
the parent's end of every other connection's user socketpair, so those sockets
keep a non-zero refcount after the parent closes them. Their peers never see
EOF, so writes block instead of failing with EPIPE. That blocking write is held
under a read lock on the user list, so guac_client_remove_user() never gets its
write lock, connected_users never reaches zero, and guacd_proc_stop() is never
called.
Descriptor counts rose strictly with fork recency (98 in the oldest process,
756 in the youngest), and one socket inode was held by 40 of 40 sampled
processes. After the fix they are flat at 42 and nothing gets stuck.
This is not a 1.6 regression. Both the inheritance and the
write-under-read-lock pattern exist in 1.5.5, and guacd/proc.c is unchanged
between the two. What changed is that guac_display writes frames from a pool of
worker threads rather than a single flush from the protocol thread, which makes
the latent deadlock near-certain under load.
PRs: [1625|https://github.com/apache/guacamole-server/pull/698] closes
inherited sockets after fork.
[2118|https://github.com/apache/guacamole-server/pull/699] covers a separate
path, where a user vanishing without "disconnect" leaves an idle process
awaiting input forever.
!image-2026-08-04-21-49-44-148.png|width=1132,height=210!
> Unable to upgrade 1.5.5 to 1.6.0 due to sporadic hanging issue
> --------------------------------------------------------------
>
> Key: GUACAMOLE-2118
> URL: https://issues.apache.org/jira/browse/GUACAMOLE-2118
> Project: Guacamole
> Issue Type: Bug
> Affects Versions: 1.6.0
> Reporter: Jason Keltz
> Assignee: Mike Jumper
> Priority: Major
> Fix For: 1.6.1 [staging]
>
> Attachments: 2026-03-11_14-56.png, backtrace threads, backtrace.log,
> guacamole-logs.txt, guacd-178388-gdb.txt, guacd-178388-lsof.txt,
> guacd-240326-gdb.txt, guacd-240326-lsof.txt, guacd-stale-report.sh,
> image-2025-09-08-15-11-59-435.png, image-2026-08-04-21-49-44-148.png,
> image-blocks.jpg
>
>
> I've been running Guacamole since around 2020, upgrading reasonably quickly
> each and every time there's been an update. I update my Tomcat to the latest
> 9.X release from time to time (currently 9.0.102) , and my JDK to the latest
> 8.X release from time to time (currently jdk8u452-b09).
> Recently, after attempting an upgrade from Guacamole 1.5.5 to 1.6.0, I ran
> into a problem. Initially, everything seemed to work just fine. I can
> connect to any of the systems I have available. However, at some point
> later, I notice in the tomcat logs a lot of "connects" and "disconnects" to
> hosts. Users start complaining that "Guacamole isn't working". What I
> noticed at this point was that when they would try to return to a connection,
> it would connect, and their existing connecting would start to redraw, but
> it would hang in the middle. If I restart guacd at this point, it starts to
> work again, but the problem comes back. Some users would see it. Other users
> were fine.
> I feel like there's a bug hiding, and it may require a lot of user activity
> to get to it. I ended up creating a devel system for testing, and I'm
> running guac 1.6.0 there, and I've enabled full debugging, but I can't seem
> to make it happen here yet. Is there any easy way I can force a bunch of
> connections? The devel system is running labtest Rocky 8.10 (RHEL8.10) with
> latest kernel and patches and this matches the production system. They are
> both installed with the same kickstart configuration.
> I'm opening this "bug" even though I don't have concrete information yet. If
> I really have to do it, I may have to re-install on the production system to
> get the debugging information that I need, but I'd rather not do it if not
> necessary since it causes user inconvenience, and Guacamole is an important
> part of our educational environment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
