Sean, thanks for the clarification. >From that reply, you've made it clear you do understand the issue, and that you've modified the guacamole-client in a way that is similar to this feature request -- making guacamole-client stop requesting your guacamole-hosts poll your back end targets once they appear to be dead (by some configurable parameter). You achieved this by writing an extension that handles this more intelligently and for your specific use case, whereas G-1126 and this email chain are saying that some portion of that logic should be inside guacamole by default as a matter of good practice so the go-to answer to resolve it isn't to write an extension, or write your own UI.
I believe you're trying to highlight the issue of performance issues because of the mention of resource constraints/load -- I think those of us facing this issue are not having an *active* issue with the Guacamole server load constraints, just seeing inefficiency that can be improved. The real constraint (to my team) is that we have to deal with log noise, and graphing and alarms based on log events/frequency are less useful since a single user stuck in an infinite loop generates a lot of noise... This issue is also closely related to how some of us are trying to manage the *target* *resource* constraints... and forcing idle users out can contribute to this issue appearing more frequently. The real fix is here better activity monitoring, which is why I started a separate discussion on that. -Lee On Wed, Jun 8, 2022 at 11:24 AM Sean Hulbert <[email protected]> wrote: > Understood, and we don’t see this. However we have modified the > responders, by making guacamole aware if a VM is up and running. > > We wrote an executable that runs on the guest VM system that notifies our > platform (StormCloud) it is up and running and awaiting a connection, once > the connection is established the program then sets a flag in the database > as in use. Once a user is done our system will do 1 of 2 things based on > initial parameters being set during the template build, hold session VM > state (persistent state), or reset state of VM to zero (Ondemand). We also > place a timer on failed attempts to 3 minutes, if an attempt moves in to > failed connection state we are notified, then our system will auto > remediate, by resetting connection tunnels, then reboot the VM, generally > this corrects the issue. > > > > Now if you’re in a high VM saturated environment (or Starved resources) > you will have connections issues due to lack of resources and/or CPU > scheduling. > > > > This works for all the connection methods. > > > > One main difference is we do not use Docker, we use a full VDI on Ubuntu > 20/22.04LTS for greater security. > > > > Specs on we use on Guacamole, > > vCPU 4x8 (32 cores) > > RAM 64G > > NIC 10G > > Drive 25GB > > > > This will handle about 1000 connections at once, we run anywhere from 5 to > 8 guacamole servers which our platform load balances the connection to the > Guacamole systems. > > > > This should help clear up on why we don’t see this issue. > > > > *Thank You* > > *Sean Hulbert* > > > > > > *www.securitycentric.net <http://www.securitycentric.net/>* > > *A Cybersecurity Enablement Company* > > *We don't just run you through the motions, Our labs teach you how to > think!* > > > > [image: logo-acad400] > > > > System Award Management > > *CAGE:** 8AUV4* > > > > *AFCEA San Francisco Chapter V.P.* > > > > If you have heard of a hacker by name, he/she has failed, fear the hacker > you haven’t heard of! > > > > *CONFIDENTIALITY NOTICE:* This communication with its contents may > contain confidential and/or legally privileged information. It is solely > for the use of the intended recipient(s). Unauthorized interception, > review, use or disclosure is prohibited and may violate applicable laws > including the Electronic Communications Privacy Act. If you are not the > intended recipient, please contact the sender and destroy all copies of the > communication. Content within this email communication is not legally > binding as a contract and no promises are guaranteed unless in a formal > contract outside this email communication. > > > > igitur qui desiderat pacem, praeparet bellum!!! > > Epitoma Rei Militaris > > > > *From:* Lee Doughty [mailto:[email protected]] > *Sent:* Wednesday, June 8, 2022 7:43 AM > *To:* [email protected] > *Subject:* Re: Guacamole re-connection attempts never stop... they should? > > > > Sean, I don't think you're seeing the problem area correctly on this > issue.. > > > > > > The issue is that guacamole-client -- the javascript for the front-end > that the server gives to users -- has an infinite loop on retrying > connections. You can modify guacamole yourself to get away from this > problem (that's what GUACAMOLE-1126 shows), but if you are using stock > guacamole-client, you have the *possibility* for this infinite loop > > > > In order to trigger this, the target machine must disconnect the user > forcefully... Such as the target experiencing a hardware failure, or the > target machine otherwise no longer working over the designated protocol > (firewall rule, or disabling the protocol's service), or the machine > "bouncing" due to a power failure (and possibly now being on a different IP > address) -- These are actions/events/issues on the target VM, which might > be performed by the user utilizing the connection, or by some other > out-of-band process/error. > > > > Once the machine is effectively offline/unreachable by guacamole, the > user's browser will continue to poll the guacamole server every 15 seconds > with an attempt to restart the session to the target. The connection > attempt usually takes some time to fail, which adds some delay on the loop > (typically ~60 seconds, I think). > > > > The server getting hassled by this process is simply the result of being > the middle-man between an open browser tab on the user side trying to > connect to a VM behind guacamole that was, but is no longer, available. > > > > My use case heavily involves education of students around firewalls, > server security, etc., so we see dozens of different ways the target > machine can fail. USUALLY the students realize the connection is not > recoverable and stop the loop themselves after a few minutes (and recovery > of the target VM is outside the scope of guacamole), but sometimes they > walk (or tab) away, and guacamole retries indefinitely (which some people > view as undesirable behaviour). > > > > The default for guacamole works very well if you connect users to > always-on machines who are utilizing unprivileged user accounts or are > otherwise skilled administrators... but starts to have the above issue for > ephemeral/on-demand targets, or when users can cause their > machines/connections to become unstable -- which admittedly are not > guacamole problems -- but I believe the fact guacamole assumes machines > will always be available or recoverable and never stops trying IS a > guacamole issue. > > > > -Lee > > > > On Tue, Jun 7, 2022 at 7:18 PM Sean Hulbert < > [email protected]> wrote: > > This has to be a Docker image issue, > > > > We run a few guacamole servers and never experienced this issue, however > we did built them from scratch and modified the kernels. > > > > > > *Thank You* > > *Sean Hulbert* > > > > *Founder / CEO* > > *Work Ph: 925.292.4309* > > > > *www.securitycentric.net <http://www.securitycentric.net/>* > > *A Cybersecurity Enablement Company* > > *We don't just run you through the motions, Our labs teach you how to > think!* > > > > [image: logo-acad400] > > > > System Award Management > > *CAGE:** 8AUV4* > > > > *AFCEA San Francisco Chapter V.P.* > > > > If you have heard of a hacker by name, he/she has failed, fear the hacker > you haven’t heard of! > > > > *CONFIDENTIALITY NOTICE:* This communication with its contents may > contain confidential and/or legally privileged information. It is solely > for the use of the intended recipient(s). Unauthorized interception, > review, use or disclosure is prohibited and may violate applicable laws > including the Electronic Communications Privacy Act. If you are not the > intended recipient, please contact the sender and destroy all copies of the > communication. Content within this email communication is not legally > binding as a contract and no promises are guaranteed unless in a formal > contract outside this email communication. > > > > igitur qui desiderat pacem, praeparet bellum!!! > > Epitoma Rei Militaris > > > > *From:* Lee Doughty [mailto:[email protected]] > *Sent:* Tuesday, June 7, 2022 2:46 PM > *To:* [email protected] > *Subject:* Re: Guacamole re-connection attempts never stop... they should? > > > > Nick, > > > > I agree https://issues.apache.org/jira/browse/GUACAMOLE-1126 looks in the > ballpark. > > > > I have no personal investment in the implementation, I would simply like > the endless server pings to stop so we can more easily monitor application > logs, have less "errors" in our logs, and reduce the baseline load on the > servers. > > > > We do somewhat inflict this problem on ourselves (I think), because we > shut down the VMs people are using, but we do so because we reasonably > believe the user is idle and wasting resources... > > > > I suppose if there was a "guacamole session time limit" that could be set > when the user initially authenticates, and they get kicked out and forced > to re-authenticate at that point (or at least click a button to extend > their session).... that would resolve my issue too... as I understand the > auth process now, once they have a guacamole-guacd session successfully > established, that seems to be almost entirely independent of their user > account or the connection's properties... I have not tried it, but I > suspect you could delete either of them, and guacamole would still let the > user continue their current session. > > > > -Lee > > > > On Wed, Jun 1, 2022 at 5:18 AM Jürgen Kuri <[email protected]> wrote: > > Suggestion: > > 1) keep endless reconnect as default > > 2) have two connection specific and / or global parameter: > a. number of retries > b. retry interval > > 3) if we have these parameters as global and on connection level, global > is overwritten for specific connection if defined > > El 31.05.22 a las 22:44, Lee Doughty escribió: > > Hello Guacamole Community, > > > > I tried asking this a few weeks ago, but it looks like there was not a > lot of traction on this idea.. but I wanted to try one more time before I > gave up on it. > > > > I think it would be a great feature to stop auto-reconnect attempts that > are simply not connecting after several dozen attempts. I've seen in our > logs that some users hit the "Reconnect" button or otherwise get into a > reconnect loop, then leave the tab open for hours *or days*. This results > in our guacamole server getting a ping every minute or so from a user > trying to connect to a VM that is not available, and they just leave it > retrying over and over again. > > > > It would be nice to at least require user interaction to resume the > connection attempts... So users have to return to the tab every N attempts > to restart the countdown, instead of the current never-ending loop... I'm > not suggesting any value for N... because any reasonable value would be > nice over infinite. My record was somewhere in the ballpark of 7,000 > attempts (5 days) before the user was kind enough to close the tab and stop > poking our Guacamole server. > > > > Is this something that can make it into an upcoming Guacamole release? > > > > -Lee > > > -- > Jürgen > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
