On Mon, Nov 23, 2020 at 10:46 AM Gianluca Renzi <[email protected]> wrote:
> Hello everybody, > > this is my first message on this list. I'm trying to troubleshoot a > strange behavior of a Guacamole server that results in suddenly terminated > connections. > > This is my deployment: > > - guacamole 1.1.0 > - tomcat 9.0.35 > - jdk 1.8.0_251 > - user auth and connection configuration through LDAP (Active > Directory) > - hardware: physical server with 2 socket, 16 CPU and 64GB of RAM > > The Tomcat server sits behind a NGINX web server acting as a reverse > proxy, with SSL and HTTP2 enabled (perhaps http2 is not needed). > > Users connect to the web application through a clientless VPN using a Palo > Alto Networks firewall as the VPN portal. > > The application runs fine most of the time, but seems that during peek > hours is tropping connections to the remote desktops (RDP protocol). From > the catalina.out log I can see that users are connected to their own > connections after authentication but are immediately disconnected with the > client trying to reconnect. Log file entries are as follows: > The fact that this happens during "Peak Hours" suggest that something may be failing from a resource perspective. The hardware you have seems robust enough, though you haven't said what your actual connection load is (100 concurrent users, 1000, 10000, etc.)? Outside of the hardware itself, have you looked at things like PAN firewall load, Internet circuit, and network links? > > User "xxx" connected to connection "xxxDesktop" >> User "xxx" disconnected from connection "xxxDesktop". Duration: 73 >> milliseconds > > > I'm trying to track the connection flow, but I'm unable to bind the > catalina.out logs with the tomcat access logs and the guacd logs. > If I search for "xxxDesktop" in the tomcat access logs, I can see entries > like this: > > GET >> /guacamole-1.1.0/api/session/data/ldap/connections/xxxDesktop?token=<LONG_HEX_STRING> > > > and if I then search for that log token string I can find some 404: > > GET /guacamole-1.1.0/api/session/tunnels/ >> *8ac12e84-b219-45fb-8912-e2fdc702c870*/activeConnection/connection/sharingProfiles?token= >> <LONG_HEX_STRING> HTTP/1.1" 404 210 > > This isn't necessarily an error - this states that Guacamole Client is looking for Share Profiles on an active connection related to that specific tunnel, and that none exist. In this case, the 404 is just how the API tells the web application that there are no Sharing Profiles for that particular tunnel. > > But I'm not able to bind this kind of log entry with any entry in the > guacd log file, thus I can't go back to the root cause of the > disconnection. Here I'm assuming that there is some mapping between the > tunnel ID (the string in the url after "tunnels" and the guacd logs, but > maybe I'm wrong. > > In the guacd log I see entries like this: > > nov 23 11:37:35 guacamole2 guacd[33384]: Last user of connection >> "$10c70db2-8d84-4b1c-aca4-28ed4f9e3a98" disconnected >> nov 23 11:37:35 guacamole2 guacd[26032]: guacd[33384]: INFO: User >> "@ee949374-6f5a-4f89-bd17-4b8931d3fdb5" disconnected (0 users remain) >> nov 23 11:37:35 guacamole2 guacd[26032]: guacd[33384]: INFO: Last >> user of connection "$10c70db2-8d84-4b1c-aca4-28ed4f9e3a98" disconnected >> nov 23 11:37:35 guacamole2 guacd[26032]: Connection >> "$10c70db2-8d84-4b1c-aca4-28ed4f9e3a98" removed. >> nov 23 11:37:35 guacamole2 guacd[26032]: guacd[26032]: INFO: >> Connection "$10c70db2-8d84-4b1c-aca4-28ed4f9e3a98" removed. > > > but no way to map those user-id and connection-id to the actual connection > in the Tomcat logs. > > Any suggestion? > My first suggestion would be to bump up the verbosity on both guacd and Guacamole Client such that you're getting more detailed error logs. Instructions for doing this can be found at the following locations: http://guacamole.apache.org/doc/gug/configuring-guacamole.html#webapp-logging http://guacamole.apache.org/doc/gug/configuring-guacamole.html#guacd.conf That should give you some more information as to why connections are closing down, and might even provide the information you need to link the connection information in guacd to that in Tomcat. There is a JIRA issue out there for improvements that better help to correlate this information, but hasn't gotten any attention, yet: https://issues.apache.org/jira/browse/GUACAMOLE-752 -Nick > > Thank you in advance, > Gianluca > >
