Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

n Thu, Apr 18, 2024 at 10:05 AM Maciej Konigsman
 wrote:

> Has this issue been resolved in v1.5.5?
>

Yes; however, a new issue was introduced where a lock is not correctly
checked/opened, and having microphone support enabled can result in RDP
connections failing. We haven't decided, yet, if we're going to do a 1.5.6
bugfix release to correct that and anything else that pops up, or just move
on to 1.6.0.

-Nick

Re: Major bug message log in guacd 1.5.4

[Maciej Konigsman]

Has this issue been resolved in v1.5.5?

On Thu, 25 Jan 2024 at 17:26, Weston Thayer  wrote:

> Hmm, I'd try openssl 1.0.2, but I don't see an alpine repository for it.
> Not quite sure how to go about building from source. I'd already tried
> downgrading FreeRDP (although there could be an interaction there).
>
> I suppose another approach would be to start a bisect of guacd commits
> between 1.5.3 and 1.5.4. Seems possible to automate...
>
> On Thu, Jan 25, 2024 at 1:17 AM Vieri  wrote:
>
>>  On Wednesday, January 24, 2024 at 11:38:35 PM GMT+1, Nick Couchman <
>> vn...@apache.org> wrote:
>>
>> > CentOS 7
>> > freerdp-libs 2.1.1-5.el7_9
>> > openssl-devel 1.0.2k-26.el7_9
>>
>>
>> I'm using:
>> openssl 1.1.1l
>> freerdp 2.4.1
>>
>> In my case, guacd 1.5.3 works flawlessly whereas 1.5.4 fails.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
>> For additional commands, e-mail: user-h...@guacamole.apache.org
>>
>>

RE: Aw: Re: Re: Major bug message log in guacd 1.5.4

[Yannick Martin]

Hello

We still have issue with connection to guacd even with the patch we suggest:
https://github.com/apache/guacamole-server/commit/9134780e347e4a75a0759a7970aaddfb0e8fa7de

Have got a stuck guacd daemon and my gdb gives me this:

Thread 4 is the parent and waits for thread 5:

|(gdb) thread ||4|
|[Switching to thread ||4| |(Thread ||0x7f2db2fd5700| |(LWP ||40612||))]|
|#||0| |0x7f2e5c016495| |in __pthread_timedjoin_ex () from 
/lib/x86_64-linux-gnu/libpthread.so.||0|

|(gdb) bt|
|#||0| |0x7f2e5c016495| |in __pthread_timedjoin_ex () from 
/lib/x86_64-linux-gnu/libpthread.so.||0|
|#||1| |0x55bcffd16fad| |in guacd_connection_io_thread 
(data=||0x7f2e2c00b6e0||) at connection.c:||150|
|#||2| |0x7f2e5c014fa3| |in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.||0|
|#||3| |0x7f2e5beb2eff| |in clone () from 
/lib/x86_64-linux-gnu/libc.so.||6|

|(gdb) frame ||1|
|#||1| |0x55bcffd16fad| |in guacd_connection_io_thread 
(data=||0x7f2e2c00b6e0||) at connection.c:||150|

|print *(guacd_connection_io_thread_params* )data|
|$||5| |= {parser = ||0x7f2e2c002930||, socket = ||0x7f2e2c00b000||, fd 
= ||87|



Thread 5 seems stuck on read() call.

|thread ||5|
|[Switching to thread ||5| |(Thread ||0x7f2db27d4700| |(LWP ||40613||))]|
|#||0| |0x7f2e5c01e544| |in read () from 
/lib/x86_64-linux-gnu/libpthread.so.||0|

|(gdb) bt|
|#||0| |0x7f2e5c01e544| |in read () from 
/lib/x86_64-linux-gnu/libpthread.so.||0|
|#||1| |0x7f2e5c03ecab| |in ?? () from 
/lib/x86_64-linux-gnu/libguac.so.||23|
|#||2| |0x55bcffd16ec8| |in guacd_connection_write_thread 
(data=||0x7f2e2c00b6e0||) at connection.c:||123|
|#||3| |0x7f2e5c014fa3| |in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.||0|
|#||4| |0x7f2e5beb2eff| |in clone () from 
/lib/x86_64-linux-gnu/libc.so.||6|

|print *(guacd_connection_io_thread_params* )data|
|$||6| |= {parser = ||0x7f2e2c002930||, socket = ||0x7f2e2c00b000||, fd 
= ||87||}|



Looking code 
https://github.com/apache/guacamole-server/blob/3782339031bdc47e3c67c5630e42f1f2fd9493a0/src/guacd/connection.c#L143, 
I wonder how the code behave if the read() fails immediately (aka we 
don't push anything in with guac_socket_write()).


Maybe something is missing to unlock the writting thread or even better 
using some select and merge read/write operations in the same thread.



Regards

On 2024/02/09 09:23:27 michael böhm wrote:
> Hi everyone,
>
>
>
> I proceeded as Antoine proposed and set "ARG ALPINE_BASE_IMAGE=3.18" in
> staging/1.5.5 Dockerfile.
>
>
>
> The build worked and I was able to start the guacd container from 
this image.
> I tried more than 100 consecutive reconnects to an RDP session 
without the

> issue appearing.
>
>
>
> So, it looks good to me. Can anyone confirm?
>
>
>
> Infos on my docker-host:
>
>
>
> NAME="Ubuntu"
> VERSION_ID="22.04"
> VERSION="22.04.3 LTS (Jammy Jellyfish)"
> VERSION_CODENAME=jammy
> ID=ubuntu
> ID_LIKE=debian
> HOME_URL="https://www.ubuntu.com/;
> SUPPORT_URL="https://help.ubuntu.com/;
> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/;
> 
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-

> policy"
>
>
>
> Kernel 5.15.0-92-generic
>
>
>
> Docker version 25.0.3, build 4debf41
>
>
>
> Thanks to everyone working on this.
>
>
>
> Best wishes
>
>
>
> Michael
>
>
>
> **Gesendet:** Freitag, 09. Februar 2024 um 09:00 Uhr
> **Von:** "Antoine Besnier" 
> **An:** "user@guacamole.apache.org" 
> **Betreff:** Re: Aw: Re: Major bug message log in guacd 1.5.4
>
>
>
> Hi,
>
>
>
> On Alpine, openssl1.1-compat-dev is available for 3.17, 3.18 and 
Edge, but not
> 3.19 (which is the version for the 'latest' tag). You could try by 
changing

> the version of Alpine.
>
>
>
> Cheers
>
> Antoine
>
>
>
>
>
> Le vendredi 9 fevrier 2024 a 07:35:42 UTC+1, michael bohm
>  a ecrit :
>
>
>
>
>
> Hi everyone
>
>
>
> I'd gladly test in our environment. However, the docker build does 
not work

> for me:
>
>
>
> /tmp/guacamole-server ‹staging/1.5.5› » git checkout staging/1.5.5
> 1 ↵
> Switched to branch 'staging/1.5.5'
> Your branch is up to date with 'origin/staging/1.5.5'.
> /tmp/guacamole-server ‹staging/1.5.5› » docker build -t guac_test .
> [+] Building 0.9s (6/13)
> docker:default
> => [internal] load build definition from Dockerfile
> 0.0s
> => => transferring dockerfile: 6.10kB
> 0.0s
> => [internal] load metadata for docker.io/library/alpine:latest
> 0.0s
> => [internal] load .dockerignore
> 0.0s
> => => transferring context: 681B
> 0.0s
> => CACHED 

Aw: Re: Re: Major bug message log in guacd 1.5.4

[michael böhm]

Hi everyone,

 

I proceeded as Antoine proposed and set "ARG ALPINE_BASE_IMAGE=3.18" in staging/1.5.5 Dockerfile.

 

The build worked and I was able to start the guacd container from this image. I tried more than 100 consecutive reconnects to an RDP session without the issue appearing.

 

So, it looks good to me. Can anyone confirm?

 

Infos on my docker-host:

 

NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

 

Kernel 5.15.0-92-generic

 

Docker version 25.0.3, build 4debf41

 

Thanks to everyone working on this.

 

Best wishes

 

Michael

 


Gesendet: Freitag, 09. Februar 2024 um 09:00 Uhr
Von: "Antoine Besnier" 
An: "user@guacamole.apache.org" 
Betreff: Re: Aw: Re: Major bug message log in guacd 1.5.4



 

Hi, 

 

On Alpine, openssl1.1-compat-dev is available for 3.17, 3.18 and Edge, but not 3.19 (which is the version for the 'latest' tag). You could try by changing the version of Alpine.

 

Cheers

Antoine

 

 




Le vendredi 9 février 2024 à 07:35:42 UTC+1, michael böhm  a écrit :

 

 





Hi everyone

 

I'd gladly test in our environment. However, the docker build does not work for me:

 

/tmp/guacamole-server ‹staging/1.5.5› » git checkout staging/1.5.5                                                                                                                                                                     1 ↵
Switched to branch 'staging/1.5.5'
Your branch is up to date with 'origin/staging/1.5.5'.
/tmp/guacamole-server ‹staging/1.5.5› » docker build -t guac_test .
[+] Building 0.9s (6/13)                                                                                                                                                                                                                                          docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                        0.0s
 => => transferring dockerfile: 6.10kB                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                           0.0s
 => => transferring context: 681B                                                                                                                                                                                                                                           0.0s
 => CACHED [builder 1/5] FROM docker.io/library/alpine:latest                                                                                                                                                                                                               0.0s
 => [internal] load build context                                                                                                                                                                                                                                           0.0s
 => => transferring context: 28.84kB                                                                                                                                                                                                                                        0.0s
 => ERROR [builder 2/5] RUN apk add --no-cache                        autoconf                              automake                              build-base                            cairo-dev                             cmake                                 git     0.8s
--                                                                                                                                                                                                                                                                           
 > [builder 2/5] RUN apk add --no-cache    

Re: Aw: Re: Major bug message log in guacd 1.5.4

[Antoine Besnier]

 Hi, 
On Alpine, openssl1.1-compat-dev is available for 3.17, 3.18 and Edge, but not 
3.19 (which is the version for the 'latest' tag). You could try by changing the 
version of Alpine.
CheersAntoine 
Le vendredi 9 février 2024 à 07:35:42 UTC+1, michael böhm 
 a écrit :  
 
 Hi everyone I'd gladly test in our environment. However, the docker build does 
not work for me: /tmp/guacamole-server ‹staging/1.5.5› » git checkout 
staging/1.5.5                                                                   
                                                                                
                  1 ↵
Switched to branch 'staging/1.5.5'
Your branch is up to date with 'origin/staging/1.5.5'.
/tmp/guacamole-server ‹staging/1.5.5› » docker build -t guac_test .
[+] Building 0.9s (6/13)                                                        
                                                                                
                                                                                
                  docker:default
 => [internal] load build definition from Dockerfile                            
                                                                                
                                                                                
                            0.0s
 => => transferring dockerfile: 6.10kB                                          
                                                                                
                                                                                
                            0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                
                                                                                
                                                                                
                            0.0s
 => [internal] load .dockerignore                                               
                                                                                
                                                                                
                            0.0s
 => => transferring context: 681B                                               
                                                                                
                                                                                
                            0.0s
 => CACHED [builder 1/5] FROM docker.io/library/alpine:latest                   
                                                                                
                                                                                
                            0.0s
 => [internal] load build context                                               
                                                                                
                                                                                
                            0.0s
 => => transferring context: 28.84kB                                            
                                                                                
                                                                                
                            0.0s
 => ERROR [builder 2/5] RUN apk add --no-cache                        autoconf  
                            automake                              build-base    
                        cairo-dev                             cmake             
                    git     0.8s
--                                                                          
                                                                                
                                                                                
                                 
 > [builder 2/5] RUN apk add --no-cache                        autoconf         
                     automake                              build-base           
                 cairo-dev                             cmake                    
             git                                   grep                         
         libjpeg-turbo-dev                     libpng-dev                       
     libtool                               libwebp-dev                          
 make                                  openssl1.1-compat-dev                 
pango-dev                             pulseaudio-dev                        
util-linux-dev:                                                                 
                                                                                
                          
0.285 fetch 
https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz         
                                                                                
                                                                                
                     
0.475 fetch 

Aw: Re: Major bug message log in guacd 1.5.4

[michael böhm]

che                \
  30 | >>>         autoconf                      \
  31 | >>>         automake                      \
  32 | >>>         build-base                    \
  33 | >>>         cairo-dev                     \
  34 | >>>         cmake                         \
  35 | >>>         git                           \
  36 | >>>         grep                          \
  37 | >>>         libjpeg-turbo-dev             \
  38 | >>>         libpng-dev                    \
  39 | >>>         libtool                       \
  40 | >>>         libwebp-dev                   \
  41 | >>>         make                          \
  42 | >>>         openssl1.1-compat-dev         \
  43 | >>>         pango-dev                     \
  44 | >>>         pulseaudio-dev                \
  45 | >>>         util-linux-dev
  46 |     

ERROR: failed to solve: process "/bin/sh -c apk add --no-cache                        autoconf                              automake                              build-base                            cairo-dev                             cmake                                 git                                   grep                                  libjpeg-turbo-dev                     libpng-dev                            libtool                               libwebp-dev                           make                                  openssl1.1-compat-dev                 pango-dev                             pulseaudio-dev                        util-linux-dev" did not complete successfully: exit code: 1

 

It seems that "openssl1.1-compat-dev" is not present in base image alpine:latest's repositories. Am I doing something wrong? On the master branch I can build the image just fine.

 

Thanks and best wishes

 

Michael


 

Gesendet: Freitag, 09. Februar 2024 um 02:59 Uhr
Von: "Michael Jumper" 
An: user@guacamole.apache.org
Betreff: Re: Major bug message log in guacd 1.5.4

On 2/7/24 05:24, Nick Couchman wrote:
> On Wed, Feb 7, 2024 at 6:40 AM Yannick Martin
> > wrote:
>
> Hello
>
> About pthread_keys leak, I wonder if
> https://github.com/apache/guacamole-server/blob/master/src/libguac/client.c#L299 <https://github.com/apache/guacamole-server/blob/master/src/libguac/client.c#L299>
> and L300 is not a duplicate call of those done in:
>  guac_rwlock_init(&(client->__users_lock));
>  guac_rwlock_init(&(client->__pending_users_lock));
>
>  which call pthread_key_create too =>
> https://github.com/apache/guacamole-server/blob/master/src/libguac/rwlock.c#L52 <https://github.com/apache/guacamole-server/blob/master/src/libguac/rwlock.c#L52>
>
>
> Two issues with this:
> * I'm not sure that duplicating a call to pthread_key_create()
> would/should result in the behavior we're seeing - where TLS-based
> connections fail after a certain, relatively well-defined number (58-60).
> * This also would not explain why this only occurs in certain
> situations, on certain platforms - that is, the same exact libguac code
> running on EL7 (RHEL, CentOS, etc.) does not result in the resource
> leak, whereas it does on some other set of platforms (Debian, Alpine,
> EulerOS). Unless the pthread library has been changed substantially
> between those versions to not clean up after itself?
>

This *might* now be fixed on master and staging/1.5.5, if folks want to
take a look. The issue does appear to be have been the duplicate
pthread_key_create() calls noted above (pending confirmation with testing).

If this is the cause, it's still unclear why the behavior varies between
platforms. It might be a matter of varying implementations, different
resource limits, or something else platform-specific.

- Mike

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org
 




-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Michael Jumper]

On 2/7/24 05:24, Nick Couchman wrote:
On Wed, Feb 7, 2024 at 6:40 AM Yannick Martin 
mailto:yannick.mar...@ovhcloud.com>> wrote:


Hello

About pthread_keys leak, I wonder if
https://github.com/apache/guacamole-server/blob/master/src/libguac/client.c#L299 

and L300 is not a duplicate call of those done in:
  guac_rwlock_init(&(client->__users_lock));
  guac_rwlock_init(&(client->__pending_users_lock));

  which call pthread_key_create too =>
https://github.com/apache/guacamole-server/blob/master/src/libguac/rwlock.c#L52 



Two issues with this:
* I'm not sure that duplicating a call to pthread_key_create() 
would/should result in the behavior we're seeing - where TLS-based 
connections fail after a certain, relatively well-defined number (58-60).
* This also would not explain why this only occurs in certain 
situations, on certain platforms - that is, the same exact libguac code 
running on EL7 (RHEL, CentOS, etc.) does not result in the resource 
leak, whereas it does on some other set of platforms (Debian, Alpine, 
EulerOS). Unless the pthread library has been changed substantially 
between those versions to not clean up after itself?




This *might* now be fixed on master and staging/1.5.5, if folks want to 
take a look. The issue does appear to be have been the duplicate 
pthread_key_create() calls noted above (pending confirmation with testing).


If this is the cause, it's still unclear why the behavior varies between 
platforms. It might be a matter of varying implementations, different 
resource limits, or something else platform-specific.


- Mike

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

On Wed, Feb 7, 2024 at 6:40 AM Yannick Martin 
wrote:

> Hello
>
> About pthread_keys leak, I wonder if
>
> https://github.com/apache/guacamole-server/blob/master/src/libguac/client.c#L299
> and L300 is not a duplicate call of those done in:
>  guac_rwlock_init(&(client->__users_lock));
>  guac_rwlock_init(&(client->__pending_users_lock));
>
>  which call pthread_key_create too =>
>
> https://github.com/apache/guacamole-server/blob/master/src/libguac/rwlock.c#L52
>
>
Two issues with this:
* I'm not sure that duplicating a call to pthread_key_create() would/should
result in the behavior we're seeing - where TLS-based connections fail
after a certain, relatively well-defined number (58-60).
* This also would not explain why this only occurs in certain situations,
on certain platforms - that is, the same exact libguac code running on EL7
(RHEL, CentOS, etc.) does not result in the resource leak, whereas it does
on some other set of platforms (Debian, Alpine, EulerOS). Unless the
pthread library has been changed substantially between those versions to
not clean up after itself?

-Nick

Re: Major bug message log in guacd 1.5.4

[Weston Thayer]

Hmm, I'd try openssl 1.0.2, but I don't see an alpine repository for it.
Not quite sure how to go about building from source. I'd already tried
downgrading FreeRDP (although there could be an interaction there).

I suppose another approach would be to start a bisect of guacd commits
between 1.5.3 and 1.5.4. Seems possible to automate...

On Thu, Jan 25, 2024 at 1:17 AM Vieri  wrote:

>  On Wednesday, January 24, 2024 at 11:38:35 PM GMT+1, Nick Couchman <
> vn...@apache.org> wrote:
>
> > CentOS 7
> > freerdp-libs 2.1.1-5.el7_9
> > openssl-devel 1.0.2k-26.el7_9
>
>
> I'm using:
> openssl 1.1.1l
> freerdp 2.4.1
>
> In my case, guacd 1.5.3 works flawlessly whereas 1.5.4 fails.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> For additional commands, e-mail: user-h...@guacamole.apache.org
>
>

Re: Major bug message log in guacd 1.5.4

[Vieri]

 On Wednesday, January 24, 2024 at 11:38:35 PM GMT+1, Nick Couchman 
 wrote: 

> CentOS 7
> freerdp-libs 2.1.1-5.el7_9
> openssl-devel 1.0.2k-26.el7_9


I'm using:
openssl 1.1.1l
freerdp 2.4.1

In my case, guacd 1.5.3 works flawlessly whereas 1.5.4 fails.

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

CentOS 7
freerdp-libs 2.1.1-5.el7_9
openssl-devel 1.0.2k-26.el7_9

-Nick

On Wed, Jan 24, 2024 at 3:35 PM Weston Thayer 
wrote:

> I am repro'ing with the Docker image (slightly customized, otherwise I
> would've shared repro steps, working on that), so that helps.
>
> I did some research. alpine:3.17.6 through alpine:3.18.5 appear to all
> have the same version of openssl1.1-compat-dev
> <https://github.com/apache/guacamole-server/blob/a575af63ef5b4c8b1804999f211ebeb270992b5a/Dockerfile#L45>
> (1.1.1u-r1). I'm checking with `apk add --no-cache --simulate
> openssl1.1-compat-dev`, let me know if that's not to be trusted for some
> reason. No surprise that it repros when the image is built with something
> older, like alpine:3.18.2.
>
> Nick, even though you can't reproduce it, could you share your openssl
> version?
>
> On Wed, Jan 24, 2024 at 11:00 AM Barnhart, Steven 
> wrote:
>
>> Would be worth using the docker versions and seeing if is present in them
>> as that is then easy to point to dependencies not being updated.
>>
>> –Steve
>> --
>> *From:* Weston Thayer 
>> *Sent:* Wednesday, January 24, 2024 1:58:14 PM
>> *To:* user@guacamole.apache.org 
>> *Subject:* Re: Major bug message log in guacd 1.5.4
>>
>> Running with Nick's theory that it could be something to do with OpenSSL,
>> could folks who have reproduced it by upgrading ONLY guacd from 1. 5. 3 to
>> 1. 5. 4 share their OpenSSL versions? Since I can easily reproduce this, I
>> can employ some
>> Running with Nick's theory that it could be something to do with OpenSSL,
>> could folks who have reproduced it by upgrading ONLY guacd from 1.5.3 to
>> 1.5.4 share their OpenSSL versions? Since I can easily reproduce this, I
>> can employ some trial & error testing to try and identify the key variable.
>>
>> On Wed, Jan 24, 2024 at 9:15 AM Vieri 
>> wrote:
>>
>>
>>
>> On Wednesday, January 24, 2024 at 04:01:52 PM GMT+1, Nick Couchman <
>> vn...@apache.org> wrote:
>>
>> > When I say "underlying system" - I mean that I'm not running Docker
>> containers, I'm installing natively on CentOS7,
>> > and when I upgraded from 1.5.3 to 1.5.4, I did not update any of the
>> other dependencies (FreeRDP, SSH, kernel, openSSL, etc.).
>> > This indicates that the issue isn't guacd itself, but some issue
>> between guacd + FreeRDP and possibly some
>> > underlying library (OpenSSL) that is causing the leakage.
>>
>> Same here.
>> I am NOT using docker containers.
>> As I said, absolutely nothing has changed except for guacd 1.5.4 ->
>> 1.5.3. No other dependency has been updated or changed. OpenSSL is exactly
>> the same. All other libs are exactly the same. FreeRDP is left untouched.
>>
>> So it could be what you mentioned earlier: guacd 1.5.4 might be using
>> something in FreeRDP that 1.5.3 wasn't.
>> Assuming it's FreeRDP that's leaking.
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
>> For additional commands, e-mail: user-h...@guacamole.apache.org
>>
>>

Re: Major bug message log in guacd 1.5.4

[Weston Thayer]

I am repro'ing with the Docker image (slightly customized, otherwise I
would've shared repro steps, working on that), so that helps.

I did some research. alpine:3.17.6 through alpine:3.18.5 appear to all have
the same version of openssl1.1-compat-dev
<https://github.com/apache/guacamole-server/blob/a575af63ef5b4c8b1804999f211ebeb270992b5a/Dockerfile#L45>
(1.1.1u-r1). I'm checking with `apk add --no-cache --simulate
openssl1.1-compat-dev`, let me know if that's not to be trusted for some
reason. No surprise that it repros when the image is built with something
older, like alpine:3.18.2.

Nick, even though you can't reproduce it, could you share your openssl
version?

On Wed, Jan 24, 2024 at 11:00 AM Barnhart, Steven 
wrote:

> Would be worth using the docker versions and seeing if is present in them
> as that is then easy to point to dependencies not being updated.
>
> –Steve
> --
> *From:* Weston Thayer 
> *Sent:* Wednesday, January 24, 2024 1:58:14 PM
> *To:* user@guacamole.apache.org 
> *Subject:* Re: Major bug message log in guacd 1.5.4
>
> Running with Nick's theory that it could be something to do with OpenSSL,
> could folks who have reproduced it by upgrading ONLY guacd from 1. 5. 3 to
> 1. 5. 4 share their OpenSSL versions? Since I can easily reproduce this, I
> can employ some
> Running with Nick's theory that it could be something to do with OpenSSL,
> could folks who have reproduced it by upgrading ONLY guacd from 1.5.3 to
> 1.5.4 share their OpenSSL versions? Since I can easily reproduce this, I
> can employ some trial & error testing to try and identify the key variable.
>
> On Wed, Jan 24, 2024 at 9:15 AM Vieri  wrote:
>
>
>
> On Wednesday, January 24, 2024 at 04:01:52 PM GMT+1, Nick Couchman <
> vn...@apache.org> wrote:
>
> > When I say "underlying system" - I mean that I'm not running Docker
> containers, I'm installing natively on CentOS7,
> > and when I upgraded from 1.5.3 to 1.5.4, I did not update any of the
> other dependencies (FreeRDP, SSH, kernel, openSSL, etc.).
> > This indicates that the issue isn't guacd itself, but some issue between
> guacd + FreeRDP and possibly some
> > underlying library (OpenSSL) that is causing the leakage.
>
> Same here.
> I am NOT using docker containers.
> As I said, absolutely nothing has changed except for guacd 1.5.4 -> 1.5.3.
> No other dependency has been updated or changed. OpenSSL is exactly the
> same. All other libs are exactly the same. FreeRDP is left untouched.
>
> So it could be what you mentioned earlier: guacd 1.5.4 might be using
> something in FreeRDP that 1.5.3 wasn't.
> Assuming it's FreeRDP that's leaking.
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> For additional commands, e-mail: user-h...@guacamole.apache.org
>
>

Re: Major bug message log in guacd 1.5.4

[Barnhart, Steven]

Would be worth using the docker versions and seeing if is present in them as 
that is then easy to point to dependencies not being updated.

–Steve

From: Weston Thayer 
Sent: Wednesday, January 24, 2024 1:58:14 PM
To: user@guacamole.apache.org 
Subject: Re: Major bug message log in guacd 1.5.4

Running with Nick's theory that it could be something to do with OpenSSL, could 
folks who have reproduced it by upgrading ONLY guacd from 1. 5. 3 to 1. 5. 4 
share their OpenSSL versions? Since I can easily reproduce this, I can employ 
some

Running with Nick's theory that it could be something to do with OpenSSL, could 
folks who have reproduced it by upgrading ONLY guacd from 1.5.3 to 1.5.4 share 
their OpenSSL versions? Since I can easily reproduce this, I can employ some 
trial & error testing to try and identify the key variable.

On Wed, Jan 24, 2024 at 9:15 AM Vieri  wrote:


On Wednesday, January 24, 2024 at 04:01:52 PM GMT+1, Nick Couchman 
mailto:vn...@apache.org>> wrote:

> When I say "underlying system" - I mean that I'm not running Docker 
> containers, I'm installing natively on CentOS7,
> and when I upgraded from 1.5.3 to 1.5.4, I did not update any of the other 
> dependencies (FreeRDP, SSH, kernel, openSSL, etc.).
> This indicates that the issue isn't guacd itself, but some issue between 
> guacd + FreeRDP and possibly some
> underlying library (OpenSSL) that is causing the leakage.

Same here.
I am NOT using docker containers.
As I said, absolutely nothing has changed except for guacd 1.5.4 -> 1.5.3. No 
other dependency has been updated or changed. OpenSSL is exactly the same. All 
other libs are exactly the same. FreeRDP is left untouched.

So it could be what you mentioned earlier: guacd 1.5.4 might be using something 
in FreeRDP that 1.5.3 wasn't.
Assuming it's FreeRDP that's leaking.


-
To unsubscribe, e-mail: 
user-unsubscr...@guacamole.apache.org<mailto:user-unsubscr...@guacamole.apache.org>
For additional commands, e-mail: 
user-h...@guacamole.apache.org<mailto:user-h...@guacamole.apache.org>

Re: Major bug message log in guacd 1.5.4

[Weston Thayer]

Running with Nick's theory that it could be something to do with OpenSSL,
could folks who have reproduced it by upgrading ONLY guacd from 1.5.3 to
1.5.4 share their OpenSSL versions? Since I can easily reproduce this, I
can employ some trial & error testing to try and identify the key variable.

On Wed, Jan 24, 2024 at 9:15 AM Vieri  wrote:

>
>
> On Wednesday, January 24, 2024 at 04:01:52 PM GMT+1, Nick Couchman <
> vn...@apache.org> wrote:
>
> > When I say "underlying system" - I mean that I'm not running Docker
> containers, I'm installing natively on CentOS7,
> > and when I upgraded from 1.5.3 to 1.5.4, I did not update any of the
> other dependencies (FreeRDP, SSH, kernel, openSSL, etc.).
> > This indicates that the issue isn't guacd itself, but some issue between
> guacd + FreeRDP and possibly some
> > underlying library (OpenSSL) that is causing the leakage.
>
> Same here.
> I am NOT using docker containers.
> As I said, absolutely nothing has changed except for guacd 1.5.4 -> 1.5.3.
> No other dependency has been updated or changed. OpenSSL is exactly the
> same. All other libs are exactly the same. FreeRDP is left untouched.
>
> So it could be what you mentioned earlier: guacd 1.5.4 might be using
> something in FreeRDP that 1.5.3 wasn't.
> Assuming it's FreeRDP that's leaking.
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> For additional commands, e-mail: user-h...@guacamole.apache.org
>
>

Re: Major bug message log in guacd 1.5.4

[Vieri]

On Wednesday, January 24, 2024 at 04:01:52 PM GMT+1, Nick Couchman 
 wrote: 

> When I say "underlying system" - I mean that I'm not running Docker 
> containers, I'm installing natively on CentOS7, 
> and when I upgraded from 1.5.3 to 1.5.4, I did not update any of the other 
> dependencies (FreeRDP, SSH, kernel, openSSL, etc.). 
> This indicates that the issue isn't guacd itself, but some issue between 
> guacd + FreeRDP and possibly some 
> underlying library (OpenSSL) that is causing the leakage.

Same here.
I am NOT using docker containers.
As I said, absolutely nothing has changed except for guacd 1.5.4 -> 1.5.3. No 
other dependency has been updated or changed. OpenSSL is exactly the same. All 
other libs are exactly the same. FreeRDP is left untouched.

So it could be what you mentioned earlier: guacd 1.5.4 might be using something 
in FreeRDP that 1.5.3 wasn't.
Assuming it's FreeRDP that's leaking.
 

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

>
>
> I myself reported having problems with SSH and Telnet. I think at least
> one other person did too. However, in my case the number of RDP connections
> are by fr the most common. So, I got SSH / Telnet issues only *after*
> hitting the so-called freerdp major bug message. So it might be because of
> how guacd uses freerdp and hits that bug that subsequent SSH/Telnet
> connections also fail. No idea.
>

The FreeRDP message is probably due to some resource leakage, perhaps in
FreeRDP, perhaps in some specific aspect of FreeRDP that 1.5.4 started
referencing, that causes the resource to leak to the point where it can no
longer create connections, particularly TLS/SSL connections.


>
> So if my host running guacd 1.5.3 has no issues at all but does with guacd
> 1.5.4 (without absolutely any other software change), what do you mean by
> "underlying system"? Any other library dependency can be the culprit?
>
>
When I say "underlying system" - I mean that I'm not running Docker
containers, I'm installing natively on CentOS7, and when I upgraded from
1.5.3 to 1.5.4, I did not update any of the other dependencies (FreeRDP,
SSH, kernel, openSSL, etc.). This indicates that the issue isn't guacd
itself, but some issue between guacd + FreeRDP and possibly some underlying
library (OpenSSL) that is causing the leakage.

-Nick

Re: Major bug message log in guacd 1.5.4

[Tobias Feller | OPC]

Hi,

i can confirm we had exactly the same problems. Docker environment. 
Downgrade to 1.5.3 and the problems are gone.


Sorry, i have no logs for the problem on this havy used production system.

Best regards,

Tobias

Am 24.01.2024 um 13:25 schrieb Vieri:

On Tuesday, January 23, 2024 at 04:02:10 PM GMT+1, Nick 
Couchman  wrote:

On Tue, Jan 23, 2024 at 6:53 AM Vieri  wrote:

Hi,

Even if the error message in the log refers to freerdp code there is something in 
guacd-1.5.4 that makes the service "in need" of a restart
(not just RDP connections are affected -- VNC, ssh, etc. are all concerned).

I have observed for a long time now that on a host where NO software has been 
changed except for guacd being downgraded from
1.5.4
to 1.5.3 (even with guacamole-client still at 1.5.4), everything is working 
flawlessly.

I'm not saying freerdp isn't somehow responsible, but it is likely that guacd 1.5.3 is 
handling "something" better than 1.5.4.


On the other hand, I have 1.5.4 on a system that was running 1.5.3 before and 
the *only* change I made was to upgrade to 1.5.4,
and I have not encountered the issue.

So, it's got to be some nuance of interaction between guacd and the underlying 
system. I don't know that it's limited to FreeRDP,
either - I think someone else responded at some point saying that they were 
seeing it with SSH, as well.

I myself reported having problems with SSH and Telnet. I think at least one 
other person did too. However, in my case the number of RDP connections are by 
fr the most common. So, I got SSH / Telnet issues only *after* hitting the 
so-called freerdp major bug message. So it might be because of how guacd uses 
freerdp and hits that bug that subsequent SSH/Telnet connections also fail. No 
idea.

So if my host running guacd 1.5.3 has no issues at all but does with guacd 1.5.4 (without 
absolutely any other software change), what do you mean by "underlying system"? 
Any other library dependency can be the culprit?

Vieri

-
To unsubscribe, e-mail:user-unsubscr...@guacamole.apache.org
For additional commands, e-mail:user-h...@guacamole.apache.org


--

OPC® cardsystems GmbH
Phone: +49 651 8408-0  | Fax: +49 651 8408-150
E-Mail:tfel...@opc.de  | Internet:www.opc.de  


Geschäftsführer: Dipl.-Ing. (FH) Klaus Dieter Schömer
Registergericht: Amtsgericht Wittlich | HRB-Nr.: 3903

Re: Major bug message log in guacd 1.5.4

[Vieri]

On Tuesday, January 23, 2024 at 04:02:10 PM GMT+1, Nick Couchman 
 wrote: 

On Tue, Jan 23, 2024 at 6:53 AM Vieri  wrote:
>> Hi,
>> 
>> Even if the error message in the log refers to freerdp code there is 
>> something in guacd-1.5.4 that makes the service "in need" of a restart 
>>(not just RDP connections are affected -- VNC, ssh, etc. are all concerned).
>> 
>> I have observed for a long time now that on a host where NO software has 
>> been changed except for guacd being downgraded from 
>> 1.5.4 
>>to 1.5.3 (even with guacamole-client still at 1.5.4), everything is working 
>>flawlessly.
>> 
>> I'm not saying freerdp isn't somehow responsible, but it is likely that 
>> guacd 1.5.3 is handling "something" better than 1.5.4.
>> 
>
> On the other hand, I have 1.5.4 on a system that was running 1.5.3 before and 
> the *only* change I made was to upgrade to 1.5.4, 
> and I have not encountered the issue.
>
> So, it's got to be some nuance of interaction between guacd and the 
> underlying system. I don't know that it's limited to FreeRDP, 
> either - I think someone else responded at some point saying that they were 
> seeing it with SSH, as well.

I myself reported having problems with SSH and Telnet. I think at least one 
other person did too. However, in my case the number of RDP connections are by 
fr the most common. So, I got SSH / Telnet issues only *after* hitting the 
so-called freerdp major bug message. So it might be because of how guacd uses 
freerdp and hits that bug that subsequent SSH/Telnet connections also fail. No 
idea.

So if my host running guacd 1.5.3 has no issues at all but does with guacd 
1.5.4 (without absolutely any other software change), what do you mean by 
"underlying system"? Any other library dependency can be the culprit?

Vieri

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

On Tue, Jan 23, 2024 at 6:53 AM Vieri  wrote:

> Hi,
>
> Even if the error message in the log refers to freerdp code there is
> something in guacd-1.5.4 that makes the service "in need" of a restart (not
> just RDP connections are affected -- VNC, ssh, etc. are all concerned).
>
> I have observed for a long time now that on a host where NO software has
> been changed except for guacd being downgraded from 1.5.4 to 1.5.3 (even
> with guacamole-client still at 1.5.4), everything is working flawlessly.
>
> I'm not saying freerdp isn't somehow responsible, but it is likely that
> guacd 1.5.3 is handling "something" better than 1.5.4.
>

On the other hand, I have 1.5.4 on a system that was running 1.5.3 before
and the *only* change I made was to upgrade to 1.5.4, and I have not
encountered the issue.

So, it's got to be some nuance of interaction between guacd and the
underlying system. I don't know that it's limited to FreeRDP, either - I
think someone else responded at some point saying that they were seeing it
with SSH, as well.

-Nick

Re: Major bug message log in guacd 1.5.4

[Vieri]

Hi,

Even if the error message in the log refers to freerdp code there is something 
in guacd-1.5.4 that makes the service "in need" of a restart (not just RDP 
connections are affected -- VNC, ssh, etc. are all concerned).

I have observed for a long time now that on a host where NO software has been 
changed except for guacd being downgraded from 1.5.4 to 1.5.3 (even with 
guacamole-client still at 1.5.4), everything is working flawlessly.

I'm not saying freerdp isn't somehow responsible, but it is likely that guacd 
1.5.3 is handling "something" better than 1.5.4.

The only observation I can make is that connections fail after a while with 
guacd 1.5.4 but work in a stable manner with guacd 1.5.3.

The number of eventfds observed are very similar in both situations (1.5.3 vs 
1.5.4).

Not much I can add for debug because I am forced to use 1.5.3 on a production 
system.

Vieri


-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Mauricio Silveira]

Hi. FollowUp:

Rebuilding the docker image with FreeRDP 2.10.0 didn't fix the issue.

In my use case, a daily restart of the guacamole-server container is 
enough ( less than 58 connections/daily I bet ).



Well, Nick Couchman states that it's working fine on CentOS7, but 
official docker images are alpine based, not CentOS7.


It must be something else, not FreeRDP per se.


I'll try a CentOS7 based docker build as soon as I can and report back.


- Mauricio


On 1/11/24 19:08, Mauricio Silveira wrote:
Anyone out there Rebuilt the docker image using FreeRDP 2.10.0 and is 
able to state it is stable now?


When using official docker image, the error started 5-7 days after 
container restart ( restarting guacamole-server container fixes the 
issue immediately ).


I've rebuild guacamole-server docker image with FreeRDP 2.10.0 and got 
it running on January 9th.


If you don´t see a reply to this thread from me in the next 10 days, 
this means it worked. ( or I might reply stating it is fixed )




On 1/2/24 15:31, Michael Jumper wrote:
While the issue may correlate with the release of 1.5.4, keep in mind 
that the log message in question is coming from FreeRDP regarding 
memory that FreeRDP itself manages, and it is reporting what FreeRDP 
identifies as a possible bug *in itself*. See:


https://lists.apache.org/thread/9xws51d32h6mqz35d9odh2kwnl04g8p9

If you suspect this is not the case, then you would need to eliminate 
FreeRDP as a possible cause by rebuilding the 1.5.4 guacd image 
against the same version of FreeRDP used by 1.5.3 (using the 
"WITH_FREERDP" build arg). The version of FreeRDP available at the 
time 1.5.3 was released was "2.10.0".


Pointing to the 1.5.3 image does not just roll back to a previous 
version of Guacamole; it rolls back to the previous version of 
*everything available at the time*, including FreeRDP.


- Mike

On 1/2/24 06:28, Anders Fogh Eriksen wrote:

We just tried rolling back to 1.5.3 and that fixes our issue.

It seems like 1.5.4 contains a lot of changes related to memory 
management: 
https://github.com/apache/guacamole-server/compare/1.5.3...1.5.4# 



On 2024/01/02 13:46:06 Vieri wrote:
 >
 > From the OP it seems that guacamole 1.5.3 did not have this 
issue, but it is yet to be confirmed.

 >
 >
 >
 > 
-

 > To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
 > For additional commands, e-mail: user-h...@guacamole.apache.org
 >
 >




-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org



-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org



-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Re: Major bug message log in guacd 1.5.4

[Weston Thayer]

 No such process
2024-01-12 11:24:34 guacd[1]: DEBUG:All child processes for connection
"$003920a5-932f-44c4-8279-8c37378990ab" have been terminated.

Is there another dependency that's worth testing an image built to match
your version?

On Fri, Jan 12, 2024 at 12:10 AM michael böhm  wrote:

> Hello,
>
> I can confirm Weston's observations. I could reproduce it by connecting 59
> times to to an RDP desktop with SSH enabled. On the 60th attempt it did not
> work anymore (I might have got the count wrong by 1, so it could have very
> well also been after the 58th attempt).
>
> On the failing attempts there is the following in docker host's dmesg:
>
> [Fri Jan 12 08:54:09 2024] guacd[2454882]: segfault at 8 ip
> 7fe9c9b5a7a2 sp 7fe9c6c4b0b8 error 4 in
> ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
> [Fri Jan 12 08:54:09 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16
> 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02
> 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
> [Fri Jan 12 08:54:40 2024] guacd[2455844]: segfault at 8 ip
> 7fe9c9b5a7a2 sp 7fe9c6c4b0b8 error 4 in
> ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
> [Fri Jan 12 08:54:40 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16
> 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02
> 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
> [Fri Jan 12 08:55:10 2024] guacd[2456174]: segfault at 8 ip
> 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in
> ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
> [Fri Jan 12 08:55:10 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16
> 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02
> 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
> [Fri Jan 12 08:55:40 2024] guacd[2456750]: segfault at 8 ip
> 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in
> ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
> [Fri Jan 12 08:55:40 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16
> 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02
> 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
> [Fri Jan 12 08:56:42 2024] guacd[2457875]: segfault at 8 ip
> 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in
> ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
> [Fri Jan 12 08:56:42 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16
> 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02
> 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
> [Fri Jan 12 08:58:39 2024] guacd[2460389]: segfault at 8 ip
> 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in
> ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
> [Fri Jan 12 08:58:39 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16
> 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02
> 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b
>
> guacd's docker logs on a failing connection:
>
> guacd[1]: INFO:Creating new client for protocol "rdp"
> guacd[1]: INFO:Connection ID is "$d39d01e0-5101-4d79-bd8d-a8620c389efa"
> guacd[1800]: WARNING:FreeRDP initialization may fail: Writability of
> the current user's home directory ("/home/guacd") could not be determined:
> Read-only file system
> guacd[1800]: INFO:No security mode specified. Defaulting to security
> mode negotiation with server.
> guacd[1800]: INFO:Resize method: display-update
> guacd[1800]: INFO:No clipboard line-ending normalization specified.
> Defaulting to preserving the format of all line endings.
> guacd[1800]: INFO:User "@ad5b9532-c914-4fc0-be0f-3e7a247157c7" joined
> connection "$d39d01e0-5101-4d79-bd8d-a8620c389efa" (1 users now present)
> guacd[1800]: ERROR:SSH handshake failed.
> guacd[1]: INFO:Connection "$d39d01e0-5101-4d79-bd8d-a8620c389efa"
> removed.
>
> More info on my environment in my thread in this mailing list on Tuesday,
> January 9, 2024 8:03:04 PM GMT+1.
>
> I added a nightly "docker restart guacd" cronjob to the host as a quick
> workaround.
>
> Best wishes
>
> Michael
> *Gesendet:* Freitag, 12. Januar 2024 um 02:42 Uhr
> *Von:* "Nick Couchman" 
> *An:* user@guacamole.apache.org
> *Betreff:* Re: Major bug message log in guacd 1.5.4
> On Thu, Jan 11, 2024 at 5:42 PM Weston Thayer 
> wrote:
>
>> Overall. Our test is just 1 user re-connecting to the same VM over RDP
>> over and over.
>>
>
> Thanks for clarifying. FWIW, I'm running Guacamole 1.5.4 (native
> build/install), on CentOS 7 (FreeRDP 2.2.0), and I've gone way past the
> 58th successive connection without hitting this issue.
>
> -Nick
>
>>
>>
>> - To
> unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org For additional
> commands, e-mail: user-h...@guacamole.apache.org

Aw: Re: Major bug message log in guacd 1.5.4

[michael böhm]

Hello,

 

I can confirm Weston's observations. I could reproduce it by connecting 59 times to to an RDP desktop with SSH enabled. On the 60th attempt it did not work anymore (I might have got the count wrong by 1, so it could have very well also been after the 58th attempt).

 

On the failing attempts there is the following in docker host's dmesg:

 

[Fri Jan 12 08:54:09 2024] guacd[2454882]: segfault at 8 ip 7fe9c9b5a7a2 sp 7fe9c6c4b0b8 error 4 in ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
[Fri Jan 12 08:54:09 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
[Fri Jan 12 08:54:40 2024] guacd[2455844]: segfault at 8 ip 7fe9c9b5a7a2 sp 7fe9c6c4b0b8 error 4 in ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
[Fri Jan 12 08:54:40 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
[Fri Jan 12 08:55:10 2024] guacd[2456174]: segfault at 8 ip 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
[Fri Jan 12 08:55:10 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
[Fri Jan 12 08:55:40 2024] guacd[2456750]: segfault at 8 ip 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
[Fri Jan 12 08:55:40 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
[Fri Jan 12 08:56:42 2024] guacd[2457875]: segfault at 8 ip 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
[Fri Jan 12 08:56:42 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b1
[Fri Jan 12 08:58:39 2024] guacd[2460389]: segfault at 8 ip 7fe9c9b5a7a2 sp 7fe9c6c620b8 error 4 in ld-musl-x86_64.so.1[7fe9c9b18000+4c000]
[Fri Jan 12 08:58:39 2024] Code: e8 ef 16 00 00 31 c0 5a c3 31 c0 c3 b8 16 00 00 00 c3 b9 0a 00 00 00 31 c0 48 89 fa f3 ab 48 85 f6 74 04 8b 06 89 02 31 c0 c3 <8b> 07 83 e0 0f 74 07 31 f6 e9 15 00 00 00 ba 10 00 00 00 f0 0f b

 

guacd's docker logs on a failing connection:

 

guacd[1]: INFO:    Creating new client for protocol "rdp"
guacd[1]: INFO:    Connection ID is "$d39d01e0-5101-4d79-bd8d-a8620c389efa"
guacd[1800]: WARNING:    FreeRDP initialization may fail: Writability of the current user's home directory ("/home/guacd") could not be determined: Read-only file system
guacd[1800]: INFO:    No security mode specified. Defaulting to security mode negotiation with server.
guacd[1800]: INFO:    Resize method: display-update
guacd[1800]: INFO:    No clipboard line-ending normalization specified. Defaulting to preserving the format of all line endings.
guacd[1800]: INFO:    User "@ad5b9532-c914-4fc0-be0f-3e7a247157c7" joined connection "$d39d01e0-5101-4d79-bd8d-a8620c389efa" (1 users now present)
guacd[1800]: ERROR:    SSH handshake failed.
guacd[1]: INFO:    Connection "$d39d01e0-5101-4d79-bd8d-a8620c389efa" removed.

 

More info on my environment in my thread in this mailing list on Tuesday, January 9, 2024 8:03:04 PM GMT+1.

 

I added a nightly "docker restart guacd" cronjob to the host as a quick workaround.

 

Best wishes

 

Michael


Gesendet: Freitag, 12. Januar 2024 um 02:42 Uhr
Von: "Nick Couchman" 
An: user@guacamole.apache.org
Betreff: Re: Major bug message log in guacd 1.5.4



On Thu, Jan 11, 2024 at 5:42 PM Weston Thayer <wes...@assistivlabs.com> wrote:



Overall. Our test is just 1 user re-connecting to the same VM over RDP over and over.


 

Thanks for clarifying. FWIW, I'm running Guacamole 1.5.4 (native build/install), on CentOS 7 (FreeRDP 2.2.0), and I've gone way past the 58th successive connection without hitting this issue.

 

-Nick



 









-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

On Thu, Jan 11, 2024 at 5:42 PM Weston Thayer 
wrote:

> Overall. Our test is just 1 user re-connecting to the same VM over RDP
> over and over.
>

Thanks for clarifying. FWIW, I'm running Guacamole 1.5.4 (native
build/install), on CentOS 7 (FreeRDP 2.2.0), and I've gone way past the
58th successive connection without hitting this issue.

-Nick

>

Re: Major bug message log in guacd 1.5.4

[Weston Thayer]

Overall. Our test is just 1 user re-connecting to the same VM over RDP over
and over.

On Thu, Jan 11, 2024 at 2:36 PM Nick Couchman  wrote:

> On Thu, Jan 11, 2024 at 5:13 PM Weston Thayer 
> wrote:
>
>> We can reliability reproduce it with the Docker image at
>> https://github.com/apache/guacamole-server/commit/94a755c90a7511969884774dbca2ef2ab5958ba6
>> on the 58th connection in the guacd container's lifetime. A bit annoying to
>> reproduce manually, but you could write a little test script.
>>
>>
> The 58th concurrent connection, or the 58th overall connection?
>
> -Nick
>

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

On Thu, Jan 11, 2024 at 5:13 PM Weston Thayer 
wrote:

> We can reliability reproduce it with the Docker image at
> https://github.com/apache/guacamole-server/commit/94a755c90a7511969884774dbca2ef2ab5958ba6
> on the 58th connection in the guacd container's lifetime. A bit annoying to
> reproduce manually, but you could write a little test script.
>
>
The 58th concurrent connection, or the 58th overall connection?

-Nick

Re: Major bug message log in guacd 1.5.4

[Weston Thayer]

We can reliability reproduce it with the Docker image at
https://github.com/apache/guacamole-server/commit/94a755c90a7511969884774dbca2ef2ab5958ba6
on the 58th connection in the guacd container's lifetime. A bit annoying to
reproduce manually, but you could write a little test script.

On Thu, Jan 11, 2024 at 2:08 PM Mauricio Silveira
 wrote:

> Anyone out there Rebuilt the docker image using FreeRDP 2.10.0 and is
> able to state it is stable now?
>
> When using official docker image, the error started 5-7 days after
> container restart ( restarting guacamole-server container fixes the
> issue immediately ).
>
> I've rebuild guacamole-server docker image with FreeRDP 2.10.0 and got
> it running on January 9th.
>
> If you don´t see a reply to this thread from me in the next 10 days,
> this means it worked. ( or I might reply stating it is fixed )
>
>
>
> On 1/2/24 15:31, Michael Jumper wrote:
> > While the issue may correlate with the release of 1.5.4, keep in mind
> > that the log message in question is coming from FreeRDP regarding
> > memory that FreeRDP itself manages, and it is reporting what FreeRDP
> > identifies as a possible bug *in itself*. See:
> >
> > https://lists.apache.org/thread/9xws51d32h6mqz35d9odh2kwnl04g8p9
> >
> > If you suspect this is not the case, then you would need to eliminate
> > FreeRDP as a possible cause by rebuilding the 1.5.4 guacd image
> > against the same version of FreeRDP used by 1.5.3 (using the
> > "WITH_FREERDP" build arg). The version of FreeRDP available at the
> > time 1.5.3 was released was "2.10.0".
> >
> > Pointing to the 1.5.3 image does not just roll back to a previous
> > version of Guacamole; it rolls back to the previous version of
> > *everything available at the time*, including FreeRDP.
> >
> > - Mike
> >
> > On 1/2/24 06:28, Anders Fogh Eriksen wrote:
> >> We just tried rolling back to 1.5.3 and that fixes our issue.
> >>
> >> It seems like 1.5.4 contains a lot of changes related to memory
> >> management:
> >> https://github.com/apache/guacamole-server/compare/1.5.3...1.5.4#
> >> 
> >>
> >> On 2024/01/02 13:46:06 Vieri wrote:
> >>  >
> >>  > From the OP it seems that guacamole 1.5.3 did not have this issue,
> >> but it is yet to be confirmed.
> >>  >
> >>  >
> >>  >
> >>  > -
> >>  > To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> >>  > For additional commands, e-mail: user-h...@guacamole.apache.org
> >>  >
> >>  >
> >>
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> > For additional commands, e-mail: user-h...@guacamole.apache.org
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> For additional commands, e-mail: user-h...@guacamole.apache.org
>
>

Re: Major bug message log in guacd 1.5.4

[Mauricio Silveira]

Anyone out there Rebuilt the docker image using FreeRDP 2.10.0 and is 
able to state it is stable now?


When using official docker image, the error started 5-7 days after 
container restart ( restarting guacamole-server container fixes the 
issue immediately ).


I've rebuild guacamole-server docker image with FreeRDP 2.10.0 and got 
it running on January 9th.


If you don´t see a reply to this thread from me in the next 10 days, 
this means it worked. ( or I might reply stating it is fixed )




On 1/2/24 15:31, Michael Jumper wrote:
While the issue may correlate with the release of 1.5.4, keep in mind 
that the log message in question is coming from FreeRDP regarding 
memory that FreeRDP itself manages, and it is reporting what FreeRDP 
identifies as a possible bug *in itself*. See:


https://lists.apache.org/thread/9xws51d32h6mqz35d9odh2kwnl04g8p9

If you suspect this is not the case, then you would need to eliminate 
FreeRDP as a possible cause by rebuilding the 1.5.4 guacd image 
against the same version of FreeRDP used by 1.5.3 (using the 
"WITH_FREERDP" build arg). The version of FreeRDP available at the 
time 1.5.3 was released was "2.10.0".


Pointing to the 1.5.3 image does not just roll back to a previous 
version of Guacamole; it rolls back to the previous version of 
*everything available at the time*, including FreeRDP.


- Mike

On 1/2/24 06:28, Anders Fogh Eriksen wrote:

We just tried rolling back to 1.5.3 and that fixes our issue.

It seems like 1.5.4 contains a lot of changes related to memory 
management: 
https://github.com/apache/guacamole-server/compare/1.5.3...1.5.4# 



On 2024/01/02 13:46:06 Vieri wrote:
 >
 > From the OP it seems that guacamole 1.5.3 did not have this issue, 
but it is yet to be confirmed.

 >
 >
 >
 > -
 > To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
 > For additional commands, e-mail: user-h...@guacamole.apache.org
 >
 >




-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org



-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

On Mon, Jan 8, 2024 at 6:07 AM Joonas Tuomisto  wrote:

> Hi,
>
> I also ran into this exact issue it seems, but we're not using the
> Guacamole Docker images, instead we're using custom built RPM packages
> on OpenSUSE Leap 15.5 as a native systemd service.
>
> To me this does indeed look like some kind of resource leak in guacd.
>
> Initially everything works, but eventually guacd stops being able to
> initiate new connections, and it's not limited to just one protocol,
> at least both RDP and SSH are impacted.
>
> I'm not entirely sure how forking processes interact with system
> limits and how to debug this to find what resource exactly is being
> leaked.
>
> I attached to guacd with strace after encountering this state and
> initiated a new RDP/SSH connection and it does appear to be failing
> after a connect() but I didn't notice anything weird.
>
> connect() goes -1 EINPROGRESS and poll, recvfrom() goes -1 EAGAIN.
>
> There's indeed a whole bunch of eventfd2() when trying an RDP connection.
>
> guacd logs (for RDP):
> > RDP server closed/refused connection: Security negotiation failed (wrong
> security type?)
>
> guacd logs (for SSH):
> > GLib (gthread-posix.c): Unexpected error from C library during
> 'pthread_key_create': Resource temporarily unavailable. Aborting.
>

Interesting. If you're able to track down how this is leaking, that'd be
really helpful. I'm running 1.5.4 on CentOS 7, and not seeing this at all.

-Nick

RE: Major bug message log in guacd 1.5.4

[Pål Hjelmeseth Myklebust]

We also experienced this. Upgraded from 1.5.3 to 1.5.4 and everything worked 
fine. The next day nobody was able to establish a connection. I did not have 
time to debug, so I downgraded the guacd server to 1.5.3, the web client and 
the extensions are still running 1.5.4, and everything works ok. 

-
Pål

-Original Message-
From: Joonas Tuomisto  
Sent: Monday, January 8, 2024 12:06 PM
To: user@guacamole.apache.org
Subject: Re: Major bug message log in guacd 1.5.4

Hi,

I also ran into this exact issue it seems, but we're not using the Guacamole 
Docker images, instead we're using custom built RPM packages on OpenSUSE Leap 
15.5 as a native systemd service.

To me this does indeed look like some kind of resource leak in guacd.

Initially everything works, but eventually guacd stops being able to initiate 
new connections, and it's not limited to just one protocol, at least both RDP 
and SSH are impacted.

I'm not entirely sure how forking processes interact with system limits and how 
to debug this to find what resource exactly is being leaked.

I attached to guacd with strace after encountering this state and initiated a 
new RDP/SSH connection and it does appear to be failing after a connect() but I 
didn't notice anything weird.

connect() goes -1 EINPROGRESS and poll, recvfrom() goes -1 EAGAIN.

There's indeed a whole bunch of eventfd2() when trying an RDP connection.

guacd logs (for RDP):
> RDP server closed/refused connection: Security negotiation failed 
> (wrong security type?)

guacd logs (for SSH):
> GLib (gthread-posix.c): Unexpected error from C library during 
> 'pthread_key_create': Resource temporarily unavailable. Aborting.

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Joonas Tuomisto]

Hi,

I also ran into this exact issue it seems, but we're not using the
Guacamole Docker images, instead we're using custom built RPM packages
on OpenSUSE Leap 15.5 as a native systemd service.

To me this does indeed look like some kind of resource leak in guacd.

Initially everything works, but eventually guacd stops being able to
initiate new connections, and it's not limited to just one protocol,
at least both RDP and SSH are impacted.

I'm not entirely sure how forking processes interact with system
limits and how to debug this to find what resource exactly is being
leaked.

I attached to guacd with strace after encountering this state and
initiated a new RDP/SSH connection and it does appear to be failing
after a connect() but I didn't notice anything weird.

connect() goes -1 EINPROGRESS and poll, recvfrom() goes -1 EAGAIN.

There's indeed a whole bunch of eventfd2() when trying an RDP connection.

guacd logs (for RDP):
> RDP server closed/refused connection: Security negotiation failed (wrong 
> security type?)

guacd logs (for SSH):
> GLib (gthread-posix.c): Unexpected error from C library during 
> 'pthread_key_create': Resource temporarily unavailable. Aborting.

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Vieri]

 On Saturday, January 6, 2024 at 06:50:38 AM GMT+1, Michael Jumper 
 wrote: 

> Interesting that there's so many eventfds. I wonder if there might be an 
> underlying library call that's leaking these...
>
> To my knowledge, we're not opening any eventfds directly within the Guacamole 
> codebase, but FreeRDP definitely does. There might be 
> something that's not being freed quite right.


Hi,

I've hit the same issue again. Before restarting guacd and resuming normal 
operation, the processes were as shown below:

166475 ?    00:00:25 guacd
170223 ?    00:00:23 guacd
216997 ?    00:00:31 guacd
227763 ?    00:05:46 guacd
235636 ?    00:00:10 guacd


COMMAND    PID  USER   FD  TYPE DEVICE SIZE/OFF  NODE NAME
guacd   235636 guacd  cwd   DIR  253,4 4096 2 /
guacd   235636 guacd  rtd   DIR  253,4 4096 2 /
guacd   235636 guacd  txt   REG  253,4    43048   6304952 
/usr/sbin/guacd
guacd   235636 guacd  mem   REG  253,4    88712   3801105 
/lib64/libresolv-2.33.so
guacd   235636 guacd  mem   REG  253,4   403016   6303450 
/usr/lib64/libopus.so.0.8.0
guacd   235636 guacd  mem   REG  253,4   280696   6303443 
/usr/lib64/libFLAC.so.8.3.0
guacd   235636 guacd  mem   REG  253,4   571256   6303544 
/usr/lib64/libsndfile.so.1.0.31
guacd   235636 guacd  mem   REG  253,4   607568  10635891 
/usr/lib64/pulseaudio/libpulsecommon-15.0.so
guacd   235636 guacd  mem   REG  253,4   725464   6303397 
/usr/lib64/libxvidcore.so.4.3
guacd   235636 guacd  mem   REG  253,4  1957104   6303394 
/usr/lib64/libx264.so.157
guacd   235636 guacd  mem   REG  253,4    81216   6303521 
/usr/lib64/libtheoradec.so.1.1.4
guacd   235636 guacd  mem   REG  253,4   201088   6303520 
/usr/lib64/libtheoraenc.so.1.1.2
guacd   235636 guacd  mem   REG  253,4   332960   6303499 
/usr/lib64/libmp3lame.so.0.0.0
guacd   235636 guacd  mem   REG  253,4  1799512   6303495 
/usr/lib64/libdav1d.so.5.1.1
guacd   235636 guacd  mem   REG  253,4    48952   6303529 
/usr/lib64/libwebpmux.so.3.0.5
guacd   235636 guacd  mem   REG  253,4   100520   6165345 
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/libgcc_s.so.1
guacd   235636 guacd  mem   REG  253,4  2161136   6165346 
/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.0/libstdc++.so.6.0.29
guacd   235636 guacd  mem   REG  253,4   398864   6303568 
/usr/lib64/libpulse.so.0.24.0
guacd   235636 guacd  mem   REG  253,4  1137224   6303457 
/usr/lib64/libasound.so.2.0.0
guacd   235636 guacd  mem   REG  253,4    21888   6303405 
/usr/lib64/libasyncns.so.0.3.1
guacd   235636 guacd  mem   REG  253,4   545576   6303597 
/usr/lib64/libavutil.so.56.70.100
guacd   235636 guacd  mem   REG  253,4 12799736   6303592 
/usr/lib64/libavcodec.so.58.134.100
guacd   235636 guacd  mem   REG  253,4  1136960   6303503 
/usr/lib64/libopenh264.so.2.1.1
guacd   235636 guacd  mem   REG  253,4   621416   6303596 
/usr/lib64/libswscale.so.5.9.100
guacd   235636 guacd  mem   REG  253,4  3785912   6305021 
/usr/lib64/libwinpr2.so.2.4.1
guacd   235636 guacd  mem   REG  253,4  8510336   6305027 
/usr/lib64/libfreerdp2.so.2.4.1
guacd   235636 guacd  mem   REG  253,4   161168   6303595 
/usr/lib64/libswresample.so.3.9.100
guacd   235636 guacd  mem   REG  253,4   311328   6304853 
/usr/lib64/libssh2.so.1.0.1
guacd   235636 guacd  mem   REG  253,4  3365648   6305030 
/usr/lib64/libfreerdp-client2.so.2.4.1
guacd   235636 guacd  mem   REG  253,4   382104   6304965 
/usr/lib64/libguac-client-rdp.so.0.0.0
guacd   235636 guacd  mem   REG  253,4    26760   3801190 
/lib64/libnss_dns-2.33.so
guacd   235636 guacd  mem   REG  253,4    47384   3801196 
/lib64/libnss_files-2.33.so
guacd   235636 guacd  mem   REG  253,4    74904   3801231 
/lib64/libbz2.so.1.0.8
guacd   235636 guacd  mem   REG  253,4   194528   6300672 
/usr/lib64/libexpat.so.1.8.1
guacd   235636 guacd  mem   REG  253,4  1304768   3801202 
/lib64/libm-2.33.so
guacd   235636 guacd  mem   REG  253,4   108904   3801161 
/lib64/libz.so.1.2.12
guacd   235636 guacd  mem   REG  253,4   899416   2104311 
/usr/lib64/libfreetype.so.6.18.0
guacd   235636 guacd  mem   REG  253,4   333784   6301726 
/usr/lib64/libfontconfig.so.1.12.0
guacd   235636 guacd  mem   REG  253,4   746480   6304850 
/usr/lib64/libpixman-1.so.0.40.0
guacd   235636 guacd  mem   REG  253,4    35640   3801255 
/lib64/librt-2.33.so
guacd   235636 guacd  mem   REG   

Re: Major bug message log in guacd 1.5.4

[Michael Jumper]

On 1/4/24 16:35, Vieri wrote:

  On Thursday, January 4, 2024 at 02:50:47 PM GMT+1, Nick Couchman 
 wrote:


I suspect that you're hitting some other limitation in the number of 
files/sockets open that is causing all new connections to fail



Hi Nick,

...
guacd   84702 guacd    8u  a_inode   0,13    0 14718 
[eventfd:32]
guacd   84702 guacd    9u  a_inode   0,13    0 14718 
[eventfd:41]
guacd   84702 guacd   10u  a_inode   0,13    0 14718 
[eventfd:46]
guacd   84702 guacd   11u  a_inode   0,13    0 14718 
[eventfd:47]
guacd   84702 guacd   12u  a_inode   0,13    0 14718 
[eventfd:48]
...
guacd   84702 guacd   73u  a_inode   0,13    0 14718 
[eventfd:109]
guacd   84702 guacd   74u  a_inode   0,13    0 14718 
[eventfd:110]
guacd   84702 guacd   75u  a_inode   0,13    0 14718 
[eventfd:111]



Interesting that there's so many eventfds. I wonder if there might be an 
underlying library call that's leaking these...


To my knowledge, we're not opening any eventfds directly within the 
Guacamole codebase, but FreeRDP definitely does. There might be 
something that's not being freed quite right.


- Mike

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Vieri]

 On Thursday, January 4, 2024 at 02:50:47 PM GMT+1, Nick Couchman 
 wrote: 

> I suspect that you're hitting some other limitation in the number of 
> files/sockets open that is causing all new connections to fail


Hi Nick,

# ps -ae | grep guacd
 84702 ?    00:00:29 guacd
227763 ?    00:02:13 guacd

# cat /proc/84702/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimited    unlimited    seconds
Max file size unlimited    unlimited    bytes
Max data size unlimited    unlimited    bytes
Max stack size    8388608  unlimited    bytes
Max core file size    0    unlimited    bytes
Max resident set  unlimited    unlimited    bytes
Max processes 127588   127588   processes
Max open files    16384    65535    files
Max locked memory 65536    65536    bytes
Max address space unlimited    unlimited    bytes
Max file locks    unlimited    unlimited    locks
Max pending signals   127588   127588   signals
Max msgqueue size 819200   819200   bytes
Max nice priority 0    0
Max realtime priority 0    0
Max realtime timeout  unlimited    unlimited    us

# cat /proc/227763/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimited    unlimited    seconds
Max file size unlimited    unlimited    bytes
Max data size unlimited    unlimited    bytes
Max stack size    8388608  unlimited    bytes
Max core file size    0    unlimited    bytes
Max resident set  unlimited    unlimited    bytes
Max processes 127588   127588   processes
Max open files    16384    65535    files
Max locked memory 65536    65536    bytes
Max address space unlimited    unlimited    bytes
Max file locks    unlimited    unlimited    locks
Max pending signals   127588   127588   signals
Max msgqueue size 819200   819200   bytes
Max nice priority 0    0
Max realtime priority 0    0
Max realtime timeout  unlimited    unlimited    us

# lsof -p 84702
COMMAND   PID  USER   FD  TYPE DEVICE SIZE/OFF  NODE NAME
guacd   84702 guacd  cwd   DIR  253,4 4096 2 /
guacd   84702 guacd  rtd   DIR  253,4 4096 2 /
guacd   84702 guacd  txt   REG  253,4    43048   6304952 
/usr/sbin/guacd
guacd   84702 guacd  mem   REG  253,4    47384   3801196 
/lib64/libnss_files-2.33.so
guacd   84702 guacd  mem   REG  253,4    88712   3801105 
/lib64/libresolv-2.33.so
guacd   84702 guacd  mem   REG  253,4   403016   6303450 
/usr/lib64/libopus.so.0.8.0
guacd   84702 guacd  mem   REG  253,4  1957104   6303394 
/usr/lib64/libx264.so.157
guacd   84702 guacd  mem   REG  253,4    73584   6303890 
/usr/lib64/freerdp2/libguacai-client.so
guacd   84702 guacd  mem   REG  253,4   280696   6303443 
/usr/lib64/libFLAC.so.8.3.0
guacd   84702 guacd  mem   REG  253,4 12799736   6303592 
/usr/lib64/libavcodec.so.58.134.100
guacd   84702 guacd  mem   REG  253,4    26760   3801190 
/lib64/libnss_dns-2.33.so
guacd   84702 guacd  mem   REG  253,4    21888   6303405 
/usr/lib64/libasyncns.so.0.3.1
guacd   84702 guacd  mem   REG  253,4   571256   6303544 
/usr/lib64/libsndfile.so.1.0.31
guacd   84702 guacd  mem   REG  253,4   607568  10635891 
/usr/lib64/pulseaudio/libpulsecommon-15.0.so
guacd   84702 guacd  mem   REG  253,4   725464   6303397 
/usr/lib64/libxvidcore.so.4.3
guacd   84702 guacd  mem   REG  253,4    81216   6303521 
/usr/lib64/libtheoradec.so.1.1.4
guacd   84702 guacd  mem   REG  253,4   201088   6303520 
/usr/lib64/libtheoraenc.so.1.1.2
guacd   84702 guacd  mem   REG  253,4   332960   6303499 
/usr/lib64/libmp3lame.so.0.0.0
guacd   84702 guacd  mem   REG  253,4  1799512   6303495 
/usr/lib64/libdav1d.so.5.1.1
guacd   84702 guacd  mem   REG  253,4    48952   6303529 
/usr/lib64/libwebpmux.so.3.0.5
guacd   84702 guacd  mem   REG  253,4   100520   6165345 

Re: Major bug message log in guacd 1.5.4

[Nick Couchman]

On Thu, Jan 4, 2024 at 2:00 AM Vieri  wrote:

> Hi,
>
> I understand what you're saying about docker images and reverting to
> 1.5.3, Mike.
> However, in my case I'm NOT using Docker.
>
> As I said, I have two different hosts and BOTH are using FreeRDP 2.4.1 (am
> not using 2.10.0 as you suggest).
> The 2 hosts are 2 different machines. The only software that changes is,
> as I said, guacd (and Apache Tomcat to be exact, but it's still 8.5 on
> both).
>
> I have NOT tried downgrading guacamole to 1.5.3.
> I am using 1.4.0 on my old system and 1.5.4 on the new one.
> Obviously, 1.4.0 and 1.5.3 have vulnerabilities that I want to get rid of,
> so I'm eager to move to 1.5.4. However, this issue is a show stopper.
>
> Also, while I understand that the error "Major bug, unable to allocate a
> TLS value for currentThread" comes from FreeRDP, I also reported in my
> previous post that all other protocols are also affected.
> My telnet and ssh connections fail too, but they obviously do not use
> FreeRDP.
> So, it seems that as soon as guacd hits the FreeRDP issue, every single
> connection type (including those that do not use FreeRDP) fail.
> A guacd restart makes every single connection type work again -- not just
> RDP.
> I might be wrong, and I don't have more information to offer (my guacd
> 1.5.4 process was restarted and working fine for now).
> However, it seems that the FreeRDP "failure" has a crippling effect on
> guacd not just for RDP connections.
>
>
I highly doubt that FreeRDP is having any impact on other guacd connections
- I suspect that you're hitting some other limitation in the number of
files/sockets open that is causing all new connections to fail, and the
FreeRDP one just happens to be the most likely one or first one to hit the
issue. You might look at things like the "nofile" hard and soft limits for
the user running guacd and see if you're running up against that. IIRC,
opening sockets counts as opening a "file" in Linux, so if you are opening
a lot of connections, you may be hitting that.

It may not be nofile specifically, so you might need to investigate other
limits, either in /etc/security/limits.conf, or sysctl, and see if you're
running up against any of those.

-Nick

Re: Major bug message log in guacd 1.5.4

[Vieri]

Hi,

I understand what you're saying about docker images and reverting to 1.5.3, 
Mike.
However, in my case I'm NOT using Docker.

As I said, I have two different hosts and BOTH are using FreeRDP 2.4.1 (am not 
using 2.10.0 as you suggest).
The 2 hosts are 2 different machines. The only software that changes is, as I 
said, guacd (and Apache Tomcat to be exact, but it's still 8.5 on both).

I have NOT tried downgrading guacamole to 1.5.3.
I am using 1.4.0 on my old system and 1.5.4 on the new one.
Obviously, 1.4.0 and 1.5.3 have vulnerabilities that I want to get rid of, so 
I'm eager to move to 1.5.4. However, this issue is a show stopper.

Also, while I understand that the error "Major bug, unable to allocate a TLS 
value for currentThread" comes from FreeRDP, I also reported in my previous 
post that all other protocols are also affected.
My telnet and ssh connections fail too, but they obviously do not use FreeRDP.
So, it seems that as soon as guacd hits the FreeRDP issue, every single 
connection type (including those that do not use FreeRDP) fail.
A guacd restart makes every single connection type work again -- not just RDP.
I might be wrong, and I don't have more information to offer (my guacd 1.5.4 
process was restarted and working fine for now).
However, it seems that the FreeRDP "failure" has a crippling effect on guacd 
not just for RDP connections.

If I understand correctly you're saying that the 1.5.3 docker image is using a 
different freeRDP version than the 1.5.4 image, and that you want people using 
1.5.4 to use FreeRDP 2.10.0.
Since I don't use the docker images I have no idea which FreeRDP version 1.5.3 
is using.
In any case, my 2 hosts (1.4.0 and 1.5.4) are not using FreeRDP 2.10.0, but 
they ARE using the same FreeRDP version (2.4.1), but I'm seeing the same issue 
as the rest of the reports on this ML.
So in my case, it's probably recommended to upgrade FreeRDP from 2.4.1 to 
2.10.0 for other reasons, but I'm wondering if it's really relevant to this 
issue.

I guess the only thing I can try is to rebuild my guacamole 1.5.4 setup with 
FreeRDP 2.10.0 (upgrade 2.4.1 to 2.10.0), and leave my old guacamole 1.4.0 with 
FreeRDP 2.4.1 untouched.

Regards,

Vieri

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Michael Jumper]

Correction, that should be:

$ git clone https://github.com/apache/guacamole-server
$ cd guacamole-server/
$ git checkout 1.5.4
$ sudo docker build -t guacamole/guacd \
 --build-arg WITH_FREERDP=2.10.0 \
 --build-arg ALPINE_BASE_IMAGE=3.18 .

- Mike

On 1/3/24 09:13, Michael Jumper wrote:
All, if you wish to test this issue further, the thing to test is 
whether the *1.5.4* images suffer from the same failure even if rebuilt 
against the same version of FreeRDP used by 1.5.3 (version 2.10.0).


There is no need to confirm that the issue does not occur with 1.5.3 
version of the Docker images. This is already known but too broad a 
result, as switching from "guacamole/guacd:1.5.4" to 
"guacamole/guacd:1.5.3" does not simply switch to an older version of 
Guacamole; it switches to older versions of absolutely everything, 
including the very software reporting the error (FreeRDP).


See:

* https://lists.apache.org/thread/9xws51d32h6mqz35d9odh2kwnl04g8p9
* https://lists.apache.org/thread/27mg4wg4lh7rnqxnjw8j3xhjlbj1v95w

You can try rebuilding against FreeRDP 2.10.0 by checking out a copy of 
the 1.5.4 source and using the "WITH_FREERDP" build arg to specify 
"2.10.0":


$ git clone https://github.com/apache/guacamole-server
$ cd guacamole-server/
$ git checkout 1.5.4
$ sudo docker build -t guacamole/guacamole \
     --build-arg WITH_FREERDP=2.10.0 \
     --build-arg ALPINE_BASE_IMAGE=3.18 .

- Mike

On 1/3/24 08:59, John Watson wrote:
1.5.3 does not have the issue. I was dealing with it in 4 different 
environments on 1.5.4 very frequently, and have had 0 incidents since 
reverting to 1.5.3.


On Tue, Jan 2, 2024 at 8:46 AM Vieri  wrote:


 From the OP it seems that guacamole 1.5.3 did not have this issue,
    but it is yet to be confirmed.



    -
    To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
    
    For additional commands, e-mail: user-h...@guacamole.apache.org
    



-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Michael Jumper]

All, if you wish to test this issue further, the thing to test is 
whether the *1.5.4* images suffer from the same failure even if rebuilt 
against the same version of FreeRDP used by 1.5.3 (version 2.10.0).


There is no need to confirm that the issue does not occur with 1.5.3 
version of the Docker images. This is already known but too broad a 
result, as switching from "guacamole/guacd:1.5.4" to 
"guacamole/guacd:1.5.3" does not simply switch to an older version of 
Guacamole; it switches to older versions of absolutely everything, 
including the very software reporting the error (FreeRDP).


See:

* https://lists.apache.org/thread/9xws51d32h6mqz35d9odh2kwnl04g8p9
* https://lists.apache.org/thread/27mg4wg4lh7rnqxnjw8j3xhjlbj1v95w

You can try rebuilding against FreeRDP 2.10.0 by checking out a copy of 
the 1.5.4 source and using the "WITH_FREERDP" build arg to specify "2.10.0":


$ git clone https://github.com/apache/guacamole-server
$ cd guacamole-server/
$ git checkout 1.5.4
$ sudo docker build -t guacamole/guacamole \
--build-arg WITH_FREERDP=2.10.0 \
--build-arg ALPINE_BASE_IMAGE=3.18 .

- Mike

On 1/3/24 08:59, John Watson wrote:
1.5.3 does not have the issue. I was dealing with it in 4 different 
environments on 1.5.4 very frequently, and have had 0 incidents since 
reverting to 1.5.3.


On Tue, Jan 2, 2024 at 8:46 AM Vieri  wrote:


 From the OP it seems that guacamole 1.5.3 did not have this issue,
but it is yet to be confirmed.



-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org

For additional commands, e-mail: user-h...@guacamole.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[John Watson]

1.5.3 does not have the issue. I was dealing with it in 4 different
environments on 1.5.4 very frequently, and have had 0 incidents since
reverting to 1.5.3.

On Tue, Jan 2, 2024 at 8:46 AM Vieri  wrote:

>
> From the OP it seems that guacamole 1.5.3 did not have this issue, but it
> is yet to be confirmed.
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> For additional commands, e-mail: user-h...@guacamole.apache.org
>
>

Re: Major bug message log in guacd 1.5.4

[Michael Jumper]

While the issue may correlate with the release of 1.5.4, keep in mind 
that the log message in question is coming from FreeRDP regarding memory 
that FreeRDP itself manages, and it is reporting what FreeRDP identifies 
as a possible bug *in itself*. See:


https://lists.apache.org/thread/9xws51d32h6mqz35d9odh2kwnl04g8p9

If you suspect this is not the case, then you would need to eliminate 
FreeRDP as a possible cause by rebuilding the 1.5.4 guacd image against 
the same version of FreeRDP used by 1.5.3 (using the "WITH_FREERDP" 
build arg). The version of FreeRDP available at the time 1.5.3 was 
released was "2.10.0".


Pointing to the 1.5.3 image does not just roll back to a previous 
version of Guacamole; it rolls back to the previous version of 
*everything available at the time*, including FreeRDP.


- Mike

On 1/2/24 06:28, Anders Fogh Eriksen wrote:

We just tried rolling back to 1.5.3 and that fixes our issue.

It seems like 1.5.4 contains a lot of changes related to memory 
management: 
https://github.com/apache/guacamole-server/compare/1.5.3...1.5.4# 



On 2024/01/02 13:46:06 Vieri wrote:
 >
 > From the OP it seems that guacamole 1.5.3 did not have this issue, 
but it is yet to be confirmed.

 >
 >
 >
 > -
 > To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
 > For additional commands, e-mail: user-h...@guacamole.apache.org
 >
 >




-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

RE: Re: Major bug message log in guacd 1.5.4

[Anders Fogh Eriksen]

We just tried rolling back to 1.5.3 and that fixes our issue.

It seems like 1.5.4 contains a lot of changes related to memory management: 
https://github.com/apache/guacamole-server/compare/1.5.3...1.5.4#

On 2024/01/02 13:46:06 Vieri wrote:
>
> From the OP it seems that guacamole 1.5.3 did not have this issue, but it is 
> yet to be confirmed.
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
> For additional commands, e-mail: user-h...@guacamole.apache.org
>
>

Re: Major bug message log in guacd 1.5.4

[Vieri]

>From the OP it seems that guacamole 1.5.3 did not have this issue, but it is 
>yet to be confirmed.



-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Anders Fogh Eriksen]

We have the exact same issue as Vieri is experiencing. We can add a bit more 
information.
The error message:

Major bug, unable to allocate a TLS value for currentThread


is thrown by FreeRDP: 
https://github.com/FreeRDP/FreeRDP/blob/87557b1e4ef636e18a7dea58f0cae76fa5dff4fb/winpr/libwinpr/thread/thread.c#L439

We have a setup running Guacd in Docker running Alpine 3.18.5 on an Ubuntu 
20.04 OS. After exactly 59 connections guacd is put in a state where it will 
throw the error above every time a new connection attempt is made.
The only way to get guacd out of this state seems to be to restart the Docker 
host. After a reboot, we're able to do another 59 connections before needing to 
restart again.


Furthermore, we can also see the following from our logs:

guacd[6368]: DEBUG: SSL_CTX_new failed
2024-01-02T08:47:53.508450591Z [2024-01-02 08:47:53 UTC] guacd[6368]: DEBUG: 
transport_connect_tls:freerdp_set_last_error_ex ERRCONNECT_CONNECT_CANCELLED 
[0x0002000B]
2024-01-02T08:47:53.509335078Z [2024-01-02 08:47:53 UTC] guacd[6368]: DEBUG: 
SVC "rdpdr" disconnected.
2024-01-02T08:47:53.509352781Z [2024-01-02 08:47:53 UTC] guacd[6368]: DEBUG: 
SVC "rdpsnd" disconnected.
2024-01-02T08:47:53.509358692Z [2024-01-02 08:47:53 UTC] guacd[6368]: INFO: RDP 
server closed/refused connection: Connection failed (server unreachable?)


Which seems to be an issue from within OpenSSL because of the SSL_CTX_new 
message.
We're currently backtracking from the FreeRDP and Guacd code to see if we can 
figure our why this state occurs; but any help from the developers would be 
hugely appreciated. This is currently holding back a product launch in our end, 
so we're very motivated to help figure out what's causing this.

From: Vieri 
Date: Tuesday, 2 January 2024 at 08.30
To: user@guacamole.apache.org 
Subject: Re: Major bug message log in guacd 1.5.4
Hi,

I'm running two different systems: one with Guacamole client & server 1.5.4 and 
another with guacamole 1.4.0. Both systems are running the same freeRDP version 
2.4.1.

The guacamole 1.4.0 system has been running fine for a very long time and still 
is.

The 1.5.4 system has been running fine for a few weeks now, but suddenly I'm 
hitting the FreeRDP "major bug" issue, and the RDP client connection fails.

Major bug, unable to allocate a TLS value for currentThread
freerdp_tcp_is_hostname_resolvable:freerdp_set_last_error_ex resetting error 
state
freerdp_tcp_connect:freerdp_set_last_error_ex resetting error state
Protocol Security Negotiation Failure
rdp_client_connect:freerdp_set_last_error_ex 
ERRCONNECT_SECURITY_NEGO_CONNECT_FAILED [0x0002000C]
Error: protocol security negotiation or connectionfailure
SVC "rdpdr" disconnected.
SVC "rdpsnd" disconnected.
RDP server closed/refused connection: Security negotiation failed (wrong 
security type?)

This is the same target RDP server, same FreeRDP version, but different 
guacamole software. Even Apache Tomcat is almost the same (8.5.72 vs. 8.5.73).

I then tried several target RDP hosts, and they all gave the same error.
So then I tried both telnet and ssh hosts. They failed too, but with no errors 
-- just a timeout.
Finally, the *only* thing I tried was to restart the guacd daemon on the 1.5.4 
system, and everything started working again (RDP, telnet, ssh).

Unfortunately, now that I've restarted guacd 1.5.4 I cannot reproduce the 
problem for further research (until maybe it starts failing again which can 
take quite a while).

Vieri


-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Vieri]

Hi,

I'm running two different systems: one with Guacamole client & server 1.5.4 and 
another with guacamole 1.4.0. Both systems are running the same freeRDP version 
2.4.1.

The guacamole 1.4.0 system has been running fine for a very long time and still 
is.

The 1.5.4 system has been running fine for a few weeks now, but suddenly I'm 
hitting the FreeRDP "major bug" issue, and the RDP client connection fails.

Major bug, unable to allocate a TLS value for currentThread
freerdp_tcp_is_hostname_resolvable:freerdp_set_last_error_ex resetting error 
state
freerdp_tcp_connect:freerdp_set_last_error_ex resetting error state
Protocol Security Negotiation Failure
rdp_client_connect:freerdp_set_last_error_ex 
ERRCONNECT_SECURITY_NEGO_CONNECT_FAILED [0x0002000C]
Error: protocol security negotiation or connectionfailure
SVC "rdpdr" disconnected.
SVC "rdpsnd" disconnected.
RDP server closed/refused connection: Security negotiation failed (wrong 
security type?)

This is the same target RDP server, same FreeRDP version, but different 
guacamole software. Even Apache Tomcat is almost the same (8.5.72 vs. 8.5.73).

I then tried several target RDP hosts, and they all gave the same error.
So then I tried both telnet and ssh hosts. They failed too, but with no errors 
-- just a timeout.
Finally, the *only* thing I tried was to restart the guacd daemon on the 1.5.4 
system, and everything started working again (RDP, telnet, ssh).

Unfortunately, now that I've restarted guacd 1.5.4 I cannot reproduce the 
problem for further research (until maybe it starts failing again which can 
take quite a while).

Vieri


-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org

Re: Major bug message log in guacd 1.5.4

[Michael Jumper]

On 12/13/2023 4:19 AM, Jorge López Díaz wrote:

Hi,

We updated our guacd Docker image to v1.5.4 and after hours working as 
expected, connections stared to failed and it was not possible to 
establish new connections. In the container logs we saw this message:


/Major bug, unable to allocate a TLS value for currentThread/

After rolling back to v1.5.3 everything is working as expected.

Do you have any issue like this?



I have not seen this before, but the log message is from FreeRDP:

https://github.com/FreeRDP/FreeRDP/blob/53cb5f8852258c8a4bb18715d34a51a31080e620/winpr/libwinpr/thread/thread.c#L439

If you can reproduce the issue reliably, given that the message itself 
suspects a bug, I'd recommend reporting the issue to FreeRDP with steps 
to reproduce:


https://github.com/FreeRDP/FreeRDP/issues

If this turns out to be a bug in a specific version of FreeRDP, it is 
possible to manually rebuild the Docker image with an older version of 
FreeRDP using the "WITH_FREERDP" build argument:


https://github.com/apache/guacamole-server/blob/f0ab6655445d72ef92c7f33683817b9cea6ef1e8/Dockerfile#L65

- Mike

-
To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org
For additional commands, e-mail: user-h...@guacamole.apache.org