Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
Thanks Mike. I will look into it. On Sat, Jul 25, 2020 at 11:00 PM Mike Jumper wrote: > On Sat, Jul 25, 2020, 05:03 Amarjeet Singh wrote: > >> ... >> >>> you've not made any modifications/customizations to the code? >> >> >> I have done few modifications in the logs. >> > > Please retry against an unmodified build of guacamole-server from the > latest release. > > https://guacamole.apache.org/faq/#test-against-latest-version > > - Mike > >
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
On Sat, Jul 25, 2020, 05:03 Amarjeet Singh wrote: > ... > >> you've not made any modifications/customizations to the code? > > > I have done few modifications in the logs. > Please retry against an unmodified build of guacamole-server from the latest release. https://guacamole.apache.org/faq/#test-against-latest-version - Mike
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
Hi Nick, Thanks for looking into it. > I'm not entirely sure what those kernel messages mean. If you look in > /var/log/messages and locate one of those lines that you posted above, are > there any hints around those messages? Perhaps other lines that do not > contain the PID, which would have been filtered out with your "grep" > command? I have looked into the /var/log/messages file around the above process ID *23724*. These logs are continuous. No other logs are there. Jul 24 17:57:17 bghjrfed guacd: guacd[20578]: INFO: Sending display channels > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[22497]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22497]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[22842]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22842]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[22584]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22584]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[22693]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22693]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:19 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:19 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:19 bghjrfed guacd[22545]: User "raj.joshi" is not responding. > Jul 24 17:57:19 bghjrfed guacd: guacd[22545]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:19 bghjrfed guacd[20578]: Sending display channels Jul 24 17:57:41 bghjrfed guacd: guacd[23688]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[23486]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23486]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[23372]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23372]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[23724]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23724]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[23533]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23533]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[24020]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[24020]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
On Sat, Jul 25, 2020 at 4:38 AM Amarjeet Singh wrote: > There are kernel logs after this USER IS NOT RESPONDING log. > > I do not have any idea what it means ? > > The user is not responding message means that guacd does has not received any traffic from the user in a certain amount of time. What's supposed to happen after that is that guacd will close the connection and then the child guacd process should terminate, which does not appear to be happening. I'm not entirely sure what those kernel messages mean. If you look in /var/log/messages and locate one of those lines that you posted above, are there any hints around those messages? Perhaps other lines that do not contain the PID, which would have been filtered out with your "grep" command? > It is on CENTOS 7. I will update the kernel version as well. > > What version of guacd are you running? I looked back in the message history and did not see that noted anywhere, though I so see libguac.so.17 which looks like the 1.2.0 version. And I assume this is the "stock" guacd (from the Guacamole website or Git repos) - you've not made any modifications/customizations to the code? Your original stack trace mentioned "accops-server", and I'm curious why that would show up in the stack track for guacd? I wonder if something else installed on the system has inserted itself into the dynamic loading order, and the libraries provided by that are causing problems?? -Nick
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
[root@x fd]# grep "23724" /var/log/messages Jul 24 17:57:42 xxx guacd[23724]: User "raj.joshi" is not responding. Jul 24 17:57:42 xxx guacd: guacd[23724]: ERROR:User "raj.joshi" is not responding. Jul 25 07:10:42 xxx kernel: [23724] 0 23724 6011219201902748 3887 0 guacd Jul 25 07:10:50 xxx kernel: [23724] 0 23724 6011219201882748 3889 0 guacd Jul 25 07:19:47 xxx kernel: [23724] 0 23724 6011219201862748 3891 0 guacd Jul 25 07:29:31 xxx kernel: [23724] 0 23724 6011219201832748 3894 0 guacd Jul 25 10:42:47 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 10:42:54 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 10:58:13 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 10:58:22 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 11:09:32 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 11:09:33 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd [root@ xxx fd]# grep "23233" /var/log/messages On Sat, Jul 25, 2020 at 10:13 AM Mike Jumper wrote: > On Fri, Jul 24, 2020 at 5:24 PM Amarjeet Singh > wrote: > >> Please let me know if you need any logs or any other trace. >> > > What do the guacd logs look like for the processes that are blocked and in > CLOSE_WAIT? > > - Mike > >
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
On Fri, Jul 24, 2020 at 5:24 PM Amarjeet Singh wrote: > Please let me know if you need any logs or any other trace. > What do the guacd logs look like for the processes that are blocked and in CLOSE_WAIT? - Mike
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
On Fri, Jul 24, 2020 at 4:42 PM Mike Jumper wrote: > You > > On Fri, Jul 24, 2020 at 4:14 PM Amarjeet Singh > wrote: > (E ... not sure how the single word "You" snuck in here, but just in case: this is just a typo, not an expression of loathing). - Mike
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
On Fri, Jul 24, 2020 at 4:36 PM Amarjeet Singh wrote: > One suggestion would be to make *write *non blocking. > No, there is nothing inherently wrong with leveraging blocking I/O, and no evidence thus far that using blocking I/O is causing any problem. Any other suggestions? > Assuming there is a bug, that bug needs to first be identified before it would make sense to suggest a fix. This would mean understanding what is causing this behavior (what is preventing normal cleanup from occurring). - Mike
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
You On Fri, Jul 24, 2020 at 4:14 PM Amarjeet Singh wrote: > Hi Team, > > > More analysis on this :- > > There are threads which are in deadlock: > > THREAD *25377 *is waiting for the mutex lock whereas THREAD *25376 *is > stuck in a *write *system call Because of which there are connections > which are in CLOSE_WAIT. > What about this leads you to believe there is deadlock? There is nothing inherently abnormal about a thread being in the middle of a write() when the process is suspended and inspected by gdb. Are you certain that this write() is blocked? If so, are you certain that the write() is blocked by a resource that will not be released until that write() succeeds (deadlock)? Your backtrace shows two instructions being written simultaneously (one "sync" and another "error"), with the socket lock ensuring those instructions do not overlap. This is also normal behavior and not deadlock; it is simply two threads sharing access to the same guac_socket. If you are seeing a great many sockets stuck in CLOSE_WAIT, that does sound like there may be a case where file descriptors are not being properly released, but the information here so far just looks like snapshots of a normal, properly-functioning guac process. If you can trace down which file descriptors specifically are being held, and by which process, then that coupled with logs might reveal what conditions are necessary for this to occur. So far, I think you're just snapshotting the processes that are functioning normally. - Mike
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
One suggestion would be to make *write *non blocking. Any other suggestions? On Sat, Jul 25, 2020 at 4:43 AM Amarjeet Singh wrote: > Hi Team, > > > More analysis on this :- > > There are threads which are in deadlock: > > THREAD *25377 *is waiting for the mutex lock whereas THREAD *25376 *is > stuck in a *write *system call Because of which there are connections > which are in CLOSE_WAIT. > guacd is not able to free the resources as well. > > (gdb) info threads > Id Target Id Frame > 7Thread 0x7fb3431ce700 (LWP 25374) "guacd" 0x7fb7ad8fcf57 in > pthread_join () from /lib64/libpthread.so.0 > * 6Thread 0x7fb441bcb700 (LWP 25376) "guacd" 0x7fb7ad9026ad in write > () from /lib64/libpthread.so.0 > 5Thread 0x7fb4423cc700 (LWP 25377) "guacd" 0x7fb7ad90242d in > __lll_lock_wait () from /lib64/libpthread.so.0 > 4Thread 0x7fb3439cf700 (LWP 25395) "guacd" 0x7fb7ac1ed7a3 in > select () from /lib64/libc.so.6 > 3Thread 0x7fb3441d0700 (LWP 25396) "guacd" 0x7fb7ac1ed7a3 in > select () from /lib64/libc.so.6 > 2Thread 0x7fb3449d1700 (LWP 25397) "guacd" 0x7fb7ac1ed7a3 in > select () from /lib64/libc.so.6 > 1Thread 0x7fb3429cd700 (LWP 23724) "guacd" 0x7fb7ad902b5d in > recvmsg () from /lib64/libpthread.so.0 > (gdb) thr 5 > [Switching to thread 5 (Thread 0x7fb4423cc700 (LWP 25377))] > #0 0x7fb7ad90242d in __lll_lock_wait () from /lib64/libpthread.so.0 > (gdb) bt > #0 0x7fb7ad90242d in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x7fb7ad8fddcb in _L_lock_812 () from /lib64/libpthread.so.0 > #2 0x7fb7ad8fdc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x7fb7ae4c5345 in guac_socket_fd_write_handler () from > /lib64/libguac.so.17 > #4 0x7fb7ae4c4733 in __guac_socket_write () from /lib64/libguac.so.17 > #5 0x7fb7ae4c4770 in guac_socket_write () from /lib64/libguac.so.17 > #6 0x7fb7ae4c4a9a in guac_socket_write_string () from > /lib64/libguac.so.17 > #7 0x7fb7ae4c2365 in guac_protocol_send_error () from > /lib64/libguac.so.17 > #8 0x7fb7ae4c63cf in vguac_user_abort () from /lib64/libguac.so.17 > #9 0x7fb7ae4c6495 in guac_user_abort () from /lib64/libguac.so.17 > #10 0x7fb7ae4c7aa8 in guac_user_input_thread () from > /lib64/libguac.so.17 > #11 0x7fb7ad8fbe25 in start_thread () from /lib64/libpthread.so.0 > #12 0x7fb7ac1f634d in clone () from /lib64/libc.so.6 > > *MUTEX IS OWNED BY 25376* > >> 2 0x7fb7ad8fdc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 >> (gdb) info reg >> rax0xfe00 -512 >> rbx0x0 0 >> rcx0x -1 >> rdx0x0 0 >> rsi0x0 0 >> rdi0x7fb7a001dc30 140426640219184 >> rbp0x7fb4423cba00 0x7fb4423cba00 >> rsp0x7fb4423cb9c8 0x7fb4423cb9c8 >> r8 0x7fb7a001dc30 140426640219184 >> r9 0x141d54 1318228 >> r100x2 2 >> r110x202514 >> r120x0 0 >> r130x7fb4423cc9c0 140412182120896 >> r140x7fb4423cc700 140412182120192 >> r150x2a 42 >> rip0x7fb7ad8fdc98 0x7fb7ad8fdc98 >> eflags 0x202[ IF ] >> cs 0x33 51 >> ss 0x2b 43 >> ds 0x0 0 >> es 0x0 0 >> fs 0x0 0 >> gs 0x0 0 >> (gdb) print *((int*)(0x7fb7a001dc30)+2) >> $6 = *25376* > > > *STRACE of the THREAD is as follows : -* > > strace -p 25376 >> Process 25376 attached >> write(4, "4.sync,10.1318124283;", 21 > > > > Can I file a bug in JIRA ? > > Any suggestions how to fix the above ? > > *NOTE *: This happens intermittently. > > Thanks and Regards, > Amarjeet Singh > > > > On Fri, Jul 17, 2020 at 8:43 AM Amarjeet Singh > wrote: > >> Hi Team, >> >> *GUACD *is consuming 100% of RAM. On analysis I have found that there >> are many process which are not in any state [ CLOSE_WAIT, ESTABLISHED etc ] >> but they are in >> recvmsg waiting for the fd. This process is there for more than 2 days. >> Below is the backtrace of the process. >> >> Reading symbols from /usr/lib64/freerdp/disp.so...Reading symbols from >>> /usr/lib64/freerdp/disp.so...(no debugging symbols found)...done. >>> (no debugging symbols found)...done. >>> Loaded symbols for /usr/lib64/freerdp/disp.so >>> 0x7fa764807b5d in recvmsg () from /lib64/libpthread.so.0 >>> Missing separate debuginfos, use: debuginfo-install >>> accops-server-8.0.0-2.x86_64 >>> (gdb) bt >>> #0 0x7fa764807b5d in recvmsg () from /lib64/libpthread.so.0 >>> #1 0x00404a64 in guacd_recv_fd () >>> #2 0x00404ed9 in guacd_exec_proc () >>> #3 0x00405297 in guacd_create_proc () >>> #4 0x0040399f in guacd_route_connection () >>> #5 0x00403ba7 in guacd_connection_thread () >>> #6 0x7fa764800e25 in start_thread () from
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
Hi Team, More analysis on this :- There are threads which are in deadlock: THREAD *25377 *is waiting for the mutex lock whereas THREAD *25376 *is stuck in a *write *system call Because of which there are connections which are in CLOSE_WAIT. guacd is not able to free the resources as well. (gdb) info threads Id Target Id Frame 7Thread 0x7fb3431ce700 (LWP 25374) "guacd" 0x7fb7ad8fcf57 in pthread_join () from /lib64/libpthread.so.0 * 6Thread 0x7fb441bcb700 (LWP 25376) "guacd" 0x7fb7ad9026ad in write () from /lib64/libpthread.so.0 5Thread 0x7fb4423cc700 (LWP 25377) "guacd" 0x7fb7ad90242d in __lll_lock_wait () from /lib64/libpthread.so.0 4Thread 0x7fb3439cf700 (LWP 25395) "guacd" 0x7fb7ac1ed7a3 in select () from /lib64/libc.so.6 3Thread 0x7fb3441d0700 (LWP 25396) "guacd" 0x7fb7ac1ed7a3 in select () from /lib64/libc.so.6 2Thread 0x7fb3449d1700 (LWP 25397) "guacd" 0x7fb7ac1ed7a3 in select () from /lib64/libc.so.6 1Thread 0x7fb3429cd700 (LWP 23724) "guacd" 0x7fb7ad902b5d in recvmsg () from /lib64/libpthread.so.0 (gdb) thr 5 [Switching to thread 5 (Thread 0x7fb4423cc700 (LWP 25377))] #0 0x7fb7ad90242d in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) bt #0 0x7fb7ad90242d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x7fb7ad8fddcb in _L_lock_812 () from /lib64/libpthread.so.0 #2 0x7fb7ad8fdc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x7fb7ae4c5345 in guac_socket_fd_write_handler () from /lib64/libguac.so.17 #4 0x7fb7ae4c4733 in __guac_socket_write () from /lib64/libguac.so.17 #5 0x7fb7ae4c4770 in guac_socket_write () from /lib64/libguac.so.17 #6 0x7fb7ae4c4a9a in guac_socket_write_string () from /lib64/libguac.so.17 #7 0x7fb7ae4c2365 in guac_protocol_send_error () from /lib64/libguac.so.17 #8 0x7fb7ae4c63cf in vguac_user_abort () from /lib64/libguac.so.17 #9 0x7fb7ae4c6495 in guac_user_abort () from /lib64/libguac.so.17 #10 0x7fb7ae4c7aa8 in guac_user_input_thread () from /lib64/libguac.so.17 #11 0x7fb7ad8fbe25 in start_thread () from /lib64/libpthread.so.0 #12 0x7fb7ac1f634d in clone () from /lib64/libc.so.6 *MUTEX IS OWNED BY 25376* > 2 0x7fb7ad8fdc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 > (gdb) info reg > rax0xfe00 -512 > rbx0x0 0 > rcx0x -1 > rdx0x0 0 > rsi0x0 0 > rdi0x7fb7a001dc30 140426640219184 > rbp0x7fb4423cba00 0x7fb4423cba00 > rsp0x7fb4423cb9c8 0x7fb4423cb9c8 > r8 0x7fb7a001dc30 140426640219184 > r9 0x141d54 1318228 > r100x2 2 > r110x202514 > r120x0 0 > r130x7fb4423cc9c0 140412182120896 > r140x7fb4423cc700 140412182120192 > r150x2a 42 > rip0x7fb7ad8fdc98 0x7fb7ad8fdc98 > eflags 0x202[ IF ] > cs 0x33 51 > ss 0x2b 43 > ds 0x0 0 > es 0x0 0 > fs 0x0 0 > gs 0x0 0 > (gdb) print *((int*)(0x7fb7a001dc30)+2) > $6 = *25376* *STRACE of the THREAD is as follows : -* strace -p 25376 > Process 25376 attached > write(4, "4.sync,10.1318124283;", 21 Can I file a bug in JIRA ? Any suggestions how to fix the above ? *NOTE *: This happens intermittently. Thanks and Regards, Amarjeet Singh On Fri, Jul 17, 2020 at 8:43 AM Amarjeet Singh wrote: > Hi Team, > > *GUACD *is consuming 100% of RAM. On analysis I have found that there are > many process which are not in any state [ CLOSE_WAIT, ESTABLISHED etc ] but > they are in > recvmsg waiting for the fd. This process is there for more than 2 days. > Below is the backtrace of the process. > > Reading symbols from /usr/lib64/freerdp/disp.so...Reading symbols from >> /usr/lib64/freerdp/disp.so...(no debugging symbols found)...done. >> (no debugging symbols found)...done. >> Loaded symbols for /usr/lib64/freerdp/disp.so >> 0x7fa764807b5d in recvmsg () from /lib64/libpthread.so.0 >> Missing separate debuginfos, use: debuginfo-install >> accops-server-8.0.0-2.x86_64 >> (gdb) bt >> #0 0x7fa764807b5d in recvmsg () from /lib64/libpthread.so.0 >> #1 0x00404a64 in guacd_recv_fd () >> #2 0x00404ed9 in guacd_exec_proc () >> #3 0x00405297 in guacd_create_proc () >> #4 0x0040399f in guacd_route_connection () >> #5 0x00403ba7 in guacd_connection_thread () >> #6 0x7fa764800e25 in start_thread () from /lib64/libpthread.so.0 >> #7 0x7fa7630fb34d in clone () from /lib64/libc.so.6 > > > Please help me to understand what is going wrong here ? This is not > happening for every connections. Is there any way we can fix this ? > There are many connections which are in CLOSE_WAIT ( parent process id ) > also. They are there
More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
Hi Team, *GUACD *is consuming 100% of RAM. On analysis I have found that there are many process which are not in any state [ CLOSE_WAIT, ESTABLISHED etc ] but they are in recvmsg waiting for the fd. This process is there for more than 2 days. Below is the backtrace of the process. Reading symbols from /usr/lib64/freerdp/disp.so...Reading symbols from > /usr/lib64/freerdp/disp.so...(no debugging symbols found)...done. > (no debugging symbols found)...done. > Loaded symbols for /usr/lib64/freerdp/disp.so > 0x7fa764807b5d in recvmsg () from /lib64/libpthread.so.0 > Missing separate debuginfos, use: debuginfo-install > accops-server-8.0.0-2.x86_64 > (gdb) bt > #0 0x7fa764807b5d in recvmsg () from /lib64/libpthread.so.0 > #1 0x00404a64 in guacd_recv_fd () > #2 0x00404ed9 in guacd_exec_proc () > #3 0x00405297 in guacd_create_proc () > #4 0x0040399f in guacd_route_connection () > #5 0x00403ba7 in guacd_connection_thread () > #6 0x7fa764800e25 in start_thread () from /lib64/libpthread.so.0 > #7 0x7fa7630fb34d in clone () from /lib64/libc.so.6 Please help me to understand what is going wrong here ? This is not happening for every connections. Is there any way we can fix this ? There are many connections which are in CLOSE_WAIT ( parent process id ) also. They are there for many days. Amarjeet Singh
