Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
Thanks Mike. I will look into it. On Sat, Jul 25, 2020 at 11:00 PM Mike Jumper wrote: > On Sat, Jul 25, 2020, 05:03 Amarjeet Singh wrote: > >> ... >> >>> you've not made any modifications/customizations to the code? >> >> >> I have done few modifications in the logs. >> > > Please retry against an unmodified build of guacamole-server from the > latest release. > > https://guacamole.apache.org/faq/#test-against-latest-version > > - Mike > >
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
On Sat, Jul 25, 2020, 05:03 Amarjeet Singh wrote: > ... > >> you've not made any modifications/customizations to the code? > > > I have done few modifications in the logs. > Please retry against an unmodified build of guacamole-server from the latest release. https://guacamole.apache.org/faq/#test-against-latest-version - Mike
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
Hi Nick, Thanks for looking into it. > I'm not entirely sure what those kernel messages mean. If you look in > /var/log/messages and locate one of those lines that you posted above, are > there any hints around those messages? Perhaps other lines that do not > contain the PID, which would have been filtered out with your "grep" > command? I have looked into the /var/log/messages file around the above process ID *23724*. These logs are continuous. No other logs are there. Jul 24 17:57:17 bghjrfed guacd: guacd[20578]: INFO: Sending display channels > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[22497]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22497]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[22842]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22842]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[22584]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22584]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[22693]: User "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd: guacd[22693]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:18 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:18 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:18 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:19 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:19 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:19 bghjrfed guacd[22545]: User "raj.joshi" is not responding. > Jul 24 17:57:19 bghjrfed guacd: guacd[22545]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:19 bghjrfed guacd[20578]: Sending display channels Jul 24 17:57:41 bghjrfed guacd: guacd[23688]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[23486]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23486]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[23372]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23372]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[23724]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23724]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[23533]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[23533]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[24020]: User "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd: guacd[24020]: ERROR:User > "raj.joshi" is not responding. > Jul 24 17:57:42 bghjrfed guacd[20578]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[20578]: INFO: Sending display > channels > Jul 24 17:57:42 bghjrfed guacd[5294]: Sending display channels > Jul 24 17:57:42 bghjrfed guacd: guacd[5294]: INFO: Sending display > channels > Jul 24
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
On Sat, Jul 25, 2020 at 4:38 AM Amarjeet Singh wrote: > There are kernel logs after this USER IS NOT RESPONDING log. > > I do not have any idea what it means ? > > The user is not responding message means that guacd does has not received any traffic from the user in a certain amount of time. What's supposed to happen after that is that guacd will close the connection and then the child guacd process should terminate, which does not appear to be happening. I'm not entirely sure what those kernel messages mean. If you look in /var/log/messages and locate one of those lines that you posted above, are there any hints around those messages? Perhaps other lines that do not contain the PID, which would have been filtered out with your "grep" command? > It is on CENTOS 7. I will update the kernel version as well. > > What version of guacd are you running? I looked back in the message history and did not see that noted anywhere, though I so see libguac.so.17 which looks like the 1.2.0 version. And I assume this is the "stock" guacd (from the Guacamole website or Git repos) - you've not made any modifications/customizations to the code? Your original stack trace mentioned "accops-server", and I'm curious why that would show up in the stack track for guacd? I wonder if something else installed on the system has inserted itself into the dynamic loading order, and the libraries provided by that are causing problems?? -Nick
Re: More than 400 PROCESS is in RECMSG for two days [ RAM CONSUMPTION IS HIGH ]
[root@x fd]# grep "23724" /var/log/messages Jul 24 17:57:42 xxx guacd[23724]: User "raj.joshi" is not responding. Jul 24 17:57:42 xxx guacd: guacd[23724]: ERROR:User "raj.joshi" is not responding. Jul 25 07:10:42 xxx kernel: [23724] 0 23724 6011219201902748 3887 0 guacd Jul 25 07:10:50 xxx kernel: [23724] 0 23724 6011219201882748 3889 0 guacd Jul 25 07:19:47 xxx kernel: [23724] 0 23724 6011219201862748 3891 0 guacd Jul 25 07:29:31 xxx kernel: [23724] 0 23724 6011219201832748 3894 0 guacd Jul 25 10:42:47 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 10:42:54 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 10:58:13 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 10:58:22 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 11:09:32 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd Jul 25 11:09:33 xxx kernel: [23724] 0 23724 6011219201632748 3914 0 guacd [root@ xxx fd]# grep "23233" /var/log/messages On Sat, Jul 25, 2020 at 10:13 AM Mike Jumper wrote: > On Fri, Jul 24, 2020 at 5:24 PM Amarjeet Singh > wrote: > >> Please let me know if you need any logs or any other trace. >> > > What do the guacd logs look like for the processes that are blocked and in > CLOSE_WAIT? > > - Mike > >
