Re: [X2Go-User] Long-running x2goagent, high memory usage
Great to hear that! This closes one point on my list ;-) Uli On Tue, Jan 28, 2020 at 9:37 AM Johannes Töger wrote: > > Hi, > > Yesterday I found some time to upgrade the packages. I did the following: > > 1. Downloaded all packages, (.22, amd64, 18.04 versions) from > http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/ > 2. sudo dpkg -i *deb to install everything > 3. Asked users to end sessions and reconnect. > > After this, the x2go memory use is <= 350 MB for each user and no longer > growing (last 18 hours). So problem fixed from my point of view! Thanks for > all your help. > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology > Lund University, Sweden > > -Original Message- > From: Ulrich Sibiller > Sent: Friday, January 24, 2020 09:44 > To: Johannes Töger > Cc: Bernd Schubert ; x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > Well, from MPOV you do not need a shutdown. Just update the nx packages. The > next new connection will use the new nxagent. > > Let me now when you want to compile an own version, so that I can provide > some instructions for sanitizer integration. (I need to figure that out again > since I mainly use valgrind). > > Uli > > On Fri, Jan 24, 2020 at 8:35 AM Johannes Töger > wrote: > > > > Yeah, I could absolutely compile my own version. I think it might be better > > to try with the .deb packages you linked first though. > > > > I need to ask my users for a window where I can shut down everything first, > > so might take a few days before I can organize this. > > > > — > > Johannes Töger > > > > Associate Senior Lecturer > > Cardiac MR Group > > Department of Clinical Sciences Lund, Clinical Physiology Lund > > University, Sweden > > > > -Original Message- > > From: Ulrich Sibiller > > Sent: Thursday, January 23, 2020 19:43 > > To: Johannes Töger > > Cc: Bernd Schubert ; > > x2go-user@lists.x2go.org > > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > > > On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger > > wrote: > > > > > > Thanks to you both for getting involved in this. > > > > > > I've been swamped with other things, so not much to report. I did manage > > > to plot the x2goagent memory use over time though. Maybe this can help. > > > > > > Link to plot: > > > https://imgur.com/a/lDqZ2zd > > > > > > As you see, several users have a constant memory use. However, 3 of them > > > have a constantly increasing memory use. So not isolated incidents that > > > leak, but rather a continuous drip. > > > > OK, well, do you think you could compile an own version of x2go? > > > > Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Hi, Yesterday I found some time to upgrade the packages. I did the following: 1. Downloaded all packages, (.22, amd64, 18.04 versions) from http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/ 2. sudo dpkg -i *deb to install everything 3. Asked users to end sessions and reconnect. After this, the x2go memory use is <= 350 MB for each user and no longer growing (last 18 hours). So problem fixed from my point of view! Thanks for all your help. — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: Ulrich Sibiller Sent: Friday, January 24, 2020 09:44 To: Johannes Töger Cc: Bernd Schubert ; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage Well, from MPOV you do not need a shutdown. Just update the nx packages. The next new connection will use the new nxagent. Let me now when you want to compile an own version, so that I can provide some instructions for sanitizer integration. (I need to figure that out again since I mainly use valgrind). Uli On Fri, Jan 24, 2020 at 8:35 AM Johannes Töger wrote: > > Yeah, I could absolutely compile my own version. I think it might be better > to try with the .deb packages you linked first though. > > I need to ask my users for a window where I can shut down everything first, > so might take a few days before I can organize this. > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology Lund > University, Sweden > > -Original Message- > From: Ulrich Sibiller > Sent: Thursday, January 23, 2020 19:43 > To: Johannes Töger > Cc: Bernd Schubert ; > x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger > wrote: > > > > Thanks to you both for getting involved in this. > > > > I've been swamped with other things, so not much to report. I did manage to > > plot the x2goagent memory use over time though. Maybe this can help. > > > > Link to plot: > > https://imgur.com/a/lDqZ2zd > > > > As you see, several users have a constant memory use. However, 3 of them > > have a constantly increasing memory use. So not isolated incidents that > > leak, but rather a continuous drip. > > OK, well, do you think you could compile an own version of x2go? > > Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Well, from MPOV you do not need a shutdown. Just update the nx packages. The next new connection will use the new nxagent. Let me now when you want to compile an own version, so that I can provide some instructions for sanitizer integration. (I need to figure that out again since I mainly use valgrind). Uli On Fri, Jan 24, 2020 at 8:35 AM Johannes Töger wrote: > > Yeah, I could absolutely compile my own version. I think it might be better > to try with the .deb packages you linked first though. > > I need to ask my users for a window where I can shut down everything first, > so might take a few days before I can organize this. > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology > Lund University, Sweden > > -Original Message- > From: Ulrich Sibiller > Sent: Thursday, January 23, 2020 19:43 > To: Johannes Töger > Cc: Bernd Schubert ; x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger > wrote: > > > > Thanks to you both for getting involved in this. > > > > I've been swamped with other things, so not much to report. I did manage to > > plot the x2goagent memory use over time though. Maybe this can help. > > > > Link to plot: > > https://imgur.com/a/lDqZ2zd > > > > As you see, several users have a constant memory use. However, 3 of them > > have a constantly increasing memory use. So not isolated incidents that > > leak, but rather a continuous drip. > > OK, well, do you think you could compile an own version of x2go? > > Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Yeah, I could absolutely compile my own version. I think it might be better to try with the .deb packages you linked first though. I need to ask my users for a window where I can shut down everything first, so might take a few days before I can organize this. — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: Ulrich Sibiller Sent: Thursday, January 23, 2020 19:43 To: Johannes Töger Cc: Bernd Schubert ; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger wrote: > > Thanks to you both for getting involved in this. > > I've been swamped with other things, so not much to report. I did manage to > plot the x2goagent memory use over time though. Maybe this can help. > > Link to plot: > https://imgur.com/a/lDqZ2zd > > As you see, several users have a constant memory use. However, 3 of them have > a constantly increasing memory use. So not isolated incidents that leak, but > rather a continuous drip. OK, well, do you think you could compile an own version of x2go? Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger wrote: > > Thanks to you both for getting involved in this. > > I've been swamped with other things, so not much to report. I did manage to > plot the x2goagent memory use over time though. Maybe this can help. > > Link to plot: > https://imgur.com/a/lDqZ2zd > > As you see, several users have a constant memory use. However, 3 of them have > a constantly increasing memory use. So not isolated incidents that leak, but > rather a continuous drip. OK, well, do you think you could compile an own version of x2go? Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Thanks to you both for getting involved in this. I've been swamped with other things, so not much to report. I did manage to plot the x2goagent memory use over time though. Maybe this can help. Link to plot: https://imgur.com/a/lDqZ2zd As you see, several users have a constant memory use. However, 3 of them have a constantly increasing memory use. So not isolated incidents that leak, but rather a continuous drip. — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: x2go-user On Behalf Of Bernd Schubert Sent: Thursday, January 23, 2020 16:11 To: Ulrich Sibiller Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage On 1/23/20 3:41 PM, Ulrich Sibiller wrote: > Ok, that's fine, too! I am curious about the results! > > Uli > > PS: Well, Johannes started the thread and I tried to help him, I am a > bit confused that now you (Bernd) kick in. Are you a colleague of > Johannes or are you also seeing that problem? Ah sorry, no, I'm just reading the thread. I had similar issues in the past with x2go, but right now long running sessions work pretty well. I'm also coding all the time and the compiler sanitizers are great help and work so much faster than valgrind - I just tried to point that out. Sorry for confusion. - Bernd ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
On 1/23/20 3:41 PM, Ulrich Sibiller wrote: > Ok, that's fine, too! I am curious about the results! > > Uli > > PS: Well, Johannes started the thread and I tried to help him, I am a > bit confused that now you (Bernd) kick in. Are you a colleague of > Johannes or are you also seeing that problem? Ah sorry, no, I'm just reading the thread. I had similar issues in the past with x2go, but right now long running sessions work pretty well. I'm also coding all the time and the compiler sanitizers are great help and work so much faster than valgrind - I just tried to point that out. Sorry for confusion. - Bernd ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Ok, that's fine, too! I am curious about the results! Uli PS: Well, Johannes started the thread and I tried to help him, I am a bit confused that now you (Bernd) kick in. Are you a colleague of Johannes or are you also seeing that problem? On Thu, Jan 23, 2020 at 3:23 PM Bernd Schubert wrote: > > > > On 1/23/20 2:52 PM, Ulrich Sibiller wrote: > > Just a little update: I have tried to integrate valgrind into the x2go > > setup but I failed. I'll try again... > > I guess I should try to compile x2go myself, but typically ASAN (which > then also runs LSAN) is much faster than valgrind. A bit tricky, though, > to get the reports for daemon processes. ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
On 1/23/20 2:52 PM, Ulrich Sibiller wrote: > Just a little update: I have tried to integrate valgrind into the x2go > setup but I failed. I'll try again... I guess I should try to compile x2go myself, but typically ASAN (which then also runs LSAN) is much faster than valgrind. A bit tricky, though, to get the reports for daemon processes. ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
On Thu, Jan 23, 2020 at 2:52 PM Ulrich Sibiller wrote: > > Just a little update: I have tried to integrate valgrind into the x2go > setup but I failed. I'll try again... > > In your initial bug report you showed some measurement where you > accumulated the memory per user. Can you verify that this amount is is > tied to x2goagent processes? Forget that, I had missed the grep.. Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Just a little update: I have tried to integrate valgrind into the x2go setup but I failed. I'll try again... In your initial bug report you showed some measurement where you accumulated the memory per user. Can you verify that this amount is is tied to x2goagent processes? Uli On Tue, Jan 21, 2020 at 11:43 AM Ulrich Sibiller wrote: > > On Tue, Jan 21, 2020 at 11:24 AM Johannes Töger > wrote: > > > > Great, I see in the blog that you did a lot of work on fixing memory leaks. > > I'll try to install the .22 version. > > > > But isn't it strange that I can't find or have an nx-libs package > > installed? What's going on there? Is the packaging for Ubuntu 18.04 LTS > > done in some other way? > > No, this is a mistake from my side. The project is called nx-libs and > produces a number of packages. You can replace the relevant x2go > packages by the arctica ones. > > Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
On Tue, Jan 21, 2020 at 11:24 AM Johannes Töger wrote: > > Great, I see in the blog that you did a lot of work on fixing memory leaks. > I'll try to install the .22 version. > > But isn't it strange that I can't find or have an nx-libs package installed? > What's going on there? Is the packaging for Ubuntu 18.04 LTS done in some > other way? No, this is a mistake from my side. The project is called nx-libs and produces a number of packages. You can replace the relevant x2go packages by the arctica ones. Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Great, I see in the blog that you did a lot of work on fixing memory leaks. I'll try to install the .22 version. But isn't it strange that I can't find or have an nx-libs package installed? What's going on there? Is the packaging for Ubuntu 18.04 LTS done in some other way? — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: Ulrich Sibiller Sent: Tuesday, January 21, 2020 10:39 To: Johannes Töger Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage On Tue, Jan 21, 2020 at 9:39 AM Johannes Töger wrote: > > OK, thanks. > > Sending the answers again, maybe they were cut by the mailing list before. I have now found it, don't know how I managed to miss it. Sorry. > Question: What version of x2goagent resp. nxagent are you using? dpkg -l > nx-libs? > > Answer: > dpkg -l nx-libs: no packages found <--- this seems strange? > x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent > 2:3.5.99.20-0~20190425 amd64 Ok, we are at 3.5.99.22 at the moment (see https://sunweavers.net/blog/node/97) and close to 3.5.99.23 although x2go is not having packages for that yet. So please try http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/ (do not use the current nightlies, they miss a patch which fixes regular crashes) Uli > > > Question: Can you please run xrestop within such a session and check if there > are numbers that are far away from those of a fresh session? > Answer: Nothing out of the ordinary there I think. No processes with high > memory usage. > > > Question: You probably cannot answer that due to your usage scenario, but > anyway: does this also happen when you do not disconnect/reconnect? > Answer: I don't know. I started logging the memory usage over time, so when > the problem arises next time I can correlate with log events and ask the > users what they have been doing at certain times etc. > > > Question: What are you running inside the session?fdasfafad > Answer: No real pattern here for good and bad sessions. Matlab, > Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning > programs etc. Maybe a browser here and there to look up stuff (Firefox). > > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology Lund > University, Sweden > > > -Original Message- > From: Ulrich Sibiller > Sent: Monday, January 20, 2020 15:45 > To: Johannes Töger > Cc: x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > Sorry, I forgot to check, but probably not today. But can you please answer > my questions in the meantime? > > Basically you must replace the x2go binary with a little wrapper that use > valgrind to start the original binary with the original arguments. > > Uli > > On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger > wrote: > > > > I have now set up a logging of the memory use over time. I hope this will > > give some data so we can understand the error. Hopefully I can correlate > > the memory use increase with user actions. > > > > Ulrich, were you able to figure out if x2go can be run with valgrind? > > > > — > > Johannes Töger > > > > Associate Senior Lecturer > > Cardiac MR Group > > Department of Clinical Sciences Lund, Clinical Physiology Lund > > University, Sweden > > > > -Original Message- > > From: x2go-user On Behalf Of > > Johannes Töger > > Sent: Friday, January 17, 2020 16:28 > > To: Ulrich Sibiller > > Cc: x2go-user@lists.x2go.org > > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > > > Thanks for quick reply. > > > > Answers to your questions below. I hope it gives some clue. Please let me > > know what else I can try. > > > > > > > > What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? > > > > dpkg -l nx-libs: no packages found <--- this seems strange? > > > > x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 > > nxagent 2:3.5.99.20-0~20190425 amd64 > > > > Can you please run xrestop within such a session and check if there are > > numbers that are far away from those of a fresh session? > > > > Nothing out of the ordinary there I think. No processes with high > > memory usage. > > > > You probably cannot answer that due to your usage scenario, but > > anyway: does this also happen when you do not disconnect/r
Re: [X2Go-User] Long-running x2goagent, high memory usage
On Tue, Jan 21, 2020 at 9:39 AM Johannes Töger wrote: > > OK, thanks. > > Sending the answers again, maybe they were cut by the mailing list before. I have now found it, don't know how I managed to miss it. Sorry. > Question: What version of x2goagent resp. nxagent are you using? dpkg -l > nx-libs? > > Answer: > dpkg -l nx-libs: no packages found <--- this seems strange? > x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 > nxagent 2:3.5.99.20-0~20190425 amd64 Ok, we are at 3.5.99.22 at the moment (see https://sunweavers.net/blog/node/97) and close to 3.5.99.23 although x2go is not having packages for that yet. So please try http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/ (do not use the current nightlies, they miss a patch which fixes regular crashes) Uli > > > Question: Can you please run xrestop within such a session and check if there > are numbers that are far away from those of a fresh session? > Answer: Nothing out of the ordinary there I think. No processes with high > memory usage. > > > Question: You probably cannot answer that due to your usage scenario, but > anyway: does this also happen when you do not disconnect/reconnect? > Answer: I don't know. I started logging the memory usage over time, so when > the problem arises next time I can correlate with log events and ask the > users what they have been doing at certain times etc. > > > Question: What are you running inside the session?fdasfafad > Answer: No real pattern here for good and bad sessions. Matlab, > Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning > programs etc. Maybe a browser here and there to look up stuff (Firefox). > > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology > Lund University, Sweden > > > -Original Message- > From: Ulrich Sibiller > Sent: Monday, January 20, 2020 15:45 > To: Johannes Töger > Cc: x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > Sorry, I forgot to check, but probably not today. But can you please answer > my questions in the meantime? > > Basically you must replace the x2go binary with a little wrapper that use > valgrind to start the original binary with the original arguments. > > Uli > > On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger > wrote: > > > > I have now set up a logging of the memory use over time. I hope this will > > give some data so we can understand the error. Hopefully I can correlate > > the memory use increase with user actions. > > > > Ulrich, were you able to figure out if x2go can be run with valgrind? > > > > — > > Johannes Töger > > > > Associate Senior Lecturer > > Cardiac MR Group > > Department of Clinical Sciences Lund, Clinical Physiology Lund > > University, Sweden > > > > -Original Message- > > From: x2go-user On Behalf Of > > Johannes Töger > > Sent: Friday, January 17, 2020 16:28 > > To: Ulrich Sibiller > > Cc: x2go-user@lists.x2go.org > > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > > > Thanks for quick reply. > > > > Answers to your questions below. I hope it gives some clue. Please let me > > know what else I can try. > > > > > > > > What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? > > > > dpkg -l nx-libs: no packages found <--- this seems strange? > > > > x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 > > nxagent 2:3.5.99.20-0~20190425 amd64 > > > > Can you please run xrestop within such a session and check if there are > > numbers that are far away from those of a fresh session? > > > > Nothing out of the ordinary there I think. No processes with high > > memory usage. > > > > You probably cannot answer that due to your usage scenario, but > > anyway: does this also happen when you do not disconnect/reconnect? > > > > I'll try to think of a way to test this. > > > > What are you running inside the session? > > > > No real pattern here for good and bad sessions. Matlab, > > Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning > > programs etc. > > > > — > > Johannes Töger > > > > Associate Senior Lecturer > > Cardiac MR Group > > Department of Clinical Sciences Lund, Clinical Physiology Lund > > University, Sweden > > > > -Original Message- > > From: Ulrich Sibiller >
Re: [X2Go-User] Long-running x2goagent, high memory usage
OK, thanks. Sending the answers again, maybe they were cut by the mailing list before. Question: What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? Answer: dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64 Question: Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? Answer: Nothing out of the ordinary there I think. No processes with high memory usage. Question: You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? Answer: I don't know. I started logging the memory usage over time, so when the problem arises next time I can correlate with log events and ask the users what they have been doing at certain times etc. Question: What are you running inside the session?fdasfafad Answer: No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc. Maybe a browser here and there to look up stuff (Firefox). — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: Ulrich Sibiller Sent: Monday, January 20, 2020 15:45 To: Johannes Töger Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage Sorry, I forgot to check, but probably not today. But can you please answer my questions in the meantime? Basically you must replace the x2go binary with a little wrapper that use valgrind to start the original binary with the original arguments. Uli On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger wrote: > > I have now set up a logging of the memory use over time. I hope this will > give some data so we can understand the error. Hopefully I can correlate the > memory use increase with user actions. > > Ulrich, were you able to figure out if x2go can be run with valgrind? > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology Lund > University, Sweden > > -Original Message- > From: x2go-user On Behalf Of > Johannes Töger > Sent: Friday, January 17, 2020 16:28 > To: Ulrich Sibiller > Cc: x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > Thanks for quick reply. > > Answers to your questions below. I hope it gives some clue. Please let me > know what else I can try. > > > > What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? > > dpkg -l nx-libs: no packages found <--- this seems strange? > > x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 > nxagent 2:3.5.99.20-0~20190425 amd64 > > Can you please run xrestop within such a session and check if there are > numbers that are far away from those of a fresh session? > > Nothing out of the ordinary there I think. No processes with high > memory usage. > > You probably cannot answer that due to your usage scenario, but > anyway: does this also happen when you do not disconnect/reconnect? > > I'll try to think of a way to test this. > > What are you running inside the session? > > No real pattern here for good and bad sessions. Matlab, > Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning > programs etc. > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology Lund > University, Sweden > > -Original Message- > From: Ulrich Sibiller > Sent: Friday, January 17, 2020 13:25 > To: Johannes Töger > Cc: x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger > wrote: > > We are using x2go for a common compute server at our department. We have > > users both on the local LAN and connections to the other side of Europe > > that work really well. Being able to close the client and have computations > > still running in a GUI environment is great. > > > > However, I think we have encountered a bug. Some (but not all) of the > > long-running x2goagent processes start taking a lot of memory, ~30 GB. This > > adds up quickly so that the other users can’t do their work. Below you can > > see how user3 and user7 have x2goagent processes with large memory use (RSS > > column). > > > > How can I go about debugging this? I have quite a bit of Linux experience, > > but I
Re: [X2Go-User] Long-running x2goagent, high memory usage
Sorry, I forgot to check, but probably not today. But can you please answer my questions in the meantime? Basically you must replace the x2go binary with a little wrapper that use valgrind to start the original binary with the original arguments. Uli On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger wrote: > > I have now set up a logging of the memory use over time. I hope this will > give some data so we can understand the error. Hopefully I can correlate the > memory use increase with user actions. > > Ulrich, were you able to figure out if x2go can be run with valgrind? > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology > Lund University, Sweden > > -Original Message- > From: x2go-user On Behalf Of Johannes Töger > Sent: Friday, January 17, 2020 16:28 > To: Ulrich Sibiller > Cc: x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > Thanks for quick reply. > > Answers to your questions below. I hope it gives some clue. Please let me > know what else I can try. > > > > What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? > > dpkg -l nx-libs: no packages found <--- this seems strange? > > x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 > nxagent 2:3.5.99.20-0~20190425 amd64 > > Can you please run xrestop within such a session and check if there are > numbers that are far away from those of a fresh session? > > Nothing out of the ordinary there I think. No processes with high > memory usage. > > You probably cannot answer that due to your usage scenario, but > anyway: does this also happen when you do not disconnect/reconnect? > > I'll try to think of a way to test this. > > What are you running inside the session? > > No real pattern here for good and bad sessions. Matlab, > Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning > programs etc. > > — > Johannes Töger > > Associate Senior Lecturer > Cardiac MR Group > Department of Clinical Sciences Lund, Clinical Physiology Lund University, > Sweden > > -Original Message----- > From: Ulrich Sibiller > Sent: Friday, January 17, 2020 13:25 > To: Johannes Töger > Cc: x2go-user@lists.x2go.org > Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage > > On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger > wrote: > > We are using x2go for a common compute server at our department. We have > > users both on the local LAN and connections to the other side of Europe > > that work really well. Being able to close the client and have computations > > still running in a GUI environment is great. > > > > However, I think we have encountered a bug. Some (but not all) of the > > long-running x2goagent processes start taking a lot of memory, ~30 GB. This > > adds up quickly so that the other users can’t do their work. Below you can > > see how user3 and user7 have x2goagent processes with large memory use (RSS > > column). > > > > How can I go about debugging this? I have quite a bit of Linux experience, > > but I don’t know where to start on this one since I don’t know much about > > how x2go works internally. > > > > Server is on Ubuntu Server 18.04.1 LTS, (x2goserver > > 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0. > > > > johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep > > x2goagent) > > What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? > > Can you please run xrestop within such a session and check if there are > numbers that are far away from those of a fresh session? > > You probably cannot answer that due to your usage scenario, but > anyway: does this also happen when you do not disconnect/reconnect? > > What are you running inside the session? > > You could run x2goagent with valgrind to identify memory leaks but I am > unsure if that is working on a stock installation ( I don't do the packaging > for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent > analysis/output). I'll check that this weekend. > > Uli > ___ > x2go-user mailing list > x2go-user@lists.x2go.org > https://lists.x2go.org/listinfo/x2go-user ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
I have now set up a logging of the memory use over time. I hope this will give some data so we can understand the error. Hopefully I can correlate the memory use increase with user actions. Ulrich, were you able to figure out if x2go can be run with valgrind? — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: x2go-user On Behalf Of Johannes Töger Sent: Friday, January 17, 2020 16:28 To: Ulrich Sibiller Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage Thanks for quick reply. Answers to your questions below. I hope it gives some clue. Please let me know what else I can try. What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64 Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? Nothing out of the ordinary there I think. No processes with high memory usage. You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? I'll try to think of a way to test this. What are you running inside the session? No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc. — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: Ulrich Sibiller Sent: Friday, January 17, 2020 13:25 To: Johannes Töger Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger wrote: > We are using x2go for a common compute server at our department. We have > users both on the local LAN and connections to the other side of Europe that > work really well. Being able to close the client and have computations still > running in a GUI environment is great. > > However, I think we have encountered a bug. Some (but not all) of the > long-running x2goagent processes start taking a lot of memory, ~30 GB. This > adds up quickly so that the other users can’t do their work. Below you can > see how user3 and user7 have x2goagent processes with large memory use (RSS > column). > > How can I go about debugging this? I have quite a bit of Linux experience, > but I don’t know where to start on this one since I don’t know much about how > x2go works internally. > > Server is on Ubuntu Server 18.04.1 LTS, (x2goserver > 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0. > > johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep > x2goagent) What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? What are you running inside the session? You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend. Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
Thanks for quick reply. Answers to your questions below. I hope it gives some clue. Please let me know what else I can try. What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64 Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? Nothing out of the ordinary there I think. No processes with high memory usage. You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? I'll try to think of a way to test this. What are you running inside the session? No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc. — Johannes Töger Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden -Original Message- From: Ulrich Sibiller Sent: Friday, January 17, 2020 13:25 To: Johannes Töger Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger wrote: > We are using x2go for a common compute server at our department. We have > users both on the local LAN and connections to the other side of Europe that > work really well. Being able to close the client and have computations still > running in a GUI environment is great. > > However, I think we have encountered a bug. Some (but not all) of the > long-running x2goagent processes start taking a lot of memory, ~30 GB. This > adds up quickly so that the other users can’t do their work. Below you can > see how user3 and user7 have x2goagent processes with large memory use (RSS > column). > > How can I go about debugging this? I have quite a bit of Linux experience, > but I don’t know where to start on this one since I don’t know much about how > x2go works internally. > > Server is on Ubuntu Server 18.04.1 LTS, (x2goserver > 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0. > > johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep > x2goagent) What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? What are you running inside the session? You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend. Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Re: [X2Go-User] Long-running x2goagent, high memory usage
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger wrote: > We are using x2go for a common compute server at our department. We have > users both on the local LAN and connections to the other side of Europe that > work really well. Being able to close the client and have computations still > running in a GUI environment is great. > > However, I think we have encountered a bug. Some (but not all) of the > long-running x2goagent processes start taking a lot of memory, ~30 GB. This > adds up quickly so that the other users can’t do their work. Below you can > see how user3 and user7 have x2goagent processes with large memory use (RSS > column). > > How can I go about debugging this? I have quite a bit of Linux experience, > but I don’t know where to start on this one since I don’t know much about how > x2go works internally. > > Server is on Ubuntu Server 18.04.1 LTS, (x2goserver > 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0. > > johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent) What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs? Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? What are you running inside the session? You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend. Uli ___ x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
