Re: Bookworm system randomly not responding (was Re: Bookworm system not responding on high memory usage)

[Xiyue Deng]

Xiyue Deng  writes:

> Xiyue Deng  writes:
>
>> Xiyue Deng  writes:
>>
>>> So after some more tries it looks like this issue is not directly memory
>>> usage related.  I've tried the following:
>>>
>>> * Using older kernel version when I was on Bullseye.
>>> * Have a cronjob to drop memory caches every minutes.
>>> * Using Gnome on Wayland by default or Xorg.
>>>
>>> And this can still happen when I was running a qemu-based Win11 VM using
>>> virtual manager.  So this rules out the possibility of a kernel issue
>>> and OOM killer issue.  All that is certain is that this issue can be
>>> reproduced when running my qemu-based Win11 VM and in a few hours it
>>> will trigger this lockup.
>>>
>>> As this system has been running Bullseye for a few years with zero
>>> problem, I'm hopeful this should work for Bookworm as well.  If you have
>>> anything in mind that may worth a try please feel free to share.  The
>>> more ideas the better.
>>>
>>> Thanks in advance!
>>
>> So, to rule out possible software issues, I've done a clean install of
>> Bookworm and Bullseye, and this issue still happens.  I guess this
>> largely lowers the possibility of a software cause.  I've also done a
>> 10-hour memtest session and it passed so I guess it was proven to be
>> clean as well.
>>
>> For the next step, I'll go with the hardware aspect.  I want to thank
>> for the helps, suggestions, and brainstorming from various people from
>> #debian{,-next} IRC channels!  Will try to get to the bottom of this.
>>
>
> Actually after I decided to contact the customer service of my box[1],
> after a few rounds of suggestions (reset CMOS, reinstall system, etc.),
> they provided an update to the BIOS that supposed to Windows 10/11
> freezing when accessing the fTPM module.  After flashing the new BIOS,
> I've been running the system on high load for 12+ hours without issue.
> Though a much longer testing period is needed to make sure the fix is
> sufficient, I think this is looking very promising!  Will report back
> after a week.
>
> Hope this is useful for anyway having similar issues.

It has been over a week after applying the BIOS update to my Minisforum
Elitemini HX90[1] and except a manual reboot my system has been running
totally fine!  So I'd consider this issue as resolved.  In case you are
using similar system from the same vendor and experiencing similar
system freezing issues, please contact the customer support for a
similar BIOS updates.

I'd like to thank the wonderful people at #debian{,-next} on IRC again
for helping me and the suggestions during the debugging!

>
> [1] https://store.minisforum.com/products/hx90
>
>>>
>>> (Replies to Timothy below inline.)
>>>
>>> Timothy M Butterworth  writes:
>>>
>>>> On Sat, Mar 11, 2023 at 3:30 AM Xiyue Deng  wrote:
>>>>
>>>>  Timothy M Butterworth  writes:
>>>>
>>>>  > On Fri, Mar 10, 2023 at 7:57 PM Xiyue Deng  wrote:
>>>>  >
>>>>  >  Hi,
>>>>  >
>>>>  >  I have an AMD64 system[1] that has been running fine on Bullseye for a
>>>>  >  few years, and recently following the soft freeze on Bookworm I 
>>>> upgraded
>>>>  >  my system to try it out, and the system has been frequently losing
>>>>  >  response.  Initially I thought it was because of some issue of my
>>>>  >  qemu-based Win11 virtual machine as it happens most frequently when it
>>>>  >  was running and filed a bug report[2].  But then it happened again
>>>>  >  without it running because some other program had slowly used up most 
>>>> of
>>>>  >  the memory again, though not as frequently as the VM was running.
>>>>  >
>>>>  >  Now in retrospect, when I was using Bullseye the total memory was also
>>>>  >  mostly used up most of the time, with a few hundreds of megabytes
>>>>  >  reported as free and a few Gigs reported as cache, and it has been
>>>>  >  running fine.  I'm not sure what has changed in Bookworm and having to
>>>>  >  manually restart the machine is a pretty annoying and unpleasant
>>>>  >  experience.
>>>>  >
>>>>  >  Does anyone seeing a similar problem as well?  What can I do to avoid
>>>>  >  this?  Any suggest is welcome.
>>>>  >
>>>>  >  Thanks in advance.
>>>>  >
>>>>  > Open the command prompt and run `su

Re: Bookworm system randomly not responding (was Re: Bookworm system not responding on high memory usage)

[Xiyue Deng]

Xiyue Deng  writes:

> Xiyue Deng  writes:
>
>> So after some more tries it looks like this issue is not directly memory
>> usage related.  I've tried the following:
>>
>> * Using older kernel version when I was on Bullseye.
>> * Have a cronjob to drop memory caches every minutes.
>> * Using Gnome on Wayland by default or Xorg.
>>
>> And this can still happen when I was running a qemu-based Win11 VM using
>> virtual manager.  So this rules out the possibility of a kernel issue
>> and OOM killer issue.  All that is certain is that this issue can be
>> reproduced when running my qemu-based Win11 VM and in a few hours it
>> will trigger this lockup.
>>
>> As this system has been running Bullseye for a few years with zero
>> problem, I'm hopeful this should work for Bookworm as well.  If you have
>> anything in mind that may worth a try please feel free to share.  The
>> more ideas the better.
>>
>> Thanks in advance!
>
> So, to rule out possible software issues, I've done a clean install of
> Bookworm and Bullseye, and this issue still happens.  I guess this
> largely lowers the possibility of a software cause.  I've also done a
> 10-hour memtest session and it passed so I guess it was proven to be
> clean as well.
>
> For the next step, I'll go with the hardware aspect.  I want to thank
> for the helps, suggestions, and brainstorming from various people from
> #debian{,-next} IRC channels!  Will try to get to the bottom of this.
>

Actually after I decided to contact the customer service of my box[1],
after a few rounds of suggestions (reset CMOS, reinstall system, etc.),
they provided an update to the BIOS that supposed to Windows 10/11
freezing when accessing the fTPM module.  After flashing the new BIOS,
I've been running the system on high load for 12+ hours without issue.
Though a much longer testing period is needed to make sure the fix is
sufficient, I think this is looking very promising!  Will report back
after a week.

Hope this is useful for anyway having similar issues.

[1] https://store.minisforum.com/products/hx90

>>
>> (Replies to Timothy below inline.)
>>
>> Timothy M Butterworth  writes:
>>
>>> On Sat, Mar 11, 2023 at 3:30 AM Xiyue Deng  wrote:
>>>
>>>  Timothy M Butterworth  writes:
>>>
>>>  > On Fri, Mar 10, 2023 at 7:57 PM Xiyue Deng  wrote:
>>>  >
>>>  >  Hi,
>>>  >
>>>  >  I have an AMD64 system[1] that has been running fine on Bullseye for a
>>>  >  few years, and recently following the soft freeze on Bookworm I upgraded
>>>  >  my system to try it out, and the system has been frequently losing
>>>  >  response.  Initially I thought it was because of some issue of my
>>>  >  qemu-based Win11 virtual machine as it happens most frequently when it
>>>  >  was running and filed a bug report[2].  But then it happened again
>>>  >  without it running because some other program had slowly used up most of
>>>  >  the memory again, though not as frequently as the VM was running.
>>>  >
>>>  >  Now in retrospect, when I was using Bullseye the total memory was also
>>>  >  mostly used up most of the time, with a few hundreds of megabytes
>>>  >  reported as free and a few Gigs reported as cache, and it has been
>>>  >  running fine.  I'm not sure what has changed in Bookworm and having to
>>>  >  manually restart the machine is a pretty annoying and unpleasant
>>>  >  experience.
>>>  >
>>>  >  Does anyone seeing a similar problem as well?  What can I do to avoid
>>>  >  this?  Any suggest is welcome.
>>>  >
>>>  >  Thanks in advance.
>>>  >
>>>  > Open the command prompt and run `su` to switch user to root. Then run 
>>> `sync && echo 1 > /proc/sys/vm/drop_caches`
>>>  as
>>>  > root. This will write RAM caches to the hard drive to free up memory. 
>>> You have to run this as root as sudo, my
>>>  preferred
>>>  > method, returns a permission disabled error.
>>>
>>>  Thanks for the tip!  I'll try it out.
>>
>> So unfortunately this doesn't help either, as it happens again with very
>> low cache usage.
>>
>> `free -h`:
>>
>>totalusedfree  shared  buff/cache   
>> available
>> Mem:30Gi13Gi16Gi   206Mi   1.4Gi
>> 17Gi
>> Swap:  979Mi  0B   979Mi
>>
>> `top` excerpt:
>>
>> top - 14:55:05 up 18 

Re: Bookworm system randomly not responding (was Re: Bookworm system not responding on high memory usage)

[Xiyue Deng]

Xiyue Deng  writes:

> So after some more tries it looks like this issue is not directly memory
> usage related.  I've tried the following:
>
> * Using older kernel version when I was on Bullseye.
> * Have a cronjob to drop memory caches every minutes.
> * Using Gnome on Wayland by default or Xorg.
>
> And this can still happen when I was running a qemu-based Win11 VM using
> virtual manager.  So this rules out the possibility of a kernel issue
> and OOM killer issue.  All that is certain is that this issue can be
> reproduced when running my qemu-based Win11 VM and in a few hours it
> will trigger this lockup.
>
> As this system has been running Bullseye for a few years with zero
> problem, I'm hopeful this should work for Bookworm as well.  If you have
> anything in mind that may worth a try please feel free to share.  The
> more ideas the better.
>
> Thanks in advance!

So, to rule out possible software issues, I've done a clean install of
Bookworm and Bullseye, and this issue still happens.  I guess this
largely lowers the possibility of a software cause.  I've also done a
10-hour memtest session and it passed so I guess it was proven to be
clean as well.

For the next step, I'll go with the hardware aspect.  I want to thank
for the helps, suggestions, and brainstorming from various people from
#debian{,-next} IRC channels!  Will try to get to the bottom of this.

>
> (Replies to Timothy below inline.)
>
> Timothy M Butterworth  writes:
>
>> On Sat, Mar 11, 2023 at 3:30 AM Xiyue Deng  wrote:
>>
>>  Timothy M Butterworth  writes:
>>
>>  > On Fri, Mar 10, 2023 at 7:57 PM Xiyue Deng  wrote:
>>  >
>>  >  Hi,
>>  >
>>  >  I have an AMD64 system[1] that has been running fine on Bullseye for a
>>  >  few years, and recently following the soft freeze on Bookworm I upgraded
>>  >  my system to try it out, and the system has been frequently losing
>>  >  response.  Initially I thought it was because of some issue of my
>>  >  qemu-based Win11 virtual machine as it happens most frequently when it
>>  >  was running and filed a bug report[2].  But then it happened again
>>  >  without it running because some other program had slowly used up most of
>>  >  the memory again, though not as frequently as the VM was running.
>>  >
>>  >  Now in retrospect, when I was using Bullseye the total memory was also
>>  >  mostly used up most of the time, with a few hundreds of megabytes
>>  >  reported as free and a few Gigs reported as cache, and it has been
>>  >  running fine.  I'm not sure what has changed in Bookworm and having to
>>  >  manually restart the machine is a pretty annoying and unpleasant
>>  >  experience.
>>  >
>>  >  Does anyone seeing a similar problem as well?  What can I do to avoid
>>  >  this?  Any suggest is welcome.
>>  >
>>  >  Thanks in advance.
>>  >
>>  > Open the command prompt and run `su` to switch user to root. Then run 
>> `sync && echo 1 > /proc/sys/vm/drop_caches`
>>  as
>>  > root. This will write RAM caches to the hard drive to free up memory. You 
>> have to run this as root as sudo, my
>>  preferred
>>  > method, returns a permission disabled error.
>>
>>  Thanks for the tip!  I'll try it out.
>
> So unfortunately this doesn't help either, as it happens again with very
> low cache usage.
>
> `free -h`:
>
>totalusedfree  shared  buff/cache   
> available
> Mem:30Gi13Gi16Gi   206Mi   1.4Gi
> 17Gi
> Swap:  979Mi  0B   979Mi
>
> `top` excerpt:
>
> top - 14:55:05 up 18 min, 11 users,  load average: 1.77, 1.65, 1.09
> Tasks: 504 total,   1 running, 503 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 12.5 us,  0.0 sy,  0.0 ni, 68.8 id,  0.0 wa,  0.0 hi,  6.2 si,  0.0 
> st 
> MiB Mem :  31519.9 total,  16972.6 free,  13759.0 used,   1447.6 buff/cache   
>   
> MiB Swap:980.0 total,980.0 free,  0.0 used.  17760.8 avail Mem 
>
> PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND
>8886 libvirt+  20   0   11.1g   8.1g  26580 S  87.5  26.4  17:38.47 
> qemu-sy+
>5434 xiyueden  20   0 4047004   1.2g 170036 S   0.0   4.0   0:41.00 
> thunder+
>5143 xiyueden  20   0 7056664 526296 191152 S   0.0   1.6   2:19.65 
> gnome-s+
> ...
>
>>
>>  >  
>>  >  
>>  >  [1] System info from inxi:
>>  >  CPU: 8-core AMD Ryzen 9 5900HX with Radeon Graphics (-MT MCP-)
>>  >  speed/min/max: 1199/1200/4679 MHz Kernel: 6.1.0-5-amd64

Re: Bookworm system randomly not responding (was Re: Bookworm system not responding on high memory usage)

[Anssi Saari]

Xiyue Deng  writes:

> As this system has been running Bullseye for a few years with zero
> problem, I'm hopeful this should work for Bookworm as well.  If you have
> anything in mind that may worth a try please feel free to share.  The
> more ideas the better.

To me the interesting question is, does the problem disappear if you go
back to Bullseye? If not then it's likely a hardware problem.

Bookworm system randomly not responding (was Re: Bookworm system not responding on high memory usage)

[Xiyue Deng]

So after some more tries it looks like this issue is not directly memory
usage related.  I've tried the following:

* Using older kernel version when I was on Bullseye.
* Have a cronjob to drop memory caches every minutes.
* Using Gnome on Wayland by default or Xorg.

And this can still happen when I was running a qemu-based Win11 VM using
virtual manager.  So this rules out the possibility of a kernel issue
and OOM killer issue.  All that is certain is that this issue can be
reproduced when running my qemu-based Win11 VM and in a few hours it
will trigger this lockup.

As this system has been running Bullseye for a few years with zero
problem, I'm hopeful this should work for Bookworm as well.  If you have
anything in mind that may worth a try please feel free to share.  The
more ideas the better.

Thanks in advance!

(Replies to Timothy below inline.)

Timothy M Butterworth  writes:

> On Sat, Mar 11, 2023 at 3:30 AM Xiyue Deng  wrote:
>
>  Timothy M Butterworth  writes:
>
>  > On Fri, Mar 10, 2023 at 7:57 PM Xiyue Deng  wrote:
>  >
>  >  Hi,
>  >
>  >  I have an AMD64 system[1] that has been running fine on Bullseye for a
>  >  few years, and recently following the soft freeze on Bookworm I upgraded
>  >  my system to try it out, and the system has been frequently losing
>  >  response.  Initially I thought it was because of some issue of my
>  >  qemu-based Win11 virtual machine as it happens most frequently when it
>  >  was running and filed a bug report[2].  But then it happened again
>  >  without it running because some other program had slowly used up most of
>  >  the memory again, though not as frequently as the VM was running.
>  >
>  >  Now in retrospect, when I was using Bullseye the total memory was also
>  >  mostly used up most of the time, with a few hundreds of megabytes
>  >  reported as free and a few Gigs reported as cache, and it has been
>  >  running fine.  I'm not sure what has changed in Bookworm and having to
>  >  manually restart the machine is a pretty annoying and unpleasant
>  >  experience.
>  >
>  >  Does anyone seeing a similar problem as well?  What can I do to avoid
>  >  this?  Any suggest is welcome.
>  >
>  >  Thanks in advance.
>  >
>  > Open the command prompt and run `su` to switch user to root. Then run 
> `sync && echo 1 > /proc/sys/vm/drop_caches`
>  as
>  > root. This will write RAM caches to the hard drive to free up memory. You 
> have to run this as root as sudo, my
>  preferred
>  > method, returns a permission disabled error.
>
>  Thanks for the tip!  I'll try it out.

So unfortunately this doesn't help either, as it happens again with very
low cache usage.

`free -h`:

   totalusedfree  shared  buff/cache   available
Mem:30Gi13Gi16Gi   206Mi   1.4Gi17Gi
Swap:  979Mi  0B   979Mi

`top` excerpt:

top - 14:55:05 up 18 min, 11 users,  load average: 1.77, 1.65, 1.09
Tasks: 504 total,   1 running, 503 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.5 us,  0.0 sy,  0.0 ni, 68.8 id,  0.0 wa,  0.0 hi,  6.2 si,  0.0 st 
MiB Mem :  31519.9 total,  16972.6 free,  13759.0 used,   1447.6 buff/cache 
MiB Swap:980.0 total,980.0 free,  0.0 used.  17760.8 avail Mem 

PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND
   8886 libvirt+  20   0   11.1g   8.1g  26580 S  87.5  26.4  17:38.47 qemu-sy+
   5434 xiyueden  20   0 4047004   1.2g 170036 S   0.0   4.0   0:41.00 thunder+
   5143 xiyueden  20   0 7056664 526296 191152 S   0.0   1.6   2:19.65 gnome-s+
...

>
>  >  
>  >  
>  >  [1] System info from inxi:
>  >  CPU: 8-core AMD Ryzen 9 5900HX with Radeon Graphics (-MT MCP-)
>  >  speed/min/max: 1199/1200/4679 MHz Kernel: 6.1.0-5-amd64 x86_64 Up: 7m
>  >  Mem: 4844.4/31521.3 MiB (15.4%) Storage: 476.94 GiB (54.5% used) Procs: 
> 535
>  >  Shell: Bash inxi: 3.3.25
>  >
>  > Your system has 32 GB of RAM, it should not be getting used up. Run `free 
> -h` What desktop are you using: KDE,
>  GNOME,
>  > LXQT etc? Are you using Wayland or X11? It looks like you have a memory 
> leak in one of your applications. Try
>  running
>  > `top` and press `m` to sort by memory utilization.
>
>  I actually have a cronjob that runs every 5 minutes and collects memory
>  usage.  As I mentioned, it usually happens when I use qemu (see [1] for
>  free and [2] for top).  At another time it happened when deluge is
>  leaking memory (see [3] for free [4] for top).
>
>  Interestingly as you can see, in all such cases, even though the free
>  amount is low, the buff/cache is still pretty large so the system is not
>  really overloaded.  Plus, 

Re: Bookworm system not responding on high memory usage

[Timothy M Butterworth]

On Sat, Mar 11, 2023 at 3:30 AM Xiyue Deng  wrote:

>
> Timothy M Butterworth  writes:
>
> > On Fri, Mar 10, 2023 at 7:57 PM Xiyue Deng  wrote:
> >
> >  Hi,
> >
> >  I have an AMD64 system[1] that has been running fine on Bullseye for a
> >  few years, and recently following the soft freeze on Bookworm I upgraded
> >  my system to try it out, and the system has been frequently losing
> >  response.  Initially I thought it was because of some issue of my
> >  qemu-based Win11 virtual machine as it happens most frequently when it
> >  was running and filed a bug report[2].  But then it happened again
> >  without it running because some other program had slowly used up most of
> >  the memory again, though not as frequently as the VM was running.
> >
> >  Now in retrospect, when I was using Bullseye the total memory was also
> >  mostly used up most of the time, with a few hundreds of megabytes
> >  reported as free and a few Gigs reported as cache, and it has been
> >  running fine.  I'm not sure what has changed in Bookworm and having to
> >  manually restart the machine is a pretty annoying and unpleasant
> >  experience.
> >
> >  Does anyone seeing a similar problem as well?  What can I do to avoid
> >  this?  Any suggest is welcome.
> >
> >  Thanks in advance.
> >
> > Open the command prompt and run `su` to switch user to root. Then run
> `sync && echo 1 > /proc/sys/vm/drop_caches` as
> > root. This will write RAM caches to the hard drive to free up memory.
> You have to run this as root as sudo, my preferred
> > method, returns a permission disabled error.
>
> Thanks for the tip!  I'll try it out.
>
> >
> >
> >  [1] System info from inxi:
> >  CPU: 8-core AMD Ryzen 9 5900HX with Radeon Graphics (-MT MCP-)
> >  speed/min/max: 1199/1200/4679 MHz Kernel: 6.1.0-5-amd64 x86_64 Up: 7m
> >  Mem: 4844.4/31521.3 MiB (15.4%) Storage: 476.94 GiB (54.5% used) Procs:
> 535
> >  Shell: Bash inxi: 3.3.25
> >
> > Your system has 32 GB of RAM, it should not be getting used up. Run
> `free -h` What desktop are you using: KDE, GNOME,
> > LXQT etc? Are you using Wayland or X11? It looks like you have a memory
> leak in one of your applications. Try running
> > `top` and press `m` to sort by memory utilization.
>
> I actually have a cronjob that runs every 5 minutes and collects memory
> usage.  As I mentioned, it usually happens when I use qemu (see [1] for
> free and [2] for top).  At another time it happened when deluge is
> leaking memory (see [3] for free [4] for top).
>
> Interestingly as you can see, in all such cases, even though the free
> amount is low, the buff/cache is still pretty large so the system is not
> really overloaded.  Plus, on Bullseye such memory usage also happens all
> the time and this never happened.  I was suspecting that maybe the
> kernel is panicking when memory hits certain limit, but I don't see it
> in kern.log or syslog.
>
> Any suggestion to restore to Bullseye status is appreciated.  Thanks in
> advance!
>
>
> [1] `free -h` when using qemu:
>totalusedfree  shared  buff/cache
>  available
> Mem:30Gi14Gi   258Mi   216Mi17Gi
>   16Gi
> Swap:  979Mi80Mi   899Mi
>

I have an  AMD Ryzen 7 4700U with Radeon Graphics and the only time I see
my RAM used up is when I am transcoding Video files.

System Idle running KDE 5.27.2, Google Chrome and Dolphin:

   totalusedfree  shared  buff/cache
  available
Mem:14Gi   3.8Gi   9.4Gi91Mi   2.2Gi
   11Gi

System with VirtualBox running Kali Linux
   totalusedfree  shared  buff/cache
  available
Mem:14Gi   8.9Gi   4.2Gi   110Mi   2.3Gi
  6.1Gi
Swap:   14Gi  0B14Gi



> [2] `top` sorted by memory when using qemu:
> top - 16:10:05 up  1:29, 11 users,  load average: 1.83, 1.86, 2.06
> Tasks: 494 total,   1 running, 493 sleeping,   0 stopped,   0 zombie
> %Cpu(s):  8.3 us,  8.3 sy,  0.0 ni, 75.0 id,  0.0 wa,  0.0 hi,  0.0 si,
> 0.0 st
> MiB Mem :  31522.7 total,257.2 free,  14430.8 used,  17504.1
> buff/cache
> MiB Swap:980.0 total,899.5 free, 80.5 used.  17091.9 avail Mem
>
> PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+
> COMMAND
>   10131 libvirt+  20   0   11.2g   8.1g  26140 S 213.3  26.2  75:08.67
> qemu-sy+
>6547 xiyueden  20   0 4432172   1.4g 207312 S   0.0   4.5   1:53.44
> thunder+
> ...
>
> [3] `free -h` when using deluge:
>totalused 

Re: Bookworm system not responding on high memory usage

[Xiyue Deng]

Timothy M Butterworth  writes:

> On Fri, Mar 10, 2023 at 7:57 PM Xiyue Deng  wrote:
>
>  Hi,
>
>  I have an AMD64 system[1] that has been running fine on Bullseye for a
>  few years, and recently following the soft freeze on Bookworm I upgraded
>  my system to try it out, and the system has been frequently losing
>  response.  Initially I thought it was because of some issue of my
>  qemu-based Win11 virtual machine as it happens most frequently when it
>  was running and filed a bug report[2].  But then it happened again
>  without it running because some other program had slowly used up most of
>  the memory again, though not as frequently as the VM was running.
>
>  Now in retrospect, when I was using Bullseye the total memory was also
>  mostly used up most of the time, with a few hundreds of megabytes
>  reported as free and a few Gigs reported as cache, and it has been
>  running fine.  I'm not sure what has changed in Bookworm and having to
>  manually restart the machine is a pretty annoying and unpleasant
>  experience.
>
>  Does anyone seeing a similar problem as well?  What can I do to avoid
>  this?  Any suggest is welcome.
>
>  Thanks in advance.
>
> Open the command prompt and run `su` to switch user to root. Then run `sync 
> && echo 1 > /proc/sys/vm/drop_caches` as
> root. This will write RAM caches to the hard drive to free up memory. You 
> have to run this as root as sudo, my preferred
> method, returns a permission disabled error.

Thanks for the tip!  I'll try it out.

>  
>  
>  [1] System info from inxi:
>  CPU: 8-core AMD Ryzen 9 5900HX with Radeon Graphics (-MT MCP-)
>  speed/min/max: 1199/1200/4679 MHz Kernel: 6.1.0-5-amd64 x86_64 Up: 7m
>  Mem: 4844.4/31521.3 MiB (15.4%) Storage: 476.94 GiB (54.5% used) Procs: 535
>  Shell: Bash inxi: 3.3.25
>
> Your system has 32 GB of RAM, it should not be getting used up. Run `free -h` 
> What desktop are you using: KDE, GNOME,
> LXQT etc? Are you using Wayland or X11? It looks like you have a memory leak 
> in one of your applications. Try running
> `top` and press `m` to sort by memory utilization.

I actually have a cronjob that runs every 5 minutes and collects memory
usage.  As I mentioned, it usually happens when I use qemu (see [1] for
free and [2] for top).  At another time it happened when deluge is
leaking memory (see [3] for free [4] for top).

Interestingly as you can see, in all such cases, even though the free
amount is low, the buff/cache is still pretty large so the system is not
really overloaded.  Plus, on Bullseye such memory usage also happens all
the time and this never happened.  I was suspecting that maybe the
kernel is panicking when memory hits certain limit, but I don't see it
in kern.log or syslog.

Any suggestion to restore to Bullseye status is appreciated.  Thanks in
advance!


[1] `free -h` when using qemu:
   totalusedfree  shared  buff/cache   available
Mem:30Gi14Gi   258Mi   216Mi17Gi16Gi
Swap:  979Mi80Mi   899Mi

[2] `top` sorted by memory when using qemu:
top - 16:10:05 up  1:29, 11 users,  load average: 1.83, 1.86, 2.06
Tasks: 494 total,   1 running, 493 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.3 us,  8.3 sy,  0.0 ni, 75.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :  31522.7 total,257.2 free,  14430.8 used,  17504.1 buff/cache 
MiB Swap:980.0 total,899.5 free, 80.5 used.  17091.9 avail Mem 

PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND
  10131 libvirt+  20   0   11.2g   8.1g  26140 S 213.3  26.2  75:08.67 qemu-sy+
   6547 xiyueden  20   0 4432172   1.4g 207312 S   0.0   4.5   1:53.44 thunder+
...

[3] `free -h` when using deluge:
   totalusedfree  shared  buff/cache   available
Mem:30Gi12Gi   1.9Gi   219Mi17Gi18Gi
Swap:  979Mi   2.2Mi   977Mi

[4] `top` sorted by memory when using deluge:
top - 10:40:05 up 3 days, 17:11, 11 users,  load average: 1.25, 1.22, 1.20
Tasks: 492 total,   1 running, 490 sleeping,   0 stopped,   1 zombie
%Cpu(s): 25.0 us,  0.0 sy,  0.0 ni, 75.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :  31521.3 total,   1909.2 free,  12762.9 used,  17529.7 buff/cache 
MiB Swap:980.0 total,977.7 free,  2.2 used.  18758.4 avail Mem 

PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND
   7287 xiyueden  20   0 9030940   6.6g 503076 S   0.0  21.3  97:11.62 deluge-+
   5271 xiyueden  20   0 4581328   1.6g 191000 S   6.7   5.2 108:23.57 thunder+
...

>
> Tim
>
>  
>  [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032400
>
>  -- 
>  Manphiz


-- 
Manphiz

Re: Bookworm system not responding on high memory usage

[Timothy M Butterworth]

On Fri, Mar 10, 2023 at 7:57 PM Xiyue Deng  wrote:

> Hi,
>
> I have an AMD64 system[1] that has been running fine on Bullseye for a
> few years, and recently following the soft freeze on Bookworm I upgraded
> my system to try it out, and the system has been frequently losing
> response.  Initially I thought it was because of some issue of my
> qemu-based Win11 virtual machine as it happens most frequently when it
> was running and filed a bug report[2].  But then it happened again
> without it running because some other program had slowly used up most of
> the memory again, though not as frequently as the VM was running.
>
> Now in retrospect, when I was using Bullseye the total memory was also
> mostly used up most of the time, with a few hundreds of megabytes
> reported as free and a few Gigs reported as cache, and it has been
> running fine.  I'm not sure what has changed in Bookworm and having to
> manually restart the machine is a pretty annoying and unpleasant
> experience.
>
> Does anyone seeing a similar problem as well?  What can I do to avoid
> this?  Any suggest is welcome.
>
> Thanks in advance.
>

Open the command prompt and run `su` to switch user to root. Then run `sync
&& echo 1 > /proc/sys/vm/drop_caches` as root. This will write RAM caches
to the hard drive to free up memory. You have to run this as root as sudo,
my preferred method, returns a permission disabled error.


>
> [1] System info from inxi:
> CPU: 8-core AMD Ryzen 9 5900HX with Radeon Graphics (-MT MCP-)
> speed/min/max: 1199/1200/4679 MHz Kernel: 6.1.0-5-amd64 x86_64 Up: 7m
> Mem: 4844.4/31521.3 MiB (15.4%) Storage: 476.94 GiB (54.5% used) Procs: 535
> Shell: Bash inxi: 3.3.25
>

Your system has 32 GB of RAM, it should not be getting used up. Run `free
-h` What desktop are you using: KDE, GNOME, LXQT etc? Are you using Wayland
or X11? It looks like you have a memory leak in one of your applications.
Try running `top` and press `m` to sort by memory utilization.

Tim



> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032400
>
> --
> Manphiz
>
>

-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀

Bookworm system not responding on high memory usage

[Xiyue Deng]

Hi,

I have an AMD64 system[1] that has been running fine on Bullseye for a
few years, and recently following the soft freeze on Bookworm I upgraded
my system to try it out, and the system has been frequently losing
response.  Initially I thought it was because of some issue of my
qemu-based Win11 virtual machine as it happens most frequently when it
was running and filed a bug report[2].  But then it happened again
without it running because some other program had slowly used up most of
the memory again, though not as frequently as the VM was running.

Now in retrospect, when I was using Bullseye the total memory was also
mostly used up most of the time, with a few hundreds of megabytes
reported as free and a few Gigs reported as cache, and it has been
running fine.  I'm not sure what has changed in Bookworm and having to
manually restart the machine is a pretty annoying and unpleasant
experience.

Does anyone seeing a similar problem as well?  What can I do to avoid
this?  Any suggest is welcome.

Thanks in advance.


[1] System info from inxi:
CPU: 8-core AMD Ryzen 9 5900HX with Radeon Graphics (-MT MCP-)
speed/min/max: 1199/1200/4679 MHz Kernel: 6.1.0-5-amd64 x86_64 Up: 7m
Mem: 4844.4/31521.3 MiB (15.4%) Storage: 476.94 GiB (54.5% used) Procs: 535
Shell: Bash inxi: 3.3.25

[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032400

-- 
Manphiz

Re: Flatpak memory usage

[paulf]

On Thu, 16 Feb 2023 17:12:12 -0500
Jeffrey Walton  wrote:

> On Wed, Feb 15, 2023 at 1:11 AM  wrote:
> >
> > On Tue, Feb 14, 2023 at 10:36:12PM -0500, pa...@quillandmouse.com
> > wrote:
> >
> > [...]
> >
> > > I find the trend disturbing. If you have a lot of apps running,
> > > and they're all these types of packages, you're going to be using
> > > considerably more memory [...]
> >
> > I'm not a friend of flatpaks and similar concepts, either. For me,
> > it's not memory use, but the shifting of power from a distrubution
> > model to single applications. I find that makes software less
> > "free".
> >
> > In a distro, applications have to get along with each other, agree
> > on a common set of libraries, file system layout, etc. I think this
> > is a Good Thing. Every app carrying its own little distro is like
> > neoliberal hell. No wonder it uses up more resources ;-D
> 
> And the distro no longer has administrative control over the package.
> The programs do not go through distro-based end-to-end testing or
> security testing and evaluation.
> 
> Knowing a distro performs end-to-end testing, monitors security events
> and performs patching is the reason I use distros. The distro owned it
> and took care of it.
> 
> Now each user owns it. The user is responsible for ensuring the code
> is fit for installation.
> 
> Jeff
> 

This brings up another tangent. Bob, a Debian user, is using a
flatpak version of the Snorgistic package, and something goes wrong. He
gets on the Debian user list for support. He's not thinking about where
his package came from, and we don't realize it's a flatpak. We try to
help him, to no avail. A waste of community and Bob's time as well.

Paul

-- 
Paul M. Foster
Personal Blog: http://noferblatz.com
Company Site: http://quillandmouse.com
Software Projects: https://gitlab.com/paulmfoster

Re: Flatpak memory usage

[Jeffrey Walton]

On Wed, Feb 15, 2023 at 1:11 AM  wrote:
>
> On Tue, Feb 14, 2023 at 10:36:12PM -0500, pa...@quillandmouse.com wrote:
>
> [...]
>
> > I find the trend disturbing. If you have a lot of apps running, and
> > they're all these types of packages, you're going to be using
> > considerably more memory [...]
>
> I'm not a friend of flatpaks and similar concepts, either. For me,
> it's not memory use, but the shifting of power from a distrubution
> model to single applications. I find that makes software less "free".
>
> In a distro, applications have to get along with each other, agree
> on a common set of libraries, file system layout, etc. I think this
> is a Good Thing. Every app carrying its own little distro is like
> neoliberal hell. No wonder it uses up more resources ;-D

And the distro no longer has administrative control over the package.
The programs do not go through distro-based end-to-end testing or
security testing and evaluation.

Knowing a distro performs end-to-end testing, monitors security events
and performs patching is the reason I use distros. The distro owned it
and took care of it.

Now each user owns it. The user is responsible for ensuring the code
is fit for installation.

Jeff

Re: Flatpak memory usage

[tomas]

On Wed, Feb 15, 2023 at 09:13:06AM +0100, Nicolas George wrote:

[...]

> > In a distro, applications have to get along with each other, agree
> > on a common set of libraries, file system layout, etc. I think this
> > is a Good Thing. Every app carrying its own little distro is like
> > neoliberal hell. No wonder it uses up more resources ;-D
> 
> I agree with that. The memory impact of code is probably not that big
> compared to the carelessness of applications with their memory
> management for data.

Right: this was stretching the analogy a bit. In real life, that's
what happens to resources, too.

> But there is an even worse side to these pseudo-package managers:
> updates.
> 
> Now that everybody is responsible for packaging there own applications
> with all its libraries, if a bug is found in an application, you can
> hope its author will issue an updated package.
> 
> But do you trust the developers of all the applications you use to make
> updates every time a bug, including a security issue, is found in any of
> the embedded libraries?

Yes, that's another technical aspect. Imagine you have 17 slightly
different versions of libc spread across your Flatlands. Imagine
further that some big, fat CVE turns up, affecting 15 of those 17
(the other two are perhaps too old).

Of those 15, two upstream "vendors" have gone bust, another one was
a private person and has lost interest. Another was picked up by
some sleazy malicious actor who is eagerly waiting for you to push
the update button (yeah, that does happen [1] in npm world!).

All that said, I was more interested in the sociological structure
of the whole thing, because it looks like a mirror image of that
"collective" vs. "individual" from political life, which we as
humankind haven't managed to solve for the last millenia, take or
give :-)

Cheers

[1] 
https://www.synopsys.com/blogs/software-security/malicious-dependency-supply-chain/
-- 
t


signature.asc
Description: PGP signature

Re: Flatpak memory usage

[tomas]

On Wed, Feb 15, 2023 at 01:34:11AM -0500, pa...@quillandmouse.com wrote:

[...]

> I trust Debian to audit and ensure my packages are secure and
> interoperable. I don't necessarily trust Canonical or Flathub.

That's a very good condensate. That's my take, too.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: Flatpak memory usage

[tomas]

On Wed, Feb 15, 2023 at 12:18:45PM -0500, Stefan Monnier wrote:
> > I'm not a friend of flatpaks and similar concepts, either. For me,
> > it's not memory use, but the shifting of power from a distrubution
> > model to single applications. I find that makes software less "free".
> 
> Indeed.  These end up reproducing the black-box model "it just works".
> If you like that, then you presumably don't care what's inside the box.
> It might as well be iOS, Windows, Oracle, ...

I think it's interesting how history repeats here. This development
resembles much Windows's answer to "DLL hell" back in the 1990s: every
application comes with its own versions of DLLs.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: Flatpak memory usage

[Stefan Monnier]

> I'm not a friend of flatpaks and similar concepts, either. For me,
> it's not memory use, but the shifting of power from a distrubution
> model to single applications. I find that makes software less "free".

Indeed.  These end up reproducing the black-box model "it just works".
If you like that, then you presumably don't care what's inside the box.
It might as well be iOS, Windows, Oracle, ...


Stefan

Re: Flatpak memory usage

[Nicolas George]

to...@tuxteam.de (12023-02-15):
> I'm not a friend of flatpaks and similar concepts, either. For me,
> it's not memory use, but the shifting of power from a distrubution
> model to single applications. I find that makes software less "free".
> 
> In a distro, applications have to get along with each other, agree
> on a common set of libraries, file system layout, etc. I think this
> is a Good Thing. Every app carrying its own little distro is like
> neoliberal hell. No wonder it uses up more resources ;-D

I agree with that. The memory impact of code is probably not that big
compared to the carelessness of applications with their memory
management for data.

But there is an even worse side to these pseudo-package managers:
updates.

Now that everybody is responsible for packaging there own applications
with all its libraries, if a bug is found in an application, you can
hope its author will issue an updated package.

But do you trust the developers of all the applications you use to make
updates every time a bug, including a security issue, is found in any of
the embedded libraries?

Yeah, me neither.

And that is not counting the fact that every time a new bug is found in
OpenSSL or libxml2 we would have to update the dozens of packages that
embed them.

Regards,

-- 
  Nicolas George

Re: Flatpak memory usage

[paulf]

On Wed, 15 Feb 2023 07:11:02 +0100
 wrote:

> On Tue, Feb 14, 2023 at 10:36:12PM -0500, pa...@quillandmouse.com
> wrote:
> 
> [...]
> 
> > I find the trend disturbing. If you have a lot of apps running, and
> > they're all these types of packages, you're going to be using
> > considerably more memory [...]
> 
> I'm not a friend of flatpaks and similar concepts, either. For me,
> it's not memory use, but the shifting of power from a distrubution
> model to single applications. I find that makes software less "free".
> 
> In a distro, applications have to get along with each other, agree
> on a common set of libraries, file system layout, etc. I think this
> is a Good Thing. Every app carrying its own little distro is like
> neoliberal hell. No wonder it uses up more resources ;-D

I'd forgotten about that angle. IIRC, snaps are controlled by
Canonical. Flathub controls flatpaks. Appimages can be built by anyone,
but they are one file with *everything* in them.

I trust Debian to audit and ensure my packages are secure and
interoperable. I don't necessarily trust Canonical or Flathub.

Paul

-- 
Paul M. Foster
Personal Blog: http://noferblatz.com
Company Site: http://quillandmouse.com
Software Projects: https://gitlab.com/paulmfoster

Re: Flatpak memory usage

[tomas]

On Tue, Feb 14, 2023 at 10:36:12PM -0500, pa...@quillandmouse.com wrote:

[...]

> I find the trend disturbing. If you have a lot of apps running, and
> they're all these types of packages, you're going to be using
> considerably more memory [...]

I'm not a friend of flatpaks and similar concepts, either. For me,
it's not memory use, but the shifting of power from a distrubution
model to single applications. I find that makes software less "free".

In a distro, applications have to get along with each other, agree
on a common set of libraries, file system layout, etc. I think this
is a Good Thing. Every app carrying its own little distro is like
neoliberal hell. No wonder it uses up more resources ;-D

> Maybe I'm weird.

I seem to be even weirder :)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: Flatpak memory usage

[paulf]

On Tue, 14 Feb 2023 23:55:03 +0100
Oliver Schoede  wrote:

> On Mon, 13 Feb 2023 09:35:34 -0500
>  wrote:
> 
> >Am I correct in assuming that package formats like Flatpak, Snap and
> >Appimage, because they package up everything with the executable,
> >would consume more system memory?

[snip]
 
> Flatpak is the odd one out and doesn't exactly work like this. Rather
> than completely self-contained images, the packages aren't that much
> different from what we have in Debian. The difference being that as
> for dependencies it's more of an all or nothing affair.

I principally wanted to confirm my suspicions about memory usage.
There's been increasing usage of Flatpaks, Snaps and Appimages. As
though it's a solution to the "problem" of distributions' own package
management systems. And now Fedora is openly embracing Flatpaks.

I find the trend disturbing. If you have a lot of apps running, and
they're all these types of packages, you're going to be using
considerably more memory. The alarming increase in the size of the
Linux kernel is yet another symptom of this idea that, because memory
is cheap, we simply use more. In my mind, it's a little like having
access to unlimited amounts of water and thus using all of it you can.
Or gasoline/petrol. Or food.

I don't have a problem with Debian's packaging system, and am generally
satisfied with the stable but older versions of Debian packages. They
get the job done.

Maybe I'm weird.

Paul

-- 
Paul M. Foster
Personal Blog: http://noferblatz.com
Company Site: http://quillandmouse.com
Software Projects: https://gitlab.com/paulmfoster

Re: Flatpak memory usage

[Oliver Schoede]

On Mon, 13 Feb 2023 09:35:34 -0500
 wrote:

>Am I correct in assuming that package formats like Flatpak, Snap and
>Appimage, because they package up everything with the executable, would
>consume more system memory? One of the reasons to use these formats is
>to avoid library version mismatches, and peg the libraries which
>accompany an executable at a certain version. But if this is true, then
>it stands to reason that the executable would use, for example, GNOME
>libraries which aren't the same as what's on your system being shared
>by other software. Thus, when you launch X flatpak, it must load its
>own version of the GNOME libraries. Which would take up more system
>memory.
>

Flatpak is the odd one out and doesn't exactly work like this. Rather
than completely self-contained images, the packages aren't that much
different from what we have in Debian. The difference being that as for
dependencies it's more of an all or nothing affair. Or what's called a
"runtime", basically a small userland mostly tied to specific desktop
environments. Clearly not as economical, especially if all you need is
a single app, which to be fair isn't Flatpak's intented use case, or
everything you're using needs a different runtime. It's also simpler
though. With nothing but, say, GTK software you might always get away
with a single runtime. I do, although with very few apps installed.
Yes, if you're then also running something installed via Debian, or yet
another package manager, with the same dependencies, there's
redundancy, this is true even where versions match. I wouldn't rack my
brains on that however, modern loaders are quite intelligent and
dynamic linking is selective in a sense. I guess it's quite attractive
for people like me who are not even using one of the full-blown DEs
like GNOME, so there's little that must be resident all the time and
I'm more flexibel in "load balancing" if need be, which should be
almost never considering today's memory supplies. And I don't think any
of these solutions is specifically catering to resource-constrained
systems. With that you're probably always better off with installing
from a single source.

Oliver

Re: Flatpak memory usage

[Greg Wooledge]

On Mon, Feb 13, 2023 at 09:35:34AM -0500, pa...@quillandmouse.com wrote:
> Am I correct in assuming that package formats like Flatpak, Snap and
> Appimage, because they package up everything with the executable, would
> consume more system memory?

"Yes, but it depends."

Let's say you have two applications linked with libjpeg -- one regular
Debian package, which uses the Debian dynamic libjpeg, and one flatpak
thing that's either statically linked with its own libjpeg, or brings
along its own dynamic libjpeg.

If you run both of those at the same time, then you'll have two different
versions of libjpeg in memory.

However, if you only run *one* of them at a time, then there'll only be
one instance of libjpeg in memory.

Flatpak memory usage

[paulf]

Folks:

Am I correct in assuming that package formats like Flatpak, Snap and
Appimage, because they package up everything with the executable, would
consume more system memory? One of the reasons to use these formats is
to avoid library version mismatches, and peg the libraries which
accompany an executable at a certain version. But if this is true, then
it stands to reason that the executable would use, for example, GNOME
libraries which aren't the same as what's on your system being shared
by other software. Thus, when you launch X flatpak, it must load its
own version of the GNOME libraries. Which would take up more system
memory.

Am I correct about this, or is there something I'm missing?

Paul

-- 
Paul M. Foster
Personal Blog: http://noferblatz.com
Company Site: http://quillandmouse.com
Software Projects: https://gitlab.com/paulmfoster

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Martin Schwarz]

On Tue, Apr 16, 2019 at 04:39:32PM +0200, Peter Wiersig wrote:
>rss RSS   resident set size, the non-swapped physical 
> memory that a task has used (in kiloBytes).
>  (alias rssize, rsz).
> ...
>vsz VSZ   virtual memory size of the process in KiB 
> (1024-byte units).  Device mappings are currently
>  excluded; this is subject to change. (alias 
> vsize)."""
> 

Thanks for pointing out the difference. However, in this case I'm trying
to find out what consumes RAM specifically, not virtual memory in
general. Besides, from all I can see, the high memory usage is NOT
caused user space processes. So the output from ps was just to show that
even though some hundred MB of RAM (!) are used, just some few MB are
consumed by processes.

Kind regards
Martin

-- 
Martin Schwarz * Karlsruhe, Germany * http://kuroi.de/

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Reco]

On Wed, Apr 17, 2019 at 01:14:36AM +0200, Peter Wiersig wrote:
> Reco  writes:
> 
> Hi Reco,
> 
> > On Tue, Apr 16, 2019 at 04:39:32PM +0200, Peter Wiersig wrote:
> >> VSZ is the Virtual Memory Size. (...),
> >>including memory that is swapped out,
> >> memory that is allocated, but not used,
>   
> >> and memory that is from shared libraries.""
> >
> > Given this:
> >
> > == cut ==
> >
> > One can expect the process to "consume" 4Gb of VSZ, and about 700kb of
> > RSS.
> >
> > Do you really suggest to treat all this VSZ as a "used memory"? A hint -
> > compile the sample, run it a hundred times.
> 
> No, but that is all in the explanation above albeit tersely. I
> underlined the part which comes into play with your example code.

That was the whole point. If it's allocated, but ain't used - one should
not account it.


> If Martin wants to find out what's using his memory during timespans
> with degraded performance, it will not be helpful to list the processes
> and sort by thenot swapped amount.

To quote the original e-mail:

root@rad-m2m-srv02:~# free -thwl
  totalusedfree  shared buffers cache   
available
Mem:   987M910M 59M  0B704K 16M 
13M
Low:   987M927M 59M
High:0B  0B  0B
Swap:  2,0G345M1,7G
Total: 3,0G1,2G1,7G

root@rad-m2m-srv02:~# smem -uktr
User Count Swap  USS  PSS  RSS
root39   332.8M10.4M12.4M44.7M
msch 6 7.0M0   607.0K 8.3M
_chrony  1   360.0K 4.0K20.0K   572.0K
messagebus 1   580.0K 4.0K17.0K   480.0K
postfix  2 1.6M013.0K   568.0K
daemon   1   208.0K 4.0K 6.0K72.0K
---
50   342.5M10.4M13.0M54.7M

A host has 1Gb of RAM and 3Gb swap.
Both "free" and "/proc/meminfo" show that there's no free memory left.
"smem" output shows us that this memory ain't used by userspace.
The question was - where all this memory has gone?


> If the system is thrashing due to swapping in and out, I always look
> at the VSZ values to find out where I should direct my attention to.

Thanks for sharing. In this particular case, swapping is a consequence,
not a root cause.
But, just in case, how exactly your method is superior to "top -o SWAP"?

Reco

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Peter Wiersig]

Martin Schwarz  writes:
>
> Here's the output from some commands I hope to be helpful:
>
> The machine in this example is a RADIUS server but has not even gone
> productive ... no incoming client requests yet.  (But the problem is not
> related to the RADIUS server software - OSC Radiator - since the same
> symptoms show on different machines: not only RADIUS servers but also
> nameservers, shell servers or jumphosts, etc.)
>
> [values while the problem persists:]
...
> USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> root 34718 12.0  0.5  29596  5672 ?D09:01   0:00 
> /usr/bin/python3 -Es /usr/bin/lsb_release --short --description
> root 26491  3.1  0.2  79328  2860 ?D08:04   1:50 apt-get 
> update -qq
> root 32551  6.8  0.2 119036  2800 ?D08:51   0:43 
> /usr/bin/python3 /usr/bin/unattended-upgrade

Disable this, do your upgrades by some schedule for the duration in
which you're debugging this problem.  Think about system orchestration
tools with push mechanisms if you want to minimize RAM allocated to
VMs.  We're thinking about deploying ansible for patch management.

> root 12792  2.2  0.1 159720  1748 ?D06:06   3:54 
> /usr/bin/perl -w /usr/bin/apt-show-versions -i
> root 15502  2.4  0.1 167660  1608 ?D06:25   3:51 
> /usr/bin/perl -w /usr/bin/apt-show-versions -i

Do they need to run on 6:06 and then parallel at 6:25?  What's their
process tree calling structure, ie. what's starting them?

> root 34527  1.7  0.1  14096  1596 ?Ss   09:01   0:00 /bin/bash 
> /usr/bin/check_mk_agent

Can you show a zoomed image of the memory graph prior to a problem?  And
a load graph of the same duration?

I had some webservers which were also prone to death spiraling, the only
real solution was to throw RAM at them until they were able to process
the requests and to optimize the database indices to speed up the time
spent fetching and sorting rows.

Peter

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Peter Wiersig]

Reco  writes:

Hi Reco,

> On Tue, Apr 16, 2019 at 04:39:32PM +0200, Peter Wiersig wrote:
>> VSZ is the Virtual Memory Size. (...),
>>including memory that is swapped out,
>> memory that is allocated, but not used,
  
>> and memory that is from shared libraries.""
>
> Given this:
>
> == cut ==
>
> One can expect the process to "consume" 4Gb of VSZ, and about 700kb of
> RSS.
>
> Do you really suggest to treat all this VSZ as a "used memory"? A hint -
> compile the sample, run it a hundred times.

No, but that is all in the explanation above albeit tersely. I
underlined the part which comes into play with your example code.

If Martin wants to find out what's using his memory during timespans
with degraded performance, it will not be helpful to list the processes
and sort by thenot swapped amount.  If the system is thrashing due to
swapping in and out, I always look at the VSZ values to find out where I
should direct my attention to.

If Martin or if you want, I can explode the the previous sentence about
what VSZ is but you'd probably find better answers if you read about
that.

Feel free to ask, I think I can explain more, but not better than
stackexchange answers or for example LWN kernel article series.

Have fun,
Peter

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Reco]

Hi.

On Tue, Apr 16, 2019 at 04:39:32PM +0200, Peter Wiersig wrote:
> VSZ is the Virtual Memory Size. It includes all memory that the process
> can access, including memory that is swapped out, memory that is
> allocated, but not used, and memory that is from shared libraries.""

Given this:

== cut ==

#include 
#include 

void main(void)
{
char* a = malloc(4l*1024l*1024l*1024l);
sleep(1200);
}

== cut ==

One can expect the process to "consume" 4Gb of VSZ, and about 700kb of
RSS.

Do you really suggest to treat all this VSZ as a "used memory"? A hint -
compile the sample, run it a hundred times.

Reco

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Peter Wiersig]

Martin Schwarz  writes:
> root@rad-m2m-srv02:~# ps aux --sort=-rss | head -15

you're choosing the wrong sort field to debug your problem here:
man ps:
"""
   rss RSS   resident set size, the non-swapped physical memory 
that a task has used (in kiloBytes).
 (alias rssize, rsz).
...
   vsz VSZ   virtual memory size of the process in KiB 
(1024-byte units).  Device mappings are currently
 excluded; this is subject to change. (alias 
vsize)."""

try the latter field if the problem is repeating.  VSZ can be very
misleading with graphical processes, but that should not occur here.

https://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management

https://stackoverflow.com/a/21049737/2911961
""RSS is the Resident Set Size and is used to show how much memory is
allocated to that process and is in RAM. It does not include memory that
is swapped out. It does include memory from shared libraries as long as
the pages from those libraries are actually in memory. It does include
all stack and heap memory.

VSZ is the Virtual Memory Size. It includes all memory that the process
can access, including memory that is swapped out, memory that is
allocated, but not used, and memory that is from shared libraries.""

Peter

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Reco]

Hi.

On Tue, Apr 16, 2019 at 02:30:57PM +0200, Martin Schwarz wrote:
> > slabtop from "procps" package, definitely.
> > Should've thought of it earlier.
> 
> I did take a look at slaptop (and at /proc/slabinfo) before. But since the
> values for "SReclaimable" and "SUnreclaim" from /proc/meminfo are not high
> enough to explain the memory consumption, I didn't investigate any further in
> that direction.
> 
> Looks unsuspicious to me:

Agreed. Nothing here that explains the behaviour observed.


> ['perf top' and 'perf record']
> > It's very indirect method. Basically it shows internal kernel functions
> > used, which may or may not be the source of the leak.
> 
> I'm afraid 'perf' is beyond my knowledge. In its default invocation, 'perf_4.9
> top' seems to be more focussed on CPU usage?

Nope, it's deeper than this. What "perf top" should show by default is
the kernel and userspace library functions called by every kernel thread
and every userspace process. For instance,

  81.98%  [kernel]  [k] load_balance
  18.02%  [kernel]  [k] __tick_nohz_idle_enter
   0.00%  [kernel]  [k] module_get_kallsym

This shows mostly idle OS, and the kernel calls load_balance() 81% of
the time, and __tick_nohz_idle_enter() 18% of the time. It eats CPU, of
course, but CPU consumption is not a concern here. The answer to the
question 'what the kernel does' is.


> And 'perf_4.9 record' is used to profile a certain command?

'perf record -a' in this case. All userspace and the kernel.
Terminate it with Ctrl+C after a minute or two.
Sorry if I didn't mention that.


> Tried with "sleep 30" as command, but not sure how
> to interpret the recording.

'perf report' from the same directory where it wrote "pref.data"
earlier.


> So I'm really unsure how to use them to further drill into kernel or module
> memory usage. Any hints?

Send it here, unless it's classified or somehow private. You'll need
those anyway, because...

> 
> > There's this SystemTap thing that can presumably do it better, but last
> > time I've checked it required a debug version of the kernel (and that's
> > unsuitable for just about anything short of kernel development).
> 
> By then I should probably go to debian-kernel? ;-)

I'd file a bug against linux-image package. You're using only in-tree
kernel modules shipped by Debian. The kernel is obviously misbehaving.
And if they won't answer you in a week (it's a small volunteer team
after all) - then you'll go at debian-kernel with a bug number at hands.

Reco

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Martin Schwarz]

Hello,

On Tue, Apr 16, 2019 at 11:49:57AM +0300, Reco wrote:
> I see nothing unusual short of somewhat low amount of RAM by today's
> standards.
> It's no excuse for the kernel to behave that way, for obvious reasons.

yes, 1 GB is not that much - but should be more than enough for the two
small RADIUSd instances (some 80 MB). Of course, if the workload
requires it, we can assign more RAM. In fact, that was what we first did
when the problem started to show, but this only lasted so long until the
additional RAM was consumed as well - it just takes a bit longer.

> slabtop from "procps" package, definitely.
> Should've thought of it earlier.

I did take a look at slaptop (and at /proc/slabinfo) before. But since the
values for "SReclaimable" and "SUnreclaim" from /proc/meminfo are not high
enough to explain the memory consumption, I didn't investigate any further in
that direction.

Looks unsuspicious to me:

root@rad-wgv-srv01:~# slabtop --once --sort=s | head -15
 Active / Total Objects (% used): 144152 / 186846 (77,2%)
 Active / Total Slabs (% used)  : 11205 / 11206 (100,0%)
 Active / Total Caches (% used) : 66 / 116 (56,9%)
 Active / Total Size (% used)   : 40726,60K / 46301,70K (88,0%)
 Minimum / Average / Maximum Object : 0,02K / 0,25K / 4096,00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME   
 0  0   0% 4096,00K  01 0K kmalloc-4194304
 0  0   0% 4096,00K  01 0K dma-kmalloc-4194304
 0  0   0% 2048,00K  01 0K kmalloc-2097152
 0  0   0% 2048,00K  01 0K dma-kmalloc-2097152
 0  0   0% 1024,00K  01 0K kmalloc-1048576
 0  0   0% 1024,00K  01 0K dma-kmalloc-1048576
 0  0   0%  512,00K  01 0K kmalloc-524288 
 0  0   0%  512,00K  01 0K dma-kmalloc-524288 
root@rad-wgv-srv01:~# 

> > Only I can't unload modules that are in use, of course. Unloading
> > vmw_balloon, vmw_vmci, and vmw_vsock_vmci_transport didn't help.
> 
> I doubt that these are the problem. Unless you're changing VM's RAM at
> runtime (and you wrote you don't).

We sometimes do increase a VM's RAM while it is running. But most of the VMs
showing the problem still have their initial memory size from the template they
were deployed from.

['perf top' and 'perf record']
> It's very indirect method. Basically it shows internal kernel functions
> used, which may or may not be the source of the leak.

I'm afraid 'perf' is beyond my knowledge. In its default invocation, 'perf_4.9
top' seems to be more focussed on CPU usage?  And 'perf_4.9 record' is used to
profile a certain command? Tried with "sleep 30" as command, but not sure how
to interpret the recording.

So I'm really unsure how to use them to further drill into kernel or module
memory usage. Any hints?

> There's this SystemTap thing that can presumably do it better, but last
> time I've checked it required a debug version of the kernel (and that's
> unsuitable for just about anything short of kernel development).

By then I should probably go to debian-kernel? ;-)

Thanks again for all your help!

Martin

-- 
Martin Schwarz * Karlsruhe, Germany * http://kuroi.de/

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Reco]

Hi.

On Tue, Apr 16, 2019 at 10:22:16AM +0200, Martin Schwarz wrote:
> Hello,
> 
> On Mon, Apr 15, 2019 at 07:03:13PM +0300, Reco wrote:
> [...]
> > What I suspect is happening here is runaway memory allocation by a
> > kernel module (at least one of them), and said kernel module is likely
> > to be VMWare-specific.
> > It could be vmxnet3 (network). It could be that LSI kernel module or
> > whatever they're using for SCSI these days (vmw_pvscsi?).
> 
> sounds interesting.
> 
> That would explain why I haven't seen this problem on one of my (few)
> personal Stretch installations running as Xen DomU. But then, I guess
> we're not the only ones who use Debian Stretch on VMware ESXi ;-)
> but haven't found any mention of this problem. I wonder what makes our
> setup so special ...

I see nothing unusual short of somewhat low amount of RAM by today's
standards.
It's no excuse for the kernel to behave that way, for obvious reasons.


> Here's the outout from lsmod:

Nothing unexpected here.


> Is there a way to show the memory consumed by each module? (besides the
> 'perf' tool you recommend below)

slabtop from "procps" package, definitely.
Should've thought of it earlier.


> Would memory consumed by a module be released when the module is
> unloaded? I guess so.

Barring kernel memory leaks - yes. Yep, they "invented" them too.
A price to pay if you write a kernel in C.


> Only I can't unload modules that are in use, of course. Unloading
> vmw_balloon, vmw_vmci, and vmw_vsock_vmci_transport didn't help.

I doubt that these are the problem. Unless you're changing VM's RAM at
runtime (and you wrote you don't).
If you can unload it - it's not used, hence no kernel memory allocations
worthy of speaking.


> > And that means - 'perf top', or better yet - 'perf record'.
> 
> I have never used perf before, will look into it.

It's very indirect method. Basically it shows internal kernel functions
used, which may or may not be the source of the leak.

There's this SystemTap thing that can presumably do it better, but last
time I've checked it required a debug version of the kernel (and that's
unsuitable for just about anything short of kernel development).

Reco

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Martin Schwarz]

Hello,

On Mon, Apr 15, 2019 at 07:03:13PM +0300, Reco wrote:
[...]
> What I suspect is happening here is runaway memory allocation by a
> kernel module (at least one of them), and said kernel module is likely
> to be VMWare-specific.
> It could be vmxnet3 (network). It could be that LSI kernel module or
> whatever they're using for SCSI these days (vmw_pvscsi?).

sounds interesting.

That would explain why I haven't seen this problem on one of my (few)
personal Stretch installations running as Xen DomU. But then, I guess
we're not the only ones who use Debian Stretch on VMware ESXi ;-)
but haven't found any mention of this problem. I wonder what makes our
setup so special ...

Yes, we're using vmw_pvscsi (VMware Paravirtual SCSI controller).

Here's the outout from lsmod:

msch@rad-wgv-srv01:~$ lsmod 
Module  Size  Used by
tcp_diag   16384  0
inet_diag  20480  1 tcp_diag
ppdev  20480  0
vmw_balloon20480  0
joydev 20480  0
evdev  24576  1
pcspkr 16384  0
serio_raw  16384  0
vmwgfx237568  1
ttm98304  1 vmwgfx
drm_kms_helper155648  1 vmwgfx
sg 32768  0
drm   360448  4 vmwgfx,ttm,drm_kms_helper
shpchp 36864  0
parport_pc 28672  0
parport49152  2 parport_pc,ppdev
ac 16384  0
button 16384  0
vmw_vsock_vmci_transport28672  0
vsock  36864  1 vmw_vsock_vmci_transport
vmw_vmci   69632  2 vmw_balloon,vmw_vsock_vmci_transport
ip_tables  24576  0
x_tables   36864  1 ip_tables
autofs440960  2
ext4  585728  3
crc16  16384  1 ext4
jbd2  106496  1 ext4
crc32c_generic 16384  0
fscrypto   28672  1 ext4
ecb16384  0
glue_helper16384  0
lrw16384  0
gf128mul   16384  1 lrw
ablk_helper16384  0
cryptd 24576  1 ablk_helper
aes_x86_64 20480  0
mbcache16384  4 ext4
dm_mod118784  16
sr_mod 24576  0
cdrom  61440  1 sr_mod
sd_mod 49152  2
ata_generic16384  0
crc32c_intel   24576  6
psmouse   135168  0
vmxnet361440  0
ata_piix   36864  0
vmw_pvscsi 24576  1
i2c_piix4  24576  0
libata249856  2 ata_piix,ata_generic
scsi_mod  225280  5 sd_mod,libata,sr_mod,sg,vmw_pvscsi
floppy 69632  0
msch@rad-wgv-srv01:~$ 

Is there a way to show the memory consumed by each module? (besides the
'perf' tool you recommend below)

Would memory consumed by a module be released when the module is
unloaded? I guess so. Only I can't unload modules that are in use, of
course. Unloading vmw_balloon, vmw_vmci, and vmw_vsock_vmci_transport
didn't help.

> And that means - 'perf top', or better yet - 'perf record'.

I have never used perf before, will look into it.

Thanks a lot for your insight!

Martin

-- 
Martin Schwarz * Karlsruhe, Germany * http://kuroi.de/

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Reco]

Hi.

On Mon, Apr 15, 2019 at 04:40:56PM +0200, Martin Schwarz wrote:
> The system from my previous example has already been rebooted, sorry!

Kind of expected. It's useful nevertheless.


> But here's from another system that currently starts showing the same
> problem and has an equally small workload:
> 
> root@rad-wgv-srv01:~# free -thwl

Nothing out of the ordinary here.

> root@rad-wgv-srv01:~# cat /proc/meminfo
> MemTotal:1010976 kB
> MemFree:   73980 kB
> MemAvailable:  38756 kB
> Buffers:9964 kB
> Cached:50340 kB

It's not the file cache who ate the memory.

> SwapCached: 2728 kB

And it's not the swap caching.

> Active(anon):  11068 kB
> Inactive(anon): 3696 kB

Memory consumption cannot be attributed to tmpfs.
I know, you've posted 'df' output earlier, but it does not take mount
namespaces into the account.

> Mapped:19904 kB

To my biggest disappointment, the problem cannot be explained by
excessive use of mmap(2) syscall. Would be easy otherwise.


> Shmem:  1120 kB

It's not the shared memory segments.


> Slab:  90744 kB
> SReclaimable:  13100 kB
> SUnreclaim:77644 kB

And it's not dentries cache (saw the thing grown once or twice. was
ugly).


> AnonHugePages: 0 kB
> ShmemHugePages:0 kB
> ShmemPmdMapped:0 kB
> HugePages_Total:   0
> HugePages_Free:0
> HugePages_Rsvd:0
> HugePages_Surp:0

And last, but not the least, there are no hugepages in use.


> root@rad-wgv-srv01:~# smem -tm | tail
> /bin/bash3  358 1076 
> /lib/systemd/systemd 3  386 1158 
> /lib/x86_64-linux-gnu/libc-2.24.so  33   54 1783 
> /usr/lib/x86_64-linux-gnu/libcrypto.so.1 5  386 1933 
> /usr/bin/python2.7   1 2220 2220 
> /lib/systemd/libsystemd-shared-232.so5  544 2723 
>  33  146 4848 
> [heap]  33  30410060 
> -
> 179922041011 

Moreover, no current running visible process consume the memory.
I suspect that this host does not utilize them anyway.


In short. I do believe that this is happening, but I never seen anything
like this. I cannot imagine the scenario that can lead to this, as long
as we're talking real hardware aka big iron.

What I suspect is happening here is runaway memory allocation by a
kernel module (at least one of them), and said kernel module is likely
to be VMWare-specific.
It could be vmxnet3 (network). It could be that LSI kernel module or
whatever they're using for SCSI these days (vmw_pvscsi?).


And that means - 'perf top', or better yet - 'perf record'.

Reco

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Martin Schwarz]

On Mon, Apr 15, 2019 at 10:44:26AM -0400, Kenneth Parker wrote:
> I had a Symptom like this a few years ago, which was tracked to something
> called "zram", which tries to use "excess RAM" as Swap Space.

thanks for your input!

We do not use zram. (I assume that would also show up in `lsmod`?)

/proc/swap shows only one "normal" swap partition (on a LVM logical
volume in this case):


root@rad-wgv-srv01:~# cat /proc/swaps
FilenameTypeSizeUsedPriority
/dev/dm-3   partition   2097148 70876   -1
root@rad-wgv-srv01:~# fgrep swap /etc/fstab 
/dev/mapper/vg0-lv_swap noneswapsw  0   0
root@rad-wgv-srv01:~# ls -l /dev/mapper/vg0-lv_swap
lrwxrwxrwx 1 root root 7 Mär 29 08:33 /dev/mapper/vg0-lv_swap -> ../dm-3
root@rad-wgv-srv01:~# ls -l /dev/dm-3
brw-rw 1 root disk 254, 3 Mär 29 08:33 /dev/dm-3
root@rad-wgv-srv01:~# lvs /dev/mapper/vg0-lv_swap
  LV  VG  Attr   LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync 
Convert
  lv_swap vg0 -wi-ao 2,00g  
  
root@rad-wgv-srv01:~# lsmod | fgrep z
Module  Size  Used by
root@rad-wgv-srv01:~# 

-- 
Martin Schwarz * Karlsruhe, Germany * http://kuroi.de/

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Kenneth Parker]

I had a Symptom like this a few years ago, which was tracked to something
called "zram", which tries to use "excess RAM" as Swap Space.

If so, it would show up on /proc/swaps

Verify that.

Best regards,

Kenneth Parker

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Martin Schwarz]

On Mon, Apr 15, 2019 at 04:35:27PM +0300, Reco wrote:
> Can you please provide unsorted outputs of /proc/meminfo? It's easier to
> compare them if they are unsorted.
> And "smem -tm | tail" would be helpful too.

Thanks for your input!

The system from my previous example has already been rebooted, sorry!
But here's from another system that currently starts showing the same
problem and has an equally small workload:


root@rad-wgv-srv01:~# free -thwl
  totalusedfree  shared buffers   cache 
  available
Mem:   987M843M 72M1,1M9,7M 61M 
37M
Low:   987M914M 72M
High:0B  0B  0B
Swap:  2,0G 75M1,9G
Total: 3,0G919M2,0G
root@rad-wgv-srv01:~# cat /proc/meminfo
MemTotal:1010976 kB
MemFree:   73980 kB
MemAvailable:  38756 kB
Buffers:9964 kB
Cached:50340 kB
SwapCached: 2728 kB
Active:58776 kB
Inactive:  15164 kB
Active(anon):  11068 kB
Inactive(anon): 3696 kB
Active(file):  47708 kB
Inactive(file):11468 kB
Unevictable:   0 kB
Mlocked:   0 kB
SwapTotal:   2097148 kB
SwapFree:2019416 kB
Dirty:   104 kB
Writeback: 0 kB
AnonPages: 13048 kB
Mapped:19904 kB
Shmem:  1120 kB
Slab:  90744 kB
SReclaimable:  13100 kB
SUnreclaim:77644 kB
KernelStack:2700 kB
PageTables: 3764 kB
NFS_Unstable:  0 kB
Bounce:0 kB
WritebackTmp:  0 kB
CommitLimit: 2602636 kB
Committed_AS: 155208 kB
VmallocTotal:   34359738367 kB
VmallocUsed:   0 kB
VmallocChunk:  0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages:0 kB
ShmemPmdMapped:0 kB
HugePages_Total:   0
HugePages_Free:0
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize:   2048 kB
DirectMap4k:  925568 kB
DirectMap2M:  122880 kB
root@rad-wgv-srv01:~# smem -tm | tail
/bin/bash3  358 1076 
/lib/systemd/systemd 3  386 1158 
/lib/x86_64-linux-gnu/libc-2.24.so  33   54 1783 
/usr/lib/x86_64-linux-gnu/libcrypto.so.1 5  386 1933 
/usr/bin/python2.7   1 2220 2220 
/lib/systemd/libsystemd-shared-232.so5  544 2723 
 33  146 4848 
[heap]  33  30410060 
-
179922041011 
root@rad-wgv-srv01:~# ps aux --sort=-rss | head -15
USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
root 61714  0.0  0.6  95180  6676 ?Ss   16:35   0:00 sshd: msch 
[priv]
msch 61716  0.0  0.5  64832  5852 ?Ss   16:35   0:00 
/lib/systemd/systemd --user
msch 61723  0.0  0.4  95452  4996 ?S16:35   0:00 sshd: 
msch@pts/0
msch 61724  0.2  0.4  23636  4924 pts/0Ss   16:35   0:00 -bash
root 62015  1.0  0.4  23740  4892 pts/0S16:36   0:00 -bash
root 1  0.0  0.4  57128  4440 ?Ss   Mär29  14:30 
/lib/systemd/systemd --system --deserialize 19
root 62014  0.0  0.3  51964  3836 pts/0S16:36   0:00 sudo -i
root 62040  0.0  0.3  41164  3584 pts/0R+   16:37   0:00 ps aux 
--sort=-rss
root   228  0.1  0.2 136620  2884 ?Ss   Mär29  31:20 
/usr/bin/vmtoolsd
root   224  0.0  0.2  6  2560 ?Ss   Mär29  13:48 
/lib/systemd/systemd-journald
root   411  0.0  0.2  46520  2408 ?Ss   Mär29   6:40 
/lib/systemd/systemd-logind
message+   419  0.0  0.1  45200  1332 ?Ss   Mär29   6:39 
/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile 
--systemd-activation
msch 61717  0.0  0.1  82652  1268 ?S16:35   0:00 (sd-pam)
_chrony531  0.0  0.1  29980  1184 ?SMär29   0:52 
/usr/sbin/chronyd
root@rad-wgv-srv01:~# uname -a
Linux rad-wgv-srv01 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3 (2019-02-02) x86_64 
GNU/Linux
root@rad-wgv-srv01:~# 


(any other commands I should repeat on the new system?)

Thanks
Martin

-- 
Martin Schwarz * Karlsruhe, Germany * http://kuroi.de/

Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Reco]

Hi.

On Mon, Apr 15, 2019 at 02:21:16PM +0200, Martin Schwarz wrote:
> I need help debugging/solving a weird memory problem. The symptoms are
> the usual ones for high memory usage: free/available memory is getting
> low, systems start swapping, disk I/O increases, performance drops.

Can you please provide unsorted outputs of /proc/meminfo? It's easier to
compare them if they are unsorted.
And "smem -tm | tail" would be helpful too.

Reco

Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)

[Martin Schwarz]

Hello,

(please let me know if this is more appropriate somewhere else, e.g. on
ebian-kernel)

I need help debugging/solving a weird memory problem. The symptoms are
the usual ones for high memory usage: free/available memory is getting
low, systems start swapping, disk I/O increases, performance drops.

However, from what I can see, the memory is not used up by user space
processes but from the Kernel (NOT caches/buffers), see commands output
at the end.

I'm still puzzled about what exactly eats all the RAM and how to reclaim
it (without rebooting the machine, of course!). Any help would be highly
appreciated!

Some findings so far:

- same problem on many systems, all Debian 9 Stretch, all running stock
  4.9 kernel from the official package, all amd64 virtual machines on
  several (different) VMware ESXi hosts.
- not all Stretch systems seem to be affected, but we haven't yet found
  the common ground.
- problem can occur after some days or some weeks, not at the same time
  on all affected machines. And not at the same time for all VMs on the
  same host
- problem only occurs on Stretch systems, not Jessie, even running on
  the same host.
- we haven't yet seen the problem on real hardware machines, only VMs
  (but since the vast majority of our systems are VMs, this may not be
  relevant)
- problem seems not directly related to the machine's load. it occurs on
  machines that are mostly idle as well as on more heavily-loaded
  systems
- problem occurs the same on single-core VMs as well as on multi-core
  VMs
- problem occurs the same on VMs running on single-socket hosts as well
  as on multi-socket hosts
- problem occurs the same on VMs running on hosts with different
  hypervisor releases, both VMware ESXi 5.5 and 6.5, both standalone and
  in a vSphere cluster.

Here's the output from some commands I hope to be helpful:

The machine in this example is a RADIUS server but has not even gone
productive ... no incoming client requests yet.  (But the problem is not
related to the RADIUS server software - OSC Radiator - since the same
symptoms show on different machines: not only RADIUS servers but also
nameservers, shell servers or jumphosts, etc.)

[values while the problem persists:]

root@rad-m2m-srv02:~# free -thwl
  totalusedfree  shared buffers   cache 
  available
Mem:   987M910M 59M  0B704K 16M 
13M
Low:   987M927M 59M
High:0B  0B  0B
Swap:  2,0G345M1,7G
Total: 3,0G1,2G1,7G
root@rad-m2m-srv02:~# smem -twk
Area   Used  Cache   Noncache 
firmware/hardware 0  0  0 
kernel image  0  0  0 
kernel dynamic memory914.9M  11.1M 903.8M 
userspace memory  13.0M   5.5M   7.4M 
free memory   59.4M  59.4M  0 
--
 987.3M  76.1M 911.2M 
root@rad-m2m-srv02:~# smem -uktr
User Count Swap  USS  PSS  RSS 
root39   332.8M10.4M12.4M44.7M 
msch 6 7.0M0   607.0K 8.3M 
_chrony  1   360.0K 4.0K20.0K   572.0K 
messagebus 1   580.0K 4.0K17.0K   480.0K 
postfix  2 1.6M013.0K   568.0K 
daemon   1   208.0K 4.0K 6.0K72.0K 
---
50   342.5M10.4M13.0M54.7M 
root@rad-m2m-srv02:~# sort -k2,2nr /proc/meminfo
VmallocTotal:   34359738367 kB
CommitLimit: 2602636 kB
SwapTotal:   2097148 kB
SwapFree:1741028 kB
MemTotal:1010976 kB
DirectMap4k: 1007488 kB
Committed_AS: 465128 kB
Slab:  79680 kB
SUnreclaim:69268 kB
MemFree:   61068 kB
DirectMap2M:   40960 kB
SReclaimable:  10412 kB
Active: 6944 kB
Inactive:   6660 kB
AnonPages:  6608 kB
PageTables: 5804 kB
Cached: 5748 kB
Mapped: 4660 kB
SwapCached: 3988 kB
Active(file):   3920 kB
Inactive(anon): 3828 kB
Active(anon):   3024 kB
KernelStack:2992 kB
Inactive(file): 2832 kB
Hugepagesize:   2048 kB
Buffers:1020 kB
Dirty: 8 kB
AnonHugePages: 0 kB
Bounce:0 kB
HardwareCorrupted: 0 kB
HugePages_Free:0
HugePages_Rsvd:0
HugePages_Surp:0
HugePages_Total:   0
MemAvailable:  0 kB
Mlocked:   0 kB
NFS_Unstable:  0 kB
Shmem: 0 kB
ShmemHugePages:0 kB
ShmemPmdMapped:0 kB
Unevictable:   0 kB
VmallocChunk:  0 kB
VmallocUsed:   0 kB
Writeback: 0 kB
WritebackTmp

Re: Inexplicable memory usage after move to Debian9

[Simon Beirnaert]

I've been out for a bit, I'll bundle multiple replies in a single mail.
Before I start: thank you to everybody taking the time to respond in this
thread :)



How are you determining what you call "consumed memory"?


Memory which isn't available to the system. So "used" minus "buffers/cache"


Keep in mind that the kernel will by default use almost all free
memory (not actually used by processes and libraries) as cache
space, because it makes no sense to leave memory just laying around.
However, once it's really needed, the caches will be dropped. Thus
"free" memory is usually reported as low. Compare with "available"
memory as reported by free.


Yup. I'm aware of this and it's not the issue I'm trying to solve. Disk cache
is good.





The free's 'available' value is computed without taking "SUnreclaim"
(unreclaimable kernel slabs) value from /proc/meminfo.
The difference is not that great, usually, but for NFS server it can
lead to funny things like "I have plenty of free swap and OOM killer was
invoked, despite 'available' telling me there's plenty of free RAM".
Can happen with Java too, as OP e-mail shows us.


Unfortunately the system I was currently seeing the issues on has apparently
been rebooted by the client, so right now I don't have a system to verify,
but it sounds like this is what I'm on about. I'll verify as soon as another
system with issues pops up.


Happened to me with kernel 4.9.0-5, continued with kernel 4.9.0-6, seems
to be solved by upgrading to backported kernel version 4.14.


Hm. This could bite me in the ass. Thanks for your feedback.





The last line from smem sticks out with high usage figures:
566 jetty /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804


Java is actually consuming the expected amount of RAM for the settings we
start it with. Also, the hight memory usage presists after shutting down
Jetty (and pretty much any other service), which is why I was hinting at
possible kernel issues (not much else was running).

Re: Inexplicable memory usage after move to Debian9

[Selim T . Erdoğan]

On Thu, Apr 26, 2018 at 03:09:15PM +0200, Simon Beirnaert wrote:
> Hi Recently I've started moving a fleet of Debian 7, 32-bit machines over to
> Debian 9, 64-bit. This migration is done by creating a fresh Debian 9 image
> with the necessary services, moving over user data (some wars and the
> content of /home) and rebooting into the new OS.
> 
> Relevant services (ones we manage and use) are:
> 
> - Jetty- Puppet - SSH - AutoSSH - NewRelic Infrastructure Through Puppet, we
> enforce system configuration is pretty much identical, save for stuff like
> host names and SSH keys. Now, we notice that on some systems, the RAM usage
> is way higher than expected, to the point where system memory is exhausted
> and processes (are) terminate(d). Investigation into what is causing this,
> leaves me at a dead end. I can't figure out where the memory is being
> consumed. Even after quitting all services we manage (leaving a "clean"
> installation), RAM usage on the system hovers just over 600MB, half a gig
> over what the same exact image consumes just after boot. The only fix I
> found so far is to reboot the system. The systems have ~1.8GB of system
> memory available. We don't use swap. Enabling swap gives the system some
> breathing room. On one system I enabled a swap volume of 512MB, which the
> kernel fills up and leaves filled up indefinitely. This points me to unused
> memory being allocated by mistake. A memory leak in the kernel, maybe? Below
> I've posted the output of some of the things I checked (with all services
> online). I also searched the internet and reached a topic about slab
> allocation (1). However, that didn't seem to solve anything. Can anyone here
> point me to some more stuff I can check out or try to debug this? Thanks!
> Simon root@mysystem:~# uname -a Linux mysystem 4.9.0-6-amd64 #1 SMP Debian
> 4.9.82-1+deb9u2 (2018-02-21) x86_64 GNU/Linux root@mysystem:~# free -m total
> used free shared buff/cache available Mem: 1831 1091 238 16 501 520 Swap:
> 511 511 0 root@mysystem:~# vmstat 1 procs ---memory--
> ---swap-- -io -system-- --cpu- r b swpd free buff cache si
> so bi bo in cs us sy id wa st 2 0 524268 244480 46044 467248 1 2 2137 36 97
> 35 9 3 85 3 0 0 0 524268 244092 46052 467244 0 0 0 16 1125 2406 3 3 94 1 0 0
> 0 524268 244092 46060 467284 0 0 4 208 1230 3906 4 3 93 0 0 0 0 524268
> 244084 46068 467256 0 0 0 32 1003 1990 1 2 97 0 0 0 0 524268 244180 46068
> 467256 0 0 0 0 1099 2121 4 1 95 0 0 0 0 524268 244204 46076 467272 0 0 0 20
> 1000 1978 1 2 97 1 0 0 0 524268 244080 46076 467272 0 0 0 0 1135 2315 2 2 96
> 0 0 0 0 524268 244080 46084 467264 0 0 0 16 1079 2103 1 3 96 1 0 0 0 524268
> 244080 46092 467272 0 0 0 56 1002 1973 2 2 96 0 0 1 0 524268 244080 46100
> 467264 0 0 0 16 997 1979 1 2 97 1 0 0 0 524268 244144 46100 467268 0 0 0 0
> 988 1957 1 2 97 0 0 0 1 524268 244228 46108 467260 0 0 0 980 1292 2700 4 5
> 81 9 0 root@mysystem:~# smem PID User Command Swap USS PSS RSS 528 root
> /sbin/agetty -f /etc/issue. 148 4 4 8 554 myuser /usr/lib/autossh/autossh -o
> 80 24 40 184 220 root /lib/systemd/systemd-udevd 552 108 140 688 10665 root
> /usr/lib/autossh/autossh -o 0 104 143 648 432 root /usr/sbin/cron -f 164 120
> 147 500 434 root /usr/sbin/irqbalance --fore 252 156 172 508 12042 mail
> /usr/sbin/nullmailer-send - 120 144 219 1228 382 systemd-timesync
> /lib/systemd/systemd-timesy 456 152 301 1060 439 messagebus
> /usr/bin/dbus-daemon --syst 272 280 348 1116 430 root
> /lib/systemd/systemd-logind 380 192 448 1224 557 myuser /usr/bin/ssh -o
> StrictHostK 564 360 515 1472 216 root /sbin/lvmetad -f 188 484 517 1028
> 10668 root /usr/bin/ssh -o StrictHostK 0 736 825 1112 14225 ntp
> /usr/sbin/ntpd -p /var/run/ 0 808 849 1404 435 root /usr/sbin/rsyslogd -n
> 716 896 980 2036 11612 root /usr/sbin/sshd -D 0 856 991 1560 500 root
> /sbin/dhclient -4 -v -pf /r 840 928 1010 2068 1330 root sudo -i 0 920 1375
> 3548 1234 myuser -bash 0 600 1407 3684 1331 root -bash 0 640 1447 3712 1233
> myuser sshd: myuser@pts/0 0 300 1697 4568 1 root /sbin/init 524 1568 1935
> 3524 1227 root sshd: posios [priv] 0 1372 2984 6380 193 root
> /lib/systemd/systemd-journa 312 4304 4521 5828 8885 root /usr/bin/python
> /usr/bin/sm 0 9100 9452 11292 14574 root /usr/bin/newrelic-infra 1440 17348
> 17378 17828 520 root /opt/puppetlabs/puppet/bin/ 20636 40140 40168 40592 566
> jetty /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804 [1] 
> https://unix.stackexchange.com/questions/244735/why-are-slab-objects-not-reclaimed-automatically

Hi,

The last line from smem sticks out with high usage figures:
566 jetty /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804

I don't use jetty (or java) so don't know if this normal but
googling "openjdk memory usage for jetty" (or &quo

Re: Inexplicable memory usage after move to Debian9

[Cindy-Sue Causey]

On 4/27/18, Reco <recovery...@gmail.com> wrote:
>   Hi.
>
> On Fri, Apr 27, 2018 at 12:26:11PM +, Eduardo M KALINOWSKI wrote:
>> On sex, 27 abr 2018, Simon Beirnaert wrote:
>> > The bottom line for me is that I when I shut down everything I install
>> > and manage on the system, it's still conuming about half a gig more
>> > than a system running the exact same base image right after use,
>> > without
>> > the extra memory being accounted for by monitoring tools.
>>
>> How are you determining what you call "consumed memory"?
>
> OP's original e-mail mentioned free(1), vmstat(1) and smem(1).
> All three lie :).
>
>
>> Keep in mind that the kernel will by default use almost all free memory
>> (not
>> actually used by processes and libraries) as cache space, because it
>> makes
>> no sense to leave memory just laying around. However, once it's really
>> needed, the caches will be dropped. Thus "free" memory is usually
>> reported
>> as low. Compare with "available" memory as reported by free.


Yeah, I took a shot at trying to help with this one, too, via
searching apt for "monitor memory usage". Saw a couple possibilities,
but so far nothing's reflecting what I know is going on.

smemstat was the current standout. As normal user, it said my browser
was using ~60MB or so. smemstate via root user shows closer to 1GB.
That's a big difference...

Meanwhile in reality and on a fresh reboot, "free -m" will lurch
"used" from ~350MB to ~5 1/2 gigabytes as soon as all... open browser
tabs.. are refreshed (offline). User experience of sporadic crashes
lately backs that up as pretty much verified.

Apparently ~600 tabs this morning. When a browser flashes that warning
that opening more than a few tabs has potentially negative
consequences.. they ain't just whistling Dixie... :)

Cindy.
-- 
Cindy-Sue Causey
Talking Rock, Pickens County, Georgia, USA

* runs... in search of more (very) cheap RAM *

Re: Inexplicable memory usage after move to Debian9

[David Wright]

On Fri 27 Apr 2018 at 12:26:11 (+), Eduardo M KALINOWSKI wrote:
> On sex, 27 abr 2018, Simon Beirnaert wrote:
> >The bottom line for me is that I when I shut down everything I install
> >and manage on the system, it's still conuming about half a gig more
> >than a system running the exact same base image right after use, without
> >the extra memory being accounted for by monitoring tools.
> 
> How are you determining what you call "consumed memory"?
> 
> Keep in mind that the kernel will by default use almost all free
> memory (not actually used by processes and libraries) as cache
> space, because it makes no sense to leave memory just laying around.
> However, once it's really needed, the caches will be dropped. Thus
> "free" memory is usually reported as low. Compare with "available"
> memory as reported by free.

In which case, this could be a useful recipe:

# free -m
 total   used   free sharedbuffers cached
Mem:  2016   1251764  0301774
-/+ buffers/cache:175   1840
Swap: 3861  0   3861
# echo 3 > /proc/sys/vm/drop_caches
# free -m
 total   used   free sharedbuffers cached
Mem:  2016119   1896  0  0 24
-/+ buffers/cache: 94   1921
Swap: 3861  0   3861
# 

Now you can watch it creep up again.

Cheers,
David.

Re: Inexplicable memory usage after move to Debian9

[Reco]

Hi.

On Fri, Apr 27, 2018 at 12:26:11PM +, Eduardo M KALINOWSKI wrote:
> On sex, 27 abr 2018, Simon Beirnaert wrote:
> > The bottom line for me is that I when I shut down everything I install
> > and manage on the system, it's still conuming about half a gig more
> > than a system running the exact same base image right after use, without
> > the extra memory being accounted for by monitoring tools.
> 
> How are you determining what you call "consumed memory"?

OP's original e-mail mentioned free(1), vmstat(1) and smem(1).
All three lie :).


> Keep in mind that the kernel will by default use almost all free memory (not
> actually used by processes and libraries) as cache space, because it makes
> no sense to leave memory just laying around. However, once it's really
> needed, the caches will be dropped. Thus "free" memory is usually reported
> as low. Compare with "available" memory as reported by free.

The free's 'available' value is computed without taking "SUnreclaim"
(unreclaimable kernel slabs) value from /proc/meminfo.
The difference is not that great, usually, but for NFS server it can
lead to funny things like "I have plenty of free swap and OOM killer was
invoked, despite 'available' telling me there's plenty of free RAM".
Can happen with Java too, as OP e-mail shows us.

Happened to me with kernel 4.9.0-5, continued with kernel 4.9.0-6, seems
to be solved by upgrading to backported kernel version 4.14.

Reco

Re: Inexplicable memory usage after move to Debian9

[Eduardo M KALINOWSKI]

On sex, 27 abr 2018, Simon Beirnaert wrote:

The bottom line for me is that I when I shut down everything I install
and manage on the system, it's still conuming about half a gig more
than a system running the exact same base image right after use, without
the extra memory being accounted for by monitoring tools.


How are you determining what you call "consumed memory"?

Keep in mind that the kernel will by default use almost all free  
memory (not actually used by processes and libraries) as cache space,  
because it makes no sense to leave memory just laying around. However,  
once it's really needed, the caches will be dropped. Thus "free"  
memory is usually reported as low. Compare with "available" memory as  
reported by free.


--
Eduardo M KALINOWSKI
edua...@kalinowski.com.br

Re: Inexplicable memory usage after move to Debian9

[Simon Beirnaert]

I'm not sure that a ratio of 512MB swap to 1.8GB RAM really proves
anything. If the swap space matched RAM in size and still filled up, I
think that would be more definitive.


The bottom line for me is that I when I shut down everything I install
and manage on the system, it's still conuming about half a gig more
than a system running the exact same base image right after use, without
the extra memory being accounted for by monitoring tools.

Enabling swap just helped me realize that the allocated memory isn't
actively used, otherwise the system would be swapping it in/out more than
it is.

These machines (thousands) have been running on without swap on Debian 7
for a *long* time without issues. I wasn't intending on activating swap
now either. It's just a debugging step.

What I'm actually looking for is a way to trace where to which process
that mistery half-a-gig is allocated. As I said: I can't find a memory
list which acocunts for everything. This is what makes me think of a
memory leak somewhere. I just want to debug this. I'm not trying to imply
that I was solving anything by allocating RAM.


Also note this comment at the top of the stackexchange article:
"UPDATE: I'm no longer having this problem on 4.9.* Not sure when 
it was fixed."


I know. Just wanted to indicate that I had looked at slab allocation
issues, but it doesn't seem to be related


Thanks
Simon

Re: Inexplicable memory usage after move to Debian9

[Patrick Bartek]

On Thu, 26 Apr 2018 15:45:23 + Kenneth Parker <sea7k...@gmail.com>
wrote:

> With the help of a "minimal" Debian 8 System, I am learning SystemD.
> My guess was, simply due to more "Moving Parts", than the prior Boot
> Process. More Open Processes, mean, practically more Memory Usage.
> 
> Thanks!
> 
> Kenneth Parker

Kenneth,

If systemd is proving to be a headache, it can be easily replaced by
sysvinit in Stretch. As root:

apt-get install sysvinit-core
cp /usr/share/sysvinit/inittab /etc/inittab

is all it takes.  systemd libraries are not removed which some apps
require, no special repos are needed, etc.  The init part is just
removed. I've been running Stretch like this for months.  No problems.

Here's the link I used: 

http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_Stretch_installation

I just did the first Required steps, nothing else.

B


> On Thu, Apr 26, 2018, 11:18 AM David Wright <deb...@lionunicorn.co.uk>
> wrote:
> 
> > On Thu 26 Apr 2018 at 15:01:38 (+), Kenneth Parker wrote:
> > > Couldn't be SystemD, could it?   That wasn't in use, in Debian 7.
> > >
> > > Just a guess...
> > >
> > > Kenneth Parker
> >
> > That's probably about as useful as saying "What do you expect?
> > You've moved from 32-bit to 64-bit, so double your memory."
> >
> > > On Thu, Apr 26, 2018, 9:09 AM Simon Beirnaert <
> > > simon.beirna...@lightspeedhq.com> wrote:
> > >
> > > > Hi Recently I've started moving a fleet of Debian 7, 32-bit
> > > > machines over to Debian 9, 64-bit. This migration is done by
> > > > creating a fresh Debian 9 image with the necessary services,
> > > > moving over user data (some wars and the content of /home) and
> > > > rebooting into the new OS.
> >
> > Cheers,
> > David.
> >
> >

Re: Inexplicable memory usage after move to Debian9

[Kenneth Parker]

With the help of a "minimal" Debian 8 System, I am learning SystemD.  My
guess was, simply due to more "Moving Parts", than the prior Boot Process.
More Open Processes, mean, practically more Memory Usage.

Thanks!

Kenneth Parker

On Thu, Apr 26, 2018, 11:18 AM David Wright <deb...@lionunicorn.co.uk>
wrote:

> On Thu 26 Apr 2018 at 15:01:38 (+), Kenneth Parker wrote:
> > Couldn't be SystemD, could it?   That wasn't in use, in Debian 7.
> >
> > Just a guess...
> >
> > Kenneth Parker
>
> That's probably about as useful as saying "What do you expect?
> You've moved from 32-bit to 64-bit, so double your memory."
>
> > On Thu, Apr 26, 2018, 9:09 AM Simon Beirnaert <
> > simon.beirna...@lightspeedhq.com> wrote:
> >
> > > Hi Recently I've started moving a fleet of Debian 7, 32-bit machines
> > > over to Debian 9, 64-bit. This migration is done by creating a fresh
> > > Debian 9 image with the necessary services, moving over user data (some
> > > wars and the content of /home) and rebooting into the new OS.
>
> Cheers,
> David.
>
>

Re: Inexplicable memory usage after move to Debian9

[David Wright]

On Thu 26 Apr 2018 at 15:01:38 (+), Kenneth Parker wrote:
> Couldn't be SystemD, could it?   That wasn't in use, in Debian 7.
> 
> Just a guess...
> 
> Kenneth Parker

That's probably about as useful as saying "What do you expect?
You've moved from 32-bit to 64-bit, so double your memory."

> On Thu, Apr 26, 2018, 9:09 AM Simon Beirnaert <
> simon.beirna...@lightspeedhq.com> wrote:
> 
> > Hi Recently I've started moving a fleet of Debian 7, 32-bit machines
> > over to Debian 9, 64-bit. This migration is done by creating a fresh
> > Debian 9 image with the necessary services, moving over user data (some
> > wars and the content of /home) and rebooting into the new OS.

Cheers,
David.

Re: Inexplicable memory usage after move to Debian9

[Kenneth Parker]

Couldn't be SystemD, could it?   That wasn't in use, in Debian 7.

Just a guess...

Kenneth Parker

On Thu, Apr 26, 2018, 9:09 AM Simon Beirnaert <
simon.beirna...@lightspeedhq.com> wrote:

> Hi Recently I've started moving a fleet of Debian 7, 32-bit machines
> over to Debian 9, 64-bit. This migration is done by creating a fresh
> Debian 9 image with the necessary services, moving over user data (some
> wars and the content of /home) and rebooting into the new OS.
>
> Relevant services (ones we manage and use) are:
>
> - Jetty- Puppet - SSH - AutoSSH - NewRelic Infrastructure Through
> Puppet, we enforce system configuration is pretty much identical, save
> for stuff like host names and SSH keys. Now, we notice that on some
> systems, the RAM usage is way higher than expected, to the point where
> system memory is exhausted and processes (are) terminate(d).
> Investigation into what is causing this, leaves me at a dead end. I
> can't figure out where the memory is being consumed. Even after quitting
> all services we manage (leaving a "clean" installation), RAM usage on
> the system hovers just over 600MB, half a gig over what the same exact
> image consumes just after boot. The only fix I found so far is to reboot
> the system. The systems have ~1.8GB of system memory available. We don't
> use swap. Enabling swap gives the system some breathing room. On one
> system I enabled a swap volume of 512MB, which the kernel fills up and
> leaves filled up indefinitely. This points me to unused memory being
> allocated by mistake. A memory leak in the kernel, maybe? Below I've
> posted the output of some of the things I checked (with all services
> online). I also searched the internet and reached a topic about slab
> allocation (1). However, that didn't seem to solve anything. Can anyone
> here point me to some more stuff I can check out or try to debug this?
> Thanks! Simon root@mysystem:~# uname -a Linux mysystem 4.9.0-6-amd64 #1
> SMP Debian 4.9.82-1+deb9u2 (2018-02-21) x86_64 GNU/Linux
> root@mysystem:~# free -m total used free shared buff/cache available
> Mem: 1831 1091 238 16 501 520 Swap: 511 511 0 root@mysystem:~# vmstat 1
> procs ---memory-- ---swap-- -io -system--
> --cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st
> 2 0 524268 244480 46044 467248 1 2 2137 36 97 35 9 3 85 3 0 0 0 524268
> 244092 46052 467244 0 0 0 16 1125 2406 3 3 94 1 0 0 0 524268 244092
> 46060 467284 0 0 4 208 1230 3906 4 3 93 0 0 0 0 524268 244084 46068
> 467256 0 0 0 32 1003 1990 1 2 97 0 0 0 0 524268 244180 46068 467256 0 0
> 0 0 1099 2121 4 1 95 0 0 0 0 524268 244204 46076 467272 0 0 0 20 1000
> 1978 1 2 97 1 0 0 0 524268 244080 46076 467272 0 0 0 0 1135 2315 2 2 96
> 0 0 0 0 524268 244080 46084 467264 0 0 0 16 1079 2103 1 3 96 1 0 0 0
> 524268 244080 46092 467272 0 0 0 56 1002 1973 2 2 96 0 0 1 0 524268
> 244080 46100 467264 0 0 0 16 997 1979 1 2 97 1 0 0 0 524268 244144 46100
> 467268 0 0 0 0 988 1957 1 2 97 0 0 0 1 524268 244228 46108 467260 0 0 0
> 980 1292 2700 4 5 81 9 0 root@mysystem:~# smem PID User Command Swap USS
> PSS RSS 528 root /sbin/agetty -f /etc/issue. 148 4 4 8 554 myuser
> /usr/lib/autossh/autossh -o 80 24 40 184 220 root
> /lib/systemd/systemd-udevd 552 108 140 688 10665 root
> /usr/lib/autossh/autossh -o 0 104 143 648 432 root /usr/sbin/cron -f 164
> 120 147 500 434 root /usr/sbin/irqbalance --fore 252 156 172 508 12042
> mail /usr/sbin/nullmailer-send - 120 144 219 1228 382 systemd-timesync
> /lib/systemd/systemd-timesy 456 152 301 1060 439 messagebus
> /usr/bin/dbus-daemon --syst 272 280 348 1116 430 root
> /lib/systemd/systemd-logind 380 192 448 1224 557 myuser /usr/bin/ssh -o
> StrictHostK 564 360 515 1472 216 root /sbin/lvmetad -f 188 484 517 1028
> 10668 root /usr/bin/ssh -o StrictHostK 0 736 825 1112 14225 ntp
> /usr/sbin/ntpd -p /var/run/ 0 808 849 1404 435 root /usr/sbin/rsyslogd
> -n 716 896 980 2036 11612 root /usr/sbin/sshd -D 0 856 991 1560 500 root
> /sbin/dhclient -4 -v -pf /r 840 928 1010 2068 1330 root sudo -i 0 920
> 1375 3548 1234 myuser -bash 0 600 1407 3684 1331 root -bash 0 640 1447
> 3712 1233 myuser sshd: myuser@pts/0 0 300 1697 4568 1 root /sbin/init
> 524 1568 1935 3524 1227 root sshd: posios [priv] 0 1372 2984 6380 193
> root /lib/systemd/systemd-journa 312 4304 4521 5828 8885 root
> /usr/bin/python /usr/bin/sm 0 9100 9452 11292 14574 root
> /usr/bin/newrelic-infra 1440 17348 17378 17828 520 root
> /opt/puppetlabs/puppet/bin/ 20636 40140 40168 40592 566 jetty
> /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804 [1]
>
> https://unix.stackexchange.com/questions/244735/why-are-slab-objects-not-reclaimed-automatically
>
>
>

Re: Inexplicable memory usage after move to Debian9

[Nicholas Geovanis]

I'm not sure that a ratio of 512MB swap to 1.8GB RAM really proves
anything. If the swap space matched RAM in size and still filled up, I
think that would be more definitive. The bottom-line is that you need
to determine what's consuming it. Also note this comment at the top of
the stackexchange article:
"UPDATE: I'm no longer having this problem on 4.9.* Not sure when it was fixed."

On Thu, Apr 26, 2018 at 8:09 AM, Simon Beirnaert
 wrote:
> Hi Recently I've started moving a fleet of Debian 7, 32-bit machines over to
> Debian 9, 64-bit. This migration is done by creating a fresh Debian 9 image
> with the necessary services, moving over user data (some wars and the
> content of /home) and rebooting into the new OS.
>
> Relevant services (ones we manage and use) are:
>
> - Jetty- Puppet - SSH - AutoSSH - NewRelic Infrastructure Through Puppet, we
> enforce system configuration is pretty much identical, save for stuff like
> host names and SSH keys. Now, we notice that on some systems, the RAM usage
> is way higher than expected, to the point where system memory is exhausted
> and processes (are) terminate(d). Investigation into what is causing this,
> leaves me at a dead end. I can't figure out where the memory is being
> consumed. Even after quitting all services we manage (leaving a "clean"
> installation), RAM usage on the system hovers just over 600MB, half a gig
> over what the same exact image consumes just after boot. The only fix I
> found so far is to reboot the system. The systems have ~1.8GB of system
> memory available. We don't use swap. Enabling swap gives the system some
> breathing room. On one system I enabled a swap volume of 512MB, which the
> kernel fills up and leaves filled up indefinitely. This points me to unused
> memory being allocated by mistake. A memory leak in the kernel, maybe? Below
> I've posted the output of some of the things I checked (with all services
> online). I also searched the internet and reached a topic about slab
> allocation (1). However, that didn't seem to solve anything. Can anyone here
> point me to some more stuff I can check out or try to debug this? Thanks!
> Simon root@mysystem:~# uname -a Linux mysystem 4.9.0-6-amd64 #1 SMP Debian
> 4.9.82-1+deb9u2 (2018-02-21) x86_64 GNU/Linux root@mysystem:~# free -m total
> used free shared buff/cache available Mem: 1831 1091 238 16 501 520 Swap:
> 511 511 0 root@mysystem:~# vmstat 1 procs ---memory--
> ---swap-- -io -system-- --cpu- r b swpd free buff cache si
> so bi bo in cs us sy id wa st 2 0 524268 244480 46044 467248 1 2 2137 36 97
> 35 9 3 85 3 0 0 0 524268 244092 46052 467244 0 0 0 16 1125 2406 3 3 94 1 0 0
> 0 524268 244092 46060 467284 0 0 4 208 1230 3906 4 3 93 0 0 0 0 524268
> 244084 46068 467256 0 0 0 32 1003 1990 1 2 97 0 0 0 0 524268 244180 46068
> 467256 0 0 0 0 1099 2121 4 1 95 0 0 0 0 524268 244204 46076 467272 0 0 0 20
> 1000 1978 1 2 97 1 0 0 0 524268 244080 46076 467272 0 0 0 0 1135 2315 2 2 96
> 0 0 0 0 524268 244080 46084 467264 0 0 0 16 1079 2103 1 3 96 1 0 0 0 524268
> 244080 46092 467272 0 0 0 56 1002 1973 2 2 96 0 0 1 0 524268 244080 46100
> 467264 0 0 0 16 997 1979 1 2 97 1 0 0 0 524268 244144 46100 467268 0 0 0 0
> 988 1957 1 2 97 0 0 0 1 524268 244228 46108 467260 0 0 0 980 1292 2700 4 5
> 81 9 0 root@mysystem:~# smem PID User Command Swap USS PSS RSS 528 root
> /sbin/agetty -f /etc/issue. 148 4 4 8 554 myuser /usr/lib/autossh/autossh -o
> 80 24 40 184 220 root /lib/systemd/systemd-udevd 552 108 140 688 10665 root
> /usr/lib/autossh/autossh -o 0 104 143 648 432 root /usr/sbin/cron -f 164 120
> 147 500 434 root /usr/sbin/irqbalance --fore 252 156 172 508 12042 mail
> /usr/sbin/nullmailer-send - 120 144 219 1228 382 systemd-timesync
> /lib/systemd/systemd-timesy 456 152 301 1060 439 messagebus
> /usr/bin/dbus-daemon --syst 272 280 348 1116 430 root
> /lib/systemd/systemd-logind 380 192 448 1224 557 myuser /usr/bin/ssh -o
> StrictHostK 564 360 515 1472 216 root /sbin/lvmetad -f 188 484 517 1028
> 10668 root /usr/bin/ssh -o StrictHostK 0 736 825 1112 14225 ntp
> /usr/sbin/ntpd -p /var/run/ 0 808 849 1404 435 root /usr/sbin/rsyslogd -n
> 716 896 980 2036 11612 root /usr/sbin/sshd -D 0 856 991 1560 500 root
> /sbin/dhclient -4 -v -pf /r 840 928 1010 2068 1330 root sudo -i 0 920 1375
> 3548 1234 myuser -bash 0 600 1407 3684 1331 root -bash 0 640 1447 3712 1233
> myuser sshd: myuser@pts/0 0 300 1697 4568 1 root /sbin/init 524 1568 1935
> 3524 1227 root sshd: posios [priv] 0 1372 2984 6380 193 root
> /lib/systemd/systemd-journa 312 4304 4521 5828 8885 root /usr/bin/python
> /usr/bin/sm 0 9100 9452 11292 14574 root /usr/bin/newrelic-infra 1440 17348
> 17378 17828 520 root /opt/puppetlabs/puppet/bin/ 20636 40140 40168 40592 566
> jetty /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804 [1]
> https://unix.stackexchange.com/questions/244735/why-are-slab-objects-not-reclaimed-automatically
>

Inexplicable memory usage after move to Debian9

[Simon Beirnaert]

Hi Recently I've started moving a fleet of Debian 7, 32-bit machines 
over to Debian 9, 64-bit. This migration is done by creating a fresh 
Debian 9 image with the necessary services, moving over user data (some 
wars and the content of /home) and rebooting into the new OS.


Relevant services (ones we manage and use) are:

- Jetty- Puppet - SSH - AutoSSH - NewRelic Infrastructure Through 
Puppet, we enforce system configuration is pretty much identical, save 
for stuff like host names and SSH keys. Now, we notice that on some 
systems, the RAM usage is way higher than expected, to the point where 
system memory is exhausted and processes (are) terminate(d). 
Investigation into what is causing this, leaves me at a dead end. I 
can't figure out where the memory is being consumed. Even after quitting 
all services we manage (leaving a "clean" installation), RAM usage on 
the system hovers just over 600MB, half a gig over what the same exact 
image consumes just after boot. The only fix I found so far is to reboot 
the system. The systems have ~1.8GB of system memory available. We don't 
use swap. Enabling swap gives the system some breathing room. On one 
system I enabled a swap volume of 512MB, which the kernel fills up and 
leaves filled up indefinitely. This points me to unused memory being 
allocated by mistake. A memory leak in the kernel, maybe? Below I've 
posted the output of some of the things I checked (with all services 
online). I also searched the internet and reached a topic about slab 
allocation (1). However, that didn't seem to solve anything. Can anyone 
here point me to some more stuff I can check out or try to debug this? 
Thanks! Simon root@mysystem:~# uname -a Linux mysystem 4.9.0-6-amd64 #1 
SMP Debian 4.9.82-1+deb9u2 (2018-02-21) x86_64 GNU/Linux 
root@mysystem:~# free -m total used free shared buff/cache available 
Mem: 1831 1091 238 16 501 520 Swap: 511 511 0 root@mysystem:~# vmstat 1 
procs ---memory-- ---swap-- -io -system-- 
--cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 
2 0 524268 244480 46044 467248 1 2 2137 36 97 35 9 3 85 3 0 0 0 524268 
244092 46052 467244 0 0 0 16 1125 2406 3 3 94 1 0 0 0 524268 244092 
46060 467284 0 0 4 208 1230 3906 4 3 93 0 0 0 0 524268 244084 46068 
467256 0 0 0 32 1003 1990 1 2 97 0 0 0 0 524268 244180 46068 467256 0 0 
0 0 1099 2121 4 1 95 0 0 0 0 524268 244204 46076 467272 0 0 0 20 1000 
1978 1 2 97 1 0 0 0 524268 244080 46076 467272 0 0 0 0 1135 2315 2 2 96 
0 0 0 0 524268 244080 46084 467264 0 0 0 16 1079 2103 1 3 96 1 0 0 0 
524268 244080 46092 467272 0 0 0 56 1002 1973 2 2 96 0 0 1 0 524268 
244080 46100 467264 0 0 0 16 997 1979 1 2 97 1 0 0 0 524268 244144 46100 
467268 0 0 0 0 988 1957 1 2 97 0 0 0 1 524268 244228 46108 467260 0 0 0 
980 1292 2700 4 5 81 9 0 root@mysystem:~# smem PID User Command Swap USS 
PSS RSS 528 root /sbin/agetty -f /etc/issue. 148 4 4 8 554 myuser 
/usr/lib/autossh/autossh -o 80 24 40 184 220 root 
/lib/systemd/systemd-udevd 552 108 140 688 10665 root 
/usr/lib/autossh/autossh -o 0 104 143 648 432 root /usr/sbin/cron -f 164 
120 147 500 434 root /usr/sbin/irqbalance --fore 252 156 172 508 12042 
mail /usr/sbin/nullmailer-send - 120 144 219 1228 382 systemd-timesync 
/lib/systemd/systemd-timesy 456 152 301 1060 439 messagebus 
/usr/bin/dbus-daemon --syst 272 280 348 1116 430 root 
/lib/systemd/systemd-logind 380 192 448 1224 557 myuser /usr/bin/ssh -o 
StrictHostK 564 360 515 1472 216 root /sbin/lvmetad -f 188 484 517 1028 
10668 root /usr/bin/ssh -o StrictHostK 0 736 825 1112 14225 ntp 
/usr/sbin/ntpd -p /var/run/ 0 808 849 1404 435 root /usr/sbin/rsyslogd 
-n 716 896 980 2036 11612 root /usr/sbin/sshd -D 0 856 991 1560 500 root 
/sbin/dhclient -4 -v -pf /r 840 928 1010 2068 1330 root sudo -i 0 920 
1375 3548 1234 myuser -bash 0 600 1407 3684 1331 root -bash 0 640 1447 
3712 1233 myuser sshd: myuser@pts/0 0 300 1697 4568 1 root /sbin/init 
524 1568 1935 3524 1227 root sshd: posios [priv] 0 1372 2984 6380 193 
root /lib/systemd/systemd-journa 312 4304 4521 5828 8885 root 
/usr/bin/python /usr/bin/sm 0 9100 9452 11292 14574 root 
/usr/bin/newrelic-infra 1440 17348 17378 17828 520 root 
/opt/puppetlabs/puppet/bin/ 20636 40140 40168 40592 566 jetty 
/usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804 [1] 
https://unix.stackexchange.com/questions/244735/why-are-slab-objects-not-reclaimed-automatically 

Re: Memory usage in Debian Jessie (stable)

[Michael Biebl]

Am 08.05.2015 um 01:45 schrieb real bas:
 Hi guys,
 It's early to talk about but the memory usage it's too much (3.5 GB RAM)
 with Debian Jessie. I using ISO (cd-1) of debian.org and have this problem.
 
 The normal usage with Debian Wheezy (7.8.0) is 800~900mb

How did you measure memory usage? And no, top or free don't count.


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Re: Memory usage in Debian Jessie (stable)

[Michael Biebl]

Am 08.05.2015 um 02:53 schrieb real bas:
 I use the monitoring system.

What monitoring system?

 I talk with other users and have the same problem. Gnome is too heavy even
 for Classic

This blanket statement makes no sense.

That aside, I regularly run GNOME/jessie in a Virtualbox VM with 512MB.
Works just fine.


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Memory usage in Debian Jessie (stable)

[real bas]

Hi guys,
It's early to talk about but the memory usage it's too much (3.5 GB RAM)
with Debian Jessie. I using ISO (cd-1) of debian.org and have this problem.

The normal usage with Debian Wheezy (7.8.0) is 800~900mb, anyone has the
same problem and there is already a solution?

My hardware configuration is:
Processor I5, memory RAM 4 GB and graphic board 1 GB nvidia v346.59.

Re: Memory usage in Debian Jessie (stable)

[real bas]

I use the monitoring system.
I talk with other users and have the same problem. Gnome is too heavy even
for Classic

2015-05-07 20:14 GMT-04:00 Michael Biebl bi...@debian.org:

 Am 08.05.2015 um 01:45 schrieb real bas:
  Hi guys,
  It's early to talk about but the memory usage it's too much (3.5 GB RAM)
  with Debian Jessie. I using ISO (cd-1) of debian.org and have this
 problem.
 
  The normal usage with Debian Wheezy (7.8.0) is 800~900mb

 How did you measure memory usage? And no, top or free don't count.


 --
 Why is it that all of the instruments seeking intelligent life in the
 universe are pointed away from Earth?

Re: Memory usage in Debian Jessie (stable)

[Ric Moore]

On 05/07/2015 09:52 PM, real bas wrote:

Sorry, is the System Monitor (app default in Debian Jessie), monitoring
CPU, memory and network history. Besides monitor the processes


Please don't top post.

System Monitor is showing total mem usage, applications and everything. 
Did you upgrade to Jessie or install fresh? If you just upgraded, 
sometimes the system will be doing a lot of collection and cleanup and 
eat some memory in the process, for a short time. Has it returned to 
normal yet? top will show you if an app is being a hog. What desktop are 
you using? KDE goes ape when nepomuk tries to catalog everything you 
have. I avoid it like a Mandrake Root. :) Ric




--
My father, Victor Moore (Vic) used to say:
There are two Great Sins in the world...
..the Sin of Ignorance, and the Sin of Stupidity.
Only the former may be overcome. R.I.P. Dad.
http://linuxcounter.net/user/44256.html


--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Archive: https://lists.debian.org/554c454b.4040...@gmail.com

Re: Memory usage in Debian Jessie (stable)

[real bas]

Sorry, is the System Monitor (app default in Debian Jessie), monitoring
CPU, memory and network history. Besides monitor the processes

2015-05-07 21:22 GMT-04:00 Michael Biebl bi...@debian.org:

 Am 08.05.2015 um 02:53 schrieb real bas:
  I use the monitoring system.

 What monitoring system?

  I talk with other users and have the same problem. Gnome is too heavy
 even
  for Classic

 This blanket statement makes no sense.

 That aside, I regularly run GNOME/jessie in a Virtualbox VM with 512MB.
 Works just fine.


 --
 Why is it that all of the instruments seeking intelligent life in the
 universe are pointed away from Earth?

Mailman memory usage after dist-upgrade

[Daniel Galambos]

Hi,

I recently upgraded a maillist server from Squeeze to Wheezy. We had no
problem with it until the upgrade.

In 1-2 weeks mailman's OutgoingRunner and BounceRunner processes eat all
the memories and a lot of swap.
The box has 1,5GB RAM and 2G swap.

Mailman version is: 1:2.1.15-1
There are 2 lists with around 40k addresses with one letter at week, the
other lists are small.

Why are the qrunner processes eat all the memory, and what can I do
against it?
I don't believe that the solution is to write a cron job to restart
periodically.

thanks,

Dancsa


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/52822592.8020...@dancsa.hu

Re: Xorg high memory usage

[Alberto Luaces]

Veljko writes:

 On Tue, May 28, 2013 at 04:20:04PM +0200, Linux-Fan wrote:
 I have tried to reproduce the problem as well but my Xorg's memory usage
 stayed constant (about 50M), iceweasel's memory usage went from 180M to
 900M during the test. I used a periodic ps invocation to get the data
 pasted to http://paste.debian.net/plain/7070, this was the peak output:
 
 -- 28.05.2013 - 16:07:01 up 43 min, 4 users,
load average: 1.51, 0.85, 0.58 --
 COMMAND  C %MEMVSZ   RSS
 Xorg 3  0.9 166480 58300
 iceweasel   53 15.2 1714556 931832
 
 I am using Debian 7.0.0 Wheezy, Iceweasel is from Experimental.
 Window Manager: Fluxbox, xcompmgr enabled.
 Graphics Driver: Debian-packaged Nvidia non-free.
 
 It seems not to depend on Iceweasel and Debian alone ... as already
 mentioned by someone else, graphics drivers and such certainly also
 influence the result.
 
 Linux-Fan

 I asked friend of mine, who is still running Squeezy, to test it and Xorg
 behaves nicely. 


 As for graphics drivers, I'm using open source radeon drivers.

 01:05.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI 
 RS780L [Radeon HD 3000]

My laptop is also using the open source radeon driver.  The problem
could be that iceweasel/firefox are trying to store the images into the
GPU's RAM, but the driver places them in the RAM instead.

-- 
Alberto


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d2sa8c4d@eps142.cdf.udc.es

Re: Xorg high memory usage

[Alberto Luaces]

Veljko writes:

 This is really weird as both Debian and Slackware use same xorg:

 dpkg -l |grep xserver-xorg-core
 ii  xserver-xorg-core 2:1.12.4-6 amd64 Xorg X server - core server


 ls /var/log/packages/ |grep xorg-server
 xorg-server-1.12.4-x86_64-1_slack14.0


And what about the version numbers for iceweasel and firefox?


 Nobody has any idea what could be the problem? Should I file a bug? Would that
 be bug for Xorg or Iceweasel?

That behavior is not a good thing, but it is not very surprising to me.
It seems that iceweasel is storing all images as X pixmaps, so the
memory requirements of the Xorg server grow.  Maybe other browsers tend
to avoid the issue by not loading all the images in memory
simultaneously.  Nevertheless, it would be a good thing to know if this
is going to stay as is, or if there are plans to reduce that ram
compsumption.

-- 
Alberto


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871u8rpngh@eps142.cdf.udc.es

Re: Xorg high memory usage

[Veljko]

On Tue, May 28, 2013 at 09:34:22AM +0200, Alberto Luaces wrote:
 And what about the version numbers for iceweasel and firefox?

Debian:
iceweasel -v
Mozilla Iceweasel 10.0.12

Slackware:
firefox -v
Mozilla Firefox 21.0

Firefox I downloaded from mozilla.org and I started on Debian:
./firefox-bin -v
Mozilla Firefox 21.0


 That behavior is not a good thing, but it is not very surprising to me.

Not good, indeed. This is very important peace of software everybody use. 

 It seems that iceweasel is storing all images as X pixmaps, so the
 memory requirements of the Xorg server grow.  Maybe other browsers tend
 to avoid the issue by not loading all the images in memory
 simultaneously.  Nevertheless, it would be a good thing to know if this
 is going to stay as is, or if there are plans to reduce that ram
 compsumption.

Firefox I downloaded behaves exactly like Iceweasel, but only on Debian.
That's why I'm not sure if problem lies only there. If picture is loaded in
RAM by browser, why to duplicate it in Xorg?
 


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130528084255.ga3...@angelina.example.com

Re: Xorg high memory usage

[Alberto Luaces]

Veljko writes:

 On Tue, May 28, 2013 at 09:34:22AM +0200, Alberto Luaces wrote:
 And what about the version numbers for iceweasel and firefox?

 Debian:
 iceweasel -v
 Mozilla Iceweasel 10.0.12

 Slackware:
 firefox -v
 Mozilla Firefox 21.0

 Firefox I downloaded from mozilla.org and I started on Debian:
 ./firefox-bin -v
 Mozilla Firefox 21.0


If I remember correctly, the same happens in my laptop with iceweasel 18.



 That behavior is not a good thing, but it is not very surprising to me.

 Not good, indeed. This is very important peace of software everybody use. 

 It seems that iceweasel is storing all images as X pixmaps, so the
 memory requirements of the Xorg server grow.  Maybe other browsers tend
 to avoid the issue by not loading all the images in memory
 simultaneously.  Nevertheless, it would be a good thing to know if this
 is going to stay as is, or if there are plans to reduce that ram
 compsumption.

 Firefox I downloaded behaves exactly like Iceweasel, but only on Debian.
 That's why I'm not sure if problem lies only there. If picture is loaded in
 RAM by browser, why to duplicate it in Xorg?
  

I don't know.  It could be an error in Xorg, or just triggered by
iceweasel/firefox.  There are similar bug reports
(http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=673349), but we would
have to compare Xorg versions and graphic drivers between Debian and
Slackware.

-- 
Alberto


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87li6zo4aw@eps142.cdf.udc.es

Re: Xorg high memory usage

[staticsafe]

On Tue, May 28, 2013 at 10:42:55AM +0200, Veljko wrote:
 On Tue, May 28, 2013 at 09:34:22AM +0200, Alberto Luaces wrote:
  And what about the version numbers for iceweasel and firefox?
 
 Debian:
 iceweasel -v
 Mozilla Iceweasel 10.0.12
 
 Slackware:
 firefox -v
 Mozilla Firefox 21.0
 
 Firefox I downloaded from mozilla.org and I started on Debian:
 ./firefox-bin -v
 Mozilla Firefox 21.0
 
 
  That behavior is not a good thing, but it is not very surprising to me.
 
 Not good, indeed. This is very important peace of software everybody use. 
 
  It seems that iceweasel is storing all images as X pixmaps, so the
  memory requirements of the Xorg server grow.  Maybe other browsers tend
  to avoid the issue by not loading all the images in memory
  simultaneously.  Nevertheless, it would be a good thing to know if this
  is going to stay as is, or if there are plans to reduce that ram
  compsumption.
 
 Firefox I downloaded behaves exactly like Iceweasel, but only on Debian.
 That's why I'm not sure if problem lies only there. If picture is loaded in
 RAM by browser, why to duplicate it in Xorg?
  
 
 
 -- 
 To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: http://lists.debian.org/20130528084255.ga3...@angelina.example.com
 
I don't think this is an Xorg issue, just tried the webpage you
suggested on Firefox in Windows 7, same issue, rapid memory usage
growth. Seems more like a Firefox issue, can you try a Nightly [0] build
and see if the issue still exists?

[0] - http://nightly.mozilla.org/
-- 
staticsafe
O ascii ribbon campaign - stop html mail - www.asciiribbon.org
Please don't top post - http://goo.gl/YrmAb
Don't CC me! I'm subscribed to whatever list I just posted on.


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130528134433.gc14...@uriel.asininetech.com

Re: Xorg high memory usage

[Alberto Luaces]

staticsafe writes:

 I don't think this is an Xorg issue, just tried the webpage you
 suggested on Firefox in Windows 7, same issue, rapid memory usage
 growth. Seems more like a Firefox issue, can you try a Nightly [0] build
 and see if the issue still exists?

I guess the problem is not firefox eating lots of memory, but Xorg doing
the same simultaneously.

-- 
Alberto


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zjvfmcjr@eps142.cdf.udc.es

Re: Xorg high memory usage

[staticsafe]

On Tue, May 28, 2013 at 03:58:16PM +0200, Alberto Luaces wrote:
 staticsafe writes:
 
  I don't think this is an Xorg issue, just tried the webpage you
  suggested on Firefox in Windows 7, same issue, rapid memory usage
  growth. Seems more like a Firefox issue, can you try a Nightly [0] build
  and see if the issue still exists?
 
 I guess the problem is not firefox eating lots of memory, but Xorg doing
 the same simultaneously.
 
 -- 
 Alberto
 
 

Might have found some answers to the Firefox side of the issue:

http://kb.mozillazine.org/Reducing_memory_usage_-_Firefox#Image-heavy_sites
https://bugzilla.mozilla.org/show_bug.cgi?id=683284
-- 
staticsafe
O ascii ribbon campaign - stop html mail - www.asciiribbon.org
Please don't top post - http://goo.gl/YrmAb
Don't CC me! I'm subscribed to whatever list I just posted on.


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130528140704.gd14...@uriel.asininetech.com

Re: Xorg high memory usage

[Linux-Fan]

On 05/28/2013 10:42 AM, Veljko wrote:
 On Tue, May 28, 2013 at 09:34:22AM +0200, Alberto Luaces wrote:
 And what about the version numbers for iceweasel and firefox?
 
 Debian:
 iceweasel -v
 Mozilla Iceweasel 10.0.12

$ Mozilla Iceweasel 20.0

 It seems that iceweasel is storing all images as X pixmaps, so the
 memory requirements of the Xorg server grow.  Maybe other browsers tend
 to avoid the issue by not loading all the images in memory
 simultaneously.  Nevertheless, it would be a good thing to know if this
 is going to stay as is, or if there are plans to reduce that ram
 compsumption.
 
 Firefox I downloaded behaves exactly like Iceweasel, but only on Debian.
 That's why I'm not sure if problem lies only there. If picture is loaded in
 RAM by browser, why to duplicate it in Xorg?

I have tried to reproduce the problem as well but my Xorg's memory usage
stayed constant (about 50M), iceweasel's memory usage went from 180M to
900M during the test. I used a periodic ps invocation to get the data
pasted to http://paste.debian.net/plain/7070, this was the peak output:

-- 28.05.2013 - 16:07:01 up 43 min, 4 users,
   load average: 1.51, 0.85, 0.58 --
COMMAND  C %MEMVSZ   RSS
Xorg 3  0.9 166480 58300
iceweasel   53 15.2 1714556 931832

I am using Debian 7.0.0 Wheezy, Iceweasel is from Experimental.
Window Manager: Fluxbox, xcompmgr enabled.
Graphics Driver: Debian-packaged Nvidia non-free.

It seems not to depend on Iceweasel and Debian alone ... as already
mentioned by someone else, graphics drivers and such certainly also
influence the result.

Linux-Fan

-- 
http://masysma.ohost.de/


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/51a4bd14.7090...@web.de

Re: Xorg high memory usage

[Veljko]

On Tue, May 28, 2013 at 03:58:16PM +0200, Alberto Luaces wrote:
 staticsafe writes:
 
  I don't think this is an Xorg issue, just tried the webpage you
  suggested on Firefox in Windows 7, same issue, rapid memory usage
  growth. Seems more like a Firefox issue, can you try a Nightly [0] build
  and see if the issue still exists?
 
 I guess the problem is not firefox eating lots of memory, but Xorg doing
 the same simultaneously.

Exactly, it is expected for browser to eat lot of RAM in this situation, but
Xorg shouldn't. 
 


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130528162453.gb12...@angelina.example.com

Re: Xorg high memory usage

[Veljko]

On Tue, May 28, 2013 at 04:20:04PM +0200, Linux-Fan wrote:
 I have tried to reproduce the problem as well but my Xorg's memory usage
 stayed constant (about 50M), iceweasel's memory usage went from 180M to
 900M during the test. I used a periodic ps invocation to get the data
 pasted to http://paste.debian.net/plain/7070, this was the peak output:
 
 -- 28.05.2013 - 16:07:01 up 43 min, 4 users,
load average: 1.51, 0.85, 0.58 --
 COMMAND  C %MEMVSZ   RSS
 Xorg 3  0.9 166480 58300
 iceweasel   53 15.2 1714556 931832
 
 I am using Debian 7.0.0 Wheezy, Iceweasel is from Experimental.
 Window Manager: Fluxbox, xcompmgr enabled.
 Graphics Driver: Debian-packaged Nvidia non-free.
 
 It seems not to depend on Iceweasel and Debian alone ... as already
 mentioned by someone else, graphics drivers and such certainly also
 influence the result.
 
 Linux-Fan

I asked friend of mine, who is still running Squeezy, to test it and Xorg
behaves nicely. 


As for graphics drivers, I'm using open source radeon drivers.

01:05.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RS780L 
[Radeon HD 3000]



-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130528162519.gc12...@angelina.example.com

Re: Xorg high memory usage

[Veljko]

On Sun, May 26, 2013 at 11:49:19PM +0200, Veljko wrote:
 On Sat, May 25, 2013 at 12:09:05PM +0200, Veljko wrote:
  Hello,
  
  I don't know if this is bug, but when I visit some thumblr blogs that have
  endless scrolling (like google images search), it is expected that Iceweasel
  eats lots of RAM (here are lots of pictures), but Xorg RAM usage is even
  bigger. More I scroll, more of RAM gets used by Iceweasel and Xorg.
  
  Currently Xorg takes 36% and Iceweasel takes 34% of RAM (I have 2GB). I'm
  using Wheezy. I'm dual booting with Slackware 14 (3.2.29 kernel) and there 
  is
  no such issue, only FIrefox takes lots of RAM and X is always on 4-5%.
  
  Both installation uses i3wm. Only difference is Debian uses GDM and 
  Slackware
  uses KDM. 
  
  Any idea what could be the problem?
  
  Regards,
  Veljko
 
 
 To update this question. I tried without GDM, just with startx, same
 situation. Then I tried Chromium and Xorg stayed at 5%. So, it seams it's only
 related to Iceweasel. To check if it has something to do with Debian's changed
 version, I downloaded Firefox binary from mozilla.org, but even with this
 version Xorg still eats RAM. 
 
 If someone wants to confirm, try with page like this:
 http://monk3y.tumblr.com/
 
 Scroll down and check memory usage.

This is really weird as both Debian and Slackware use same xorg:

dpkg -l |grep xserver-xorg-core
ii  xserver-xorg-core 2:1.12.4-6 
amd64Xorg X server - core server


ls /var/log/packages/ |grep xorg-server
xorg-server-1.12.4-x86_64-1_slack14.0


Nobody has any idea what could be the problem? Should I file a bug? Would that
be bug for Xorg or Iceweasel?




-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130527155706.gg8...@angelina.example.com

Re: Xorg high memory usage

[Wayne Topa]

On 05/27/2013 11:57 AM, Veljko wrote:
 On Sun, May 26, 2013 at 11:49:19PM +0200, Veljko wrote:
 On Sat, May 25, 2013 at 12:09:05PM +0200, Veljko wrote:
 Hello,

 I don't know if this is bug, but when I visit some thumblr blogs that have
 endless scrolling (like google images search), it is expected that Iceweasel
 eats lots of RAM (here are lots of pictures), but Xorg RAM usage is even
 bigger. More I scroll, more of RAM gets used by Iceweasel and Xorg.

 Currently Xorg takes 36% and Iceweasel takes 34% of RAM (I have 2GB). I'm
 using Wheezy. I'm dual booting with Slackware 14 (3.2.29 kernel) and there 
 is
 no such issue, only FIrefox takes lots of RAM and X is always on 4-5%.

 Both installation uses i3wm. Only difference is Debian uses GDM and 
 Slackware
 uses KDM. 

 Any idea what could be the problem?

 Regards,
 Veljko


 To update this question. I tried without GDM, just with startx, same
 situation. Then I tried Chromium and Xorg stayed at 5%. So, it seams it's 
 only
 related to Iceweasel. To check if it has something to do with Debian's 
 changed
 version, I downloaded Firefox binary from mozilla.org, but even with this
 version Xorg still eats RAM. 

 If someone wants to confirm, try with page like this:
 http://monk3y.tumblr.com/

 Scroll down and check memory usage.
 
 This is really weird as both Debian and Slackware use same xorg:
 
 dpkg -l |grep xserver-xorg-core
 ii  xserver-xorg-core 2:1.12.4-6 
 amd64Xorg X server - core server
 
 
 ls /var/log/packages/ |grep xorg-server
 xorg-server-1.12.4-x86_64-1_slack14.0
 
 
 Nobody has any idea what could be the problem? Should I file a bug? Would that
 be bug for Xorg or Iceweasel?
 

wtopa@dj:~$ apt-listbugs list iceweasel
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
serious bugs of iceweasel (- ) marked as done in some version
 #696041 - ia64 (Itanium) Mozilla JS engine needs pointers have their
high 17 bits cleared (Fixed: iceweasel/10.0.12esr-1+nmu1)
 #705067 - FTBFS on powerpc: Missing WebRTC entries in configure.ac
(Fixed: 21.0-1)
 #692053 - [ia64] Iceweasel 10.0 (and above?) randomly stops responding,
eating 100% CPU (Fixed: iceweasel/10.0.12esr-1+nmu1)
grave bugs of iceweasel (- ) unfixed
 #709841 - iceweasel: spurious out of memory + crash (segmentation fault)
 #703472 - iceweasel freezes on some site due to bad handling of mailcap
file
 #703071 - CVE-2011-1187, CVE-2012-0475,
CVE-2013-{0773,0775,0776,0780,0782,0783} (Fixed: iceweasel/19.0-1)
 #704019 - iceweasel: Iceweasel Crashes upon loading pages using JavaScript
 #674908 - [sparc] iceweasel: JavaScript crash on some sites
serious bugs of iceweasel (- ) unfixed
 #708765 - iceweasel: FTBFS with gawk
Summary:
 iceweasel(9 bugs)



-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/51a38bb2.6050...@gmail.com

Re: Xorg high memory usage

[Veljko]

On Sat, May 25, 2013 at 12:09:05PM +0200, Veljko wrote:
 Hello,
 
 I don't know if this is bug, but when I visit some thumblr blogs that have
 endless scrolling (like google images search), it is expected that Iceweasel
 eats lots of RAM (here are lots of pictures), but Xorg RAM usage is even
 bigger. More I scroll, more of RAM gets used by Iceweasel and Xorg.
 
 Currently Xorg takes 36% and Iceweasel takes 34% of RAM (I have 2GB). I'm
 using Wheezy. I'm dual booting with Slackware 14 (3.2.29 kernel) and there is
 no such issue, only FIrefox takes lots of RAM and X is always on 4-5%.
 
 Both installation uses i3wm. Only difference is Debian uses GDM and Slackware
 uses KDM. 
 
 Any idea what could be the problem?
 
 Regards,
 Veljko


To update this question. I tried without GDM, just with startx, same
situation. Then I tried Chromium and Xorg stayed at 5%. So, it seams it's only
related to Iceweasel. To check if it has something to do with Debian's changed
version, I downloaded Firefox binary from mozilla.org, but even with this
version Xorg still eats RAM. 

If someone wants to confirm, try with page like this:
http://monk3y.tumblr.com/

Scroll down and check memory usage.
 


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130526214919.ga3...@angelina.example.com

Xorg high memory usage

[Veljko]

Hello,

I don't know if this is bug, but when I visit some thumblr blogs that have
endless scrolling (like google images search), it is expected that Iceweasel
eats lots of RAM (here are lots of pictures), but Xorg RAM usage is even
bigger. More I scroll, more of RAM gets used by Iceweasel and Xorg.

Currently Xorg takes 36% and Iceweasel takes 34% of RAM (I have 2GB). I'm
using Wheezy. I'm dual booting with Slackware 14 (3.2.29 kernel) and there is
no such issue, only FIrefox takes lots of RAM and X is always on 4-5%.

Both installation uses i3wm. Only difference is Debian uses GDM and Slackware
uses KDM. 

Any idea what could be the problem?

Regards,
Veljko


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130525100905.ga3...@angelina.example.com

Limit hd quota and memory usage by user at linux

[Pablo Augusto]

Hello

i need limit hd quita and memory usage by user
whats the best way to do this?

for example: lets supose i have a server with 4gb ram and 40gb hd,
i want share all resources with 4 users, so each one must have garanteed 1gb
ram (can use more if are avaiable)
and use max 10gb hd.

how can i do this?

obs: i dont use panels like cpanel, plesk or any other

Re: Limit hd quota and memory usage by user at linux

[Boyd Stephen Smith Jr.]

In 907f512f0904230240r19c01f8chc0a9e173e12ad...@mail.gmail.com, Pablo 
Augusto wrote:
i need limit hd quita and memory usage by user
whats the best way to do this?

For RAM: ulimit and friends.
For HD: filesystem quotas.
-- 
Boyd Stephen Smith Jr.   ,= ,-_-. =.
b...@iguanasuicide.net   ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/\_/



signature.asc
Description: This is a digitally signed message part.

Re: A question about memory usage

[Douglas Allan Tutty]

On Tue, Jun 12, 2007 at 06:05:08PM +0200, Arnau wrote:
 
 In fact what I really want to do is 
 monitor how much memory PostgreSQL is using.

What about top?  If you need to, you can limit top to a list of PIDs

Doug.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: A question about memory usage

[Scott Gifford]

Douglas Allan Tutty [EMAIL PROTECTED] writes:

 On Tue, Jun 12, 2007 at 06:05:08PM +0200, Arnau wrote:
  
 In fact what I really want to do is 
 monitor how much memory PostgreSQL is using.

 What about top?  If you need to, you can limit top to a list of PIDs

top has the same issues as ps: you can't tell what memory is shared
between processes.

Scott.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: A question about memory usage

[Douglas Allan Tutty]

On Wed, Jun 13, 2007 at 01:28:31PM -0400, Scott Gifford wrote:
 Douglas Allan Tutty [EMAIL PROTECTED] writes:
  On Tue, Jun 12, 2007 at 06:05:08PM +0200, Arnau wrote:
   
  In fact what I really want to do is 
  monitor how much memory PostgreSQL is using.
 
  What about top?  If you need to, you can limit top to a list of PIDs
 
 top has the same issues as ps: you can't tell what memory is shared
 between processes.
 

I think I'm beginning to understand: e.g. 10 postgres processes, each
saying its using e.g. 100 MB VIRT yet 1 GB is not being used.  

What is needed is something that does for VM as du does for diskspace.

That would be a most useful tool, especially on boxes with low free
memory.

Doug.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: A question about memory usage

[Andrew Sackville-West]

On Wed, Jun 13, 2007 at 07:45:50PM -0400, Douglas Allan Tutty wrote:
 On Wed, Jun 13, 2007 at 01:28:31PM -0400, Scott Gifford wrote:
  Douglas Allan Tutty [EMAIL PROTECTED] writes:
   On Tue, Jun 12, 2007 at 06:05:08PM +0200, Arnau wrote:

   In fact what I really want to do is 
   monitor how much memory PostgreSQL is using.
  
   What about top?  If you need to, you can limit top to a list of PIDs
  
  top has the same issues as ps: you can't tell what memory is shared
  between processes.
  
 
 I think I'm beginning to understand: e.g. 10 postgres processes, each
 saying its using e.g. 100 MB VIRT yet 1 GB is not being used.  
 
 What is needed is something that does for VM as du does for diskspace.
 
 That would be a most useful tool, especially on boxes with low free
 memory.

http://lwn.net/Articles/230975/

its not the article I was looking for (which actually shows you how to
do the calculations...), but its pretty cool

A


signature.asc
Description: Digital signature

Re: A question about memory usage

[David Fox]

http://lwn.net/Articles/230975/

its not the article I was looking for (which actually shows you how to
do the calculations...), but its pretty cool



As I recall, there was a utility somewhat like vmstat(1)  but unique in that
it could  crawl through
the system and  find out all the different regions occupied by various
processes and dump the result
out. I remember playing with it once or twice but didn't keep it around
because it takes a rather long
time to run on a slow system. Pity, because it seems to fit the bill - I
just can't seem to remember the name of the tool (it's not a standard
utility in that sense).


A

A question about memory usage

[Arnau]

Hi all,

  I have a server with 4GB of RAM and I wanted to know how much memory 
is being used by a PostgreSQL. To do so I have executed the following:


  ps -A -o rss,vsz,command|grep postgres | awk '{rss += $1; vsz += $2 } 
END { print Real: ,rss/1024MB Virtual: ,vsz/1024MB }'


And the result is:
  Real:  157.59MB Virtual:  6359.04MB

So I understand that amount of virtual memory must be swapped to disk as 
it's bigger that my physical memory. To check this I have executed the 
free command and I get the following results:


total used   free sharedbuffers cached
Mem:   4153180 3988536 164644  0 1614003117900
-/+ buffers/cache:  7092363443944
Swap:  6215672  646215608

  So my question is, what I'm doing wrong? because the swap it's almost 
empty


Thank you very much
--
Arnau


--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: A question about memory usage

[Scott Gifford]

Arnau [EMAIL PROTECTED] writes:

 Hi all,

I have a server with 4GB of RAM and I wanted to know how much
 memory is being used by a PostgreSQL. To do so I have executed the
 following:

ps -A -o rss,vsz,command|grep postgres | awk '{rss += $1; vsz += $2
 } END { print Real: ,rss/1024MB Virtual: ,vsz/1024MB }'

 And the result is:
Real:  157.59MB Virtual:  6359.04MB

 So I understand that amount of virtual memory must be swapped to disk
 as it's bigger that my physical memory. 

Hi Arnau,

Your calculation is incorrect.  Many of the PostgreSQL processes are
sharing the same memory, so while each may be using, for example,
500MB of memory, probably 450MB may be the same in each process.
Memory is shared between parent processes and children and when items
are mapped into memory with mmap, including the executable and shared
libraries.

Figuring out exactly how much memory a set of processes are using is a
difficult problem.  Looking in /proc/$pid/maps is a good place to
start.

Good luck!

Scott.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: A question about memory usage

[Andrew Sackville-West]

On Tue, Jun 12, 2007 at 11:32:00AM -0400, Scott Gifford wrote:
 Arnau [EMAIL PROTECTED] writes:
 
  Hi all,
 
 I have a server with 4GB of RAM and I wanted to know how much
  memory is being used by a PostgreSQL. To do so I have executed the
  following:
 
 ps -A -o rss,vsz,command|grep postgres | awk '{rss += $1; vsz += $2
  } END { print Real: ,rss/1024MB Virtual: ,vsz/1024MB }'
 
  And the result is:
 Real:  157.59MB Virtual:  6359.04MB
 
  So I understand that amount of virtual memory must be swapped to disk
  as it's bigger that my physical memory. 
 
 Hi Arnau,
 
 Your calculation is incorrect.  Many of the PostgreSQL processes are
 sharing the same memory, so while each may be using, for example,
 500MB of memory, probably 450MB may be the same in each process.
 Memory is shared between parent processes and children and when items
 are mapped into memory with mmap, including the executable and shared
 libraries.
 
 Figuring out exactly how much memory a set of processes are using is a
 difficult problem.  Looking in /proc/$pid/maps is a good place to
 start.
 

probably the easiest thing is to use free in two states: with the
process running and without. 

if all other factors are the same, then the difference would be the
amount of memory used by the process in question.

or is there something wrong with that?

A



signature.asc
Description: Digital signature

Re: A question about memory usage

[Arnau]

Andrew Sackville-West wrote:

On Tue, Jun 12, 2007 at 11:32:00AM -0400, Scott Gifford wrote:

Arnau [EMAIL PROTECTED] writes:


Hi all,

   I have a server with 4GB of RAM and I wanted to know how much
memory is being used by a PostgreSQL. To do so I have executed the
following:

   ps -A -o rss,vsz,command|grep postgres | awk '{rss += $1; vsz += $2
} END { print Real: ,rss/1024MB Virtual: ,vsz/1024MB }'

And the result is:
   Real:  157.59MB Virtual:  6359.04MB

So I understand that amount of virtual memory must be swapped to disk
as it's bigger that my physical memory. 

Hi Arnau,

Your calculation is incorrect.  Many of the PostgreSQL processes are
sharing the same memory, so while each may be using, for example,
500MB of memory, probably 450MB may be the same in each process.
Memory is shared between parent processes and children and when items
are mapped into memory with mmap, including the executable and shared
libraries.

Figuring out exactly how much memory a set of processes are using is a
difficult problem.  Looking in /proc/$pid/maps is a good place to
start.



probably the easiest thing is to use free in two states: with the
process running and without. 


if all other factors are the same, then the difference would be the
amount of memory used by the process in question.

or is there something wrong with that?


  I'm afraid that's not possible :) In fact what I really want to do is 
monitor how much memory PostgreSQL is using.


--
Arnau


--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: A question about memory usage

[Scott Gifford]

Andrew Sackville-West [EMAIL PROTECTED] writes:

 On Tue, Jun 12, 2007 at 11:32:00AM -0400, Scott Gifford wrote:

[...]

 Figuring out exactly how much memory a set of processes are using is a
 difficult problem.  Looking in /proc/$pid/maps is a good place to
 start.
 

 probably the easiest thing is to use free in two states: with the
 process running and without. 

 if all other factors are the same, then the difference would be the
 amount of memory used by the process in question.

 or is there something wrong with that?

It's a good idea for many circumstances.  It's a bit imprecise on a
normal system with all of its running processes, since the memory
usage of other things will fluctuate all the time.  And it requires
killing the database, which isn't always acceptable in a production
environment.

-Scott.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

firefox / Xorg memory usage / swap

[David A.]

Hi,

sometimes when I have many tabs open in firefox (10-15 or so) the
system begins to swap and become virtually unresponsive. If I'm quick I
can kill firefox.

Below is a snapshot from top while swapping. As you can see Xorg is the
heavy Mem-consumer.
My questions;

1) why does Xorg  consume much memory when I surf with many tabs? (I
would have expected the firefox-process to increase more.)

2) How can i prevent this from happening? (switch of swap? warning
before swapping begins?)


Cpu(s):  0.8%us, 12.8%sy,  0.0%ni,  0.0%id, 85.4%wa,  1.0%hi,  0.0%si,
0.0%st
Mem:483252k total,   478364k used, 4888k free,   96k
buffers
Swap:0k total,0k used,0k free, 6568k cached

PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
23097 david 17   0 23444 2164  792 D  3.1  0.4   5:54.96 gkrellm
22993 root   6 -10  446m 373m  528 D  2.6 79.1  71:35.06 Xorg
  153 root  10  -5 000 D  2.4  0.0   0:54.74 kswapd0
17554 david 26  10  168m  55m 1516 S  2.4 11.8  12:28.85
firefox-bin
26230 david 16   0  2232  500  240 R  1.0  0.1   0:04.19 top

system; EPIA PD6000e, 2.6.18, SID, 512mb ram, firefox 2 from
mozilla.org (same was happening in debian-FF 1.5)

regards, David.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Managing memory usage per *user* (or group) not per *process*?

[Dave Ewart]

I have a system which I manage which has many users.  Now and again,
during busy periods, if many users are working at once the machine
starts swapping and performance goes through the floor.  This is because
the main purpose of the machine is to run statistical analyses of large
datasets which results in heavy RAM usage.

I'd like to impose resource limits on RAM usage.

The standard approach to this appears to be to use ulimit/setrlimit,
which allows one to set process limits for various things.  But, the
underlying flaw with this approach is that the limits operate per
*process*, not per user.

This means that if there is a process limit of 1GB RAM, there is nothing
stopping a user running many processes each of which are 'only' 900MB,
for example.

Is there a way to impose resource limits (spec. RAM usage) per user?
Or, even better, per system group (so I could say all users in group
'staff' are limited to a total memory usage at any one time of 4GB RAM
or similar)?

Any suggestions and ideas most welcome!

Thanks,

Dave.
-- 
Dave Ewart - [EMAIL PROTECTED] - jabber: [EMAIL PROTECTED]
All email from me is now digitally signed, key from http://www.sungate.co.uk/
Fingerprint: AEC5 9360 0A35 7F66 66E9 82E4 9E10 6769 CD28 DA92



signature.asc
Description: Digital signature

Re: Managing memory usage per *user* (or group) not per *process*?

[Steve Block]

On Wed, Mar 08, 2006 at 03:19:50PM +, Dave Ewart wrote:

I have a system which I manage which has many users.  Now and again,
during busy periods, if many users are working at once the machine
starts swapping and performance goes through the floor.  This is because
the main purpose of the machine is to run statistical analyses of large
datasets which results in heavy RAM usage.

I'd like to impose resource limits on RAM usage.

The standard approach to this appears to be to use ulimit/setrlimit,
which allows one to set process limits for various things.  But, the
underlying flaw with this approach is that the limits operate per
*process*, not per user.

This means that if there is a process limit of 1GB RAM, there is nothing
stopping a user running many processes each of which are 'only' 900MB,
for example.

Is there a way to impose resource limits (spec. RAM usage) per user?
Or, even better, per system group (so I could say all users in group
'staff' are limited to a total memory usage at any one time of 4GB RAM
or similar)?

Any suggestions and ideas most welcome!


I think /etc/security/limits.conf may be what you're looking for.

--
Steve Block
http://ev-15.com/
http://steveblock.com/
[EMAIL PROTECTED]


--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Managing memory usage per *user* (or group) not per *process*?

[Dave Ewart]

On Wednesday, 08.03.2006 at 09:25 -0600, Steve Block wrote:

 On Wed, Mar 08, 2006 at 03:19:50PM +, Dave Ewart wrote:
 I have a system which I manage which has many users.  Now and again,
 during busy periods, if many users are working at once the machine
 starts swapping and performance goes through the floor.  This is
 because the main purpose of the machine is to run statistical
 analyses of large datasets which results in heavy RAM usage.
 
 I'd like to impose resource limits on RAM usage.
 
 The standard approach to this appears to be to use ulimit/setrlimit,
 which allows one to set process limits for various things.  But, the
 underlying flaw with this approach is that the limits operate per
 *process*, not per user.
 
 This means that if there is a process limit of 1GB RAM, there is
 nothing stopping a user running many processes each of which are
 'only' 900MB, for example.
 
 Is there a way to impose resource limits (spec. RAM usage) per user?
 Or, even better, per system group (so I could say all users in group
 'staff' are limited to a total memory usage at any one time of 4GB
 RAM or similar)?
 
 Any suggestions and ideas most welcome!
 
 I think /etc/security/limits.conf may be what you're looking for.

Thanks Steve.  In my experience testing this out just now,
/etc/security/limits.conf still sets limits per process, but these
limits can be set differently for each user or for each group.

It is still possible to run many processes that are just under the
radar, as it were, so this doesn't solve my problem unfortunately.

Dave.
-- 
Please don't CC me on list messages!
...
Dave Ewart - [EMAIL PROTECTED] - jabber: [EMAIL PROTECTED]
All email from me is now digitally signed, key from http://www.sungate.co.uk/
Fingerprint: AEC5 9360 0A35 7F66 66E9 82E4 9E10 6769 CD28 DA92



signature.asc
Description: Digital signature

Re: Managing memory usage per *user* (or group) not per *process*?

[Hugo Vanwoerkom]

Dave Ewart wrote:

I have a system which I manage which has many users.  Now and again,
during busy periods, if many users are working at once the machine
starts swapping and performance goes through the floor.  This is because
the main purpose of the machine is to run statistical analyses of large
datasets which results in heavy RAM usage.

I'd like to impose resource limits on RAM usage.

The standard approach to this appears to be to use ulimit/setrlimit,
which allows one to set process limits for various things.  But, the
underlying flaw with this approach is that the limits operate per
*process*, not per user.

This means that if there is a process limit of 1GB RAM, there is nothing
stopping a user running many processes each of which are 'only' 900MB,
for example.

Is there a way to impose resource limits (spec. RAM usage) per user?
Or, even better, per system group (so I could say all users in group
'staff' are limited to a total memory usage at any one time of 4GB RAM
or similar)?

Any suggestions and ideas most welcome!

Thanks,

Dave.


I don't know the answer to your query. But if I was in your place I 
would ask Con Kolivas at:

http://members.optusnet.com.au/ckolivas/kernel/
I run his kernel (2.6.15-ck5) and the discussion in the forum he 
maintains and the kernel patches he provides are along the line of your 
query.


H



--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Managing memory usage per *user* (or group) not per *process*?

[Ishwar Rattan]

On Wed, 8 Mar 2006, Dave Ewart wrote:

 On Wednesday, 08.03.2006 at 09:25 -0600, Steve Block wrote:

  Is there a way to impose resource limits (spec. RAM usage) per user?
  Or, even better, per system group (so I could say all users in group
  'staff' are limited to a total memory usage at any one time of 4GB
  RAM or similar)?
  
  Any suggestions and ideas most welcome!
 
  I think /etc/security/limits.conf may be what you're looking for.

 Thanks Steve.  In my experience testing this out just now,
 /etc/security/limits.conf still sets limits per process, but these
 limits can be set differently for each user or for each group.

 It is still possible to run many processes that are just under the
 radar, as it were, so this doesn't solve my problem unfortunately.

Design philosophy of UNIX is multiuser and multitasking system.
To achiev what you want is either limit number of user on the
box or rewrite the kernel to your needs.

-ishwar


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Managing memory usage per *user* (or group) not per *process*?

[Dave Ewart]

On Wednesday, 08.03.2006 at 12:34 -0500, Ishwar Rattan wrote:

   Is there a way to impose resource limits (spec. RAM usage) per
   user?  Or, even better, per system group (so I could say all
   users in group 'staff' are limited to a total memory usage at any
   one time of 4GB RAM or similar)?
   
   Any suggestions and ideas most welcome!
  
   I think /etc/security/limits.conf may be what you're looking for.
 
  Thanks Steve.  In my experience testing this out just now,
  /etc/security/limits.conf still sets limits per process, but these
  limits can be set differently for each user or for each group.
 
  It is still possible to run many processes that are just under the
  radar, as it were, so this doesn't solve my problem unfortunately.
 
 Design philosophy of UNIX is multiuser and multitasking system.  To
 achiev what you want is either limit number of user on the box or
 rewrite the kernel to your needs.

Well that's a rather abrupt reply to what I consider to be a reasonable
question, Ishwar...  I understand the UNIX philosophy, but given the
flexibility of the many and varied tools available on any/all Unix-like
variants, I didn't think I'm asking for anything particularly out of the
ordinary.

Given that the facilities provided by /etc/security/limits.conf are very
*close* to being what I want, it's not much of a technical/logical leap
to what I was asking about.

If it's truly not possible then, yes, I may need to rethink.

Dave.

-- 
Please don't CC me on list messages!
...
Dave Ewart - [EMAIL PROTECTED] - jabber: [EMAIL PROTECTED]
All email from me is now digitally signed, key from http://www.sungate.co.uk/
Fingerprint: AEC5 9360 0A35 7F66 66E9 82E4 9E10 6769 CD28 DA92



signature.asc
Description: Digital signature

Re: Memory usage (avoid swapping, top accuracy...)

[Hugo Vanwoerkom]

Vincent Lefevre wrote:

On 2006-01-01 07:08:23 -0600, Hugo Vanwoerkom wrote:

See:
http://members.optusnet.com.au/ckolivas/kernel/

Has Swap Prefetch patch to make that easier. Especially for laptops.


Thanks, I'll try that in a few days.


To see who runs what check out:
http://klive.cpushare.com/


No-one has ck on ppc. I hope there won't be any problem.



Subscribe to the list and ask.
H


--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Memory usage (avoid swapping, top accuracy...)

[Hugo Vanwoerkom]

Vincent Lefevre wrote:

Hi,

I'd like to have some information about memory usage and how to
improve it.



snip

See:
http://members.optusnet.com.au/ckolivas/kernel/

Has Swap Prefetch patch to make that easier. Especially for laptops.
Check out the mailing list. Ask the question there.
Kolivas is a memory usage guru.
I currently run 2.6.14-ck8.
To see who runs what check out:
http://klive.cpushare.com/

H


--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Memory usage (avoid swapping, top accuracy...)

[Vincent Lefevre]

On 2006-01-01 07:08:23 -0600, Hugo Vanwoerkom wrote:
 See:
 http://members.optusnet.com.au/ckolivas/kernel/
 
 Has Swap Prefetch patch to make that easier. Especially for laptops.

Thanks, I'll try that in a few days.

 To see who runs what check out:
 http://klive.cpushare.com/

No-one has ck on ppc. I hope there won't be any problem.

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Memory usage (avoid swapping, top accuracy...)

[Steve Lamb]

Vincent Lefevre wrote:
 Yes, precisely because this is a laptop (i.e. it is sometimes offline).
 Also, I currently have only the laptop here; so, I have the choice
 between having the news locally or read/post after ssh'ing a remote
 machine. I prefer the former solution.

Again, do you NEED Apache and innd?  Answer, no.  wwwoffle and noffle are
far better solutions for on/offline usage.

 I don't think so. Well, after the reboot, my machine is quite
 responsive. Disabling spamd probably helped (and the reboot too).

This is a FAQ.  Lemme ask this one question.  Are you concerned about swap
size or are you concerned about active swapping?  Two different things.  If
your concerned about the former and there is very little of the latter here's
your answer.  Don't pay attention to how much swap is used.  Swap used without
a large amount of active swapping is a *good* thing and you're fretting over
nothing.  Honest.  Check the archives for the why and how's of it.

If it is the latter then please clarify that you're concerned about active
swapping.

-- 
 Steve C. Lamb | I'm your priest, I'm your shrink, I'm your
   PGP Key: 8B6E99C5   | main connection to the switchboard of souls.
---+-


signature.asc
Description: OpenPGP digital signature

Re: Memory usage (avoid swapping, top accuracy...)

[Vincent Lefevre]

On 2005-12-30 19:12:22 +, Dave Ewart wrote:
 FWIW Try 'leafnode' instead - that's a much lighter 'offline news' setup.

I need a news server to which an external machine can connect (in
addition to the local connection). Does leafnode support that?

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Memory usage (avoid swapping, top accuracy...)

[Vincent Lefevre]

On 2005-12-31 12:14:12 -0800, Steve Lamb wrote:
 Vincent Lefevre wrote:
  Yes, precisely because this is a laptop (i.e. it is sometimes offline).
  Also, I currently have only the laptop here; so, I have the choice
  between having the news locally or read/post after ssh'ing a remote
  machine. I prefer the former solution.
 
 Again, do you NEED Apache and innd? Answer, no. wwwoffle and
 noffle are far better solutions for on/offline usage.

Does wwwoffle support content negociation, CGI and Perl filtering?
Does noffle support remote connections?

  I don't think so. Well, after the reboot, my machine is quite
  responsive. Disabling spamd probably helped (and the reboot too).

BTW, I may need to reactivate spamd later; I'm currently testing
the antivirus/antispam service of my ISP, but I've had several
false positives with it, without any report to the remote side.
I may also try spamd on the machine hosting my virtual domain,
where I have much more control, hoping it won't take too much
CPU time.

 This is a FAQ. Lemme ask this one question. Are you concerned
 about swap size or are you concerned about active swapping?

My machine swaps a lot, in particular after running a process that
needed a lot of memory (several dozens of MB), for instance dpkg
(it can take more than 100 MB). Also, spamd (with bayesian filter)
was one of the culprits: since I've disabled it, the PowerBook is
much more responsive (but see above).

  Two different things. If your concerned about the former and there
 is very little of the latter here's your answer.

Well, I don't know what you mean exactly, but I have enough swap
space: I've almost never run out of (virtual) memory (when I did,
it was due to a bug).

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Memory usage (avoid swapping, top accuracy...)

[Vincent Lefevre]

On 2006-01-01 01:24:14 +0100, Vincent Lefevre wrote:
 Does wwwoffle support content negociation, CGI and Perl filtering?

and I forgot: dav-svn (which was the only reason why I switched from
Apache 1 to Apache 2).

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Memory usage (avoid swapping, top accuracy...)

[John Hasler]

Vincent Lefevre writes:
 I need a news server to which an external machine can connect (in
 addition to the local connection). Does leafnode support that?

Yes.  It does not support any authentication, though, so you only want to
use it on your LAN.  Don't expose it to the Internet.

My biggest gripe is the lack of local groups.
-- 
John Hasler


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Memory usage (avoid swapping, top accuracy...)

[Vincent Lefevre]

On 2005-12-31 18:41:37 -0600, John Hasler wrote:
 Vincent Lefevre writes:
  I need a news server to which an external machine can connect (in
  addition to the local connection). Does leafnode support that?
 
 Yes. It does not support any authentication, though, so you only
 want to use it on your LAN. Don't expose it to the Internet.

This is a bit annoying. In fact, I just need IP-based filtering.
With inn, I currently have:

auth local {
hosts: localhost, 127.0.0.1, 192.168.0.0/24
default: local
}

I could add iptables rules, though, but I'm not sure how to do
this without affecting the other rules.

-- 
Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/
100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/
Work: CR INRIA - computer arithmetic / SPACES project at LORIA


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]