Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 9, 2019 at 5:34 PM Kevin Oberman wrote:
>
>
> On Fri, Aug 9, 2019 at 2:16 PM Mark Johnston wrote:
>
>> On Fri, Aug 09, 2019 at 01:05:50PM -0700, Kevin Oberman wrote:
>> > On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston
>> wrote:
>> >
>> > > On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
>> > > > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing
>> issues
>> > > > resuming my Win7 VM on VirtualBox. My prior kernel was built on
>> 24-Jul.
>> > > If
>> > > > there is not sufficient memory available to reload the system (4
>> Meg.),
>> > > the
>> > >
>> > > Where does this number come from? What memory usage stats do you see
>> in
>> > > top(1) when the error occurs?
>> > >
>> >
>> > I am monitoring memory usage with gkrellm. It appears to define "Free"
>> as
>> > the sum of "Inactive" and "Free". If you are referring to size of the
>> VM,
>> > was supposed to be the memory specified when I created the VM, but my
>> > fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
>> > What's a few orders of magnitude?
>> >
>> > Oddly, when I watch memory space closely I note that, as the VM loads, I
>> > started seeing swap utilization increase as free space was exhausted at
>> > about 80% loaded. Loading continued to 98%. at that point loading
>> stopped
>> > and swap use continued to grow for a bit. Then free space started to
>> > increase from about 300M to about 700M before the error window popped
>> up.
>> >
>> >
>> > > > resume fails with a message that memory was exhausted. Usually I
>> can try
>> > > > resuming again and it will work. Sometimes I get the error two or
>> three
>> > > > times before the system resumes.
>> > >
>> > > What exactly is the error message?
>> > >
>> > Failed to open a session for the virtual machine Win7.
>> >
>> > Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
>> >
>> > Result Code: NS_ERROR_FAILURE (0x80004005)
>> > Component: ConsoleWrap
>> > Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
>> >
>> >
>> > >
>> > > > Since I have not touched VirtualBox other than to rebuild the kmod
>> after
>> > > > the kernel build, it looks like something in the OS triggered this.
>> Since
>> > > > the system frees up some memory each time so that the VM eventually
>> > > > resumes, it looks like the memory request is made to the OS, but VB
>> is
>> > > not
>> > > > waiting or not enough memory is freed to allow the VB to complete
>> the
>> > > > resume.
>> > > >
>> > > > Any clue what might have changed over those 13 days? I am running
>> GENERIC
>> > > > except that I run the 4BSD scheduler.
>> > >
>> > > Possible culprits are r350374 and r350375, but I can't really see how.
>> > >
>> >
>> > This started after the 6-Aug build (r350664). My prior build was
>> r350292,
>> > so just before these two commits.
>> >
>> > Can I try just reverting these two? Once I do, it will need to run for a
>> > while or do something to tie up a lot of memory before the error will
>> > recur. In normal use it is a matter of firefox increasing resident
>> memory
>> > until there is not enough free memory to load the VM without swapping.
>> > (These days I often see the sum of all firefox process resident memory
>> > exceeding 3G after it's been up for a day or two. Still, not worse than
>> > chromium.)
>>
>> Those commits can simply be reverted, but I am skeptical that they will
>> help. You should also verify that these same conditions don't lead to
>> errors on your prior build, if you haven't already.
>>
>
> OK. Running identical kernel except for 350374-5.
>
> Yes, I am sure that it was not happening with the r350292 kernel. I hit
> this quite consistently when firefox has been running for a while.
>
> Firefox has rss of just under 3G on startup and will slowly grow until I
> don't have the resources to run the Win7 VM without the error. Right now
> the VM completes loading with no swapping and about 800M of memory free
> after it is running. I'll let you know when it gets big enough to cause a
> problem and whether it fails. Probably won't happen until tomorrow.
>
For three days I have been trying to repeat the failure with the to changed
backed out and the problem does not occur with 350374-5 reverted.
I have no idea how these changes trigger the issue, but it sure looks like
they do. It happens when a 4G file is being loaded into memory. I have not
looked at the VB code to see if anything unusual is being done. I did
noticed that the problem only seems to show up when the VM is almost fully
loaded... usually 98%, but I have seen it all the way to 100%. That was an
odd "failure" where the error was not fatal. The VM was in a "Paused"
state. I was able to resume it and it ran normally.
Let me know what else, if anything, you would like me to try or test.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 9, 2019 at 2:16 PM Mark Johnston wrote:
> On Fri, Aug 09, 2019 at 01:05:50PM -0700, Kevin Oberman wrote:
> > On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston wrote:
> >
> > > On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
> > > > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing
> issues
> > > > resuming my Win7 VM on VirtualBox. My prior kernel was built on
> 24-Jul.
> > > If
> > > > there is not sufficient memory available to reload the system (4
> Meg.),
> > > the
> > >
> > > Where does this number come from? What memory usage stats do you see
> in
> > > top(1) when the error occurs?
> > >
> >
> > I am monitoring memory usage with gkrellm. It appears to define "Free" as
> > the sum of "Inactive" and "Free". If you are referring to size of the VM,
> > was supposed to be the memory specified when I created the VM, but my
> > fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
> > What's a few orders of magnitude?
> >
> > Oddly, when I watch memory space closely I note that, as the VM loads, I
> > started seeing swap utilization increase as free space was exhausted at
> > about 80% loaded. Loading continued to 98%. at that point loading stopped
> > and swap use continued to grow for a bit. Then free space started to
> > increase from about 300M to about 700M before the error window popped up.
> >
> >
> > > > resume fails with a message that memory was exhausted. Usually I can
> try
> > > > resuming again and it will work. Sometimes I get the error two or
> three
> > > > times before the system resumes.
> > >
> > > What exactly is the error message?
> > >
> > Failed to open a session for the virtual machine Win7.
> >
> > Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
> >
> > Result Code: NS_ERROR_FAILURE (0x80004005)
> > Component: ConsoleWrap
> > Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
> >
> >
> > >
> > > > Since I have not touched VirtualBox other than to rebuild the kmod
> after
> > > > the kernel build, it looks like something in the OS triggered this.
> Since
> > > > the system frees up some memory each time so that the VM eventually
> > > > resumes, it looks like the memory request is made to the OS, but VB
> is
> > > not
> > > > waiting or not enough memory is freed to allow the VB to complete the
> > > > resume.
> > > >
> > > > Any clue what might have changed over those 13 days? I am running
> GENERIC
> > > > except that I run the 4BSD scheduler.
> > >
> > > Possible culprits are r350374 and r350375, but I can't really see how.
> > >
> >
> > This started after the 6-Aug build (r350664). My prior build was r350292,
> > so just before these two commits.
> >
> > Can I try just reverting these two? Once I do, it will need to run for a
> > while or do something to tie up a lot of memory before the error will
> > recur. In normal use it is a matter of firefox increasing resident memory
> > until there is not enough free memory to load the VM without swapping.
> > (These days I often see the sum of all firefox process resident memory
> > exceeding 3G after it's been up for a day or two. Still, not worse than
> > chromium.)
>
> Those commits can simply be reverted, but I am skeptical that they will
> help. You should also verify that these same conditions don't lead to
> errors on your prior build, if you haven't already.
>
OK. Running identical kernel except for 350374-5.
Yes, I am sure that it was not happening with the r350292 kernel. I hit
this quite consistently when firefox has been running for a while.
Firefox has rss of just under 3G on startup and will slowly grow until I
don't have the resources to run the Win7 VM without the error. Right now
the VM completes loading with no swapping and about 800M of memory free
after it is running. I'll let you know when it gets big enough to cause a
problem and whether it fails. Probably won't happen until tomorrow.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 09, 2019 at 01:05:50PM -0700, Kevin Oberman wrote:
> On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston wrote:
>
> > On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
> > > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues
> > > resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul.
> > If
> > > there is not sufficient memory available to reload the system (4 Meg.),
> > the
> >
> > Where does this number come from? What memory usage stats do you see in
> > top(1) when the error occurs?
> >
>
> I am monitoring memory usage with gkrellm. It appears to define "Free" as
> the sum of "Inactive" and "Free". If you are referring to size of the VM,
> was supposed to be the memory specified when I created the VM, but my
> fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
> What's a few orders of magnitude?
>
> Oddly, when I watch memory space closely I note that, as the VM loads, I
> started seeing swap utilization increase as free space was exhausted at
> about 80% loaded. Loading continued to 98%. at that point loading stopped
> and swap use continued to grow for a bit. Then free space started to
> increase from about 300M to about 700M before the error window popped up.
>
>
> > > resume fails with a message that memory was exhausted. Usually I can try
> > > resuming again and it will work. Sometimes I get the error two or three
> > > times before the system resumes.
> >
> > What exactly is the error message?
> >
> Failed to open a session for the virtual machine Win7.
>
> Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
>
> Result Code: NS_ERROR_FAILURE (0x80004005)
> Component: ConsoleWrap
> Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
>
>
> >
> > > Since I have not touched VirtualBox other than to rebuild the kmod after
> > > the kernel build, it looks like something in the OS triggered this. Since
> > > the system frees up some memory each time so that the VM eventually
> > > resumes, it looks like the memory request is made to the OS, but VB is
> > not
> > > waiting or not enough memory is freed to allow the VB to complete the
> > > resume.
> > >
> > > Any clue what might have changed over those 13 days? I am running GENERIC
> > > except that I run the 4BSD scheduler.
> >
> > Possible culprits are r350374 and r350375, but I can't really see how.
> >
>
> This started after the 6-Aug build (r350664). My prior build was r350292,
> so just before these two commits.
>
> Can I try just reverting these two? Once I do, it will need to run for a
> while or do something to tie up a lot of memory before the error will
> recur. In normal use it is a matter of firefox increasing resident memory
> until there is not enough free memory to load the VM without swapping.
> (These days I often see the sum of all firefox process resident memory
> exceeding 3G after it's been up for a day or two. Still, not worse than
> chromium.)
Those commits can simply be reverted, but I am skeptical that they will
help. You should also verify that these same conditions don't lead to
errors on your prior build, if you haven't already.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston wrote:
> On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
> > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues
> > resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul.
> If
> > there is not sufficient memory available to reload the system (4 Meg.),
> the
>
> Where does this number come from? What memory usage stats do you see in
> top(1) when the error occurs?
>
I am monitoring memory usage with gkrellm. It appears to define "Free" as
the sum of "Inactive" and "Free". If you are referring to size of the VM,
was supposed to be the memory specified when I created the VM, but my
fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
What's a few orders of magnitude?
Oddly, when I watch memory space closely I note that, as the VM loads, I
started seeing swap utilization increase as free space was exhausted at
about 80% loaded. Loading continued to 98%. at that point loading stopped
and swap use continued to grow for a bit. Then free space started to
increase from about 300M to about 700M before the error window popped up.
> > resume fails with a message that memory was exhausted. Usually I can try
> > resuming again and it will work. Sometimes I get the error two or three
> > times before the system resumes.
>
> What exactly is the error message?
>
Failed to open a session for the virtual machine Win7.
Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
Result Code: NS_ERROR_FAILURE (0x80004005)
Component: ConsoleWrap
Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
>
> > Since I have not touched VirtualBox other than to rebuild the kmod after
> > the kernel build, it looks like something in the OS triggered this. Since
> > the system frees up some memory each time so that the VM eventually
> > resumes, it looks like the memory request is made to the OS, but VB is
> not
> > waiting or not enough memory is freed to allow the VB to complete the
> > resume.
> >
> > Any clue what might have changed over those 13 days? I am running GENERIC
> > except that I run the 4BSD scheduler.
>
> Possible culprits are r350374 and r350375, but I can't really see how.
>
This started after the 6-Aug build (r350664). My prior build was r350292,
so just before these two commits.
Can I try just reverting these two? Once I do, it will need to run for a
while or do something to tie up a lot of memory before the error will
recur. In normal use it is a matter of firefox increasing resident memory
until there is not enough free memory to load the VM without swapping.
(These days I often see the sum of all firefox process resident memory
exceeding 3G after it's been up for a day or two. Still, not worse than
chromium.)
Thanks, Mark, for the quick response.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote: > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues > resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul. If > there is not sufficient memory available to reload the system (4 Meg.), the Where does this number come from? What memory usage stats do you see in top(1) when the error occurs? > resume fails with a message that memory was exhausted. Usually I can try > resuming again and it will work. Sometimes I get the error two or three > times before the system resumes. What exactly is the error message? > Since I have not touched VirtualBox other than to rebuild the kmod after > the kernel build, it looks like something in the OS triggered this. Since > the system frees up some memory each time so that the VM eventually > resumes, it looks like the memory request is made to the OS, but VB is not > waiting or not enough memory is freed to allow the VB to complete the > resume. > > Any clue what might have changed over those 13 days? I am running GENERIC > except that I run the 4BSD scheduler. Possible culprits are r350374 and r350375, but I can't really see how. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Memory management changes after kernel update on 6-Aug
Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul. If there is not sufficient memory available to reload the system (4 Meg.), the resume fails with a message that memory was exhausted. Usually I can try resuming again and it will work. Sometimes I get the error two or three times before the system resumes. Since I have not touched VirtualBox other than to rebuild the kmod after the kernel build, it looks like something in the OS triggered this. Since the system frees up some memory each time so that the VM eventually resumes, it looks like the memory request is made to the OS, but VB is not waiting or not enough memory is freed to allow the VB to complete the resume. Any clue what might have changed over those 13 days? I am running GENERIC except that I run the 4BSD scheduler. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkober...@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS memory management
2012/11/27 Freddie Cash fjwc...@gmail.com: Read any ZFS tuning manual on the web, including the ones direct from SUN/Oracle, and they all list: - if you are running processes that need a lot of memory, then limit the ARC to allow the apps to have access to that memory Or you could have at least a little swap (good practice) to allow ARC take the time to evict some memory when under pressure. :) On Tue, Nov 27, 2012 at 12:10 PM, Nikolay Denev nde...@gmail.com wrote: Hello list, I have the following question : I have several machines with 196G of RAM that are using RELENG_9 with ZFS, and are running a very memory intensive java applications - ElasticSearch The machines are without swap configured and have vm.swap_enabled=0 in /etc/sysctl.conf. The ElasticSearch processes are using mlockall(2) to pin down their memory (configured at 40G). And at this point I thought that there would be no problems, but from time to time, when the machine grows it's ARC memory and there are some other running processes like nginx with passenger and uwsgi the ElasticSearch process would get killed by the kernel OOM killer with reason no swap space available Of course, I've now tuned down arc_max in /boot/loader.conf, but isn't this supposed to work automatically? Like ZFS releasing some memory when there is a pressure, instead of the OOM killer going postal? (at the moment when the process was killed the ZFS ARC was 132G). I understand that this might be problematic as AFAIK ZFS releases memory asynchronously when the arc_reclaim_thread() is run, which might take some time to be scheduled and complete. Cheers, Nikolay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org- against HTML email vCards X www: http://www.gid0.org- against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS memory management
On Nov 29, 2012, at 4:53 PM, Olivier Smedts oliv...@gid0.org wrote: 2012/11/27 Freddie Cash fjwc...@gmail.com: Read any ZFS tuning manual on the web, including the ones direct from SUN/Oracle, and they all list: - if you are running processes that need a lot of memory, then limit the ARC to allow the apps to have access to that memory Or you could have at least a little swap (good practice) to allow ARC take the time to evict some memory when under pressure. Yes, this was already suggested off-list, and it seems like a solution. Thanks to all for the input! :) On Tue, Nov 27, 2012 at 12:10 PM, Nikolay Denev nde...@gmail.com wrote: Hello list, I have the following question : I have several machines with 196G of RAM that are using RELENG_9 with ZFS, and are running a very memory intensive java applications - ElasticSearch The machines are without swap configured and have vm.swap_enabled=0 in /etc/sysctl.conf. The ElasticSearch processes are using mlockall(2) to pin down their memory (configured at 40G). And at this point I thought that there would be no problems, but from time to time, when the machine grows it's ARC memory and there are some other running processes like nginx with passenger and uwsgi the ElasticSearch process would get killed by the kernel OOM killer with reason no swap space available Of course, I've now tuned down arc_max in /boot/loader.conf, but isn't this supposed to work automatically? Like ZFS releasing some memory when there is a pressure, instead of the OOM killer going postal? (at the moment when the process was killed the ZFS ARC was 132G). I understand that this might be problematic as AFAIK ZFS releases memory asynchronously when the arc_reclaim_thread() is run, which might take some time to be scheduled and complete. Cheers, Nikolay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: oliv...@gid0.org- against HTML email vCards X www: http://www.gid0.org- against proprietary attachments / \ Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS memory management
on 29/11/2012 19:16 Nikolay Denev said the following: On Nov 29, 2012, at 4:53 PM, Olivier Smedts oliv...@gid0.org wrote: 2012/11/27 Freddie Cash fjwc...@gmail.com: Read any ZFS tuning manual on the web, including the ones direct from SUN/Oracle, and they all list: - if you are running processes that need a lot of memory, then limit the ARC to allow the apps to have access to that memory Or you could have at least a little swap (good practice) to allow ARC take the time to evict some memory when under pressure. Yes, this was already suggested off-list, and it seems like a solution. Thanks to all for the input! I think that various VM thresholds are not very well auto-tuned for a swap-less system. So, perhaps, something to _experiment_ with... I could make sense to increase (e.g. double or triple) vm.v_cache_min, so that the pager is waken up earlier. At the same time vm.v_free_target could be decreased so that difference between it and vm.v_free_reserved is smaller (but greater than zero). My understanding is that OOM handling is activated when the pager can not get number of available (free + cached) pages above v_cache_min + v_free_target after two passes. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
ZFS memory management
Hello list, I have the following question : I have several machines with 196G of RAM that are using RELENG_9 with ZFS, and are running a very memory intensive java applications - ElasticSearch The machines are without swap configured and have vm.swap_enabled=0 in /etc/sysctl.conf. The ElasticSearch processes are using mlockall(2) to pin down their memory (configured at 40G). And at this point I thought that there would be no problems, but from time to time, when the machine grows it's ARC memory and there are some other running processes like nginx with passenger and uwsgi the ElasticSearch process would get killed by the kernel OOM killer with reason no swap space available Of course, I've now tuned down arc_max in /boot/loader.conf, but isn't this supposed to work automatically? Like ZFS releasing some memory when there is a pressure, instead of the OOM killer going postal? (at the moment when the process was killed the ZFS ARC was 132G). I understand that this might be problematic as AFAIK ZFS releases memory asynchronously when the arc_reclaim_thread() is run, which might take some time to be scheduled and complete. Cheers, Nikolay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS memory management
Read any ZFS tuning manual on the web, including the ones direct from SUN/Oracle, and they all list: - if you are running processes that need a lot of memory, then limit the ARC to allow the apps to have access to that memory :) On Tue, Nov 27, 2012 at 12:10 PM, Nikolay Denev nde...@gmail.com wrote: Hello list, I have the following question : I have several machines with 196G of RAM that are using RELENG_9 with ZFS, and are running a very memory intensive java applications - ElasticSearch The machines are without swap configured and have vm.swap_enabled=0 in /etc/sysctl.conf. The ElasticSearch processes are using mlockall(2) to pin down their memory (configured at 40G). And at this point I thought that there would be no problems, but from time to time, when the machine grows it's ARC memory and there are some other running processes like nginx with passenger and uwsgi the ElasticSearch process would get killed by the kernel OOM killer with reason no swap space available Of course, I've now tuned down arc_max in /boot/loader.conf, but isn't this supposed to work automatically? Like ZFS releasing some memory when there is a pressure, instead of the OOM killer going postal? (at the moment when the process was killed the ZFS ARC was 132G). I understand that this might be problematic as AFAIK ZFS releases memory asynchronously when the arc_reclaim_thread() is run, which might take some time to be scheduled and complete. Cheers, Nikolay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS memory management
- if you are running processes that need a lot of memory, then limit the ARC to allow the apps to have access to that memory And this is what Nikolay did after the incident apparently. The question was more like: Shouldn't the OS release this memory automatically when it's starved as opposed to nuking processes? It's only cache after all. Lots of it. On Tue, Nov 27, 2012 at 12:22 PM, Freddie Cash fjwc...@gmail.com wrote: Read any ZFS tuning manual on the web, including the ones direct from SUN/Oracle, and they all list: - if you are running processes that need a lot of memory, then limit the ARC to allow the apps to have access to that memory :) On Tue, Nov 27, 2012 at 12:10 PM, Nikolay Denev nde...@gmail.com wrote: Hello list, I have the following question : I have several machines with 196G of RAM that are using RELENG_9 with ZFS, and are running a very memory intensive java applications - ElasticSearch The machines are without swap configured and have vm.swap_enabled=0 in /etc/sysctl.conf. The ElasticSearch processes are using mlockall(2) to pin down their memory (configured at 40G). And at this point I thought that there would be no problems, but from time to time, when the machine grows it's ARC memory and there are some other running processes like nginx with passenger and uwsgi the ElasticSearch process would get killed by the kernel OOM killer with reason no swap space available Of course, I've now tuned down arc_max in /boot/loader.conf, but isn't this supposed to work automatically? Like ZFS releasing some memory when there is a pressure, instead of the OOM killer going postal? (at the moment when the process was killed the ZFS ARC was 132G). I understand that this might be problematic as AFAIK ZFS releases memory asynchronously when the arc_reclaim_thread() is run, which might take some time to be scheduled and complete. Cheers, Nikolay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Rumen Telbizov http://telbizov.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Memory management
Hello there, I have a computer running FreeBSD 6.1. As time passing by, the memory fills up. When the machine starts, memory is occupied to 30 %, and after two or three weeks memory is occupied to 100 % and it begins to use swap. It is inactive pages that fills up the memory. I tried to restart every process, but memory usage does not decrease. Only a reboot can fix that. And I'm not able to see which process leaks. I was not able to find a correct definition of what inactive memory is. First, I would like to know what are these kind of pages : wired, active, inactive, cache and free. Is that normal that inactive memory usage grows ? What should I do ? Do you have any tools to monitor memory usage of processes ? Many thanks, regards, ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Memory management
On Wed, 2006-Jul-26 11:26:34 +0200, Stephane Dupille wrote: As time passing by, the memory fills up. When the machine starts, memory is occupied to 30 %, and after two or three weeks memory is occupied to 100 % and it begins to use swap. How are you monitoring memory usage? Do you mean 'swap' or 'page'? A level of page-in's is normal because text and data areas for processes are loaded by paging them in. I was not able to find a correct definition of what inactive memory is. First, I would like to know what are these kind of pages : wired, active, inactive, cache and free. Wired pages are pages that the kernel has wired to RAM so they cannot be paged out. Active pages are being mapped by virtual memory and in use by running processes. Inactive pages are not currently mapped but the kernel knows their contents and can re-map them without needing to retrieve them from disk - they may be dirty. Cache pages are similar to active pages but aren't dirty and are higher-priority candidates for being freed. Free pages have no useful content and will be used to fulfil page-in requests. Is that normal that inactive memory usage grows ? Yes. 'Free' memory is basically wasted and so the kernel tries to limit it, subject to having sufficient free memory to meet page-faults. Most of your RAM should be wired, active or inactive. Inactive memory will start at 0 and grow as active pages are released. What should I do ? Nothing. Why do you think you have a problem? Do you have any tools to monitor memory usage of processes ? ps(1) -- Peter Jeremy pgpYlM6L9c5ab.pgp Description: PGP signature
Re: Memory management
On Wed, 26 Jul 2006, Stephane Dupille wrote: I have a computer running FreeBSD 6.1. As time passing by, the memory fills up. When the machine starts, memory is occupied to 30 %, and after two or three weeks memory is occupied to 100 % and it begins to use swap. It is inactive pages that fills up the memory. I tried to restart every process, but memory usage does not decrease. Only a reboot can fix that. And I'm not able to see which process leaks. I was not able to find a correct definition of what inactive memory is. First, I would like to know what are these kind of pages : wired, active, inactive, cache and free. Is that normal that inactive memory usage grows ? What should I do ? Do you have any tools to monitor memory usage of processes ? You can find an article discussing some of the FreeBSD VM system design here: http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html This document is gradually aging, and doesn't cover certain elements of the design, including some recent optimizations, SMP behavior, etc, but still makes quite a good read. It could be that you have applications leaking memory, but it's more likely that the VM system is simply being efficient. Free memory is, in effect, wasted memory, since it's not being used. FreeBSD will agressively cache file system data, and either drop or page out unused pages (depending on whether they are dirty) in order to maximize the amount of memory available for actively used data. This means it will swap out dirty pages allocated by a process if the process isn't using them, rather than keep them in memory preventing that memory from being used by processes that need it. The result is that at any given moment, an active system should have almost no free memory, and instead should be providing as much memory as possible to active pages, and the largest possible file system cache. This may seem counter-intuitive compared to some other systems where a premium is placed on free pages as representing the resources available to run additional applications. We consider memory floating around unused to be a waste of memory that could be used to improve system performance. systat -vmstat 1 is a good tool to monitor the VM system, as it will let you monitor memory use, in particular, the vnode and swap paging rates. You can use ps(1) with various parameters to inspect process memory use. A popular combination is aux, which views all processes and displays, among other things, their virtual and resident sizes. The resident size is the figure you want to look at, as it represents the number of pages actually in memory, rather than pages that could be paged in. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Memory management
Peter Jeremy [EMAIL PROTECTED] écrit : As time passing by, the memory fills up. When the machine starts, memory is occupied to 30 %, and after two or three weeks memory is occupied to 100 % and it begins to use swap. How are you monitoring memory usage? Using top, mainly. And ps, swapinfo, vmstat... Do you mean 'swap' or 'page'? swap. A level of page-in's is normal because text and data areas for processes are loaded by paging them in. OK for that, but when I have 700 MB of inactive memory, and free memory reaching zero, the system begins to use the swap : swapinfo says that swap is used (paged out). Is that normal ? Wired pages are pages that the kernel has wired to RAM so they cannot be paged out. Active pages are being mapped by virtual memory and in use by running processes. Inactive pages are not currently mapped but the kernel knows their contents and can re-map them without needing to retrieve them from disk - they may be dirty. Cache pages are similar to active pages but aren't dirty and are higher-priority candidates for being freed. Free pages have no useful content and will be used to fulfil page-in requests. OK, thanks for the definitions. Why there is two states inactive and cache, if they are so similar ? (it's just curiosity, my questions have answers now.) Yes. 'Free' memory is basically wasted and so the kernel tries to limit it, subject to having sufficient free memory to meet page-faults. Most of your RAM should be wired, active or inactive. Inactive memory will start at 0 and grow as active pages are released. OK, sounds clear. What should I do ? Nothing. Why do you think you have a problem? As long as I was not sure what inactive means, I was not sure of what to think. My first guess was that inactive memory are like active memory but was not accessed for some time, and as such have more priority to be pages out to swap than active memory. So i tried to search for the real definition of what inactive memory is, and what I found was a little bit fuzzy. Do you have any tools to monitor memory usage of processes ? ps(1) ps(1) is ok to check snapshots of process states, but not the evolution of the memory usages. I looked for a tool mainly to find processes with memory leaks. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Memory management
Stephane Dupille [EMAIL PROTECTED] wrote: Peter Jeremy [EMAIL PROTECTED] écrit : As time passing by, the memory fills up. When the machine starts, memory is occupied to 30 %, and after two or three weeks memory is occupied to 100 % and it begins to use swap. How are you monitoring memory usage? Using top, mainly. And ps, swapinfo, vmstat... Do you mean 'swap' or 'page'? swap. Note that it is normal that processes get swapped out if they haven't been in the run queue for a very long time and free memory has reached near zero (which is also normal). That's a feature, because it improves efficiency in most situations, because more RAM is available to processes which really need it. For example, if you use X11 and don't log into syscons' virtual terminals (/dev/vty*), you will notice that the getty(8) processes will be swapped out after some time when there is not many free memory left: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 382 root 3 0 996K 0K ttyin 0:00 0.00% 0.00% getty 383 root 3 0 996K 0K ttyin 0:00 0.00% 0.00% getty 384 root 3 0 996K 0K ttyin 0:00 0.00% 0.00% getty 385 root 3 0 996K 0K ttyin 0:00 0.00% 0.00% getty 386 root 3 0 996K 0K ttyin 0:00 0.00% 0.00% getty If you want to check whether there's a _real_ shortage of RAM (or a memory leak somewhere), a good way is to watch the po and sr columns in the output of vmstat 5: procsmemorypagedisksfaults cpu r b w avm fre flt re pi po fr sr ad0 da0 in sy cs us sy id 1 0 0 39960 44944 62 0 0 0 61 3 0 0 157 6 36 2 1 97 0 0 0 39960 44944 2 0 0 0 0 0 0 0 1132 19 8 0 0 100 0 0 0 39960 44944 1 0 0 0 0 0 0 1 1132 19 8 0 0 100 0 0 0 34300 44944 1 0 0 0 0 0 0 0 1132 23 9 0 0 100 The po (page-out) column indicates paging activity, i.e. data is moved to the swap. The sr (scan-rate) column measures the speed at which inactive pages are scanned for becomeing cache or free pages; this number is a good measure of the pressure on the VM system. If both numbers are almost always near zero, you don't have to worry at all. If the numbers are constantly high, you either have a leak somewhere that you need to discover, or you need to add more RAM to your machine. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
