Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 9, 2019 at 2:16 PM Mark Johnston wrote:
> On Fri, Aug 09, 2019 at 01:05:50PM -0700, Kevin Oberman wrote:
> > On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston wrote:
> >
> > > On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
> > > > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing
> issues
> > > > resuming my Win7 VM on VirtualBox. My prior kernel was built on
> 24-Jul.
> > > If
> > > > there is not sufficient memory available to reload the system (4
> Meg.),
> > > the
> > >
> > > Where does this number come from? What memory usage stats do you see
> in
> > > top(1) when the error occurs?
> > >
> >
> > I am monitoring memory usage with gkrellm. It appears to define "Free" as
> > the sum of "Inactive" and "Free". If you are referring to size of the VM,
> > was supposed to be the memory specified when I created the VM, but my
> > fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
> > What's a few orders of magnitude?
> >
> > Oddly, when I watch memory space closely I note that, as the VM loads, I
> > started seeing swap utilization increase as free space was exhausted at
> > about 80% loaded. Loading continued to 98%. at that point loading stopped
> > and swap use continued to grow for a bit. Then free space started to
> > increase from about 300M to about 700M before the error window popped up.
> >
> >
> > > > resume fails with a message that memory was exhausted. Usually I can
> try
> > > > resuming again and it will work. Sometimes I get the error two or
> three
> > > > times before the system resumes.
> > >
> > > What exactly is the error message?
> > >
> > Failed to open a session for the virtual machine Win7.
> >
> > Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
> >
> > Result Code: NS_ERROR_FAILURE (0x80004005)
> > Component: ConsoleWrap
> > Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
> >
> >
> > >
> > > > Since I have not touched VirtualBox other than to rebuild the kmod
> after
> > > > the kernel build, it looks like something in the OS triggered this.
> Since
> > > > the system frees up some memory each time so that the VM eventually
> > > > resumes, it looks like the memory request is made to the OS, but VB
> is
> > > not
> > > > waiting or not enough memory is freed to allow the VB to complete the
> > > > resume.
> > > >
> > > > Any clue what might have changed over those 13 days? I am running
> GENERIC
> > > > except that I run the 4BSD scheduler.
> > >
> > > Possible culprits are r350374 and r350375, but I can't really see how.
> > >
> >
> > This started after the 6-Aug build (r350664). My prior build was r350292,
> > so just before these two commits.
> >
> > Can I try just reverting these two? Once I do, it will need to run for a
> > while or do something to tie up a lot of memory before the error will
> > recur. In normal use it is a matter of firefox increasing resident memory
> > until there is not enough free memory to load the VM without swapping.
> > (These days I often see the sum of all firefox process resident memory
> > exceeding 3G after it's been up for a day or two. Still, not worse than
> > chromium.)
>
> Those commits can simply be reverted, but I am skeptical that they will
> help. You should also verify that these same conditions don't lead to
> errors on your prior build, if you haven't already.
>
OK. Running identical kernel except for 350374-5.
Yes, I am sure that it was not happening with the r350292 kernel. I hit
this quite consistently when firefox has been running for a while.
Firefox has rss of just under 3G on startup and will slowly grow until I
don't have the resources to run the Win7 VM without the error. Right now
the VM completes loading with no swapping and about 800M of memory free
after it is running. I'll let you know when it gets big enough to cause a
problem and whether it fails. Probably won't happen until tomorrow.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
FreeBSD CI Weekly Report 2019-08-04
(Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2019-08-04 === Here is a summary of the FreeBSD Continuous Integration results for the period from 2019-07-29 to 2019-08-04. During this period, we have: * 1662 builds (95.8% (-1.3) passed, 4.2% (+1.3) failed) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 309 test runs (87.4% (+6.6) passed, 12.6% (-6.3) unstable, 0% (-0.3) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 18 doc builds (100% (+0) passed) Test case status (on 2019-07-28 23:59): | Branch/Architecture | Total | Pass | Fail| Skipped | | --- | -- | - | --- | | | head/amd64 | 7460 (+0) | 7409 (+0) | 0 (+0) | 51 (+0) | | head/i386 | 7458 (+0) | 7400 (+0) | 0 (+0) | 58 (+0) | | 12-STABLE/amd64 | 7388 (+0) | 7344 (+0) | 0 (+0) | 44 (+0) | | 12-STABLE/i386 | 7386 (+0) | 7332 (-3) | 0 (+0) | 54 (+3) | | 11-STABLE/amd64 | 6845 (+0) | 6800 (+3) | 1 (+0) | 44 (-3) | | 11-STABLE/i386 | 6843 (+0) | 6760 (+1) | 34 (-1) | 49 (+0) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/s/rkEiIYr7B and archive is available at http://hackfoldr.org/freebsd-ci-report/, any help is welcome. ## News * [FCP 20190401-ci_policy: CI policy ](https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md) is in "feedback" state, please check and provide comments. * We are skipping the test cases failing in CI system only when "ci" configuration variable is true. This value can be set in kyua.conf or cli as: `kyua -v test_suites.FreeBSD.ci=true test` ## Failing Tests * https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/ * local.kyua.* (31 cases) * local.lutok.* (3 cases) ## Failing and Flaky Tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d https://bugs.freebsd.org/237641 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~60 failing cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details ## Disabled Tests * lib.libc.sys.mmap_test.mmap_truncate_signal https://bugs.freebsd.org/211924 * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * usr.bin.procstat.procstat_test.command_loogle.com/ine_arguments https://bugs.freebsd.org/233587 * usr.bin.procstat.procstat_test.environment https://bugs.freebsd.org/233588 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero (new) https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.netpfil.pf.forward.v4 (i386 only) * sys.netpfil.pf.forward.v6 (i386 only) * sys.netpfil.pf.set_tos.v4 (i386 only) https://bugs.freebsd.org/239380 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 ## Issues ### Cause build fails * https://bugs.freebsd.org/233735 Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s * https://bugs.freebsd.org/238828 Possible build race: lib/libsysdecode/tables.h:948: error: 'IPV6_MIN_MEMBERSHIPS' undeclared Patch available: https://reviews.freebsd.org/D21069 ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic Patch exists: * https://reviews.freebsd.org/D20868 * https://reviews.freebsd.org/D20869 ### Open * https://bugs.freebsd.org/237077 possible race in build: /usr/src/sys/amd64/linux/linux_support.s:38:2: error: expected relocatable expression * https://bugs.freebsd.org/237403 Tests in
Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 09, 2019 at 01:05:50PM -0700, Kevin Oberman wrote:
> On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston wrote:
>
> > On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
> > > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues
> > > resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul.
> > If
> > > there is not sufficient memory available to reload the system (4 Meg.),
> > the
> >
> > Where does this number come from? What memory usage stats do you see in
> > top(1) when the error occurs?
> >
>
> I am monitoring memory usage with gkrellm. It appears to define "Free" as
> the sum of "Inactive" and "Free". If you are referring to size of the VM,
> was supposed to be the memory specified when I created the VM, but my
> fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
> What's a few orders of magnitude?
>
> Oddly, when I watch memory space closely I note that, as the VM loads, I
> started seeing swap utilization increase as free space was exhausted at
> about 80% loaded. Loading continued to 98%. at that point loading stopped
> and swap use continued to grow for a bit. Then free space started to
> increase from about 300M to about 700M before the error window popped up.
>
>
> > > resume fails with a message that memory was exhausted. Usually I can try
> > > resuming again and it will work. Sometimes I get the error two or three
> > > times before the system resumes.
> >
> > What exactly is the error message?
> >
> Failed to open a session for the virtual machine Win7.
>
> Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
>
> Result Code: NS_ERROR_FAILURE (0x80004005)
> Component: ConsoleWrap
> Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
>
>
> >
> > > Since I have not touched VirtualBox other than to rebuild the kmod after
> > > the kernel build, it looks like something in the OS triggered this. Since
> > > the system frees up some memory each time so that the VM eventually
> > > resumes, it looks like the memory request is made to the OS, but VB is
> > not
> > > waiting or not enough memory is freed to allow the VB to complete the
> > > resume.
> > >
> > > Any clue what might have changed over those 13 days? I am running GENERIC
> > > except that I run the 4BSD scheduler.
> >
> > Possible culprits are r350374 and r350375, but I can't really see how.
> >
>
> This started after the 6-Aug build (r350664). My prior build was r350292,
> so just before these two commits.
>
> Can I try just reverting these two? Once I do, it will need to run for a
> while or do something to tie up a lot of memory before the error will
> recur. In normal use it is a matter of firefox increasing resident memory
> until there is not enough free memory to load the VM without swapping.
> (These days I often see the sum of all firefox process resident memory
> exceeding 3G after it's been up for a day or two. Still, not worse than
> chromium.)
Those commits can simply be reverted, but I am skeptical that they will
help. You should also verify that these same conditions don't lead to
errors on your prior build, if you haven't already.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 9, 2019 at 11:35 AM Mark Johnston wrote:
> On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote:
> > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues
> > resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul.
> If
> > there is not sufficient memory available to reload the system (4 Meg.),
> the
>
> Where does this number come from? What memory usage stats do you see in
> top(1) when the error occurs?
>
I am monitoring memory usage with gkrellm. It appears to define "Free" as
the sum of "Inactive" and "Free". If you are referring to size of the VM,
was supposed to be the memory specified when I created the VM, but my
fingers got ahead of my brain and it should have been 4G, not 4M. Hey!
What's a few orders of magnitude?
Oddly, when I watch memory space closely I note that, as the VM loads, I
started seeing swap utilization increase as free space was exhausted at
about 80% loaded. Loading continued to 98%. at that point loading stopped
and swap use continued to grow for a bit. Then free space started to
increase from about 300M to about 700M before the error window popped up.
> > resume fails with a message that memory was exhausted. Usually I can try
> > resuming again and it will work. Sometimes I get the error two or three
> > times before the system resumes.
>
> What exactly is the error message?
>
Failed to open a session for the virtual machine Win7.
Failed to load unit 'pgm' (VERR_EM_NO_MEMORY).
Result Code: NS_ERROR_FAILURE (0x80004005)
Component: ConsoleWrap
Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
>
> > Since I have not touched VirtualBox other than to rebuild the kmod after
> > the kernel build, it looks like something in the OS triggered this. Since
> > the system frees up some memory each time so that the VM eventually
> > resumes, it looks like the memory request is made to the OS, but VB is
> not
> > waiting or not enough memory is freed to allow the VB to complete the
> > resume.
> >
> > Any clue what might have changed over those 13 days? I am running GENERIC
> > except that I run the 4BSD scheduler.
>
> Possible culprits are r350374 and r350375, but I can't really see how.
>
This started after the 6-Aug build (r350664). My prior build was r350292,
so just before these two commits.
Can I try just reverting these two? Once I do, it will need to run for a
while or do something to tie up a lot of memory before the error will
recur. In normal use it is a matter of firefox increasing resident memory
until there is not enough free memory to load the VM without swapping.
(These days I often see the sum of all firefox process resident memory
exceeding 3G after it's been up for a day or two. Still, not worse than
chromium.)
Thanks, Mark, for the quick response.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Memory management changes after kernel update on 6-Aug
On Fri, Aug 09, 2019 at 11:09:24AM -0700, Kevin Oberman wrote: > Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues > resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul. If > there is not sufficient memory available to reload the system (4 Meg.), the Where does this number come from? What memory usage stats do you see in top(1) when the error occurs? > resume fails with a message that memory was exhausted. Usually I can try > resuming again and it will work. Sometimes I get the error two or three > times before the system resumes. What exactly is the error message? > Since I have not touched VirtualBox other than to rebuild the kmod after > the kernel build, it looks like something in the OS triggered this. Since > the system frees up some memory each time so that the VM eventually > resumes, it looks like the memory request is made to the OS, but VB is not > waiting or not enough memory is freed to allow the VB to complete the > resume. > > Any clue what might have changed over those 13 days? I am running GENERIC > except that I run the 4BSD scheduler. Possible culprits are r350374 and r350375, but I can't really see how. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Memory management changes after kernel update on 6-Aug
Since I updated my 12.0-STABLE system on 6-Aug I have been seeing issues resuming my Win7 VM on VirtualBox. My prior kernel was built on 24-Jul. If there is not sufficient memory available to reload the system (4 Meg.), the resume fails with a message that memory was exhausted. Usually I can try resuming again and it will work. Sometimes I get the error two or three times before the system resumes. Since I have not touched VirtualBox other than to rebuild the kmod after the kernel build, it looks like something in the OS triggered this. Since the system frees up some memory each time so that the VM eventually resumes, it looks like the memory request is made to the OS, but VB is not waiting or not enough memory is freed to allow the VB to complete the resume. Any clue what might have changed over those 13 days? I am running GENERIC except that I run the 4BSD scheduler. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkober...@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
