Re: Unreclaimed swap space upon process termination?

[Richard Braun]

On Tue, Nov 29, 2016 at 10:11:34AM +0100, Thomas Schwinge wrote:
> Early after a system reboot (second run of the executable), I ran into a
> GNU Mach "panic ../vm/vm_page.c:2058: vm_page_evict: vm_page: unable to
> recycle any page"; see attached.  As this is new code in GNU Mach (commit
> 5d1258459ad618481a4f239e8ce020bdecda1d3f "Rework pageout to handle
> multiple segments"): anything you'd like me to do/preserve before
> rebooting?

It's known, and just needs to be fixed, something I plan to do when I
have more free time. For information, the problem is caused by the lack
of threshold on the number of dirty pages.

-- 
Richard Braun

Re: Unreclaimed swap space upon process termination?

[Thomas Schwinge]

Hi!

On Mon, 28 Nov 2016 19:55:20 +0100, Samuel Thibault  
wrote:
> Thomas Schwinge, on Mon 28 Nov 2016 17:10:26 +0100, wrote:
> > ..., but on the new ("bad") system, the first non-sensical (huge;
> > -2147479552 is 0x80001000) vm_allocate call actually succeeds:
> 
> Yes, the userland address space is now 3G again, so it can now indeed
> allocate that much.

ACK.

> > I have not yet figured out where these vm_allocate calls and/or their
> > huge size parameters are coming from.
> 
> Probably the real source of the issue :)

Will look into that.

Early after a system reboot (second run of the executable), I ran into a
GNU Mach "panic ../vm/vm_page.c:2058: vm_page_evict: vm_page: unable to
recycle any page"; see attached.  As this is new code in GNU Mach (commit
5d1258459ad618481a4f239e8ce020bdecda1d3f "Rework pageout to handle
multiple segments"): anything you'd like me to do/preserve before
rebooting?


Grüße
 Thomas

Re: Unreclaimed swap space upon process termination?

[Samuel Thibault]

Hello,

Thomas Schwinge, on Mon 28 Nov 2016 17:10:26 +0100, wrote:
> ..., but on the new ("bad") system, the first non-sensical (huge;
> -2147479552 is 0x80001000) vm_allocate call actually succeeds:

Yes, the userland address space is now 3G again, so it can now indeed
allocate that much.

> I have not yet figured out where these vm_allocate calls and/or their
> huge size parameters are coming from.

Probably the real source of the issue :)

Samuel

Re: Unreclaimed swap space upon process termination?

[Thomas Schwinge]

Hi!

On Mon, 28 Nov 2016 16:03:44 +0100, I wrote:
> Updating a Debian GNU/Hurd virtual machine to recent packages after many
> months, and then running the GCC testsuite, I observe the following
> behavior, which should be reproducible with the executable in the
> attached tarball:
> 
> $ vmstat | grep swap\ free
> swap free: 4096M
> $ ./1.exe 
> $ vmstat | grep swap\ free
> swap free: 3288M
> $ ./1.exe 
> $ vmstat | grep swap\ free
> swap free: 2495M
> $ ./1.exe 
> $ vmstat | grep swap\ free
> swap free: 1726M
> $ ./1.exe 
> $ vmstat | grep swap\ free
> swap free:  931M
> $ ./1.exe 
> $ vmstat | grep swap\ free
> swap free:  164M
> $ ./1.exe 
> Bus error
> $ vmstat | grep swap\ free
> swap free:0 
> 
> At this point, the system doesn't recover from this low memory situation.
> 
> For each invocation of the executable, there are three "no more room in
> [...]  (./1.exe([...])" messages on the Mach console.
> 
> The executable is compiled from
> [gcc]/libstdc++-v3/testsuite/21_strings/basic_string/modifiers/insert/char/1.cc
> from commit a050099a416f013bda35832b878d9a57b0cbb231 (gcc-6-branch branch
> point; 2016-04-15), which doesn't look very spectacular -- apart from
> maybe the __gnu_test::set_memory_limits call, which I'll try to figure
> out what it does.

That uses setrlimit for RLIMIT_DATA, RLIMIT_RSS, RLIMIT_VMEM, RLIMIT_AS,
but there is no change with that call removed.

> But nevertheless, unreclaimed swap space upon process
> termination sounds like a bug?
> 
> Unless this a known issue, or somebody can quickly pinpoint the problem,
> I'll try to bisect core system packages, between the version of the
> "good" and "bad" disk images.

Running with rpctrace, I see that on the old ("good") system, at the end
of the process, we got:

[...]
task52(pid2198)->vm_deallocate (16973824 16) = 0 
task52(pid2198)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space 
available) 
task52(pid2198)->vm_allocate (0 -2147348480 1) = 0x3 ((os/kern) no space 
available) 
task52(pid2198)->vm_map (0 2097152 0 1  (null) 0 1 0 7 1) = 0 21405696
task52(pid2198)->vm_deallocate (21405696 614400) = 0 
task52(pid2198)->vm_deallocate (23068672 434176) = 0 
task52(pid2198)->vm_protect (22020096 135168 0 3) = 0 
task52(pid2198)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space 
available) 
  61<--68(pid2198)->proc_mark_exit_request (0 0) = 0 
task52(pid2198)->task_terminate () = 0 
Child 2198 exited with 0

..., but on the new ("bad") system, the first non-sensical (huge;
-2147479552 is 0x80001000) vm_allocate call actually succeeds:

[...]
task154(pid1080)->vm_deallocate (16973824 16) = 0 
task154(pid1080)->vm_allocate (0 -2147479552 1) = 0 268742656
task154(pid1080)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space 
available) 
task154(pid1080)->vm_allocate (0 -2147348480 1) = 0x3 ((os/kern) no space 
available) 
task154(pid1080)->vm_map (0 2097152 0 1  (null) 0 1 0 7 1) = 0 2162
task154(pid1080)->vm_deallocate (2162 364544) = 0 
task154(pid1080)->vm_deallocate (23068672 684032) = 0 
task154(pid1080)->vm_protect (22020096 135168 0 3) = 0 
task154(pid1080)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space 
available) 
task154(pid1080)->vm_deallocate (268742656 -2147479552) = 0 
  163<--170(pid1080)->proc_mark_exit_request (0 0) = 0 
task154(pid1080)->task_terminate () = 0 
Child 1080 exited with 0

I have not yet figured out where these vm_allocate calls and/or their
huge size parameters are coming from.


Grüße
 Thomas

Re: Unreclaimed swap space upon process termination?

[Samuel Thibault]

Hello,

Thomas Schwinge, on Mon 28 Nov 2016 16:03:44 +0100, wrote:
> But nevertheless, unreclaimed swap space upon process
> termination sounds like a bug?

It's actually not really a new problem.  It has just been emphasized
more with recent memory management changes.

Samuel

Re: Unreclaimed swap space upon process termination?

[Richard Braun]

On Mon, Nov 28, 2016 at 04:03:44PM +0100, Thomas Schwinge wrote:
> Unless this a known issue, or somebody can quickly pinpoint the problem,
> I'll try to bisect core system packages, between the version of the
> "good" and "bad" disk images.

It's a known issue, a side effect of the page cache changes that I
merged some months ago. Because of increased cache usage, there is
more memory pressure, and therefore more swap usage (mostly through
double paging, since the kernel always tries to evict external pages
first). The default pager has never been able to correctly reclaim
these pages in the past, but now there are a lot more of them.

-- 
Richard Braun

Unreclaimed swap space upon process termination?

[Thomas Schwinge]

Hi!

Updating a Debian GNU/Hurd virtual machine to recent packages after many
months, and then running the GCC testsuite, I observe the following
behavior, which should be reproducible with the executable in the
attached tarball:

$ vmstat | grep swap\ free
swap free: 4096M
$ ./1.exe 
$ vmstat | grep swap\ free
swap free: 3288M
$ ./1.exe 
$ vmstat | grep swap\ free
swap free: 2495M
$ ./1.exe 
$ vmstat | grep swap\ free
swap free: 1726M
$ ./1.exe 
$ vmstat | grep swap\ free
swap free:  931M
$ ./1.exe 
$ vmstat | grep swap\ free
swap free:  164M
$ ./1.exe 
Bus error
$ vmstat | grep swap\ free
swap free:0 

At this point, the system doesn't recover from this low memory situation.

For each invocation of the executable, there are three "no more room in
[...]  (./1.exe([...])" messages on the Mach console.

The executable is compiled from
[gcc]/libstdc++-v3/testsuite/21_strings/basic_string/modifiers/insert/char/1.cc
from commit a050099a416f013bda35832b878d9a57b0cbb231 (gcc-6-branch branch
point; 2016-04-15), which doesn't look very spectacular -- apart from
maybe the __gnu_test::set_memory_limits call, which I'll try to figure
out what it does.  But nevertheless, unreclaimed swap space upon process
termination sounds like a bug?

Unless this a known issue, or somebody can quickly pinpoint the problem,
I'll try to bisect core system packages, between the version of the
"good" and "bad" disk images.


Grüße
 Thomas




1.tar.xz
Description: application/xz