Re: Unreclaimed swap space upon process termination?
On Tue, Nov 29, 2016 at 10:11:34AM +0100, Thomas Schwinge wrote: > Early after a system reboot (second run of the executable), I ran into a > GNU Mach "panic ../vm/vm_page.c:2058: vm_page_evict: vm_page: unable to > recycle any page"; see attached. As this is new code in GNU Mach (commit > 5d1258459ad618481a4f239e8ce020bdecda1d3f "Rework pageout to handle > multiple segments"): anything you'd like me to do/preserve before > rebooting? It's known, and just needs to be fixed, something I plan to do when I have more free time. For information, the problem is caused by the lack of threshold on the number of dirty pages. -- Richard Braun
Re: Unreclaimed swap space upon process termination?
Hi! On Mon, 28 Nov 2016 19:55:20 +0100, Samuel Thibaultwrote: > Thomas Schwinge, on Mon 28 Nov 2016 17:10:26 +0100, wrote: > > ..., but on the new ("bad") system, the first non-sensical (huge; > > -2147479552 is 0x80001000) vm_allocate call actually succeeds: > > Yes, the userland address space is now 3G again, so it can now indeed > allocate that much. ACK. > > I have not yet figured out where these vm_allocate calls and/or their > > huge size parameters are coming from. > > Probably the real source of the issue :) Will look into that. Early after a system reboot (second run of the executable), I ran into a GNU Mach "panic ../vm/vm_page.c:2058: vm_page_evict: vm_page: unable to recycle any page"; see attached. As this is new code in GNU Mach (commit 5d1258459ad618481a4f239e8ce020bdecda1d3f "Rework pageout to handle multiple segments"): anything you'd like me to do/preserve before rebooting? Grüße Thomas
Re: Unreclaimed swap space upon process termination?
Hello, Thomas Schwinge, on Mon 28 Nov 2016 17:10:26 +0100, wrote: > ..., but on the new ("bad") system, the first non-sensical (huge; > -2147479552 is 0x80001000) vm_allocate call actually succeeds: Yes, the userland address space is now 3G again, so it can now indeed allocate that much. > I have not yet figured out where these vm_allocate calls and/or their > huge size parameters are coming from. Probably the real source of the issue :) Samuel
Re: Unreclaimed swap space upon process termination?
Hi! On Mon, 28 Nov 2016 16:03:44 +0100, I wrote: > Updating a Debian GNU/Hurd virtual machine to recent packages after many > months, and then running the GCC testsuite, I observe the following > behavior, which should be reproducible with the executable in the > attached tarball: > > $ vmstat | grep swap\ free > swap free: 4096M > $ ./1.exe > $ vmstat | grep swap\ free > swap free: 3288M > $ ./1.exe > $ vmstat | grep swap\ free > swap free: 2495M > $ ./1.exe > $ vmstat | grep swap\ free > swap free: 1726M > $ ./1.exe > $ vmstat | grep swap\ free > swap free: 931M > $ ./1.exe > $ vmstat | grep swap\ free > swap free: 164M > $ ./1.exe > Bus error > $ vmstat | grep swap\ free > swap free:0 > > At this point, the system doesn't recover from this low memory situation. > > For each invocation of the executable, there are three "no more room in > [...] (./1.exe([...])" messages on the Mach console. > > The executable is compiled from > [gcc]/libstdc++-v3/testsuite/21_strings/basic_string/modifiers/insert/char/1.cc > from commit a050099a416f013bda35832b878d9a57b0cbb231 (gcc-6-branch branch > point; 2016-04-15), which doesn't look very spectacular -- apart from > maybe the __gnu_test::set_memory_limits call, which I'll try to figure > out what it does. That uses setrlimit for RLIMIT_DATA, RLIMIT_RSS, RLIMIT_VMEM, RLIMIT_AS, but there is no change with that call removed. > But nevertheless, unreclaimed swap space upon process > termination sounds like a bug? > > Unless this a known issue, or somebody can quickly pinpoint the problem, > I'll try to bisect core system packages, between the version of the > "good" and "bad" disk images. Running with rpctrace, I see that on the old ("good") system, at the end of the process, we got: [...] task52(pid2198)->vm_deallocate (16973824 16) = 0 task52(pid2198)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space available) task52(pid2198)->vm_allocate (0 -2147348480 1) = 0x3 ((os/kern) no space available) task52(pid2198)->vm_map (0 2097152 0 1 (null) 0 1 0 7 1) = 0 21405696 task52(pid2198)->vm_deallocate (21405696 614400) = 0 task52(pid2198)->vm_deallocate (23068672 434176) = 0 task52(pid2198)->vm_protect (22020096 135168 0 3) = 0 task52(pid2198)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space available) 61<--68(pid2198)->proc_mark_exit_request (0 0) = 0 task52(pid2198)->task_terminate () = 0 Child 2198 exited with 0 ..., but on the new ("bad") system, the first non-sensical (huge; -2147479552 is 0x80001000) vm_allocate call actually succeeds: [...] task154(pid1080)->vm_deallocate (16973824 16) = 0 task154(pid1080)->vm_allocate (0 -2147479552 1) = 0 268742656 task154(pid1080)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space available) task154(pid1080)->vm_allocate (0 -2147348480 1) = 0x3 ((os/kern) no space available) task154(pid1080)->vm_map (0 2097152 0 1 (null) 0 1 0 7 1) = 0 2162 task154(pid1080)->vm_deallocate (2162 364544) = 0 task154(pid1080)->vm_deallocate (23068672 684032) = 0 task154(pid1080)->vm_protect (22020096 135168 0 3) = 0 task154(pid1080)->vm_allocate (0 -2147479552 1) = 0x3 ((os/kern) no space available) task154(pid1080)->vm_deallocate (268742656 -2147479552) = 0 163<--170(pid1080)->proc_mark_exit_request (0 0) = 0 task154(pid1080)->task_terminate () = 0 Child 1080 exited with 0 I have not yet figured out where these vm_allocate calls and/or their huge size parameters are coming from. Grüße Thomas
Re: Unreclaimed swap space upon process termination?
Hello, Thomas Schwinge, on Mon 28 Nov 2016 16:03:44 +0100, wrote: > But nevertheless, unreclaimed swap space upon process > termination sounds like a bug? It's actually not really a new problem. It has just been emphasized more with recent memory management changes. Samuel
Re: Unreclaimed swap space upon process termination?
On Mon, Nov 28, 2016 at 04:03:44PM +0100, Thomas Schwinge wrote: > Unless this a known issue, or somebody can quickly pinpoint the problem, > I'll try to bisect core system packages, between the version of the > "good" and "bad" disk images. It's a known issue, a side effect of the page cache changes that I merged some months ago. Because of increased cache usage, there is more memory pressure, and therefore more swap usage (mostly through double paging, since the kernel always tries to evict external pages first). The default pager has never been able to correctly reclaim these pages in the past, but now there are a lot more of them. -- Richard Braun
Unreclaimed swap space upon process termination?
Hi! Updating a Debian GNU/Hurd virtual machine to recent packages after many months, and then running the GCC testsuite, I observe the following behavior, which should be reproducible with the executable in the attached tarball: $ vmstat | grep swap\ free swap free: 4096M $ ./1.exe $ vmstat | grep swap\ free swap free: 3288M $ ./1.exe $ vmstat | grep swap\ free swap free: 2495M $ ./1.exe $ vmstat | grep swap\ free swap free: 1726M $ ./1.exe $ vmstat | grep swap\ free swap free: 931M $ ./1.exe $ vmstat | grep swap\ free swap free: 164M $ ./1.exe Bus error $ vmstat | grep swap\ free swap free:0 At this point, the system doesn't recover from this low memory situation. For each invocation of the executable, there are three "no more room in [...] (./1.exe([...])" messages on the Mach console. The executable is compiled from [gcc]/libstdc++-v3/testsuite/21_strings/basic_string/modifiers/insert/char/1.cc from commit a050099a416f013bda35832b878d9a57b0cbb231 (gcc-6-branch branch point; 2016-04-15), which doesn't look very spectacular -- apart from maybe the __gnu_test::set_memory_limits call, which I'll try to figure out what it does. But nevertheless, unreclaimed swap space upon process termination sounds like a bug? Unless this a known issue, or somebody can quickly pinpoint the problem, I'll try to bisect core system packages, between the version of the "good" and "bad" disk images. Grüße Thomas 1.tar.xz Description: application/xz