[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-19 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 Andi Kleen andi-gcc at firstfloor dot org changed: What|Removed |Added Status|UNCONFIRMED |RESOLVED

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-17 Thread ak at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #15 from ak at gcc dot gnu.org 2011-10-17 14:43:45 UTC --- Author: ak Date: Mon Oct 17 14:43:37 2011 New Revision: 180093 URL: http://gcc.gnu.org/viewcvs?root=gccview=revrev=180093 Log: Use MADV_DONTNEED for freeing in garbage

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #11 from Andi Kleen andi-gcc at firstfloor dot org 2011-10-08 16:47:54 UTC --- Created attachment 25445 -- http://gcc.gnu.org/bugzilla/attachment.cgi?id=25445 patchkit I tested this patchkit which implements most of the ideas from

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 Andi Kleen andi-gcc at firstfloor dot org changed: What|Removed |Added Attachment #25445|0 |1 is

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #13 from Jakub Jelinek jakub at gcc dot gnu.org 2011-10-08 20:14:14 UTC --- In the 0001-initial-madvise.patch patch I think if you subtract the size MADV_DONTNEEDed from G.bytes_mapped, then you should add it again in alloc_page (i.e.

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #14 from Andi Kleen andi-gcc at firstfloor dot org 2011-10-08 21:10:13 UTC --- Thanks for the review. Fixed the accounting I'll leave the xmalloc_failed hook out for now: it would need a retry path which is somewhat complicated. If

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-07 Thread rguenth at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #8 from Richard Guenther rguenth at gcc dot gnu.org 2011-10-07 08:25:37 UTC --- (In reply to comment #4) Right now alloc_page will (usually) allocate in GGC_QUIRE_SIZE * G.pagesize chunks, but release_pages will munmap individual

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-07 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #9 from Jakub Jelinek jakub at gcc dot gnu.org 2011-10-07 10:01:32 UTC --- (In reply to comment #2) Perhaps on Linux, at least with recentish kernel, we could change release_pages into keeping the pages in the G.free_pages chain,

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-07 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #10 from Andi Kleen andi-gcc at firstfloor dot org 2011-10-07 14:44:10 UTC --- To track the pattern you can simply use strace or ftrace (I did ftrace) I checked the kernel code now and if the madvise is big enough it won't split up

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread hubicka at ucw dot cz
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #1 from Jan Hubicka hubicka at ucw dot cz 2011-10-06 19:20:45 UTC --- When doing a very large LTO build I fail with out of virtual memory Some investigation showed the problem was not actually running out of memory, but gcc

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 Jakub Jelinek jakub at gcc dot gnu.org changed: What|Removed |Added CC||jakub at gcc dot

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #3 from Andi Kleen andi-gcc at firstfloor dot org 2011-10-06 21:31:56 UTC --- I would prefer to free in 2MB chunks if possible I was experimenting with increasing the quire size from 1 to 2MB so that a modern kernel with transparent

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #4 from Jakub Jelinek jakub at gcc dot gnu.org 2011-10-06 21:41:41 UTC --- Right now alloc_page will (usually) allocate in GGC_QUIRE_SIZE * G.pagesize chunks, but release_pages will munmap individual pages immediately during

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #5 from Andi Kleen andi-gcc at firstfloor dot org 2011-10-06 21:46:32 UTC --- If it's a 2MB page then madvise MADV_DONTNEED will split it if it's not 2MB aligned. It would be good to optimize the freeing pattern so that this happens

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #6 from Andi Kleen andi-gcc at firstfloor dot org 2011-10-07 05:47:54 UTC --- *** Bug 50302 has been marked as a duplicate of this bug. ***

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #7 from Andi Kleen andi-gcc at firstfloor dot org 2011-10-07 05:50:40 UTC --- *** Bug 50511 has been marked as a duplicate of this bug. ***