[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 Sam James changed: What|Removed |Added Resolution|--- |FIXED Status|UNCONFIRMED |RESOLVED --- Comment #24 from Sam James --- I think we can call this fixed via the split improvements. It's in 14 but it's also in 13 downstream in Gentoo where Stan reported it from.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #23 from Richard Biener --- Just to summarize we are talking about the six commits r14-496-g580cda3c2799b1 r14-497-ge487fcc0f7466e r14-498-gc0ce29bc1ce329 r14-499-g27fcf994c5515e r14-500-g703417a030b3d8 r14-501-g0a85544e1aaeca there were related followups in r14-535-g319aef8d2fde32, r14-570-g31c70a7daa3687, r14-599-g2ed6dd975d1da8, r14-600-g425198bb5d2e30 and r14-601-g153eafaa0b7449
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #22 from Sam James --- Thanks for your help. We've had this in production since 26th May and no complaints (but a lot of happy customers who couldn't build 12 or earlier 13 versions): https://gitweb.gentoo.org/proj/gcc-patches.git/commit/?id=7a6b2d23ec02e75475a6123254ccd44d73827f39. Now that 13.2 is done, could this be pushed to the branch for 13.3?
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #21 from rguenther at suse dot de --- On Thu, 25 May 2023, sjames at gcc dot gnu.org wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #20 from Sam James --- > When I looked at it, I think I got it to apply to 13 with no hassle and it > seemed to work okay, but I didn't test it that hard. > > It's a considerable win so even if not backported upstream, if you think it's > safe enough, let us know if it's ok for us to do it downstream in Gentoo. > > (That could also serve as a form of testing to do it upstream on > releases/gcc-13 if desired.) yes, it's safe (unless there are not yet noticed portability issues with the Makefile setup)
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #20 from Sam James --- When I looked at it, I think I got it to apply to 13 with no hassle and it seemed to work okay, but I didn't test it that hard. It's a considerable win so even if not backported upstream, if you think it's safe enough, let us know if it's ok for us to do it downstream in Gentoo. (That could also serve as a form of testing to do it upstream on releases/gcc-13 if desired.)
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #19 from rguenther at suse dot de --- On Wed, 24 May 2023, userm57 at yahoo dot com wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #18 from Stan Johnson --- > $ git clone git://gcc.gnu.org/git/gcc.git > $ cd gcc > $ git checkout master > > I'm testing a manual bootstrap of "gcc version 14.0.0 20230524 (experimental) > (GCC)" now, accessed via git as shown above. > > It will still take about 24 more hours for the bootstrap to finish (I'll send > an update if it fails), but with gimple-match.cc (and generic-match.cc, which > was not affected in my tests) split up, it looks like it will finish ok (it's > currently in about the middle of stage 2 and has successfully compiled all the > gimple-match-n.cc files). > > Note that Gentoo's emerge of gcc-13 behaves a little differently than a manual > bootstrap. I don't know why, since I think I'm using Gentoo's ./configure > options in the manual bootstrap, but in Gentoo's emerge of gcc, they seem to > run cc1plus and "as" simultaneously for each compilation, perhaps aggravating > the memory issue for gimple-match.cc (or maybe not, since the problem is > virtual memory exhausted, not swap space exhausted). Possibly -pipe is used somehow. > Anyway, it looks like the solution was already close. Does anyone know whether > the change will be backported to gcc-12 or gcc-13 available from > ftp.gnu.org/pub/gnu/gcc? That wasn't planned. It's very unlikely to happen for gcc 12, it should be possible for gcc 13 I think.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #18 from Stan Johnson --- $ git clone git://gcc.gnu.org/git/gcc.git $ cd gcc $ git checkout master I'm testing a manual bootstrap of "gcc version 14.0.0 20230524 (experimental) (GCC)" now, accessed via git as shown above. It will still take about 24 more hours for the bootstrap to finish (I'll send an update if it fails), but with gimple-match.cc (and generic-match.cc, which was not affected in my tests) split up, it looks like it will finish ok (it's currently in about the middle of stage 2 and has successfully compiled all the gimple-match-n.cc files). Note that Gentoo's emerge of gcc-13 behaves a little differently than a manual bootstrap. I don't know why, since I think I'm using Gentoo's ./configure options in the manual bootstrap, but in Gentoo's emerge of gcc, they seem to run cc1plus and "as" simultaneously for each compilation, perhaps aggravating the memory issue for gimple-match.cc (or maybe not, since the problem is virtual memory exhausted, not swap space exhausted). Anyway, it looks like the solution was already close. Does anyone know whether the change will be backported to gcc-12 or gcc-13 available from ftp.gnu.org/pub/gnu/gcc? Thanks to all of the GNU developers who continue to make modern tools available for use on old hardware!
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #17 from Andreas Schwab --- The linker just uses TEXT_START_ADDR=0x8000, but mmap can use any address.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #16 from Richard Biener --- (In reply to Andreas Schwab from comment #15) > TASK_SIZE is 0xF000UL on m68k. That would mean ~3.75GB virtual address space is available. The cited /proc/maps though looks like the lower half isn't available? It would be probably interesting to strace the process that will ultimatively run out of virtual memory to see how glibc places allocations. A simple test program repeatedly creating anonymous mmaps of a larger size (2MB?) might also get an upper bound.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #15 from Andreas Schwab --- TASK_SIZE is 0xF000UL on m68k.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #14 from Stan Johnson --- I can try to capture the offending "cc1plus" and "as" processes just before the compilation of gimple-match.cc fails in stage2; the output will look something like this (for a different cc1plus process number 19431 that is currently running): # cat /proc/19431/maps 8000-82027000 r-xp 08:05 1823425 /var/tmp/portage/sys-devel/gcc-13.1.1_p20230520/work/build/prev-gcc/cc1plus 82028000-8202a000 r--p 02028000 08:05 1823425 /var/tmp/portage/sys-devel/gcc-13.1.1_p20230520/work/build/prev-gcc/cc1plus 8202a000-82036000 rw-p 0202a000 08:05 1823425 /var/tmp/portage/sys-devel/gcc-13.1.1_p20230520/work/build/prev-gcc/cc1plus 82036000-8207 rwxp 00:00 0 [heap] 8207-82601000 rwxp 00:00 0 [heap] c000-c002 r-xp 08:05 1682956/lib/ld.so.1 c002-c0022000 r--p 0002 08:05 1682956/lib/ld.so.1 c0022000-c0023000 rw-p 00022000 08:05 1682956/lib/ld.so.1 c0023000-c0026000 rw-p 00:00 0 c0028000-c0044000 r-xp 08:05 1515049/usr/lib/libmpc.so.3.3.1 c0044000-c0045000 ---p 0001c000 08:05 1515049/usr/lib/libmpc.so.3.3.1 c0045000-c0046000 r--p 0001b000 08:05 1515049/usr/lib/libmpc.so.3.3.1 c0046000-c0047000 rw-p 0001c000 08:05 1515049/usr/lib/libmpc.so.3.3.1 c0048000-c00b3000 r-xp 08:05 1672972/usr/lib/libmpfr.so.6.2.0 c00b3000-c00b5000 ---p 0006b000 08:05 1672972/usr/lib/libmpfr.so.6.2.0 c00b5000-c00b6000 r--p 0006b000 08:05 1672972/usr/lib/libmpfr.so.6.2.0 c00b6000-c00b7000 rw-p 0006c000 08:05 1672972/usr/lib/libmpfr.so.6.2.0 c00b8000-c0113000 r-xp 08:05 1515186/usr/lib/libgmp.so.10.4.1 c0113000-c0115000 ---p 0005b000 08:05 1515186/usr/lib/libgmp.so.10.4.1 c0115000-c0116000 r--p 0005b000 08:05 1515186/usr/lib/libgmp.so.10.4.1 c0116000-c0117000 rw-p 0005c000 08:05 1515186/usr/lib/libgmp.so.10.4.1 c0118000-c012a000 r-xp 08:05 411479 /lib/libz.so.1.2.13 c012a000-c012b000 ---p 00012000 08:05 411479 /lib/libz.so.1.2.13 c012b000-c012c000 r--p 00011000 08:05 411479 /lib/libz.so.1.2.13 c012c000-c012d000 rw-p 00012000 08:05 411479 /lib/libz.so.1.2.13 c012e000-c0172000 r-xp 08:05 1682625/lib/libm.so.6 c0172000-c0173000 ---p 00044000 08:05 1682625/lib/libm.so.6 c0173000-c0174000 r--p 00043000 08:05 1682625/lib/libm.so.6 c0174000-c0176000 rw-p 00044000 08:05 1682625/lib/libm.so.6 c0176000-c02df000 r-xp 08:05 1682906/lib/libc.so.6 c02df000-c02e ---p 00169000 08:05 1682906/lib/libc.so.6 c02e-c02e2000 r--p 0016a000 08:05 1682906/lib/libc.so.6 c02e2000-c02e6000 rw-p 0016c000 08:05 1682906/lib/libc.so.6 c02e6000-c0ad2000 rw-p 00:00 0 c0ad5000-c25dd000 rw-p 00:00 0 ef91f000-ef948000 rw-p 00:00 0 [stack] > Git checkout master Thanks, I'll try compiling that version manually and also capture the relevant /proc output.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #13 from rguenther at suse dot de --- > Am 23.05.2023 um 19:44 schrieb userm57 at yahoo dot com > : > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #12 from Stan Johnson --- >> That’s indeed unhelpful. Cat /proc/maps of some executable might help > > Sorry, I don't know how to do that. I don't see /proc/maps. Try /proc/1/maps It’s a file in the per process directories > Regarding downloading mainline gcc-14, it's not at ftp.gnu.org. I tried the > following, but I must be using the wrong repository. Please advise on where I > should find gcc-14. > > $ git clone git://gcc.gnu.org/git/gcc.git > $ cd gcc > $ git reset --hard > HEAD is now at 0d2ab259bc5 Improve cost computation for single-bit bit > insertions. > $ git checkout releases/gcc-14 > error: pathspec 'releases/gcc-14' did not match any file(s) known to git > $ git checkout releases/gcc-13 > Already on 'releases/gcc-13' > Your branch is up to date with 'origin/releases/gcc-13'. > $ git checkout mainline/gcc-14 > error: pathspec 'mainline/gcc-14' did not match any file(s) known to git Git checkout master Should do what I meant
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #11 from rguenther at suse dot de --- > Am 23.05.2023 um 19:28 schrieb userm57 at yahoo dot com > : > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #10 from Stan Johnson --- >> The question is how much virtual memory is exposed to a user >> process, that is - how large is the address space? > > I'm not sure, but I see this: > > $ prlimit > RESOURCE DESCRIPTION SOFT HARD UNITS > AS address space limitunlimited unlimited bytes That’s indeed unhelpful. Cat /proc/maps of some executable might help > CORE max core file size 0 unlimited bytes > CPUCPU time unlimited unlimited seconds > DATA max data size unlimited unlimited bytes > FSIZE max file size unlimited unlimited bytes > LOCKS max number of file locks held unlimited unlimited locks > MEMLOCKmax locked-in-memory address space 8388608 8388608 bytes > MSGQUEUE max bytes in POSIX mqueues819200819200 bytes > NICE max nice prio allowed to raise 0 0 > NOFILE max number of open files1024 4096 files > NPROC max number of processes26236 26236 processes > RSSmax resident set size unlimited unlimited bytes > RTPRIO max real-time priority 0 0 > RTTIME timeout for real-time tasksunlimited unlimited microsecs > SIGPENDING max number of pending signals 26236 26236 signals > STACK max stack size 8388608 unlimited bytes > > As long as those are also the limits for portage, it should be ok. > >> Note mainline would be gcc 14.0, you can probably download a recent >> snapshot. > > ok, I've downloaded the latest gcc snapshot using git. I'll try a manual > bootstrap of that. Do you recommend that I use QEMU m68k (virt) running Debian > SID or Gentoo, or doesn't it matter? I don’t think that matters.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #10 from Stan Johnson --- > The question is how much virtual memory is exposed to a user > process, that is - how large is the address space? I'm not sure, but I see this: $ prlimit RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limitunlimited unlimited bytes CORE max core file size 0 unlimited bytes CPUCPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited bytes LOCKS max number of file locks held unlimited unlimited locks MEMLOCKmax locked-in-memory address space 8388608 8388608 bytes MSGQUEUE max bytes in POSIX mqueues819200819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files1024 4096 files NPROC max number of processes26236 26236 processes RSSmax resident set size unlimited unlimited bytes RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasksunlimited unlimited microsecs SIGPENDING max number of pending signals 26236 26236 signals STACK max stack size 8388608 unlimited bytes As long as those are also the limits for portage, it should be ok. > Note mainline would be gcc 14.0, you can probably download a recent > snapshot. ok, I've downloaded the latest gcc snapshot using git. I'll try a manual bootstrap of that. Do you recommend that I use QEMU m68k (virt) running Debian SID or Gentoo, or doesn't it matter?
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #9 from rguenther at suse dot de --- On Tue, 23 May 2023, userm57 at yahoo dot com wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #8 from Stan Johnson --- > > How much virtual memory does the m68k host have? > > Swap space in the m68k virt VM is configurable; I'm using 1 GiB in two 512 MiB > partitions. I noticed that compilation goes further with two 512 MiB > partitions > that with one 1 GiB partition, even though the second 512 MiB swap partition > isn't used (see above). And I know I tested 2 GiB swap at one point, and that > also failed. The virt VM is configured with 3399672K memory. The question is how much virtual memory is exposed to a user process, that is - how large is the address space? > While observing the failure, it appears to happen when the assembler "as" > becomes more active (so probably when cc1plus is about done). Assembler memory > increases quickly, and swap space is not really exhausted, despite the error > message. That's odd, I never noticed the assembler taking much memory. > > Can you try current mainline to see if that's enough to pass native > > bootstrap in qemu? > > Native (manual) bootstrap of gcc-13.1 in QEMU (virt and q800) failed in > Gentoo, > but the failure was slightly different (it failed in stage 3 instead of stage > 2). And I used Gentoo's configure options, as documented above. I could try no > configure options, followed by "make bootstrap"; would that be helpful? If > yes, > should I use Gentoo or Debian SID? Note mainline would be gcc 14.0, you can probably download a recent snapshot.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #8 from Stan Johnson --- > How much virtual memory does the m68k host have? Swap space in the m68k virt VM is configurable; I'm using 1 GiB in two 512 MiB partitions. I noticed that compilation goes further with two 512 MiB partitions that with one 1 GiB partition, even though the second 512 MiB swap partition isn't used (see above). And I know I tested 2 GiB swap at one point, and that also failed. The virt VM is configured with 3399672K memory. While observing the failure, it appears to happen when the assembler "as" becomes more active (so probably when cc1plus is about done). Assembler memory increases quickly, and swap space is not really exhausted, despite the error message. > Can you try current mainline to see if that's enough to pass native > bootstrap in qemu? Native (manual) bootstrap of gcc-13.1 in QEMU (virt and q800) failed in Gentoo, but the failure was slightly different (it failed in stage 3 instead of stage 2). And I used Gentoo's configure options, as documented above. I could try no configure options, followed by "make bootstrap"; would that be helpful? If yes, should I use Gentoo or Debian SID?
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 Richard Biener changed: What|Removed |Added CC||schwab at gcc dot gnu.org --- Comment #7 from Richard Biener --- How much virtual memory does the m68k host have? Is it possible to extend that to 2GB? ISTR that linux on 32bit platforms had a 1GB userland virtual memory setup at some point (and maybe for some platforms). In theory it's possible to backport the split of the gimple-match.cc TUs. Can you try current mainline to see if that's enough to pass native bootstrap in qemu?
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #6 from Stan Johnson --- Thanks, let me know if you need me to check anything here. The problem has existed since at least gcc-12. Initially I thought the QEMU q800 VM had run out of memory, but switching to a virt VM with ~3 GB memory and 1 GiB swap also failed. Further checking showed that the virt VM was not actually out of memory (unless I wasn't able to monitor memory changes quickly enough looking at /proc/swaps and /proc/meminfo every minute).
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 Sam James changed: What|Removed |Added CC||sjames at gcc dot gnu.org, ||toolchain at gentoo dot org --- Comment #5 from Sam James --- Thanks for reporting this upstream, Stan. We've been hitting this for a while in our QEMU builds in Gentoo but we hadn't got to the point where we'd figured out who to report it to.
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 --- Comment #4 from Stan Johnson --- Adding more swap space doesn't help. The latest Gentoo update attempted to merge gcc-13.1.1_p20230520, and it failed while compiling gimple-match.cc in stage2. The error message is "virtual memory exhausted: Cannot allocate memory". Here are some memory details just before the failure: ./work/build/./prev-gcc/cc1plus VSZ=920260 RSS=632200 /usr/m68k-unknown-linux-gnu/bin/as VSZ=265260 RSS=258608 There is still swap space available: # cat /proc/swaps Filename Type Size Used Priority /dev/sda7 partition 524284225008-2 /dev/sda8 partition 5242840 -3
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 Richard Biener changed: What|Removed |Added Target||m68k --- Comment #3 from Richard Biener --- assign more virtual memory? the other option is to figure out what exactly uses so much memory in a native m68k compiler ... if stage2 is fine then using -g0 for the object might help. x86_64 native needs 1.3GB and using -O1 doesn't make a difference there but -g1 shrinks memory requirements to 750MB (similar to using -g -fno-var-tracking).
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 Xi Ruoyao changed: What|Removed |Added CC||xry111 at gcc dot gnu.org --- Comment #2 from Xi Ruoyao --- (In reply to Andrew Pinski from comment #1) > I wonder if splitting up gimple-match that happened on the trunk fixes this > issue. I guess the issue is just "compiling giant TU has exhausted the memory" so it should be fixed now. But how to handle it for stable branches? :(
[Bug target/109927] Bootstrap fails for m68k in stage2 compilation of gimple-match.cc
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 Andrew Pinski changed: What|Removed |Added Keywords||build Component|bootstrap |target --- Comment #1 from Andrew Pinski --- I wonder if splitting up gimple-match that happened on the trunk fixes this issue.
