[Bug ld/22831] ld causes massive thrashing if object files are not fully memory-resident: new algorithm needed
https://sourceware.org/bugzilla/show_bug.cgi?id=22831 --- Comment #35 from Luke Kenneth Casson Leighton --- On Sat, Jul 23, 2022 at 3:04 PM amodra at gmail dot com wrote: > And "new algorithm needed" is really saying "rewrite the linker". i mention this very early on in this bugreport: back in the early 90s it was indeed rewritten, to remove Dr Stallman's algorithms, on the flawed assumption "640k^H^H^H^H 4GB should be enough for anybody". > That's low > priority. Also, there are other linkers, eg. gold and lld, that are much > newer > than ld.bfd. gold suffers from similar problems - i was able to make it keel over just as easily. i've not heard of lld before: if it likewise makes the same flawed assumption that going into swap is acceptable, it will likewise result in the exact same problem. > They don't do much better at memory usage, do they? if Dr Stallman's carefully-crafted original algorithms had been left in place, which, just as in gcc, made *really certain* to only use *resident* RAM, we would not be having this conversation as this bugreport would not need to be raised. the fundamental flawed assumption is that it's "ok to use swap". the sheer overwhelming amount of cross-referencing required in a linker *100% guarantee* that even 10 kbytes over resident RAM will result in thrashing. any rewrite or redesign that does not take that into account is 100% guaranteed to be problematic. this is just how it is: it's basic fundamental computer science that a linker *has* to jump around across the entirety of *all* of the objects it's trying to link. this makes the "Working Set" *equal* to 100% of the available Swap, which is unfortunately the very definition of "thrash conditions". -- You are receiving this mail because: You are on the CC list for the bug.
[Bug ld/22831] ld causes massive thrashing if object files are not fully memory-resident: new algorithm needed
https://sourceware.org/bugzilla/show_bug.cgi?id=22831 --- Comment #34 from Alan Modra --- I'll note that the priority and severity fields in bugzilla are primarily for the use of maintainers, or at least that should be the way they are treated. They are not for bug reporters to say "this bug is really, really important!" That said, I've experienced exactly the pain you ran into with a machine swapping like crazy, in fact it used to happen to me quite regularly. And some things we do, like trying to free memory before exit to pacify people crying "memory leak!" only make things worse when you run into swap. I had one link take 30 minutes extra just freeing memory.. In putting the severity at "enhancement" I'm merely reflecting reality. Using more memory than necessary is not a bug, at least not until you run out of memory. Even with ideal memory usage you will always be able to generate a workload that is just too big to handle. And "new algorithm needed" is really saying "rewrite the linker". That's low priority. Also, there are other linkers, eg. gold and lld, that are much newer than ld.bfd. They don't do much better at memory usage, do they? -- You are receiving this mail because: You are on the CC list for the bug.
[Bug ld/22831] ld causes massive thrashing if object files are not fully memory-resident: new algorithm needed
https://sourceware.org/bugzilla/show_bug.cgi?id=22831 --- Comment #33 from luke.leighton at gmail dot com --- that was supposed to be a private reply, bugzilla masked the email address "amodra ". the comment still stands though. i apologise for the toppost context. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug ld/22831] ld causes massive thrashing if object files are not fully memory-resident: new algorithm needed
https://sourceware.org/bugzilla/show_bug.cgi?id=22831 --- Comment #32 from luke.leighton at gmail dot com --- (replying privately) dealing with this one was deeply unpleasant. i gave up as people were not listening. i refer people to it frequently whenever they encounter serious build problems. the torture demo is dead easy to autogenerate programs that crash both ld and gold, for both 32 and 64 bit. for 64 bit just keep increasing the parameters until programs exceed 16 gbytes in size and in some cases they won't even link at all. there are multiple complaints by distro builders that their 128 GB and 256 GB build farms actually kernel panic if they happen to accidentally have e.g. firefox, libreoffice and other massive linking occur simultaneously, due to thrashing. with 128 GB of RAM! i have had my very expensive laptop hit 1,200 loadavg due to this problem, it nearly lost me a year's work and took 25 minutes to get the cursor to move so i could hold down Ctrl-C and terminate the build. it's exacerbated significantly by debug builds. l. On July 23, 2022 3:37:03 AM GMT+01:00, amodra at gmail dot com wrote: >https://sourceware.org/bugzilla/show_bug.cgi?id=22831 > >Alan Modra changed: > > What|Removed |Added > > Severity|critical|enhancement > Status|WAITING |NEW > Priority|P1 |P3 > >--- Comment #31 from Alan Modra --- >Putting priority and severity back where they belong. > >-- >You are receiving this mail because: >You reported the bug. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug ld/12564] GNU ld internal ironly section should not be leaking warnings
https://sourceware.org/bugzilla/show_bug.cgi?id=12564 Alan Modra changed: What|Removed |Added Status|REOPENED|RESOLVED Resolution|--- |FIXED --- Comment #4 from Alan Modra --- Fixed a long time ago. -- You are receiving this mail because: You are on the CC list for the bug.
