Hey Gabe, How hard would it be to have an option for partial linking? If we could have a --force-lto option that did the opposite (enable lto and disable partial linking), that would be great. But if this would be a giant scons pain, I understand :).
Jason On Mon, Jun 5, 2017 at 5:21 PM Andreas Hansson <[email protected]> wrote: > I think dropping LTO for the affected compilers is probably the right > option, but it is potentially quite a blow to performance if we simply > leave it at that. Back when I profiled LTO usage I remember seeing 10+ > percent difference with and without LTO. > > Andreas > > From: gem5-users <[email protected]> on behalf of Jason > Lowe-Power <[email protected]> > Reply-To: gem5 users mailing list <[email protected]> > Date: Monday, 5 June 2017 at 23:16 > To: Gabe Black <[email protected]> > Cc: gem5 users mailing list <[email protected]> > > Subject: Re: [gem5-users] Link error building gem5.fast > > I'm fine with dropping LTO on gcc 4.9-6. (It also works on gcc 4.8 for me, > BTW.) I it's also failing for gcc 6 (see > https://travis-ci.org/powerjg/gem5-ci-test). > > If you make a patch for this, could you add some detailed comments so we > can "undo" this change when we all move to gcc 7+? > > Thanks for tracking this down! > > Jason > > On Mon, Jun 5, 2017 at 4:47 PM Gabe Black <[email protected]> wrote: > >> I also used this thread/bug as a basis for my diagnosis. >> >> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67548 >> >> On Mon, Jun 5, 2017 at 2:40 PM, Gabe Black <[email protected]> wrote: >> >>> Ok, I did some digging around about this, and while a lot of the talk >>> between gcc devs sounds like gibberish to me, I did pick out bits and >>> pieces which makes me think this is just something that's broken in gcc of >>> particular versions. There's some documentation of the state of things in >>> gcc version 6 where it says things are fixed to the point where they can be >>> worked with, and that in version 7 they expect them to be totally fixed up: >>> >>> https://gcc.gnu.org/gcc-6/changes.html >>> >>> The problem seems to be that there were assumptions being made in the >>> LTO gcc plugin that the link was a final link all the time, and that weak >>> external symbols (the inline functions that might show up multiple times) >>> should be promoted to regular external symbols. That breaks things since, >>> in subsequent links, there are more copies of them and they all look like >>> they're supposed to be unique. >>> >>> What I think we should do is detect what situation we're in and react >>> accordingly. In gcc version 5.0-6.0 (maybe 4.9 or 4.8, although 4.8 seems >>> to work for me) we need to avoid doing either partial linking or LTO since >>> the combination doesn't work. It would be easier to make scons avoid LTO >>> since that would just be changing the compiler flags and not structurally >>> changing how the build flows or how the scons scripts are written. >>> >>> In versions 6.0 to 7.0, if those release notes are to be believed, we >>> would need to decide between doing the partial link with gcc or with ld. It >>> sounds like ld is better since it would still do whole program >>> optimization, although again it might be more difficult to twist scon's arm >>> and get it to do things that way since its linker command line uses gcc. We >>> could pass -r to ld directly by wrapping it in -Wl,-r, but that apparently >>> caused issues for people on macs. >>> >>> And then, in the glorious future where we're using gcc v7 which works >>> properly, or if you're using clang, we can stop worrying about it since -r >>> and -lto will play nicely together. >>> >>> Thoughts? >>> >>> Gabe >>> >>> On Sun, Jun 4, 2017 at 3:11 PM, Jason Lowe-Power <[email protected]> >>> wrote: >>> >>>> Hi Gabe, >>>> >>>> I think the follow "just works" by downloading the image from dockerhub. >>>> >>>> > docker run -v `pwd`:`pwd` -it powerjg/gem5-gcc-6-build /bin/bash >>>> >>>> Or, to just download the image: >>>> >>>> > docker pull powerjg/gem5-gcc-6-build >>>> >>>> Let me know if that doesn't work for you. >>>> >>>> I searched around some to try to track down the issue. It seems that >>>> GCC doesn't love doing LTO and partial linking. There have been a number of >>>> bugs related to this in the past. But, all of the bugs that I found had >>>> been fixed before gcc 5. >>>> >>>> Cheers, >>>> Jason >>>> >>>> On Sun, Jun 4, 2017 at 4:37 PM Gabe Black <[email protected]> wrote: >>>> >>>>> I was hoping we could leave the lto in the .cc->.o but take it out of >>>>> the partial links, and that that would fix the problem. If you have a >>>>> docker image I can try this all out on, that would be helpful. If I can >>>>> get >>>>> ahold of that somehow, I can try to figure out how to fix this on Monday. >>>>> I'd speculate what's happening is that little inline functions, etc., >>>>> which >>>>> I think are called "common" symbols are showing up more than once. >>>>> Normally >>>>> that's ok and the linker will figure that out and get rid of the extra >>>>> copies, but apparently here it isn't doing that. There's a scons variable >>>>> which you can use to disable lto in the fast build if you want a >>>>> workaround >>>>> for right now, although using gem5.fast is a bit dangerous in the first >>>>> place since I think it disables asserts, etc. >>>>> >>>>> Gabe >>>>> >>>>> On Sun, Jun 4, 2017 at 2:11 PM, Jason Lowe-Power <[email protected]> >>>>> wrote: >>>>> >>>>>> I tried passing -no-lto with ['-r', '-nostdlib'] on line >>>>>> SConstruct:688. However, that caused the error/warning: >>>>>> "/usr/bin/ld: build/X86/dev/io_device.fo: plugin needed to handle >>>>>> lto object" for every lib.fo.partial. I guess the -lto option during >>>>>> .cc->.o puts some data in the .o. Then, when partially linking with -r >>>>>> gcc >>>>>> gets confused? >>>>>> >>>>>> Should I filter out the -lto=8 from all of the .cc->.o? I could have >>>>>> mis-understood how to do this. >>>>>> >>>>>> Jason >>>>>> >>>>>> On Sun, Jun 4, 2017 at 4:08 PM Gabe Black <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Did you try filtering out the lto option from the partial link steps >>>>>>> like I suggested? Look at how I do that for -shared as an example. >>>>>>> >>>>>>> Gabe >>>>>>> >>>>>>> On May 24, 2017 2:00 PM, "Jason Lowe-Power" <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi everyone, >>>>>>>> >>>>>>>> So I've been able to reproduce the problem. I would bet it's due to >>>>>>>> the new partial linking code ( >>>>>>>> https://gem5.googlesource.com/public/gem5/+/6bdd897f04f4efdf90d0761c6d31d3f960f4eacf). >>>>>>>> I'm not sure what the solution is, yet, or if I'll have time to look >>>>>>>> at it >>>>>>>> in the next few day. Gabe might have an idea, though, if that is the >>>>>>>> problem. >>>>>>>> >>>>>>>> Here's a matrix of what compilers are working and which aren't >>>>>>>> (gcc-4.8 is working, too, though not tested on travis). >>>>>>>> https://travis-ci.org/powerjg/gem5-ci-test/builds/235779432 >>>>>>>> >>>>>>>> Jason >>>>>>>> >>>>>>>> On Tue, May 23, 2017 at 4:33 PM Moussa, Ayman < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> How can I check which compiler scons uses? These are the compilers >>>>>>>>> on my system >>>>>>>>> >>>>>>>>> >>>>>>>>> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 >>>>>>>>> g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 >>>>>>>>> Linux 4.4.0-75-generic #96-Ubuntu SMP >>>>>>>>> >>>>>>>>> ------------------------------ >>>>>>>>> *From:* gem5-users <[email protected]> on behalf of >>>>>>>>> Jason Lowe-Power <[email protected]> >>>>>>>>> *Sent:* 23 May 2017 22:27:34 >>>>>>>>> >>>>>>>>> *To:* gem5 users mailing list >>>>>>>>> *Subject:* Re: [gem5-users] Link error building gem5.fast >>>>>>>>> >>>>>>>>> I just tried again and still cannot reproduce the error. What >>>>>>>>> compiler are you using? >>>>>>>>> >>>>>>>>> Jason >>>>>>>>> >>>>>>>>> On Tue, May 23, 2017 at 3:41 PM Moussa, Ayman < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hey >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I've encountered this exact problem with x86 and it only seems to >>>>>>>>>> be for gem5.fast (gem5.opt works fine). I still have problems with a >>>>>>>>>> clean >>>>>>>>>> build as Jason suggested so I reverted back to some random commit on >>>>>>>>>> the >>>>>>>>>> gem5 repository and it works but it's not what I was looking for >>>>>>>>>> though. >>>>>>>>>> Hope this gets fixed soon. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* gem5-users <[email protected]> on behalf of >>>>>>>>>> Alec Roelke <[email protected]> >>>>>>>>>> *Sent:* 23 May 2017 21:14:10 >>>>>>>>>> *To:* gem5 users mailing list >>>>>>>>>> *Subject:* [gem5-users] Link error building gem5.fast >>>>>>>>>> >>>>>>>>>> Hi Everyone, >>>>>>>>>> >>>>>>>>>> When I try to build gem5.fast using any ISA, I get a lot of >>>>>>>>>> multiple definition errors during the final linking stage. For >>>>>>>>>> example, >>>>>>>>>> with x86: >>>>>>>>>> >>>>>>>>>> [ LINK] -> X86/gem5.fast.unstripped >>>>>>>>>> build/X86/arch/x86/bios/lib.fo.partial: In function >>>>>>>>>> `Drainable::drainResume()': >>>>>>>>>> (.text+0x5b00): multiple definition of `Drainable::drainResume()' >>>>>>>>>> build/X86/dev/x86/lib.fo.partial:(.text+0x0): first defined here >>>>>>>>>> >>>>>>>>>> There are way too many of these to list them all, but they're all >>>>>>>>>> multiple definitions of symbols. Has anyone else encountered this? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Alec Roelke >>>>>>>>>> _______________________________________________ >>>>>>>>>> gem5-users mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> gem5-users mailing list >>>>>>>>> [email protected] >>>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>> >>>>>>>> >>>>> >>> >> IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ > gem5-users mailing list > [email protected] > http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
_______________________________________________ gem5-users mailing list [email protected] http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
