Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
"Mike Snitzer" writes: > On Sat, Jan 10, 2009 at 8:28 PM, Ingo Molnar wrote: >> >> Note, back when kdump was added to the kernel many moons ago i strongly >> supported it and helped out with the patches, etc. I still think it might >> have the potential to become big - but it needs a ton of tech and care to >> reach that level of convenience. >> >> 'kdump light' perhaps that dumps the most important data structures like >> registers of all CPUs, task struct and the symbol tables, the current task >> itself including the kernel stack plus the surrounding 4K of all pointers >> that are in current registers and that point into kernel memory - maybe >> straight to kerneloops.org [if the user agrees] - or something like that. > > I think 'kdump light' is a good idea. I'm all for infrastructure that > works better for more people. Having to deal with multi-gigabyte dump > files can be a chore. > > The mechanics of dumping your suggested 'light' amount of data vs. all > memory should be configurable (e.g. /sys/kernel/kexec_crash_light). Not in sys because this is a user space configuration issue. All of the dumping happens from user space. The kernel just provides access to the state of the previous kernel. > And this obviously doesn't change the potentially fragile nature of > kexec'ing to a crash kernel from an arbitrary context; or the fact > that drivers can easily be incompatible with cleanly shutting down and > restarting on kexec. Yep. Although the general answer in the kdump case is that if the kdump kernel is running you have gotten past all of the driver problems. > But honestly 99+% of my filesystem/storage enduced Linux crashes > kexec/kdump properly (with RHEL5, 2.6.22, 2.6.25, and 2.6.28); so all > the hard work of people like yourself and other kexec/kdump hackers > (upstream and at RedHat) really is paying off for real Linux users! Thanks. It is good to hear that the code works in the field. > Now if only I could fix line numbers when debugging crashes in x86_64 > modules with the crash utility! :) It's a userspace problem... All of the little usability things are userspace problems. I won't claim that it is trivial because it is a userspace problem, at the same time there is no reason to wait for any kernel features to merge etc. Someone just has to scratch an itch and go fix it. Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 09 Jan 2009 08:34:57 PST, "H. Peter Anvin" said: > A lot of noise is being made about the naming of the levels (and I > personally believe we should have a different annotation for "inline > unconditionally for correctness" and "inline unconditionally for > performance", as a documentation issue), but those are the four we get. I know we use __builtin_return_address() in several places, and several other places we introspect the stack and need to find the right frame entry. Are there any other places that need to be inlined for correctness? pgpeUP6xCt7Nj.pgp Description: PGP signature
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Sun, Jan 11, 2009 at 06:11:35PM +0800, Andreas Dilger wrote: > I'm sad that netconsole/netdump never made it big. It was fairly useful, > and extending the eth drivers to add the polling mode was trivial to do. > We were using that for a few years, but it got replaced by kdump and it > appears to be less usable IMHO. The netdump I'm familiar with had the misfeature that it didn't do packet retransmission, so when it was used on a customer network with any amount of traffic, packets would get dropped and the crash dump would utterly fail. I honestly can't remember which enterprise distro shipped it, but I can't say I was terribly impressed. :-( - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
> 'kdump light' perhaps that dumps the most important data structures like > registers of all CPUs, task struct and the symbol tables, the current task > itself including the kernel stack plus the surrounding 4K of all pointers > that are in current registers and that point into kernel memory - maybe > straight to kerneloops.org [if the user agrees] - or something like that. All you would need for that would be a new custom level in makedumpfile that dumps only this data (except that for live dumps it can be difficult to get the full register contents) No kernel changes needed. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Sun, Jan 11, 2009 at 5:11 AM, Andreas Dilger wrote: > On Jan 10, 2009 16:15 -0500, Theodore Ts'o wrote: >> In my experience, there are very few kernel versions and hardware for >> which kdump works. I've talked to the people who have to make kdump >> work, and every 12-18 months, with a new set of enterprise kernels >> comes out, they have to go and fix kdump so it works again for the set >> of hardware that they care about, and for the kernel version involved. > > I'm sad that netconsole/netdump never made it big. It was fairly useful, > and extending the eth drivers to add the polling mode was trivial to do. > We were using that for a few years, but it got replaced by kdump and it > appears to be less usable IMHO. Less usable in terms of ease of configuration and use? Or reliability? In practice netdump's non-interrupt-driven polling mode proved fairly reliable but it seems to me kdump provides more reliable dumping than netdump. Because you're performing the kdump from a reliable new dump kernel (provided the initial kexec transition to the dump kernel works properly). Mike -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Jan 10, 2009 16:15 -0500, Theodore Ts'o wrote: > In my experience, there are very few kernel versions and hardware for > which kdump works. I've talked to the people who have to make kdump > work, and every 12-18 months, with a new set of enterprise kernels > comes out, they have to go and fix kdump so it works again for the set > of hardware that they care about, and for the kernel version involved. I'm sad that netconsole/netdump never made it big. It was fairly useful, and extending the eth drivers to add the polling mode was trivial to do. We were using that for a few years, but it got replaced by kdump and it appears to be less usable IMHO. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Sat, Jan 10, 2009 at 8:28 PM, Ingo Molnar wrote: > > Note, back when kdump was added to the kernel many moons ago i strongly > supported it and helped out with the patches, etc. I still think it might > have the potential to become big - but it needs a ton of tech and care to > reach that level of convenience. > > 'kdump light' perhaps that dumps the most important data structures like > registers of all CPUs, task struct and the symbol tables, the current task > itself including the kernel stack plus the surrounding 4K of all pointers > that are in current registers and that point into kernel memory - maybe > straight to kerneloops.org [if the user agrees] - or something like that. I think 'kdump light' is a good idea. I'm all for infrastructure that works better for more people. Having to deal with multi-gigabyte dump files can be a chore. The mechanics of dumping your suggested 'light' amount of data vs. all memory should be configurable (e.g. /sys/kernel/kexec_crash_light). And this obviously doesn't change the potentially fragile nature of kexec'ing to a crash kernel from an arbitrary context; or the fact that drivers can easily be incompatible with cleanly shutting down and restarting on kexec. I worked with Eric Biederman testing in the early days of his kexec work and the e1000 driver was incompatible with kexec at that time (IFF it was built into the kernel, workaround was to use a module and unload it before kexec, *shudder*; I was using kexec in a custom bootloader for a storage appliance, not for kdump). But honestly 99+% of my filesystem/storage enduced Linux crashes kexec/kdump properly (with RHEL5, 2.6.22, 2.6.25, and 2.6.28); so all the hard work of people like yourself and other kexec/kdump hackers (upstream and at RedHat) really is paying off for real Linux users! The fairly recent kvm kdump compatibility work (2340b62f) is a perfect example of how hard things can be. But it is encouraging to see such commendable effort being put to making kdump workable for all. Now if only I could fix line numbers when debugging crashes in x86_64 modules with the crash utility! :) Regards, Mike -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
* Mike Snitzer wrote: > On Sat, Jan 10, 2009 at 10:34 AM, Ingo Molnar wrote: > > > > * Mike Snitzer wrote: > > > >> Yes, especially from someone who lacks the ability to properly > >> configure kdump. I'm fairly surprised others are giving you a free > >> pass when you keep asserting how broken kdump is with such hollow > >> criticism. I rely heavily on kdump and it works quite well (kvm > >> integration was lacking but has improved). > > > > hm, you say you rely heavily on kdump ... for what exactly, and how > > does it help the upstream Linux kernel? > > > > I see a single fix from you in the whole repository: > > > > ffc41cf: nbd: prevent sock_xmit from attempting to use a NULL socket > > > > ... and that single fix is a NULL pointer dereference that ought to have > > been quite debuggable from a plain oops alone. > > I've reported various bugs and helped with prototypes for fixes (e.g. > a0da84f3). But by all means belittle me... must be fun. I really did not want to belittle you - but in hindsight it really reads that way ... sorry about that and how insensitive it was from me! :( I just wanted to point out that if kdump is useful it must be _visible_. Ask commit logs to include "this was debugged via kdump" lines, etc. The upstream kernel must feel that it all matters. The upstream kernel really has to be ruthless about such things and must react to how things are not to how things are wished to be - one of the most critical things is that keeps Linux ticking is people like you who report and debug problems. Note, back when kdump was added to the kernel many moons ago i strongly supported it and helped out with the patches, etc. I still think it might have the potential to become big - but it needs a ton of tech and care to reach that level of convenience. 'kdump light' perhaps that dumps the most important data structures like registers of all CPUs, task struct and the symbol tables, the current task itself including the kernel stack plus the surrounding 4K of all pointers that are in current registers and that point into kernel memory - maybe straight to kerneloops.org [if the user agrees] - or something like that. Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
* Linus Torvalds wrote: > As far as I'm concerned, digital cameras have been more useful than > kernel dumps to kernel debugging. Yes, especially ones with VGA video capture. (I caught a "oops+triple-fault" crash via that trick once, which was not serial-console capture-able and which was just a single frame in the 25 fps video of the incident.) Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Sat, 10 Jan 2009, Andi Kleen wrote: > > I think that's mostly because kexec from arbitary context is a > somewhat unstable concept. I think that's the understatement of the year. We have tons of problems with standard suspend-to-ram, and that's when the suspend sequence has done its best to make everything quiescent. Expecting that we can reinitialize all the hardware at some random point when things are going haywire is "optimistic" at best. So of course it will work on some hardware and not others. I think we've been fairly successful at keeping a running system for _most_ of our bugs. Even when things go bad with X running, it's quite often possible to ssh in over the network (although it's often better if you were already connected) and see the dump. Not always, obviously. Many dumps really are painful. I'm hoping that kernel-mode-setting will at least give us the oops message _more_ of the time. As far as I'm concerned, digital cameras have been more useful than kernel dumps to kernel debugging. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
> In my experience, there are very few kernel versions and hardware for > which kdump works. I think that's mostly because kexec from arbitary context is a somewhat unstable concept. It requires all drivers to be able to reinit the hardware from an arbitary state, and that's just hard (it's kind of "make suspend/resume work everywhere" and then a little harder and we know how long that took) We also don't really have any tools to help making this easier to implement for driver developers. Like e.g. some self test that restarted drivers regularly to check this. But you often don't need kdump for crash dumps. In many cases the system is still alive after an oops or other problem and you can just do a live dump or even live crash session to look at data structures. I used to do this with gdb regularly, but now usually use crash because it has better tools. > they aren't enterprise users. Heck, until July of last year, > Systemtap wouldn't even ***compile*** out of the box on a > non-enterprise distribution like Ubuntu or Debian. At least on opensuse releases it tends to work for me. The biggest PITA used to be the elfutils dependency which seemed to come out of a all-world-is-redhat mindset at the developers, but that can be worked around now. Sometimes on has to patch it up when updating kernels because some interface by the runtime has changed again, but that shouldn't be a demanding task for kernel hackers really. At least I didn't find it particularly difficult. I wish they would merge the runtime into mainline though. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Sat, Jan 10, 2009 at 01:21:06PM -0500, Mike Snitzer wrote: > > In practice i rarely see bugfixes that were debugged via kdump. Normal > > oops based fixes outnumber kdump based fixes by a ratio of 1:100 or worse > > - and kdump is readily available these days - just nobody configures it. > > So you're telling me RedHat doesn't rely on kdump at enterprise > customer installations? I find that hard to believe. Few enterprise > customers allow defects to be debugged on-site, sometimes collecting a > crash dump is all you can hope for to make progress. I have to > believe you know this fairly well; if not with direct experience then > through your co-workers? Or am I living in Ingo's version of Linux > hell where kdump is actually useful? In my experience, there are very few kernel versions and hardware for which kdump works. I've talked to the people who have to make kdump work, and every 12-18 months, with a new set of enterprise kernels comes out, they have to go and fix kdump so it works again for the set of hardware that they care about, and for the kernel version involved. Part of the problem is one which has infected nearly every single RAS technology out there, from kdump to Systemtap, which is the people who architect and fund these RAS technologies delude themselves into thinking that they only have to worry about making it work for enterprise kernels and enterprise users, and to hell with everyone else --- specifically, kernel developers, which don't matter since they aren't enterprise users. Heck, until July of last year, Systemtap wouldn't even ***compile*** out of the box on a non-enterprise distribution like Ubuntu or Debian. And I still have yet to make kdump work on a Thinkpad, although I've tried. Since pretty much no one uses these RAS technologies except enterprise users, and no one bothers to make it easy for kernel developers, kernel developers have developed alternate mechanisms for debugging the Linux kernel --- and they don't involve using Systemtap or kdump, because in practice, it doesn't work for them at all, or it's too hard to make it work for them. And this becomes a vicious cycle; since no one is bothered to spend time making RAS technologies work for everyday use by kernel developers, bitrot inevitably sets in, and so the RAS developers get no help from other kernel developers, who are busy fixing their own problems via different means; and so the RAS developers hunker down, and spend even more time fixing the bitrot and complaining that no one helps them or takes them seriously, and the problem gets worse and worse and worse --- until now there are people who are busily developing alternatives to Systemtap, just because too many RAS architects and developers and had their priorities wrong, and forgot to focus on every day kernel developers instead of just enterprise users. It's very sad, and it means a lot of investment gets wasted, and work is getting duplicated as a result. Oh, well. - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Sat, Jan 10, 2009 at 10:34 AM, Ingo Molnar wrote: > > * Mike Snitzer wrote: > >> Yes, especially from someone who lacks the ability to properly configure >> kdump. I'm fairly surprised others are giving you a free pass when you >> keep asserting how broken kdump is with such hollow criticism. I rely >> heavily on kdump and it works quite well (kvm integration was lacking >> but has improved). > > hm, you say you rely heavily on kdump ... for what exactly, and how does > it help the upstream Linux kernel? > > I see a single fix from you in the whole repository: > > ffc41cf: nbd: prevent sock_xmit from attempting to use a NULL socket > > ... and that single fix is a NULL pointer dereference that ought to have > been quite debuggable from a plain oops alone. I've reported various bugs and helped with prototypes for fixes (e.g. a0da84f3). But by all means belittle me... must be fun. Baiting me into this fairly irrelevant tangent like you have really shows class. I've read hundreds of posts from you over the years and don't recall you be so overtly antagonistic for absolutely no reason. I was simply saying "kdump doesn't suck"; certainly not as bad as Nicholas would have us all believe. Yes, I'm not Ingo Molnar. I don't rewrite the core of Linux with ease but for the past 10 years I've developed solely on Linux to pay the bills. I started hacking Linux distributions and have progressed to kernel development where I primarily focus on storage and filesystems. As things relate to upstream Linux, I'm particularly good at backporting and cherry picking upstream advances to help stabilize enterprise solutions. I have to believe that you understand not all Linux kernel development happens upstream. > In practice i rarely see bugfixes that were debugged via kdump. Normal > oops based fixes outnumber kdump based fixes by a ratio of 1:100 or worse > - and kdump is readily available these days - just nobody configures it. So you're telling me RedHat doesn't rely on kdump at enterprise customer installations? I find that hard to believe. Few enterprise customers allow defects to be debugged on-site, sometimes collecting a crash dump is all you can hope for to make progress. I have to believe you know this fairly well; if not with direct experience then through your co-workers? Or am I living in Ingo's version of Linux hell where kdump is actually useful? > For example, in the whole kernel repo there's just 45 commits that mention > 'kdump' [excluding those commits that develop kdump itself]: > > $ git log --pretty=format:"%h: %s" --no-merges -i --grep="kdump" | >grep -viE 'kdump|kexec|dump|mem' | wc -l > 45 > > Contrast that to the 1954 commits that contain the string 'oops' or > 'crash': > > $ git log --pretty=format:"%h: %s" --no-merges -i -E --grep="oops|crash" | >wc -l > 5900 > > That's a ratio of 1:131. (and probably optimistic in favor of kdump.) > > Note, i dont have any negative feelings towards kdump - some people use it > and enterprise folks with their frozen, immutable kernels love it - it > just has not yet given me a reason to have particularly positive feelings > towards it in the upstream kernel space. Clearly you don't care about kdump; but please don't abuse your standing to turn this into a referendum on kdump's existence in the upstream kernel. Is upstream Linux somehow less pure by the existence of kdump? I'm left with a certain disappointment that the amazing Ingo Molnar took the time to respond to my post yet thought it best to immediately go negative _and_ off-topic on me with some vendetta against kdump. Your kdump vs oops statistics don't help defend Nicholas' "kump sucks" assertion either. So just what was your point (other than to flame me cause your cornflakes were nasty this morning)? Mike -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
* Mike Snitzer wrote: > Yes, especially from someone who lacks the ability to properly configure > kdump. I'm fairly surprised others are giving you a free pass when you > keep asserting how broken kdump is with such hollow criticism. I rely > heavily on kdump and it works quite well (kvm integration was lacking > but has improved). hm, you say you rely heavily on kdump ... for what exactly, and how does it help the upstream Linux kernel? I see a single fix from you in the whole repository: ffc41cf: nbd: prevent sock_xmit from attempting to use a NULL socket ... and that single fix is a NULL pointer dereference that ought to have been quite debuggable from a plain oops alone. In practice i rarely see bugfixes that were debugged via kdump. Normal oops based fixes outnumber kdump based fixes by a ratio of 1:100 or worse - and kdump is readily available these days - just nobody configures it. For example, in the whole kernel repo there's just 45 commits that mention 'kdump' [excluding those commits that develop kdump itself]: $ git log --pretty=format:"%h: %s" --no-merges -i --grep="kdump" | grep -viE 'kdump|kexec|dump|mem' | wc -l 45 Contrast that to the 1954 commits that contain the string 'oops' or 'crash': $ git log --pretty=format:"%h: %s" --no-merges -i -E --grep="oops|crash" | wc -l 5900 That's a ratio of 1:131. (and probably optimistic in favor of kdump.) Note, i dont have any negative feelings towards kdump - some people use it and enterprise folks with their frozen, immutable kernels love it - it just has not yet given me a reason to have particularly positive feelings towards it in the upstream kernel space. Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]
On Sat, Jan 10, 2009 at 1:44 AM, Nicholas Miell wrote: > On Fri, 2009-01-09 at 20:05 -0800, Linus Torvalds wrote: >> >> On Fri, 9 Jan 2009, Nicholas Miell wrote: >> > >> > It's only too big if you always keep it in memory, and I wasn't >> > suggesting that. >> >> Umm. We're talking kernel panics here. If it's not in memory, it doesn't >> exist as far as the kernel is concerned. >> >> If it doesn't exist, it cannot be reported. > > The idea was that the kernel would generate a crash dump and then after > the reboot, a post-processing tool would do something with it. (e.g. run > the dump through crash to get a stack trace using separate debug info or > ship the entire dump off to a collection server or something). > >> > And this is where we disagree. I believe that crash dumps should be the >> > norm and all the reasons you have against crash dumps in general are in >> > fact reasons against Linux's sub-par implementation of crash dumps in >> > specific. >> >> Good luck with that. Go ahead and try it. You'll find it wasn't so easy >> after all. >> >> > So, here I am, a non-enterprise end user with a non-stale kernel who'd >> > love to be able to give you a crash dump (or, more likely, a stack trace >> > created from that crash dump), but I can't because Linux crash dumps are >> > stuck in the enterprise ghetto. >> >> No, you're stuck because you apparently have your mind stuck on a >> crash-dump, and aren't willing to look at alternatives. >> >> You could use a network console. Trust me - if you can't set up a network >> console, you have no business mucking around with crash dumps. > > netconsole requires a second computer. Feel free to mail me one. :) > >> And if the crash is hard enough that you can't any output from that, >> again, a crash dump wouldn't exactly help, would it? >> >> > Hell, I'd be happy if I could get the the normal panic text written to >> > disk, but since the hard part is the actual writing to disk, there's no >> > reason not to do the full crash dump if you can. >> >> Umm. And why do you think the two have anything to do with each other? >> >> Only insane people want the kernel to write to disk when it has problems. >> Sane people try to write to something that doesn't potentially overwrite >> their data. Like the network. >> >> Which is there. Try it. Trust me, it's a _hell_ of a lot more likely to >> wotk than a crash dump. > > Well, yes, but that has everything to do with how terrible kdump is and > nothing to do with the idea of crash dumps in general. > > Anyway, we've strayed off topic long enough, I'm sure everyone in the Cc > list would be happy to stop getting an earful about the merits of crash > dumps. Yes, especially from someone who lacks the ability to properly configure kdump. I'm fairly surprised others are giving you a free pass when you keep asserting how broken kdump is with such hollow criticism. I rely heavily on kdump and it works quite well (kvm integration was lacking but has improved). Now that said, I too value crash dumps and will stray a bit off-topic (subject changed), I do have one significant debugability issue when using crash dumps: on x86_64 I'm unable to get line number information from symbols that reside in a module. I tried to get some insight on this from Dave Anderson (crash utility developer/maintainer) here: http://www.mail-archive.com/crash-util...@redhat.com/msg01101.html Dave was helpful but ultimately couldn't explain why I'm unable to get line numbers with my x86_64 kernel.org kernels (and he/redhat with RHEL4 kernels). I strongly respect those cc'd and have to believe someone can help me cut through this. I've updated scripts/package/mkspec so that the 'make rpm' target produces RedHat-style kernel rpms (models redhat's kernel.spec). This includes creating debuginfo rpms. I've attached a patch that works on 2.6.28; but again the resulting debuginfo doesn't provide line numbers for the kernel modules under crash!? If anyone has some insight on what I might be missing (as part of the kernel build) I'd _really_ appreciate it. I provided the mkspec patch just as a reference, if the RPM stuff is too opaque please just help me understand the bare mechanics of what is needed without mkspec (in the meantime I'll try the debuginfo generation patch Sam Ravnborg recently posted too). regards, Mike diff --git a/scripts/package/mkspec b/scripts/package/mkspec index 2500886..a71b4f8 100755 --- a/scripts/package/mkspec +++ b/scripts/package/mkspec @@ -24,33 +24,53 @@ fi PROVIDES="$PROVIDES kernel-$KERNELRELEASE" __KERNELRELEASE=`echo $KERNELRELEASE | sed -e "s/-//g"` +echo "%define _enable_debug_packages 1" +echo "" + +# for compatibility with RedHat-based kernel specfile-isms +echo "%define KVERREL $KERNELRELEASE" +echo "" + echo "Name: kernel" echo "Summary: The Linux Kernel" echo "Version: $__KERNELRELEASE" # we need to determine the NEXT version number so that uname and # rpm -q will agree -echo "Release: `. $srctree/scrip
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Sat, 10 Jan 2009 6:56:02 am H. Peter Anvin wrote: > This does bring up the idea of including a compiler with the kernel > sources again, doesn't it? Oh please don't give him any more ideas; first it was a terminal emulator that had delusions of grandeur, then something to help him manage that code, do we really want lcc too ? ;-) -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP signature.asc Description: This is a digitally signed message part.
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 2009-01-09 at 20:05 -0800, Linus Torvalds wrote: > > On Fri, 9 Jan 2009, Nicholas Miell wrote: > > > > It's only too big if you always keep it in memory, and I wasn't > > suggesting that. > > Umm. We're talking kernel panics here. If it's not in memory, it doesn't > exist as far as the kernel is concerned. > > If it doesn't exist, it cannot be reported. The idea was that the kernel would generate a crash dump and then after the reboot, a post-processing tool would do something with it. (e.g. run the dump through crash to get a stack trace using separate debug info or ship the entire dump off to a collection server or something). > > And this is where we disagree. I believe that crash dumps should be the > > norm and all the reasons you have against crash dumps in general are in > > fact reasons against Linux's sub-par implementation of crash dumps in > > specific. > > Good luck with that. Go ahead and try it. You'll find it wasn't so easy > after all. > > > So, here I am, a non-enterprise end user with a non-stale kernel who'd > > love to be able to give you a crash dump (or, more likely, a stack trace > > created from that crash dump), but I can't because Linux crash dumps are > > stuck in the enterprise ghetto. > > No, you're stuck because you apparently have your mind stuck on a > crash-dump, and aren't willing to look at alternatives. > > You could use a network console. Trust me - if you can't set up a network > console, you have no business mucking around with crash dumps. netconsole requires a second computer. Feel free to mail me one. :) > And if the crash is hard enough that you can't any output from that, > again, a crash dump wouldn't exactly help, would it? > > > Hell, I'd be happy if I could get the the normal panic text written to > > disk, but since the hard part is the actual writing to disk, there's no > > reason not to do the full crash dump if you can. > > Umm. And why do you think the two have anything to do with each other? > > Only insane people want the kernel to write to disk when it has problems. > Sane people try to write to something that doesn't potentially overwrite > their data. Like the network. > > Which is there. Try it. Trust me, it's a _hell_ of a lot more likely to > wotk than a crash dump. Well, yes, but that has everything to do with how terrible kdump is and nothing to do with the idea of crash dumps in general. Anyway, we've strayed off topic long enough, I'm sure everyone in the Cc list would be happy to stop getting an earful about the merits of crash dumps. -- Nicholas Miell -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Arjan van de Ven wrote: > > thinking about this.. making a "pastebin" like thing for oopses is > relatively trivial for me; all the building blocks I have already. > > The hard part is getting the vmlinux files in place. Right now I do > this manually for popular released kernels.. if the fedora/suse guys > would help to at least have the vmlinux for their released updates > easily available that would be a huge help without that it's going > to suck. > We could just pick them up automatically from the kernel.org mirrors with a little bit of scripting. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009 14:52:33 -0500 Theodore Tso wrote: > Kerneloops.org does this, so the code is mostly written; but it does > this in a blinded fashion, so it only makes sense for oops which are > very common and for which we don't need to ask the user, "so what were > you doing at the time". In cases where the user has already stepped > up and reported the oops on a mailing list, it would be nice if > kerneloops.org had a way of decoding the oops via some web page. > > Arjan, would something like this be doable, hopefully without too much > effort? thinking about this.. making a "pastebin" like thing for oopses is relatively trivial for me; all the building blocks I have already. The hard part is getting the vmlinux files in place. Right now I do this manually for popular released kernels.. if the fedora/suse guys would help to at least have the vmlinux for their released updates easily available that would be a huge help without that it's going to suck. -- Arjan van de VenIntel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Nicholas Miell wrote: > > It's only too big if you always keep it in memory, and I wasn't > suggesting that. Umm. We're talking kernel panics here. If it's not in memory, it doesn't exist as far as the kernel is concerned. If it doesn't exist, it cannot be reported. > My point was that you can get completely accurate stack traces in the > face of gcc's inlining, and that blaming gcc because you can't get good > stack traces because the kernel's debugging infrastructure isn't up to > snuff isn't exactly fair. No. I'm blaming inlining for making debugging harder. And that's ok - IF IT IS WORTH IT. It's not. Gcc inlining decisions suck. gcc inlines stuff that doesn't really help from being inlined, and doesn't inline stuff that _does_. What's so hard to accept in that? > And this is where we disagree. I believe that crash dumps should be the > norm and all the reasons you have against crash dumps in general are in > fact reasons against Linux's sub-par implementation of crash dumps in > specific. Good luck with that. Go ahead and try it. You'll find it wasn't so easy after all. > So, here I am, a non-enterprise end user with a non-stale kernel who'd > love to be able to give you a crash dump (or, more likely, a stack trace > created from that crash dump), but I can't because Linux crash dumps are > stuck in the enterprise ghetto. No, you're stuck because you apparently have your mind stuck on a crash-dump, and aren't willing to look at alternatives. You could use a network console. Trust me - if you can't set up a network console, you have no business mucking around with crash dumps. And if the crash is hard enough that you can't any output from that, again, a crash dump wouldn't exactly help, would it? > Hell, I'd be happy if I could get the the normal panic text written to > disk, but since the hard part is the actual writing to disk, there's no > reason not to do the full crash dump if you can. Umm. And why do you think the two have anything to do with each other? Only insane people want the kernel to write to disk when it has problems. Sane people try to write to something that doesn't potentially overwrite their data. Like the network. Which is there. Try it. Trust me, it's a _hell_ of a lot more likely to wotk than a crash dump. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 2009-01-09 at 16:05 -0800, Linus Torvalds wrote: > > On Fri, 9 Jan 2009, Nicholas Miell wrote: > > > In the general case is it does nothing at all to debugging (beyond the > > usual weird control flow you get from any optimized code) -- the > > compiler generates line number information for the inlined functions, > > the debugger interprets that information, and your backtrace is > > accurate. > > The thng is, we do not use line number information, and never will - > because it's too big. MUCH too big. > It's only too big if you always keep it in memory, and I wasn't suggesting that. My point was that you can get completely accurate stack traces in the face of gcc's inlining, and that blaming gcc because you can't get good stack traces because the kernel's debugging infrastructure isn't up to snuff isn't exactly fair. > We do end up saving function start information (although even that is > actually disabled if you're doing embedded development), so that we can at > least tell which function something happened in. > > > It is only in the specific case of the kernel's broken backtrace code > > that this becomes an issue. It's failure to function correctly is the > > direct result of a failure to keep up with modern compiler changes that > > everybody else in the toolchain has dealt with. > > Umm. You can say that. But the fact is, most others care a whole lot > _less_ about those "modern compiler changes". In user space, when you > debug something, you generally just stop optimizing. In the kernel, we've > tried to balance the "optimize vs debug info" thing. The bulk of the libraries that your userspace app links to are distro-built with full optimization and figuring out why an optimized build crashed does come up from time to time, so those changes still matter. > > I think that the answer to that is that the kernel should do its best to > > be as much like userspace apps as it can, because insisting on special > > treatment doesn't seem to be working. > > The problem with that is that the kernel _isn't_ a normal app. An it > _definitely_ isn't a normal app when it comes to debugging. > > You can hand-wave and talk about it all you want, but it's just not going > to happen. A kernel is special. We don't get dumps, and only crazy people > even ask for them. > > The fact that you seem to think that we should get them just shows that > you either don't udnerstand the problems, or you live in some sheltered > environment wher crash-dumps _could_ work, but also by definition those > environments aren't where they buy kernel developers anything. > > The thing is, a crash dump in a "enterprise environment" (and that is the > only kind where you can reasonably dump more than the minimal stuff we do > now) is totally useless - because such kernels are usually at least a year > old, often more. As such, debug information from enterprise users is > almost totally worthless - if we relied on it, we'd never get anything > done. > > And outside of those kinds of very rare niches, big kernel dumps simply > are not an option. Writing to disk when things go hay-wire in the kernel > is the _last_ thing you must ever do. People can't have dedicated dump > partitions or network dumps. > > That's the reality. I'm not making it up. We can give a simple trace, and > yes, we can try to do some off-line improvement on it (and kerneloops.org > to some degree does), but that's just about it. > And this is where we disagree. I believe that crash dumps should be the norm and all the reasons you have against crash dumps in general are in fact reasons against Linux's sub-par implementation of crash dumps in specific. I can semi-reliably panic my kernel, and it's fairly recent, too (2.6.27.9-159.fc10.x86_64). Naturally, I run X, so the result of a panic is a frozen screen, blinking keyboard lights, the occasional endless audio loop, and no useful information whatsoever. I looked into kdump, only to discover that it doesn't work (but it could, it's a simple matter of fixing the initrd's script to support LVM), but I've already found a workaround, and after fiddling with kdump, I just don't care anymore So, here I am, a non-enterprise end user with a non-stale kernel who'd love to be able to give you a crash dump (or, more likely, a stack trace created from that crash dump), but I can't because Linux crash dumps are stuck in the enterprise ghetto. You're right, the bulk of the people who do use kdump these days are enterprise people running ancient enterprise kernels, but that has more to do with kdump being an unusable bastard red-headed left-handed stepchild that you only use if you can't avoid it than it has to do with the crash dump concept being useless. Hell, I'd be happy if I could get the the normal panic text written to disk, but since the hard part is the actual writing to disk, there's no reason not to do the full crash dump if you can. -- Nicholas Miell -- To unsubscribe
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> I thought -Os actually disabled the basic-block reordering, doesn't it? Not in current gcc head no (just verified by stepping through) > > And I thought it did that exactly because it generates bigger code and > much worse I$ patterns (ie you have a lot of "conditional branch to other > place and then unconditional branch back" instead of "conditional branch > over the non-taken code". > > Also, I think we've had about as much good luck with guessing > "likely/unlikely" as we've had with "inline" ;) That's true. But if you look at the default heuristics that gcc has (gcc/predict.def in the gcc sources) like == NULL, < 0, branch guarding etc. I would expect a lot of them to DTRT for the kernel. Honza at some point even fixed goto to be unlikely after I complained :) > Sadly, apart from some of the "never happens" error cases, the kernel > doesn't tend to have lots of nice patterns. We have almost no loops (well, > there are loops all over, but most of them we hopefully just loop over > once or twice in any good situation), and few really predictable things. That actually makes us well suited to gcc, it has a relatively poor loop optimizer compared to other compilers ;-) > Or rather, they can easily be very predictable under one particular load, > and the totally the other way around under another .. Yes that is why we got good branch predictors in CPUs I guess. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Sat, 10 Jan 2009, Andi Kleen wrote: > > > What's the cost/benefit of that 4%? Does it actually improve performance? > > Especially if you then want to keep DWARF unwind information in memory in > > order to fix up some of the problems it causes? At that point, you lost > > dwarf unwind information has nothing to do with this, it doesn't tell > you anything about inlining or not inlining. It just gives you > finished frames after all of that has been done. > > Full line number information would help, but I don't think anyone > proposed to keep that in memory. Yeah, true. Although one of the reasons inlining actually ends up causing problems is because of the bigger stack frames. That leaves a lot of space for old stale function pointers to peek through. With denser stack frames, the stack dumps look better, even without an unwinder. > > Does it help I$ utilization (which can speed things up a lot more, and is > > probably the main reason -Os actually tends to perform better)? Likely > > not. Sure, shrinking code is good for I$, but on the other hand inlining > > can actually be bad for I$ density because if you inline a function that > > doesn't get called, you now fragmented your footprint a lot more. > > Not sure that is always true; the gcc basic block reordering > based on its standard branch prediction heuristics (e.g. < 0 or > == NULL unlikely or the unlikely macro) might well put it all out of line. I thought -Os actually disabled the basic-block reordering, doesn't it? And I thought it did that exactly because it generates bigger code and much worse I$ patterns (ie you have a lot of "conditional branch to other place and then unconditional branch back" instead of "conditional branch over the non-taken code". Also, I think we've had about as much good luck with guessing "likely/unlikely" as we've had with "inline" ;) Sadly, apart from some of the "never happens" error cases, the kernel doesn't tend to have lots of nice patterns. We have almost no loops (well, there are loops all over, but most of them we hopefully just loop over once or twice in any good situation), and few really predictable things. Or rather, they can easily be very predictable under one particular load, and the totally the other way around under another .. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> What's the cost/benefit of that 4%? Does it actually improve performance? > Especially if you then want to keep DWARF unwind information in memory in > order to fix up some of the problems it causes? At that point, you lost dwarf unwind information has nothing to do with this, it doesn't tell you anything about inlining or not inlining. It just gives you finished frames after all of that has been done. Full line number information would help, but I don't think anyone proposed to keep that in memory. > Does it help I$ utilization (which can speed things up a lot more, and is > probably the main reason -Os actually tends to perform better)? Likely > not. Sure, shrinking code is good for I$, but on the other hand inlining > can actually be bad for I$ density because if you inline a function that > doesn't get called, you now fragmented your footprint a lot more. Not sure that is always true; the gcc basic block reordering based on its standard branch prediction heuristics (e.g. < 0 or == NULL unlikely or the unlikely macro) might well put it all out of line. -Andi -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Nicholas Miell wrote: > > So take your complaint about gcc's decision to inline functions called > once. Actually, the "called once" really is a red herring. The big complaint is "too aggressively when not asked for". It just so happens that the called once logic is right now the main culprit. > Ignore for the moment the separate issue of stack growth and let's > talk about what it does to debugging, which was the bulk of your > complaint that I originally responded to. Actually, stack growth is the one that ends up being a correctness issue. But: > In the general case is it does nothing at all to debugging (beyond the > usual weird control flow you get from any optimized code) -- the > compiler generates line number information for the inlined functions, > the debugger interprets that information, and your backtrace is > accurate. The thng is, we do not use line number information, and never will - because it's too big. MUCH too big. We do end up saving function start information (although even that is actually disabled if you're doing embedded development), so that we can at least tell which function something happened in. > It is only in the specific case of the kernel's broken backtrace code > that this becomes an issue. It's failure to function correctly is the > direct result of a failure to keep up with modern compiler changes that > everybody else in the toolchain has dealt with. Umm. You can say that. But the fact is, most others care a whole lot _less_ about those "modern compiler changes". In user space, when you debug something, you generally just stop optimizing. In the kernel, we've tried to balance the "optimize vs debug info" thing. > I think that the answer to that is that the kernel should do its best to > be as much like userspace apps as it can, because insisting on special > treatment doesn't seem to be working. The problem with that is that the kernel _isn't_ a normal app. An it _definitely_ isn't a normal app when it comes to debugging. You can hand-wave and talk about it all you want, but it's just not going to happen. A kernel is special. We don't get dumps, and only crazy people even ask for them. The fact that you seem to think that we should get them just shows that you either don't udnerstand the problems, or you live in some sheltered environment wher crash-dumps _could_ work, but also by definition those environments aren't where they buy kernel developers anything. The thing is, a crash dump in a "enterprise environment" (and that is the only kind where you can reasonably dump more than the minimal stuff we do now) is totally useless - because such kernels are usually at least a year old, often more. As such, debug information from enterprise users is almost totally worthless - if we relied on it, we'd never get anything done. And outside of those kinds of very rare niches, big kernel dumps simply are not an option. Writing to disk when things go hay-wire in the kernel is the _last_ thing you must ever do. People can't have dedicated dump partitions or network dumps. That's the reality. I'm not making it up. We can give a simple trace, and yes, we can try to do some off-line improvement on it (and kerneloops.org to some degree does), but that's just about it. But debugging isn't even the only issue. It's just that debuggability is more important than a DUBIOUS improvement in code quality. See? Note the DUBIOUS. Let's take a very practical example on a number that has been floated around here: letting gcc do inlining decisions apparently can help for up to about 4% of code-size. Fair enough - I happen to believe that we could cut that down a bit by just doing things manually with a checker, but that's neither here nor there. What's the cost/benefit of that 4%? Does it actually improve performance? Especially if you then want to keep DWARF unwind information in memory in order to fix up some of the problems it causes? At that point, you lost all the memory you won, and then some. Does it help I$ utilization (which can speed things up a lot more, and is probably the main reason -Os actually tends to perform better)? Likely not. Sure, shrinking code is good for I$, but on the other hand inlining can actually be bad for I$ density because if you inline a function that doesn't get called, you now fragmented your footprint a lot more. So aggressively inlining has to be shown to be a real _win_. You try to say "well, do better debug info", but that turns inlining into a _loss_, so then the proper response is "don't inline". So when is inlining a win? It's a win when the thing you inline is clearly not bigger than the call site. Then it's totally unambiguous. It's also often a win if it's a unconditional call from a single site, and you only inline one such, so that you avoid all of the downsides (you may be able to _shrink_ stack usage, and you're hopefully making I$ accesses _denser_ rather t
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 11:44:10PM +0100, Andi Kleen wrote: > > Fetch a gigabyte's worth of data for the debuginfo RPM? > > The suse 11.0 kernel debuginfo is ~120M. How is this debuginfo generated? Someone have posted the following patch which I did not apply in lack of any real need. But maybe distroes have something similar already so it makes sense to apply it. Sam Subject: [PATCH] Kbuild: generate debug info in building This patch will generate kernel debuginfo in Kbuild when invoking "make debug_info". The separate debug files are in .debug under building tree. They can help the cases of requiring debug info for tracing/debug tools, especially cross-compilation. Moreover, it can simplify or standardize the packaging process for the distributions those will provide kernel-debuginfo. Signed-off-by: Wenji Huang --- Makefile | 14 ++ scripts/Makefile.modpost | 14 ++ 2 files changed, 28 insertions(+), 0 deletions(-) diff --git a/Makefile b/Makefile index 7f9ff9b..eed7510 100644 --- a/Makefile +++ b/Makefile @@ -814,6 +814,20 @@ define rule_vmlinux-modpost $(Q)echo 'cmd_$@ := $(cmd_vmlinux-modpost)' > $(dot-target).cmd endef +ifdef CONFIG_DEBUG_INFO +quiet_cmd_vmlinux_debug = GEN $<.debug + cmd_vmlinux_debug = mkdir -p .debug; \ + $(OBJCOPY) --only-keep-debug \ + $< .debug/$<.debug +targets += vmlinux.debug +endif + +debug_info: vmlinux FORCE +ifdef CONFIG_DEBUG_INFO + $(call if_changed,vmlinux_debug) + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost $@ +endif + # vmlinux image - including updated kernel symbols vmlinux: $(vmlinux-lds) $(vmlinux-init) $(vmlinux-main) vmlinux.o $(kallsyms.o) FORCE ifdef CONFIG_HEADERS_CHECK diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost index f4053dc..0df73b2 100644 --- a/scripts/Makefile.modpost +++ b/scripts/Makefile.modpost @@ -137,6 +137,20 @@ $(modules): %.ko :%.o %.mod.o FORCE targets += $(modules) +modules-debug := $(modules:.ko=.ko.debug) +ifdef CONFIG_DEBUG_INFO +quiet_cmd_debug_ko = GEN $@ + cmd_debug_ko = mkdir -p .debug/`dirname $...@`; \ +$(OBJCOPY) --only-keep-debug $< .debug/$@ +targets += $(modules-debug) +endif + +debug_info: $(modules-debug) FORCE + +$(modules-debug): $(modules) FORCE +ifdef CONFIG_DEBUG_INFO + $(call if_changed,debug_ko) +endif # Add FORCE to the prequisites of a target to force it to be always rebuilt. # --- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 2009-01-09 at 12:29 -0800, Linus Torvalds wrote: > > On Fri, 9 Jan 2009, Nicholas Miell wrote: > > > > Maybe the kernel's backtrace code should be fixed instead of blaming > > gcc. > > And maybe people who don't know what they are talking about shouldn't > speak? I may not know what I'm talking about, but you don't have to be rude about it, at least not until I've made a nuisance of myself, and a single mail surely isn't yet a nuisance. > You just loaded the whole f*cking debug info just to do that exact > analysis. Guess how big it is for the kernel? It's huge, I know. My understanding is that the DWARF size could be reduced rather dramatically if redundant DIEs were coalesced into a single entry. Even then it would still be too large to load into the kernel for runtime stack trace generation, but that's what the offline analysis of crash dumps is for. > Did you even read this discussion? No, I didn't read most of the discussion, I only started skimming it after I saw the gcc people talking negatively about the latest brouhaha. (And this thread being an offshoot of a large thread that is itself an offshoot of another large thread certainly didn't help me find the relevant parts of the discussion, either). So I'm sorry that I missed whatever comments you made on the subject (As an aside, I'm amazed that anything works at all when the kernel, compiler and runtime library people all seem to mutually loathe each other to the point of generally refusing to communicate at all.) > Did you see my comments about why > kernel backtrace debugging is different from regular user mode debugging? I think I found the mail in question[1], and after reading it, I found that it touches on one of the things I was thinking about while looking through the thread. The majority of code built by gcc was, is and always will be userspace code. The kernel was, is and always will be of minor importance to gcc. gcc will never be perfect for kernel use. So take your complaint about gcc's decision to inline functions called once. Ignore for the moment the separate issue of stack growth and let's talk about what it does to debugging, which was the bulk of your complaint that I originally responded to. In the general case is it does nothing at all to debugging (beyond the usual weird control flow you get from any optimized code) -- the compiler generates line number information for the inlined functions, the debugger interprets that information, and your backtrace is accurate. It is only in the specific case of the kernel's broken backtrace code that this becomes an issue. It's failure to function correctly is the direct result of a failure to keep up with modern compiler changes that everybody else in the toolchain has dealt with. So putting "helpfully" in sarcastic quotes or calling what gcc does a gcc problem is outright wrong. For most things, gcc's behavior does actually help, and it is a kernel's problem (by virtue of the kernel being different and strange), not gcc's. The question then becomes, how does the kernel deal with the fact that it is of minor importance to gcc and significantly different from the bulk of gcc's consumers to the point where those differences become serious problems? I think that the answer to that is that the kernel should do its best to be as much like userspace apps as it can, because insisting on special treatment doesn't seem to be working. In this specific case, that would mean make kernel debugging as much like userspace debugging as you can -- stop pretending that stack traces generated by the kernel at runtime are adequate, get kernel crash dumps enabled and working 100% of the time, and then use a debugger to examine the kernel's core dump and find your stack trace. As an added bonus, real crash dumps aren't limited just to backtraces, so you'd have even more information to work with to find the root cause of the failure. [1] Message ID: alpine.lfd.2.00.0901090947080.6...@localhost.localdomain -- Nicholas Miell -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 09 Jan 2009 14:35:29 -0800 "H. Peter Anvin" wrote: > Andi Kleen wrote: > >> Fetch a gigabyte's worth of data for the debuginfo RPM? > > > > The suse 11.0 kernel debuginfo is ~120M. > > Still, though, hardly worth doing client-side when it can be done > server-side for all the common distro kernels. For custom kernels, > not so, but there you should already have the debuginfo locally.h and if you have the debug info local, all you need is dmesg | scripts/markup_oops.pl vmlinux and it nicely decodes it for you -- Arjan van de VenIntel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Andi Kleen wrote: Fetch a gigabyte's worth of data for the debuginfo RPM? The suse 11.0 kernel debuginfo is ~120M. Still, though, hardly worth doing client-side when it can be done server-side for all the common distro kernels. For custom kernels, not so, but there you should already have the debuginfo locally. And yes, there are probably residual holes, but it's questionable if it matters. -hpa -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> Fetch a gigabyte's worth of data for the debuginfo RPM? The suse 11.0 kernel debuginfo is ~120M. -Andi -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009 14:52:33 -0500 Theodore Tso wrote: > On Fri, Jan 09, 2009 at 07:55:09PM +0100, Andi Kleen wrote: > > > But _users_ just get their oopses sent automatically. So it's not > > > about > > > > If they send it from distro kernels the automated oops sender could > > just fetch the debuginfo rpm and decode it down to a line. > > My old automatic user segfault uploader I did originally > > for the core pipe code did that too. > > Fetch a gigabyte's worth of data for the debuginfo RPM? Um, I think > most users would not be happy with that, especially if they are behind > a slow network. Including the necessary information so someone who > wants to investigate the oops, or having kerneloops.org pull apart the > oops makes more sense, I think, and is already done. > it is. > Something that would be **really** useful would be a web page where if > someone sends me an oops message from Fedora or Open SUSE kernel to > linux-kernel or linux-ext4, I could take the oops message, cut and > paste it into a web page, along with the kernel version information, > and the kernel developer could get back a decoded oops information > with line number information. > > Kerneloops.org does this, so the code is mostly written; but it does > this in a blinded fashion, so it only makes sense for oops which are > very common and for which we don't need to ask the user, "so what were > you doing at the time". In cases where the user has already stepped > up and reported the oops on a mailing list, it would be nice if > kerneloops.org had a way of decoding the oops via some web page. > > Arjan, would something like this be doable, hopefully without too much > effort? I suppose it could be done if the exact idea is there, and it would be nice if I'd get help from the distro kernel mainainers so that they'll send me the vmlinux'n ;) -- Arjan van de VenIntel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Theodore Tso wrote: > I'm beginning to think that for the kernel, we should just simply > remove CONFIG_OPTIMIZE_INLINING (so that inline means > "always_inline"), and -fno-inline-functions > -fno-inline-functions-called-one (so that gcc never inlines functions > behind our back) --- and then we create tools that count how many times > functions get used, and how big functions are, so that we can flag if some > function really should be marked inline when it isn't or vice versa. > > But given that this is a very hard thing for an automated program > todo, let's write some tools so we can easily put a human in the loop, > who can add or remove inline keywords where it makes sense, and let's > give up on gcc being able to "guess" correctly. > > For some things, like register allocation, I can accept that the > compiler will usually get these things right. But whether or not to > inline a function seems to be one of those things that humans (perhaps > with some tools assist) can still do a better job than compilers. Adding a function histogram in ftrace should be trivial. I can write one up if you want. It will only count the functions not inlined. -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
I'm beginning to think that for the kernel, we should just simply remove CONFIG_OPTIMIZE_INLINING (so that inline means "always_inline"), and -fno-inline-functions -fno-inline-functions-called-one (so that gcc never inlines functions behind our back) --- and then we create tools that count how many times functions get used, and how big functions are, so that we can flag if some function really should be marked inline when it isn't or vice versa. But given that this is a very hard thing for an automated program todo, let's write some tools so we can easily put a human in the loop, who can add or remove inline keywords where it makes sense, and let's give up on gcc being able to "guess" correctly. For some things, like register allocation, I can accept that the compiler will usually get these things right. But whether or not to inline a function seems to be one of those things that humans (perhaps with some tools assist) can still do a better job than compilers. - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> > And you do have to realize that Linux has been using gcc for a _loong_ > while. You can talk all you want about how "inline" is just a hint, but > the fact is, it didn't use to be. gcc people _made_ it so, and are having > a damn hard time admitting that it's causing problems. The kernel has used: # define inline inline __attribute__((always_inline)) For a looong time. So anyone in the kernel when they said "inline" actually said to gcc: if you have any possible way to do so inline this sucker. Now we have a config option that changes this so inline is only a hint. gcc does not pay enough attention to the hint, especially compared to the days where the hint was actually a command. Sam -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 9, 2009 at 9:26 PM, Linus Torvalds
wrote:
>
>
> On Fri, 9 Jan 2009, Richard Guenther wrote:
>>
>> This is a case where the improved IPA-CP (interprocedural constant
>> propagation) of GCC 4.4 may help. In general GCC cannot say how a call
>> argument may affect optimization if the function was inlined, so the
>> size estimates are done with just looking at the function body, not the
>> arguments (well, for GCC 4.4 this is not completely true, there is now
>> some "heuristics"). With IPA-CP GCC will clone the function for the
>> constant arguments, optimize it and eventually inline it if it is small
>> enough. At the moment this happens only if all callers call the
>> function with the same constant though (at least I think so).
>
> Ok, that's useless. The whole point is that everybody gives different -
> but still constant - arguments.
Btw, both GCC 4.3 and upcoming GCC 4.4 inline the bit-test. This is what I
used as a testcase (to avoid the single-call and single-constant cases):
#define BITS_PER_LONG 32
static inline int constant_test_bit(int nr, const volatile unsigned long *addr)
{
return ((1UL << (nr % BITS_PER_LONG)) &
(((unsigned long *)addr)[nr / BITS_PER_LONG])) != 0;
}
#define test_bit(nr, addr) \
(__builtin_constant_p((nr)) \
? constant_test_bit((nr), (addr)) \
: variable_test_bit((nr), (addr)))
int foo(unsigned long *addr)
{
return test_bit (5, addr);
}
int bar(unsigned long *addr)
{
return test_bit (6, addr);
}
at -Os even.
>> The above is definitely one case where using a macro or forced inlining is
>> a better idea than to trust a compiler to figure out that it can optimize the
>> function to a size suitable for inlining if called with a constant parameter.
>
> .. and forced inlining is what we default to. But that's when "let's try
> letting gcc optimize this" fails. And macros get really unreadable, really
> quickly.
As it happens to work with your simple case it may still apply for more
complex (thus appearantly big) cases.
>> > Maybe there was something else going on, and maybe Ingo's tests were off,
>> > but this is an example of gcc not inlining WHEN WE TOLD IT TO, and when
>> > the function was a single instruction.
>> >
>> > How can anybody possibly not consider that to be "stupid"?
>>
>> Because it's a hard problem, it's not stupid to fail here - you didn't tell
>> the
>> compiler the function optimizes!
>
> Well, actually we did. It's that "inline" there. That's how things used to
> work. It's like "no". It means "no". It doesn't mean "yes, I really want
> to s*ck your d*ck, but I'm just screaming no at the top of my lungs
> because I think I should do so".
>
> See?
See below.
> And you do have to realize that Linux has been using gcc for a _loong_
> while. You can talk all you want about how "inline" is just a hint, but
> the fact is, it didn't use to be. gcc people _made_ it so, and are having
> a damn hard time admitting that it's causing problems.
We made it so 10 years ago.
>> Experience tells us that people do not know better. Maybe the kernel is
>> an exception here
^^^
> Oh, I can well believe it.
>
> And I don't even think that kernel people get it right nearly enough, but
> since for the kernel it can even be a _correctness_ issue, at least if we
> get it wrong, everybody sees it.
>
> When _some_ compiler versions get it wrong, it's a disaster.
Of course. If you use always_inline then it's even a compiler bug.
>> But would you still want small functions be inlined even if they are not
>> marked inline?
>
> If you can really tell that they are that small, yes.
>
>> They do - just constant arguments are obviously not used for optimizing
>> before inlining. Otherwise you'd scream bloody murder at us for all the
>> increase in compile-time ;)
>
> A large portion of that has gone away now that everybody uses ccache. And
> if you only did it for functions that we _mark_ inline, it wouldn't even
> be true. Because those are the ones that presumably really should be
> inlined.
>
> So no, I don't believe you. You much too easily dismiss the fact that
> we've explicitly marked these functions for inlining, and then you say
> "but we were too stupid".
>
> If you cannot afford to do the real job, then trust the user. Don't guess.
We're guessing way better than the average programmer. But if you are
asking for a compiler option to disable guessing you can have it (you can
already use #define inline always_inline and -fno-inline to get it).
Richard.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Nicholas Miell wrote: > > Maybe the kernel's backtrace code should be fixed instead of blaming > gcc. And maybe people who don't know what they are talking about shouldn't speak? You just loaded the whole f*cking debug info just to do that exact analysis. Guess how big it is for the kernel? Did you even read this discussion? Did you see my comments about why kernel backtrace debugging is different from regular user mode debugging? Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Nicholas Miell wrote:
> On Fri, 2009-01-09 at 08:28 -0800, Linus Torvalds wrote:
> >
> > We get oopses that have a nice symbolic back-trace, and it reports an
> > error IN TOTALLY THE WRONG FUNCTION, because gcc "helpfully" inlined
> > things to the point that only an expert can realize "oh, the bug was
> > actually five hundred lines up, in that other function that was just
> > called once, so gcc inlined it even though it is huge".
> >
> > See? THIS is the problem with gcc heuristics. It's not about quality of
> > code, it's about RELIABILITY of code.
>
> [bt]$ cat backtrace.c
> #include
>
> static void called_once()
> {
> abort();
> }
>
> int main(int argc, char* argv[])
> {
> called_once();
> return 0;
> }
> [bt]$ gcc -Wall -O2 -g backtrace.c -o backtrace
> [bt]$ gdb --quiet backtrace
> (gdb) disassemble main
> Dump of assembler code for function main:
> 0x004004d0 : sub$0x8,%rsp
> 0x004004d4 : callq 0x4003b8
> End of assembler dump.
> (gdb) run
> Starting program: /home/nicholas/src/bitbucket/bt/backtrace
>
> Program received signal SIGABRT, Aborted.
> 0x003d9dc32f05 in raise (sig=) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> (gdb) bt
> #0 0x003d9dc32f05 in raise (sig=) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x003d9dc34a73 in abort () at abort.c:88
> #2 0x004004d9 in called_once () at backtrace.c:5
> #3 main (argc=3989, argv=0xf95) at backtrace.c:10
> (gdb)
>
>
> Maybe the kernel's backtrace code should be fixed instead of blaming
> gcc.
Try doing the same without compiling with -g.
I believe Andi has a patch to use the DWARF markings for backtrace (I'm
sure he'll correct me if I'm wrong ;-), but things like ftrace that use
kallsyms to get the names of functions and such does better when the
functions are not inlined. Not to mention that the function tracer does
not trace inlined functions so that's another downside of inlining.
-- Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Richard Guenther wrote: > > This is a case where the improved IPA-CP (interprocedural constant > propagation) of GCC 4.4 may help. In general GCC cannot say how a call > argument may affect optimization if the function was inlined, so the > size estimates are done with just looking at the function body, not the > arguments (well, for GCC 4.4 this is not completely true, there is now > some "heuristics"). With IPA-CP GCC will clone the function for the > constant arguments, optimize it and eventually inline it if it is small > enough. At the moment this happens only if all callers call the > function with the same constant though (at least I think so). Ok, that's useless. The whole point is that everybody gives different - but still constant - arguments. > The above is definitely one case where using a macro or forced inlining is > a better idea than to trust a compiler to figure out that it can optimize the > function to a size suitable for inlining if called with a constant parameter. .. and forced inlining is what we default to. But that's when "let's try letting gcc optimize this" fails. And macros get really unreadable, really quickly. > > Maybe there was something else going on, and maybe Ingo's tests were off, > > but this is an example of gcc not inlining WHEN WE TOLD IT TO, and when > > the function was a single instruction. > > > > How can anybody possibly not consider that to be "stupid"? > > Because it's a hard problem, it's not stupid to fail here - you didn't tell > the > compiler the function optimizes! Well, actually we did. It's that "inline" there. That's how things used to work. It's like "no". It means "no". It doesn't mean "yes, I really want to s*ck your d*ck, but I'm just screaming no at the top of my lungs because I think I should do so". See? And you do have to realize that Linux has been using gcc for a _loong_ while. You can talk all you want about how "inline" is just a hint, but the fact is, it didn't use to be. gcc people _made_ it so, and are having a damn hard time admitting that it's causing problems. > Experience tells us that people do not know better. Maybe the kernel is > an exception here Oh, I can well believe it. And I don't even think that kernel people get it right nearly enough, but since for the kernel it can even be a _correctness_ issue, at least if we get it wrong, everybody sees it. When _some_ compiler versions get it wrong, it's a disaster. > But would you still want small functions be inlined even if they are not > marked inline? If you can really tell that they are that small, yes. > They do - just constant arguments are obviously not used for optimizing > before inlining. Otherwise you'd scream bloody murder at us for all the > increase in compile-time ;) A large portion of that has gone away now that everybody uses ccache. And if you only did it for functions that we _mark_ inline, it wouldn't even be true. Because those are the ones that presumably really should be inlined. So no, I don't believe you. You much too easily dismiss the fact that we've explicitly marked these functions for inlining, and then you say "but we were too stupid". If you cannot afford to do the real job, then trust the user. Don't guess. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 2009-01-09 at 08:28 -0800, Linus Torvalds wrote:
>
> We get oopses that have a nice symbolic back-trace, and it reports an
> error IN TOTALLY THE WRONG FUNCTION, because gcc "helpfully" inlined
> things to the point that only an expert can realize "oh, the bug was
> actually five hundred lines up, in that other function that was just
> called once, so gcc inlined it even though it is huge".
>
> See? THIS is the problem with gcc heuristics. It's not about quality of
> code, it's about RELIABILITY of code.
[bt]$ cat backtrace.c
#include
static void called_once()
{
abort();
}
int main(int argc, char* argv[])
{
called_once();
return 0;
}
[bt]$ gcc -Wall -O2 -g backtrace.c -o backtrace
[bt]$ gdb --quiet backtrace
(gdb) disassemble main
Dump of assembler code for function main:
0x004004d0 :sub$0x8,%rsp
0x004004d4 : callq 0x4003b8
End of assembler dump.
(gdb) run
Starting program: /home/nicholas/src/bitbucket/bt/backtrace
Program received signal SIGABRT, Aborted.
0x003d9dc32f05 in raise (sig=) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x003d9dc32f05 in raise (sig=) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x003d9dc34a73 in abort () at abort.c:88
#2 0x004004d9 in called_once () at backtrace.c:5
#3 main (argc=3989, argv=0xf95) at backtrace.c:10
(gdb)
Maybe the kernel's backtrace code should be fixed instead of blaming
gcc.
--
Nicholas Miell
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 9, 2009 at 8:44 PM, Linus Torvalds
wrote:
>
>
> On Fri, 9 Jan 2009, Richard Guenther wrote:
>>
>> -fno-inline-functions-called-once disables the heuristic that always
>> inlines (static!) functions that are called once. Other heuristics
>> still apply, like inlining the static function if it is small.
>> Everything else would be totally stupid - which seems to be the "default
>> mode" you think GCC developers are in.
>
> Well, I don't know about you, but the "don't inline a single instruction"
> sounds a bit stupid to me. And yes, that's exactly what triggered this
> whole thing.
>
> We have two examples of gcc doing that, one of which was even a modern
> version of gcc, where we had sone absolutely _everything_ on a source
> level to make sure that gcc could not possibly screw up. Yet it did:
>
>static inline int constant_test_bit(int nr, const volatile unsigned
> long *addr)
>{
>return ((1UL << (nr % BITS_PER_LONG)) &
>(((unsigned long *)addr)[nr / BITS_PER_LONG])) != 0;
>}
>
>#define test_bit(nr, addr) \
>(__builtin_constant_p((nr)) \
> ? constant_test_bit((nr), (addr)) \
> : variable_test_bit((nr), (addr)))
>
> in this case, Ingo said that changing that _single_ inline to forcing
> inlining made a difference.
>
> That's CRAZY. The thing isn't even called unless "nr" is constant, so
> absolutely _everything_ optimizes away, and that whole function was
> designed to give us a single instruction:
>
>testl $constant,constant_offset(addr)
>
> and nothing else.
This is a case where the improved IPA-CP (interprocedural constant propagation)
of GCC 4.4 may help. In general GCC cannot say how a call argument may
affect optimization if the function was inlined, so the size estimates are done
with just looking at the function body, not the arguments (well, for
GCC 4.4 this
is not completely true, there is now some "heuristics"). With IPA-CP GCC will
clone the function for the constant arguments, optimize it and eventually inline
it if it is small enough. At the moment this happens only if all
callers call the
function with the same constant though (at least I think so).
The above is definitely one case where using a macro or forced inlining is
a better idea than to trust a compiler to figure out that it can optimize the
function to a size suitable for inlining if called with a constant parameter.
> Maybe there was something else going on, and maybe Ingo's tests were off,
> but this is an example of gcc not inlining WHEN WE TOLD IT TO, and when
> the function was a single instruction.
>
> How can anybody possibly not consider that to be "stupid"?
Because it's a hard problem, it's not stupid to fail here - you didn't tell the
compiler the function optimizes!
> The other case (with a single "cmpxchg" inline asm instruction) was at
> least _slightly_ more understandable, in that (a) Ingo claims modern gcc's
> did inline it and (b) the original function actually has a "switch()"
> statement that depends on the argument that is constant, so a stupid
> inliner might believe that it's a big function. But again, we _told_ the
> compiler to inline the damn thing, because we knew better. But gcc didn't.
Experience tells us that people do not know better. Maybe the kernel is
an exception here, but generally trusting "inline" up to an absolute is
a bad idea (we stretch heuristics if you specify "inline" though). We can,
for the kernels purpose and maybe other clever developers, invent a
-fobey-inline mode, only inline functions marked inline and inline
all of them (if possible - which would be the key difference to always_inline).
But would you still want small functions be inlined even if they are not
marked inline?
> The other part that is crazy is when gcc inlines large functions that
> aren't even called most of the time (the "ioctl()" switch statements tend
> to be a great example of this - gcc inlines ten or twenty functions, and
> we can guarantee that only one of them is ever called). Yes, maybe it
> makes the code smaller, but it makes the code also undebuggable and often
> BUGGY, because we now have the stack frame of all ten-to-twenty functions
> to contend with.
Use -fno-inline-functions-called-once then. But if you ask for -Os you get -Os.
Also recent GCC estimate and limit stack growth - which you can tune
reliably with --param large-stack-frame-growth and --param large-stack-frame.
> And notice how "static" has absolutely _zero_ meaning for the above
> example. Yes, the thing is called just from one place - that's how
> something like that very much works. It's a special case. It's not _worth_
> inlining, especially if it causes bugs. So "called once" or "static" is
> actually totally irrelevant.
Static makes inlining a single call cheap, because the out-of-line body
can be reclaimed. If you do not like that, turn i
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Linus Torvalds wrote: Because then they are _our_ mistakes, not some random compiler version that throws a dice! This does bring up the idea of including a compiler with the kernel sources again, doesn't it? -hpa (ducks & runs) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 07:55:09PM +0100, Andi Kleen wrote: > > But _users_ just get their oopses sent automatically. So it's not about > > If they send it from distro kernels the automated oops sender could > just fetch the debuginfo rpm and decode it down to a line. > My old automatic user segfault uploader I did originally > for the core pipe code did that too. Fetch a gigabyte's worth of data for the debuginfo RPM? Um, I think most users would not be happy with that, especially if they are behind a slow network. Including the necessary information so someone who wants to investigate the oops, or having kerneloops.org pull apart the oops makes more sense, I think, and is already done. Something that would be **really** useful would be a web page where if someone sends me an oops message from Fedora or Open SUSE kernel to linux-kernel or linux-ext4, I could take the oops message, cut and paste it into a web page, along with the kernel version information, and the kernel developer could get back a decoded oops information with line number information. Kerneloops.org does this, so the code is mostly written; but it does this in a blinded fashion, so it only makes sense for oops which are very common and for which we don't need to ask the user, "so what were you doing at the time". In cases where the user has already stepped up and reported the oops on a mailing list, it would be nice if kerneloops.org had a way of decoding the oops via some web page. Arjan, would something like this be doable, hopefully without too much effort? - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Matthew Wilcox wrote: > > Now, I'm not going to argue the directIO code is a shining example of > how we want things to look, but we don't really want ten arguments > being marshalled into a function call; we want gcc to inline the > direct_io_worker() and do its best to optimise the whole thing. Well, except we quite probably would be happier with gcc not doing that, than with gcc doing that too often. There are exceptions. If the caller is really small (ie a pure wrapper that perhaps just does some locking around the call), then sure, inlining a large function that only gets called from one place does make sense. But if both the caller and the callee is large, like in your example, then no. DON'T INLINE IT. Unless we _tell_ you, of course, which we probably shouldn't do. Why? Becuase debugging is more important. And deciding to inline that, you probably decided to inline something _else_ too. And now you've quite possibly blown your stackspace. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Richard Guenther wrote:
>
> -fno-inline-functions-called-once disables the heuristic that always
> inlines (static!) functions that are called once. Other heuristics
> still apply, like inlining the static function if it is small.
> Everything else would be totally stupid - which seems to be the "default
> mode" you think GCC developers are in.
Well, I don't know about you, but the "don't inline a single instruction"
sounds a bit stupid to me. And yes, that's exactly what triggered this
whole thing.
We have two examples of gcc doing that, one of which was even a modern
version of gcc, where we had sone absolutely _everything_ on a source
level to make sure that gcc could not possibly screw up. Yet it did:
static inline int constant_test_bit(int nr, const volatile unsigned
long *addr)
{
return ((1UL << (nr % BITS_PER_LONG)) &
(((unsigned long *)addr)[nr / BITS_PER_LONG])) != 0;
}
#define test_bit(nr, addr) \
(__builtin_constant_p((nr)) \
? constant_test_bit((nr), (addr)) \
: variable_test_bit((nr), (addr)))
in this case, Ingo said that changing that _single_ inline to forcing
inlining made a difference.
That's CRAZY. The thing isn't even called unless "nr" is constant, so
absolutely _everything_ optimizes away, and that whole function was
designed to give us a single instruction:
testl $constant,constant_offset(addr)
and nothing else.
Maybe there was something else going on, and maybe Ingo's tests were off,
but this is an example of gcc not inlining WHEN WE TOLD IT TO, and when
the function was a single instruction.
How can anybody possibly not consider that to be "stupid"?
The other case (with a single "cmpxchg" inline asm instruction) was at
least _slightly_ more understandable, in that (a) Ingo claims modern gcc's
did inline it and (b) the original function actually has a "switch()"
statement that depends on the argument that is constant, so a stupid
inliner might believe that it's a big function. But again, we _told_ the
compiler to inline the damn thing, because we knew better. But gcc didn't.
The other part that is crazy is when gcc inlines large functions that
aren't even called most of the time (the "ioctl()" switch statements tend
to be a great example of this - gcc inlines ten or twenty functions, and
we can guarantee that only one of them is ever called). Yes, maybe it
makes the code smaller, but it makes the code also undebuggable and often
BUGGY, because we now have the stack frame of all ten-to-twenty functions
to contend with.
And notice how "static" has absolutely _zero_ meaning for the above
example. Yes, the thing is called just from one place - that's how
something like that very much works. It's a special case. It's not _worth_
inlining, especially if it causes bugs. So "called once" or "static" is
actually totally irrelevant.
And no, they are not marked "inline" (although they are clearly also not
marked "uninline", until we figure out that gcc is causing system crashes,
and we add the thing).
If these two small problems were fixed, gcc inlining would work much
better. But the first one, in particular, means that the "do I inline or
not" decision would have to happen after expanding and simplifying
constants. And then, if the end result is big, the inlining gets aborted.
Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 9, 2009 at 8:40 PM, Andi Kleen wrote: > On Fri, Jan 09, 2009 at 08:10:20PM +0100, Richard Guenther wrote: >> On Fri, Jan 9, 2009 at 8:21 PM, Andi Kleen wrote: >> >> GCC 4.3 should be good in compiling the >> >> kernel with default -Os settings. >> > >> > It's unfortunately not. It doesn't inline a lot of simple asm() inlines >> > for example. >> >> Reading Ingos posting with the actual numbers states the opposite. > > Hugh had some numbers upto 4.4.0 20090102 in > > http://thread.gmane.org/gmane.linux.kernel/775254/focus=777231 > > which demonstrated the problem. How about GCC bugzillas with testcases that we can fix and enter into the testsuite to make sure future GCC versions won't regress? Richard. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 08:35:06PM +0100, Andi Kleen wrote:
> - Also inline everything static that is only called once
> [on the theory that this shrinks code size which is true
> according to my measurements]
>
> -fno-inline-functions-called once disables this new rule.
> It's very well and clearly defined.
It's also not necessarily what we want. For example, in fs/direct-io.c,
we have:
static ssize_t
direct_io_worker(int rw, struct kiocb *iocb, struct inode *inode,
const struct iovec *iov, loff_t offset, unsigned long nr_segs,
unsigned blkbits, get_block_t get_block, dio_iodone_t end_io,
struct dio *dio)
{
[150 lines]
}
ssize_t
__blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
struct block_device *bdev, const struct iovec *iov, loff_t offset,
unsigned long nr_segs, get_block_t get_block, dio_iodone_t end_io,
int dio_lock_type)
{
[100 lines]
retval = direct_io_worker(rw, iocb, inode, iov, offset,
nr_segs, blkbits, get_block, end_io, dio);
[10 lines]
}
Now, I'm not going to argue the directIO code is a shining example of
how we want things to look, but we don't really want ten arguments
being marshalled into a function call; we want gcc to inline the
direct_io_worker() and do its best to optimise the whole thing.
--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 08:10:20PM +0100, Richard Guenther wrote: > On Fri, Jan 9, 2009 at 8:21 PM, Andi Kleen wrote: > >> GCC 4.3 should be good in compiling the > >> kernel with default -Os settings. > > > > It's unfortunately not. It doesn't inline a lot of simple asm() inlines > > for example. > > Reading Ingos posting with the actual numbers states the opposite. Hugh had some numbers upto 4.4.0 20090102 in http://thread.gmane.org/gmane.linux.kernel/775254/focus=777231 which demonstrated the problem. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Richard Guenther wrote: On Fri, Jan 9, 2009 at 8:21 PM, Andi Kleen wrote: GCC 4.3 should be good in compiling the kernel with default -Os settings. It's unfortunately not. It doesn't inline a lot of simple asm() inlines for example. Reading Ingos posting with the actual numbers states the opposite. Well, Andi's patch forcing inlining of the bitops chops quite a bit of size off the kernel, so there is clearly room for improvement. From my post yesterday: : voreg 64 ; size o.*/vmlinux textdata bss dec hex filename 57590217 24940519 15560504 98091240 5d8c0e8 o.andi/vmlinux 59421552 24912223 15560504 99894279 5f44407 o.noopt/vmlinux 57700527 24950719 15560504 98211750 5da97a6 o.opty/vmlinux 110 KB of code size reduction by force-inlining the small bitops. -hpa -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> What happens when you say -fno-inline-functions-called-once? Does it > disable inlining for those functions IN GENERAL, or just for the LARGE It does disable it in general, unless they're marked inline explicitely : The traditional gcc 2.x rules were: I Only inline what is marked inline (but it can decide to not inline) II Also inline others heuristically only with -O3 / -finline-functions [which we don't set in the kernel] Then at some point this additional rule was added: - Also inline everything static that is only called once [on the theory that this shrinks code size which is true according to my measurements] -fno-inline-functions-called once disables this new rule. It's very well and clearly defined. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Richard Guenther wrote: But it's also not inconceivable that gcc adds a -fkernel-inlining or similar that changes the parameters if we ask nicely. I suppose actually such a parameter would be useful for far more programs than the kernel. I think that the kernel is a perfect target to optimize default -Os behavior for (whereas template heavy C++ programs are a target to optimize -O2 for). And I think we did a good job in listening to kernel developers if once in time they tried to talk to us - GCC 4.3 should be good in compiling the kernel with default -Os settings. We, unfortunately, cannot retroactively fix old versions that kernel developers happen to like and still use. Unfortunately I think there have been a lot of "we can't talk to them" on both sides of the kernel-gcc interface, which is incredibly unfortunate. I personally try to at least observe gcc development, including monitoring #gcc and knowing enough about gcc internals to write a (crappy) port, but I can hardly call myself a gcc expert. Still, I am willing to spend some significant time interfacing with anyone in the gcc community willing to spend the effort. I think we can do good stuff. -hpa -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 9, 2009 at 8:21 PM, Andi Kleen wrote: >> GCC 4.3 should be good in compiling the >> kernel with default -Os settings. > > It's unfortunately not. It doesn't inline a lot of simple asm() inlines > for example. Reading Ingos posting with the actual numbers states the opposite. Richard. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 9, 2009 at 6:54 PM, Linus Torvalds wrote: > > > On Fri, 9 Jan 2009, Matthew Wilcox wrote: >> >> That seems like valuable feedback to give to the GCC developers. > > Well, one thing we should remember is that the kernel really _is_ special. (sorry for not threading properly here) Linus writes: "Thirdly, you're just replacing _one_ random gcc choice with _another_ random one. What happens when you say -fno-inline-functions-called-once? Does it disable inlining for those functions IN GENERAL, or just for the LARGE ones? See?" -fno-inline-functions-called-once disables the heuristic that always inlines (static!) functions that are called once. Other heuristics still apply, like inlining the static function if it is small. Everything else would be totally stupid - which seems to be the "default mode" you think GCC developers are in. Richard. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> GCC 4.3 should be good in compiling the > kernel with default -Os settings. It's unfortunately not. It doesn't inline a lot of simple asm() inlines for example. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Andi Kleen wrote: > > Ok you're saying we should pay the 4.1% by default for this? The thing is, YOU ARE MAKING THAT NUMBER UP! First off, the size increase only matters if it actually increases the cache footprint. And it may, but.. Secondly, my whole point here has been that we should not rely on gcc doing things behind our back, because gcc will generally do the wrong thing. If we decided to be more active about this, we could just choose to find the places that matter (in hot code) and fix _those_. Thirdly, you're just replacing _one_ random gcc choice with _another_ random one. What happens when you say -fno-inline-functions-called-once? Does it disable inlining for those functions IN GENERAL, or just for the LARGE ones? See? Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 9, 2009 at 7:19 PM, Andi Kleen wrote: >> So we do have special issues. And exactly _because_ we have special issues >> we should also expect that some compiler defaults simply won't ever really >> be appropriate for us. > > I agree that the kernel needs quite different inlining heuristics > than let's say a template heavy C++ program. I guess that is > also where our trouble comes from -- gcc is more tuned for the > later. Perhaps because the C++ programmers are better at working > with the gcc developers? > > But it's also not inconceivable that gcc adds a -fkernel-inlining or > similar that changes the parameters if we ask nicely. I suppose > actually such a parameter would be useful for far more programs > than the kernel. I think that the kernel is a perfect target to optimize default -Os behavior for (whereas template heavy C++ programs are a target to optimize -O2 for). And I think we did a good job in listening to kernel developers if once in time they tried to talk to us - GCC 4.3 should be good in compiling the kernel with default -Os settings. We, unfortunately, cannot retroactively fix old versions that kernel developers happen to like and still use. Richard. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> But _users_ just get their oopses sent automatically. So it's not about If they send it from distro kernels the automated oops sender could just fetch the debuginfo rpm and decode it down to a line. My old automatic user segfault uploader I did originally for the core pipe code did that too. > "debugging kernels", it's about _normal_ kernels. They are the ones that > need to be debuggable, and the ones that care most about things like the > symbolic EIP being as helpful as possible. Ok you're saying we should pay the 4.1% by default for this? If you want that you can apply the appended patch. Not sure if it's really a good idea though. 4.1% is a lot. -Andi --- Disable inlining of functions called once by default This makes oopses easier to read because it's clearer in which function the problem is. Disadvantage: costs ~4.1% of text size (measured with allyesconfig on gcc 4.1 on x86-64) Signed-off-by: Andi Kleen --- Makefile |4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) Index: linux-2.6.28-git12/Makefile === --- linux-2.6.28-git12.orig/Makefile2009-01-09 07:06:44.0 +0100 +++ linux-2.6.28-git12/Makefile 2009-01-09 07:16:21.0 +0100 @@ -546,10 +546,8 @@ KBUILD_CFLAGS += -pg endif -# We trigger additional mismatches with less inlining -ifdef CONFIG_DEBUG_SECTION_MISMATCH +# Disable too aggressive inlining because it makes oopses harder to read KBUILD_CFLAGS += $(call cc-option, -fno-inline-functions-called-once) -endif # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include) -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Linus Torvalds wrote: So we do have special issues. And exactly _because_ we have special issues we should also expect that some compiler defaults simply won't ever really be appropriate for us. That is, of course, true. However, the Linux kernel (and quite a few other kernels) is a very important customer of gcc, and adding sustainable modes for the kernel that we can rely on is probably something we can work with them on. I think the relationship between the gcc and Linux kernel people is unnecessarily infected, and cultivating a more constructive relationship would be good. I suspect a big part of the reason for the oddities is that the timeline for the kernel community from making a request into gcc until we can actually rely on it is *very* long, and so we end up having to working things around no matter what (usually with copious invective), and the gcc people have other customers with shorter lead times which therefore drive their development more. -hpa -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> So we do have special issues. And exactly _because_ we have special issues > we should also expect that some compiler defaults simply won't ever really > be appropriate for us. I agree that the kernel needs quite different inlining heuristics than let's say a template heavy C++ program. I guess that is also where our trouble comes from -- gcc is more tuned for the later. Perhaps because the C++ programmers are better at working with the gcc developers? But it's also not inconceivable that gcc adds a -fkernel-inlining or similar that changes the parameters if we ask nicely. I suppose actually such a parameter would be useful for far more programs than the kernel. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Andi Kleen wrote: > > Universal noinline would also be a bad idea because of its > costs (4.1% text size increase). Perhaps should make it > a CONFIG option for debugging though. That's _totally_ the wrong way. If you can reproduce an issue on your machine, you generally don't care about inline, because you can see the stack, do the whole "gdb vmlinux" thing, and you generally have tools to help you decode things. Including just recompiling the kernel with an added noinline. But _users_ just get their oopses sent automatically. So it's not about "debugging kernels", it's about _normal_ kernels. They are the ones that need to be debuggable, and the ones that care most about things like the symbolic EIP being as helpful as possible. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Matthew Wilcox wrote: > > That seems like valuable feedback to give to the GCC developers. Well, one thing we should remember is that the kernel really _is_ special. The kernel not only does things no other program tends to do (inline asms are unusual in the first place - many of them are literally due to system issues like atomic accesses and interrupts that simply aren't an issue in user space, or that need so much abstraction that they aren't inlinable anyway). But the kernel also has totally different requirements in other ways. When was the last time you did user space programming and needed to get a backtrace from a user with register info because you simply don't have the hardware that he has? IOW, debugging in user space tends to be much more about trying to reproduce the bug - in a way that we often cannot in the kernel. User space in general is much more reproducible, since it's seldom as hardware- or timing-dependent (threading does change the latter, but usually user space threading is not _nearly_ as aggressive as the kernel has to be). So the thing is, even if gcc was "perfect", it would likely be perfect for a different audience than the kernel. Do you think _any_ user space programmer worries about the stack space being a few hundred bytes larger because the compiler inlined two functions, and caused stack usage to be sum of them instead of just the maximum of the two? So we do have special issues. And exactly _because_ we have special issues we should also expect that some compiler defaults simply won't ever really be appropriate for us. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> I think that's the point. gcc will not get it right. I don't think that's necessary an universal truth. It can be probably fixed. > So we need to do it ourselves in the kernel sources. > We may not like it, but it's the only way to guarantee reproducable > reliable inline / noinline decisions. For most things we don't really need it to be reproducable, the main exception are the inlines in headers. Universal noinline would also be a bad idea because of its costs (4.1% text size increase). Perhaps should make it a CONFIG option for debugging though. -Andi > > /D > > -- > Dirk Hohndel > Intel Open Source Technology Center > -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009 18:47:19 +0100 Andi Kleen wrote: > On Fri, Jan 09, 2009 at 10:28:01AM -0700, Matthew Wilcox wrote: > > On Fri, Jan 09, 2009 at 06:20:11PM +0100, Andi Kleen wrote: > > > Also cc Honza in case he has comments (you might want > > > to review more of the thread in the archives) > > > > I think this particular bug is already known and discussed: > > I thought so initially too, but: > > > > > http://gcc.gnu.org/ml/gcc/2008-12/msg00365.html > > > > and it hints at being fixed with gcc 4.4. Does anyone want to test > > that? > > Hugh already tested with 4.4 and it didn't work well. At least > a lot of the low level asm inlines were not inlined. > So it looks like it's still mistuned for the kernel. I think that's the point. gcc will not get it right. So we need to do it ourselves in the kernel sources. We may not like it, but it's the only way to guarantee reproducable reliable inline / noinline decisions. /D -- Dirk Hohndel Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 06:47:19PM +0100, Andi Kleen wrote: > On Fri, Jan 09, 2009 at 10:28:01AM -0700, Matthew Wilcox wrote: > > On Fri, Jan 09, 2009 at 06:20:11PM +0100, Andi Kleen wrote: > > > Also cc Honza in case he has comments (you might want > > > to review more of the thread in the archives) > > > > I think this particular bug is already known and discussed: > > I thought so initially too, but: > > > > > http://gcc.gnu.org/ml/gcc/2008-12/msg00365.html > > > > and it hints at being fixed with gcc 4.4. Does anyone want to test > > that? > > Hugh already tested with 4.4 and it didn't work well. At least > a lot of the low level asm inlines were not inlined. > So it looks like it's still mistuned for the kernel. That seems like valuable feedback to give to the GCC developers. Richi, you can find the whole thread at http://marc.info/?l=linux-fsdevel&m=123150610901773&w=2 and http://marc.info/?l=linux-fsdevel&m=123150834405285&w=2 is also relevant. -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 10:28:01AM -0700, Matthew Wilcox wrote: > On Fri, Jan 09, 2009 at 06:20:11PM +0100, Andi Kleen wrote: > > Also cc Honza in case he has comments (you might want > > to review more of the thread in the archives) > > I think this particular bug is already known and discussed: I thought so initially too, but: > > http://gcc.gnu.org/ml/gcc/2008-12/msg00365.html > > and it hints at being fixed with gcc 4.4. Does anyone want to test > that? Hugh already tested with 4.4 and it didn't work well. At least a lot of the low level asm inlines were not inlined. So it looks like it's still mistuned for the kernel. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 09:11:47AM -0800, Linus Torvalds wrote: > IIRC, the numbers mean different things for different versions of gcc, and > I think using the parameters was very strongly discouraged by gcc > developers. IOW, they were meant for gcc developers internal tuning > efforts, not really for external people. Which means that using them would When I asked last time that was not what I heard. Apparently at least some --params are considered ready for user consumption these days. > put us _more_ at the mercy of random compiler versions rather than less. Yes it would basically be a list in the Makefile keyed on compiler version giving different options and someone would need to do the work to do that for each new compiler version. That would be some work, but it might be less work than going all over 9.7MLOCs and changing inlines around manually. Also the advantage is that that you wouldn't need to teach the rules to hundreds of new driver programmers. Anyways I'm not very strongly wedded to this idea, but I think it's an alternative that should be at least considered before doing anything else drastic. -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 06:20:11PM +0100, Andi Kleen wrote: > Also cc Honza in case he has comments (you might want > to review more of the thread in the archives) I think this particular bug is already known and discussed: http://gcc.gnu.org/ml/gcc/2008-12/msg00365.html and it hints at being fixed with gcc 4.4. Does anyone want to test that? -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
> I vote for the, get rid of the current inline, rename __always_inline to There is some code that absolutely requires inline for correctness, like the x86-64 vsyscall code. I would advocate to keep the explicit __always_inline at least there to make it very clear. > inline, and then remove all non needed inlines from the kernel. Most inlines in .c should be probably dropped. > > We'll, probably start adding a lot more noinlines. That would cost you, see the numbers I posted (~4.1% text increase) -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Steven Rostedt wrote: > > I vote for the, get rid of the current inline, rename __always_inline to > inline, and then remove all non needed inlines from the kernel. This is what we do all the time, and historically have always done. But - CONFIG_OPTIMIZE_INLINING=y screws that up and - gcc still inlines even big functions static that have no markings at all. > We'll, probably start adding a lot more noinlines. That's going to be very painful. Especially since the cases we really want to not inline is the random drivers etc - generally not "hot in the cache", but they are the ones that cause the most oopses (not per line, but globally - because there's just so many drivers). Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 09:03:12AM -0500, jim owens wrote: > They also know inlining may increase program object size. > That inlining will reduce object size on many architectures > if the function is small is just a happy side effect to them. The problem is that the threshold for that is architecture specific. While e.g. x86 has relatively low overhead of prologue/epilogue other architectures like s390 have enormous overhead. So handling this in the compiler would be optimal, but it would need at least whole-program optimization and a compiler aware of the inline assembly to get it half-way right. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Andi Kleen wrote: > > There's also one alternative: gcc's inlining algorithms are extensibly > tunable with --param. We might be able to find a set of numbers that > make it roughly work like we want it by default. We tried that. IIRC, the numbers mean different things for different versions of gcc, and I think using the parameters was very strongly discouraged by gcc developers. IOW, they were meant for gcc developers internal tuning efforts, not really for external people. Which means that using them would put us _more_ at the mercy of random compiler versions rather than less. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, H. Peter Anvin wrote: > Dirk Hohndel wrote: > > > > Does gcc actually follow the "promise"? If that's the case (and if it's > > considered a bug when it doesn't), then we can get what Linus wants by > > annotating EVERY function with either __always_inline or noinline. > > > > __always_inline and noinline does work. I vote for the, get rid of the current inline, rename __always_inline to inline, and then remove all non needed inlines from the kernel. We'll, probably start adding a lot more noinlines. -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, Jan 09, 2009 at 08:46:20AM -0800, Dirk Hohndel wrote: > On Fri, 09 Jan 2009 08:34:57 -0800 > "H. Peter Anvin" wrote: > > > > As far as naming is concerned, gcc effectively supports four levels, > > which *currently* map onto macros as follows: > > > > __always_inline Inline unconditionally > > inline Inlining hint > >Standard heuristics > > noinlineUninline unconditionally > > > > A lot of noise is being made about the naming of the levels (and I > > personally believe we should have a different annotation for "inline > > unconditionally for correctness" and "inline unconditionally for > > performance", as a documentation issue), but those are the four we > > get. > > Does gcc actually follow the "promise"? If that's the case (and if it's > considered a bug when it doesn't), then we can get what Linus wants by > annotating EVERY function with either __always_inline or noinline. There's also one alternative: gcc's inlining algorithms are extensibly tunable with --param. We might be able to find a set of numbers that make it roughly work like we want it by default. Disadvantage: the whole thing will be compiler version dependent so we might need to have different numbers for different compiler versions and it will be an area that will need constant maintenance in the future. I'm not sure that's really a good path to walk down to. Also cc Honza in case he has comments (you might want to review more of the thread in the archives) -Andi -- a...@linux.intel.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Dirk Hohndel wrote: > > Does gcc actually follow the "promise"? If that's the case (and if it's > considered a bug when it doesn't), then we can get what Linus wants by > annotating EVERY function with either __always_inline or noinline. > __always_inline and noinline does work. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009 08:44:47 -0800 (PST) Linus Torvalds wrote: > On Fri, 9 Jan 2009, H. Peter Anvin wrote: > > As far as naming is concerned, gcc effectively supports four levels, > > which *currently* map onto macros as follows: > > > > __always_inline Inline unconditionally > > inline Inlining hint > >Standard heuristics > > noinlineUninline unconditionally > > > > A lot of noise is being made about the naming of the levels > > The biggest problem is the . > > The standard heuristics for that are broken, in particular for the > "single call-site static function" case. > > If gcc only inlined truly trivial functions for that case, I'd > already be much happier. Size be damned. See my other email. Maybe we should just stop trusting gcc and annotate every single function call. Ugly, but effective. /D -- Dirk Hohndel Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 09 Jan 2009 08:34:57 -0800 "H. Peter Anvin" wrote: > > As far as naming is concerned, gcc effectively supports four levels, > which *currently* map onto macros as follows: > > __always_inline Inline unconditionally > inlineInlining hint > Standard heuristics > noinline Uninline unconditionally > > A lot of noise is being made about the naming of the levels (and I > personally believe we should have a different annotation for "inline > unconditionally for correctness" and "inline unconditionally for > performance", as a documentation issue), but those are the four we > get. Does gcc actually follow the "promise"? If that's the case (and if it's considered a bug when it doesn't), then we can get what Linus wants by annotating EVERY function with either __always_inline or noinline. /D -- Dirk Hohndel Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, H. Peter Anvin wrote: > As far as naming is concerned, gcc effectively supports four levels, > which *currently* map onto macros as follows: > > __always_inline Inline unconditionally > inlineInlining hint > Standard heuristics > noinline Uninline unconditionally > > A lot of noise is being made about the naming of the levels The biggest problem is the . The standard heuristics for that are broken, in particular for the "single call-site static function" case. If gcc only inlined truly trivial functions for that case, I'd already be much happier. Size be damned. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Ingo Molnar wrote: > > My goal is to make the kernel smaller and faster, and as far as the > placement of 'inline' keywords goes, i dont have too strong feelings about > how it's achieved: they have a certain level of documentation value > [signalling that a function is _intended_ to be lightweight] but otherwise > they are pretty neutral attributes to me. > As far as naming is concerned, gcc effectively supports four levels, which *currently* map onto macros as follows: __always_inline Inline unconditionally inline Inlining hint Standard heuristics noinlineUninline unconditionally A lot of noise is being made about the naming of the levels (and I personally believe we should have a different annotation for "inline unconditionally for correctness" and "inline unconditionally for performance", as a documentation issue), but those are the four we get. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
On Fri, 9 Jan 2009, Ingo Molnar wrote: > > Core kernel developers tend to be quite inline-conscious and generally do > not believe that making something inline will make it go faster. Some of us core kernel developers tend to believe that: - inlining is supposed to work like macros, and should make the compiler do decisions BASED ON CALL-SITE. This is one of the most _common_ reasons for inlining. Making the compiler select static code rather than dynamic code, and using inlining as a nice macro. We can pass in a flag with a constant value, and only the case that matters will be compiled. It's not about size - or necessarily even performance - at all. It's about abstraction, and a way of writing code. And the thing is, as long as gcc does what we ask, we can notice when _we_ did something wrong. We can say "ok, we should just remove the inline" etc. But when gcc then essentially flips a coin, and inlines things we don't want to, it dilutes the whole value of inlining - because now gcc does things that actually does hurt us. We get oopses that have a nice symbolic back-trace, and it reports an error IN TOTALLY THE WRONG FUNCTION, because gcc "helpfully" inlined things to the point that only an expert can realize "oh, the bug was actually five hundred lines up, in that other function that was just called once, so gcc inlined it even though it is huge". See? THIS is the problem with gcc heuristics. It's not about quality of code, it's about RELIABILITY of code. The reason people use C for system programming is because the language is a reasonably portable way to get the expected end results WITHOUT the compiler making a lot of semantic changes behind your back. Inlining is also the wrong thing to do _even_ if it makes code smaller and faster if you inline the unlikely case, or inlining causes more live variables that cause stack pressure. And we KNOW this happens. Again, I'd be much happier if we had a compiler option to just does "do what I _say_, dammit", and then we can fix up the mistakes. Because then they are _our_ mistakes, not some random compiler version that throws a dice! Linus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
* jim owens wrote: > Ingo Molnar wrote: >> >> One interpretation of the numbers would be that core kernel hackers are >> more inline-happy, maybe because they think that their functions are >> more important to inline. >> >> Which is generally a fair initial assumption, but according to the >> numbers it does not appear to pay off in practice as it does not result >> in a smaller kernel image. > > I think people over-use inline for the opposite reason. Note that i talked about the core kernel (kernel/*.c) specifically. > They are taught: >- use inline functions instead of macros >- inlining functions makes your code run faster > > They also know inlining may increase program object size. That inlining > will reduce object size on many architectures if the function is small > is just a happy side effect to them. Core kernel developers tend to be quite inline-conscious and generally do not believe that making something inline will make it go faster. That's why i picked kernel/built-in.o as a good "best of breed" entity to measure - if then that is an area where we have at least the chance to do a "kernel coders know best when to inline" manual inlining job. But despite a decade of tuning and systematic effort in that area, the numbers suggest that we dont. (if someone has different numbers or different interpretation, please share it with us.) My goal is to make the kernel smaller and faster, and as far as the placement of 'inline' keywords goes, i dont have too strong feelings about how it's achieved: they have a certain level of documentation value [signalling that a function is _intended_ to be lightweight] but otherwise they are pretty neutral attributes to me. So we want all the mechanisms in place to constantly press towards a smaller and faster kernel, with the most efficient use of development resources. Some techniques work in practice despite looking problematic, some dont, despite looking good on paper. This might be one of those cases. Or not :-) Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
* Ingo Molnar wrote: > > I suspect gcc has some pre-inlining heuristics that don't take > > constant folding and simplifiation into account - if you look at just > > the raw tree of the function without taking the optimization into > > account, it will look big. > > Yeah. In my tests GCC 4.3.2 does properly inline that particular asm. here is how that function ended up looking like here (with Peter's v7, which should be close to what Chris tried, looking at his crash dump): 81491347 <__mutex_lock_common>: 81491347: 55 push %rbp 81491348: 48 89 e5mov%rsp,%rbp 8149134b: 41 57 push %r15 8149134d: 4c 8d 7f 08 lea0x8(%rdi),%r15 81491351: 41 56 push %r14 81491353: 49 89 f6mov%rsi,%r14 81491356: 41 55 push %r13 81491358: 41 54 push %r12 8149135a: 4c 8d 67 18 lea0x18(%rdi),%r12 8149135e: 53 push %rbx 8149135f: 48 89 fbmov%rdi,%rbx 81491362: 48 83 ec 38 sub$0x38,%rsp 81491366: 65 4c 8b 2c 25 00 00mov%gs:0x0,%r13 8149136d: 00 00 8149136f: b8 01 00 00 00 mov$0x1,%eax 81491374: 31 d2 xor%edx,%edx 81491376: f0 0f b1 13 lock cmpxchg %edx,(%rbx) 8149137a: 83 f8 01cmp$0x1,%eax 8149137d: 75 18 jne81491397 <__mutex_lock_common+0x50> 8149137f: 65 48 8b 04 25 10 00mov%gs:0x10,%rax 81491386: 00 00 81491388: 48 2d d8 1f 00 00 sub$0x1fd8,%rax 8149138e: 48 89 43 18 mov%rax,0x18(%rbx) 81491392: e9 dd 00 00 00 jmpq 81491474 <__mutex_lock_common+0x12d> 81491397: 85 c0 test %eax,%eax 81491399: 79 06 jns814913a1 <__mutex_lock_common+0x5a> 8149139b: 4c 39 7b 08 cmp%r15,0x8(%rbx) 8149139f: 75 19 jne814913ba <__mutex_lock_common+0x73> 814913a1: 49 8b 34 24 mov(%r12),%rsi 814913a5: 48 85 f6test %rsi,%rsi 814913a8: 74 0c je 814913b6 <__mutex_lock_common+0x6f> 814913aa: 48 89 dfmov%rbx,%rdi 814913ad: e8 6c dd b9 ff callq 8102f11e 814913b2: 85 c0 test %eax,%eax 814913b4: 74 04 je 814913ba <__mutex_lock_common+0x73> 814913b6: f3 90 pause 814913b8: eb b5 jmp8149136f <__mutex_lock_common+0x28> 814913ba: 4c 8d 63 04 lea0x4(%rbx),%r12 814913be: 4c 89 e7mov%r12,%rdi 814913c1: e8 d2 0f 00 00 callq 81492398 <_spin_lock> 814913c6: 48 8b 53 10 mov0x10(%rbx),%rdx 814913ca: 48 8d 45 b0 lea-0x50(%rbp),%rax 814913ce: 4c 89 7d b0 mov%r15,-0x50(%rbp) 814913d2: 48 89 43 10 mov%rax,0x10(%rbx) 814913d6: 48 89 02mov%rax,(%rdx) 814913d9: 48 89 55 b8 mov%rdx,-0x48(%rbp) 814913dd: 48 83 ca ff or $0x,%rdx 814913e1: 4c 89 6d c0 mov%r13,-0x40(%rbp) 814913e5: 48 89 d0mov%rdx,%rax 814913e8: 87 03 xchg %eax,(%rbx) 814913ea: ff c8 dec%eax 814913ec: 74 55 je 81491443 <__mutex_lock_common+0xfc> 814913ee: 44 88 f0mov%r14b,%al 814913f1: 44 89 f2mov%r14d,%edx 814913f4: 83 e0 81and$0xff81,%eax 814913f7: 83 e2 01and$0x1,%edx 814913fa: 88 45 afmov%al,-0x51(%rbp) 814913fd: 89 55 a8mov%edx,-0x58(%rbp) 81491400: 48 83 c8 ff or $0x,%rax 81491404: 87 03 xchg %eax,(%rbx) 81491406: ff c8 dec%eax 81491408: 74 39 je 81491443 <__mutex_lock_common+0xfc> 8149140a: 80 7d af 00 cmpb $0x
Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
Ingo Molnar wrote: One interpretation of the numbers would be that core kernel hackers are more inline-happy, maybe because they think that their functions are more important to inline. Which is generally a fair initial assumption, but according to the numbers it does not appear to pay off in practice as it does not result in a smaller kernel image. I think people over-use inline for the opposite reason. They are taught: - use inline functions instead of macros - inlining functions makes your code run faster They also know inlining may increase program object size. That inlining will reduce object size on many architectures if the function is small is just a happy side effect to them. jim -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact
* Linus Torvalds wrote: > On Thu, 8 Jan 2009, H. Peter Anvin wrote: > > > > Right. gcc simply doesn't have any way to know how heavyweight an > > asm() statement is > > I don't think that's relevant. > > First off, gcc _does_ have a perfectly fine notion of how heavy-weight > an "asm" statement is: just count it as a single instruction (and count > the argument setup cost that gcc _can_ estimate). > > That would be perfectly fine. If people use inline asms, they tend to > use it for a reason. > > However, I doubt that it's the inline asm that was the biggest reason > why gcc decided not to inline - it was probably the constant "switch()" > statement. The inline function actually looks pretty large, if it wasn't > for the fact that we have a constant argument, and that one makes the > switch statement go away. > > I suspect gcc has some pre-inlining heuristics that don't take constant > folding and simplifiation into account - if you look at just the raw > tree of the function without taking the optimization into account, it > will look big. Yeah. In my tests GCC 4.3.2 does properly inline that particular asm. But because we cannot really trust GCC in this area yet (or at all), today i've conducted extensive tests measuring GCC's interaction with inlined asm statements. I built hundreds of vmlinux's and distilled the numbers. Here are my findings. Firstly i've written 40 patches that gradually add an __asm_inline annotation to all inline functions of arch/x86/include/asm/*.h - i've annotated 224 inline functions that way. Then i've conducted an x86 defconfig and an x86 allyesconfig test for each of the 40 patches. Realizing that there's assymetry in the inlining practices of drivers and kernel code, i've also conducted a separate series of tests: measuring the size increase/decrease of the core kernel, kernel/built-in.o. The first table containts the size numbers of kernel/built-in.o, on allyesconfig builds, and the size delta (and percentage): ( Object size tests on kernel: v2.6.28-7939-g2150edc ( using: gcc (GCC) 4.3.2 20081007 (Red Hat 4.3.2-6) ( target: kernel/ kernel/built-in.o ( config: x86.64.allyes name text-size( delta) ( pct) - always-inline.patch: 1039591( 0) ( 0.000%) optimize-inlining.patch: 967724( -71867) ( -7.426%) asm-inline-0.patch: 967724( 0) ( 0.000%) asm-inline-bitops-simple.patch: 967691( -33) ( -0.003%) asm-inline-bitops-constant-set.patch: 966947(-744) ( -0.077%) asm-inline-bitops-test-and-set.patch: 966735(-212) ( -0.022%) asm-inline-bitops-ffs.patch: 966735( 0) ( 0.000%) asm-inline-__ffs.h: 966735( 0) ( 0.000%) asm-inline-__fls.h: 966735( 0) ( 0.000%) asm-inline-fls.h: 966735( 0) ( 0.000%) asm-inline-fls64.h: 966735( 0) ( 0.000%) asm-inline-apic.h: 966735( 0) ( 0.000%) asm-inline-atomic_32.h: 966735( 0) ( 0.000%) asm-inline-atomic_64.h: 966735( 0) ( 0.000%) asm-inline-checksum_32.h: 966735( 0) ( 0.000%) asm-inline-checksum_64.h: 966735( 0) ( 0.000%) asm-inline-cmpxchg_32.h: 966735( 0) ( 0.000%) asm-inline-cmpxchg_64.h: 966735( 0) ( 0.000%) asm-inline-desc.h: 966735( 0) ( 0.000%) asm-inline-futex.h: 966735( 0) ( 0.000%) asm-inline-i387.h: 966735( 0) ( 0.000%) asm-inline-io.h: 966735( 0) ( 0.000%) asm-inline-io_32.h: 966735( 0) ( 0.000%) asm-inline-io_64.h: 966735( 0) ( 0.000%) asm-inline-irqflags.h: 966735( 0) ( 0.000%) asm-inline-kexec.h: 966735( 0) ( 0.000%) asm-inline-kgdb.h: 966735( 0) ( 0.000%) asm-inline-kvm_host.h: 966735( 0) ( 0.000%) asm-inline-kvm_para.h: 966735( 0) ( 0.000%) asm-inline-lguest_hcall.h: 966735( 0) ( 0.000%) asm-inline-local.h: 966735( 0) ( 0.000%) asm-inline-msr.h: 966735( 0) ( 0.000%) asm-inline-paravirt.h: 966735(
