Re: OK to get rid of scaddins?
Hi Michael, On Mon, 2012-02-13 at 16:06 +0100, Michael Stahl wrote: (of course i don't care if you do it for a special merged libs mode, but C++ development is already a sufficiently unproductive activity that we shouldn't make it even more so...) Is it necessary to build with full debug enabled ? how slow is it really ? [ if it takes ten minutes - how slow is it to re-build with just the bits you want symbols for re-run whatever you're debugging ?]. i find it works quite well with 8GB of RAM, except that linking takes much longer (and you better not have 3 unit tests crash concurrently otherwise gdb will lock up the box for 15 minutes until OOM killer is invoked...). So - can you give some concrete ideas of time space it is taking to link our shared libraries ? and also the growth in size that we get - what is the stripped vs. non-stripped output ? Mark prolly has some great insights as to how to improve that. I wonder if the new 'gold' linker will help performance wise - have you tried it ? no, but the problem is really the space that the object files take up: they don't fit all into RAM cache, and ld is blocked on I/O most of the time (in tail_build). If we have 8Gb of debug symbols per module there is a -real- problem here; Lubos was talking of using some more magic / smaller debug option in the past: -gdwarf-4 Are you using that ? I believe we turned it off by default again for some reason or other: potentially we want to add a check for a tolerably recent toolchain and debugger on the system before defaulting to that [ it supposedly saved 30% of the size ], but you need gdb 7.3 really. Either way, it sucks to hinder ourselves from creating a more efficient library structure because of un-necessary performance problems in the toolchain ;-) All the best, Michael. -- michael.me...@suse.com , Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
linking performance (was: Re: OK to get rid of scaddins?)
On 14/02/12 11:52, Michael Meeks wrote: Hi Michael, On Mon, 2012-02-13 at 16:06 +0100, Michael Stahl wrote: (of course i don't care if you do it for a special merged libs mode, but C++ development is already a sufficiently unproductive activity that we shouldn't make it even more so...) Is it necessary to build with full debug enabled ? how slow is it really ? [ if it takes ten minutes - how slow is it to re-build with just the bits you want symbols for re-run whatever you're debugging ?]. i find it works quite well with 8GB of RAM, except that linking takes much longer (and you better not have 3 unit tests crash concurrently otherwise gdb will lock up the box for 15 minutes until OOM killer is invoked...). So - can you give some concrete ideas of time space it is taking to link our shared libraries ? and also the growth in size that we get - what is the stripped vs. non-stripped output ? Mark prolly has some great insights as to how to improve that. so the objects from gbuild modules with debug are about 6.5G: du -sh workdir/unxlngx6/*Object 1,1M workdir/unxlngx6/CObject 6,3G workdir/unxlngx6/CxxObject 960K workdir/unxlngx6/GenCObject 164M workdir/unxlngx6/GenCxxObject the libraries and executables linked from these, 2.9G: du -sh workdir/unxlngx6/LinkTarget/ 2,9G workdir/unxlngx6/LinkTarget/ i don't have a non-dbgutil tree to compare to. I wonder if the new 'gold' linker will help performance wise - have you tried it ? no, but the problem is really the space that the object files take up: they don't fit all into RAM cache, and ld is blocked on I/O most of the time (in tail_build). If we have 8Gb of debug symbols per module there is a -real- problem here; the problem is more likely that in tail_build we first compile all the object files, and only after they have all been built they are linked into libraries/executables. that probably results in sub-optimal use of disk cache :) perhaps i'll look into fixing that in gbuild when i have some time, but LinkTarget is annoyingly complex so that would need some testing... Lubos was talking of using some more magic / smaller debug option in the past: -gdwarf-4 Are you using that ? I believe we turned it off by default again for some reason or other: potentially we want to add a check for a tolerably recent toolchain and debugger on the system before defaulting to that [ it supposedly saved 30% of the size ], but you need gdb 7.3 really. no, AFAIR Lubos enabled that on master and there were some problems with it (gdb was horribly slow or something), so he disabled it again a week later. Either way, it sucks to hinder ourselves from creating a more efficient library structure because of un-necessary performance problems in the toolchain ;-) well perhaps there are really 2 different target audiences, product builds need fast start-up, while development builds need fast re-build cycles... All the best, Michael. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: linking performance (was: Re: OK to get rid of scaddins?)
Hi Michael, On Tue, 2012-02-14 at 13:07 +0100, Michael Stahl wrote: the problem is more likely that in tail_build we first compile all the object files, and only after they have all been built they are linked into libraries/executables. Perhaps; could it also be that we like to compile with gcc in some eight way parallel way, but when it comes to linking, we -really- don't want to bog our machine down in that way ? I wonder if we could explicitly limit parallelism of linking in some way (?) - we should prolly also do this for the java compilation which is often quite memory intensive and doesn't do well with umpteen in parallel (at least on my machine). that probably results in sub-optimal use of disk cache :) Riight; but the parallelism creating a problem with a working set that is avoidably larger than memory footprint sounds like a more likely culprit (perhaps) ? Either way, it sucks to hinder ourselves from creating a more efficient library structure because of un-necessary performance problems in the toolchain ;-) well perhaps there are really 2 different target audiences, product builds need fast start-up, while development builds need fast re-build cycles... Completely :-) trading one vs. the other sucks - hence the merged libs option, that we can use for product builds, and the non-merged version that we can use all these little, bitty libraries for (I guess). At least, until we have working incremental linking - it's always encouraging to see things like: http://gcc.gnu.org/wiki/GoldIncrementalLinking but - I wonder how the mozilla guys cope with this problem. All the best, Michael. -- michael.me...@suse.com , Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: linking performance (was: Re: OK to get rid of scaddins?)
Hi, Added Tom to the CC, see thread here: http://thread.gmane.org/gmane.comp.documentfoundation.libreoffice.devel/24055/focus=24096 (Although it seems posts from non-subscribers, aka me, are not allowed, so you won't see my earlier reply there. I'll send it to you separately.) On Tue, Feb 14, 2012 at 01:07:20PM +0100, Michael Stahl wrote: On 14/02/12 11:52, Michael Meeks wrote: On Mon, 2012-02-13 at 16:06 +0100, Michael Stahl wrote: (of course i don't care if you do it for a special merged libs mode, but C++ development is already a sufficiently unproductive activity that we shouldn't make it even more so...) Is it necessary to build with full debug enabled ? how slow is it really ? [ if it takes ten minutes - how slow is it to re-build with just the bits you want symbols for re-run whatever you're debugging ?]. i find it works quite well with 8GB of RAM, except that linking takes much longer (and you better not have 3 unit tests crash concurrently otherwise gdb will lock up the box for 15 minutes until OOM killer is invoked...). So - can you give some concrete ideas of time space it is taking to link our shared libraries ? and also the growth in size that we get - what is the stripped vs. non-stripped output ? Mark prolly has some great insights as to how to improve that. so the objects from gbuild modules with debug are about 6.5G: du -sh workdir/unxlngx6/*Object 1,1Mworkdir/unxlngx6/CObject 6,3Gworkdir/unxlngx6/CxxObject 960Kworkdir/unxlngx6/GenCObject 164Mworkdir/unxlngx6/GenCxxObject the libraries and executables linked from these, 2.9G: du -sh workdir/unxlngx6/LinkTarget/ 2,9Gworkdir/unxlngx6/LinkTarget/ i don't have a non-dbgutil tree to compare to. I wonder if the new 'gold' linker will help performance wise - have you tried it ? no, but the problem is really the space that the object files take up: they don't fit all into RAM cache, and ld is blocked on I/O most of the time (in tail_build). If we have 8Gb of debug symbols per module there is a -real- problem here; the problem is more likely that in tail_build we first compile all the object files, and only after they have all been built they are linked into libraries/executables. that probably results in sub-optimal use of disk cache :) perhaps i'll look into fixing that in gbuild when i have some time, but LinkTarget is annoyingly complex so that would need some testing... Lubos was talking of using some more magic / smaller debug option in the past: -gdwarf-4 Are you using that ? I believe we turned it off by default again for some reason or other: potentially we want to add a check for a tolerably recent toolchain and debugger on the system before defaulting to that [ it supposedly saved 30% of the size ], but you need gdb 7.3 really. no, AFAIR Lubos enabled that on master and there were some problems with it (gdb was horribly slow or something), so he disabled it again a week later. That really needs some investigation (which is why I added Tom to the CC) recent GDB versions improved startup performance a lot, so if enabling DWARF4 would make GDB startup slower again (which surprises me) that would be really bad. Especially since DWARF4 is now the default at least on Fedora. Could you give some more specifics about what was tried, what the original startup times for gdb were and how much change you saw? Either way, it sucks to hinder ourselves from creating a more efficient library structure because of un-necessary performance problems in the toolchain ;-) well perhaps there are really 2 different target audiences, product builds need fast start-up, while development builds need fast re-build cycles... Thanks, Mark ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: OK to get rid of scaddins?
On Tue, Feb 14, 2012 at 10:52:31AM +, Michael Meeks wrote: On Mon, 2012-02-13 at 16:06 +0100, Michael Stahl wrote: (of course i don't care if you do it for a special merged libs mode, but C++ development is already a sufficiently unproductive activity that we shouldn't make it even more so...) Is it necessary to build with full debug enabled ? how slow is it really ? [ if it takes ten minutes - how slow is it to re-build with just the bits you want symbols for re-run whatever you're debugging ?]. i find it works quite well with 8GB of RAM, except that linking takes much longer (and you better not have 3 unit tests crash concurrently otherwise gdb will lock up the box for 15 minutes until OOM killer is invoked...). So - can you give some concrete ideas of time space it is taking to link our shared libraries ? and also the growth in size that we get - what is the stripped vs. non-stripped output ? Mark prolly has some great insights as to how to improve that. Actually I was working on reducing the debug size for distro packaging, that won't immediately help in this case. There is some effort to reduce the size of the generated debug info between compiler and linker: http://gcc.gnu.org/wiki/DebugFission I wonder if the new 'gold' linker will help performance wise - have you tried it ? no, but the problem is really the space that the object files take up: they don't fit all into RAM cache, and ld is blocked on I/O most of the time (in tail_build). If we have 8Gb of debug symbols per module there is a -real- problem here; Lubos was talking of using some more magic / smaller debug option in the past: -gdwarf-4 In general using DWARF4 will allow the compiler to use a more compact representation. So if at all possible make sure you use -gdwarf-4 (on some distros this is already the default). Are you using that ? I believe we turned it off by default again for some reason or other: potentially we want to add a check for a tolerably recent toolchain and debugger on the system before defaulting to that [ it supposedly saved 30% of the size ], but you need gdb 7.3 really. Some tools don't yet support the new DWARF4 .debug_types sections. In that case you can use -gdwarf-4 -fno-debug-types-section to get the more compact DWARF4 representation except for the separate .debug_types. (This is the default for Fedora now.) Either way, it sucks to hinder ourselves from creating a more efficient library structure because of un-necessary performance problems in the toolchain ;-) The toolchain is constantly being updated to make the debuginfo more expressive and more compact, but it definitely helps to use the latest and greatest (gcc 4.6 or the 4.7-pre-release highly recommended). And for C++ linking speedups switching to the gold linker should also help. Cheers, Mark ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
faster java compilation (was Re: linking performance (was: Re: OK to get rid of scaddins?))
On 2012-02-14 14:22, Michael Meeks wrote: Perhaps; could it also be that we like to compile with gcc in some eight way parallel way, but when it comes to linking, we -really- don't want to bog our machine down in that way ? I wonder if we could explicitly limit parallelism of linking in some way (?) - we should prolly also do this for the java compilation which is often quite memory intensive and doesn't do well with umpteen in parallel (at least on my machine). Java compilation would be a hang of a lot faster if the build process passed all of the java files to the compiler in one go. Java is really not meant to be compiled in a file-at-a-time-model, it is meant to be compiled in a single-project-at-a-time go. Regards, Noel Grandin Disclaimer: http://www.peralex.com/disclaimer.html ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
OK to get rid of scaddins?
Do we really need to keep analysis and date as separate shared libraries? Would it be OK to move their source code over to the sc module, and merge their objects into the sc library, and their .component files into sc.component? Probably the code that refers to them could be simplified a bit then, too, but that I don't know how to do. --tml ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: OK to get rid of scaddins?
On 13/02/12 12:31, Tor Lillqvist wrote: Do we really need to keep analysis and date as separate shared libraries? Would it be OK to move their source code over to the sc module, and merge their objects into the sc library, and their ..component files into sc.component? is that really necessary? the sc, sd, sw libraries already take forever to link with full debug, and adding more stuff to them would also impact startup performance for the respective application. (of course i don't care if you do it for a special merged libs mode, but C++ development is already a sufficiently unproductive activity that we shouldn't make it even more so...) Probably the code that refers to them could be simplified a bit then, too, but that I don't know how to do. --tml ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: OK to get rid of scaddins?
is that really necessary? No, so OK, not then. --tml ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: OK to get rid of scaddins?
On Mon, 2012-02-13 at 12:38 +0100, Michael Stahl wrote: the sc, sd, sw libraries already take forever to link with full debug, You link with full debug ? :-) and adding more stuff to them would also impact startup performance for the respective application. Not necessarily; by merging libraries together we can potentially improve startup performance a fair bit. Fewer scattered libraries on disk means better access times, more scope for LTO, and faster run-time linking. (of course i don't care if you do it for a special merged libs mode, but C++ development is already a sufficiently unproductive activity that we shouldn't make it even more so...) Is it necessary to build with full debug enabled ? how slow is it really ? [ if it takes ten minutes - how slow is it to re-build with just the bits you want symbols for re-run whatever you're debugging ?]. I wonder if the new 'gold' linker will help performance wise - have you tried it ? Regards, Michael. -- michael.me...@suse.com , Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: OK to get rid of scaddins?
On Mon, 2012-02-13 at 13:31 +0200, Tor Lillqvist wrote: Do we really need to keep analysis and date as separate shared libraries ? Ho hum; well - they are not used at calc startup, but they are at document load - as soon as you start to enter a formula they are loaded, and if you load a non-trivial calc document (ie. not just a time-empty-doc-creation) they are loaded too. Would it be OK to move their source code over to the sc module, and merge their objects into the sc library, and their .component files into sc.component? They could certainly both be merged together into a libscaddins or somesuch; but I'd say in a 'merged' world, we'd want them all inside a 'libsc' I think. Probably the code that refers to them could be simplified a bit then, too, but that I don't know how to do. The whole idea of doing that UNO de-coupling there is somewhat strange, but I guess it provides a template for others that want to do that, and prolly those fn's are not performance criticial. HTH, Michael.; -- michael.me...@suse.com , Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: OK to get rid of scaddins?
On 13/02/12 15:47, Michael Meeks wrote: On Mon, 2012-02-13 at 12:38 +0100, Michael Stahl wrote: the sc, sd, sw libraries already take forever to link with full debug, You link with full debug ? :-) well i was actually surprised once that i do, but soon found out that somebody has changed gbuild.mk so that it enables debug when --enable-dbgutil is active. i'm not sure if _that_ is a good idea (after all we have --enable-debug as well, and i think it broke --enable-dbgutil --disable-debug), but actually i got used to always having symbols everywhere, because that way i get useful backtraces etc. on hard-to-reproduce crashes and deadlocks, which is quite useful. and adding more stuff to them would also impact startup performance for the respective application. Not necessarily; by merging libraries together we can potentially improve startup performance a fair bit. Fewer scattered libraries on disk means better access times, more scope for LTO, and faster run-time linking. sounds plausible in principle, but to me scaddins doesn't sound like something needed at startup; i think the fastest startup can only be attained by only loading what is actually necessary to start. (of course i don't care if you do it for a special merged libs mode, but C++ development is already a sufficiently unproductive activity that we shouldn't make it even more so...) Is it necessary to build with full debug enabled ? how slow is it really ? [ if it takes ten minutes - how slow is it to re-build with just the bits you want symbols for re-run whatever you're debugging ?]. i find it works quite well with 8GB of RAM, except that linking takes much longer (and you better not have 3 unit tests crash concurrently otherwise gdb will lock up the box for 15 minutes until OOM killer is invoked...). I wonder if the new 'gold' linker will help performance wise - have you tried it ? no, but the problem is really the space that the object files take up: they don't fit all into RAM cache, and ld is blocked on I/O most of the time (in tail_build). Regards, Michael. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: OK to get rid of scaddins?
Hi Tor, On Monday, 2012-02-13 13:31:11 +0200, Tor Lillqvist wrote: Do we really need to keep analysis and date as separate shared libraries? Would it be OK to move their source code over to the sc module, and merge their objects into the sc library, and their .component files into sc.component? Actually I'm glad that these are not lumped together under sc. They also serve as an entry point for people who want to develop their own set of UNO Add-In functions. If this is for the mere library count limited on Android I'd prefer if there was a mechanism that allowed to link those objects in only on that platform. Probably the code that refers to them could be simplified a bit then, too, but that I don't know how to do. I don't see easy simplification there even if we merged them. Eike -- LibreOffice Calc developer. Number formatter stricken i18n transpositionizer. GnuPG key 0x293C05FD : 997A 4C60 CE41 0149 0DB3 9E96 2F1A D073 293C 05FD pgplOo1nm4nif.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
