Re: build time optimization
Hi, Then next step would be not to build external libs during LO build. There is a magic option: --with-system-libs. If you try this option on debian based systems (x86_64) you would probably end up with clucene dependency not met error. ACK. While hacking on dropping all bundled 3rdparty libs, I faced similar problems. My workaround for now is to install all the deps locally (in the user who's running the build) and pass proper pkg-config env variables to ./configure. Just a matter of installing those two available clucene packages? Unfortunately not: as it turns out those packages are hopelessly outdated (2008). Yes, it seems there had been a major rewrite of clucene, and the two versions actually seem to be separate packages (from deployment side) now. I was also considering providing an recent clucene package for ubuntu (just didnt have the time to properly set up the ubuntu build machinery on my side). I'm thinking about another magic configure option, like --with-external-lib-directory=/opt/libo this would install all dependend libs to this directory during first build and this would *survive* the make clean call, so the next time all external lib artifacts would be reused from there. I wouldn't recommend this, as it makes the build machinery even more complex. Better write a small shell script that builds the missing deps (those not already in your distro) to some proper prefix and then calls the LO build machinery with proper parameters (eg. pkg-config search pathes). I've already hacked up a little bit, but not really useful yet. My plan is to package all the yet missing deps for the major distros (in my case: ubuntu) and throw them out of the LO tree entirely. cu -- Mit freundlichen Grüßen / Kind regards Enrico Weigelt VNC - Virtual Network Consult GmbH Head Of Development Äußere Bayreuther Str. 55, D - 90409 Nürnberg Tel: +49 911 72303-30 Fax: +49 911 72303-50 enrico.weig...@vnc.biz; www.vnc.de ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
Hi, It is not that certain that ramdisk on linux really helps. ramdisk eat-up memory that cannot be used for fs-cache... so ramdisk will increase the cache-miss at the kernel level... I've used ramdisks for automatic mass builds on some other projects, and it really helped (IIRC factor 10). But the scenario there was a bit different: much smaller codebases and the machines had a lot of ram (16GB), and i didn't want to keep the actual build output, just know whether it builds at all and certain post-tests succeed. I guess, we only find out if we try ;-o I have not tested ramdisk on Windows, so I do not know how effective it is. On what I remember from certain large enterprise hosting projects, Windows (at least pre Win7) was pretty bad in IO (the OS itself keeps the stores quite busy for certain housekeeping tasks, so at least on short load peaks that was quite unpleasent - no idea if that really matters much in this scenario). cu -- Mit freundlichen Grüßen / Kind regards Enrico Weigelt VNC - Virtual Network Consult GmbH Head Of Development Äußere Bayreuther Str. 55, D - 90409 Nürnberg Tel: +49 911 72303-30 Fax: +49 911 72303-50 enrico.weig...@vnc.biz; www.vnc.de ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
On Thu, 2012-05-24 at 09:23 +0200, David Ostrovsky wrote: I'm trying to accelerate LO (developer) build on linux_x86-64 system. :-) one systematic approach to this might be to try to unwind which complex[1] headers are most included and see if, in fact, with some more strategic forward definitions we can avoid that. For the gnumake world the workdir/*/*.d files have all those deps that can easily be sedded / sorted / uniq -c'd / sorted etc. into a list of the top includes. That of course has the benefit of creating a long-term speedup for all of us that won't involve custom tweaking the system. IIRC there was some easy hack related to that too. ATB, Michael. [1] - where complex is presumably defined by 'slow to parse' (perhaps we could time a compile of include that-header.hxx in a .cxx) -- michael.me...@suse.com , Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
On 2012-05-24 19:02, Norbert Thiebaud wrote: On Thu, May 24, 2012 at 10:44 AM, Noel Grandinnoelgran...@gmail.com wrote: I'm running a tinderbox with a ramdisk on Windows - reduces build time dramatically. can you post an How-to ? (and possibly some numbers) Note: I do not have enough memory to use it... but I may consider investing in some RAM if that is worth it... http://lists.freedesktop.org/archives/libreoffice/2012-May/031335.html ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
On Thu, May 24, 2012 at 2:23 AM, David Ostrovsky d.ostrov...@idaia.de wrote: Hi, I'm trying to accelerate LO (developer) build on linux_x86-64 system. Before you point me to wiki (distributed build, ccache and friends) and tell me to use ramdisk: I already did it all and it is still need to be optimized ;-) It is not that certain that ramdisk on linux really helps. ramdisk eat-up memory that cannot be used for fs-cache... so ramdisk will increase the cache-miss at the kernel level... ramdisk on Mac help tremendously because of how horrible HFS is lock-wise. I have not tested ramdisk on Windows, so I do not know how effective it is. Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
On Thu, May 24, 2012 at 5:08 PM, Norbert Thiebaud nthieb...@gmail.com wrote: I have not tested ramdisk on Windows, so I do not know how effective it is. I'm running a tinderbox with a ramdisk on Windows - reduces build time dramatically. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
On Thu, May 24, 2012 at 10:44 AM, Noel Grandin noelgran...@gmail.com wrote: On Thu, May 24, 2012 at 5:08 PM, Norbert Thiebaud nthieb...@gmail.com wrote: I have not tested ramdisk on Windows, so I do not know how effective it is. I'm running a tinderbox with a ramdisk on Windows - reduces build time dramatically. can you post an How-to ? (and possibly some numbers) Note: I do not have enough memory to use it... but I may consider investing in some RAM if that is worth it... Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
build time optimization
Hi, I'm trying to accelerate LO (developer) build on linux_x86-64 system. Before you point me to wiki (distributed build, ccache and friends) and tell me to use ramdisk: I already did it all and it is still need to be optimized ;-) There are many disable configure options already, like disable mozilla and friends, and for some libs the combination does the job, like disable xmlsec and disable mozilla does disable nss. Then next step would be not to build external libs during LO build. There is a magic option: --with-system-libs. If you try this option on debian based systems (x86_64) you would probably end up with clucene dependency not met error. Just a matter of installing those two available clucene packages? Unfortunately not: as it turns out those packages are hopelessly outdated (2008). So, just pick the version, that is shipped with LO, apply the patches in clucene directory and make install and we are done? Unfortunately not, because the next error probably would be clucene-contrib-lib is missing. As it turns out, there is still one clucene patch missing: to force clucene-contrib-lib to be actually installed. After applying this missing patch to clucene-contrib-lib build file the installation was sucessfull, and then next missing external dependency error appears ;-) (And with other external libs you would probably end up messing around with pkg-config because pc files are missing) But wait, LO does it already during the build, doesn't it? It picks up the really fresh external lib, applies his own patches and install it to his working dir ... just to throw it away after make clean ;-( I'm thinking about another magic configure option, like --with-external-lib-directory=/opt/libo this would install all dependend libs to this directory during first build and this would *survive* the make clean call, so the next time all external lib artifacts would be reused from there. The best is, this option would also help to accelerate the build on other platforms! Unfortunately this option would not be so easy to implement, I guess. Another thoughts on that? Ciao David ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
On 05/24/2012 09:23 AM, David Ostrovsky wrote: I'm thinking about another magic configure option, like --with-external-lib-directory=/opt/libo this would install all dependend libs to this directory during first build and this would *survive* the make clean call, so the next time all external lib artifacts would be reused from there. The line what to keep across a make clean and what to through away could be a gray one. On the one hand, some low-level LO modules (e.g., soltools) do not really change much these days, so you could argue to not rebuild them, either. On the other hand, the external modules *do* depend on LO code (its build machinery), so not re-building them after a make clean is unsound if you modified any LO code they depend on. In the end, I'd question why you do make clean anyway, if you are after fast build times. These days, our dependency tracking has apparently become fine enough that there are hardly any apparent build failures resulting from missing dependencies any longer. (And, btw, most of the --with-external-libs should work just fine. You typically only need a few select --without-external-* overrides.) Stephan ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
On Thursday 24 of May 2012, David Ostrovsky wrote: Then next step would be not to build external libs during LO build. There is a magic option: --with-system-libs. If you try this option on debian based systems (x86_64) you would probably end up with clucene dependency not met error. ... But wait, LO does it already during the build, doesn't it? It picks up the really fresh external lib, applies his own patches and install it to his working dir ... just to throw it away after make clean ;-( I'm thinking about another magic configure option, like --with-external-lib-directory=/opt/libo this would install all dependend libs to this directory during first build and this would *survive* the make clean call, so the next time all external lib artifacts would be reused from there. The best is, this option would also help to accelerate the build on other platforms! Unfortunately this option would not be so easy to implement, I guess. Another thoughts on that? We had a discussion about this recently, in the thread started at [1]. You have roughly the following options: - do not use 'make clean' - install as many system libraries as you can (or even build them explicitly from source), use --with-system-libraries and explicit --without-system-foo as needed - implement the --with-system-libraries=auto mentioned in [2] to save you the explicit --without-system-foo hassle Implementing your suggestion is possibly more work than any of these. [1] http://lists.freedesktop.org/archives/libreoffice/2012-April/030645.html [2] http://lists.freedesktop.org/archives/libreoffice/2012-April/030897.html -- Lubos Lunak l.lu...@suse.cz ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: build time optimization
Hi David, On Thu, May 24, 2012 at 9:23 AM, David Ostrovsky d.ostrov...@idaia.de wrote: I'm trying to accelerate LO (developer) build on linux_x86-64 system. Before you point me to wiki (distributed build, ccache and friends) and tell me to use ramdisk: I already did it all and it is still need to be optimized ;-) Well - do the few external modules that you don't have in your system really matter that much with regards to build time when you're already using ccache? Given that you build in the same directory always or use ccache's prefix-option so that the cache can actually be used, the time needed to build those few external libs is negligible, isn't it? I rather have make clean really do make clean for everything. (And with other external libs you would probably end up messing around with pkg-config because pc files are missing) Nah, you would just set PKG_CONFIG_PATH accordingly. I'm thinking about another magic configure option, like --with-external-lib-directory=/opt/libo Not necessary IMHO. As you also want to be able to run your build LO on your machine, don't you? So you need those external libs available on your system, thus you need to add the library path to your ld.so.conf or similar mechanism. this would install all dependend libs to this directory during first build and this would *survive* the make clean It is pointless to have system libraries in a system that is not available in the system, but only for the build. Unfortunately this option would not be so easy to implement, I guess. Another thoughts on that? Yes, run ./configure --prefix /opt/lodpes ; make ; make install on the external libs you're missing and export PKG_CONFIG_PATH=/opt/lodeps/share/pkgconfig:/usr/share/pkgconfig before running configure/autogen.sh And see above for the runtime thing. you can also just install them to /usr/local/ so you don't have to bother with LD_LIBRARY_PATH/ld.so.conf ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
