Re: [Libreoffice] ccache support
On Saturday 10 of September 2011, Norbert Thiebaud wrote: On Fri, Sep 9, 2011 at 1:22 PM, Lubos Lunak l.lu...@suse.cz wrote: Since there (AFAIR) haven't been any actual data presented in the discussion here are some number for my linux buildbot. Ccache hit statistics from a buildbot is probably the least realistic example possible. Of course the hit ratio is almost 100% when it repeatedly rebuilds almost the same source. For normal development builds the hit ratio should be much much lower, for many reasons (building noticeably less often and building when something does change being the primary two). I consider even my 40% hit ratio to be unusually high. The only useful numbers I can see is the ~5% ccache overhead, which should mean here the break even ratio is 10%, which I guess should be doable for LO, but without any real numbers this is still just guesswork. Note: when icecream is enable configure.in does _not_ auto-enable ccache (iow if you want ccache _with_ icecream you need to actively say --enable-ccahe or set up up transparently on your environment I have manual setup for either/both icecream and cccache, if this was directed at me. -- Lubos Lunak l.lu...@suse.cz ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Mon, Sep 12, 2011 at 10:08 AM, Lubos Lunak l.lu...@suse.cz wrote: On Saturday 10 of September 2011, Norbert Thiebaud wrote: On Fri, Sep 9, 2011 at 1:22 PM, Lubos Lunak l.lu...@suse.cz wrote: Since there (AFAIR) haven't been any actual data presented in the discussion here are some number for my linux buildbot. Ccache hit statistics from a buildbot is probably the least realistic example possible. Of course the hit ratio is almost 100% when it repeatedly rebuilds almost the same source. For normal development builds the hit ratio should be much much lower, for many reasons (building noticeably less often and building when something does change being the primary two). I consider even my 40% hit ratio to be unusually high. The only useful numbers I can see is the ~5% ccache overhead, which should Agreed. that was indeed the most interesing part. the fact that ccache give good benefit for a completely hot cache is indeed quite expected. mean here the break even ratio is 10%, which I guess should be doable for LO, but without any real numbers this is still just guesswork. Well that is real number for overhead... since the difference between the no ccache run and the empty cache ccache run is purely the ccache overhead. so it is a very good indicator of the ccache overhead. Note: when icecream is enable configure.in does _not_ auto-enable ccache (iow if you want ccache _with_ icecream you need to actively say --enable-ccahe or set up up transparently on your environment I have manual setup for either/both icecream and cccache, if this was directed at me. I was wondering if the difference of overhead observed between your numbers and mine may have been related to ice-cream... All that being said. I am not overly attached to the default. I mean I can just as easily add --enable-ccache to my autogen.lastrun as you can add --disable-ccache to yours. and as I said earlier, as partial rebuild becomes more reliable it may very well make sens to have it 'disabled' by default. The problem is that today, when someone hit a build issue, the first advice we give is make clean make... which is typically 4 to 5 time more painful if you did not use ccache That is why I think that --enable-ccache is a much more causal/new-dev friendly. and more experience dev can pretty easily chose the setting that fit best their build pattern Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Tue, 2011-08-30 at 12:56 -0500, Norbert Thiebaud wrote: 2/ I had a spike in IRC questions/confusions from new tentative contributors on that topic, so I wanted to make it as newbie friendly as possible. iow a 'just fscking work' approach. Cache compression is advertised as saving space in the cache in return for a modest increase in compilation time. For me, it is the clear choice: I see the improved cache usage, and I do not notice the increased time. I wonder if any of our hackers using ccache would have the contrary opinion. IOW, should our configuration enable compression unconditionally? As an option to autogen.sh? (FWIW, I have changed my local set_soenv.in to enable compression unconditionally, and I have completed one successful build.) Terry. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, Sep 9, 2011 at 11:10 AM, Terrence Enger ten...@iseries-guru.com wrote: On Tue, 2011-08-30 at 12:56 -0500, Norbert Thiebaud wrote: 2/ I had a spike in IRC questions/confusions from new tentative contributors on that topic, so I wanted to make it as newbie friendly as possible. iow a 'just fscking work' approach. Cache compression is advertised as saving space in the cache in return for a modest increase in compilation time. For me, it is the clear choice: I see the improved cache usage, and I do not notice the increased time. Well, my boxes are usually cpu bound (even with a fully hot ccache) and I have plenty of space for my ccache, so that is not that clear a choice to me. I wonder if any of our hackers using ccache would have the contrary opinion. IOW, should our configuration enable compression unconditionally? As an option to autogen.sh? (FWIW, I have changed my local set_soenv.in to enable compression unconditionally, and I have completed one successful build.) you can set that up with an env variable right ? so you can easily make ccache behave the way you want, 'automatically', for you. (i.e export CCACHE_COMPRESS=1 in your bashrc) I don't do that in configure for the same reason we don't change CCACHE_DIR or the cache dir size. Using ccache if available is one thing... but trying to 'auto-magically' optimize ccache is another can of worms altogether... Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Talking about compression of the ccache cache, ... On Fri, 2011-09-09 at 11:35 -0500, Norbert Thiebaud wrote: Well, my boxes are usually cpu bound (even with a fully hot ccache) and I have plenty of space for my ccache, so that is not that clear a choice to me. So, unconditional compression is no good. you can set that up with an env variable right ? so you can easily make ccache behave the way you want, 'automatically', for you. (i.e export CCACHE_COMPRESS=1 in your bashrc) Yes, it is the existence of the envvar that controls. Setting it in .bashrc makes a lot of sense in that the relative availability of I/O capacity vs. CPU capacity does not depend on what product the user is building. The use of .bashrc is so easy and natural (thank you for reminding me about it!) that I think LO should not even provide a configuration option for compression. I notice, BTW, that How to build LibreOffice http://wiki.documentfoundation.org/Development/How_to_build no longer says that we automatically enable ccache if it is found on the system. Should that statement be added back to the page, now that it is true? I don't do that in configure for the same reason we don't change CCACHE_DIR or the cache dir size. Using ccache if available is one thing... but trying to 'auto-magically' optimize ccache is another can of worms altogether... Agree. Especially so, considering that sometimes a cache may be shared between different users and different systems. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, Sep 9, 2011 at 12:35 PM, Terrence Enger ten...@iseries-guru.com wrote: I notice, BTW, that How to build LibreOffice http://wiki.documentfoundation.org/Development/How_to_build no longer says that we automatically enable ccache if it is found on the system. Should that statement be added back to the page, now that it is true? Sure. Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Friday 09 of September 2011, Norbert Thiebaud wrote: On Fri, Sep 9, 2011 at 11:10 AM, Terrence Enger ten...@iseries-guru.com wrote: On Tue, 2011-08-30 at 12:56 -0500, Norbert Thiebaud wrote: 2/ I had a spike in IRC questions/confusions from new tentative contributors on that topic, so I wanted to make it as newbie friendly as possible. iow a 'just fscking work' approach. Cache compression is advertised as saving space in the cache in return for a modest increase in compilation time. For me, it is the clear choice: I see the improved cache usage, and I do not notice the increased time. Well, my boxes are usually cpu bound (even with a fully hot ccache) and I have plenty of space for my ccache, so that is not that clear a choice to me. Since there (AFAIR) haven't been any actual data presented in the discussion about enabling ccache, let me present some. I admit they are somewhat special, but they are real nevertheless. So, the build setup is Intel Xeon @2.53GHz (4 cores), 12G RAM, quite powerful icecream network (--with-max-jobs=40) and HDD with 120MB/s speed reported by hdparm (but it's not SSD, so the seek time is awful as it is with HDDs). I also build with --enable-debug. - cd svx; make clean; make (just to make the disk caches hot) - make clean; time make (i.e. no ccache) real0m52.378s user1m37.430s sys 0m21.630s - make clean; ccache -C; time cmakeset make (cmakeset enables ccache use here) real1m1.170s user1m58.422s sys 0m27.661s - make clean; time cmakeset make real0m14.562s user0m28.270s sys 0m6.972s So if ccache has 0% hit ratio, there is 17% overhead in this specific case. With 100% hit ratio, 73% is saved. That, if my math is right, means at least 19% ccache hit ratio is required for breaking even. My ccache -s reports cache hit (direct)173894 cache hit (preprocessed) 43418 cache miss323627 which is 40% hit ratio, but this is with using ccache only explicitly for full rebuilds (i.e. when I develop e.g. in sw/ and build just there, it's without ccache). I'm also not sure how many hits come from rebuilding everything from scratch, which is not something people without this powerful build setup would do that easily (I sometimes just give up on a problem and rebuilding everything, and after pull -r I usually also rebuild from scratch in another dir before switching to it). Finally, it doesn't say anything about how many of those hits are tiny .c files and how many of the misses are huge .cxx files, which are more likely to be misses because of having more dependencies. Could somebody else, preferably with a somewhat more common system :), post their results, so that we can compare? I assume the ccache overhead (caused by I/O I'd say) is exceptionally high here, but I still doubt ccache warrants being enabled by default just like that. I don't do that in configure for the same reason we don't change CCACHE_DIR or the cache dir size. Using ccache if available is one thing... but trying to 'auto-magically' optimize ccache is another can of worms altogether... I think the case above shows that there are some worms even in the one thing. -- Lubos Lunak l.lu...@suse.cz ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, 2011-09-09 at 12:47 -0500, Norbert Thiebaud wrote: On Fri, Sep 9, 2011 at 12:35 PM, Terrence Enger ten...@iseries-guru.com wrote: I notice, BTW, that How to build LibreOffice http://wiki.documentfoundation.org/Development/How_to_build no longer says that we automatically enable ccache if it is found on the system. Should that statement be added back to the page, now that it is true? Sure. Uh oh. That page is restricted to administrators. Just as well, as I would have had to send a message to the discuss list saying that I do not know anything about the Mac and so had not restored the simultaneously deleted text about the Mac, and inviting someone who knows something to write something. Terry. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, Sep 9, 2011 at 1:53 PM, Terrence Enger ten...@iseries-guru.com wrote: On Fri, 2011-09-09 at 12:47 -0500, Norbert Thiebaud wrote: On Fri, Sep 9, 2011 at 12:35 PM, Terrence Enger ten...@iseries-guru.com wrote: I notice, BTW, that How to build LibreOffice http://wiki.documentfoundation.org/Development/How_to_build no longer says that we automatically enable ccache if it is found on the system. Should that statement be added back to the page, now that it is true? Sure. Uh oh. That page is restricted to administrators. no it is not. you just need to register to the wiki... but there is no special restriction on that. Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, Sep 9, 2011 at 1:22 PM, Lubos Lunak l.lu...@suse.cz wrote: So if ccache has 0% hit ratio, there is 17% overhead in this specific case. With 100% hit ratio, 73% is saved. That, if my math is right, means at least 19% ccache hit ratio is required for breaking even. I'm am gathering numbers, in the mean time http://tinderbox.libreoffice.org/MASTER/status.html give me some good data about the ccache hit. by looking at the time it take for each build iteration... most of them show that the hit ration is almost always very very high (90%+), unless some commit touch a commonly included header. Now granted, once we can reliably do incremental build, the question of the effectiveness of ccache will be much more acute. presumable incremental build only try to compile stuff that _need_ to be compiled and therefore should be a ccache miss... so with perfect incremental build the ccache hit should be 0% when that is true, we could indeed change the default for ccache to disable FYI: my ccachs -s for my MacOSX buildbot is n_th@tpamac ~/.ssh $ccache -s cache directory /Volumes/ccache_ramdisk cache hit (direct)481347 cache hit (preprocessed) 19049 cache miss 28588 called for link25314 multiple source files102 compile failed 8524 preprocessor error 4318 bad compiler arguments 1768 unsupported source language 408 autoconf compile/link 35802 unsupported compiler option60295 no input file 7446 files in cache 43163 cache size 1.8 Gbytes max cache size 2.3 Gbytes so a hit ration of 94.6% (481347 + 19049) / (481347 + 19049 + 25588) (note that cache had been wiped out about 2 days ago due to an IPL of the box -- the cache is in ramdisk, so it does not survive a reboot) Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, 2011-09-09 at 20:22 +0200, Lubos Lunak wrote: On Friday 09 of September 2011, Norbert Thiebaud wrote: Could somebody else, preferably with a somewhat more common system :), post their results, so that we can compare? I assume the ccache overhead (caused by I/O I'd say) is exceptionally high here, but I still doubt ccache warrants being enabled by default just like that. I don't do that in configure for the same reason we don't change CCACHE_DIR or the cache dir size. Using ccache if available is one thing... but trying to 'auto-magically' optimize ccache is another can of worms altogether... I think the case above shows that there are some worms even in the one thing. Well, let me throw some more numbers on the pile, about space, number of cached files, and hits. Short version: cache compression reduces cache size by very roughly a factor of 5. My RAM is so small, at 512MB, that I omit timings; besides I kept using the system for other things at the same time. All compiles yesterday were with cache compression turned on. Size is in KB as reported by `du --summarize ~/.ccache`. Files, hits, and misses are from `ccache -s`. I started yesterday with a cache holding files from a number of previous builds of LibreOffice, some of them compressed, but lots not (just because I keep forgetting to set CCACHE_COMPRESS; yes, I have now added that to my .bashrc): size : 9563228 files : 79453 hits : 54662 + 510 misses : 39198 I calculate average file size 120.02KB After some unsuccessful attempts to rebuild: size : 9618256 files : 81585 hits : 54874 + 629 misses : 39990 I calculate average file size 117.85KB and average size 21.03KB for the 2184 new files. After successful complete build, which cleaned out part of the cache: size : 9123940 files : 99112 hits : 62377 + 3080 misses : 48614 I calculate average file size 92.06KB. FWIW, and HTH Terry. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, 2011-09-09 at 14:22 -0500, Norbert Thiebaud wrote: Now granted, once we can reliably do incremental build, the question of the effectiveness of ccache will be much more acute. presumable incremental build only try to compile stuff that _need_ to be compiled and therefore should be a ccache miss... so with perfect incremental build the ccache hit should be 0% Not quite, I think. Consider a header file changed within in #if-skipped region in a way which does not change the number of skipped lines. I expect this would trigger a recompile and that ccache, other things being equal, would find a hit on the cache. Does this sound plausible. Still, I would expect the hit rate to go really close to 0. Terry. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, Sep 9, 2011 at 3:55 PM, Terrence Enger ten...@iseries-guru.com wrote: On Fri, 2011-09-09 at 14:22 -0500, Norbert Thiebaud wrote: Now granted, once we can reliably do incremental build, the question of the effectiveness of ccache will be much more acute. presumable incremental build only try to compile stuff that _need_ to be compiled and therefore should be a ccache miss... so with perfect incremental build the ccache hit should be 0% Not quite, I think. Consider a header file changed within in #if-skipped region in a way which does not change the number of skipped lines. I expect this would trigger a recompile and that ccache, other things being equal, would find a hit on the cache. Does this sound plausible. Still, I would expect the hit rate to go really close to 0. And as Lubos remarked earlier... you need to be in the 20% hit rate to get your money-back... Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Fri, Sep 9, 2011 at 1:22 PM, Lubos Lunak l.lu...@suse.cz wrote: Since there (AFAIR) haven't been any actual data presented in the discussion here are some number for my linux buildbot. no ccache, clean full build real56m2.046s user187m2.925s sys 20m45.332s ccache, empty cache, clean full build real59m8.259s user193m59.283s sys 23m41.392s cache directory /home/n_th/.ccache cache hit 78 cache miss 11983 called for link 480 multiple source files 3 compile failed 845 preprocessor error28 not a C/C++ file 103 autoconf compile/link616 unsupported compiler option 602 no input file 1247 files in cache 23966 cache size 629.2 Mbytes max cache size 8.0 Gbytes ccache fully hot, clean full build real14m9.809s user35m0.468s sys 10m8.120s cache directory /home/n_th/.ccache cache hit 11915 cache miss 12186 called for link 959 multiple source files 6 compile failed 1687 preprocessor error56 not a C/C++ file 206 autoconf compile/link 1196 unsupported compiler option 1203 no input file 2492 files in cache 24372 cache size 630.0 Mbytes max cache size 8.0 Gbytes Note: when icecream is enable configure.in does _not_ auto-enable ccache (iow if you want ccache _with_ icecream you need to actively say --enable-ccahe or set up up transparently on your environment ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Hi all, On Wed, Aug 10, 2011 at 11:27:05AM +0200, Jan Holesovsky wrote: On 2011-08-08 at 11:33 +0200, Lubos Lunak wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378 ca38f59518b018fccca8f This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX I disagree with this change, as it does not improve anything for anybody. Well, I am not that happy about using ccache by default either ;-) It collides with the icecream support, I'm not happy about this commit either: when packaging LibreOffice under pkgsrc, it doubles the size of the build tree for no gain in speed. The pkgsrc framework sanitizes the environment; $HOME is a temporary directory which is always removed and created anew for each build. I'm using ccache for development builds (from git), and I _don't_ want to use it when creating packages on the same machine. Could this change be reverted ? Instead of improving things, it degrades them. -- Francois Tigeot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Tue, Aug 30, 2011 at 11:58 AM, Norbert Thiebaud nthieb...@gmail.com wrote: On Tue, Aug 30, 2011 at 11:57 AM, Francois Tigeot ftig...@wolfpond.org wrote: Hi all, On Wed, Aug 10, 2011 at 11:27:05AM +0200, Jan Holesovsky wrote: On 2011-08-08 at 11:33 +0200, Lubos Lunak wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378 ca38f59518b018fccca8f This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX I disagree with this change, as it does not improve anything for anybody. Well, I am not that happy about using ccache by default either ;-) It collides with the icecream support, I'm not happy about this commit either: when packaging LibreOffice under pkgsrc, it doubles the size of the build tree for no gain in speed. then use --disable-ccache as in CONFIGURE_ARGS+= --disable-ccache :-) Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Tue, Aug 30, 2011 at 12:00:54PM -0500, Norbert Thiebaud wrote: On Tue, Aug 30, 2011 at 11:57 AM, Francois Tigeot ftig...@wolfpond.org wrote: I'm not happy about this commit either: when packaging LibreOffice under pkgsrc, it doubles the size of the build tree for no gain in speed. then use --disable-ccache as in CONFIGURE_ARGS+= --disable-ccache Nice, I don't even have to do the job anymore ;-) Still, this ccache stuff is too clever for its own good IMHO. -- Francois Tigeot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Tue, Aug 30, 2011 at 12:16 PM, Francois Tigeot ftig...@wolfpond.org wrote: On Tue, Aug 30, 2011 at 12:00:54PM -0500, Norbert Thiebaud wrote: On Tue, Aug 30, 2011 at 11:57 AM, Francois Tigeot ftig...@wolfpond.org wrote: I'm not happy about this commit either: when packaging LibreOffice under pkgsrc, it doubles the size of the build tree for no gain in speed. then use --disable-ccache as in CONFIGURE_ARGS+= --disable-ccache Nice, I don't even have to do the job anymore ;-) Still, this ccache stuff is too clever for its own good IMHO. for reference here is the motivations that lead me to do it 1/ the configure.in help advertised that we where doing it (as in claimed that if ccache was present we would automatically use it) 2/ I had a spike in IRC questions/confusions from new tentative contributors on that topic, so I wanted to make it as newbie friendly as possible. iow a 'just fscking work' approach. 3/ I had issue with MacOS: in order to use ccache I had to set CC and CXX before autogen (or as autogen.sh parameters). but doing so meant that I had to take into account which compiler should be use. and on Mac that depend on the SDK being used... so basically I had to do manually the job of configure... to set the proper CC and CXX manually 4/ I considered that one-time builder were not the audience I cared about, and that distrib-builder are savvy enough to make the build system do what they want (iow add --disable-ccache if they decided that ccache is not useful to them) and really it is not _that_ clever: if ccache is disabled or icecream is enabled or there is no ccache on the system we do nothing special else if CC or CCX are manually preset we do nothing special (presumably if you bothered to explicitly pre-set it and wanted ccache you would have set it so) else (that was added later on remark that CCACHE_DIR may be to small to be useful) if the ccache size is less than 1GB then we do nothing and warn about it else if ccache size is between 1GB and 5GB then we warn that you should prolly set a bigger ccache size especially if you intend to do debug/symbol build fi if CC or CXX respectively point to ccache already (case when you alias gcc to ccache) then we do nothing special else we prepend ccache to CC or CXX respectively fi fi fi fi Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Tue, Aug 30, 2011 at 12:56:50PM -0500, Norbert Thiebaud wrote: On Tue, Aug 30, 2011 at 12:16 PM, Francois Tigeot ftig...@wolfpond.org wrote: Still, this ccache stuff is too clever for its own good IMHO. for reference here is the motivations that lead me to do it [Snip all sorts of good reasons] Well, don't be bothered too much by what I said, sometimes there aren't perfect solutions -- Francois Tigeot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Hi Norbert, Lubos, On 2011-08-08 at 11:33 +0200, Lubos Lunak wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378 ca38f59518b018fccca8f This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX I disagree with this change, as it does not improve anything for anybody. Well, I am not that happy about using ccache by default either ;-) It collides with the icecream support, and while it would be possible to combine ccache and icecream [1], it is generally something that will save you just very little with big enough icecream farm. For now I've changed it so that when there is --enable-icecream, the ccache is disabled, unless explicitly enabled; but it is quite inconsistent: http://cgit.freedesktop.org/libreoffice/core/commit/?id=d865fed229035440eccb8a89aa3f44f64e0f696c Regards, Kendy [1] http://en.opensuse.org/Icecream#How_to_combine_icecream_with_ccache ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Mon, 2011-08-08 at 11:40 +0200, Christian Lohmaier wrote: Hi Lubos, (snip) Not true either. Default ccache size is 1GB, and that is enough when you focus on one branch. (a build uses less than 500MB) Just a note about what I have observed, for anybody who happens to care. I have configured LO with options --disable-mozilla --enable-symbols --enable-dbgutil --enable-crashdump --disable-build-mozilla --without-junit and completed my first build of LO using ccache up to the step `make`. Cache statistics look like ... cache directory /home/terry/.ccache cache hit (direct) 6158 cache hit (preprocessed) 37 cache miss 11986 called for link 563 multiple source files 3 compile failed 212 preprocessor error34 bad compiler arguments60 unsupported source language 15 autoconf compile/link611 unsupported compiler option 1426 no input file 1309 files in cache 30789 cache size 4.1 Gbytes max cache size 10.0 Gbytes This makes me really glad that I set a max cache size larger than the default 1 GB, For comparison, after I had completed several non-product builds of OpenOffice, my cache statistics looked like ... cache directory /media/2ced347d-3cdc-49f5-956d-099d36784957/terry/OOo_hacking/SRC_DIR_ccache cache hit (direct) 0 cache hit (preprocessed) 15499 cache miss 14993 called for link 1192 multiple source files 9 compile failed91 preprocessor error24 unsupported source language 186 autoconf compile/link666 no input file450 files in cache 29986 cache size 1.5 Gbytes max files 10 max cache size 10.0 Gbytes I do not remember turning on cache compression back then, but that could well explain the big difference in cache size. Cheers, Terry. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Tue, Aug 9, 2011 at 1:58 PM, Terrence Enger ten...@iseries-guru.com wrote: On Mon, 2011-08-08 at 11:40 +0200, Christian Lohmaier wrote: Hi Lubos, (snip) Not true either. Default ccache size is 1GB, and that is enough when you focus on one branch. (a build uses less than 500MB) Just a note about what I have observed, for anybody who happens to care. Since we are sharing data point, here is ccache -s for the MacOSX tinderbox: cache directory /Volumes/ccache_ramdisk cache hit (direct) 5595178 cache hit (preprocessed) 217622 cache miss362720 called for link 297911 multiple source files 1281 compile failed 98681 preprocessor error 49165 couldn't find the compiler 5 bad compiler arguments 21278 unsupported source language 4445 autoconf compile/link 420383 unsupported compiler option 689280 no input file 89750 files in cache 59804 cache size 2.0 Gbytes max cache size 2.3 Gbytes ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Hi *, On Tue, Aug 9, 2011 at 8:58 PM, Terrence Enger ten...@iseries-guru.com wrote: On Mon, 2011-08-08 at 11:40 +0200, Christian Lohmaier wrote: Not true either. Default ccache size is 1GB, and that is enough when you focus on one branch. (a build uses less than 500MB) Just a note about what I have observed, for anybody who happens to care. cache directory /home/terry/.ccache cache hit (direct) 6158 cache hit (preprocessed) 37 cache miss 11986 [...] files in cache 30789 cache size 4.1 Gbytes max cache size 10.0 Gbytes This makes me really glad that I set a max cache size larger than the default 1 GB, So you showed some irrelevant numbers, at least for the current topic. Your cache is filled with other stuff already, not with just the files from one single build. And of course ccache will happily use whatever size is available before it starts to expire old files. But yes, If you build with debug symbols, you need larger cache size. The 500MB for a build was referring to product builds, without any debugging options. For comparison, after I had completed several non-product builds of OpenOffice, my cache statistics looked like ... cache directory /media/2ced347d-3cdc-49f5-956d-099d36784957/terry/OOo_hacking/SRC_DIR_ccache cache hit (direct) 0 cache hit (preprocessed) 15499 cache miss 14993 That is a rather bad hit ratio - something between 5:1 and 10:1 when building often would be what I expect. 1:1 is when you build versions that did have lots of changes in between. And no direct hits? Did you disable it? Norbert's ratio is of course when you build multiple times a day, so might not be that representative either :-) But you see that the cache-size doesn't need to be big to get hits. When you stay on one branch, then the files that didn't match won't match in future, so no need to keep them around, they can be expired. When you switch back and forth in history and between different branches, you should consider using different cache-directories (when you build the other branches seldomly compared to your main tree), or use a bigger size (if the other branch gets build regularly, and thus the files in the cache are touched and thus not considered candidates for expiry). max files 10 Any reason why you set a limit on the number of files? ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
actually the more interesting numbers are: --disable-ccache make clean time make ccache -C --enable-ccache make clean time make make clean time make Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Tue, 2011-08-09 at 21:33 +0200, Christian Lohmaier wrote: Hi *, On Tue, Aug 9, 2011 at 8:58 PM, Terrence Enger ten...@iseries-guru.com wrote: On Mon, 2011-08-08 at 11:40 +0200, Christian Lohmaier wrote: Not true either. Default ccache size is 1GB, and that is enough when you focus on one branch. (a build uses less than 500MB) Just a note about what I have observed, for anybody who happens to care. cache directory /home/terry/.ccache cache hit (direct) 6158 cache hit (preprocessed) 37 cache miss 11986 [...] files in cache 30789 cache size 4.1 Gbytes max cache size 10.0 Gbytes This makes me really glad that I set a max cache size larger than the default 1 GB, So you showed some irrelevant numbers, at least for the current topic. Your cache is filled with other stuff already, not with just the files from one single build. And of course ccache will happily use whatever size is available before it starts to expire old files. Actually, my cache was empty when I started just after midnight ... cache directory /home/terry/.ccache cache hit (direct) 0 cache hit (preprocessed) 0 cache miss 0 files in cache 0 cache size 0 Kbytes max cache size 1.0 Gbytes Then I increased the maximum size. But yes, If you build with debug symbols, you need larger cache size. The 500MB for a build was referring to product builds, without any debugging options. For comparison, after I had completed several non-product builds of OpenOffice, my cache statistics looked like ... cache directory /media/2ced347d-3cdc-49f5-956d-099d36784957/terry/OOo_hacking/SRC_DIR_ccache cache hit (direct) 0 cache hit (preprocessed) 15499 cache miss 14993 That is a rather bad hit ratio - something between 5:1 and 10:1 when building often would be what I expect. 1:1 is when you build versions that did have lots of changes in between. And no direct hits? Did you disable it? This is from last year. I suspect the numbers result from two builds, the first giving almost all misses and second giving almost all hits. I dunno about direct hits. Direct hits may be a feature newer than the ccache I was running then. (snip) max files 10 Any reason why you set a limit on the number of files? No, I think ccache did that all on its own. OTOH, the man page says there is no default value for --max-files. The output from `script` does not contain the string max-files. Sigh. Terry. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
If one of us (cloph) strongly feels that we should use # comments in configure.in and not dnl comments (well, keywords), and most others (?) (including me) are indifferent, why don't just change them all in one go then? I wonder if there is some cases where the dnl keyword can not just be changed to # without changing the semantics of the autoconf/shell code? --tml ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Hi Norbert, *, On Mon, Aug 8, 2011 at 2:46 AM, Norbert Thiebaud nthieb...@gmail.com wrote: On Sun, Aug 7, 2011 at 7:21 PM, Christian Lohmaier lohmaier+libreoff...@googlemail.com wrote: if it's already an alias for ccache, it won't be prefixed with an additional ccace [...] I'm not sure I'm following: are you talking of the case where someone, on MacOS, has set-up ccache to masquerade as the compiler ? I was using your words.. I'm talking about having setup ccache using symlinks. gcc → ccache g++ → ccache gcc-4.0.1 → ccache [...] And having those in a PATH that comes before actual compiler. It just doesn't make much sense to do magic with CC (i.e check whether it is ccache driven or not) and then later set CC to a completely different compiler. This is the part that is/was MacOS specific. You changed the order of the blocks, so thanks for that :-) ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Hi Tor, *, On Mon, Aug 8, 2011 at 9:28 AM, Tor Lillqvist tlillqv...@novell.com wrote: If one of us (cloph) strongly feels that we should use # comments in configure.in and not dnl comments (well, keywords), and most others (?) (including me) are indifferent, why don't just change them all in one go then? I don't feel /that/ strongly about it. But if configure.in is touched, I indeed would prefer it to be changed in the touched part, and not introduced for new stuff. I wonder if there is some cases where the dnl keyword can not just be changed to # without changing the semantics of the autoconf/shell code? The section headings are enough to have in the generated configure, and those are safe to change to #. ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Monday 08 of August 2011, Norbert Thiebaud wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378 ca38f59518b018fccca8f This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX I disagree with this change, as it does not improve anything for anybody. Ccache needs an explicit setup for LO anyway, so people who have already done their setup will need to adjust for this. People who only have ccache installed but haven't done anything will get slower builds, as the default ccache cache size is vastly unsufficient for LO builds, so ccache will only create cache contents that will be eventually thrown away without any use. -- Lubos Lunak l.lu...@suse.cz ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Hi Lubos, On Mon, Aug 8, 2011 at 11:33 AM, Lubos Lunak l.lu...@suse.cz wrote: On Monday 08 of August 2011, Norbert Thiebaud wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378 ca38f59518b018fccca8f This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX I disagree with this change, as it does not improve anything for anybody. Ccache needs an explicit setup for LO anyway, No, no specific setup necessary, using the symlinks or exporting CC/CXX is standard way. so people who have already done their setup will need to adjust for this. People who only have ccache installed but haven't done anything will get slower builds, as the default ccache cache size is vastly unsufficient for LO builds, Not true either. Default ccache size is 1GB, and that is enough when you focus on one branch. (a build uses less than 500MB) so ccache will only create cache contents that will be eventually thrown away without any use. At least this is not the case. But of course compiling LO with ccache might throw out your other project's content, so I understand your point. As I have ccache setup using the symlink approach on all my systems, I don't care... (but I just realized that --disable-ccache doesn't actually disable ccache when using symlinks - you would need to set CCACHE_DISABLE in this case) ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Monday 08 of August 2011, Christian Lohmaier wrote: On Mon, Aug 8, 2011 at 11:33 AM, Lubos Lunak l.lu...@suse.cz wrote: On Monday 08 of August 2011, Norbert Thiebaud wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707 378 ca38f59518b018fccca8f This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX I disagree with this change, as it does not improve anything for anybody. Ccache needs an explicit setup for LO anyway, No, no specific setup necessary, using the symlinks or exporting CC/CXX is standard way. By setup I meant increasing the ccache size, although I didn't realize that LO build might fit if one builds without any debug info. so people who have already done their setup will need to adjust for this. People who only have ccache installed but haven't done anything will get slower builds, as the default ccache cache size is vastly unsufficient for LO builds, Not true either. Default ccache size is 1GB, and that is enough when you focus on one branch. (a build uses less than 500MB) This is not true if the build uses any kind of debug info, and I would find it a really nasty surprise that one would need to explicitly adjust ccache when enabling debug build. I think this feature is trying to be way too clever. -- Lubos Lunak l.lu...@suse.cz ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
I think this feature is trying to be way too clever. I think I agree with Lubos here. Better to let those who know what ccache is, and who think/know they will benefit from it, decide themselves whether to use it, and exactly how. --tml ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Mon, Aug 8, 2011 at 1:21 AM, Norbert Thiebaud nthieb...@gmail.com wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378ca38f59518b018fccca8f Note on the Mac-OSX case: The previous check was not to completely disable ccache, but only to check whether --ccache-skip can be used safely. Even with unsuitable versions (like unpatched 2.4 or 3.0) you could use ccache, it was just that objective-c(++) was not cached. This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX The latter is not done for the Mac OSX case when gcc 4.0 is forced, or am I overlooking something? I personally would prefer if the heading-comments would use # and not dnl - any reason why you're using dnl and thus remove any hints on the structure of the checks from the generated configure? ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Sun, Aug 7, 2011 at 6:55 PM, Christian Lohmaier lohmaier+libreoff...@googlemail.com wrote: On Mon, Aug 8, 2011 at 1:21 AM, Norbert Thiebaud nthieb...@gmail.com wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378ca38f59518b018fccca8f Note on the Mac-OSX case: The previous check was not to completely disable ccache, but only to check whether --ccache-skip can be used safely. Even with unsuitable versions (like unpatched 2.4 or 3.0) you could use ccache, it was just that objective-c(++) was not cached. humm... I need to take a closer look... Is that what the USE_CCACHE was for ? This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX The latter is not done for the Mac OSX case when gcc 4.0 is forced, or am I overlooking something? if you 'forced' gcc 4.0 via CC= then no there is no automatic ccache if 4.0 is 'forced' because you did not specify CC and the gcc picked-up is too recent, then yes ccache is used if allowed: + if test $_os = Darwin -a $with_macosx_sdk = 10.4 -a $_gpp_majmin -ge 401 ; then + if test -z $save_CXX -a -x $GCC_HOME/bin/g++-4.0 ; then + if test -z $CCACHE ; then + export CXX=$GCC_HOME/bin/g++-4.0 + else + export CXX=$CCACHE $GCC_HOME/bin/g++-4.0 + fi I personally would prefer if the heading-comments would use # and not dnl - any reason why you're using dnl and thus remove any hints on the structure of the checks from the generated configure? Just mimetic behavior... I have no preference either way Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
Hi *, On Mon, Aug 8, 2011 at 2:08 AM, Norbert Thiebaud nthieb...@gmail.com wrote: On Sun, Aug 7, 2011 at 6:55 PM, Christian Lohmaier lohmaier+libreoff...@googlemail.com wrote: On Mon, Aug 8, 2011 at 1:21 AM, Norbert Thiebaud nthieb...@gmail.com wrote: I've pushed http://cgit.freedesktop.org/libreoffice/core/commit/?id=57cf026739a3d707378ca38f59518b018fccca8f Note on the Mac-OSX case: The previous check was not to completely disable ccache, but only to check whether --ccache-skip can be used safely. Even with unsuitable versions (like unpatched 2.4 or 3.0) you could use ccache, it was just that objective-c(++) was not cached. humm... I need to take a closer look... Is that what the USE_CCACHE was for ? Yes, that also was the only part where this was evaluated. Current versions of ccache do support the objective-c++ flags, so there is no need to use --ccache-skip anymore. The problem was ccache did not support objectve-c++ flags, and had a argument-corruption bug when --ccache-skip was used. Hence the need for the check for a patched version that identifies itself as version 2.4_OOo. Its fine to drop this one and also remove the use of --ccache-skip in the mac-makefile. This try to automatically use ccache when present, unless CC= or CXX= was specified by the user or --disable-ccache was passed to autogen This also detect if CC or CXX is already an alias for ccache, in which case ccache is not prefixed to CC and/or CXX The latter is not done for the Mac OSX case when gcc 4.0 is forced, or am I overlooking something? if you 'forced' gcc 4.0 via CC= then no there is no automatic ccache if 4.0 is 'forced' because you did not specify CC and the gcc picked-up is too recent, then yes ccache is used if allowed: That's not the point is it is if it's already an alias for ccache, it won't be prefixed with an additional ccace And that's not the case. CC is checked for ccache and possibly altered, then gcc version is checked for mac and it just disregards everything and sets CC no matter of the previous check. Same for CXX. the ccache handling should occur *after* modifying CC/CXX, at least that would me more logical. I personally would prefer if the heading-comments would use # and not dnl - any reason why you're using dnl and thus remove any hints on the structure of the checks from the generated configure? Just mimetic behavior... I have no preference either way OK, please use my preference in future then :-) ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] ccache support
On Sun, Aug 7, 2011 at 7:21 PM, Christian Lohmaier lohmaier+libreoff...@googlemail.com wrote: Hi *, That's not the point is it is if it's already an alias for ccache, it won't be prefixed with an additional ccace And that's not the case. CC is checked for ccache and possibly altered, then gcc version is checked for mac and it just disregards everything and sets CC no matter of the previous check. Same for CXX. the ccache handling should occur *after* modifying CC/CXX, at least that would me more logical. I'm not sure I'm following: are you talking of the case where someone, on MacOS, has set-up ccache to masquerade as the compiler ? Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice