Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On 3 December 2010 11:29, Justin Lebar wrote: >>> Even on a ccache *hit* both copies of the .o file wind up occupying >>> buffer cache space, because the ccached .o is read from disk [paging >>> it in] in order to write the .o file to the build output directory. >>> On a ccache miss the copy runs the other direction but you still wind >>> up with both sets of pages in the buffer cache. >> >> In the hit case I would have thought that the .o file you read would >> still create less memory pressure than the working memory of running >> the real compiler on that file? Perhaps the difference is that the >> kernel knows that when the compiler exits, its anonymous pages can be >> thrown away, whereas it doesn't know which .o file it ought to retain. >> So perhaps madvise might help. (Just speculating.) > > I'm curious about this. I guess you'd madvise to tell the kernel that > the .o you just wrote shouldn't be cached? But presumably it should > be, because you're going to link your program. > > Alternatively, you could madvise and tell the kernel not to cache the > .o file from ccache's cache. But if you re-compile, you want ccache's > cache to be in memory. I meant the latter. It seems like the common case is that you're more likely to link next than to recompile the same source. I don't know if this would actually work. It may be that madvise is too weak to make a difference, or it may be it's so strong that it causes potentially useful things to be thrown out too soon. For that matter there are vm tuning knobs too. Perhaps this is getting too far into futzing. I'm not going to do it myself because I don't build such huge trees at the moment. But it's interesting to contemplate. -- Martin ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
>> Even on a ccache *hit* both copies of the .o file wind up occupying >> buffer cache space, because the ccached .o is read from disk [paging >> it in] in order to write the .o file to the build output directory. >> On a ccache miss the copy runs the other direction but you still wind >> up with both sets of pages in the buffer cache. > > In the hit case I would have thought that the .o file you read would > still create less memory pressure than the working memory of running > the real compiler on that file? Perhaps the difference is that the > kernel knows that when the compiler exits, its anonymous pages can be > thrown away, whereas it doesn't know which .o file it ought to retain. > So perhaps madvise might help. (Just speculating.) I'm curious about this. I guess you'd madvise to tell the kernel that the .o you just wrote shouldn't be cached? But presumably it should be, because you're going to link your program. Alternatively, you could madvise and tell the kernel not to cache the .o file from ccache's cache. But if you re-compile, you want ccache's cache to be in memory. I'm not sure how one might win here without hardlinking. -Justin On Thu, Dec 2, 2010 at 4:24 PM, Martin Pool wrote: > On 3 December 2010 03:42, Christopher Tate wrote: >>> I'd love to know whether you also tried distcc for it, and if so what >>> happened or what went wrong. (Obviously it can only help for the >>> C/C++ phases.) >> >> distcc can certainly help a great deal. For us, it's a bit >> problematic to use because more than half of our total build is >> non-C/C++ that depends on the C/C++ targets [e.g. Java-language >> modules that have partially native implementations], > > ... and you suspect that the Makefile dependencies are not solid > enough to safely do a parallel build? > >> plus we have a >> highly heterogeneous set of build machines: both Mac hosts and Linux, >> not all the same distro of Linux, etc. The inclusion of Macs in >> particular makes distcc more of a pain to get up and running cleanly. > > That can certainly be a problem. > >>> I'm just trying to understand how this happens. Is it that when >>> ccache misses it writes out an object file both to the cache directory >>> and into the build directory, and both will be in the buffer cache? >>> So it's not so much they're paged in, but they are dirtied in memory >>> and will still be held there. >> >> Even on a ccache *hit* both copies of the .o file wind up occupying >> buffer cache space, because the ccached .o is read from disk [paging >> it in] in order to write the .o file to the build output directory. >> On a ccache miss the copy runs the other direction but you still wind >> up with both sets of pages in the buffer cache. > > In the hit case I would have thought that the .o file you read would > still create less memory pressure than the working memory of running > the real compiler on that file? Perhaps the difference is that the > kernel knows that when the compiler exits, its anonymous pages can be > thrown away, whereas it doesn't know which .o file it ought to retain. > So perhaps madvise might help. (Just speculating.) > > -- > Martin > ___ > ccache mailing list > ccache@lists.samba.org > https://lists.samba.org/mailman/listinfo/ccache > ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On 3 December 2010 03:42, Christopher Tate wrote: >> I'd love to know whether you also tried distcc for it, and if so what >> happened or what went wrong. (Obviously it can only help for the >> C/C++ phases.) > > distcc can certainly help a great deal. For us, it's a bit > problematic to use because more than half of our total build is > non-C/C++ that depends on the C/C++ targets [e.g. Java-language > modules that have partially native implementations], ... and you suspect that the Makefile dependencies are not solid enough to safely do a parallel build? > plus we have a > highly heterogeneous set of build machines: both Mac hosts and Linux, > not all the same distro of Linux, etc. The inclusion of Macs in > particular makes distcc more of a pain to get up and running cleanly. That can certainly be a problem. >> I'm just trying to understand how this happens. Is it that when >> ccache misses it writes out an object file both to the cache directory >> and into the build directory, and both will be in the buffer cache? >> So it's not so much they're paged in, but they are dirtied in memory >> and will still be held there. > > Even on a ccache *hit* both copies of the .o file wind up occupying > buffer cache space, because the ccached .o is read from disk [paging > it in] in order to write the .o file to the build output directory. > On a ccache miss the copy runs the other direction but you still wind > up with both sets of pages in the buffer cache. In the hit case I would have thought that the .o file you read would still create less memory pressure than the working memory of running the real compiler on that file? Perhaps the difference is that the kernel knows that when the compiler exits, its anonymous pages can be thrown away, whereas it doesn't know which .o file it ought to retain. So perhaps madvise might help. (Just speculating.) -- Martin ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On Wed, 2010-12-01 at 21:47 -0500, Paul Smith wrote: > Now I'm on to my next problem. In order to get this to happen I have > to set CCACHE_BASEDIR to strip off the workspace directory prefix, so > that the per-workspace filenames are not embedded in the cache. This > works (see above), however the result is not so nice. Ugh. I lied. Actually GDB handles this just fine with no special instruction; I had a problem on my test server (and then I misunderstood how the GDB substitute-path feature worked). This works because GDB remembers not only the path of the source file, but also the working directory when the compile happened: (gdb) info source Current source file is ../../../src/subdir/foo/foo.c Compilation directory is /path/to/ONE/obj/subdir/foo Source language is c. Compiled with DWARF 2 debugging format. Does not include preprocessor macro info. So GDB intelligently notices that the source file path is relative and appends it to the compilation directory, and viola! [1] I _think_ I'm all set now. I'll check back if more issues surface. Cheers! - [1] sic ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On Wed, Dec 1, 2010 at 9:00 PM, Martin Pool wrote: > On 11 November 2010 10:56, Christopher Tate wrote: >> I don't want to rain on peoples' parade here, because ccache is a >> great product that has real benefits, but I do want to share some of >> our findings regarding the use of ccache in our very large product -- >> we were surprised by them, and you may be as well. These findings are >> specifically for *large products*. In our case, the total source code >> file size is on the order of 3 gigabytes (which includes not only >> C/C++ but also Java source files, a couple hundred thousand lines of >> makefiles, etc). It's the Android mobile phone OS, fwiw: it builds >> something like 1-2 gigabytes of .o files from C/C++ during a full >> build, and does a ton of Java compilation, resource compilation, >> Dalvik compilation, etc as well. > > I'd love to know whether you also tried distcc for it, and if so what > happened or what went wrong. (Obviously it can only help for the > C/C++ phases.) distcc can certainly help a great deal. For us, it's a bit problematic to use because more than half of our total build is non-C/C++ that depends on the C/C++ targets [e.g. Java-language modules that have partially native implementations], plus we have a highly heterogeneous set of build machines: both Mac hosts and Linux, not all the same distro of Linux, etc. The inclusion of Macs in particular makes distcc more of a pain to get up and running cleanly. >> The issue is around VM/file system buffer cache management. If you're >> using ccache, then you'll effectively be doubling the number of .o >> files that are paged into memory during the course of a build. > > I'm just trying to understand how this happens. Is it that when > ccache misses it writes out an object file both to the cache directory > and into the build directory, and both will be in the buffer cache? > So it's not so much they're paged in, but they are dirtied in memory > and will still be held there. Even on a ccache *hit* both copies of the .o file wind up occupying buffer cache space, because the ccached .o is read from disk [paging it in] in order to write the .o file to the build output directory. On a ccache miss the copy runs the other direction but you still wind up with both sets of pages in the buffer cache. > It seems like turning on compression would reduce the effect. At the expense of the extra cpu time, sure. That might be a decent tradeoff; modern cpus are getting quite fast relative to I/O. > Turning on hardlinking might eliminate it altogether, though that > could have other bad effects. Right. We haven't tried pursuing this because for other reasons the marginal returns are still pretty low, and tinkering with the build system is fraught with peril. :) -- christopher tate android framework engineer ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
>... >When ccache invokes my compiler, it runs like this: > >cd /path/to/ONE/obj/subdir/foo >my-special-gcc -c -E ../../../src/subdir/foo/foo.c > >This works, as you can see above. But, it's a serious drag from a >debugging standpoint, because now the pathname for that file in my debug >information is _based on the obj/subdir/foo directory_. > >That means that basically in gdb I have to add EVERY DIRECTORY (or at >least, one directory at every level) to my source file search path, so >that when the "../../../" is added the right thing is found. > >In other words I can't just add "/path/to/ONE" to the GDB search path, >because "/path/to/ONE/../../../src/subdir/foo/foo.c" obviously doesn't >exist. I have to add a directory at each level so that when GDB adds >the relative path to it, the right value is found. > >Ouch. >... >What I'd really like is for ccache to do this: > >cd /path/to/ONE >my-special-gcc -c -E src/subdir/foo/foo.c > >then write the .o out into /path/to/ONE/obj/subdir/foo as it would have >normally. > >I do realize it is very dangerous in general for ccache to use a >different working directory than the one it started in, because there >could be -I options etc. on the command line which are relative to the >working directory and not fully-qualified. In my particular case, >though, that's not an issue since all my paths ARE fully-qualified. > >But, I'm not sure I know how else to get this working; it seems like a >pretty serious problem. Has anyone else on the list gotten sharing >between workspaces to work properly, and if so how did you deal with >these issues? I solved this problem with a different build process and caching tool once. What we did was intercept the preprocessed code and edit the "#" directives that indicate the file name and line number to the compiler (and eventually the debugger). Specifically we specified in an environment variable the common prefix (/user_a_source_path/) then replaced it with a standard relative prefix (project/). When running the code we always had a symlink from /user_b_source_path/... to project. Files futher down just worked out, as the prefix was always the top of the build tree. This all worked wonderfully. Our current build doesn't need it, but it would still be something we'd use if it was added to ccache (maybe as a list of strings to search/replace?). -Wilson ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On 11 November 2010 10:56, Christopher Tate wrote: > I don't want to rain on peoples' parade here, because ccache is a > great product that has real benefits, but I do want to share some of > our findings regarding the use of ccache in our very large product -- > we were surprised by them, and you may be as well. These findings are > specifically for *large products*. In our case, the total source code > file size is on the order of 3 gigabytes (which includes not only > C/C++ but also Java source files, a couple hundred thousand lines of > makefiles, etc). It's the Android mobile phone OS, fwiw: it builds > something like 1-2 gigabytes of .o files from C/C++ during a full > build, and does a ton of Java compilation, resource compilation, > Dalvik compilation, etc as well. I'd love to know whether you also tried distcc for it, and if so what happened or what went wrong. (Obviously it can only help for the C/C++ phases.) Your basic point that ccache can only help if you're doing strictly unnecessary builds is quite correct; eliminating these at the build tool level may be more elegant or efficient. > The issue is around VM/file system buffer cache management. If you're > using ccache, then you'll effectively be doubling the number of .o > files that are paged into memory during the course of a build. I'm just trying to understand how this happens. Is it that when ccache misses it writes out an object file both to the cache directory and into the build directory, and both will be in the buffer cache? So it's not so much they're paged in, but they are dirtied in memory and will still be held there. Perhaps in an ideal world ccache could tell the kernel that it's less useful to hold the .ccache o file (which probably won't be used until the next rebuild) than it is to hold the working tree o file (which may be used soon by the kernel.) We could try madvise MADV_DONTNEED, at least to see what happens. It seems like turning on compression would reduce the effect. Turning on hardlinking might eliminate it altogether, though that could have other bad effects. -- Martin ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
Hi all; I got sidetracked on this for a while but I'm back at it. So, I think I have the encapsulated compiler working OK now; I am able to get almost complete sharing of objects between two workspaces; after a clean build of (just a small part of) the first workspace my cache has: cache hit (direct) 0 cache hit (preprocessed) 11 cache miss 214 files in cache 439 Then after an identical build command in my second workspace my cache has: cache hit (direct) 225 cache hit (preprocessed) 11 cache miss 214 files in cache 439 Nice! Now I'm on to my next problem. In order to get this to happen I have to set CCACHE_BASEDIR to strip off the workspace directory prefix, so that the per-workspace filenames are not embedded in the cache. This works (see above), however the result is not so nice. When my compiler is invoked it is run from an object output directory then uses the fully-qualified name of the source file/include dirs etc.; for example: cd /path/to/ONE/obj/subdir/foo my-special-gcc -c -o /path/to/ONE/obj/subdir/foo/foo.o \ /path/to/ONE/src/subdir/foo/foo.c Then I run ccache with CCACHE_BASEDIR=/path/to/ONE for example. When ccache invokes my compiler, it runs like this: cd /path/to/ONE/obj/subdir/foo my-special-gcc -c -E ../../../src/subdir/foo/foo.c This works, as you can see above. But, it's a serious drag from a debugging standpoint, because now the pathname for that file in my debug information is _based on the obj/subdir/foo directory_. That means that basically in gdb I have to add EVERY DIRECTORY (or at least, one directory at every level) to my source file search path, so that when the "../../../" is added the right thing is found. In other words I can't just add "/path/to/ONE" to the GDB search path, because "/path/to/ONE/../../../src/subdir/foo/foo.c" obviously doesn't exist. I have to add a directory at each level so that when GDB adds the relative path to it, the right value is found. Ouch. What I'd really like is for ccache to do this: cd /path/to/ONE my-special-gcc -c -E src/subdir/foo/foo.c then write the .o out into /path/to/ONE/obj/subdir/foo as it would have normally. I do realize it is very dangerous in general for ccache to use a different working directory than the one it started in, because there could be -I options etc. on the command line which are relative to the working directory and not fully-qualified. In my particular case, though, that's not an issue since all my paths ARE fully-qualified. But, I'm not sure I know how else to get this working; it seems like a pretty serious problem. Has anyone else on the list gotten sharing between workspaces to work properly, and if so how did you deal with these issues? Cheers! ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On 2010-11-11 00:56, Christopher Tate wrote: > [...] > The issue is around VM/file system buffer cache management. If you're > using ccache, then you'll effectively be doubling the number of .o > files that are paged into memory during the course of a build. > [...] By the way, have you tried setting CCACHE_HARDLINK and if so, did it make a difference? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On 2010-11-11 00:56, Christopher Tate wrote: I don't want to rain on peoples' parade here, because ccache is a great product that has real benefits, but I do want to share some of our findings regarding the use of ccache in our very large product > [...] Many thanks for sharing your findings! I have added a reference to your mail on the performance page: http://ccache.samba.org/performance.html -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On 2010-11-10 23:54, Paul Smith wrote: > [...] So, my question is, is the NAME of the compiler part of the hash as well as the mtime/size, so that ccache won't consider the second one to be a copy of the first? Yes, the compiler name is included in the hash. As you note, this is not clear from the current documentation. I will add a clarification. Thanks! Of course I can always resort to CCACHE_COMPILERCHECK='%compiler% -v' which will tell me what I want to know (that these compilers are very different). But it would be nice to avoid that overhead if possible. Yes, if it's enough for you to rely on the compiler name, then it should work to use CCACHE_COMPILERCHECK=mtime if the wrapper script changes each time a new compiler is installed. (But that will of course invalidate other compilers' cached results as well.) I added the CCACHE_COMPILERCHECK='%compiler% -v' thing exactly for the wrapper script scenario, so if it's fast enough I think it would be the best setup. If the overhead is too high, then maybe you could write a fast helper program that knows how to print suitable compiler version information on stdout and then set CCACHE_COMPILERCHECK='helper_program %compiler%'. Also if I DO go with CCACHE_COMPILERCHECK, is ONLY that output part of the hash? Or is the mtime/size for the compiler also part of the hash? mtime/size is only hashed when CCACHE_COMPILERCHECK=mtime. It would be nice for debugging/etc. if there was a way to see exactly what elements are going into the hash for a given target. That would indeed be very useful, and it's something I plan to add when implementing ability to select different hash algorithms. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
On Wed, 2010-11-10 at 15:56 -0800, Christopher Tate wrote: > Very short version: if your 'make' dependencies or equivalent are > well-written, using ccache will almost always *increase* your > incremental build times. This wasn't immediately obvious to us but > makes sense in hindsight: if your dependencies are well constructed, > then when you run 'make' it won't try to build something unless it > really has changed. We see *very* few ccache hits over time when > doing incremental builds. Yes, I believe that. Our code is probably on the cusp so I'll do some testing. We do have very good dependency recognition. However, what I was really hoping for was to gain advantage by leveraging the cache across multiple _workspaces_, not necessarily incremental builds within the same workspace. If I have 2 workspaces and I update and build one of them, then go update and build the other which shares a good portion of the same code, then my hope was I could leverage the cache to make the second build much faster by avoiding recompilation (we do have a lot of too large, pretty gnarly C++ source files that take a while to compile, especially with optimization enabled). Because we support many (too many!) parallel lines of development we're often moving workspaces around back and forth; something that could improve build times in these cases would be welcome. But we will need to measure this, to be sure, to see if the cost during the more common incremental builds pays for the savings during larger, "updated workspace" builds. Thanks for that note; I hadn't quite considered it that way! ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Stumbling blocks with ccache and embedded/encapsulated environments
I don't want to rain on peoples' parade here, because ccache is a great product that has real benefits, but I do want to share some of our findings regarding the use of ccache in our very large product -- we were surprised by them, and you may be as well. These findings are specifically for *large products*. In our case, the total source code file size is on the order of 3 gigabytes (which includes not only C/C++ but also Java source files, a couple hundred thousand lines of makefiles, etc). It's the Android mobile phone OS, fwiw: it builds something like 1-2 gigabytes of .o files from C/C++ during a full build, and does a ton of Java compilation, resource compilation, Dalvik compilation, etc as well. Very short version: if your 'make' dependencies or equivalent are well-written, using ccache will almost always *increase* your incremental build times. This wasn't immediately obvious to us but makes sense in hindsight: if your dependencies are well constructed, then when you run 'make' it won't try to build something unless it really has changed. We see *very* few ccache hits over time when doing incremental builds. Slightly longer version: if your 'make' dependencies are well-written, then using ccache will almost always increase your incremental build times. Even if your dependencies are slightly inefficient (i.e. you're getting some unnecessary compilation on a regular basis, but not tons), ccache may well still be slowing you down overall on incremental builds unless your computers have lots of RAM relative to the size of your project. It turns out that fitting the build inputs *and* outputs into the VM/filesystem buffer cache usually provides much more build-time benefit than ccache. (Unless your project is very large, you're almost certainly fitting everything in ram as a matter of course and so ccache is a fine idea.) If you're regularly doing something equivalent to a 'clean' build from make's perspective, but with a hot ccache, then ccache is solving exactly your problem and you definitely want to use it. Long answer; only applicable to very large projects: The issue is around VM/file system buffer cache management. If you're using ccache, then you'll effectively be doubling the number of .o files that are paged into memory during the course of a build. If the extra VM paging winds up pushing source code out of the VM buffer cache, then that is going to be a significant hit when your build system actually needs to build that source file -- it'll have to go to the disk for it rather that just reading it out of memory. Ideally your build machines will have enough RAM to hold in the buffer cache the entire source base, plus the entire set of .o / .class / etc intermediate build output files, *plus* the entire ccache contents for the project. That's on top of the actual memory usage of the compilers etc that run during the course of the build. As soon as things are being kicked out of the buffer cache during the course of a build, you'll take a speed hit that will more than eradicate the benefits due to using ccache. Our product may be a bit pathological in this way: a lot of the RAM-hungry tools and source files are not C/C++, which means a heavy demand on RAM in ways that ccache can't help with. The ordering of RAM in a typical build system makes things worse, in fact; the basic C/C++ compilation units during which ccache pushes source files and .o contents through the buffer cache are typically earlier during compilation, then later on in the build things like linkers run, which are RAM hungry themselves and wind up pushing the sources out of the buffer cache on memory-constrained machines. Then when you want to run an incremental build the sources have to be paged right back in from disk, and there goes your time benefit. Android's ccache footprint is one or two gigabytes(!) of .o files. We've found that on computers with less than around 20-24 gigs of RAM (!!), ccache tends to increase build times -- 12 gigs isn't enough to hold everything in the buffer cache; 24 is. Once everything is in the buffer cache, ccache's ability to avoid running gcc is a win, especially for clean builds with a hot cache. -- chris tate android framework engineer On Wed, Nov 10, 2010 at 2:54 PM, Paul Smith wrote: > Hi all; I've been considering for a long time enabling ccache support in > our build environment. However, our environment has a few unusual > aspects and I wondered if anyone had any thoughts about steps we might > take to ensure things are working properly. The documentation I've > found is just not _quite_ precise enough about exactly what is added to > the hash. > > Very briefly we have multiple "workspaces" per user, mostly stored on > their own systems. These workspaces are typically pointing to different > lines of development, and in those some files are the same and some are > different (pretty basic). What I'd like to do is have one ccache per > user per host, so that
