[ccache] ccache compression patch (was: The future of ccache)
On Thu, 7 Jan 2010 09:22:27 +1100 tridge at samba.org wrote: For example, the compression patch is badly done I think (it doesn't cope with changing the compression option - it should either use a .gz extension for compressed cache files, or recognise them by reading the header). Hm, I don't see that problem with the patch. Can you elaborate? Regardless of the CCACHE_NOCOMPRESS setting and the type of file, it uses gzdopen+gzread to read the file, and gzread works on uncompressed files too. The only problem I saw with the patch is that it breaks forward compatibility of ccache versions without the compression patch since the hash sum and therefore the file name is the same for compressed and uncompressed cache files. I think that's unfortunate, so to avoid this, I made the hash sums for compressed and uncompressed files different. A .gz extension for compressed files would of course work too, with different trade-offs. Regards, Joel
Re: [ccache] [PATCH] Always close(fd) as soon as file is mmap()'d.
On Sat, 27 Feb 2010 22:15:54 -0300 Ramiro Polla ramiro.po...@gmail.com wrote: On remember_include_file(), close() wasn't even being called. On unify_hash() and process_preprocessed_file(), it might not be closed if mmap fails. Thanks for finding this! But I think you forgot to attach your patch? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Always close(fd) as soon as file is mmap()'d.
On Sun, 28 Feb 2010 12:22:56 -0300 Ramiro Polla ramiro.po...@gmail.com wrote: On Sun, Feb 28, 2010 at 5:52 AM, Joel Rosdahl j...@rosdahl.net wrote: On Sat, 27 Feb 2010 22:15:54 -0300 Ramiro Polla ramiro.po...@gmail.com wrote: On remember_include_file(), close() wasn't even being called. On unify_hash() and process_preprocessed_file(), it might not be closed if mmap fails. Thanks for finding this! But I think you forgot to attach your patch? Hmm, gmail says it's attached. Let's try again. Thanks, got it. Apparently, the mailing list settings were quite paranoid regarding attachments and stripped your patch. I've now loosened the attachment type filter somewhat. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Release plan
Hi, I now don't have any features left on my TODO list for the next ccache release, which I plan to name 3.0. The main thing left, except more testing and fixing any bugs that are found, is some work on the documentation, and maybe some portability stuff. There are still some features that I would like to work on, though, but that I think should wait until after 3.0: - Support for a configuration file in the cache directory. I will send a mail to the list with some thoughts on this later. - Support for choosing hash algorithm (at least MD4, MD5 and SHA-1). - Support for optionally hashing the output of $CC $flag (e.g., gcc -v or gcc -dumpspecs) to identify the compiler instead of relying on mtime/size. Does this seem reasonable? And does anyone have other feature requests or semi-forks with features that could be incorporated in later ccache versions? Comments are welcome. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] ccache 3.0pre0 has been released
Hi, I'm pleased to announce ccache version 3.0pre0. I expect this prerelease (or call it beta release if you want) to not differ much from the upcoming 3.0 release functionality-wise, so I encourage everyone to try it out. And please report any bugs or misbehaviours you find to this mailing list or the bug tracker. The source archive can be found here: http://samba.org/ftp/ccache/ccache-3.0pre0.tar.bz2 http://samba.org/ftp/ccache/ccache-3.0pre0.tar.gz To find out what has changed since the previous version (2.4), please see the NEWS file: http://samba.org/ftp/ccache/ccache-3.0pre0-NEWS -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Release plan
On Sun, 28 Feb 2010 17:29:55 -0300 Ramiro Polla ramiro.po...@gmail.com wrote: I have a win32 port almost done. But since it touches *a lot* of code I'd rather wait for after 3.0 so it can receive more testing. Nice. Yes, that sounds like a good plan. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Release plan
On Sun, 28 Feb 2010 22:26:48 + William S Fulton w...@fultondesigns.co.uk wrote: I have a semi-fork of ccache for SWIG. See http://swig.svn.sourceforge.net/viewvc/swig/trunk/CCache/. It adds support for the SWIG compiler in addition to C/C++ compilers. The main difference to a traditional compiler is that SWIG can generate more than one file and ccache works equally well for it. I don't know if the changes are suitable for incorporation into the main ccache at some point - comments welcome. Yes, maybe. I have to read up on your work later. However, it would be good to port the CCACHE_VERBOSE option from this fork to ccache - it displays all the compiler invocations it makes and it is really useful for debugging ccache problems as both an end user and a developer. Thanks, this seems like a good idea. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] GPLv3
On Mon, 01 Mar 2010 23:12:45 + William S Fulton w...@fultondesigns.co.uk wrote: [...] As the copyright holders have already licensed it under GPLv2 or later (later version is the key point here) and as GPL is copyleft, anyone can take the source and license it as GPLv3 or GPLv3 or later, so there is nothing stopping you or anyone else from doing this should they so wish. Of course, it would be polite to inform the contributors and discussion on this mailing list partly goes down that route, but you don't need consent... you only need to conform to the license and that includes redistributing under a later version. I think it's the politeness part that bugs me, at least in theory, regardless of what the licenses actually imply. If I have created, say, some GPLv2+-licensed code called goodstuff.c, I would be a bit annoyed if I found the file in projectX and the file had a GPLv3 header but was otherwise unmodified, since people who stumble on goodstuff.c in projectX then can't use it (or rather: don't know they can use it) in their GPLv2-only project even though my original code was GPLv2+. If the projectX guys have changed goodstuff.c, then I wouldn't mind since they of course may decide to GPLv3(+) their changes and thereby the combination of goodstuff.c and the changes. That said, this is all theory, and I'm not debating what the licenses say, only what I think should be common sense. I don't see any problem in practise in the ccache case, so I'll switch ccache to GPLv3+, unless anyone strongly objects. Contributors, be informed! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Compression on or off by default?
Hi, Lars Gustäbel's compression patch (which will be incorporated in ccache 3.0) enables compression by default, and if you don't want compression you have to set CCACHE_NOCOMPRESS. I'm still a bit undecided about whether defaulting to compression is a good idea, though. Maybe we should be more conservative here and require CCACHE_COMPRESS to be set to enable compression instead? (Note that the question only is about the default behaviour when storing files in the cache -- ccache will still be able to read compressed and uncompressed files from the cache regardless of the CCACHE_(NO)COMPRESS setting.) The main argument I see for making compression opt-in is that hard-linking doesn't work for compressed files (where doesn't work means that ccache will fall back to copying), so if you would like to try out hard-linking, you must set both CCACHE_NOCOMPRESS and CCACHE_HARDLINK, and also build up the cache again. Or, if you currently have enabled hard-linking with ccache 2.4, you need to take the explicit action of disabling compression after an upgrade to get the previous behaviour. Another argument is maybe that disk space is cheap nowadays, and most people probably want to optimize for speed instead of disk space. On the other hand, the overhead of using compression is very small. In fact, I am unable to consistently measure any performance impact whatsoever. (Lars Gustäbel's own measurements can be found at http://gustaebel.de/lars/ccache/.) And, by compressing the cached files, more files will fit in the cache and also in the OS disk cache. Does anyone have an opinion to share about this? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Compression on or off by default?
On Wed, 3 Mar 2010 05:00:48 -0500
Mike Frysinger vapier@gmail.com wrote:
make the default a ./configure option that defaults to off
I would prefer not to make a build-time option of this, because then the
documentation has say something in line with whether files are compressed by
default depends on how ccache was built, which I think would be unfortunate. I
chose to not include the --{en,dis}able-zlib option from the original patch for
the same reason. I think it's better to strive for good defaults and few
build-time options.
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Compression on or off by default?
On Wed, 3 Mar 2010 10:39:47 + Dalton, Tom tdal...@hp.com wrote: I would say for the first release of a new feature, default it to off. That gives people a chance to 'play' with it without potentially breaking existing installations that are simply upgrading. If there are no major problems with the feature in that first release then make it a default in the next release after that (e.g. ccache-3.1). The compression feature is actually quite well-tested: it has been enabled by default in Debian's ccache packages since 2007. But I'm leaning on disabling compression by default for ccache anyway. Score so far: 4 votes (2 off-list) for default to off and 0 votes for default to on. My direct mode feature, on the other hand, is a much larger and less tested change, which I'm very well aware of. My secret plan was to try to get people to test my first announced release, and now the 3.0pre0 release, but neither release has triggered much reaction. So, maybe 3.0 will have to have direct mode disabled by default, unless I find a way to get more testers. :-) (Opinions are welcome, as usual.) -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Inline assembly .incbin directive
On Tue, 2 Mar 2010 22:21:08 + Stefan Hajnoczi stefa...@gmail.com wrote: ccache does not detect when the GNU assembler .incbin directive [1] is used to include a binary file. [...] Since this bug breaks the idea that ccache does not affect the semantics of the build, only the performance, I feel it makes sense to treat it as a bug and fix it. Ouch, that's a nasty feature from a ccache perspective. To do this right, ccache would have to parse the preprocessed source code to find __asm__ statements (and if using the direct mode, parse the raw source code, which means implementing most functionality of a preprocessor) and then parse the strings in the __asm__ sections. I think this is way beyond what's reasonable for ccache to do, unfortunately. Another way of detecting it would be to use strace or LD_PRELOAD tricks to find out which files are read by the compiler, but then ccache would be a very different program. The only reasonable way I see to tackle this is to avoid caching any source code that has a string literal containing .incbin. It would be easy to fool, and there will be a few false positives, but maybe it's good enough. On the other hand, I guess a very tiny part of ccache's users compile things with .incbin. Do you think it's worth the effort to try to detect this or is it enough to just document it as limitation? (It would be easy to make a slightly better work-around available than setting CCACHE_DISABLE, though: Introduce a CCACHE_EXTRA_FILES_TO_HASH variable containing paths to files whose content ccache should include in the hash.) -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Temporary Files vs NFS
On 2010-04-13 19:56, DATACOM - Samuel Lucas wrote: I'm considering using a shared cache over NFS, but I'm a bit concerned about creating the temporary files over NFS. Wouldn't the network become a performance bottleneck? I haven't done any measurements of NFS-based caches myself, but it sounds probable that it will affect performance. [...] If we allow temporary files in local machine, would it create all temporary files (.i, etc) locally and then transfer in the network only the hash, speeding up the process? In ccache 2.4, CCACHE_TEMPDIR needs to be on the same file system as CCACHE_DIR, but in 3.0pre0 that requirement is relaxed. In fact, it just occurred to me that CCACHE_TEMPDIR isn't needed anymore since it's only used to store truly temporary files that aren't moved into the cache. Unless I find any problem, I will remove the CCACHE_TEMPDIR setting and just use mkstemp(3) to create temporary files. Please try ccache 3.0pre0 and set CCACHE_TEMPDIR to some local directory like /tmp and see if that improves performance. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Question about features and release date
On 2010-04-14 03:58, Yoda, Kiwamu wrote: Q1. In future, will ccache support other compilers besides gcc? Especially RealView. I have no personal interest in working on supporting other compilers than GCC, but if anyone wants to, please go ahead. But keep in mind that to do this acceptably, a lot of refactoring needs to be made since the current implementation is a bit hard-wired. There's also another project, gocache, which has the goal of supporting other compilers than GCC: http://sourceforge.net/projects/gocache/ Q2. In future, will ccache support another language? Especially Java. Not likely. Q3. When do you intend to release 3.0? Unfortunately, I have no other answer than when it's ready. :-) I haven't been able to work much on ccache lately, but there's still some things I want to do before releasing. There will be at least one more prerelease before 3.0. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Add support for the -x GCC switch
On 2010-04-16 22:11, Dirk Best wrote: this patch adds support for the -x GCC switch that allows one to specify the language of the source file independently of the file extension. Sorry, but I guess the mailing list software has eaten your patch (or you forgot to attach it). I've loosened the mailing list filter rules now, so please resend. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Add support for the -x GCC switch
On 2010-04-16 22:11, Dirk Best wrote: this patch adds support for the -x GCC switch that allows one to specify the language of the source file independently of the file extension. Thanks! I see that you reuse the extensions array to find out the language, but that will only handle the c and c++ cases. It would be nice to handle c-cpp-output and c++-cpp-output as well. Another minor problem is that ccache will accept things like -x cc, while gcc will not. Also, it seems that GCC accepts both -x language and -xlanguage forms; we should handle this as well. I think it would be better to have a separate array to map language to source code extension: c - c c-cpp-output - i c++ - c++ c++-cpp-output - ii otherwise - stats_update(STATS_UNSUPPORTED); failed(); And then we can use the extensions array to find out i_extension. What do you think? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Temporary Files vs NFS
On 2010-04-14 18:48, Joel Rosdahl wrote: Unless I find any problem, I will remove the CCACHE_TEMPDIR setting and just use mkstemp(3) to create temporary files. After thinking about this, I will keep CCACHE_TEMPDIR. While ccache would perform better by default for caches on NFS if /tmp were to be used for temporary files, this would also open up for attacks since the temporary files need to be accessed by name (see for instance http://seclists.org/bugtraq/2002/Dec/210). Very unlikely, yes, but still a potential security problem not worth ignoring. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Add support for the -x GCC switch
On 2010-04-19 10:11, Dirk Best wrote: I see that you reuse the extensions array to find out the language, but that will only handle the c and c++ cases. It would be nice to handle c-cpp-output and c++-cpp-output as well. I haven't really looked into that yet, as the project I added support for doesn't require it. Is there any special handling required for those or can we handle this with an entry in the extension table? No entry is neeed in the extension table if we add a separate language array. [...] I'll prepare an updated patch that takes care about some of those issues. Excellent! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Corrupt objects from three colliding compiles
On 2010-05-06 19:14, Wilson Snyder wrote: [...] [1273161404.432174,2424 ] === 2010-05-06 11:56:44 === [1273161404.432636,2424 ] Hostname: maserati [1273161404.432643,2424 ] Cwd: /nfs/compiling/dir02 [1273161404.432646,2424 ] Source file: INV.c [1273161404.432648,2424 ] Object file: INV.o [1273161404.432657,2424 ] Trying direct lookup [1273161404.439730,2424 ] Looking for object file hash in /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest [1273161404.440126,2424 ] Did not find object file hash in manifest [1273161404.440134,2424 ] Running preprocessor [1273161404.469188,2084 ] === 2010-05-06 11:56:44 === [1273161404.469515,2084 ] Hostname: bird07 [1273161404.469519,2084 ] Cwd: /nfs/compiling/dir03 [1273161404.469521,2084 ] Source file: INV.c [1273161404.469523,2084 ] Object file: INV.o [1273161404.469532,2084 ] Trying direct lookup [1273161404.471093,2084 ] Looking for object file hash in /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest [1273161404.475427,2084 ] Did not find object file hash in manifest [1273161404.475434,2084 ] Running preprocessor [1273161404.487509,2424 ] Got object file hash from preprocessor [1273161404.492494,2424 ] Object file /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o not in cache Job 2424 doesn't see the object file in the cache... [1273161404.492522,2424 ] Running real compiler ...and decides to create it. [1273161404.511123,21936] === 2010-05-06 11:56:44 === [1273161404.511506,21936] Hostname: bat05 [1273161404.511510,21936] Cwd: /nfs/compiling/dir04 [1273161404.511513,21936] Source file: INV.c [1273161404.511515,21936] Object file: INV.o [1273161404.511523,21936] Trying direct lookup [1273161404.514058,21936] Looking for object file hash in /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest [1273161404.517648,21936] Did not find object file hash in manifest [1273161404.517654,21936] Running preprocessor [1273161404.523658,2084 ] Got object file hash from preprocessor [1273161404.542730,2084 ] Created INV.o from /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o But here, job 2084 sees the object file in the cache ~34 ms before the object file was created... [1273161404.564688,21936] Got object file hash from preprocessor [1273161404.569955,2084 ] Added object file hash to /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest [1273161404.570387,2084 ] Succeded getting cached result [1273161404.576412,2424 ] Stored in cache: /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o ...by job 2424 here. Maybe the clocks are more than 12 ms apart? [1273161404.588776,21936] Created INV.o from /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o ***BAD OBJECT*** [1273161404.596098,2424 ] Created INV.o from /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o [1273161404.630619,21936] Added object file hash to /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest [1273161404.634864,21936] Succeded getting cached result [1273161404.638443,2424 ] Added object file hash to /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest It looks like the third job's read of the cache is stuck in between the first's write and the second's. Note that times may be off by 8ms, this would put the bad object read right on top of the object update by the second compile when it hits the NFS server. Since renames should be atomic on NFS, I don't see any problem here, but I'm by no means an NFS expert. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] ccache 3.0pre1 has been released
Hi,
I'm pleased to announce ccache version 3.0pre1. I expect this prerelease
(or call it beta release if you want) to be very close to the upcoming
3.0 release functionality-wise, so I encourage everyone to try it out.
And please report any bugs or misbehaviours you find to this mailing
list or the bug tracker.
The source archive can be found here:
http://samba.org/ftp/ccache/ccache-3.0pre1.tar.bz2
http://samba.org/ftp/ccache/ccache-3.0pre1.tar.gz
To find out what has changed since the previous stable version (2.4),
please see the NEWS file:
http://ccache.samba.org/news-3.0pre1.html
You can also read the HTML version of the manual:
http://ccache.samba.org/manual-3.0pre1.html
Here's a list of changes between 3.0pre0 and 3.0pre1:
Major changes:
- ccache is now GPLv3+ instead of GPLv2+.
- Compression of object files in the cache is now off by default.
Enable by setting CCACHE_COMPRESSION.
- CCACHE_BASEDIR is now off by default since it affects the compiler
semantics too much.
- The way the hash sums are calculated has changed again. Sorry about
that.
New features and improvements:
- Added CCACHE_VERBOSE option.
- Added CCACHE_EXTRAFILES option.
- Improved debug logging.
- The compiler options -imultilib, -iquote, -isysroot, -nostdinc,
-nostdinc++ and -U are no longer hashed in preprocessor mode.
Bug fixes:
- Fixed caching of empty source files in direct mode.
- Avoid storing file hash in manifest in read-only mode.
- Don't compile the preprocessed source code when using -g3 (bug
#7190).
- The compiler option -finput-charset is now handled correctly (bug
#7292).
- umask is now set earlier since ccache_main() may create
files/directories.
- Corrected implementation of strndup() replacement.
- Improved handling of -Wp,* compiler options.
- Various code cleanups.
- The compiler options -Xassembler, -b, -G and -V are now correctly
recognized as taking an argument.
- Computation of the hash sum has been improved to decrease the risk
of collisions.
- Code that uses (or rather: potentially uses) __{DATE,FILE,TIME}__
macros is handled correctly in direct mode.
- Bail out early on the compiler option -arch since it's too hard to
handle.
- Include hostname in temporary file names to avoid potential NFS
race.
- The temporary object file is now renamed (instead of copy+delete) if
possible.
- Improved error checking when reading compressed files from the
cache.
- Improved detection of recursive invocation (bug #7411).
Other:
- Rewrote the manual and other documents into AsciiDoc format.
- Documented that the .incbin inline assembler directive causes
problems.
- Improved documentation of how the direct mode works.
- Added Compiling in different directories section in the manual.
- Added Sharing a cache on NFS section in the manual.
- Improved benchmarking script.
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] amd64 buildworld (FreeBSD 8.0p2) consistently fails with ccache
On 2010-05-10 03:24, Christer Edwards wrote: I've enabled logging and I would attach the logfile, but it is 12M total size. I've included the tail end, just leading up to the failure: [...] [78047] Running real compiler [78047] Compiler gave exit status 1 This build had no CFLAGS or CXXFLAGS and exited at the same point. What other information can I provide? The most important thing is to find out the exact command line used when the compilation failed. Your build system doesn't seem to print that, so I think you either should try to increase verbosity in some way or upgrade to ccache 3.0pre1, which was just released. 3.0pre1 logs the executed compiler commands to CCACHE_LOGFILE, so it would be enough to get the log snippet of the failed compilation. Possibly, the compiled source code could also be of interest. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] amd64 buildworld (FreeBSD 8.0p2) consistently fails with ccache
On 2010-05-11 01:38, Christer Edwards wrote: I've got another failure (same place) on 3.0pre1.Below are the details: I executed 'ccache make buildworld' at /usr/src. Putting ccache before make doesn't gain anything (but doesn't do any harm either, except that an error counter in ccache's statistics will be increased). [...] [2010-05-10T17:30:38.825974 53291] Running preprocessor [2010-05-10T17:30:38.827775 53292] Executing /usr/obj/usr/src/tmp/usr/bin/cc -O2 -pipe -I/usr/src/lib/csu/i386-elf/../common -I/usr/src/lib/csu/i386-elf/../../libc/include -std=gnu99 -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wcast-align -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wno-pointer-sign -c -E /usr/src/lib/csu/i386-elf/crt1.c [...] [2010-05-10T17:30:38.853797 53291] Running real compiler [2010-05-10T17:30:38.855228 53294] Executing /usr/obj/usr/src/tmp/usr/bin/cc -O2 -pipe -I/usr/src/lib/csu/i386-elf/../common -I/usr/src/lib/csu/i386-elf/../../libc/include -std=gnu99 -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wcast-align -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wno-pointer-sign -c -o /usr/.ccache/7/e/00e1bb753cccab0b3367e6287435a8-23935.o.tmp.mouzone.part.net.53291 /usr/.ccache/tmp/crt1.tmp.mouzone.part.net.53291.i [2010-05-10T17:30:38.903089 53291] Compiler gave exit status 1 [2010-05-10T17:30:38.903377 53291] Copying /usr/.ccache/7/e/00e1bb753cccab0b3367e6287435a8-23935.o.tmp.mouzone.part.net.53291 to crt1.o (uncompressed) [2010-05-10T17:30:38.906771 53291] open error: No such file or directory Nothing strange there, I'm afraid. Just to make it clear: The build works without ccache, right? Does it work to run the first command in the log above (after Executing), redirect the output to crt1.i and then run the second command with -o crt1.o crt1.i? If it doesn't work, it's probably a compiler problem since it can't compile its own preprocessed output. A workaround for that is to set CCACHE_CPP2 (preferably only when building the troublesome file). If it works when you run the two commands manually but not when ccache runs them, then I'm out of guesses. By the way, I found another thread about the same problem here: http://www.pubbs.net/200912/freebsd/12812-8-stable-broken-on-amd64.html -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Some patches for Xcode support
On 2010-05-25 00:58, Bernhard Bauer wrote: Turns out, the -x option was overriding the language auto-detection based on the filename. I ended up rewriting the -x patch similar to the way you suggested to Dirk, so now it supports properly changing the language for preprocessed files, using a table mapping from file extension to language, and one mapping from language to preprocessed language (and it incorporates the first patch from my original mail). You can find the patch at http://github.com/sheepmaster/ccache/commits/gcc_x . Thanks! Applied with some additional polishing. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Some patches for Xcode support
On 2010-06-11 16:33, Bernhard Bauer wrote: Also, I added the -x option for compiling preprocessed code again if there was an -x option on the original command line. Otherwise, when compiling Objective-C code on a different machine, distcc wouldn't recognize the code as preprocessed Objective-C. Is this because distcc doesn't know about .mi/.mii being precompiled code? Why is this a problem? And in that case, don't you need to add -x unconditionally (not only if explicit_language has been set)? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Some patches for Xcode support
On 2010-06-13 20:22, Bernhard Bauer wrote: On Sun, Jun 13, 2010 at 18:38, Joel Rosdahl j...@rosdahl.net wrote: [...] #!/bin/sh CCACHE_PREFIX=distcc exec ccache $@ That should work, provided PATH is modified to exclude /X, otherwise ccache would call this script again when calling distcc (because it's not a symlink to ccache). Right. Or use an absolute path to the real distcc in CCACHE_PREFIX. [...] I'll try to find out if it helps performance, and come back complaining to you if it doesn't ;-) :-) Yes, it could be hard to measure any difference. I had distcc failing complaining about a compiler option that's specific to Objective-C, but only when running the job on another host. My guess is that distcc (or at least the old version Apple is shipping) doesn't recognize .mi/.mii as being Objective-C files, so on the other host it writes them to a temp file ending in .i/.ii before calling gcc, then gcc assumes they're plain c/c++ files. But yeah, in that case I probably need -x always, I just didn't want to muck around with the command line options any more than necessary. I see. distcc 2.18.3 (the oldest release I can find on distcc.org) seems to know about .mi/.mii. Do you think it's acceptable to upgrade to a newer distcc to fix this? It would be nice to avoid having a workaround for older distcc versions. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Some patches for Xcode support
On 2010-06-14 19:13, Bernhard Bauer wrote: The bug is actually more subtle: distcc has global variables indicating whether whether it has seen an -x option and the last value of its argument, and forgets to clear them between running multiple jobs on the server, so it assumes an incorrect language override when no -x option is present. This bug is in the Apple-patched distcc version that's shipped with the 10.5 Developer Tools (source at http://opensource.apple.com/tarballs/distcc/distcc-1201.tar.gz), so it's -- sadly -- not a really old version. I see. Thanks for the investigation. In the light of this, I think that the work-around is acceptable, so I've committed a variant of your patch now. Please try it out! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] globally unique hashes
On 2010-06-20 17:26, Reuben Hawkins wrote: I was wondering if the hash values generated by ccache are globally unique. I was thinking about adding a multicast feature where after not finding a cached object file in the local cache, but before trying to compile the file, ccache will multicast to it's neighbors the hash value to see if another local machine has the cached object file corresponding the the hash value. For this to work, however, I would think the hash values MUST be globally unique across platforms and compilers. Is that the case? The hash sum currently doesn't include any information about the platform and also nothing about the compiler except the mtime and size of the compiler binary. Maybe hashing some of the data printed by the -dump* options to GCC would be enough for your use case. You could maybe also think about how your idea compares to sharing the cache over NFS. a) I've been assigned a 5 day project to do whatever I'm interested in Nice! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache on windows
On 2010-06-23 22:18, Ramiro Polla wrote: I sent an e-mail to the list a couple of days ago regarding a github fork I had created with those patches and pointing out some initial uncontroversial patches that could get applied early but got no answer yet. [...] Don't worry, your recent mail is still marked as follow up in my mailbox, so it's not lost. :-) I've just been busy with other things. Stay tuned. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Win32 support.
On Sun, Jun 20, 2010 at 7:07 PM, Ramiro Polla ramiro.po...@gmail.com wrote: Now that 3.0 has been released, do you intend to keep a branch of backports only (for 3.0.1 or something) or can we start working on the bigger changes required for win32 support? I will use the branch maint for changes that will end up in the next bugfix release (now 3.0.1) and master for changes that will end up in the next feature release (now 3.1). If needed, I also plan to publish feature branches called dev/X (where X is a feature name) for larger features that may be unstable, experimental or whose target release hasn't been decided yet. I've set up a fork on github at http://github.com/ramiropolla/ccache Excellent! Thanks. General: Please don't put a period in the commit summary (i.e., first line of the commit message). There are some uncontroversial patches that I think could get reviewed/applied right away http://github.com/ramiropolla/ccache/commit/60446f19421f4882e155e6533586935f131ed29a almost cosmetics with the current code... Applied. http://github.com/ramiropolla/ccache/commit/6bafa334a18c4bf13285bde717fe05383cdd0189 there already is some code using O_BINARY Applied. http://github.com/ramiropolla/ccache/commit/159ac29abcb771a9a77e8d81753c67b4d6b47adc this is already checked in configure Applied. http://github.com/ramiropolla/ccache/commit/fe62388c8e22a72de293ac0dbd28107b2e40421f this seems very weird since it links fine on Linux both with and without that file being compiled, but on mingw32 it's already provided by libmingwex.a and therefore doesn't link. Applied with a minor tweak. I also applied Split find_executable_in_path() out of find_executable(). I haven't looked at the others in detail yet. I'll get back to you when I have more time. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache and precompiled headers
Hi,
On 2010-06-21 13:09, Tor Arne Vestbø wrote:
I ran into a problem where ccache would log Preprocessor gave exit
status 1 and fail to cache builds. Closer inspection revealed that
this was due to the use of precompiled headers though an -include
argument to gcc. When ccache then tried to run the preprocessor on the
source file gcc reported file not found for the precompiled header.
Aparenly gcc appends .gch and some other magic to the filename
during build but not during preprocessing.
The solution seems to be to add the -fpch-preprocess flag, which will
add a custom #pragma include to the preprocessed output, see:
http://gcc.gnu.org/onlinedocs/gcc-4.5.0/gcc/Preprocessor-Options.html#index-fpch_002dpreprocess-877
I came to mostly the same conclusion some months ago but decided to
postpone thinking about support for preprocessed headers until after the
3.0 release. So, now's a good time to continue. :-)
The question is, is it safe to use ccache with precompiled headers at all?
If so, would a patch like this work, basically adding the flag if a
pch include is detected, and making sure it's included and hashed in
the direct mode manifest's list of includes?
See comments below.
[...]
@@ -1557,6 +1562,12 @@ static void process_args(int argc, char **argv, ARGS
**preprocessor_args,
}
args_add(stripped_args, argv[i]);
+ if (strcmp(argv[i], -include)
+strstr(argv[i+1],
pch)) {
+ found_pch = 1;
+ cc_log(Found precompiled
header: %s,
+ argv[i+1]);
+ }
But this means that you have to include pch in name of the precompiled
header file, which is quite limiting. There's also the case when using
'#include preprocessed.h' in the source code to include the
precompiled header instead of using the -include compiler option. Also,
if the name contains pch when it's not a precompiled header,
-fpch-preprocess will be added anyway, and that will make the
preprocessor fail if the compiler isn't GCC 4.0 or higher.
I don't see how ccache in a good way could figure out if the
-fpch-preprocess option should be added or not, so the easy option is
this: Just document that if you want ccache caching to work with
precompiled headers, you have make sure to add the -fpch-preprocess
option to the compilation command. That may be done by altering the
build system (adding -fpch-preprocess to CPPFLAGS or similar), but it
could also be done by adding -fpch-preprocess to CCACHE_CPPFLAGS if we
introduce such a variable. We also need the first part of your patch, of
course.
Another way would be to ask the compiler if it supports the
-fpch-preprocess option and, if so, always add it. This could be done by
checking the status code of running something like compiler
-fpch-preprocess -x c -c /dev/null -o /dev/null. The result of this
command could be cached somewhere in CCACHE_DIR (keyed with the
compiler's mtime) so that there's no need to run it more than once.
(Storing other types of information about the compiler may be useful for
other things as well, like doing compiler-specific handling of flags or
maybe enabling direct mode only for compilers known to work with it.)
There's also at least one more subtle thing that needs to be addressed:
the __TIME__ macro in direct mode. As can be read in the Performance
section in the manual
(http://ccache.samba.org/manual.html#_performance), the direct mode is
turned off when __TIME__ is present in the source code. But when
precompiled headers are being used, ccache can't easily read the code
that was included, so it can't tell whether __TIME__ is potentially
being used and thus has to turn off direct mode as a safety measure
(unless CCACHE_SLOPPY=time_macros). This also needs to be implemented.
What do you think?
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] globally unique hashes
On 2010-06-24 00:58, Reuben Hawkins wrote: On Wed, Jun 23, 2010 at 1:05 PM, Joel Rosdahl j...@rosdahl.net wrote: If your version become unknown, then I guess you neither built an official release nor from a Git tree? [...] $ echo `git describe --dirty` fatal: No names found, cannot describe anything. Then I think you don't have any tags in your repo (check with git tag). How did you create your local repo? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Win32 support.
On 2010-06-25 11:01, Joel Rosdahl wrote:
I haven't looked at the others in detail yet. I'll get back to you when I
have more time.
Introduce and use x_fmmap(): Nice cleanup in general. A comment with
some documention of x_fmmap's semantics and parameters would be nice.
And maybe remove trailing space from last argument when calling x_fmmap
and add a space in the format strings in x_fmmap instead?
Don't dup fd for zlib, and gzclose() gzdopen()d file: The gzclose() is
fine, but I'm not sure about the removal of dup() since that will make
copy_fd close fd_in because gzclose() will close it.
Don't overwrite a file that is currently opened: I agree that it
ideally should be done only on Windows, but I think it's OK since it's a
rare case. Applied.
Close manifest files before moving them: You can remove if (f1) {
gzclose(f1) } in out:. Also, close(fd) shouldn't be needed after
gzclose(f) when f was created with gzdopen(fd, ...).
Win32 support: Haven't looked at this one yet. And I'm going on
vacation to Greece for one week tomorrow, so I'll continue when I'm home
again.
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Win32 support.
On 2010-07-15 22:07, Ramiro Polla wrote: I updated the fork at http://github.com/ramiropolla/ccache I also went through all the issues from other forks from http://code.google.com/p/ccache-win32/ and http://gitorious.org/ccache-win and fixed them. Splendid! Run stat() before open() on some files to allow bailing out earlier: Applied. Include getopt.h when getopt_long() is provided by the system: Applied. Unlink destination file before renames: Applied with commit message tweak. Introduce and use x_fmmap() and x_munmap(): Applied with commit message tweak. Close file opened with gzdopen(): Applied. Duplicate path string inside find_executable_in_path() instead of expecting caller to duplicate it before.: Applied with commit message tweak. Close manifest files after reading/writing them: Applied. Implement is_absolute_path() helper function.: Applied with commit message tweak. Implement compare_executable_name() helper function.: Applied with commit message tweak. Implement is_full_path() helper function: This doesn't work if the path is ../foo or any other relative path other than ./foo. Doesn't just searching for / or \\ work on Windows too? Clear exit_functions after the exit functions have been called: Applied. Add a define for path delimiter: Applied. Also made CCACHE_EXTRAFILES use PATH_DELIM. Use a temporary file for sed_in_place(): Applied. Allow testsuite to run on Windows: Applied. Win32 support: In compare_executable_name, you can use x_asprintf(tmp, %s.exe, s2) instead. Otherwise, looks good to me. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache and distcc-pump
On 2010-08-05 00:24, Bernhard Bauer wrote: what is the current state of ccache together with distcc pump mode? I'd assume that direct mode cache hits would go nicely with pump mode, but for cache misses we would still need to run the preprocessor locally to calculate the hash for the manifest file, which would ruin the benefit of pump mode. Yes, that's my current understanding too. Disclaimer: I have actually never run distcc myself, only read some of the code. Is there any work under way / planned to improve that? Not that I know of. I guess it could be possible to come up with some solution where distcc tells ccache which header files it read when compiling. (Cc: the distcc list.) -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache and precompiled headers
On 2010-07-03 14:22, Tor Arne Vestbø wrote: [...] As you can see removing pch from the build increases the total build time for the initial build (empty cache), but speeds up builds when there's something in the cache since we're hitting direct mode in almost all cases. When pch and ccache is used together the initial build time decreases, _but_ builds with a filled cache now take longer. Seems like we're hitting a lot more preprocessed cases for some reason. Dunno why that might be? I've started implementing support for preprocessed headers (.gch) and now I know why you get lots of preprocessed hits with your suggested patch. The thing is that a .gch file created by GCC is apparently not stable; the content is different each time for the same input. And since ccache currently doesn't know how to cache the creation of a precompiled header, a new and thus different .gch file is created, so the direct mode will therefore miss. Another issue is that the direct mode hashing of source code skips over comments. This means that if there happens to be a /* or // in the .gch (which is a binary file), parts of the file will not contribute to the hash sum, which is bad. Both issues should be easy to handle, though. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache and precompiled headers
On 2010-08-27 15:59, Tor Arne Vestbø wrote: On 27.08.10 13.15, Tor Arne Vestbø wrote: I'm sorry to report that I'm seeing build issues with master when building Qt on Mac OS X (but the issue should be fixable): I'm assuming the hash is based solely on the contents of the preprocessed output, not the flags/defines that were passed? But it seems that the defines are stored somewhere in the precompiled header, since it's barfing out when detecting a mismatch. It seems we do include a few flags in the hash as well, but not -D. Using the following patch, with CCACHE_SLOPPINESS=time_macros set everything seems to work fine: Great! Thanks for investigating this. I moved output_is_precompiled_header to be a global variable so it could be read in calculate_object_hash(), there's probably a nicer way to do that, passing it back from cc_process_args in an out-argument and then back into calculate_object_hash() or something similar. Since there's already lots of global state, I think adding one more variable doesn't make it worse. (I have plans to deglobalize state in the future.) I also include all flags in the hash when output_is_precompiled_header is set, just to see if things would work. We could perhaps limit this to -D and a few others if we can figure out exactly what GCC puts into the precompiled header. Right. Finally, I added an improved check for use of precompiled headers, checking for the existence of a .gch directory in -include arguments (instead of pch in the filename as before). You're right that we can't detect all uses, and modifying the build system is the sure-safe way, but I think we can at least try to be smart here? I like that approach better, yes. I don't think we need to check that pchpath is a directory, though, since the preprocessed header may be a file as well. I'll apply your patch and remove the directory check. With this setup I do get mostly direct hits on subsequent runs. The few preprocessed hits I get happens when looking for the object file hash in the manifest and not finding the manifest. Not sure why this would happen? Then the hash of the arguments and the input file has changed. It's a bit fiddly but possible to debug this. If you want to do it, you can compile ccache with -DCCACHE_DEBUG_HASH and set CCACHE_DEBUG_HASH=1 in the environment. ccache will then write the bytes it hashes to the file ccache-debug-hash.bin so that you can compare the hashed bytes from two different runs in a binary editor. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] ccache version 3.1 has been released
I'm happy to announce ccache version 3.1, which contains new features and some bug fixes. Details may be read in the release notes: http://ccache.samba.org/releasenotes.html Source archives are available here: http://samba.org/ftp/ccache/ccache-3.1.tar.bz2 http://samba.org/ftp/ccache/ccache-3.1.tar.gz http://samba.org/ftp/ccache/ccache-3.1.tar.xz The source archives have been signed with my OpenPGP key (BB845E97): http://samba.org/ftp/ccache/ccache-3.1.tar.bz2.asc http://samba.org/ftp/ccache/ccache-3.1.tar.gz.asc http://samba.org/ftp/ccache/ccache-3.1.tar.xz.asc You can run gpg --verify ccache-3.1.tar.*.asc to check the source archive integrity after download. Changes since version 3.0.1: New features and improvements: - Added support for hashing the output of a custom command (e.g. $compiler --version) to identify the compiler instead of stat-ing or hashing the compiler binary. This can improve robustness when the compiler (as seen by ccache) actually isn't the real compiler but another compiler wrapper. - Added support for caching compilations that use precompiled headers. (See the manual for important instructions regarding this.) - Locking of the files containing statistics counters is now done using symlinks instead of POSIX locks. This should make ccache behave a lot better on file systems where POSIX locks are slow or broken (e.g. NFS on some systems). - Manifest files are now updated without the need of taking locks. - Updates of statistics counters are now always done in one of the sub-level statistics files. This reduces lock contention, which especially improves performance on slow NFS mounts. - Reading and writing of statistics counters has been made forward-compatible (unknown counters are retained). - Files are now read without using mmap(). This has two benefits: it's more robust against file changes during reading and it improves performance on poor systems where mmap() doesn't use the disk cache. - Added .cp and .CP as known C++ suffixes. - Improved logging. - Added -install_name as an option known to take an argument. (This improves statistics when using the Darwin linker.) Bug fixes: - Non-fatal error messages are now never printed to stderr but logged instead. - Fixed a bug affecting failing commands when --ccache-skip is used. - Made --ccache-skip work for all options. - EINTR is now handled correctly. Other: - Work on porting ccache to win32 (native), mostly done by Ramiro Polla. The port is not yet finished, but will hopefully be complete in some subsequent release. - Added a --nostats flag to the performance benchmark program. - Made the performance benchmark program more accurate when measuring cache hits. - Added a new test framework for unit tests written in C. - Got rid of configure-dev; dev mode is now given by dev.mk.in presence. - Improved documentation on how to combine ccache with other compiler wrappers (like distcc). - New LICENSE.txt file with licensing and copyright details about bundled source code. - New AUTHORS.txt file with a list of ccache contributors. - New HACKING.txt file with some notes about ccache code conventions. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] PATCH: Print temp file name in debug message
On 2010-09-17 21:56, Wilson Snyder wrote: First, congrats on finishing the latest release. It's so wonderful to have great progress. Thanks. Anyhow, here's a trivial patch I'd like considered. I had to resolve a conflict around it in git when I updated so might as well feed it up, as it may help others debug too! Sure, why not. Applied! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] More win32 patches
On 2010-09-22 05:17, Ramiro Polla wrote: I've updated my github fork at: http://github.com/ramiropolla/ccache All applied, except for the strtok_r and AR patches. - There is no strtok_r() on Windows, I've implemented an ugly replacement using strtok(). I switched to use strtok_r because I needed nested parsing somewhere. Although I don't see anything in the current code that needs it (I probably rewrote it), I think we should use a reentrant replacement. Maybe this: http://www.mail-archive.com/mutt-...@mutt.org/msg03067.html - The ar check on configure.ac fails for cross-compilation. I haven't investigated why, I've just removed it, but I suppose it's there for a reason so it's better to fix than to remove =) Perhaps already fixed in e1b287d0ca3cdb48a5d3e8b0d46b9af51ee39abe? - Make test on Windows requires a certain dd to work (one that can emulate /dev/zero). It's the one found here: http://www.chrysocome.net/dd If it's easier, I guess we could use any existing, large enough file instead of /dev/zero, e.g. ccache.c. [...] Joel, is there a way we can time how long each part of ccache takes? Not just a whole batch of tests, but it would be nice if under a certain ifdef we could print to the ccache log the time it takes to perform each step. That would be a nice thing to have. Added to the TODO list. It seems to have gotten slower on Windows since the first port I've made (either that or FFmpeg has grown a lot too). Off the top of my head, here are some things that have changed since 3.0pre0 that may affect the speed: - The parsing in hash_source_code_string. Last time I measured, it's actually about 2 times slower than the actual hashing, so there's room for improvement there. - The parsing in process_preprocessed_file is probably also possible to improve. - read() is now used instead of mmap(); maybe it makes a difference on Windows. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] PATCH: Fix version.c for older gits
On 2010-09-24 20:46, Wilson Snyder wrote: git describe --dirty is a very recent addition. I would suggest the following so that the version works properly on older versions. Thanks, applied. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] More win32 patches
On 2010-09-27 06:32, Ramiro Polla wrote:
At first I tried using a naive wrapper of strtok_r() around strtok()
but I've been bitten by the reentrancy issue. It involved that
function that hashes multiple commands and find_executable. I'm
certain my version works, but I'm certain it's a hack too =). I don't
really mind which replacement is used as long as it works and the
functions it uses exist/work as expected on win32. I've checked
quickly and all of strspn, strpbrk, and strchr seem to be available on
windows.
Right. Now I've actually read your version and I see no problem except
that it requires *saveptr == NULL on the first invocation, which the
real strtok_r doesn't. But if I'm not mistaken, that's easily fixed like
this:
--- a/util.c
+++ b/util.c
@@ -176,7 +176,7 @@ strtok_r(char *str, const char *delim, char **saveptr)
{
int len;
char *ret;
- if (*saveptr)
+ if (!str)
str = *saveptr;
len = strlen(str);
ret = strtok(str, delim);
I've applied your patch and the above tweak.
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] PATCH: Fix NFS object corruption
On 2010-10-08 13:28, Wilson Snyder wrote: The following patch fixes this issue by always calling rename before a unlink - a new x_unlink function. There are some places where temp files are being unlinked; for performance these can remain as the ordinary unlink. Seems good to me. Thanks! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Would backdating source files allow sharing hardlinks between trees?
On 2010-11-25 21:56, Justin Lebar wrote: [...] 1) Update the object file's mtime to now. If we do this, then when we rebuild in another tree, the object file will be newer than the things which depend on it (e.g. shared libraries which include the file), so we'll have to relink. 2) Leave the object file's mtime unmodified. If we do this, then when we rebuild this same tree, the object file will appear out of date. I believe ccache takes the first approach, which seems sensible. Yes. However, ccache also updates the object file's mtime for another reason: it's used to mark the object as used so that the cleanup algorithm knows which least recently used results it should remove. (So an option like you suggest below must also change ccache's bookkeeping of usedness in some way.) But as an alternative, what if we backdated the source file's mtime to before the object file was created and left the object file's mtime unchanged? (If we didn't want to arbitrarily choose a time, we could save the mtime of the source file which originally generated the object file and use that.) That's an interesting idea! This would be an option, of course -- it's kind of strange behavior, so it certainly shouldn't be on unless the user asks for it. But I wonder if this actually solves the problem. I suspect it might screw up some important tool, but I can't think of a potential bad interaction off the top of my head. One issue I see is that if foo.c is generated from foo.y, then a backdated foo.c may become older than foo.y, which will trigger generation of foo.c, which will trigger building of foo.o, which makes ccache backdate foo.c, which leads back to square one. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] ccache version 3.1.4 has been released
I'm happy to announce ccache version 3.1.4, which is a bug fix release. Details may be read in the release notes: http://ccache.samba.org/releasenotes.html Source archives are available here: http://samba.org/ftp/ccache/ccache-3.1.4.tar.bz2 http://samba.org/ftp/ccache/ccache-3.1.4.tar.gz http://samba.org/ftp/ccache/ccache-3.1.4.tar.xz The source archives have been signed with my OpenPGP key (BB845E97): http://samba.org/ftp/ccache/ccache-3.1.4.tar.bz2.asc http://samba.org/ftp/ccache/ccache-3.1.4.tar.gz.asc http://samba.org/ftp/ccache/ccache-3.1.4.tar.xz.asc You can run gpg --verify ccache-3.1.4.tar.*.asc to check the source archive integrity after download. Changes since version 3.1.4: Bug fixes: - Made a work-around for a bug in gzputc() in zlib 1.2.5. - Corrupt manifest files are now removed so that they won't block direct mode hits. - ccache now copes with file systems that don't know about symbolic links. - The file handle in now correctly closed on write error when trying to create a cache dir tag. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] How to make ccache not interpret 'multiple' compiler options
On 2011-01-20 21:01, NITIN KHOSLA, BLOOMBERG/ 731 LEXIN wrote:
Facing an issue with solaris compiler options.
Our solaris compiler command takes many -xflags.
Command: ccache CC -xcode=pic32 -xdebugformat=stabs -c -o file.o file.c
[2011-01-20T14:54:50.499872 9962 ] Unsupported language: debugformat=stabs
[2011-01-20T14:54:50.509438 9962 ] Result: unsupported source language
As you have noticed, ccache currently interprets -x flags like GCC does,
which does not work well with compilers that use -x for other things.
There's a bug report about it here:
https://bugzilla.samba.org/show_bug.cgi?id=7556
I used --ccache-skip but it seems I have to use this BEFORE every flag.
Yes, that's correct, and there's currently no other way, unless it's OK
for you to patch your locally built ccache like this:
--- a/ccache.c
+++ b/ccache.c
@@ -1323,6 +1323,7 @@ cc_process_args(struct args *orig_args, struct args
**preprocessor_args,
continue;
}
+#if 0
/*
* Special handling for -x: remember the last specified
language before the
* input file and strip all -x options from the arguments.
@@ -1346,6 +1347,7 @@ cc_process_args(struct args *orig_args, struct args
**preprocessor_args,
}
continue;
}
+#endif
/* we need to work out where the output was meant to go */
if (str_eq(argv[i], -o)) {
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] How to make ccache not interpret 'multiple'
On 2011-01-20 23:29, NITIN KHOSLA, BLOOMBERG/ 731 LEXIN wrote: Though it works even when I do 'not' use --ccache-skip flag at all. a) Is that what you expect? Yes. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache and gcc triplet
On 2011-01-23 22:43, Gilles Espinasse wrote: [...] I had think to add triplet symlink (i486-linux-gnu-gcc) to ccache. Is there another solution? The other option is to add ccache as a prefix to the compilation row, so that i486-linux-gnu-gcc arg ... becomes ccache i486-linux-gnu-gcc arg ... instead. How to do that depends on the build system, of course. Should not that issue be referenced in the doc? I think that adding that kind of detail to the docs would not be very useful since it's not a common scenario, but maybe we could improve the current manual (http://ccache.samba.org/manual.html#_run_modes) in more generic terms. Do you have any ideas on how? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache and gcc triplet
On 2011-01-25 08:32, Gilles Espinasse wrote: Then I had the issue that the 1 G default size is slightly less than what is needed. If you look at the stats after the full distrib build, you may see a 'cache size' a bit below the limit and think the size is enought. But that may be wrong. I agree that could be a bit misleading. Tracing stats after each packet build, I haven't seen a size bigger than 922 MB referenced. Then size decrease to approximately 850 MB due old hits purge and increase again with usage. The reason for the behaviour is that each of the 16 first-level cache directories are cleaned up individually. When the size of one of them exceeds max_cache_size/16, files are removed from it until the size is below 0.8*max_cache_size/16. This means that the cache size (once it has been filled one time) will be 0.9*max_cache_size on average. How do you know that 'max size' is too small? Is-it when you reach more than 'max size' x 0.85? If the max cache size is too small, you get cache misses because of too aggressive cache cleanup. If the max cache size is too large, you waste disk space. I don't think there's a good rule of thumb for what the max cache size should be, since it depends on the circumstances. But if you have lots of disk, just set it high (or maybe even unlimited). Otherwise, a good indicator of a too small cache size is if you get cache misses for things that should have been taken from the cache. I just saw yesterday that ccache-3 ubuntu package have an update-ccache-symlink that parse /usr/lib/gcc to find installed compiler triplet (with some c89/c99 variations). Yes, made by me. :-) -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Preprocessor stdout regression since v2.4
On 2011-02-15 13:51, Mike Crowe wrote: It might not be expected for cpp to be a symlink to ccache but with ccache-v2.4 it did used to work so I found it easier for our build system just to symlink everything in the toolchain directory to ccache. Unfortunately between v2.4 (Debian Lenny version) and v3.1.4 the behaviour seems to have changed. When running the preprocessor ccache doesn't seem to realise that it should write the output to stdout. Thanks for noting this. The reason for the changed behaviour is that ccache compiler header.h is a supported invocation (meaning: build a precompiled header from header.h) in ccache 3.1 and higher, but it wasn't in ccache 2.4 (which fell back to running the real compiler, i.e. cpp in this case). So, don't symlink cpp to ccache. :-) -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache shared cache - fails to rename
On 2011-04-02 08:37, devsk wrote: I see a lot of [2011-04-01T23:31:16.190840 26563] Failed to rename /var/tmp/ccache/8/7/007a9f0d4b791e3c439260a8b86724-2049.manifest.tmp.localhost.26563 to /var/tmp/ccache/8/7/007a9f0d4b791e3c439260a8b86724-2049.manifest in the log file. The source is owned by one user running the ccache currently and destination is owned by another user. Both files are 'rw' by everybody. Any ideas why would it fail to rename the file? CCACHE_UMASK is set to . You need write permission for all directories in the cache, not only the files. Check the permissions of the /var/tmp/ccache/8/7 directory. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] PATCH: -F/-iframework GCC options
On 2011-04-07 14:16, Grigory Entin wrote: Attached is a simple ccache patch for -F/-iframework GCC options that affect search paths for frameworks (that are directories that are searched for header files). I use it daily for half a year already, for compiling lots of Objective-C code, without a problem (and I do have 100% hits on recompiling with different search paths but same sources). Thanks! Applied. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Configuration file plan
On 2011-04-06 23:25, Owen Mann wrote: Question: for an NFS-mounted cache, wouldn't different users' CCACHE_MAX values conflict? It seems to me it's better as a one-time or per-cache setting. I can definitely see how it would have value in a single-user setup, though. One of my reasons for wanting to introduce a configuration file is to improve the multi-user case so that it's easier to make all users use the same settings (umask, compression, hardlink, nlevels, max_size, etc.), so I haven't considered having a per-user configuration file -- ccachedir/ccache.conf is per-cache. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Configuration file plan
On 2011-04-07 02:30, Martin Pool wrote: I would think about not putting ccache.conf inside .ccache, because that directory might often be excluded from backups or in some other way treated as disposable. Aside from that it sounds really good. I'd prefer the configuration to be tied to the cache, so it will be hard to do it another way given the current cache directory layout. I guess the only reasonable alternative is $(dirname $cache_dir)/ccache.conf. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Generated dep file newer than object file
On 2011-05-01 23:02, Andrew Gallagher wrote: [...] foo.o: foo.c gcc -c -MD ... foo.o: foo.d .d files aren't supposed to be dependencies, so I'd argue that such a Makefile is badly written. However, letting ccache update mtime of the .d file before the .o file couldn't hurt, so that would be OK with me. I think the fix is pretty simple (I've been adding an update_mtime(output_obj); if a dep file is produce). What would be the best way to send diffs/patches for bug fixes like this? You can either send the patch to this list or create a bug report; see http://ccache.samba.org/bugs.html. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] problems with buildworld gcc45 on ports
Hi,
On 2011-06-21 20:29, Beeblebrox wrote:
[...FreeBSD-specific questions...]
You will will probably get better answers to these questions on some
FreeBSD list. At least I don't know much about FreeBSD. I have one
comment, though:
2. Is is necessary to include cpp in the 1st conditional, or does ccache
detect the correct pre-processor on its own from here? (I have created the
symlinks)
CPP:=${CPP:cpp45,/usr/local/libexec/ccache/world/cpp45,1}
ccache only understands how to cache object files when invoked as the
compiler -- using ccache with cpp will only result in a slightly slower
build. So: it's not necessary and not recommended.
Here are some general tips on troubleshooting ccache:
http://ccache.samba.org/manual.html#_troubleshooting
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Automatic fallback from preprocessed to direct mode in case of cache miss
Hi, On 2011-06-23 17:15, Rémi Delmas wrote: [...] Another solution would be for ccache to automatically fallback to preprocessed mode in case direct mode fails. That's precisely what ccache does already (and direct mode is the default; you don't need to set CCACHE_BASEDIR to activate it). See http://ccache.samba.org/manual.html#_how_ccache_works for more details. If I understand your description correctly, your problem has to do with the -I options being different in the two scenarios, like this: Case A: cd ~A/dev/ProjectA gcc -I../ProjectB -c header.c Case B: cd ~B/ProjectA gcc -I../development/ProjectB -c header.c Right? The direct mode can't be made to work for this case, so there's nothing to configure to make it work. The direct mode simply needs to see exactly the same compiler options (including -I) to kick in, and then it will search for header files in exactly the same places the previous compilation found them. So, I'm afraid you have to encourage or enforce users A and B to use the same directory layout to benefit from the direct mode (or invent something more clever). -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] problems with buildworld gcc45 on ports
On 2011-06-24 06:40, Beeblebrox wrote: Thanks for the answer. The main question has been posted on a forum for a while, and judging from the number of views / hits the post has had, this appears to be a broader issue. I have also gone through the ccache manual many times, to the best of my current ability, without success. The post in question: http://forums.freebsd.org/showthread.php?t=24286 Regarding issue 4 (about CCACHE_CPP2): I'm not familiar with BSD make syntax and semantics, but I suspect that setting CCACHE_CPP2=yes in make.conf only sets a make variable, not an environment variable that ccache is able to see. I don't know how you're supposed to set environment variables in FreeBSD (maybe in /etc/profile?), but you can always set it yourself in your shell (export CCACHE_CPP2=yes or so) before building. As mentioned in the troubleshooting part of the manual, you can set the environment variable CCACHE_LOGFILE to a file name, then perform the build and then look at the file to get some clues about what happens. Also, one Major question I have is: does ccache handle a dual-gcc-setup (I suppose It should) without problems I don't know what you mean with dual-gcc-setup, but if you have two compilers with different names, then there should be nothing that can go wrong. However, if there are multiple compilers behind the same name, then ccache needs to be able to tell them apart. This is by default done by looking at the compiler's mtime (and size). If the mtime doesn't change when compilers are switched, then ccache will happily return cached results from the previous compiler. One example when this can happen is if (for instance) /usr/bin/gcc is a wrapper script that runs one of several compilers depending on an environment variable or configuation file. In that case, you can configure ccache to run something like gcc -v instead to identify the compiler. See CCACHE_COMPILERCHECK in the ccache manual. and what is the correct syntax for the gcc45 section? I don't know what that means. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for -fprofile-generate, -fprofile-use, -fprofile-arcs
On 2011-07-21 20:22, Chris AtLee wrote: I recently did some work to get ccache to support gcc's various -fprofile-* options. In some local testing in works great. Thanks! I've got the code up on github right now: https://github.com/catlee/ccache/compare/jrosdahl:master...catlee:profile Does this approach look like it will work? Justin Lebar's point that the cwd probably needs to be hashed seems valid. Other than that, I think it looks generally fine, but I only have limited knowledge about the -fprofile-* options so I can't say I understand their interaction completely. Some comments and questions, though: I think that the output_to_real_object_first mode should really be the default and only mode. That would of course mean that the things in from_cache() that need to be done after storing an object in the cache should be refactored out somehow, but we can do that later. What about the -fprofile-generate=path and -fprofile-use=path forms? Should those be detected as well? tmp_obj is freed at the end of to_cache(), so output_obj will refer to freed memory when dereferenced in from_cache() later. An x_strdup is needed. //-style comments won't compile on systems with C89 compilers, so use /**/ instead. You should probably use hash_delimiter(hash, -fprofile-use) or so before hashing the gcda file (and then I guess that the hashing of no data won't be necessary). -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for -fprofile-generate, -fprofile-use, -fprofile-arcs
On 2011-07-25 21:19, Chris AtLee wrote: I've pushed some changes up here that I hope addresses all the comments: https://github.com/catlee/ccache/compare/jrosdahl:master...catlee:profile Looks mostly good! Thanks for working in this. Regarding TODO: Disable base_dir if profile_dir is NULL?: Yes, perhaps; it doesn't really matter since you hash CWD in that case, so base_dir will have no effect anyway. (And by the same logic, base_dir could also be disabled if CCACHE_HASHDIR is set.) Some code paths are missing a free(arg) in cc_process_args. I think it would be better to always allocate profile_dir on the heap, thus simplifying cc_reset. I guess output_to_real_object_first = false is a left-over from debugging? There are still some //-style comments. The added cc_log that says Failed to move in to_cache should be Failed to copy. And like last time, Justin Lebar's comments are also relevant (thanks, Justin!). -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for -fprofile-generate, -fprofile-use, -fprofile-arcs
On 2011-08-15 15:58, Chris AtLee wrote: Assuming -fprofile-generate/-fprofile-use also work on windows, that code shouldn't be ifdef'ed out. Instead I need an equivalent to x_realpath for windows (or windows support for x_realpath). Right. I'm blissfully ignorant about Windows APIs; does anyone else have a clue? I guess that we could just skip canonicalizing the path on Windows as a work-around. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] ccache version 3.1.6 has been released
I'm happy to announce ccache version 3.1.6, which is a release with bug fixes and other minor improvements. Details may be read in the release notes: http://ccache.samba.org/releasenotes.html Source archives are available here: http://samba.org/ftp/ccache/ccache-3.1.6.tar.bz2 http://samba.org/ftp/ccache/ccache-3.1.6.tar.gz http://samba.org/ftp/ccache/ccache-3.1.6.tar.xz The source archives have been signed with my OpenPGP key (BB845E97): http://samba.org/ftp/ccache/ccache-3.1.6.tar.bz2.asc http://samba.org/ftp/ccache/ccache-3.1.6.tar.gz.asc http://samba.org/ftp/ccache/ccache-3.1.6.tar.xz.asc You can run gpg --verify ccache-3.1.6.tar.*.asc to check the source archive integrity after download. Changes since version 3.1.6: New features and improvements: - Rewrite argument to --sysroot if CCACHE_BASEDIR is used. Bug fixes: - Don't crash if getcwd() fails. - Fixed alignment of called for preprocessing counter. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] basename of source file in hashed name in ccache
On 5 November 2011 01:26, Frank Klotz frank.kl...@alcatel-lucent.com wrote: [...] I remember quite clearly (and I just confirmed with a colleague who is still there) that the file names in the cache contained BOTH the hash AND the basename of the object file. As far as I know, the object files have always been stored using only the hash. However, temporary files (stored in $CCACHE_DIR in ccache =2.4 and $CCACHE_DIR/tmp in ccache =3.0) include (a prefix of) the basename. [...] (and another string that the ccache code refers to as size, although I can't quite figure out what it's the size OF) It's the size of the hashed text, i.e. output from the preprocessor. This is just a way of making filename collisions somewhat less likely. One place we found the basenames invaluable was tracking down a corrupted object file in the cache. Once we confirmed that we had a corrupt object file foo.o, we simple searched for all *foo.o files in the cache, compared those in size and content to an actual corrupted object file in the user directory, and easily removed the corrupted file from the cache. Much harder (not impossible, but harder) to do this without the basenames. An easy way to do that is: 1. Remove foo.o from the build tree. 2. Build with CCACHE_LOGFILE set to a log file. 3. Look for Created foo.o from X (where X is a file in the cache) in the log file. 4. Remove X. Or even easier: 1. Remove foo.o from the build tree. 2. Build with CCACHE_RECACHE set. [...] Anyway, is there a general consensus on whether this would be valuable? It doesn't sound like a good idea to me, at least, since you would need to store duplicate copies of the object file for two compilations where the source content is the same but the file names differ. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Duplicate object files in the ccache - possible optimization?
On 7 November 2011 18:49, Frank Klotz frank.kl...@alcatel-lucent.com wrote: [...] That aside, however, with the advent of direct mode, there ARE two hashes possible for any given object file - the direct mode hash (hashing all the sources that go into the compilation) and the preprocessed hash (hashing the result of running all those sources through the preprocessor). Well, yes and no. There's only one hash for a given object file: the hash of the output of the preprocessor. This hash is used to look up the object file in the cache (i.e., the object file is named after the hash). Then, for the direct mode, there is one hash for each combination of source code files (i.e., the file to compile and all its include files) and compiler flags that results in the same preprocessor output. The mapping between different source code hashes and the resulting preprocessor hashes is stored in .manifest files in the cache. A manifest file is looked up using (and thus named after) a hash of only the main file and compilation flags. And any time there is a cache miss, ccache has computed both those hashes, hasn't it? As mentioned above, it starts by computing a hash of the input source file and the command line options. It then looks up the manifest file, continues hashing include file sets found in the manifest and compares them with the actual include files. If there's a match, the object file name (i.e., the preprocessor hash) can be read in the manifest. This is documented in the manual under The direct mode: http://ccache.samba.org/manual.html#_the_direct_mode If it's hard to understand, I would be happy for any suggestions on how to improve it. :-) [...] And it appears to me that in many cases, the resulting object file occurs twice in the cache, once under each hash. Well, the object file is only stored once for a given preprocessor hash. And currently, those two occurrences are two separate files, which could be combined into a single inode with two hard-linked directory entries. If there are multiple object files in the cache with the same content, then that's because different preprocessor outputs have resulted in identical object files. I can imagine two ways of storing identical object files only once: - Introduce an object file store indexed by the object file hash. Entries in the manifest files would then refer directly to those file names and the files would also be stored under their preprocessor hash name. However, on a cache miss, there will be extra performance penalty since the hash of the object file needs to be calculated as well. That's probably measurably bad. - Or: Create a compactation tool which can be run on the cache once in a while. I think a good search engine term for this would be data deduplication. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Duplicate object files in the ccache - possible optimization?
On 8 November 2011 23:18, Frank Klotz frank.kl...@alcatel-lucent.com wrote: On 11/08/2011 01:49 PM, Joel Rosdahl wrote: It doesn't sound like a good idea to me, at least, since you would need to store duplicate copies of the object file for two compilations where the source content is the same but the file names differ. 1: Would that EVER happen? (I am having trouble visualizing a situation where this would be a good thing.) Don't know, but yes, it's probably quite rare. 2: If it DID happen, rather than 2 copies, could store one inode with 2 directory entries (hard links) with the 2 names. I don't think you can in an acceptable way find an existing file to reuse when you're about to store a new file -- you'll have to list the directory and iterate through its items to look for a reusable file. (That is, unless you introduce some kind of higher-level index that makes it possible to efficiently list existing files with a given hash.) -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] basename of source file in hashed name in ccache
On 9 November 2011 18:46, Frank Klotz frank.kl...@alcatel-lucent.com wrote: On 11/08/2011 11:08 PM, Joel Rosdahl wrote: I don't think you can in an acceptable way find an existing file to reuse when you're about to store a new file -- you'll have to list the directory and iterate through its items to look for a reusable file. (That is, unless you introduce some kind of higher-level index that makes it possible to efficiently list existing files with a given hash.) Maybe I'm confused again, but since they have the same hash hhh, wouldn't it in fact be trivially EASY to find them? Easy: yes. Fast: no. The subdirectory selection still depends on the hash, so both files would be in the same directory, and when you are looking to create hhh.foo.o in that directory, you can first look for hhh.*.o, and when you find hhh.bar.o [...] To look for hhh.*.o in directory D, you need to get a directory listing of D and iterate through its items (i.e., file names) and see if they match the pattern. This will be expensive for directories with many files. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] basename of source file in hashed name in ccache
On 2011-12-20 20:09, Frank Klotz wrote: Suggestion: Put the below 2-line solution in the Corrupt object files section of http://ccache.samba.org/manual.html [...] It's a LOT lighter-weight solution than clearing the cache, and much easier than messing with the log. Good suggestion. I'll do that in the next version. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] What to do when there are problems writing to the cache directory?
Hi, I would like to get some feedback on what ccache users think ccache's behavior should be when it fails to create files or directories in the cache directory. Here are some different cases to consider: 1. CCACHE_DIR is set to a directory that doesn't exist and can't be created. 2. CCACHE_DIR is set to a directory that exists but is unwritable. 3. CCACHE_DIR is set to a directory that exists, but one of the subdirectories storing the cache data (e.g. $CCACHE_DIR/a/b) is unwritable or not possible to create. 4. CCACHE_DIR is set to a directory that exists, but CCACHE_TEMPDIR (defaults to $CCACHE_DIR/tmp for ccache 3 and $CCACHE_DIR for ccache 2) is unwritable or not possible to create. Here's what happens with ccache 2.4: 1. Fatal error, non-zero exit code. 2. Silent failure, falling back to running the compiler. 3. Silent failure, falling back to running the compiler. 4. Silent failure, falling back to running the compiler. ccache 3.0-3.1.6 (I think, haven't checked all versions): 1. Fatal error, non-zero exit code. 2. Fatal error, non-zero exit code. 3. Silent failure, falling back to running the compiler. 4. Silent failure, falling back to running the compiler. ccache 3.1.7: 1. Fatal error, non-zero exit code. 2. Fatal error, non-zero exit code. 3. Fatal error, non-zero exit code. 4. Silent failure, falling back to running the compiler. ccache has previously been more forgiving, so one might argue that ccache 3.x more strict behavior is a regression from version 2.4 that should be fixed. Also, one might argue that ccache should never fail fatally so that case 1 should silently fall back to running the compiler as well. On the other hand, my general opinion is that tools should not fail silently. As a ccache user, I would like to be informed if something is badly configured, for instance if the cache directory is unwritable by me, so that I get alerted that something is wrong. What do you think? -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Minor enhancement request: Handling of CCACHE_PREFIX when it is defined as a null string
On 7 February 2012 19:38, Frank Klotz frank.kl...@alcatel-lucent.com wrote: Could a CCACHE_PREFIX defined as a null string please be treated as if it were undefined, rather than as a fatal error? [...] And you might want to consider refactoring this almost-but-not-quite-identical code into a function or macro - not sure why there are two versions These are both good suggestions, and they have actually already been done over a year ago on the master branch along with allowing several binaries in CCACHE_PREFIX: http://git.samba.org/?p=ccache.git;a=commit;h=b453d43ac0dc7097bb8afb74db70ccf5cde3a914 -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Configuration file: please test and comment
Hi, Many moons ago, I mentioned that I planned to add a ccache configuration file (see http://www.mail-archive.com/ccache@lists.samba.org/msg00656.html). I've now implemented it mostly according to the sketch in the aforementioned mail and merged it to the master branch. If you're interested in trying it out in its present state, please do so. Feedback and opinions are welcome. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for -fprofile-generate, -fprofile-use, -fprofile-arcs
Hi, It will be included in ccache 3.2. There's no time plan for when it will be released, though. I've merged the main 3.2 feature (configuration file support) to master, so it's mostly a matter of finding time. Testing of the development version (i.e., the current master branch) by other people than me would also speed up the process. -- Joel On 31 March 2012 06:31, Chris AtLee ch...@atlee.ca wrote: Hi, When will this patch be included in a released version of ccache? fwiw I've done some timings with building firefox with ccache 3.1.7 with and without this patch. without: real59m48.040s user150m38.626s sys 24m12.441s cache directory /builds/ccache cache hit (direct) 879 cache hit (preprocessed) 16 cache miss16 called for link 128 preprocessor error 1 unsupported source language6 autoconf compile/link 11 unsupported compiler option 6948 no input file 4 with patch real35m12.100s user77m37.852s sys 18m11.073s cache directory /builds/ccache cache hit (direct) 2032 cache hit (preprocessed)2638 cache miss 3107 called for link 208 preprocessor error 1 unsupported source language8 autoconf compile/link 11 no input file 4 On Sun, Aug 14, 2011 at 1:53 PM, Joel Rosdahl j...@rosdahl.net wrote: On 2011-08-02 03:52, Chris AtLee wrote: My latest code on github should address all the comments so far: https://github.com/catlee/ccache/compare/jrosdahl:master...catlee:profile Merged now. Thanks! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Suggested patch to add local strtok_r for systems lacking it
Hi, On 12 April 2012 08:32, Jürgen Buchmüller pullm...@t-online.de wrote: here's a suggested patch for config.h.in and util.c to add a local implementation of strtok_r for systems that don't have it (e.g. mingw32 plus libgw32c). A lot of Windows build related work has already been done in the development version (i.e., on the master branch), including a strtok_r implementation. It would be great if you could try it out: http://ccache.samba.org/repo.html -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] PATCH: Look at include files' mtimes
On 22 May 2012 04:00, Justin Lebar justin.le...@gmail.com wrote: I can't say I'm particularly interested in supporting two manifest versions simultaneously I agree that supporting only the latest should be enough. Coming to think of it, perhaps it would be a good idea to include the manifest version in the manifest hash so that ccache versions using different manifest versions won't even try to read each others' manifest files? (Currently, ccache overwrites incompatible manifest files with its own version.) -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Old code ccache hit always come back unless CCACHE_DISABLE is used
On 1 June 2012 12:12, g@free.fr wrote: On the no-fPIE case, using CCACHE_CPP2=1 change compilation to work. Any next recompilation even without CCACHE_CPP2=1 result in cache hit. Yes, so this means that the compiler has a bug. You should be able to reproduce it by running these two commands: /usr/bin/gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -include ../config.h -I../include -DLOCALEDIR=/usr/share/locale -fsigned-char -Os -march=i486 -mtune=pentium -pipe -fomit-frame-pointer -c -MT lscpu.o -MD -MP -MF .deps/lscpu.Tpo -E lscpu.c -o lscpu.i /usr/bin/gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -include ../config.h -I../include -DLOCALEDIR=/usr/share/locale -fsigned-char -Os -march=i486 -mtune=pentium -pipe -fomit-frame-pointer -c lscpu.i -o lscpu.o As mentioned, the ccache workaround for this kind of bug is to set CCACHE_CPP2=1. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Support for Clang precompiled headers
On 13 July 2012 11:48, Lubos Lunak l.lu...@suse.cz wrote: Right. I guess that was an oversight when I moved the Multiple precompiled headers used error to one place. Fixed version attached. Thanks, applied. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache problem with CCACHE_BASEDIR
Hi, It seems you forgot to make cwd_canonicalized static? Currently, it has no effect. Also, the test suite crashes with your patch. I recommend running the test suite during development even for minor patches (and for bonus points: add new tests for the new functionality :-)). The crash happens because realpath returns NULL for nonexisting paths, and the test suite sets up fake paths for some parameters; this could happen in real usage as well. So, the code needs to handle x_realpath returning NULL. Thinking more about this, it would probably be better to (instead of realpath) use some function that doesn't expand symlinks and doesn't care whether the path exists or not. I'm not aware of such an existing (and portable) function, though, so we likely will have to write it ourselves. -- Joel On 20 July 2012 19:45, Eric Blau eric.b...@tekelec.com wrote: On 2012-07-20 13:09, Joel Rosdahl wrote: Thanks. Two things: 1. The patch canonicalizes the from parameter, but I think that the to parameter should be canonicalized as well, for two reasons: a) realpath() expands symlinks, and comparing (see common_dir_prefix_length) an expanded path with an unexpanded path won't work well when symlinks are present. b) The to parameter (which for instance may be a path given on the command line) could also contain path elements that could benefit from canonicalization (to increase cache hits). 2. The patch canonicalizes the from parameter every time get_relative_path is called, which is unnecessary since it's always the same. Therefore, as mentioned earlier, I think that its' better to do the canonicalization in make_relative_path where current_working_dir can be canonicalized when created. Thanks for the comments. Please discard the previous patch. This attached version of the patch should be better. It canonicalizes the current working directory once and canonicalizes the path each time get_relative_path is called. Thanks, Eric ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache problem with CCACHE_BASEDIR
On 27 July 2012 14:33, Joel Rosdahl j...@rosdahl.net wrote: [...] So, the code needs to handle x_realpath returning NULL. Eric, I've now applied your patch with some tweaks in 2df269a3121889ebcdfa5d98dfb4d675f690e039 (see http://git.samba.org/?p=ccache.git;a=commit;h=2df269a3121889ebcdfa5d98dfb4d675f690e039). Thanks! Thinking more about this, it would probably be better to (instead of realpath) use some function that doesn't expand symlinks and doesn't care whether the path exists or not. [...] On the other hand, such a function would still need to expand symlinks when there are .. components in the path, which feels quite complex. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] CCACHE_BASEDIR and dependency file creation (-MD/-MMD)
On 27 July 2012 17:27, Lalit Chhabra lchha...@linuxmail.org wrote: Here is an updated patch that also fixes -MT/-MQ options on the command line. It also keeps the arg formatting the same, so that make test passes. Applied, thanks! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Hash also Clang plugins
On 30 July 2012 19:03, Lubos Lunak l.lu...@suse.cz wrote: please review and push the attached patch, which adds Clang plugins to the hash, similarly as is already done with GCC plugins. Thanks, applied. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] make git revision detection code work if building outside of src dir
On 2 August 2012 16:51, Patrick von Reth patrick.vonr...@gmail.com wrote: This fixes git description if building outside of src. Applied, thanks! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] tmpfs + aufs use with ccache
On 3 August 2012 23:39, Craig A. Cox craig...@yahoo.com wrote: I am using version 2.4 of ccache Time to upgrade? [...] I have setup an aufs filesystem for ccache. That way a reboot will have a completely fresh initialized ccache setup. What do you mean with a completely fresh initialized ccache setup? Is it a cache where you only have run ccache -M 10 or so? [...] The possible issue that I need to ask about is files that are on the read-only filesystem that should completely disappear will not, you can only have a newer version in the tmpfs. Will this work or will I be required to just use ccache on the tmpfs and initialize it in a startup script? With ccache 2.4, I think you are OK. There might be problems with ccache newer than 2.4 because then the statistics files are removed as part of process of updating counters. Initializing in a startup script should be OK. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] permit ccache to build with clang
On 18 September 2012 18:21, Eitan Adler li...@eitanadler.com wrote: I needed the following patch for ccache to build with clang. [...] -all_cppflags = @DEFS@ @extra_cppflags@ -DSYSCONFDIR=$(sysconfdir) -I. -I$(srcdir) $(CPPFLAGS) (Your patch doesn't apply since it's garbled. The above two lines should be one.) +all_cppflags = @DEFS@ @extra_cppflags@ -DSYSCONFDIR=$(sysconfdir) $(CPPFLAGS) This breaks out-of-source builds: % mkdir build % cd build % ../configure CC=clang % make test [...] clang -DHAVE_CONFIG_H -DSYSCONFDIR=/usr/local/etc -MD -MP -MF .deps/test_main.c.d -g -O2 -Wall -W -Werror -c -o test/main.o ../test/main.c ../test/main.c:20:10: fatal error: 'test/framework.h' file not found #include test/framework.h ^ 1 error generated. make: *** [test/main.o] Error 1 So, I can't apply it. I think it's quite silly of clang to warn about unused -I options; it would be a hassle to add -I options potentially tailored for each source code file. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] permit ccache to build with clang
On 23 September 2012 14:40, Eitan Adler li...@eitanadler.com wrote: After some more experimentation this seems to only happen under ccache OK. Then this is another instance of https://bugzilla.samba.org/show_bug.cgi?id=8118 (which I should have realized, sorry about that), which has been discussed previously on the list (see http://www.mail-archive.com/ccache@lists.samba.org/msg00807.html). Off list, David Givone has started working on a fix for this, but I don't know the progress. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Why not cache compile failures?
On 24 September 2012 10:44, Andrew Stubbs a...@codesourcery.com wrote: On 22/09/12 13:13, Joel Rosdahl wrote: I would be OK with caching failures if it's off by default and doesn't affect the default's performance. Have you thought about how to store the exit code in the cache? The simplest way would be a tiny file in the cache that just contained the code. Say with extension .error. I'd prefer it to be on by default One reason for having it off by default is that it will decrease performance, especially on NFS, to let ccache stat a .error file in the cache for each compilation. Side note: I have previously toyed with an idea to store a compilation result as a single file in the cache instead of up to three different files (as it's done currently) to potentially improve performance (this will make the hardlink mode impossible, however). Another reason is that I feel that there are many subtleties about when a result can be reliably cached, as discussed in this mail thread already. if only to prevent other devs breaking it, but that's your decision to make, of course. That's what automatic test cases are for. :-) Printing an extra warning message would break ccache's transparency, but perhaps that would be acceptable for failures. Have to think more about it. [...] Otherwise, I don't know of any cases where scripts check error/warning messages beyond whether they exist or not. Since I would have it only emit the warning when there are already error messages, it would only break in unusual cases. Right, that sounds reasonable. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] direct mode design bug
Hi, The direct mode, which was introduced in version 3.0 almost three years ago, has a design bug. The essence of the problem is that in the direct mode, ccache records header files that were used by the compiler, but it doesn't record header files that were not used but could have been used if they existed. So, when ccache checks if a result could be taken from the cache, it can't check if the existence of a new header file should invalidate the result. This scenario is probably quite rare since only few people have reported it during the years (there are two public bug reports: https://bugzilla.samba.org/show_bug.cgi?id=8424 and https://bugzilla.samba.org/show_bug.cgi?id=8728), but the problem may of course happen without the user reporting it or knowing about it. Anyway, regardless of frequency, it makes ccache's behavior differ from that of the unwrapped compiler. Unfortunately, I don't know how to fix the issue in a good way. One obvious way would be to try to figure out in which directories the preprocessor has looked for header files, store that information and do the same search when considering a cache result. But how to do that? 1. ccache could use strace or similar ways of monitoring the compiler and tracing the performed system calls to find out where headers were probed. I haven't measured, but I suspect that this would be slow. 2. ccache could override strategic functions using LD_PRELOAD, thus snooping on system calls without involving the kernel. This should be possible and quite fast, but it's tricky to get right, and it's not very portable. (By the way: This is what http://audited-objects.sourceforge.netdoes, although I don't know if it monitors and acts on probes of nonexistent files.) 3. ccache could try to imitate what the preprocessor does. That is, read the source code file and follow #include statements instead of looking at the preprocessor output. This essentially means implementing a dumbed down version of a preprocessor, a task that doesn't sound trivial: It has to be significantly faster than the real preprocessor to make a difference, it will be more coupled to the behavior of the compiler and its various options (-I, -idirafter, -isystem, etc), and it probably has to know the compiler's default include directories. Anybody got other ideas? Regarding option 3: If I understand correctly, distcc's pump mode does something similar, so perhaps there is code to borrow or be inspired by? Since a quick fix likely isn't possible in the short term, and I would like to release ccache 3.2 soon, we have to decide whether the direct mode should default to off or on. Please share any opinions! -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] direct mode design bug
Many thanks for the answer! On 5 November 2012 14:53, Andrew Stubbs a...@codesourcery.com wrote: My first reaction to this issue, rightly or wrongly, is that it's more of a documentation issue than a real bug. I mean, it can only occur if two people share a cache, or if the user installs new software and then reuses an old cache. It can happen in other cases as well. Contrieved example, but still: rm -rf subdir file.c config.h echo '#include config.h' file.c mkdir subdir echo '#warning subdir/config.h used' subdir/config.h sleep 1 ccache gcc -Isubdir -c file.c # User: Oops, forgot to create ./config.h. echo '#warning config.h used' config.h sleep 1 ccache gcc -Isubdir -c file.c # User: Wat? Why isn't ./config.h used? For a real life example, see https://bugzilla.samba.org/show_bug.cgi?id=8424#c0. If the documentation simply says that you have to wipe your cache whenever you do that sort of thing then does that solve the problem? It would be nice if ccache were only used and enabled by conscious users who have read and understood the documentation, but in reality that doesn't happen in many cases. For instance, Linux distributions like Fedora install and enable ccache by default (masquerading the system compiler), at least when installing the development environment or similar. That's not surprising given that ccache works very well for most people and that it is advertised as being very safe. There are several other cases where ccache's behavior doesn't fully match that of the real compiler - I'm just a bit worried that the direct mode issue we're discussing perhaps is too much of a behavior mismatch. Hm. Coming to think of it, nothing stops Fedora et al from disabling direct mode by default even if ccache's own default is to enable it. A similar issue, albeit not so interesting, perhaps, is what happens when a user changes some part of the toolchain, but does not alter the gcc binary. Ccache won't notice a new back-end compiler, a new assembler, a new linker, a new default specs file or anything like that. Chances are that any differences in the output are harmless, but the cached objects are technically invalid. Right. However, isn't the the fact that ccache may be affected by toolchain changes much less surprising than the fact that ccache may fail to pick up header files correctly? [In fact, I have a use-case in which I have multiple users sharing a cache, and I wanted to be able to uniquely identify the same toolchain across all the installations. The mtime etc. varies from machine to machine, as might the exact tool mix, so I have some local patches to do a much deeper hash of the toolchain binaries, and include those in the object hashes. Even then, in the interests of performance, those toolchain IDs are cached according to the location and mtime, so changing the binutils will cause temporarily wrong toolchain hashes. Would you be interested in such a feature upstream?] Perhaps, it depends on how intrusive it is and how toolchain-specific it is. 3. ccache could try to imitate what the preprocessor does. Yuck. If you can program a faster preprocessor I'm sure the GCC folks would love to see it. Thankfully, my suggestion wasn't to create a preprocessor substitute. :-) You wouldn't get to dumb much down unless you're fine with running both your own preprocessor and then the real one for the preprocessor mode cache check. Yes, that's of course what I had in mind. Even if you only wanted to look for #include statements you'd still need to evaluate all the #if directives. Not sure about that. I maybe overlook something, but ccache would only have to follow all #include statements and note all header files that don't exist in the include path list. (When #include is used with a #defined token for the filename, fall back to the real compiler.) When considering a potential cache hit, reject it if any of the header files that didn't exist then exist now. Anybody got other ideas? Running the compiler with -v prints the header search directories. You could use that to do your own scan. To use the directories from cpp -v (plus directories from the command line) to do some optimistic validation was my first thought as well, but after thinking more about it I came to the conclusion that it wouldn't buy much safety because no subdirectories will be checked, and you can't tell which subdirectories to check unless you have parsed the #include statements. Also, it would trigger many false negatives. BTW, gcc has an option --trace-includes that might be faster than scanning the preprocessor output, although the compiler still has to do all the same work. Like this: gcc -E hello.c -o /dev/null. How do you use --trace-includes? It doesn't seem to be documented and nothing happens when I try it. Please leave it on. The difference is like night and day, and the bug is rare and avoidable. OK, we so far have one vote for and zero against. Any other?
Re: [ccache] direct mode design bug
On 5 November 2012 16:31, Andrew Stubbs a...@codesourcery.com wrote: Incidentally, you appear to have committed a patch updating the documentation stating that direct mode is off by default, but in the code direct_mode is still true, by default. Yes, I started sketching on disabling it by default but stopped halfway because I couldn't make up my mind at the time. I'll fix it, thanks. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Fwd: [PATCH] Detect __DATE__ and __TIME__ correctly.
On 5 November 2012 13:38, Andrew Stubbs andrew_stu...@mentor.com wrote: That one look right to me. Applied, thanks. I also wrote a bunch of unit tests for check_for_temporal_macros. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Use bash for test.sh.
On 5 November 2012 18:55, Andrew Stubbs a...@codesourcery.com wrote: The test script fails with shells other than bash. At least dash doesn't work. test.sh is intended to work even on Solaris's /bin/sh, which is even more restricted than dash. Current test.sh on master and maint works well with dash for me. Please report any failures you see. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache: FATAL: Could not create xxx.o.tmp.stdout.yyy.zzz (permission denied?)
Hi Lalit, On 2 November 2012 00:40, Lalit Chhabra lchha...@linuxmail.org wrote: My initial guess was that there is some race condition between one users' ccache process doing a clean-up, while another users' compile is going on. But looking at the code, temp files have to be at least one hour old for them to get cleaned up. Are the server and client clocks synchronized? If one of the clients has a clock which is more than one hour off, it may delete the tmp files for another client. Sanity check: Has the filesystem of the cache been full? Do all clients use the same ccache version (3.1.7)? Another odd thing that I have seen (just once so far), is that the cache max size got reduced to 18.8G (which seems to be close to 20G*15/16). Sounds like one of the stat files disappeared. That shouldn't be possible. :-) This caused a lot more of the FATAL: Could not create... messages to appear, until I reset the cache size back to 20G. If this happens reproducably, it would be helpful if you could run strace -f on a compilation with ccache to get some clues on what's happenening. On 9 November 2012 02:19, Lalit Chhabra lchha...@linuxmail.org wrote: I saw this error again today. The $CCACHE_DIR/f/stats file has the max size counter re-set to 65536 (i.e. 1024*1024/16). If the stats file somehow has disappeared, it will be recreated with the default cache size value, which in 3.1.7 is 65536. I am looking at the code a bit more, but do the experts know what conditions would cause the max size counter to get reset? I'm not aware of any bug in that area. Is it possible for the stats file to get deleted? It looks like the subroutine that updates the compile statistics creates a tmp file that it renames to the actual stats file (i.e. atomic rename under NFS). That's correct. So, although not very constructive, I'm tempted to guess that your problem is due to some NFS flakiness. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Single 'source' collection, multiple targets
On 9 November 2012 15:48, Mathias De Maré mathias.de_m...@alcatel-lucent.com wrote: [...] Is there any other way to achieve the desired effect? As you already mentioned, building identical objects only once or changing working directory to the target directory would solve the problem. I have no other ideas, sorry. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] ccache cache dir
Hi Matthew, On 11 November 2012 01:35, Matthew Millar mattmil...@googlemail.com wrote: I've noticed that ccache stores it's cached data within /var/tmp, and I'm wondering whether the developer has chosen not to store the data within /var/cache as the File System Hierarchy standard states? ccache's default cache directory is $HOME/.ccache, not /var/tmp, so either you or your operating system vendor has configured ccache not to use the default. Search for the CCACHE_DIR variable in your shell's startup files or similar. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] [PATCH] Speed up copy4, copy64 on little-endian systems.
On 21 November 2012 18:46, Andrew Stubbs a...@codesourcery.com wrote: The copy64 function implements an endian-safe copy routine for an array of 16 32-bit integers, but this is sub-optimal on machines where the byte-order is already correct. Likewise for copy4. This patch replaces the implementation with a simple memcpy when appropriate, and retains the old implementation otherwise. Note that the compiler will always inline calls to memcpy for small byte-counts, so this is a big win. Applied, thanks! By the way, the patch is missing some headers, so git am doesn't understand it. I suggest using git format-patch instead of git diff for generating future patches. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] basedir testsuite fails as non-root
On 29 November 2012 00:56, Matt m...@use.net wrote: I'm getting this on RHEL 6.1 when trying to build from source RPM, or with a fresh checkout. Thanks in advance for any suggestions/fixes :) mhargett@pyrite:~/rpmbuild/BUILD/ccache-3.1.8$ CC='gcc' ./test.sh starting testsuite base starting testsuite link starting testsuite hardlink starting testsuite cpp2 starting testsuite nlevels4 starting testsuite nlevels1 starting testsuite basedir SUITE: basedir, TEST: set CCACHE_BASEDIR - Expected cache miss to be 1, got 0 cache directory /u/mhargett/rpmbuild/BUILD/ccache-3.1.8/testdir.32495/.ccache primary config /u/mhargett/rpmbuild/BUILD/ccache-3.1.8/testdir.32495/ccache.conf secondary config (readonly) cache hit (direct) 0 cache hit (preprocessed) 0 cache miss 0 preprocessor error 1 This means that the preprocessor failed unexpectedly. Is gcc perhaps a wrapper or a masquerading symlink (what does which gcc say?)? One way of troubleshooting the problem is to locate Result: preprocessor error in testdir.X/ccache.log, then look for the previous Running preprocessor log. After that log line, it says Executing /path/to/gcc ...arguments That's the command that failed. Try running it manually to get some clues. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Bug? ccache: error: Could not stat cache file (permission denied when creating?): No such file or directory
On 7 December 2012 03:51, Michael Ellerman mich...@ellerman.id.au wrote: [...] The error we see is like: ccache: error: Could not stat /scratch/kisskb/ccache/0/d/01dfc7d17f46f8d9b361ba9323e8c9-1921690.o.tmp.stdout.Sprygo.8282 (permission denied when creating?): No such file or directory [...] So basically 11295 is running cleanup_dir() and deleting files that are still being written to. The full log shows 11295 deleting lots of files. From a brief look at the code I don't see any logic which would prevent this from happening, other than that usually there is enough space in the cache so that only old files are deleted. I'm not sure what the best way to avoid this is. The old logic of noticing the stdout is missing and calling failed() at least doesn't break the build. Thanks for the bug report and analysis! I have addressed the issue by detecting failure to create the temporary stdout/stderr files earlier so that no fatal() call is needed for the cleanup scenario. The fix will be included in v3.1.9. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] ccache version 3.1.9 has been released
I'm happy to announce ccache version 3.1.9, which is a bug fix release. Details may be read in the release notes: http://ccache.samba.org/releasenotes.html Source archives are available here: http://samba.org/ftp/ccache/ccache-3.1.9.tar.bz2 http://samba.org/ftp/ccache/ccache-3.1.9.tar.gz http://samba.org/ftp/ccache/ccache-3.1.9.tar.xz The source archives have been signed with my OpenPGP key (BB845E97): http://samba.org/ftp/ccache/ccache-3.1.9.tar.bz2.asc http://samba.org/ftp/ccache/ccache-3.1.9.tar.gz.asc http://samba.org/ftp/ccache/ccache-3.1.9.tar.xz.asc You can run gpg --verify ccache-3.1.9.tar.*.asc to check the source archive integrity after download. Bug fixes: - The EAGAIN signal is now handled correctly when emitting cached stderr output. This fixes a problem triggered by large error outputs from the compiler. - Subdirectories in the cache are no longer created in read-only mode. - Fixed so that ccache's log file descriptor is not made available to the compiler. - Improved error reporting when failing to create temporary stdout/stderr files when executing the compiler. - Disappearing temporary stdout/stderr files are now handled gracefully. Other: - Fixed test suite to work on ecryptfs. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] fcntl - windows build broken
Hi, On 8 January 2013 10:16, Patrick von Reth patrick.vonr...@gmail.com wrote: http://gitweb.samba.org/?p=ccache.git;a=commitdiff ;h=6b6a109779033ffbb8ddab7cca6574f24c2e4bfb broke the windows build because the FD_ flags are not available on windows, see https://groups.google.com/forum/?fromgroups =#!topic/beanstalk-talk/Re9AZ8GbtuA Thanks for the notice. I don't build ccache myself on Windows, so patches are welcome. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Support for Clang plugins (still) not working in 3.1.9
Hi, On 8 January 2013 10:28, Lubos Lunak l.lu...@suse.cz wrote: it looks like my patch 06be1c4cc33bf2def9c6e64483b397128f57dbe3 got included incorrectly into the maint branch. Well, it was included in 3.1.8 as intended. I don't know how exactly the maint branch works, I expect my patch didn't make it into 3.1.8 because you didn't want to include the patch from master that soon The patch was applied directly on maint (the branch for 3.1.x releases), not via master (development branch heading for 3.2). but even though it's been included in 3.1.9, that ccache version still sees such compiles as 'multiple source files', because it requires 8e005b067d8c2423e24ee14ffdee8343f650f1e8 too. Right, I didn't realize that the patch had a depencency, so I applied it to the maint branch because it looked like a good bug fix for a maintenance release. Could you please include that commit as well? Thank you. I've applied it to maint now, so it will be included in 3.1.10. Please try if current maint works like you expect. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] PATCH: Look at include files' mtimes
Hi Justin, I've resurrected these patches to look at files' mtimes and ctimes. [...] I just found out that I forgot to have a look at your patches. Sorry about the delay. I seem fine, so I've applied them. I did need to fix the unit tests since they failed, though. Please have a look and see if it looks all right. Thanks, -- Joel On 25 December 2012 08:18, Justin Lebar justin.le...@gmail.com wrote: Hi, all. I've resurrected these patches to look at files' mtimes and ctimes. Hopefully the three patches here (with their commit messages) don't need further explanation. Note that the second patch here increases safety for everyone, not just those who choose to have mtime matching on. These patches seem to be working, but I'm not seeing a significant speedup on my Mac. I think that may be a separate issue, as this machine isn't particularly good at I/O. I don't have access to my Linux box for a while, so I'd certainly appreciate if someone could verify whether there's a speedup here. I'd also appreciate if some of you could test this patch by turning on CCACHE_SLOPPINESS=file_stat_matches and letting me know if you have any problems. Happy holidays. -Justin On Sun, May 20, 2012 at 4:49 PM, Justin Lebar justin.le...@gmail.com wrote: This patch lets ccache examine an include file's mtime and size in lieu of hashing it, during direct mode. If the mtime and size don't match, we fall back to hashing. The net result is roughly a factor-of-two speedup in ccache hits (*), on my machine. I'm not sure if this is a desirable feature, because obviously mtimes can be tampered with. I didn't provide a way to disable the feature in this patch because, presuming we wanted to take this patch, I'm not sure if we'd want mtime-snooping enabled by default. Since most projects already rely on accurate mtimes in their build systems, turning this on by default doesn't seem particularly outrageous to me. Please let me know what you think about this. Regards, -Justin (*) Experimental procedure: In a Firefox debug objdir (CCACHE_HARDLINK, Linux-64, Ubuntu 12.04, 4 CPU cores), let * Let |nop| be the average real time from a few runs of $ time make -C dom -sj16 when there's nothing to do. * Let |orig| be the average real time from a few runs of $ find dom -name '*.o' time make -C dom -sj16 with ccache master (701f13192ee) (discarding the first run, of course). * Let |mtime| be the real time from the same command as |orig|, but with patched ccache. Speedup is (orig - nop) / (mtime - nop). On my machine, nop = 3.71, orig = 4.88, mtime = 4.31. Yes, our nop build times are atrocious. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Bug compiling package freetype
Hi Giacomo, On 18 March 2013 15:30, Giacomo Comes co...@naic.edu wrote: I have found a bug in ccache as it fails to compile the package freetype. [...] make CC=ccache gcc It looks like a bug in freetype's build system, not in ccache - you will get the same error if you run this: make CC=gcc If you let ccache masquerade as the compiler (see http://ccache.samba.org/manual.html#_run_modes), then it works fine (if you don't specify CC= to make, that is). It also seems like it works if you specify the compiler to configure: CC=ccache gcc ./configure make -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Backing up ccache
Hi Radovan, I was woundering if it is possible to backup the .ccache and use it on another device. Sure, that's possible. However, to get cache hits you need to (by default) use exactly the same compiler (same modification timestamp, same size) in both contexts. I have just tried this but I got build error. What kind of build error? -- Joel On 9 April 2013 23:41, radovan lekanovic lekano...@gmail.com wrote: Hi, I was woundering if it is possible to backup the .ccache and use it on another device. Why do I want to do this? Well I have an raspberry pi and compiling xbmc on the device takes like ~15h. And from time to time the sdcard get corrupted and the need reinstallation of SW. So to avoid the 15h build time each time it would be really nice to backup the .ccache and use it each time an reinstallation has been done. I have just tried this but I got build error. So my question is should this work at all? Thanks! ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
