[ccache] Fw: [NFS] nfs/mmap/rename file corruption
In summary: to make ccache work safely on an NFS filesystem, the filesystem must be exported with no_subtree_check. Otherwise data can be lost when it renames recently-written files. :-( -- Begin forwarded message: Date: 27 Aug 2003 21:37:38 -0400 From: Trond Myklebust trond.mykleb...@fys.uio.no To: Martin Pool m...@sourcefrog.net Cc: n...@lists.sourceforge.net Subject: Re: [NFS] nfs/mmap/rename file corruption == Martin Pool m...@sourcefrog.net writes: - ccache runs distcc with output to a temporary file - distcc opens, mmaps, writes to, munmaps, and closes the temporary file - distcc exits - ccache renames the temporary file to its proper location in the ccache - ccache opens the file read only, and reads from it Is this a rename from one directory to the other? If so, are you using the 'no_subtree_check' option on the server? Without the latter option enabled, I would indeed expect the behaviour that you describe. Cheers, Trond -- Martin
[ccache] lilypond doesn't compile with ccache (DEPENDENCIES_OUTPUT)
On 10 Aug 2003, Koblinger Egmont egm...@uhulinux.hu wrote: ./out/axis.o: axis.cc include/axes.hh include/string.hh \ include/arithmetic-operator.hh include/flower-proto.hh include/real.hh \ include/string-handle.hh include/string-handle.icc \ include/string-data.hh include/string-data.icc include/international.hh \ include/compare.hh include/string.icc but when ccache is used, this file has one more line: ./out/axis.o: axis.cc include/axes.hh include/string.hh \ include/arithmetic-operator.hh include/flower-proto.hh include/real.hh \ include/string-handle.hh include/string-handle.icc \ include/string-data.hh include/string-data.icc include/international.hh \ include/compare.hh include/string.icc ./out/axis.o: /home/egmont/.ccache/tmp.stdout.boci.5961.ii All the other *.dep files have a similar extra line appended which point to some ccache-internal filename, which means that somehow ccache fails to remain transparent for lilypond. The command that generates this .dep file is: DEPENDENCIES_OUTPUT=./out/axis.dep ./out/axis.o c++ -c -DHAVE_CONFIG_H -DSTRING_UTILS_INLINED -Iinclude -I./out -I../flower/include -I../flower/./out -O2 -finline-functions -g -O2 -finline-functions -g -Wall -W -Wmissing-prototypes -Wconversion -o out/axis.o axis.cc Thankyou for reporting this. It looks like the problem is that DEPENDENCIES_OUTPUT is read by both cpp and cc1, so it gets dependencies for both .h-.i and .i-.o. Almost certainly you only want the first one. You can work around this in lilypond by using -MD -MF rather than DEPENDENCIES_OUTPUT. I think ccache (and distcc) ought to strip DEPENDENCIES_OUTPUT before invoking the compiler, so that you only get the first dependencies line. Please take a look at this problem. It seems to me that ccache should fix the DEPENDENCIES_OUTPUT files after running gcc itself, but real developers should know it better than me :-) Another trivial workaround might be to let an existing DEPENDENCIES_OUTPUT env variable simply imply CCACHE_DISABLE. A small patch is attached. I don't think such severe measures are needed. Have a nice holiday, Thankyou! How did you know? :-) -- Martin
[ccache] ccache doesn't handle .i files
If I run ccache 2.2 on a .i file, it thinks it's not a C/C++ file and it doesn't cache it. This could be naively fixed by updating check_extension(), but it would be far better for ccache to know that it doesn't need to run cpp on such files. distcc does so. In particular this means that if ccache is run by distcc or distccd it will never hit. This is not a terribly big deal but might be nice to fix. -- Martin
[ccache] sharing a cache
Enno Rehling wrote: we're considering using ccache and sharing the cache between developers here. I've followed the instructions in the manpage, but ccache doesn't seem to realize that the object file is already in the cache, and compiles a new one for every user. - The output files have the same md5sum. - The input files have the same datestamp (I rsynced them) - I'm compiling from different home directories and machines Does any of this play into ccache's decision about whether two compilations are equal? Apart from the command line, what parameters go into the hash? When both developers use the same machine, they can share the cache. If they are on separate machines, it seems that they can't. Is there a way around this? Enno. Hi Enno, you get these misses only if different hosts are used? In that case, have you verified that the same compiler (its size and modification time) is used on the separate machines? It does not matter that the trees are not in the same place unless your code for instance uses #defines to compile the path into the preprocessed output. The timestamp of the source code does not matter at all. Don't give up, ccache and distcc are wonderful :) Regards, Anders Furuhed
[ccache] Re: How to debug ccache
Sorry, I don't understand what you're trying to say. Sorry for my english but may i try again. the ccache program works perfectly for me. on a machine first compile something it add its work to the cache and the second time a compile the same source it takes the objects from the cache what speed up the compilation time. prima job but now i have other machine and want to have also the speed up in the network setup so if i compile a source on a machine the first time it at his work to the cache that is for every pc the same nfs shared map. ( man ccache SHARING A CACHE is ok) an if i compile the source on a other machine than it takes the object form the network shared cache, but thats my problem a simple hello_world.c program works fine, this say that i have configured the ccache good. but it doesnt work correctly with complexer program's , and like you say al the headers, compilers options, etc etc i dont know all the parameters, must be the same, or there is no hit for it . my question is, how can i let say set a DEBUG_OPTION flag to see all these options and see thats ccache saying there is header.h or gcc-options-x that is not the same like in the ccache files i must recompile this. of course i can make a copie of the disk from my first pc to the second pc this weekend change the ip an go on that must be good, but if i later have installed something newer on the one and not on the other pc than i have the problem again and it should be easy perhaps to backtrace it with toggling a debug option or so in ccache. now i go working but i do a emerge world because it is a gentoo distro to be sure it al the tools are in sync. i hope you understand my problem ? Try ls -l /usr/bin/gcc or whatever compiler name is appropriate. Do you have *exactly* the same headers on both machines? If not, you are likely to get misses. Kris Coryn
[ccache] Intel compiler
Or perhaps ccache should just cache stdout. distcc sends it across the network alongside stderr. I'm not sure why writing to stdout forces a cache miss. It's because is also very hard to accurately cache both stderr and stdout because of the timing of lines between them. To really accurately cache them ccache would need to record not only what came out of stdout and stderr, but exactly what order the lines came in, so when it is replayed the output looks the same. I'd be happy to add a CCACHE_IGNORE_STDOUT option which would help for the intel compiler. I'm not convinced it is really worth actually cacheing stdout, as the output will probably look pretty strange if we just put all the stdout at the start or end. Cheers, Tridge
[ccache] Intel compiler
I was just about to start using ccache with distcc and didn't know the best way to go about, only to discover a just-in-time release that makes it easy! Thanks! Last week we started to use the Intel compiler in parallel with gcc. The Intel compilers produce a number of cool but unnecessary messages about optimizations that take place. Some of these messages can't be turned off and they appear on stdout. The result is that ccache discards the result instead of putting it into the cache. The Intel guys said they would consider introducing an option that turns the remarks off. I'm considering whether to make a patch for our own use that accepts a parameterized CCACHE_WHATEVER regexp that is used to filter the stdout result before deciding not to cache. There's no way we will let something get in the way of us and ccache :). However, it feels like a temporary workaround rather than a nice feature to push for inclusion. If someone has an opinion or wants to have a patch, please mail me (I'm subscribed to the list). Thanks again for today's release! Anders
[ccache] ccache 2.2.1
I released ccache 2.0 a couple of days ago, and have done two minor releases since then to fix small bugs that were noticed in the 2.0 release. Main changes are: - minor Makefile install fixes - fixed some compilation warnings - added -C option for 'clear cache completely' - added CCACHE_CC option - added CCACHE_RECACHE option - added CCACHE_EXTENSION option - added CCACHE_HASHDIR option - embed the hostname in temporary file names for NFS sharing - better handling of the various -M compiler options - handle attempts to compile to stdout - set default max cache size to 1GB - better signal handling Cheers, Tridge
[ccache] PRB: ccache directory not available
ccache: version 2.2 OS: Windows XP Professional cygwin: latest version Hello, first of all: we like ccache - its a great utlility - thanks a lot! We're using ccache to share compiler output between several people. All works fine - but if the ccache file server is not available - it would be nice if ccache compile local instead of generating an error message like the following: ccache: failed to create //ad26080/ccache (No such host or network path) Perhaps a new flag or environement variable or per default fall back to compile on local host without try fetching/saving files to ccache directory. I searched the man pages for such an option - but without success! Perhaps you can add a feature like this in the next release. best regards Heiko Elger ARBURG GmbH + Co Heiko Elger - Softwareentwicklung - / - Research and Development - Arthur-Hehl-Strasse D-72290 Lossburg Tel.: +49 (0) 7446 33-3659 Fax.: +49 (0) 7446 33-3365 mailto:heiko_elgernos...@arburg.com http://www.arburg.com Bitte entfernen Sie NOSPAM von der Email-Adresse Please remove NOSPAM from the email address
[ccache] CCACHE_PREFIX included in hash
On 31 Oct 2003, Brian Poynor bri...@redback.com wrote: Is CCACHE_PREFIX intentionally included in the hash? Yes. For my project it means that objects built locally never match objects built remotely with CCACHE_PREFIX=distcc, even though they are identical, and I'd like to avoid duplicate cache entries. If interested, I have a patch which avoids hashing it. Currently it is implemented as an alternate environment variable, CCACHE_NHPREFIX. Please post it. -- Martin linux.conf.au -- Adelaide, January 2004
[ccache] ccache and distcc not hitting caches
Hello, I've just started to use this cool tool, but came up with the following problem: If I run ccache from my local machine I build a cache, where if I rebuild the same [unchanged] files, every file is hit in the cache. If I then set the following environment variable: CCACHE_PREFIX=distcc and re-run the same build [using ONLY my machine for building], not one file is hit in the cache. My setup is as follows: SunOS Release 5.8 Is there an option, or a patch I could use to remedy this problem? I would like to do simple builds locally, while more indepth builds using multiple hosts. Thanks, Mike
[ccache] Re: disk full and distcc/ccache weirdness
On 23 Sep 2002, Tim Potter t...@samba.org wrote: Hi Tridge. I've just had some distcc/ccache wierdness that caused my ccache to be corrupted. One of the machines in our DISTCC_HOSTS list filled up /tmp and a whole bunch of object files were created containing text like: DOMAIN_GRP:t(239,22)=(239,21) domain_grp_member_info:T(239,23)=s257name:(211,2),0,2048;attr:(0,11),2048,8;; DOMAIN_GRP_MEMBER:t(239,24)=(239,23) time_info:T(239,25)=s4time:(0,4),0,32;; The exact error returned was: Linking bin/smbd lib/fsusage.o: file not recognized: File truncated I haven't been able to reproduce it by filling up /tmp again and rebuilding so I'm not sure what sort of bug it is. )-: The bug is that execute() in execute.c only checks WEXITSTATUS(), not WIFSIGNALLED(). If the compiler exits with a signal ccache will proceed as if it succeeded, and therefore probably corrupt its cache. Old versions of distcc would raise SIGABRT if something strange like ENOSPC when writing a file happened. (New ones are practically perfect in every possible way. :-) -- Martin
[ccache] Patch: Xcode compatibility
On Fri, Nov 21, 2003 at 03:28:32PM -0800, Sean Gies wrote: 2. Added a check to the argument processing logic to handle this sort of invocation: /usr/share/bin/ccache /usr/bin/distcc /usr/bin/gcc-3.3 ... In order to handle multiple compiler versions, Xcode uses full paths to invoke an explicit version of distcc or gcc. This confused ccache because it thought the gcc-3.3 argument to distcc was a source file. Can you avoid that problem by using CCACHE_PREFIX? -Brian
[ccache] Patch: Xcode compatibility
Looks like mailman ate my attachment. Here's the patch again: EOT
Index: ccache.c
===
RCS file: /cvsroot/ccache/ccache.c,v
retrieving revision 1.91
diff -u -r1.91 ccache.c
--- ccache.c28 Sep 2003 04:47:59 - 1.91
+++ ccache.c21 Nov 2003 23:03:32 -
@@ -64,6 +64,11 @@
/* a list of supported file extensions, and the equivalent
extension for code that has been through the pre-processor
+
+ i = C
+ mi = Objective-C
+ ii = C++
+ mii = Objective-C++
*/
static struct {
char *extension;
@@ -74,14 +79,21 @@
{m, mi},
{cc, ii},
{CC, ii},
+ {cp, ii},
+ {CP, ii},
{cpp, ii},
{CPP, ii},
{cxx, ii},
{CXX, ii},
{c++, ii},
{C++, ii},
+ {M, mii},
+ {mm, mii},
+ {MM, mii},
{i, i},
{ii, ii},
+ {mi, mi},
+ {mii, mii},
{NULL, NULL}};
/*
@@ -616,6 +628,7 @@
int found_c_opt = 0;
int found_S_opt = 0;
struct stat st;
+ int file_missing;
char *e;
stripped_args = args_init(0, NULL);
@@ -729,11 +742,21 @@
/* if an argument isn't a plain file then assume its
an option, not an input file. This allows us to
cope better with unusual compiler options */
- if (stat(argv[i], st) != 0 || !S_ISREG(st.st_mode)) {
+ if ((file_missing = stat(argv[i], st)) != 0 ||
!S_ISREG(st.st_mode)) {
args_add(stripped_args, argv[i]);
continue;
}
+ /* if the first argument contains distcc and the second
argument
+ is a regular, executable file, then assume it's the compiler
+ being specified to distcc:
+ ccache distcc gcc ... */
+ if (i == 1 strstr(argv[0], distcc) != NULL
!file_missing
+ S_ISREG(st.st_mode) (st.st_mode (S_IXUSR | S_IXGRP |
S_IXOTH))) {
+ args_add(stripped_args, argv[i]);
+ continue;
+ }
+
if (input_file) {
if (check_extension(argv[i], NULL)) {
cc_log(multiple input files (%s and %s)\n,
EOT
-Sean
On Nov 21, 2003, at 3:28 PM, Sean Gies wrote:
Here's a patch I created so that I can use ccache with Apple's Xcode
IDE.
1. Added more extensions to better support Objective-C and
Objective-C++.
2. Added a check to the argument processing logic to handle this sort
of invocation:
/usr/share/bin/ccache /usr/bin/distcc /usr/bin/gcc-3.3 ...
In order to handle multiple compiler versions, Xcode uses full paths
to invoke an explicit version of distcc or gcc. This confused ccache
because it thought the gcc-3.3 argument to distcc was a source file.
-Sean
___
ccache mailing list
ccache@lists.samba.org
http://lists.samba.org/mailman/listinfo/ccache
[ccache] Patch: Xcode compatibility
Here's a patch I created so that I can use ccache with Apple's Xcode IDE. 1. Added more extensions to better support Objective-C and Objective-C++. 2. Added a check to the argument processing logic to handle this sort of invocation: /usr/share/bin/ccache /usr/bin/distcc /usr/bin/gcc-3.3 ... In order to handle multiple compiler versions, Xcode uses full paths to invoke an explicit version of distcc or gcc. This confused ccache because it thought the gcc-3.3 argument to distcc was a source file. -Sean
[ccache] PRB: using ccache and distcc: multiple input files
ccache: version 2.2 distcc: version 2.11 OS: Windows XP Professional cygwin: latest version Hello, I' using the calling distcc with the full compiler name cause of using different cross compiler versions for example compiling distcc distribution (using native gcc it's just the same behaviour): make -j10 CC=distcc c:/programme/cygwin/bin/gcc -- All works fine If I use distcc and ccache make -j10 CC=ccache distcc c:/programme/cygwin/bin/gcc -- I never will get any ccache results. See following log file snippet of ccache. here is a snippet of the ccache log file: - snip snip -- multiple input files (c:/programme/cygwin/bin/gcc and src/backoff.c) multiple input files (c:/programme/cygwin/bin/gcc and src/climasq.c) multiple input files (c:/programme/cygwin/bin/gcc and src/clinet.c) multiple input files (c:/programme/cygwin/bin/gcc and src/clirpc.c) multiple input files (c:/programme/cygwin/bin/gcc and src/compile.c) - snip snip -- I'm not sure what's wrong, but cause calling distcc without the full path name, all works fine, I assume that this is perhaps a problem of distcc. Using gcc is just an example - I want to use discc for cross compiling with the gnu cross compiler ccpentium from Wind River System. And cause we have serveral different compiler versions we have to use the full name of the compiler. I'm not sure why - but setting environment variable or make variable CCACHE_PREFIX=distcc all works fine. make -j10 CC=ccache distcc c:/programme/cygwin/bin/gcc -- Error make -j10 CC=ccache distcc c:/programme/cygwin/bin/gcc CCACHE_PREFIX=distcc -- OK But why? What's the difference? Perhaps anyone can give me a hint Mit freundlichen Gruessen best regards Heiko Elger ARBURG GmbH + Co Heiko Elger - Softwareentwicklung - / - Research and Development - Arthur-Hehl-Strasse D-72290 Lossburg Tel.: +49 (0) 7446 33-3659 Fax.: +49 (0) 7446 33-3365 mailto:heiko_elgernos...@arburg.com http://www.arburg.com Bitte entfernen Sie NOSPAM von der Email-Adresse Please remove NOSPAM from the email address
[ccache] ccache - -frepo
On 1 Oct 2003, andrew andrew.sla...@discus.anu.edu.au wrote:
Hi,
Should ccache always ignore compiles with (g++) -frepo, since object
files might have to be rebuilt at link time, due to c++ template object
code placement? I bypassed caching for -frepo'd source files so my
template stuff would compile.
Yes, I think it should.
:)
andrew
diff -Naur ccache-2.3/ccache.c ccache-2.3-nofrepo/ccache.c
+++ ccache-2.3-nofrepo/ccache.c 2003-10-01 14:06:12.0 +1000
@@ -630,6 +630,7 @@
/* these are too hard */
if (strcmp(argv[i], -fbranch-probabilities)==0 ||
+ strcmp(argv[i], -frepo) == 0 ||
strcmp(argv[i], -M) == 0 ||
strcmp(argv[i], -MM) == 0 ||
strcmp(argv[i], -x) == 0) {
___
ccache mailing list
ccache@lists.samba.org
http://lists.samba.org/mailman/listinfo/ccache
--
Martin
[ccache] ccache 2.3 released
Hi, I've just released ccache 2.3. Grab it from http://ccache.samba.org/ The checksum generated by 2.3 is not compatible with previous ccache versions. (I'm using ccache since 1.9 and it was always compatible.) When I compile something, upgrade ccache from 2.2 to 2.3, and recompile it again, there's no cache hit, everything is placed again. Is it a known issue (well, should be mentioned on the homepage) or a bug? -- Egmont
[ccache] ccache 2.3 released
I've just released ccache 2.3. Grab it from http://ccache.samba.org/ Major changes: * Added CCACHE_UMASK option * Added support for compilation of .i files * Fixed bug with DEPENDENCIES_OUTPUT flag * Added support for more -Mx options * Added separate stderr channel for distcc * Improved test suite Cheers, Tridge
[ccache] Re: [distcc] ccache doesn't handle .i files
On Wed, 24 Sep 2003 ee...@gmx.net wrote: + close(creat(path_stderr, 0644)); Should be 0666, the rest is the job of umask. -- Egmont
[ccache] Re: [distcc] ccache doesn't handle .i files
+ } else {
+ copy_file(input_file, path_stdout);
+ x_asprintf(command, touch %s, path_stderr);
+ system(command);
+ free(command);
+ }
Huh?
i know, it's been a while since i last did c. i'm sure there are
faster ways to provide the empty preprocessor error file, or maybe
it's not even needed if the hashing gets moved into the
conditional. i was just curious if it would be that hard to get it to
work, and it seemed to do when i tested it locally.
isidor
[ccache] Re: [distcc] ccache doesn't handle .i files
On 23 Sep 2003, ee...@gmx.net wrote:
on Mon, 21 Jul 2003 23:52:45 -0700, Martin Pool wrote:
If I run ccache 2.2 on a .i file, it thinks it's not a C/C++ file and
it doesn't cache it.
This could be naively fixed by updating check_extension(), but it
would be far better for ccache to know that it doesn't need to run cpp
on such files. distcc does so.
In particular this means that if ccache is run by distcc or distccd it
will never hit.
This is not a terribly big deal but might be nice to fix.
imho it's a better way to combine ccache and distcc to ccache on the
distcc hosts after distributing, as compiler upgrades are handled
correctly this way.
That was what I meant by running it from distccd.
maybe this will do(I know it's kludgy but to nicely implement it some
redesign of ccache would be needed):
@@ -64,19 +70,23 @@
static struct {
char *extension;
char *i_extension;
+ int preprocess;
Okay.
+ if (preprocess) {
+ /* now the run */
args_add(args, -E);
args_add(args, input_file);
status = execute(args-argv, path_stdout, path_stderr);
@@ -327,6 +339,12 @@
failed();
}
+ } else {
+ copy_file(input_file, path_stdout);
+ x_asprintf(command, touch %s, path_stderr);
+ system(command);
+ free(command);
+ }
Huh?
--
Martin
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.samba.org/archive/ccache/attachments/20030924/d47b8340/attachment.bin
