Re: [ccache] PATCH: Add cwd/hostname to logfiles
On 2010-05-06 18:46, Wilson Snyder wrote: > > I'm debugging a problem which will be the topic of the next > mail. To do this, I needed the cwd of the compile, and > hostname printed in the logfile, otherwise it's impossible > to sort out parallel builds; a patch for this is below. > > I also found the timestamp useful, but have this as a > separate patch as it may be less worth adding and it > probably needs work for portability. I won't loose sleep > if you want to reject it. Thanks, applied the first one with some minor changes. And yes, I'll pass on the second one for now. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Corrupt objects from three colliding compiles
On 2010-05-06 19:14, Wilson Snyder wrote: > [...] > [1273161404.432174,2424 ] === 2010-05-06 11:56:44 === > [1273161404.432636,2424 ] Hostname: maserati > [1273161404.432643,2424 ] Cwd: /nfs/compiling/dir02 > [1273161404.432646,2424 ] Source file: INV.c > [1273161404.432648,2424 ] Object file: INV.o > [1273161404.432657,2424 ] Trying direct lookup > [1273161404.439730,2424 ] Looking for object file hash in > /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest > [1273161404.440126,2424 ] Did not find object file hash in manifest > [1273161404.440134,2424 ] Running preprocessor > [1273161404.469188,2084 ] === 2010-05-06 11:56:44 === > [1273161404.469515,2084 ] Hostname: bird07 > [1273161404.469519,2084 ] Cwd: /nfs/compiling/dir03 > [1273161404.469521,2084 ] Source file: INV.c > [1273161404.469523,2084 ] Object file: INV.o > [1273161404.469532,2084 ] Trying direct lookup > [1273161404.471093,2084 ] Looking for object file hash in > /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest > [1273161404.475427,2084 ] Did not find object file hash in manifest > [1273161404.475434,2084 ] Running preprocessor > [1273161404.487509,2424 ] Got object file hash from preprocessor > [1273161404.492494,2424 ] Object file > /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o not in cache Job 2424 doesn't see the object file in the cache... > [1273161404.492522,2424 ] Running real compiler ...and decides to create it. > [1273161404.511123,21936] === 2010-05-06 11:56:44 === > [1273161404.511506,21936] Hostname: bat05 > [1273161404.511510,21936] Cwd: /nfs/compiling/dir04 > [1273161404.511513,21936] Source file: INV.c > [1273161404.511515,21936] Object file: INV.o > [1273161404.511523,21936] Trying direct lookup > [1273161404.514058,21936] Looking for object file hash in > /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest > [1273161404.517648,21936] Did not find object file hash in manifest > [1273161404.517654,21936] Running preprocessor > [1273161404.523658,2084 ] Got object file hash from preprocessor > [1273161404.542730,2084 ] Created INV.o from > /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o But here, job 2084 sees the object file in the cache ~34 ms before the object file was created... > [1273161404.564688,21936] Got object file hash from preprocessor > [1273161404.569955,2084 ] Added object file hash to > /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest > [1273161404.570387,2084 ] Succeded getting cached result > [1273161404.576412,2424 ] Stored in cache: > /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o ...by job 2424 here. Maybe the clocks are more than 12 ms apart? > [1273161404.588776,21936] Created INV.o from > /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o ***BAD OBJECT*** > [1273161404.596098,2424 ] Created INV.o from > /nfs/ccache-dir/2/b/2/eedf0591498af8e5f3a27fcc89c4f-69975.o > [1273161404.630619,21936] Added object file hash to > /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest > [1273161404.634864,21936] Succeded getting cached result > [1273161404.638443,2424 ] Added object file hash to > /nfs/ccache-dir/0/3/8/f19fe0a63fc088a9c570845a0a4eb-584.manifest > > It looks like the third job's read of the cache is stuck in > between the first's write and the second's. Note that times > may be off by >8ms, this would put the bad object read right > on top of the object update by the second compile when it > hits the NFS server. Since renames should be atomic on NFS, I don't see any problem here, but I'm by no means an NFS expert. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Corrupt objects from three colliding compiles
On 2010-05-06 22:22, Wilson Snyder wrote:
>
> Two followups.
>
> First, though not the problem I'm seeing, as I understand
> it, mkstemp isn't guaranteed to work on NFS v2 mounted
> drives. Thus I suggest including the hostname; there's
> already tmp_string() which is perfect. Patch below.
Applied.
> Second, I'm still not sure of the cause, but if I count
> bytes copied, and fail if zero bytes are moved this works
> around the issue.
Hmm. Have you tried checking gzeof/gzerror after gzread? I guess it
wouldn't hurt to act on read errors detected by zlib:
--- a/util.c
+++ b/util.c
@@ -190,6 +190,18 @@ int copy_file(const char *src, const char *dest,
int compress_dest)
return -1;
}
}
+ if (n == 0 && !gzeof(gz_in)) {
+ int errnum;
+ cc_log("gzread error: %s", gzerror(gz_in, &errnum));
+ gzclose(gz_in);
+ if (gz_out) {
+ gzclose(gz_out);
+ }
+ close(fd_out);
+ unlink(tmp_name);
+ free(tmp_name);
+ return -1;
+ }
gzclose(gz_in);
if (gz_out) {
> [...]
> One thing I wonder is the unlink() before the rename. I
> presume that is needed to prevent permission issues, but I
> wonder if it contributes to the race involving
> host1:unlink(), host1:rename(), host2:unlink(),
> host2:rename(), host3:open().
I'm not sure why there's an unlink before the rename -- it seems
redundant to me. If you remove it, do you see any improvement?
-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
[ccache] amd64 buildworld (FreeBSD 8.0p2) consistently fails with ccache
I've been testing the use of ccache on my FreeBSD machines and I've found that 'make buildworld' will consistently fail when ccache is activated on my amd64 machines, while it performs as expected on my i386 machines. I have tested on both platforms with and without compile options, such as -jX, CFLAGS and CXXFLAGS. With or without these options on i386 it compiles successfully, but with or without these options on amd64 it fails. It appears that it fails at building lib32 related files but I'm not familiar enough with buildworld or ccache to be sure. What more information can I provide toward troubleshooting and hopefully resolving this situation? Cheers, -- Christer Edwards ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] amd64 buildworld (FreeBSD 8.0p2) consistently fails with ccache
On 2010-05-09 19:03, Christer Edwards wrote: > I've been testing the use of ccache on my FreeBSD machines and I've > found that 'make buildworld' will consistently fail when ccache is > activated on my amd64 machines, while it performs as expected on my > i386 machines. > [...] That's strange. Which ccache version are you using? If you're using 2.4, please try 3.0pre0. > What more information can I provide toward troubleshooting and > hopefully resolving this situation? You can enable debug logging by setting CCACHE_LOGFILE to a log file path and then send the log of a failed compilation here. -- Joel ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] amd64 buildworld (FreeBSD 8.0p2) consistently fails with ccache
On Sun, May 9, 2010 at 11:17 AM, Joel Rosdahl wrote:
> That's strange. Which ccache version are you using? If you're using 2.4,
> please try 3.0pre0.
[r...@stringer /usr/src]# ccache -V
ccache version 3.0pre0
It just failed for me again. I did not turn on logging, but this is
where it failed, if this gives you any hints:
===> lib/csu/i386-elf (obj,depend,all,install)
rm -f .depend
CC='/usr/local/libexec/ccache/world-cc' mkdep -f .depend -a
-I/usr/src/lib/csu/i386-elf/../common
-I/usr/src/lib/csu/i386-elf/../../libc/include
/usr/src/lib/csu/i386-elf/crt1.c /usr/src/lib/csu/i386-elf/crti.S
/usr/src/lib/csu/i386-elf/crtn.S
/usr/local/libexec/ccache/world-cc -march=native -O2
-fno-strict-aliasing -pipe -I/usr/src/lib/csu/i386-elf/../common
-I/usr/src/lib/csu/i386-elf/../../libc/include -std=gnu99
-Wsystem-headers -Werror -Wall -Wno-format-y2k -W
-Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes
-Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch
-Wshadow -Wcast-align -Wunused-parameter -Wchar-subscripts -Winline
-Wnested-externs -Wredundant-decls -Wno-pointer-sign -c
/usr/src/lib/csu/i386-elf/crt1.c
{standard input}: Assembler messages:
{standard input}:27: Error: suffix or operands invalid for `mov'
*** Error code 1
Stop in /usr/src/lib/csu/i386-elf.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
I can run it again with logging enabled if you need.
Ooh, and this build was using CFLAGS/CXXFLAGS. I can test again
without those if needed as well.
--
Christer Edwards
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
[ccache] [PATCH] Move mkstemp() replacement to before its use.
$subj 0001-Move-mkstemp-replacement-to-before-its-use.patch Description: Binary data ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] amd64 buildworld (FreeBSD 8.0p2) consistently fails with ccache
On Sun, May 9, 2010 at 11:17 AM, Joel Rosdahl wrote: > You can enable debug logging by setting CCACHE_LOGFILE to a log file > path and then send the log of a failed compilation here. I've enabled logging and I would attach the logfile, but it is 12M total size. I've included the tail end, just leading up to the failure: [78023] === 2010-05-09 14:54:37 === [78023] Base directory: /usr/obj/lib32/usr/src/gnu/lib/csu [78023] Source file: /usr/src/gnu/lib/csu/../../../contrib/gcc/crtstuff.c [78023] Object file: crtend.So [78023] Trying direct lookup [78023] Looking for object file hash in /usr/.ccache/7/d/95616197a1156b28cdd39a8047d5db-9623.manifest [78023] Did not find object file hash in manifest [78023] Running preprocessor [78023] Got object file hash from preprocessor [78023] Object file /usr/.ccache/c/a/512e749085241d05f1a72bfb1fa910-77439.o not in cache [78023] Running real compiler [78023] Stored in cache: /usr/.ccache/c/a/512e749085241d05f1a72bfb1fa910-77439.o [78023] Created crtend.So from /usr/.ccache/c/a/512e749085241d05f1a72bfb1fa910-77439.o [78023] Added object file hash to /usr/.ccache/7/d/95616197a1156b28cdd39a8047d5db-9623.manifest [78040] === 2010-05-09 14:54:37 === [78040] Base directory: /usr/obj/lib32/usr/src/lib/csu/i386-elf [78040] Compiler option -E is unsupported [78040] Failed; falling back to running the real compiler [78047] === 2010-05-09 14:54:37 === [78047] Base directory: /usr/obj/lib32/usr/src/lib/csu/i386-elf [78047] Source file: /usr/src/lib/csu/i386-elf/crt1.c [78047] Object file: crt1.o [78047] Trying direct lookup [78047] Looking for object file hash in /usr/.ccache/c/9/ea7ce3c6c0dcbf8e347e950e9ff6ea-1903.manifest [78047] Did not find object file hash in manifest [78047] Running preprocessor [78047] Got object file hash from preprocessor [78047] Object file /usr/.ccache/f/a/86dcc9067c2597ce7a0dbd718899ac-23567.o not in cache [78047] Running real compiler [78047] Compiler gave exit status 1 This build had no CFLAGS or CXXFLAGS and exited at the same point. What other information can I provide? [r...@stringer /usr/src]# ccache -s cache directory /usr/.ccache cache hit (direct) 871 cache hit (preprocessed)1039 cache miss 30393 called for link 1342 multiple source files 5 compile failed 2 not a C/C++ file3316 no input file 2 files in cache 30393 cache size 107.8 Mbytes max cache size 1.0 Gbytes -- Christer Edwards ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
