Bug#853834: Ccache storing and restoring file paths that may no longer be correct

2017-02-18 Thread Ben Hutchings 
On Sat, 2017-02-18 at 17:27 +0100, Joel Rosdahl wrote:
> On 18 February 2017 at 00:54, Ben Hutchings  wrote:
[...]
> > It seems to be assumed that if the object files are identical then so any
> other output files
> > will also be identical, and that's not the case.
> 
> I wouldn't call it an assumption but rather an unforeseen side effect of
> the "names of included files are no longer included in the hash of the
> compiler's
> preprocessed output" feature that snuck into ccache 3.3.

Oh I see.  I didn't realise that the preprocessed output was being
filtered before hashing.

Ben.

-- 
Ben Hutchings
Knowledge is power.  France is bacon.



signature.asc
Description: This is a digitally signed message part

Bug#853834: Ccache storing and restoring file paths that may no longer be correct

2017-02-18 Thread Joel Rosdahl 
On 18 February 2017 at 00:54, Ben Hutchings  wrote:
> I've also found that disabling direct mode (CCACHE_NODIRECT=1) is a
workaround,
> at least in this case.

Yes. When the direct mode is disabled, dependency files are created fresh
by the preprocessor instead of taken from the cache.

> The problem seems to be that a direct mode manifest contains a hash of
> the source and object files, but not any other associated output files.

Hm, not quite. The manifest (looked up from a hash of source code file +
command line arguments) contains a mapping from a set of include file
hashes to output file names (used for object file, dependency file and a
handful other types of output files). The output file names are in turn
based on a hash of the preprocessed output of the compilation. There's no
hash of the object file itself or any other output file. However, what I
call "output file name" above is called "object" in the manifest source
code, which is misleading. I'll change it! :-) See
https://ccache.samba.org/manual.html#_the_direct_mode if you're interested
in more details on how the direct mode works.

> It seems to be assumed that if the object files are identical then so any
other output files
> will also be identical, and that's not the case.

I wouldn't call it an assumption but rather an unforeseen side effect of
the "names of included files are no longer included in the hash of the
compiler's
preprocessed output" feature that snuck into ccache 3.3.

-- Joel

Bug#853834: Ccache storing and restoring file paths that may no longer be correct

2017-02-17 Thread Ben Hutchings 
Control: tag -1 upstream fixed-upstream

On Fri, 2017-02-17 at 16:41 +, Ben Hutchings wrote:
> On Fri, 2017-02-17 at 14:12 +, Ben Hutchings wrote:
> [...]
> > I also just hit this with the linux package.  On some architectures it
> > builds kernels with and without the PREEMPT_RT patches.  I found that a
> > dependency list generated from a PREEMPT_RT build may be reused for a
> > standard build; this caused a build failure because the patched source
> > directory had not yet been created.
> > 
> > The two commands are:
> > 
> >   gcc -Wp,-MD,scripts/genksyms/.parse.tab.o.d -Iscripts/genksyms -Wall 
> > -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer 
> > -std=gnu89  -I/build/linux-4.9.10/scripts/genksyms -c -o 
> > scripts/genksyms/parse.tab.o scripts/genksyms/parse.tab.c
> >   gcc -Wp,-MD,scripts/genksyms/.parse.tab.o.d -Iscripts/genksyms -Wall 
> > -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer 
> > -std=gnu89  -I/build/linux-4.9.10/debian/build/source_rt/scripts/genksyms 
> > -c -o scripts/genksyms/parse.tab.o scripts/genksyms/parse.tab.c
> 
> [...]
> 
> By the way, these two commands are run with different working
> directories.  At first glance the documentation seems to say that the
> working directory is included in the hash by default, but actually it's
> only included when generating debug info and there's no debug prefix
> map.

I've also found that disabling direct mode (CCACHE_NODIRECT=1) is a
workaround, at least in this case.

The problem seems to be that a direct mode manifest contains a hash of
the source and object files, but not any other associated output files.
  It seems to be assumed that if the object files are identical then so
any other output files will also be identical, and that's not the case.

I'm attaching a patch that adds a test case for this bug.

However, having checked upstream, I've now verified that upstream
version 3.3.4 fixes the bug (at least, it passes this test case).  This
appears to have only bug fixes and documentation improvements, so I
think it should be eligible for a freeze exception.

Ben.

-- 
Ben Hutchings
Any sufficiently advanced bug is indistinguishable from a feature.
Author: Ben Hutchings 
Description: Add test case for direct mode and extra output files
Bug-Debian: https://bugs.debian.org/853834

--- a/test.sh
+++ b/test.sh
@@ -2231,6 +2231,27 @@ EOF
 if [ -n "$data" ]; then
 test_failed "$manifest contained ignored header: $data"
 fi
+
+# -
+TEST "Identical source and object files, different dependencies"
+
+mkdir -p a/include b/include
+echo '#include "header.h"' > a/source.c
+backdate a/include/header.h
+ln a/source.c b/source.c
+ln a/include/header.h b/include/header.h
+
+(cd a && $CCACHE_COMPILE -MD -I$PWD/include -c source.c)
+(cd b && $CCACHE_COMPILE -MD -I$PWD/include -c source.c)
+expect_equal_object_files a/source.o b/source.o
+if cmp -s a/source.d b/source.d; then
+	test_failed "Dependencies unexpectedly identical"
+fi
+(cd a && $CCACHE_COMPILE -MD -I$PWD/include -c source.c)
+expect_equal_object_files a/source.o b/source.o
+if cmp -s a/source.d b/source.d; then
+	test_failed "Wrong dependencies restored from cache"
+fi
 }
 
 # =


signature.asc
Description: This is a digitally signed message part

Bug#853834: Ccache storing and restoring file paths that may no longer be correct

2017-02-17 Thread Joel Rosdahl 
Hi,

Upstream release 3.3.4 contains a fix for this problem. I uploaded a Debian
package of it to unstable just now and will look into requesting a freeze
unblock.

Regards,
-- Joel

Bug#853834: Ccache storing and restoring file paths that may no longer be correct

2017-02-17 Thread Ben Hutchings 
On Fri, 2017-02-17 at 14:12 +, Ben Hutchings wrote:
[...]
> I also just hit this with the linux package.  On some architectures it
> builds kernels with and without the PREEMPT_RT patches.  I found that a
> dependency list generated from a PREEMPT_RT build may be reused for a
> standard build; this caused a build failure because the patched source
> directory had not yet been created.
> 
> The two commands are:
> 
>   gcc -Wp,-MD,scripts/genksyms/.parse.tab.o.d -Iscripts/genksyms -Wall 
> -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89  
> -I/build/linux-4.9.10/scripts/genksyms -c -o scripts/genksyms/parse.tab.o 
> scripts/genksyms/parse.tab.c
>   gcc -Wp,-MD,scripts/genksyms/.parse.tab.o.d -Iscripts/genksyms -Wall 
> -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89  
> -I/build/linux-4.9.10/debian/build/source_rt/scripts/genksyms -c -o 
> scripts/genksyms/parse.tab.o scripts/genksyms/parse.tab.c
[...]

By the way, these two commands are run with different working
directories.  At first glance the documentation seems to say that the
working directory is included in the hash by default, but actually it's
only included when generating debug info and there's no debug prefix
map.

Ben.

-- 
Ben Hutchings
Any sufficiently advanced bug is indistinguishable from a feature.



signature.asc
Description: This is a digitally signed message part

Bug#853834: Ccache storing and restoring file paths that may no longer be correct

2017-02-17 Thread Ben Hutchings 
Control: retitle -1 ccache may wrongly output a cached dependency list
Control: severity -1 important

This is at least important, maybe serious.

On Wed, 1 Feb 2017 12:58:03 +0200 Marko Lindqvist  wrote:
> Package: ccache
> Version: 3.3.3-1
> 
> Building SDL2_mixer-2.0.1
> (https://www.libsdl.org/projects/SDL_mixer/release/) multiple times
> with ccache enabled, following reproducible problem occurs.
> 
> 1) Build SDL2_mixer in first source tree. I've used separate builddir
> when doing so.
> 2) Delete source and build trees used in step 1 so that files in them
> will not be found later
> 3) Rebuild SDL2_mixer in another (differently named) tree
> 
>  Latter build fails with error messages complaining about files
> missing in the first, deleted, tree. Such paths should be stored
> nowhere as we are talking about completely clean builds here. Ccache
> is the thing that does store things between builds, so seems like
> likely culprint. Further, issue does not occur if one does 'ccache -C'
> before the second build.
> 
>  I'll send the exact error message once I have opportunity to
> reproduce this again. Anyway, it was about .deps (again; ccache
> involvement makes sense as those are created by compiler for which
> ccache caches stuff)

I also just hit this with the linux package.  On some architectures it
builds kernels with and without the PREEMPT_RT patches.  I found that a
dependency list generated from a PREEMPT_RT build may be reused for a
standard build; this caused a build failure because the patched source
directory had not yet been created.

The two commands are:

  gcc -Wp,-MD,scripts/genksyms/.parse.tab.o.d -Iscripts/genksyms -Wall 
-Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89  
-I/build/linux-4.9.10/scripts/genksyms -c -o scripts/genksyms/parse.tab.o 
scripts/genksyms/parse.tab.c
  gcc -Wp,-MD,scripts/genksyms/.parse.tab.o.d -Iscripts/genksyms -Wall 
-Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89  
-I/build/linux-4.9.10/debian/build/source_rt/scripts/genksyms -c -o 
scripts/genksyms/parse.tab.o scripts/genksyms/parse.tab.c

and the output in .parse.tab.o.d is (for the latter):

scripts/genksyms/parse.tab.o: scripts/genksyms/parse.tab.c \
 /usr/include/stdc-predef.h /usr/include/assert.h /usr/include/features.h \
 /usr/include/x86_64-linux-gnu/sys/cdefs.h \
 /usr/include/x86_64-linux-gnu/bits/wordsize.h \
 /usr/include/x86_64-linux-gnu/gnu/stubs.h \
 /usr/include/x86_64-linux-gnu/gnu/stubs-64.h /usr/include/stdlib.h \
 /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h \
 /usr/include/x86_64-linux-gnu/bits/waitflags.h \
 /usr/include/x86_64-linux-gnu/bits/waitstatus.h \
 /usr/include/x86_64-linux-gnu/sys/types.h \
 /usr/include/x86_64-linux-gnu/bits/types.h \
 /usr/include/x86_64-linux-gnu/bits/typesizes.h /usr/include/time.h \
 /usr/include/endian.h /usr/include/x86_64-linux-gnu/bits/endian.h \
 /usr/include/x86_64-linux-gnu/bits/byteswap.h \
 /usr/include/x86_64-linux-gnu/bits/byteswap-16.h \
 /usr/include/x86_64-linux-gnu/sys/select.h \
 /usr/include/x86_64-linux-gnu/bits/select.h \
 /usr/include/x86_64-linux-gnu/bits/sigset.h \
 /usr/include/x86_64-linux-gnu/bits/time.h \
 /usr/include/x86_64-linux-gnu/sys/sysmacros.h \
 /usr/include/x86_64-linux-gnu/bits/pthreadtypes.h /usr/include/alloca.h \
 /usr/include/x86_64-linux-gnu/bits/stdlib-bsearch.h \
 /usr/include/x86_64-linux-gnu/bits/stdlib-float.h /usr/include/string.h \
 /usr/include/xlocale.h /usr/include/x86_64-linux-gnu/bits/string.h \
 /usr/include/x86_64-linux-gnu/bits/string2.h \
 /build/linux-4.9.10/debian/build/source_rt/scripts/genksyms/genksyms.h \
 /usr/include/stdio.h /usr/include/libio.h /usr/include/_G_config.h \
 /usr/include/wchar.h /usr/lib/gcc/x86_64-linux-gnu/6/include/stdarg.h \
 /usr/include/x86_64-linux-gnu/bits/stdio_lim.h \
 /usr/include/x86_64-linux-gnu/bits/sys_errlist.h \
 /usr/include/x86_64-linux-gnu/bits/stdio.h

Ben.

-- 
Ben Hutchings
Any sufficiently advanced bug is indistinguishable from a feature.



signature.asc
Description: This is a digitally signed message part

Bug#853834: Ccache storing and restoring file paths that may no longer be correct

2017-02-01 Thread Marko Lindqvist 
Package: ccache
Version: 3.3.3-1

Building SDL2_mixer-2.0.1
(https://www.libsdl.org/projects/SDL_mixer/release/) multiple times
with ccache enabled, following reproducible problem occurs.

1) Build SDL2_mixer in first source tree. I've used separate builddir
when doing so.
2) Delete source and build trees used in step 1 so that files in them
will not be found later
3) Rebuild SDL2_mixer in another (differently named) tree

 Latter build fails with error messages complaining about files
missing in the first, deleted, tree. Such paths should be stored
nowhere as we are talking about completely clean builds here. Ccache
is the thing that does store things between builds, so seems like
likely culprint. Further, issue does not occur if one does 'ccache -C'
before the second build.

 I'll send the exact error message once I have opportunity to
reproduce this again. Anyway, it was about .deps (again; ccache
involvement makes sense as those are created by compiler for which
ccache caches stuff)


 - ML