[ccache] Ccache version 4.9.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.9.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_9_1

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.9 has been released

[Joel Rosdahl via ccache]

Ccache version 4.9 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_9

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Looking to contribute to ccache

[Joel Rosdahl via ccache]

Hi Stefan,

On Tue, Sep 19, 2023, at 20:27, Stefan Constantin via ccache wrote:
> I would like to contribute some time/code to ccache if there are areas
> where help is needed.

Nice!

> For example some time ago I added the smallest of features (expanding an
> env var in ccache's config) and the maintainer mentioned that it will do
> for the time being but he would like to eventually have a solution for all
> config keys, not just the one I needed.
> Is that something I could work on if no one else is doing it?

I later made that change in 3c7fa42f147f8323deed84a87f29db483b319de3[1].

[1]: 
https://github.com/ccache/ccache/commit/3c7fa42f147f8323deed84a87f29db483b319de3

> Is there a list of open urgent issues where it would be better to look?

There is no other list than the issues list in GitHub. You could perhaps have a
look at the "good first issue"[2] and "help wanted"[3] labels. There are no 
urgent
issues as far as I know. 

[2]: 
https://github.com/ccache/ccache/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
[3]: 
https://github.com/ccache/ccache/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.8.3 has been released

[Joel Rosdahl via ccache]

Ccache version 4.8.3 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_8_3

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.8.2 has been released

[Joel Rosdahl via ccache]

Ccache version 4.8.2 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_8_2

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.8.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.8.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_8_1

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.8 has been released

[Joel Rosdahl via ccache]

Ccache version 4.8 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_8

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.7.4 has been released

[Joel Rosdahl via ccache]

Ccache version 4.7.4 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_7_4

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.7.3 has been released

[Joel Rosdahl via ccache]

Ccache version 4.7.3 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_7_3

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.7.2 has been released

[Joel Rosdahl via ccache]

Ccache version 4.7.2 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_7_2

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.7.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.7.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_7_1

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.7 has been released

[Joel Rosdahl via ccache]

Ccache version 4.7 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_7

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.6.3 has been released

[Joel Rosdahl via ccache]

Ccache version 4.6.3 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_6_3

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.6.2 has been released

[Joel Rosdahl via ccache]

Ccache version 4.6.2 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_6_2

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.6.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.6.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_6_1

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache changes file permissions unexpectedly

[Joel Rosdahl via ccache]

Aleksey Midenkov wrote:

> After the software upgrade I noticed my IDE cannot compile a project
> after it was built from the terminal. It turned out that build
> directory now contains objects with such strange permissions:
>
> -r--r--r--
>
> When I removed ccache from the compiling command it created normal rw
> files. My umask is 0002

Do you use the hard link mode? When the hard link mode is enabled, ccache
intentionally makes hard-linked output files (currently only object files)
read-only. From :

"If the resulting file is modified, the file in the cache will also be
modified since they share content, which corrupts the cache entry. As of
version 4.0, ccache makes stored and fetched object files read-only as a
safety measure."

So if your build fails because files have read-only permissions, it sounds like
ccache has saved you from corrupting the cache since something tried to write to
a file that is hard-linked to a file in ccache's cache. The solution to this is
to either not use tools that write to object files after compilation or to
disable the hard link mode.

If you don't use the hard link mode then it sounds like a ccache bug. If so,
please describe a way to reproduce the problem.

> I suspected it is because of the new `umask` setting in ccache which works
> differently by default (which is a really bad design choice).

There is no design decision to change how umask works, and I'm not aware of any
change in behavior between ccache 3.x and 4.x.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.6 has been released

[Joel Rosdahl via ccache]

Ccache version 4.6 is now available. Release archives can be found here:

https://ccache.dev/download.html

Release notes:

https://ccache.dev/releasenotes.html#_ccache_4_6

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.5.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.5.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

Bug fixes:

  - Fixed entry_size field for result entries. This bug affected the
recompression feature (“-X”/“--recompress”) in ccache 4.5.

  - The actual compression level is now once again stored in the cache entry
header.

  - Corrected error handling for non-constructible secondary storage backends.
For instance, this avoids a crash when a Redis server can't be reached.

Contributors (in alphabetical order) to this release:

  - Joel Rosdahl

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.5 has been released

[Joel Rosdahl via ccache]

Ccache version 4.5 is now available. Release archives can be found here:

https://ccache.dev/download.html

New features:

  - Made various improvements to the cache entry format. Among other things,
the header now contains a creation timestamp and a note of the ccache
version used to create the entry. The cache entry checksum has also been
upgraded to use 128-bit XXH3 instead 64-bit XXH3.

  - Added support for cache namespaces. If a namespace configured, e.g. using
“CCACHE_NAMESPACE=some_string”, the namespace string will be added to the
hashed data for each compilation. This will make the associated cache
entries logically separate from cache entries in other namespaces, but they
will still share the same storage space. Cache entries can also be
selectively removed from the primary cache with the new command line option
“--evict-namespace”, potentially in combination with “--evict-older-than”.

  - Made HTTP keep-alive configurable, defaulting to off for now.

  - Added support for rewriting absolute path to Clang option
“--gcc-toolchain”.

Compatibility notes:

  - A consequence of the changed cache entry format is that ccache 4.5 will not
share cache entries with earlier versions. Different ccache versions can
however still use the same cache storage without any issues.

Bug fixes:

  - Fixed a problem with special characters in the user info part of URLs for
HTTP storage.

  - Fixed win32 log messages about file locks.

  - Fixed debug directory handling on Windows.

  - The hard link and file clone features are now disabled when secondary
storage is used since they only work for the local primary cache.

Contributors (in alphabetical order) to this release:

  - Alexander Falkenstern
  - Gregor Jasny
  - Joel Rosdahl
  - Luboš Luňák
  - Russell McClellan

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Upgrading Ccache -> cache invalid?

[Joel Rosdahl via ccache]

On Tue, 12 Oct 2021 at 18:35, Sebert, Holger.ext via ccache
 wrote:
> I would like to upgrade Ccache from version 3.4.1 to 4.2 (or newer).
>
> Are the caches from the old version still valid or should I clean them up 
> before upgrading?

The cached results from the old version will not be used by the new version. You
can clear the cache if you want, but you don't have to since the new version
will remove the old results eventually when the cache size limit is reached
anyway.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.4.2 has been released

[Joel Rosdahl via ccache]

Ccache version 4.4.2 is now available. Release archives can be found here:

https://ccache.dev/download.html

Bug fixes:

  - Fixed a bug introduced in 4.4 where ccache could produce false direct cache
hits in some situations if it decides to disable the direct mode
temporarily (e.g. due to “too new header” file).

Test improvements:

  - Use shell builtin pwd command for basedir test.

  - Cope with CC being a wrapper script that uses ccache.

Contributors (in alphabetical order) to this release:

  - Joel Rosdahl
  - Kira Bruneau

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.4.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.4.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

New features:

  - The secondary storage statistics section of “-s/--show-stats” is now shown
only if it's non-empty or with two verbose options.

  - Added display of statistics counters for misses. Previously they were only
implicit in the “hits + misses” sums.

Bug fixes:

  - Fixed spurious crashes when using the HTTP or Redis backend and the remote
connection hung up.

  - Made sure to always store configuration origin value.

Build improvements:

  - The matching version of lld is now used for Clang.

  - The standard linker is now used if IPO (LTO) is enabled.

  - Disabled IPO (LTO) for MinGW toolchains since they seem to be generally
broken.

  - Fixed build errors with Clang on Windows.

Test improvements:

  - Fixed .incbin test with newer binutil versions.

  - Fixed basedir test suite failure when using a symlinked CWD.

  - Improved output of differing text files on failure.

Contributors (in alphabetical order) to this release:

  - Gregor Jasny
  - Joel Rosdahl
  - Orgad Shaneh

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.3 has been released

[Joel Rosdahl via ccache]

Ccache version 4.3 is now available. Release archives can be found here:

https://ccache.dev/download.html

New features:

  - Ccache now ignores the Clang compiler option “-ivfsoverlay” and its
argument if you opt in to “ivfsoverlay sloppiness”. This is useful if you
use Xcode, which uses a virtual file system (VFS) for things like combining
Objective-C and Swift code.

  - When using “-P” in combination with “-E”, ccache now reports this as
“called for preprocessing” instead of “unsupported compiler option”.

  - Added support for “-specs file.specs” and “--specs file.specs” without an
equal sign between the arguments.

Bug fixes:

  - "Split dwarf" code paths are now disabled when outputting to “/dev/null”.
This avoids an attempt to delete “/dev/null.dwo”.

  - Made the “stat”/“lstat” wrapper function for Windows treat pending deletes
as missing files.

  - Fixed a bug that made ccache process header files redundantly for some
relative headers when using Clang.

  - The object path is now included in the input hash when using “-fprofile-
arcs” (or “--coverage”) since the object file embeds a “.gcda” path based
on the object file path.

Build improvements:

  - Added an “ENABLE_DOCUMENTATION” build option (default: true) that can be
used to disable the build of documentation.

  - Fixed detection of pthread features.

  - Quote CMake variables expansions to avoid errors when
“${CMAKE_C_FLAGS_RELWITHDEBINFO}” or “${CMAKE_CXX_FLAGS_RELWITHDEBINFO}”
expands to the empty string.

Contributors (in alphabetical order) to this release:

  - Bin Li
  - Joel Rosdahl
  - Josh Triplett
  - Nicholas Hutchinson
  - Olle Liljenzin
  - Peter Steinberger
  - Petr Štetiar
  - Yoshimasa Niwa

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Long pauses due to automatic cache clean-up

[Joel Rosdahl via ccache]

On Sun, 18 Apr 2021 at 10:04, R. Diez via ccache  wrote:
> First of all, I have been using ccache for years. Many thanks for this great 
> tool.

You're welcome.

> I have come to the conclusion that these particular pauses are due to ccache
> automatically pruning its cache.

I haven't really observed pauses due to cleanup myself since I have
used SSDs for
a long time, but yes, they will occur with a slow enough disk (or loaded system)
and many parallel ccache invocations.

> I was surprised that completely clearing the cache with option '--clear' can
> take a very long time, 12 minutes the last time around, because ccache
> probably does not need to calculate anything beforehand, just delete all
> files. But I haven't benchmarked the filesystem itself, so perhaps it's just
> Linux being slow deleting so many small files.

Yes, what you're seeing is a slow disk (or filesystem) in combination with many
files in the cache directory. You will get similar performance from "rm -r".
Upgrading to ccache 4.x will actually help since then only one file will be
stored per cache entry instead of up to seven files (but more likely two) with
ccache 3.x, so there will be fewer files to consider and remove when cleaning or
clearing.

> What happens if you are building in parallel, with "make -j32", and all 32
> concurrent instances decide to clean the cache at the same time? Will all of
> them attempt to delete the same files at the same time?

Yes, that could happen. A ccache invocation that writes the result of a
compilation to one of the 16 cache buckets will start cleaning up that bucket if
the cache size threshold has been reached. If another parallel compilation
finishes and writes to the same bucket before the first has finished cleaning
then it too will start cleaning and so on. There is no coordination between such
cleanups.

This is how the automatic cleanup has worked since ccache was initially created
in 2002. I have ideas on how this can be improved but I don't have much time and
interest to work on ccache these days so things are unfortunately slow. (I no
longer use ccache personally except for building ccache...)

> I wonder if there is a way to increase the number of buckets, from 16 to say
> to 1024. There is not reason to have so few of them.

There is a reason, and it's spelled "backward compatibility". Other minor
reasons also exist.

> I have also being thinking of triggering a manual cache clean-up on start-up
> [...]
> Is that a good strategy? Or are there better ways?

I think it's a good strategy given that you have a problem with automatic
cleanups. You could consider disabling automatic cleanup completely (set
max_size to 0) and run "CCACHE_MAXSIZE=20G ccache -c" or so each night or more
often. The limit will then only affect that ccache invocation; it won't be
stored in the config file. Also, you don't have to parse the ccache
configuration since the cron job has the definition of what the max cache size
should be.

> Is there some script able to parse the ccache config file that I could use as
> an example?

Not that I know of. You can query the size with "ccache -k max_size", but it
will include a size suffix that you need to parse if you want to make it
generic.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.2.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.2.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

Bug fixes:

  - Ccache now only “dup2”s stderr into “$UNCACHED_ERR_FD” for calls to the
preprocessor/compiler. This works around a complex bug in interaction with
GNU Make, LTO linking and the Linux PTY driver.

  - Fixed detection of color diagnostics usage when using “-Xclang
-fcolor-diagnostics” options.

  - The “-frecord-gcc-switches” compiler option is now handled correctly to
avoid false positive cache hits.

  - Made it possible for per-compilation debug log files to be written in most
argument processing error scenarios. Previously, ccache would only write
debug log files if the argument processing phase was successful.

  - Made ccache bail out on too hard Clang option “-gen-cdb-fragment-path”.

  - The “run_second_cpp” made is now enforced on macOS if “-g” is used since
newer Clang versions on macOS produce different debug information when
compiling preprocessed code.

  - Made ccache only reject “-f(no-)color-diagnostics” for a known GCC
compiler. This fixes a problem when using said option with Clang on macOS.

  - Implemented a better “stat”/“lstat” wrapper function for Windows.

  - Fixed a bug where ccache could return stale cache results on Windows.

  - Fixed handling of long command lines on Windows.

Portability and build improvements:

  - Build configuration scripts now probe for atomic increment as well. This
fixes a linking error on Sparc.

  - An existing CMake log message level is now used when warning about not
finding asciidoc.

  - Added support for building ccache with xlclang++ on AIX 7.2.

  - Fixed assertion in the "Debug option" test.

  - Made build configuration skip using ccache when building with MSVC.

  - Upgraded to doctest 2.4.6. This fixes a build error with glibc >= 2.34.

Documentation:

  - Fixed markup of “compiler_type” value "other".

  - Fixed markup of “debug_dir” documentation.

  - Fixed references to the “extra_files_to_hash” configuration option.

Contributors (in alphabetical order) to this release:

  - Abubakar Nur Khalil
  - Aleksander Salwa
  - Evangelos Foutras
  - Joel Rosdahl
  - Jon Petrissans
  - Nicholas Hutchinson
  - Pawel Krysiak
  - Sergei Trofimovich
  - Thomas Otto

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] gmake and ccache conspiring together in creating gremlins

[Joel Rosdahl via ccache]

On Sun, 7 Feb 2021 at 12:26, Sam Varshavchik via ccache
 wrote:

> I've been chasing down non-deterministic, random build fails.

Thanks for the thorough investigation!

> So, now what? Both ccache, and the linker are contributing to this situation.
> ccache leaks the file descriptor (to distcc apparently?).

Yes, or rather for anyone who wants to use it. Icecream also uses it.

> The linker invokes make for its own use, but it doesn't sanitize the
> environment. make closes the job server file descriptors, but not removing
> them from MAKEFLAGS.

>From my perspective, I don't see anything wrong with ccache dup-ing stderr – I
guess the bug can be triggered by the user running something like

  exec 3>&2

in the shell as well.

I can think of an easy workaround for the problem: only dup2 stderr when running
the preprocessor or compiler. Done in
.

-- Joel

On Sun, 7 Feb 2021 at 12:26, Sam Varshavchik via ccache
 wrote:
>
> I've been chasing down non-deterministic, random build fails. make
> randomly blows up with
>
> write error: stdout
>
> A lot of stracing, experimentation, and pains, determined that this is
> due to a sequence of the following events:
>
> 1) how make uses the job server file descriptor
>
> 2) an intentional leak of the standard error file descriptor in ccache
>
> 3) LTO linking results in linker recursively invoking make (that was a
> new one to me)
>
> 4) A rather nuanced behavior of the Linux PTY driver that can be best
> explained with the following short demo:
>
> #include 
> #include 
> #include 
> #include 
> #include 
>
> int main()
> {
> struct stat state_buf;
>
> for (int i=0; i<8; ++i)
> {
> if (fstat(i, _buf) == 0)
> {
> printf("%d: %d/%d\n", i,
>(int)state_buf.st_dev,
>(int)state_buf.st_ino);
> }
> }
>
> printf("NON BLOCKING: %s\n", fcntl(0, F_GETFL) & O_NONBLOCK ? 
> "y":"n");
>
> fcntl(2, F_SETFL, O_NONBLOCK);
>
> printf("NON BLOCKING: %s\n", fcntl(0, F_GETFL) & O_NONBLOCK ? 
> "y":"n");
> return 0;
> }
>
> I'm sshed into my server, and the output of the above is, for me:
>
> 0: 25/3
> 1: 25/3
> 2: 25/3
> NON BLOCKING: n
> NON BLOCKING: y
>
> My stdin, stdout, and stderr is the same dev/ino, and setting non-blocking
> mode one any one of them puts them all in non-blocking mode. I'm setting non-
> blocking on standard error, and my standard input also becomes non-blocking.
>
> After sifting through sources, and straces, this is what I see is happening.
>
> Starting with make, and how it sets up its job file descriptors:
>
> jobserver_parse_auth (const char *auth)
> {
>   /* Given the command-line parameter, parse it.  */
>   if (sscanf (auth, "%d,%d", _fds[0], _fds[1]) != 2)
>
>   // …
>
>   /* Make sure our pipeline is valid, and (possibly) create a duplicate pipe,
>  that will be closed in the SIGCHLD handler.  If this fails with EBADF,
>  the parent has closed the pipe on us because it didn't think we were a
>  submake.  If so, warn and default to -j1.  */
>
>   if (!FD_OK (job_fds[0]) || !FD_OK (job_fds[1]) || make_job_rfd () < 0)
> }
>
> The TLDR of the above: make reads the job server's file descriptors
> from the MAKEFLAGS environment variable, then checks here if they
> actually exist. If they don't exist, make will create the job server
> pipe. Important: by default they will be file descriptors 3 and 4.
> This becomes a key player in this mystery, a little bit later.
>
> When make spawns a child job (other than a recursive make) the job
> server file descriptors get closed (marked O_CLOEXEC before the actual
> execve), but MAKEFLAGS remains in the environment. They remain open
> for a recursive make call, and the recursive instance picks up the
> ball from here.
>
> The problem with this approach is that if, somehow, a child process
> runs make, and there happens to be a pair of file descriptors here,
> make will assume that they're the job server file descriptors.
>
> Another thing that make does is that it sets one of the file
> descriptors to non-blocking mode. This is in jobserver_setup:
>
>   /* When using pselect() we want the read to be non-blocking.  */
>   set_blocking (job_fds[0], 0);
>
> Now we get to ccache which, amongst things, does the following, in its
> ccache.cpp:
>
> // Make a copy of stderr that will not be cached, so things like distcc can
> // send networking errors to it.
> static void
> set_up_uncached_err()
> {
>   int uncached_fd =
> dup(STDERR_FILENO); // The file descriptor is intentionally leaked.
>   if (uncached_fd == -1) {
> log("dup(2) failed: {}", strerror(errno));
> throw Failure(Statistic::internal_error);
>   }
>
>   Util::setenv("UNCACHED_ERR_FD", fmt::format("{}", uncached_fd));
> }
>
> TLDR: it intentionally leaks a file 

[ccache] Ccache version 4.2 has been released

[Joel Rosdahl via ccache]

Ccache version 4.2 is now available. Release archives can be found here:

https://ccache.dev/download.html

New features:

  - Improved calculation of relative paths when using “base_dir” to also
consider canonical paths (i.e. paths with dereferenced symlinks) as
candidates.

  - Added a “debug_dir” (“CCACHE_DEBUGDIR”) configuration setting for
specifying a directory for files written in debug mode.

  - Added support for compiler option “-x cuda”, understood by Clang.

  - The value of the “SOURCE_DATE_EPOCH” variable is now only hashed if it
potentially affects the output from ccache. This means that ccache now
(like before version 4.0) will be able to produce cache hits for source code
that doesn't contain “__DATE__” or “__TIME__” macros regardless of the
value of “SOURCE_DATE_EPOCH”.

Bug fixes:

  - Fixed a bug where a non-Clang compiler would silently accept the Clang-
specific “-f(no-)color-diagnostics” option when run via ccache. This
confused feature detection made by e.g. CMake.

  - Improved creation of temporary files on Windows. Previously, ccache would
in practice reuse temporary filenames on said platform resulting in various
problems with parallel builds.

  - Fixed creation of parent directories when creating a lock file on Windows.

  - Fixed a race condition related to removal of temporary files.

  - Improved calculation of directory name for a Windows-style path.

  - A compilation result is now not stored in the cache if an included
preprocessed header file is too new. This fixes a bug where the content of
a newly created preprocessed header file could be missing from the hash,
resulting in a false positive cache hit.

  - Fixed calculation of the split DWARF filename for an object filename with
zero or multiple dots.

  - Fixed retrieval of the object file the destination is “/dev/null”.

Portability and build improvements:

  - Additional compiler flags like “-Wextra -Werror” are now only added when
building ccache in developer mode.

  - The developer build mode no longer enables “-Weverything” for Clang.

  - “_XOPEN_SOURCE” is now defined appropriately on FreeBSD to fix missing
declaration of “isascii”.

  - Improved detection of buildability of BLAKE3 assembler files.

  - Disabled build of inode cache code on OSes without
“pthread_mutexattr_setpshared”, such as OpenBSD.

  - Made static linking the default for a Windows MinGW build.

  - Removed legacy fallback replacements of “mkstemp” and “realpath”.

  - Improved detection of SSE/AVX support.

  - Improved detection of support for the AVX2 target attribute.

  - Configuration scripts now try to detect and enable BLAKE3's Neon support.

  - Made it possible to run the integration test suite on macOS.

  - Fixed building of 32-bit unit tests on macOS.

  - Made it possible to compile ccache for C++17.

  - Fixed printing of 64-bit “time_t” on 32-bit architectures like RISCV32.

  - Made sure to only use ASCII characters in the manual's AsciiDoc source code
to make it possible to generate documentation in non-UTF8 locales.

  - Upgraded to optional-lite 3.4.0, fmt 7.1.3, doctest 2.4.4 and zstd 1.4.8.

  - Took steps towards being able to run the test suite on Windows.

Documentation:

  - Improved wording of “compiler_check” string values.

  - Improved documentation of compression levels and the “-X/--recompress”
option.

  - Improved consistency of terms in the manual.

  - HTML documentation is now built and installed by default if possible.

  - Fixed incorrect documentation of configuration option “cache_dir”.

  - Added hint on how to link statically with libzstd.

  - Mention that ccache requires the “-c” compiler option.

Contributors (in alphabetical order) to this release:

  - Alexander Lanin
  - Azat Khuzhin
  - Erik Flodin
  - Joel Rosdahl
  - Ka Ho Ng
  - Khem Raj
  - Nicholas Hutchinson
  - Orgad Shaneh
  - Sergei Trofimovich
  - Stuart Henderson

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Migrating to ccache 4: a "drop-in replacement"?

[Joel Rosdahl via ccache]

On Mon, 7 Dec 2020 at 22:55, Steffen Dettmer via ccache
 wrote:
> I just tested with ccache 3.7.12 and "ccache -s" that it seems to work as I
> expect

Cool, my memory was wrong about this! Or alternatively, it might be that the
final implementation of the config file handling did not become what I first
had planned since the manual has documented the behavior I described since
added in 2012:
https://github.com/ccache/ccache/commit/99396f88#diff-426db03993d0f56a62aa3f8976b22d2424d152c49b1cb4c893e71b151d0f846cR242-R246

Hopefully improved in .

> Thank you for sharing your expertise! Although not ccache specific,
> I'm glad to get your input because your input usually is just great!
> [...systemd...]

It was just a tongue-in-cheek jab at your tongue-in-cheek characterization of
XDG. I have nothing in particular against not using systemd, but I find it
convenient that it has become the norm (and as a Debian developer I didn't vote
against choosing systemd). Let's stop here.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Migrating to ccache 4: a "drop-in replacement"?

[Joel Rosdahl via ccache]

On Thu, 3 Dec 2020 at 11:28, Steffen Dettmer wrote:
> Is upgrading to ccache 4 a "drop-in replacement" even if sharing the cache
> directory with still running ccache 3 versions?
> [...]
> Does it really make new versions a "plug-in replacement"?
> [...]
> So in short: I would not need to change anything at all. Just deploy it. Is 
> this correct?

Yes to all instances of the question.

> [...] even cleanup should work: it is backward compatible. If I understand
> correctly, this means as long as the new ccache version runs from time to
> time, it will cleanup both old and new files and avoid ccache directory grow
> (much) beyond max_size.

There is no difference in how cleanup works between old and new versions. In
other words, both new and old ccache versions will clean up files generated by
both new and old versions.

> [...] release notes tell that if ~/.ccache/ccache.conf exists, it will be
> used (release notes are not clear for me if only as fallack or in any case
> when it exists).

See
.

> Also CCACHE_CONFIGPATH could be set to point to it if needed, but it is not
> needed.

There is almost never a reason to set CCACHE_CONFIGPATH. Instead, just set
CCACHE_DIR to ~/.ccache if you want the old behavior.

>   $ grep -v '^#' ~/.ccache/ccache.conf
>   cache_dir=/...shared_local_path.../tmp/ccache

This won't have any effect. cache_dir only has an effect if set in the
secondary (system-wide read-only) configuration file (typically
/etc/ccache.conf) or in the environment variable form ($CCACHE_DIR).

>   hard_link=false

(This is already the default.)

> NB: Our reference system is "Devuan 2.0 ASCII", a Debian fork without
> SystemD.

Sorry to hear that (SCNR).

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.1 has been released

[Joel Rosdahl via ccache]

Ccache version 4.1 is now available. Release archives can be found here:

https://ccache.dev/download.html

New features:

  - Symlinks are now followed when guessing the compiler. This makes ccache
able to guess compiler type “GCC” for a common symlink chain like this:
“/usr/bin/cc” → “/etc/alternatives/cc” → “/usr/bin/gcc” → “gcc-9” →
“x86_64-linux-gnu-gcc-9”.

  - Added a new “compiler_type” (“CCACHE_COMPILERTYPE”) configuration option
that allows for overriding the guessed compiler type.

  - Added support for caching compilations with “-fsyntax-only”.

  - Added a command line option “--config-path”, which specifies the
configuration file to operate on. It can be used instead of setting
“CCACHE_CONFIGPATH” temporarily.

Bug fixes:

  - The original color diagnostics options are now retained when forcing
colored output. This fixes a bug where feature detection of the
“-fcolor-diagnostics” option would succeed when run via ccache even though
the actual compiler doesn’t support it (e.g. GCC <4.9).

  - Fixed a bug related to umask when using the “umask” (“CCACHE_UMASK”)
configuration option.

  - Allow “ccache ccache compiler ...” (repeated “ccache”) again.

  - Fixed parsing of dependency file in the “depend mode” so that filenames
with space or other special characters are handled correctly.

  - Fixed rewriting of the dependency file content when the object filename
includes space or other special characters.

  - Fixed runtime detection of AVX2 support, not relying on the sometimes
broken “__builtin_cpu_support” routine.

  - Added missing parameters to a log call, thus avoiding a crash when it is
found out at runtime that file cloning is unsupported by the OS.

Portability and build fixes:

  - The ccache binary is now linked with “libatomic” if needed. This fixes
build problems with GCC on ARM and PowerPC.

  - Fixed build of BLAKE3 code with Clang 3.4 and 3.5.

  - Fixed “use of undeclared identifier 'CLONE_NOOWNERCOPY'” build error on
macOS 10.12.

  - Fixed build problems related to missing AVX2 and AVX512 support on older
macOS versions.

  - Fixed static linkage with libgcc and libstdc++ for MinGW and made it
optional.

  - Fixed conditional compilation of “robust mutex” code for the inode cache
routines.

  - Fixed badly named man page filename (“Ccache.1” instead of “ccache.1”).

  - Disabled some tests on ancient Clang versions.

Other improvements and fixes:

  - The man page is now built by default if the required tools are available.

  - Use CMake “A2X_EXE” variable instead of hardcoded “a2x”.

  - Improved build errors when building ccache with very old compiler versions.

  - Fall back to version “unknown” when Git is not installed.

  - Documented the relationship between “CCACHE_DIR” and “-d/--directory”.

  - Fixed incorrect reference and bad markup in the manual.

Contributors (in alphabetical order) to this release:

  - Alexander Lanin
  - Erik Flodin
  - Joel Rosdahl
  - Orgad Shaneh
  - Yoshimasa Niwa

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Ccache version 4.0 has been released

[Joel Rosdahl via ccache]

 available.

  - Fixed test suite problems on macOS.

  - Disabled hardlink tests on AFS since it lacks such support.

  - Disabled read-only tests on file systems that lack such support.

Contributors (in alphabetical order) to this release:

  - Alexander Korsunsky
  - Alexander Lanin
  - Anders F Björklund
  - Arne Hasselbring
  - Breno Guimaraes
  - Chris Burr
  - Cristian Adam
  - Deepak Yadav
  - Enrico Sorichetti
  - Erik Flodin
  - Gregor Jasny
  - Harsh Shandilya
  - Igor Pylypiv
  - Ivan Volnov
  - Joel Rosdahl
  - Loïc Yhuel
  - Luboš Luňák
  - Matt Whitlock
  - Mizuha Himuraki
  - Olle Liljenzin
  - Paul Bunch
  - Paul Fultz II
  - Pavol Sakac
  - Philipp Storz
  - Rafael Kitover
  - Ryan Egesdahl
  - Steffen Dettmer
  - Sumit Jamgade
  - Thomas Otto

Thanks!

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.12 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.12 is now available. Release archives can be found here:

https://ccache.dev/download.html

Bug fixes:

  - Coverage files (“.gcno”) produced by GCC 9+ when using “-fprofile-dir=dir”
are now handled gracefully by falling back to running the compiler.

  - Fixed writing to a log file larger than 2 GiB when running ccache
compiled in
32-bit mode.

Other:

  - Improved documentation about sharing a cache on NFS.

  - Fixed test case failures with old objdump versions.

  - Fixed test case failures with GCC 4.4.

Contributors (in alphabetical order) to this release:

  - Deepak Yadav
  - Joel Rosdahl

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Increase in cache size on every run

[Joel Rosdahl via ccache]

Hi Vijay,

On Fri, 28 Aug 2020 at 13:17, Vijay Tandeker  wrote:
> Every night I do the compilation. I don't know but whenever I do the
> compilation, cache size increases by few GBs (sometimes by few MBs).

That sounds perfectly normal for a build that generates parts of its source
code with volatile things like timestamps or usage of __DATE__ in the source
code, etc.

> And every after 10-14 days, it hits the max and I need to again clean it
> using -C option.

No, you don't need to use -C since ccache performs automatic cleanup (see
https://ccache.dev/manual/3.7.11.html#_automatic_cleanup).

> My understanding of ccache is:
> 1) in case of HIT, don't add anything. It should not increase the cache size.

Correct.

> 2) in case of MISS, if no entry is present, add new object file in the cache.
> It should increase the "cache size" by size_of_object_file.

Correct.

> 3) in case of MISS, If old entry is present, delete the old object file and
> add new object file. It should increase (or decrease) the cache size by
> (size_of_new_entry - size_of_old_entry). [...] ld entry should get replaced
> by new entry and should not consume space.

No, that's not how ccache works. A new entry (due to a cache miss) does not
replace a previous entry. See
https://ccache.dev/manual/3.7.11.html#_how_ccache_works for details.

By the way, you asked me more or less the same question in private mail more
than two years ago (2018-02-21, subject "query on ccache miss"). See that
conversation for further information.

Regards,
Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Speeding up builds with clang-tidy, clang-analyzer and iwyu

[Joel Rosdahl via ccache]

On Mon, 17 Aug 2020 at 15:34, Christian Ledergerber
 wrote:
> [...] The point is: I think this is not a coner-case, but people use this
> configuration abundantly.

I just wanted to stress the point that tools need to handle all configurations
correctly, regardless of whether they are popular or not.

> > 3. The method is also brittle since the clang-tidy step will be skipped if
> > the source is built (and therefore cached) by a normal non-clang-tidy
> > build.
>
> This I do not understand. Would this not be caught by the extra files to
> cache? So: if the ccache caches also the configuration of clang-tidy then
> there cannot be a cache hit due to binaries built without using clang-tidy?

The case I was thinking about is this: if a user sets CCACHE_EXTRAFILES (or
extra_files_to_hash in the configuration) to the .clang-tidy configuration file
and builds without running clang-tidy then user will have circumvented the
logic by placing a result in the cache without being stopped by clang-tidy.
This likely won't happen in practice; I'm just trying to find out if there are
edge cases where the assumptions can break down.

> Thanks for this comment! I feel like for easy integration into cmake (and
> other tools) one would need another command line flag which allows adding
> extra files to cache. E.g. --add-extra-files=file1;file2;file3.

That would be possible.

The main thing I think feels problematic with your idea is that it will require
tight coupling between CMake and ccache, something I wouldn't like to have if I
were a CMake maintainer. But since I'm not, I'll leave that discussion to you
and them.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Speeding up builds with clang-tidy, clang-analyzer and iwyu

[Joel Rosdahl via ccache]

On Fri, 14 Aug 2020 at 08:39, Christian Ledergerber
 wrote:
>
> I have created the following issue for cmake to launch the discussion:
> https://gitlab.kitware.com/cmake/cmake/-/issues/21092

When you described your idea I didn't think you meant making changes
for incorporation in the upstream CMake software but to some local
CMake version. I think that it will be hard to implement it in a
generic enough way for CMake since

1. The solution will require a specific ccache version to work at all.
2. The method of using ccache to probe whether clang-tidy should be
run won't work for users who don't use warnings-as-errors. Also, how
would CMake know that warnings-as-errors is used? It could be part of
WarningsAsErrors in the config file or it could be on the command line
but only for some warnings, etc. Sounds brittle and complex.
3. The method is also brittle since the clang-tidy step will be
skipped if the source is built (and therefore cached) by a normal
non-clang-tidy build.

(But if you apply your changes to a local CMake version you can make
any assumptions you want, of course.)

I spontaneously think that it would be a better idea to either (a)
stop running clang-tidy for a normal build and instead run it as a
special build once in a while, or (b) improve
https://github.com/ejfitzgerald/clang-tidy-cache if you find it
deficient in some way (I know nothing about it), or (c) implement
clang-tidy support in ccache or another compiler cache program.

Just a comment on your CMake GitLab issue:

> 2. Add Class.clang-tidy to configuration of extra_files_to_hash of ccache 
> using the command line syntax: --set-config=KEY=VALUE see 
> https://ccache.dev/manual/latest.html

That's not a good idea. --set-config is just a convenience alternative
to manually editing the ccache configuration file. If you run it once
for each compilation you will have race conditions between invocations
and you will mess up the user's ccache configuration. To enable some
ccache setting for a specific compilation you should set the
corresponding environment variable, i.e. CCACHE_EXTRAFILES in this
case.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Speeding up builds with clang-tidy, clang-analyzer and iwyu

[Joel Rosdahl via ccache]

On Sun, 2 Aug 2020 at 19:49, Christian Ledergerber via ccache
 wrote:
> In the mean-time I have been thinking some more and what I am afraid of
> is that the result will not be correct if the flags to clang-tidy change.
>
> A typical command run by make looks like this:
>
> cd /home/dev/erx-soft/cmake-build-debug/modules/utils && ccache
> /home/dev/clion-2020.1.3/bin/cmake/linux/bin/cmake -E __run_co_compile [...]

If I interpret the example command correctly ccache is used as a
prefix to cmake? That won't have any effect; ccache can't cache the
result of cmake so it will just fall back to executing cmake with the
given arguments (and increase some statistics counter).

> --launcher=ccache

I guess this means that cmake will execute "ccache $compiler ..." later.

> [...] In other words: it seems like ccache could also hash the flags to
> clang-tidy. The question: does it?

No, since ccache doesn't see any clang-tidy arguments when cmake
executes "ccache $compiler ..." as mentioned above.

> Now that I am looking at the above command line: Will this require to
> also modify the cmake binary to generate other command lines?

I have no idea, but it sounds likely.

Here's an idea to consider: You could write the clang-tidy arguments
to a file and set CCACHE_EXTRAFILES to the file. Or just set it to
preexisting .clang-tidy files? Then the ccache results will be
invalidated when the clang-tidy arguments change. That in combination
with the hypothetical mode where ccache would exit with failure on a
cache miss would maybe enough for your initial idea.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Speeding up builds with clang-tidy, clang-analyzer and iwyu

[Joel Rosdahl via ccache]

On Sun, 2 Aug 2020 at 18:55, Christian Ledergerber via ccache
 wrote:
> [...]
> To me it seems like the following should work:
>
> 1. try to use ccache
>
> 2. if no hit:
>
> - run clang-tidy
>
> - run clang
>
> For this I would need to know whether ccache generated a hit - lets say
> as return code of the executable.

I don't think there's a good way of accomplishing this without adding
code to ccache to optionally exit with failure on a cache miss. But it
would be easy to add it.

Regards,
Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.11 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.11 is now available. Release archives can be found here:

https://ccache.dev/download.html

Bug fixes:

  - Added knowledge about “-fprofile-{correction,reorder-functions,values}”.

  - ccache now handles the Intel compiler option “-xCODE” (where “CODE” is a
processor feature code) correctly.

  - Added support for NVCC’s “-Werror” and “--Werror” options.

Other:

  - ccache’s “Directory is not hashed if using -gz[=zlib]” tests are now
skipped for GCC 6.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.10 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.10 is now available. Release archives can be found here:

https://ccache.dev/download.html

Bug fixes:

  - Improved handling of profiling options. ccache should now work correctly
for profiling options like “-fprofile-{generate,use}[=path]” for GCC ≥ 9
and Clang as well as “-fauto-profile[=path]” and the Clang-specific
“-fprofile-instr-{generate,use}[=path]” and
“-fprofile-sample-{use,accurate}” options.

  - ccache now copies files directly from the cache to the destination file
instead of via a temporary file. This avoids problems when using filenames
long enough to be near the file system's filename max limit.

  - When the hard-link mode is enabled, ccache now only uses hard links for
object files, not other files like dependency files. This is because
compilers unlink object files before writing to them but they don't do that
for dependency files, so the latter can become overwritten and therefore
corrupted in the cache.

  - Fixed a glitch related to hard-link mode and an empty cache.

  - ccache now supports the ccache.conf file to be a symlink.

  - Temporary files are now deleted immediately on signals like SIGTERM and
SIGINT instead of some time later in a cleanup phase.

  - Fixed a bug that affected ccache's “-o/--set-config” option for the
“base_dir” and “cache_dir_levels” keys.

Contributors (in alphabetical order) to this release:

  - Alexander Lanin
  - Joel Rosdahl
  - Olle Liljenzin

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.9 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.9 is now available. Release archives can be found here:

https://ccache.dev/download.html

Bug fixes:

  - Fixed replacing of /dev/null when building as root with hard link mode
enabled and using “-o /dev/null”.

  - Removed incorrect assertion resulting in “ccache: error: Internal error in
format” when using “-fdebug-prefix-map=X=” with X equal to “$PWD”.

Other:

  - Improved CUDA/NVCC support: Recognize “-dc” and “-x cu” options.

  - Improved name of temporary file used in NFS-safe unlink.

Contributors (in alphabetical order) to this release:

  - Alexander Korsunsky
  - Joel Rosdahl

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Using a shared ccache in cmake environment (linux)

[Joel Rosdahl via ccache]

On Mon, 23 Mar 2020 at 17:11, Steffen Dettmer via ccache
 wrote:
> Just BTW, isn't it common to build in some $builddir different from top
> $srcdir (e.g. automake, cmake) and in that case couldn't the common case need
> two base directories?

Note that base_dir doesn't have to be the top source directory – it can be any
parent directory of the source and build directories, for instance /home. But
sure, it would make sense to be able to specify several base_dir directories,
for instance if you build in /top_level_1 and have the source code in
/top_level_2. (Using "base_dir = /" works as well but has the side effect of
making paths to system include files in /usr relative as well, which isn't
optimal.)

I could have sworn that there already exists an issue about implementing this
but I can't find it so it seems I've only thought about it without writing it
down.

> This is again is a great idea. Will clean recover from corrupted caches,
> or should I add some script like "when each cache value is zero, clear it"?

"ccache -c" first recalculates the size counters and then trims the cache if
needed, so it should be fine.

> Thank you for your great support again!!

You're welcome!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Using a shared ccache in cmake environment (linux)

[Joel Rosdahl via ccache]

On Tue, 17 Mar 2020 at 10:06, Steffen Dettmer via ccache
 wrote:
> As workaround for a special unrelated issue currently we redefine
> __FILE__ (and try to remove that redefinition). I understand that
> ccache still works thanks to CCACHE_BASEDIR even for __FILE__ usage
> inside files. Is that correct?

Yes, if basedir is a prefix of the the source file path then __FILE__
will expand to a relative path since ccache passes a relative source
file path to the compiler.

> I understood that CCACHE_SLOPPINESS=file_macro means that cache
> results may be used even if __FILE__ is different, i.e. using a
> __FILE__ from another user (fine for our usecases), is this correct?

That used to be the case, but the file_macro sloppiness was removed in
3.7.6; see .

> How to find a reasonable max_size? For now I just arbitrarily picked
> 25 GB (approximately the build tree size) and I never saw it "full"
> according to ccache -s.

"cache size" will never reach "max cache size", so that is not a good
indicator of whether "max cache size" is enough. See
 for details on
how automatic cache cleanup works. The TLDR is that "cache size" will
stay around 90% (assuming limit_multiple is the default 0.8) of "max
cache size" in steady state. This is because each of the 16
subdirectories will be between 80% and 100% full with uniform
probability.

Instead you can have a look at "cleanups performed". Each time a
subdirectory gets full that counter will increase by 1.

> On build servers we usually run "make -j 25" (24 cores). Often,
> several such jobs are running by different users (and Jenkins;
> sometimes 400 compiler processes or even more). I assume ccache of
> course safely handles parallel invocation, is this correct?

Yes, assuming that ccache's locking works on the filesystem in question.
(Locking is only used for the stats files; the actual cache data files
are handled via atomic renames.) The cache files can however get corrupt
if a server crashes, depending on how writeback/journaling of filesystem
metadata is configured.

Especially with network caches it might be a good idea to disable
automatic cleanup and instead perform explicit cleanup periodically on
one server, preferably the server that hosts the filesystem. That way
the cleanup won't happen over slow network and several clients won't
compete to clean up. One way of doing this is to set an unlimited cache
size and then run something like "CCACHE_MAXSIZE=25G ccache -c"
periodically the server.

> We have some team mates that have slow (old) laptops only. They
> benefit from using a network shared ccache. Technically, they "mount
> -t cifs" the cache_dir (NFS is firewalled unfortunately). We have
> different Ubuntu/Mint/Debian/Devuan machines, but exactly the same
> compilers (own toolchains).
>
> Is sharing via CIFS possibly at all or could it have bad effects?

Don't know, but I wouldn't be surprised if ccache's locking doesn't work
properly with SMB/CIFS. Locking is based on creating symlinks atomically
and I guess that doesn't translate well to Windows filesystems.

> One issue that occures from time to time is that the ccache -s stats
> become zero (all values expect max cache size are 0). I first didn't
> notice because stats are shared so I assume someone zeroed the stats,
> but with alternate directories we found that it sometimes happens
> without ccache -s. "du -hs $CCACHE_DIR" still shows gigabytes used. We
> didn't find a cause yet, but several candidates exist.

Sounds like the stats file locking fails...

> Can ccache be used on CIFS?

Answered above.

> Are cache and/or stats version dependent?

The cache data and stats files are intended to be backward and forward
compatible from ccache 3.2.

> A few times we noticed that ccache -s reports few GB size but "du -hs"
> reports 40 or 50 GB, although "max_size = 25.0G". Is this expected?

No. But if stats file locking doesn't work then this could happen.

> I'm also still facing scmake issues (using "physical" and "logical" in
> several mixed combinations). Complex topic.

What is scmake?

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.8 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.8 is now available. Release archives are
available here:

https://ccache.dev/download.html

Note: There are now 32 and 64 bit Windows binary releases available as well.

Bug fixes:

  - Use “$PWD” instead of the real CWD (current working directory) when
checking for CWD in preprocessed output. This fixes a problem when “$PWD”
includes a symlink part and the user has set “hash_dir = false”.

  - Rewrote the Windows version of the lockfile routines. This should mitigate
several problems with the old implementation.

  - If “localtime_r” fails the epoch time is now logged instead of garbage.

Other:

  - Improved error message when a boolean environment variable has an invalid
value.

  - Improved the regression fix in ccache 3.7.5 related to not passing
compilation-only options to the preprocessor.

  - ccache’s PCH test suite now skips running the tests if it detects broken
PCH compiler support.

  - Fixed unit test failure on Windows.

  - Fixed “stringop-truncation” build warning on Windows.

  - Improved “x_rename” implementation on Windows.

  - Improved removal of temporary file when rewriting absolute paths to
relative in the dependency file.

  - Clarified “include_file_ctime sloppiness” in the Performance section in the
manual.

Contributors (in alphabetical order) to this release:

  - Cristian Adam
  - Joel Rosdahl
  - Steffen Dettmer

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.7 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.7 is now available.

Source archives:


https://github.com/ccache/ccache/releases/download/v3.7.7/ccache-3.7.7.tar.gz

https://github.com/ccache/ccache/releases/download/v3.7.7/ccache-3.7.7.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7.7/ccache-3.7.7.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7.7/ccache-3.7.7.tar.xz.asc

You can run “gpg --verify ccache-3.7.7.tar.*.asc” to check the source
archive integrity after download.

Bug fixes:

  - Fixed a bug related to object file location in the dependency file (if
using “-MD” or “-MMD” but not “-MF” and the build directory is not the same
as the source directory then the object file location in the “.d” file
would become incorrect). This fixes regression in ccache 3.7.5 introduced
by the bug fix related to EDG-based compilers. Note that this removes
support for EDG-based compilers again. (A better fix for this is planned
for ccache 4.0.)

  - Removed the unify mode since it has bugs and shortcomings that are non-
trivial or impossible to fix: it doesn’t work with the direct mode, it
doesn’t handle C++ raw strings correctly, it can give false cache hits for
“.incbin” directives, it’s turned off when using “-g” and it can make line
numbers in warning messages and “__LINE__” macros incorrect.

  - mtime and ctime values are now stored in the manifest files only when
sloppy_file_stat is set. This avoids adding superfluous manifest file
entries on direct mode cache misses.

  - A “Result:” line is now always printed to the log.

  - The “cache miss” statistics counter will now be updated for read-only cache
misses, making it consistent with the cache hit case.

Contributors (in alphabetical order) to this release:

  - Joel Rosdahl
  - Mizuha Himuraki

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.6 has been released

[Joel Rosdahl via ccache]

Source archives:


https://github.com/ccache/ccache/releases/download/v3.7.6/ccache-3.7.6.tar.gz

https://github.com/ccache/ccache/releases/download/v3.7.6/ccache-3.7.6.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7.6/ccache-3.7.6.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7.6/ccache-3.7.6.tar.xz.asc

You can run “gpg --verify ccache-3.7.6.tar.*.asc” to check the source
archive integrity after download.

Bug fixes:

  - The opt-in “file_macro sloppiness” mode has been removed so that the input
file path now is always included in the direct mode hash. This fixes a bug
that could result in false cache hits in an edge case when “file_macro
sloppiness” is enabled and several identical source files include a
relative header file with the same name but in different directories.

  - Statistics files are no longer lost when the filesystem of the cache is
full.

  - Bail out on too hard Clang option “-MJarg” (in addition to the previous
bailout of “-MJ arg”).

  - Properly handle color diagnostics in the depend mode as well.

Contributors (in alphabetical order) to this release:

  - Joel Rosdahl
  - Luboš Luňák
  - Russell King
  - Thomas Otto

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.5 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.5 is now available.

Source archives:


https://github.com/ccache/ccache/releases/download/v3.7.5/ccache-3.7.5.tar.gz

https://github.com/ccache/ccache/releases/download/v3.7.5/ccache-3.7.5.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7.5/ccache-3.7.5.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7.5/ccache-3.7.5.tar.xz.asc

You can run “gpg --verify ccache-3.7.5.tar.*.asc” to check the source
archive integrity after download.

New features:

  - Added support for “-MF=arg” (with an extra equal sign) as understood by
EDG-based compilers.

Bug fixes:

  - Fixed a regression in 3.7.2 that could result in a warning message instead
of an error in an edge case related to usage of “-Werror”.

  - An implicit “-MQ” is now passed to the preprocessor only if the object file
extension is non-standard. This will make it easier to use EDG-based
compilers (e.g. GHS) which don’t understand “-MQ”. (This is a bug fix of
the corresponding improvement implemented in ccache 3.4.)

  - ccache now falls back to running the real compiler instead of failing
fataly if an internal temporary file is missing after compilation.

  - Fixed a crash if localtime returns null pointer in localtime_r replacement.

  - Fixed header file dependency tracking when building ccache itself.

  - Fixed warning during configure in out-of-tree build in developer mode.

Contributors (in alphabetical order) to this release:

  - Erik Johansson
  - Joel Rosdahl

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.4 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.4 is now available.

Source archives:


https://github.com/ccache/ccache/releases/download/v3.7.4/ccache-3.7.4.tar.gz

https://github.com/ccache/ccache/releases/download/v3.7.4/ccache-3.7.4.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7.4/ccache-3.7.4.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7.4/ccache-3.7.4.tar.xz.asc

You can run “gpg --verify ccache-3.7.4.tar.*.asc” to check the source
archive integrity after download.

Improvements:

  - Added support for GCC 9’s “-gz[=type]” option (previously ccache would
think that “-gz” alone would enable debug information, thus potentially
including the current directory in the hash).

  - Added support for converting paths like “/c/users/...” into
relative paths on
Windows.

Contributors (in alphabetical order) to this release:

  - Cristian Adam
  - Joel Rosdahl

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.3 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.3 is now available.

Source archives:


https://github.com/ccache/ccache/releases/download/v3.7.3/ccache-3.7.3.tar.gz

https://github.com/ccache/ccache/releases/download/v3.7.3/ccache-3.7.3.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7.3/ccache-3.7.3.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7.3/ccache-3.7.3.tar.xz.asc

You can run “gpg --verify ccache-3.7.3.tar.*.asc” to check the source
archive integrity after download.

Bug fixes:

  - The cache size (which is counted in “used disk blocks”) is now correct on
filesystems that use more or less disk blocks than conventional
filesystems, e.g. ecryptfs or btrfs/zfs with transparent compression. This
also fixes a related problem with ccache's own test suite when run on such
filesystems.

  - Fixed a regression in 3.7.2 when using the compiler option “-Werror” and
then “-Wno-error” later on the command line.

Contributors (in alphabetical order) to this release:

  - Ivan Volnov
  - Joel Rosdahl

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.2 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.2 is now available.

Source archives:


https://github.com/ccache/ccache/releases/download/v3.7.2/ccache-3.7.2.tar.gz

https://github.com/ccache/ccache/releases/download/v3.7.2/ccache-3.7.2.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7.2/ccache-3.7.2.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7.2/ccache-3.7.2.tar.xz.asc

You can run “gpg --verify ccache-3.7.2.tar.*.asc” to check the source
archive integrity after download.

Bug fixes:

  - The compiler option “-gdwarf*” no longer forces “run_second_cpp = true”.

  - Added verification that the value passed to the “-o/--set-config” option is
valid.

  - Fixed detection of precompiled headers in the depend mode.

  - Bail out on too hard Clang option “-ftime-trace”.

  - ccache now updates the correct stats file when adding/updating manifest
files. This bug previously made the file and size statistics counters
incorrect over time.

  - Fixed warnings from Clang about unused arguments during preprocessing.

  - Unknown manifest versions are now handled gracefully in “--dump-manifest”.

  - Fixed “make check” with “funny” locales.

Documentation:

  - Added a hint about not running “autogen.sh” when building from a release
archive.

Contributors (in alphabetical order) to this release:

  - Anders F Björklund
  - Joel Rosdahl
  - Luboš Luňák
  - luzpaz

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache on generated C++ files...

[Joel Rosdahl via ccache]

On Tue, 25 Jun 2019 at 00:28, Malfettone, Kris via ccache
 wrote:
> So what I seem to be observing is only when I do a ninja -t clean or make
> clean. This removes all of my generated files (.h and .cpp files) and of
> course the normal build outputs. A subsequent run is then mostly cache
> misses. If I say manually do a rm $(find -name "*.o") then I get a 100% cache
> hit rate. I had assumed it was due to the fact that the generated files would
> have a newer mtime and ctime. My assumption must be wrong.

Maybe there is a timestamp in the generated files?

> Is there an easy way to print diagnostics on why a cache miss occurred?

Yes, assuming you use ccache 3.5 or newer, you can enable the debug mode and
compare the hashed input for two different builds. See "To debug why you don’t
get an expected cache hit for an object file [...]" on
.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache on generated C++ files...

[Joel Rosdahl via ccache]

Hi Kris,

On Fri, 21 Jun 2019 at 02:43, Malfettone, Kris wrote:
> My project uses a large number of generated C++ files (.h/.cpp files). On a
> clean build these files are cleaned up as well. As a result during the next
> build since they are generated again they will all have a newer mtime and
> ctime than the previous runs. Their content is the same though. Is there a
> way for those files to disable checking both the mtime and ctime checks?

I'm not sure I understand your question. Since the content of the generated
files is the same, they will get the same hash sum and will therefore be seen
as the same for the purposes of caching compilation results.

Or are you talking about the "sloppiness = file_stat_matches" option, where the
normal content hashing is sidestepped and matching in the direct mode can be
done by checking only size/mtime/ctime? If so, the answer is that it's not
possible to only check the size. Or rather: It would of course be possible to
implement such an option, but that does not seem like a good idea – it would be
way too dangerous since files of course can differ in many ways but still have
the same size.

But if you really want to try it yourself, you can simply change

  if (fi->fsize != st->size) {
return false;
  }

to

  if (fi->fsize == st->size) {
continue;
  }

in verify_result in src/manifest.c.

Is the problem that you think that hashing the generated files takes too much
time? If so, have you in some way been able to measure that hashing indeed is a
problem?

> I would love to be able to check based on some sort of "check file size then
> check content checksum" ignoring mtime and ctime.

Again, not sure I understand what you mean. In the normal case (no "sloppiness
= file_stat_matches"), the mtime/ctime is not checked for verification (but
they are checked to rule out "too new" files; see
include_file_mtime/include_file_ctime in the manual), so what you describe is
how it already works. In the "sloppiness = file_stat_matches" case, the
verification also falls back to check the content if mtime/ctime don't match.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] BUG: ccache -o = -- illegal values not rejected

[Joel Rosdahl via ccache]

Hi Henri,

Verifying the value sounds like a good improvement.

The fix you suggested doesn't quite work since there is no conf variable
available, so I guess you have made other changes as well. Also, there is
an optional "verifier" for some configuration items that should be run as
well. I made a fix based on your suggestion:
https://github.com/jrosdahl/ccache/commit/3d05ccd8c8607a051cb80beeff340cc9aa1489c6

-- Joel

On Mon, 17 Jun 2019 at 13:49, Houder via ccache 
wrote:

> Hi list,
>
> Just reporting (it is not my intention to create an account
> on github in order to report a bug).
>
> Downloaded 3.7.1 and compiled it on Cygwin.
>
> Call flow for "ccache -o =
>
> ccache_main // src/ccache.c
>ccache_main_options // src/ccache.c
>  conf_set_value_in_file // src/conf.c
>
> conf_set_value_in_file
>find_conf(key)
>"insert (or replace) key/value in cfg file"
>
> Function conf_set_value_in_file() only verifies that the key
> makes sense; it does not check the user provided value.
>
> Resulting in an _illegal_ value in the cfg file, if the user
> provided an illegal value.
>
> Subsequently, a command like "ccache -s" fails, reporting an
> illegal cfg file.
>
> Adding
>
>   if (!item->parser(value, (char *)conf + item->offset, errmsg)) {
>   *errmsg = format("Henri: unknown value \"%s\"", value);
>   return false;
>   }
>
> after the call to find_key (in conf_set_value_in_file() )
>
> rejects an illegal value.
>
> Henri
>
> ___
> ccache mailing list
> ccache@lists.samba.org
> https://lists.samba.org/mailman/listinfo/ccache
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Compression

[Joel Rosdahl via ccache]

Hi Francios,

On Tue, 7 May 2019 at 12:17, Francois E via ccache
 wrote:
> We use a shared ccache server hosted on a remote server, some of us
> experience slow connection to that server. I was wondering if
> compression would help in this case, i.e does the (un)compression occur
> at the server level or at the local level ?

Ccache does not have a server part, so compression is done locally. But I guess
you mean that you use a shared network filesystem, in which case compression
very well could help since the file server then won't have to receive and send
as much data.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Intent to package: Snaps are universal Linux packages

[Joel Rosdahl via ccache]

On Wed, 10 Apr 2019 at 17:46, 林博仁 via ccache  wrote:
> I've [...] built a working snap [of ccache]

Cool, thanks for the notice.

> I'm also looking forward to working with you to make the packaging
> work upstream (if you're interested).

I'm happy if you would like to continue maintaining the ccache snap.

One thing that you could consider documenting on
https://snapcraft.io/ccache is that there seems to be a slight
overhead of executing ccache via snap (0.02 seconds on my system), so
for scenarios where ccache is called very frequently the snap overhead
is small but clearly measurable.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7.1 has been released

[Joel Rosdahl via ccache]

ccache version 3.7.1 is now available.

Source archives:


https://github.com/ccache/ccache/releases/download/v3.7.1/ccache-3.7.1.tar.bz2

https://github.com/ccache/ccache/releases/download/v3.7.1/ccache-3.7.1.tar.gz

https://github.com/ccache/ccache/releases/download/v3.7.1/ccache-3.7.1.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7.1/ccache-3.7.1.tar.bz2.asc

https://github.com/ccache/ccache/releases/download/v3.7.1/ccache-3.7.1.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7.1/ccache-3.7.1.tar.xz.asc

You can run “gpg --verify ccache-3.7.1.tar.*.asc” to check the source
archive integrity after download.

Changes:

- Fixed a problem when using the compiler option “-MF /dev/null”.

- Long commandlines are now handled gracefully on Windows by using the “@file”
  syntax to avoid hitting the commandline size limit.

- Fixed complaint from GCC 9’s “-Werror=format-overflow” when compiling ccache
  itself.

Contributors (in alphabetical order) to this release:

- Anders F Björklund
- Joel Rosdahl
- Loïc Yhuel
- Mike Blumenkrantz

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7 has been released

[Joel Rosdahl via ccache]

ccache version 3.7 is now available.

Details may be read in the release notes:

http://ccache.dev/releasenotes.html#_ccache_3_7

Source archives are available here:

https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.bz2
https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.gz
https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.bz2.asc

https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.xz.asc

You can run “gpg --verify ccache-3.7.tar.*.asc” to check the source
archive integrity after download.

Note: New release files will no longer be uploaded to
https://www.samba.org/ftp/ccache, but old releases will remain there.

Changes:

- ccache now has an opt-in “depend mode”. When enabled, ccache never
  executes the preprocessor, which results in much lower cache miss
  overhead at the expense of a lower potential cache hit rate. The
  depend mode is only possible to use when the compiler option “-MD” or
  “-MMD” is used.

- Added support for GCC's “-ffile-prefix-map” option. The
  “-fmacro-prefix-map” option is now also skipped from the hash.

- Added support for multiple “-fsanitize-blacklist” arguments.

- ccache now includes the environment variables “LANG”, “LC_ALL”,
  “LC_CTYPE” and “LC_MESSAGES” in the hash since they may affect
  localization of compiler warning messages. Set sloppiness to “locale”
  to opt out of this.

- Fixed a problem due to Clang overwriting the output file when
  compiling an assembler file.

- Clarified the manual to explain the reasoning behind the “file_macro”
  sloppiness setting in a better way.

- ccache now handles several levels of nonexistent directories when
  rewriting absolute paths to relative.

- A new sloppiness setting “clang_index_store” makes ccache skip the
  Clang compiler option “-index-store-path” and its argument when
  computing the manifest hash. This is useful if you use Xcode, which
  uses an index store path derived from the local project path. Note
  that the index store won't be updated correctly on cache hits if you
  enable this option.

- Rename sloppiness “no_system_headers” to “system_headers” for
  consistency with other options. “no_system_headers” can still be used
  as an (undocumented) alias.

- The GCC variables “DEPENDENCIES_OUTPUT” and “SUNPRO_DEPENDENCIES” are
  now supported correctly.

- The algorithm that scans for “__DATE_” and “__TIME__” tokens in the
  hashed source code now doesn't produce false positives for tokens
  where “__DATE__” or “__TIME__” is a substring.

Contributors (in alphabetical order) to this release:

- Anders F Björklund
- Doug Anderson
- Joel Rosdahl
- Pavol Sakac
- Robert Yang

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.7 has been released (correction of changelog)

[Joel Rosdahl via ccache]

[The previous mail contained changelog entries for ccache 3.6. Sorry
about that, I've rewritten my scripts for doing releases and this one
slipped through.]

ccache version 3.7 is now available.

Source archives:

https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.bz2
https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.gz
https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):


https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.bz2.asc

https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.gz.asc

https://github.com/ccache/ccache/releases/download/v3.7/ccache-3.7.tar.xz.asc

You can run “gpg --verify ccache-3.7.tar.*.asc” to check the source
archive integrity after download.

Note: New release files will no longer be uploaded to
https://www.samba.org/ftp/ccache, but old releases will remain there.

Changes:

- Fixed crash when the debug mode is enabled and the output file is in a
  non-writable directory, e.g. when the output file is `/dev/null`.

- Fixed an issue when printing very large log messages to the debug log.

- Fixed bugs related to support for `-gsplit-dwarf`. Previously ccache could
  produce an incorrect link to the `.dwo` file in the `.o` file.

- Compilations with /dev/null as the input file are now cached.

- ccache has learned how to contruct the object filename if no `-o` option is
  given and the source filename does not include a `.` or ends with a `.`.

- Fixed a temporary file leak when the depend mode is enabled and the compiler
  produces standard error output.

- Fixed a bug in the depend mode where a manifest hash only could be associated
  with one set of header dependencies.

- Manifest files did not get marked as used on direct cache hits, so the LRU
  cache cleanup would incorrectly remove them eventually. This has been fixed.

- The rewriting of absolute paths into relative paths in the dependency file
  has been enabled in the depend mode as well.

- ccache now ignores unknown keys in configuration files for forward
  compatibility.

- Rearranged command-line options into sections in the help text.

- Documented the previously undocumented `--dump-manifest` and `--hash-file`
  options (only useful for debugging ccache itself).

- Added missing documentation for the command-line option `-k/--get-config`
  added in ccache 3.5.

- Renamed the `--print-config` command to `--show-config`.

- Added a new `--print-stats` command that prints statistics counters in
  machine-parsable (tab-separated) format.

- ccache no longer creates a missing output directory, thus mimicking the
  compiler behavior for `-o out/obj.o` when “out” doesn’t exist.

- `-fdebug-prefix-map=ARG`, `-ffile-prefix-map=ARG` and
  `-fmacro-prefix-map=ARG` are now included in the hash, but only the part
  before “ARG”. This fixes a bug where compiler feature detection of said flags
  would not work correctly with ccache.

- Bail out on too hard compiler option `-gtoggle`.

- Bail out on too hard Clang options `--analyze` and `-analyze`.

- Improved debug logging of file hashes in depend mode.

- Improved handling of various `-g*` options. In particular, ccache now
  understands that `-g0` cancels out previous `-g* options`.

- Worked around a problem with Automake related to `.d` files when using the
  hard link mode.

- Added opt-in (at configure time) support for enabling trace logs for
  profiling ccache itself. See `doc/DEVELOPER.md` in the code tree for more
  information

- Removed support for Fortran 77 again. Some Fortran support was added in
  ccache 3.3, but the implementation did not work when Fortran modules are
  involved.

Contributors (in alphabetical order) to this release:

- Anders F Björklund
- Doug Anderson
- Joel Rosdahl
- Pavol Sakac
- Robert Yang

Thanks!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] icecc with cache: cache -s always 0.00 % ?

[Joel Rosdahl via ccache]

On Tue, 5 Mar 2019 at 08:08, Haio Maio via ccache
 wrote:

> On February 22, 2019 at 8:42 PM Joel Rosdahl  wrote:
>
> > Before trying to set up icecc as a wrapper together with ccache, you need
> > to make sure that ccache is used by the command(s) that you abbreviated as
> > "compile".
> >
> > When you wrote "compile", which commands do you actually run to compile?
>
> I followed the instructions
> https://github.com/icecc/icecream#how-to-combine-icecream-with-ccache which
> didnt leat to any results.
>
> Do you have any other suggestions?

My suggestion is that you should re-read my previous answers and try to learn
how ccache works. There are no magic commands to run to solve your problem as
long as we don't know what the problem is.

Start by trying to get ccache to work properly without using icecc. Since you
haven't answered my question about which commands you actually run, this is
something you need to do yourself.

Good luck!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] icecc with cache: cache -s always 0.00 % ?

[Joel Rosdahl via ccache]

On Fri, 22 Feb 2019 at 19:42, Haio Maio via ccache
 wrote:
> Im afraid, I didnt understand your question?
>
> Im trying to use icecc/icecream as a wrapper for the compiler. Could you be 
> more specific of what did you mean?

Before trying to set up icecc as a wrapper together with ccache, you
need to make sure that ccache is used by the command(s) that you
abbreviated as "compile".

When you wrote "compile", which commands do you actually run to compile?

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] icecc with cache: cache -s always 0.00 % ?

[Joel Rosdahl via ccache]

On Wed, 20 Feb 2019 at 12:46, Haio Maio via ccache
 wrote:
> Any suggestion of how to use successfully icecc with ccache? The following 
> doesnt work ...
>
> export CCACHE_PREFIX=icecc
>
> compile
>
> ccache -s
> [...]

It looks like ccache is never called in the compile step, so you need to make
sure that ccache usage is set up properly there. What does the "compile" step
do?

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] [PATCH 2/2] Fix Segmentation fault error when gcc -o /dev/null

[Joel Rosdahl via ccache]

On Tue, 22 Jan 2019 at 10:36, Robert Yang via ccache
 wrote:
> Fixed:
> $ export CCACHE_DEBUG=1
> $ ccache gcc -c hello.c -o /dev/null
>
> Segmentation fault (core dumped)
>
> This is because failed to open /dev/null.foo (Permission denied), check file
> stream before write to it can fix the problem.

Thanks, applied on master.

On a side note, I had to edit the author information before applying your
patches via "git am" since the mails were sent from "Robert Yang via ccache
". I don't know why that happened in your case.
Regardless, I prefer patches via GitHub pull requests nowadays if possible. I
have updated
https://github.com/ccache/ccache/blob/master/CONTRIBUTING.md#contributing-code
accordingly.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] [PATCH 1/2] dev.mk.in: fix file name too long error

[Joel Rosdahl via ccache]

On Tue, 22 Jan 2019 at 10:36, Robert Yang via ccache
 wrote:
> The all_cppflags changes path to filename which causes file name too long
> error when the path is longer than NAME_MAX (usually 255). Strip srcdir
> to fix the problem.

Thanks, applied on master.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.6 has been released

[Joel Rosdahl via ccache]

ccache version 3.6 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_6

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.6.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.6.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.6.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.6.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.6.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.6.tar.xz.asc

You can run “gpg --verify ccache-3.6.tar.*.asc” to check the source
archive integrity after download.

Changes:

- ccache now has an opt-in “depend mode”. When enabled, ccache never
  executes the preprocessor, which results in much lower cache miss
  overhead at the expense of a lower potential cache hit rate. The
  depend mode is only possible to use when the compiler option “-MD” or
  “-MMD” is used.

- Added support for GCC's “-ffile-prefix-map” option. The
  “-fmacro-prefix-map” option is now also skipped from the hash.

- Added support for multiple “-fsanitize-blacklist” arguments.

- ccache now includes the environment variables “LANG”, “LC_ALL”,
  “LC_CTYPE” and “LC_MESSAGES” in the hash since they may affect
  localization of compiler warning messages. Set sloppiness to “locale”
  to opt out of this.

- Fixed a problem due to Clang overwriting the output file when
  compiling an assembler file.

- Clarified the manual to explain the reasoning behind the “file_macro”
  sloppiness setting in a better way.

- ccache now handles several levels of nonexistent directories when
  rewriting absolute paths to relative.

- A new sloppiness setting “clang_index_store” makes ccache skip the
  Clang compiler option “-index-store-path” and its argument when
  computing the manifest hash. This is useful if you use Xcode, which
  uses an index store path derived from the local project path. Note
  that the index store won't be updated correctly on cache hits if you
  enable this option.

- Rename sloppiness “no_system_headers” to “system_headers” for
  consistency with other options. “no_system_headers” can still be used
  as an (undocumented) alias.

- The GCC variables “DEPENDENCIES_OUTPUT” and “SUNPRO_DEPENDENCIES” are
  now supported correctly.

- The algorithm that scans for “__DATE_” and “__TIME__” tokens in the
  hashed source code now doesn't produce false positives for tokens
  where “__DATE__” or “__TIME__” is a substring.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Help with adding support to custom language

[Joel Rosdahl via ccache]

On Thu, 3 Jan 2019 at 23:02, Andre Calfa  wrote:
> One of the motivations is that our current compiler cache is only supported
> locally for each build server. In some cases the compilation of a single
> input file takes a long time due to its size, and we cannot avoid its
> compilation when the correct cached cpp and h are located in a different
> build server. So if we can take advantage of using a distributed approach
> which ccache supports, that would be great.

No, ccache does not support distributed compilation, unless you use a cache
directory shared over the network (e.g. using NFS), which I assume you could
use for your compiler cache as well. Are you thinking about sccache, distcc or
Icecream?

> Unfortunately our compiler is not GCC like, as our flags are not analogous to
> the GCC ones. I am guessing this would require lots of changes in ccache code
> to make it work then?

Yes, to the point that it will be an almost completely different program.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Help with adding support to custom language

[Joel Rosdahl via ccache]

Hi Andre,

On Thu, 3 Jan 2019 at 20:39, Andre Calfa via ccache
 wrote:
> We currently have our own compiler cache for the compiler we developed, but
> we would like to switch to ccache if possible. Our compiler takes as input
> a custom language and generates both a .cpp file and a .h file. I would
> like to know if it would be feasible for me to work on applying a patch to
> ccache so that I can use ccache for this compiler, and if so, would there
> be any documentation/example that I should follow to get this implemented?

What problems do you see with your own compiler cache that makes you consider
using ccache instead?

Does your compiler behave similar to GCC, for instance using "-c" for
compilation, "-E" for generating preprocessed output similar to that of GCC,
"-x" for selecting source language, etc? If not, my guess is that it's not
really feasible.

There is no documentation or examples for adding new compiler support to
ccache, mainly because ccache in a sense only supports one ("GCC and compilers
that behave similar enough").

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] New source archives for ccache 3.5.1 are now available

[Joel Rosdahl via ccache]

The ccache 3.5.1 release archives announced yesterday unfortunately
included an incorrectly generated configure script, so they have been
pulled.

Fixed source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.5.1a.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.5.1a.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.5.1a.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.5.1a.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.5.1a.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.5.1a.tar.xz.asc

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.5.1 has been released

[Joel Rosdahl via ccache]

ccache version 3.5.1 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_5_1

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.5.1.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.5.1.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.5.1.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.5.1.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.5.1.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.5.1.tar.xz.asc

You can run “gpg --verify ccache-3.5.1.tar.*.asc” to check the source
archive integrity after download.

Changes:

- Added missing getopt_long.c source file to release archive.

- Fixed (harmless) compiler warnings when building ccache object files.

- CFLAGS is no longer passed to the linker when linking ccache.

- Improved development mode build flags.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Support for guile scheme?

[Joel Rosdahl via ccache]

Hi,

On Sat, 22 Dec 2018 at 04:25, swedebugia--- via ccache
 wrote:
> The site-ccache of guile does not support partials of files and every
> update to our scheme-files.

What is "site-ccache"? After reading
https://www.gnu.org/software/guile/manual/html_node/Build-Config.html, it does
not sound like site-ccache has anything to do with ccache?

> This means i'm stuck compiling big files again and again though only small
> parts of the files has changed in between.

What do you mean with "again"? Has something worked before that does not work
now? I'm afraid your questions are too vague to be answered in a good way.

> Would you be willing to include Guile scheme support in ccache or are you
> aiming to support only the C-family?

I don't know anything about how compilation with Guile works, but assuming it
does not work similar to GCC, support is out of scope for ccache. It's not tied
to the C family but to how the compiler works and which options it takes.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Header-file paths in external build directories

[Joel Rosdahl via ccache]

On Fri, 16 Nov 2018 at 19:46, Nico Huber via ccache
 wrote:
> [...]
> If I'm looking at the right spot in the code, I can see that 1. happens
> on purpose (in process_preprocessed_file(), ccache.c:945). Though, I
> wonder why?

The history of that code is that https://github.com/ccache/ccache/pull/88
removed it to implement a new feature in ccache 3.3: "Names of included files
are no longer included in the hash of the compiler's preprocessed output. This
leads to more potential cache hits when not using the direct mode." Then
https://github.com/ccache/ccache/issues/134 was discovered, leading to a revert
in https://github.com/ccache/ccache/commit/7cad4ac5.

> Could this be handled with the `base_dir` normalization? I already tried to 
> set
> $(obj) as base_dir, without success. I guess it would require to normalize 
> paths
> relative to `base_dir` even if they are not under `base_dir`.

base_dir should be set to a parent directory of all paths seen by ccache. How
is $(obj) related to your source code directory? Are both absolute paths? Can
$(obj) be reached by a relative path that is the same for different
compilations for which you expect cache hits?

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.5 has been released

[Joel Rosdahl via ccache]

ccache version 3.5 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_5

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.5.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.5.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.5.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.5.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.5.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.5.tar.xz.asc

You can run “gpg --verify ccache-3.5.tar.*.asc” to check the source
archive integrity after download.

Changes:

- Added a boolean “debug” (“CCACHE_DEBUG”) configuration option. When
  enabled, ccache will create per-object debug files that are helpful
  e.g. when debugging unexpected cache misses. See also the new “Cache
  debugging” section in the manual.

- Renamed “CCACHE_CC” to “CCACHE_COMPILER” (keeping the former as a
  deprecated alias).

- Added a new command-line option “-k/--get-config” that prints the
  value of a config key.

- It is now possible to let ccache hash a precomputed checksum file
  instead of the full content of a precompiled header. This can save
  time for large precompiled headers. Note that the build system needs
  to keep the checksum file in sync with the precompiled header for this
  to work.

- Improved performance substantially when using “hash_dir = false” on
  platforms like macOS where “getcwd()” is slow.

- Added “stats updated” timestamp in “ccache -s” output. This can be
  useful if you wonder whether ccache actually was used for your last
  build.

- Renamed “stats zero time” to “stats zeroed” and documented it. The
  counter is also now only present in “ccache -s” output when “ccache
  -z” actually has been called.

- The content of the “-fsanitize-blacklist” file is now included in the
  hash, so updates to the file will now correctly result in separate
  cache entries.

- It’s now possible to opt out of building and installing man pages when
  running “make install” in the source repository.

- If the compiler type can’t be detected (e.g. if it is named “cc”), use
  safer defaults that won’t trip up Clang.

- Made the ccache test suite work on FreeBSD.

- Added “file_stat_matches_ctime” option to disable ctime check if
  “file_stat_matches” is enabled.

- Made “./configure --without-bundled-zlib” do what’s intended.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.4.3 has been released

[Joel Rosdahl via ccache]

ccache version 3.4.3 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_4_3

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.4.3.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.4.3.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.4.3.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.4.3.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.4.3.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.4.3.tar.xz.asc

You can run “gpg --verify ccache-3.4.3.tar.*.asc” to check the source
archive integrity after download.

Bug fixes:

- Fixed a race condition when creating the initial config file in the cache
  directory.

- Bail out on too hard clang option “-MJ”.

- Bail out on too hard option “-save-temps=obj”.

- Handle separate parameter to clang option “-target” correctly.

- Upgraded bundled zlib to version 1.2.11.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.4.2 has been released

[Joel Rosdahl via ccache]

ccache version 3.4.2 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_4_2

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.4.2.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.4.2.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.4.2.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.4.2.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.4.2.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.4.2.tar.xz.asc

You can run "gpg --verify ccache-3.4.2.tar.*.asc" to check the source
archive integrity after download.

Bug fixes:

- The cleanup algorithm has been fixed to not misbehave when files are
  removed by another process while the cleanup process is running.
  Previously, too many files could be removed from the cache if multiple
  cleanup processes were triggered at the same time, in extreme cases
  trimming the cache to a much smaller size than the configured limits.

- Correctly hash preprocessed headers located in a ".gch directory".
  Previously, ccache would not pick up changes to such precompiled
  headers, risking false positive cache hits.

- Fixed build failure when using the bundled zlib sources.

- ccache 3.3.5 added a workaround for not triggering Clang errors when a
  precompiled header's dependency has an updated timestamp (but
  identical content). That workaround is now only applied when the
  compiler is Clang.

- Made it possible to perform out-of-source builds in dev mode again.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache test failures

[Joel Rosdahl via ccache]

On 16 March 2018 at 12:52, Forte, Greg via ccache
 wrote:
> Tried this with and without CCACHE_DIR set and exported - with it, I see the
> same output you showed, including the unset, and the test still fails: [...]

OK.

What's common between the two tests that fail for you is that you don't get a
cache hit for a second compilation in another directory. One way to understand
why is to log all data that is hashed by ccache in the two cases and see where
it starts to differ. There is a crude way of doing that already: Build ccache
with -DCCACHE_DEBUG_HASH and set CCACHE_DEBUG_HASH at runtime. Each ccache
invocation will then create a ccache-debug-hash.bin that contains the hashed
data, so you could compare the files in the two different directories. Or send
them to me (private mail) and I'll have a look.

By the way, here's a sketch of a yet unimplemented feature what would make this
kind of debugging easier: https://github.com/ccache/ccache/issues/226

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.4.1 has been released

[Joel Rosdahl via ccache]

ccache version 3.4.1 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_4_1

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.4.1.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.4.1.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.4.1.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.4.1.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.4.1.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.4.1.tar.xz.asc

You can run "gpg --verify ccache-3.4.1.tar.*.asc" to check the source
archive integrity after download.

Bug fixes:

- Fixed printing of version number in "ccache --version".

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.4 has been released

[Joel Rosdahl via ccache]

ccache version 3.4 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_4

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.4.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.4.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.4.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.4.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.4.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.4.tar.xz.asc

You can run "gpg --verify ccache-3.4.tar.*.asc" to check the source
archive integrity after download.

New features and enhancements:

- The compiler option form "--sysroot arg" is now handled like the
  documented "--sysroot=arg" form.

- Added support for caching ".su" files generated by GCC flag
  "-fstack-usage".

- ccache should now work with distcc's "pump" wrapper.

- The optional unifier is no longer disabled when the direct mode is
  enabled.

- Added support for nvcc compiler options "--compiler-bindir/-ccbin",
  "--output-directory/-odir" and "--libdevice-directory/-ldir".

- Boolean environment variable settings no longer accept the following
  (case-insensitive) values: "0", "false", "disable" and "no". All other
  values are accepted and taken to mean "true". This is to stop users
  from setting e.g. "CCACHE_DISABLE=0" and then expect the cache to be
  used.

- Improved support for "run_second_cpp = false": If combined with
  passing "-fdirectives-only" (GCC) or "frewrite-includes" (Clang) to
  the compiler, diagnostics warnings and similar will be correct.

- An implicit "-MQ" is now passed to the preprocessor only if the object
  file extension is non-standard. This should make it easier to use
  EDG-based compilers (e.g. GHS) which don't understand "-MQ".

- ccache now treats an unreadable configuration file just like a missing
  configuration file.

- Documented more pitfalls with enabling "hard_links"
  ("CCACHE_HARDLINK").

- Documented caveats related to colored warnings from compilers.

Bug fixes:

- File size and number counters are now updated correctly when files are
  overwritten in the cache, e.g. when using "CCACHE_RECACHE".

- "run_second_cpp" is now forced for nvcc.

- Fixed how the nvcc options "-optf" and "-odir" are handled.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] why is limit_multiple ignored?

[Joel Rosdahl via ccache]

On 29 January 2018 at 07:14, Scott Bennett <benn...@sdf.org> wrote:

> Countless data base software implementations handle these situations
> acceptably well.

Sigh.

I see. You're talking about a completely different model than what
ccache currently uses, which was not clear to me when I read your
initial description. What you seem to don't understand, or choose to
ignore, is that ccache can't stop supporting the simple server-less
file-based model since that would drop support for two important use
cases:

1. Using ccache on a personal account without having access to a system
   service and without having to start a personal server.
2. Using a shared cache on NFS.

(For case 1, it would be feasible with a model where the client starts a
server on demand, unless the cache is shared.)

Why are they important? Simply because people have used ccache like that
for many years.

It would certainly be possible to add optional client-server backends,
and for them I fully agree that using a centralized index is obvious,
but that's just an entirely separate discussion as I see it.

If we would drop support for simple file-based caches, then it would no
longer be ccache but something else. Which would be fine, but that's
another project. (It could of course be ccache version 4, but then I
expect that ccache version 3 would continue living its own life, so it
would be a separate project in practice.)

> (FWIW, I use cache_dir_levels = 5, which may not be optimal in terms
> of performance. I don't have a good way of determining the optimal
> depth to use for the cache directory trees. It seems to be very, very
> fast for use in building things, but may well be a killer for
> cleanups.)

Let's see. cache_dir_levels = 5 means 16⁵ ≈ 1 million directories on the
lowest level. A large cache might hold, say, 10 million files? Then 10
files per directory is clearly not optimal. How many files do you have
in your cache?

I think that a good rule of thumb would be to store a couple of thousand
or tens of thousands of files per directory, depending on the file
system characteristics. That would mean that cache_dir_levels = 3 would
be enough even for very large caches.

Perhaps lowering cache_dir_levels could partly solve the bad cleanup
performance you have?

> Very possibly you have the requisite knowledge/experience yourself.

Actually yes, so you could have saved both your and my time by just
asking something like "Have you considered using a client-server model,
perhaps using a standard database, instead of a file-based cache?"
instead of trying to educate me.

> To modify ccache to use data base software is admittedly a major
> rewriting job, so I expect such an idea to put you off, but it's a
> project that should ultimately yield a far superior product, IMO.

I don't disagree, but as I said, that would be another project, and I
neither have time nor interest in that personally.

-- Joel

On 29 January 2018 at 07:14, Scott Bennett <benn...@sdf.org> wrote:
> Joel Rosdahl <j...@rosdahl.net> wrote:
>
>> On 7 January 2018 at 14:02, Scott Bennett wrote:
>>
>> > The design problem is that there is no centralized index maintained of
>> > cache entries' paths, their sizes, and their timestamps, necessitating
>> > the plumbing of the directory trees. [...]
>>
>> Thanks for sharing your ideas!
>
>  You may wish to retract any thanks once you've read what follows.  The
> current independence of ccache from any other third-party software is valued
> and for good reasons.  However, I hope to show below a better way to do 
> things.
> That independence can still be maintained, but only at the cost of another
> wheel reinvention. :-(
>>
>> I fully agree that the cleanup algorithm/design hasn't aged well. It has
>> essentially stayed the same since Tridge created ccache in 2002, when
>> storage devices were much smaller and a cache of one GB or two probably
>> was considered quite large.
>>
>> Trying to improve the cleanup algorithm/design has not been a priority
>> since I personally haven't seen such pathological behavior that you
>> describe ("cleanups that can take over a half hour to run and hammer a
>
>  I don't know whether users of other operating systems are using ccache
> in building their systems, but many FreeBSD users do so because the time
> savings are so great.  When one can cut a build time of six hours to, say,
> an hour and a half, one tends to appreciate the tool(s) that make(s) it
> possible.  I.e., we use and love ccache because, in general, it works so well
> and improves performance so much.
>  However, compiling an operating system means a pretty large cache area
> is needed if one is to fit the working set within the cache.  Similarly,
> FreeBSD users who compile th

[ccache] ccache version 3.3.6 has been released

[Joel Rosdahl via ccache]

ccache version 3.3.6 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_3_6

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.3.6.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.3.6.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.3.6.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.3.6.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.3.6.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.3.6.tar.xz.asc

You can run "gpg --verify ccache-3.3.6.tar.*.asc" to check the source
archive integrity after download.

Improvements:

- Improved instructions on how to get cache hits between different
  working directories.

Bug fixes:

- Fixed regression in ccache 3.3.5 related to the UNCACHED_ERR_FD
  feature.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] why is limit_multiple ignored?

[Joel Rosdahl via ccache]

On 7 January 2018 at 14:02, Scott Bennett wrote:

> The design problem is that there is no centralized index maintained of
> cache entries' paths, their sizes, and their timestamps, necessitating
> the plumbing of the directory trees. [...]

Thanks for sharing your ideas!

I fully agree that the cleanup algorithm/design hasn't aged well. It has
essentially stayed the same since Tridge created ccache in 2002, when
storage devices were much smaller and a cache of one GB or two probably
was considered quite large.

Trying to improve the cleanup algorithm/design has not been a priority
since I personally haven't seen such pathological behavior that you
describe ("cleanups that can take over a half hour to run and hammer a
hard drive mercilessly"). However, I'm not at all convinced that
introducing a centralized index is the panacea you describe.

Do you have a sketch design of how to maintain a centralized index? Here
are some requirements to consider for the design:

A. It should cope with a ccache process being killed at any time.
B. It should work reasonably well on flaky and/or slow file systems,
   e.g. NFS.
C. It should not introduce lock contention for reasonable use cases.
D. It should be quick for cache misses (not only for cleanup).
E. It should handle cleanup quickly and gracefully.

I'm guessing that you envision having one centralized lock for the
index. The tiny stats files already suffer from lock contention in some
scenarios because they are so few. That's why ideas like
https://github.com/ccache/ccache/issues/168 and comments like
https://www.mail-archive.com/ccache@lists.samba.org/msg01011.html
(comment number 2) pop up. Even if a centralized index only needs a lock
for writing, it would still serialize writes to the cache. I have
trouble seeing how that would work out well. But I'll gladly be proved
wrong.

For reference: When updating the stats files, the current method is to
acquire a lock, write the new content to a temporary file, rename the
temporary file to the target file and release the lock. Writing the full
content to a temporary file and renaming it to the target solves A and
B, and having 16 files instead of 1 improves on C. Having no index
trivially solves D. (And E is not solved well at all.)

> The lack of a centralized index can also result in cache evictions
> that are not actually LRU.

Not having true LRU eviction doesn't bother me at all. I think that it's
a very reasonable trade-off to have "approximate LRU eviction" if the
performance is better and/or the implementation is easier.

> Where does the hysteresis of (0.9-0.8)max_size=0.1*max_size come from?

When the cache has filled up at least once, the fill grade of one of the
16 subdirectories is a random variable between 0.8 and 1.0 with uniform
distribution, so the probability of the total size of the 16
subdirectories is approximately a normal distribution with 0.9 as the
mean. In other words, it's likely that the cache size is around 0.9 and
much less likely that it's near 0.8 or 1.0. For serial usage of ccache,
that is.

> What I've seen is that the cleanups are usually triggered by
> 0.8*max_size, and that does not change when I set limit_multiple =
> 0.95.

As already explained, nothing is triggered at 0.8*max_size or even at
limit_multiple*max_size, so the reason for your 24 GB cache is something
else. And that something else is most likely that when several ccache
invocations trigger a cleanup of the same subdirectory at the same time,
the net effect will be removal of more than
(1-limit_multiple)*max_size/16, potentially much more. I bet that if you
run something like "du $CCACHE_DIR/$X" for each X in [0-9a-f], or just
count the number of files in each subdirectory, you'll see some
subdirectories that are much smaller than limit_multiple*max_size/16 but
some that are near max_size/16.

***   ***   ***   ***   ***   ***   ***   ***   ***   ***   ***   ***   ***

During a couple of recent walks with my daughter in the stroller, I've
been thinking more about how to improve ccache's cleanup. I think that I
have come up with something that will be significantly better, but I
don't have time to describe any details right now. Stay tuned.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Possible to merge ccache dirs?

[Joel Rosdahl via ccache]

On 14 January 2018 at 12:01, Michael Bazzinotti via ccache
 wrote:
> My ccache was separate for both my root user and main user account. Now, I 
> have a shared ccache pointing to the main user's cache.
> How can I merge the root and user ccache directories?

Try something like this:

cd ~mainuser/.ccache
cp -a --parents ? ~root/.ccache
ccache -c

The last step will recalculate the file counters to match the new content.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.3.5 has been released

[Joel Rosdahl via ccache]

ccache version 3.3.5 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_3_5

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.3.5.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.3.5.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.3.5.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.3.5.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.3.5.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.3.5.tar.xz.asc

You can run "gpg --verify ccache-3.3.5.tar.*.asc" to check the source
archive integrity after download.

Improvements:

- Documented how automatic cache cleanup works.


Bug fixes:

- Fixed a regression where the original order of debug options could be
  lost. This reverts the "Improved parsing of -g* options" feature in
  ccache 3.3.

- Multiple -fdebug-prefix-map options should now be handled correctly.

- Fixed matching of directories in the ignore_headers_in_manifest
  configuration option.

- Fixed detection of missing argument to -opt/--options-file.

- ccache now bails out when building a precompiled header if any of the
  corresponding header files have an updated timestamp. This fixes
  complaints from clang.

- Fixed a bug related to erroneously storing a dependency file with
  absolute paths in the cache on a preprocessed hit.

- "ccache -c/--cleanup" now works like documented: it just recalculates
  size counters and trims the cache to not exceed the max size and file
  number limits. Previously, the forced cleanup took "limit_multiple"
  into account, so that "ccache -c/--cleanup" by default would trim the
  cache to 80% of the max limit.

- ccache no longer ignores linker arguments for clang since clang warns
about
  them.

- Plugged a couple of file descriptor leaks.

- Fixed a bug where ccache would skip hashing the compiler argument
  following a -fno-working-directory, -fworking-directory, -nostdinc,
  -nostdinc++, -remap or -trigraphs option in preprocessor mode.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Insufficient environment sanitization in test.sh

[Joel Rosdahl via ccache]

On 23 December 2017 at 14:49, Radu Rendec via ccache  wrote:

> I ran across a weird issue where test.sh fails in the CCACHE_NOHASHDIR
> test because the CCACHE_HASHDIR variable happens to be set in my
> environment.

[...]

My idea is to iterate through the output of `env` and unset all
> variables starting with CCACHE_ instead of listing them explicitly
> (since this is pretty much what conf_update_from_environment() does).
> Any thoughts?
>

Sounds good, I'll fix it. Thanks!

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] why is limit_multiple ignored?

[Joel Rosdahl via ccache]

Hi Scott,

On 19 December 2017 at 02:16, Scott Bennett via ccache <
ccache@lists.samba.org> wrote:

>  I set "limit_multiple = 0.95" in ccache.conf and "max_size = 30.0G"
> in ccache.conf, but cleanups are triggered when space usage reaches 24 GB,
> which is the default of 0.8.  Why is this happening with ccache 3.3.4?
>

The ccache manual is not very good at describing what actually happens at
cleanup. I'll try to improve it.

Here's how cleanup works: After a cache miss, ccache stores the object file
in (a subdirectory of) one of the 16 top level directories in the cache
(0-9, a-f). It then checks if that top level directory holds more than
max_cache_size/16 bytes (and similar for max_files). If yes, ccache removes
files from that top level directory until it contains at most
limit_multiple*max_cache_size/16 bytes. This means that if limit_multiple
is 0.8, the total cache size is expected to hover around 0.9*max_cache_size
when it has filled up. But due to the pseudo-randomness of the hash
algorithm, the cache size can be closer to 0.8*max_cache_size or
1.0*max_cache_size.

The above should be true for any serial usage of ccache. However, ccache is
of course very often called in parallel, and then there is a race condition
since several ccache processes that have stored an object to the same top
level directory may start the cleanup process simultaneously. Since
performing cleanup in a large cache with a low limit_multiple can take a
lot of time, more ccache processes may start to perform cleanup of the same
directory. The race can lead to the final cache size being below
limit_multiple*max_cache_size, perhaps very much so. This is a known
problem. We have had some ideas to improve the admittedly naive cleanup
logic, but nothing has been done yet.

Maybe the above described problem is why you get a 24 GB cache size?

Or maybe you ran "ccache -c"? Unlike what the manual indicates, "ccache -c"
will delete files until each top level directory holds at most
limit_multiple*max_size/16...

why is limit_multiple ignored?


It isn't. Or don't you see a difference if you e.g. set it to 0.5?

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] use_relative_paths_in_depfile not called in preprocessor mode

[Joel Rosdahl via ccache]

Hi Venkat,

> cache hit(preprocessed) puts the .d file from the workspace as-is into the
> cache, thus, if such a compilation resulted in absolute paths in the .d
> file(in workspace), then, the cache will also contain .d files with
> absolute paths, resulting in troubles later on(from another users
workspace)
> [...]

Thanks for noting this!

I had a look at it and I think that the

put_file_in_cache(output_dep, cached_dep);


call in from_cache is just a mistake in commit 38301c0d – it should have
been removed when I added the corresponding call in to_cache. So currently
on a preprocessed hit, the .d file that already exists in the cache (with
absolute paths adjusted to relative) will be overwritten with a new version
(potentially with absolute paths).

Thus I think that the proper fix looks like this:

--- a/ccache.c
+++ b/ccache.c
@@ -1964,11 +1964,6 @@ from_cache(enum fromcache_call_mode mode, bool
put_object_in_manifest)
update_mtime(cached_dwo);
}

-   if (generating_dependencies && mode == FROMCACHE_CPP_MODE
-   && !conf->read_only && !conf->read_only_direct) {
-   put_file_in_cache(output_dep, cached_dep);
-   }
-
send_cached_stderr();

if (put_object_in_manifest) {


What do you think?

-- Joel


On 6 October 2017 at 15:50, Karanam Suryanarayana Rao Venkata Krishna via
ccache  wrote:

> Hello,
> cache hit(preprocessed) puts the .d file from the workspace as-is into the
> cache, thus, if such a compilation resulted in absolute paths in the .d
> file(in workspace), then, the cache will also contain .d files with
> absolute paths, resulting in troubles later on(from another users
> workspace)
>
> I mean, https://github.com/ccache/ccache/blob/3.3-maint/ccache.c#L1969
> should precede a function call use_relative_paths_in_depfile(output_dep);
> just the way it is on
> https://github.com/ccache/ccache/blob/3.3-maint/ccache.c#L1307
>
> Please comment.
>
> best regards,
> Venkat.
> ___
> ccache mailing list
> ccache@lists.samba.org
> https://lists.samba.org/mailman/listinfo/ccache
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] base_dir and symbolic links

[Joel Rosdahl via ccache]

Hi Andreas,

> [...] I understand that it makes
> sense to make the current working directory canonic, but I do not
> see why this done for the path given to `make_relative_path`.  Is
> it really necessary?

Some background can be found in this mail thread:

*https://www.mail-archive.com/ccache@lists.samba.org/msg00802.html
*

The summary is that we want to normalize the paths to increase cache hits,
and one simple and robust way of doing that is to use realpath. Writing a
custom function that both normalizes paths and doesn't expand symlinks
might be possible, but note that there are tricky edge cases that need to
be handled.

> Thank you for considering it.  A patch is in the attachment.

ccache has a test suite and it fails like this with your patch:


Running test suite basedir...
FAILED

Test suite: basedir
Test case:  Path normalization
Failure reason: Expected "cache hit (direct)" to be 1, actual 0


[From your original mail:]
> For directories they do not work on, but are required to build the sources
> they work on, they create a symbolic link to a reference area in which
> only stuff required for building resides. [...] Now, assume user1 sets
> `base_dir` to `/user1` and user2 sets `base_dir` to `/user2`.

If I understand your setup correctly, would it be possible to instead put
/reference, /user1 and /user2 under a /common directory and then let the
users set basedir to /common?

-- Joel


On 17 September 2017 at 10:43, Andreas Wettstein via ccache <
ccache@lists.samba.org> wrote:

> > Seems like a pretty niche use case, but if you can come up with a
> > patch that doesn't break too much of current behaviour - why not.
>
> Thank you for considering it.  A patch is in the attachment.
>
> > Worst case, it could be an option (that defaulted to false) ?
>
> I would not consider this the worst case.  For me, it would be still
> more convenient than keeping my own modified version around.
>
> Andreas
>
>
> ___
> ccache mailing list
> ccache@lists.samba.org
> https://lists.samba.org/mailman/listinfo/ccache
>
>
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache - .d files with absolute system header paths

[Joel Rosdahl via ccache]

Hi Karanam,

> From just ccache's point of view, is there a way to force relative paths into
the
> .d files at all?

Yes, ccache version 3.3 and newer knows how to do this. See the release
notes for ccache 3.3: http://ccache.samba.org/releasenotes.html#_ccache_3_3

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.3.4 has been released

[Joel Rosdahl]

ccache version 3.3.4 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_3_4

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.3.4.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.3.4.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.3.4.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.3.4.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.3.4.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.3.4.tar.xz.asc

You can run "gpg --verify ccache-3.3.4.tar.*.asc" to check the source
archive integrity after download.

Improvements:

- Documented the different cache statistics counters.


Bug fixes:

- Fixed a regression in ccache 3.3 related to potentially bad content of
  dependency files when compiling identical source code but with different
  source paths. This was only partially fixed in 3.3.2 and reverts the new
  "Names of included files are no longer included in the hash of the
compiler's
  preprocessed output" feature in 3.3.

- Corrected statistics counter for -optf/--options-file failure.

- Fixed undefined behavior warnings in ccache found by -fsanitize=undefined.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] CCACHE_DISABLE vs. unreadable ccache.conf

[Joel Rosdahl]

On 31 January 2017 at 13:00, mar...@mejor.pl  wrote:
> So if I set CCACHE_DISABLE=1 is any reason to read other variables from config
files?

Well, ccache currently notes in the log file (if enabled) that it is has
been disabled before handing over to the compiler. And how to enable
logging? You can either set the CCACHE_LOGFILE variable or you can set
log_file in the configuration file... Another dependency on the
configuration is that CCACHE_PREFIX (prefix_command in the configuration
file) is applied even when ccache is disabled. So yes, I think it needs to
stay this way.

> Should be "Permission denied" error in reading conf files critical? Wouldn't
be better to
> omit such conf file and emit warning only?

Sounds reasonable, I'll look into it.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Fortran support issue

[Joel Rosdahl]

On 11 October 2016 at 18:07, Giacomo Comes  wrote:
> I have been using in the past the program f90cache to cache fortran compiling 
> jobs.
> Now that ccache 3.3 has support for Fortran 77 I have done some tests to see 
> if
> I can replace f90cache with ccache.

As you already have found out, ccache doesn't support Fortran 90
(modules). This is intentional, see
https://github.com/ccache/ccache/pull/90 for background. Sorry.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.3.2 has been released

[Joel Rosdahl]

ccache version 3.3.2 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_3_2

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.3.2.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.3.2.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.3.2.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.3.2.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.3.2.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.3.2.tar.xz.asc

You can run "gpg --verify ccache-3.3.2.tar.*.asc" to check the source
archive integrity after download.

Bug fixes:

- Fixed a regression in ccache 3.3 related to potentially bad content of
  dependency files when compiling identical source code but with
  different source paths.

- Fixed a regression in ccache 3.3.1: ccache could get confused when
  using the compiler option -Wp, to pass multiple options to the
  preprocessor, resulting in missing dependency files from direct mode
  cache hits.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] [PATH]: test: elide which stderr

[Joel Rosdahl]

Hi Eric,

On 23 September 2016 at 06:41, Eric Bavier  wrote:
> While running ccache's test suite via `make check` I get output from
> 'which': [...]

Thanks for the report and suggestion!

I made an alternative fix like this (which also should work with
various shells' builtin "which" implementations):

--- a/test.sh
+++ b/test.sh
@@ -3380,11 +3380,11 @@ upgrade
 input_charset
 "

-compiler_location=$(which $COMPILER)
+compiler_location=$(which $(echo "$COMPILER" | awk '{print $1}'))
 if [ "$compiler_location" = "$COMPILER" ]; then
 echo "Compiler: $COMPILER"
 else
-echo "Compiler: $COMPILER ($(which $COMPILER))"
+echo "Compiler: $COMPILER ($compiler_location)"
 fi
 echo "Compiler version: $($COMPILER --version | head -n 1)"
 echo

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.3.1 has been released

[Joel Rosdahl]

ccache version 3.3.1 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_3_1

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.3.1.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.3.1.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.3.1.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.3.1.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.3.1.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.3.1.tar.xz.asc

You can run "gpg --verify ccache-3.3.1.tar.*.asc" to check the source archive
integrity after download.

Bug fixes:

- Fixed a problem in the "multiple -arch options" support introduced in
  3.3. When using the direct mode (the default), different combinations
  of -arch options were not detected properly.

- Fixed an issue when compiler option "-Wp,-MT,path" is used instead of
  "-MT path" (and similar for -MF, -MP and -MQ) and run_second_cpp
  (CCACHE_CPP2) is enabled.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.2.8 has been released

[Joel Rosdahl]

ccache version 3.2.8 is now available.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_2_8

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.2.8.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.2.8.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.2.8.tar.xz

The source archives have been signed with my OpenPGP key (ID 5594ADB8,
fingerprint 5A93 9A71 A467 92CF 5786 6A51 996D DA07 5594 ADB8):

https://www.samba.org/ftp/ccache/ccache-3.2.8.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.2.8.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.2.8.tar.xz.asc

You can run "gpg --verify ccache-3.2.8.tar.*.asc" to check the source archive
integrity after download.

Bug fixes:

- Fixed an issue when compiler option "-Wp,-MT,path" is used instead of
  "-MT path" (and similar for -MF, -MP and -MQ) and run_second_cpp
  (CCACHE_CPP2) is enabled.

- ccache now understands the undocumented "-coverage" (only one dash)
  GCC option.

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] clang modules (-fmodules) support

[Joel Rosdahl]

On 24 August 2016 at 22:51, Piotr Wach  wrote:
> I have seen that "-fmodules" compiler flag has been recently marked as
> "hard" / unsupported (?)

Yes, "too hard" here means unsupported, which could mean either
impossible to implement or not yet implemented.

> is this because no one had time to implement it or are there real problems
> with supporting it in ccache?

For what it's worth, I haven't tried, and I don't recall hearing
anybody else mention it either. So it would be great if you could have
a look at it!

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Migration from autotools to cmake

[Joel Rosdahl]

On 12 July 2016 at 12:03, Иван Вайгульт  wrote:
> In order to enable visual studio build I did full build replica using
> cmake. [...]
> Any feedback from you is desired.

Thanks for working on this.

After some thinking I'm positive to getting rid of autotools. Although
I don't find cmake to be very elegant, it feels less bad than
autotools, at least for modern systems.

Other opinions from list members are welcome!

> Please let me know if I left something unimplemented.

I still haven't looked much at details of your implementation, but one
thing (also mentioned in previous off-list conversation) is that
everything related to creating archives for distribution is missing,
i.e. "make dist" and "make distcheck".

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] Cache misses are extremely expensive.

[Joel Rosdahl]

On 23 June 2016 at 21:14, Guillaume Campagna 
 wrote:

> Just started using ccache based on this tutorial and so far I like it.
> However, caches miss being extremely slow. [..] Note that I do use a pch,
> if that changes something. [...]


Since you included pch_defines and time_macros in CCACHE_SLOPPINESS, I
guess you've seen the instructions on
https://ccache.samba.org/manual.html#_precompiled_headers already? If so,
did you also follow the instructions about using -include or
-fpch-preprocess?

To get more information on what's happening, you can set CCACHE_LOGFILE to
a file path and then look in that file for clues.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.2.6 has been released

[Joel Rosdahl]

I'm happy to announce ccache version 3.2.6.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_2_6

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.2.6.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.2.6.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.2.6.tar.xz

The source archives have been signed with my OpenPGP key (5594ADB8):

https://www.samba.org/ftp/ccache/ccache-3.2.6.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.2.6.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.2.6.tar.xz.asc

You can run "gpg --verify ccache-3.2.6.tar.*.asc" to check the source
archive integrity after download.

Bug fixes:

- Fixed build problem on QNX, which lacks "SA_RESTART".

- Bail out on compiler option -fstack-usage since it creates a .su
  file which ccache currently doesn't handle.

- ccache now knows how to work around a glitch in the output of GCC 6's
  preprocessor.

- Fixed a bug where (due to ccache rewriting paths) the compiler could
  choose incorrect include files if CCACHE_BASEDIR is used and the
  source file path is absolute and is a symlink.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] ccache version 3.1.12 has been released

[Joel Rosdahl]

I'm happy to announce ccache version 3.1.12.

Details may be read in the release notes:

http://ccache.samba.org/releasenotes.html#_ccache_3_1_12

Source archives are available here:

https://www.samba.org/ftp/ccache/ccache-3.1.12.tar.bz2
https://www.samba.org/ftp/ccache/ccache-3.1.12.tar.gz
https://www.samba.org/ftp/ccache/ccache-3.1.12.tar.xz

The source archives have been signed with my OpenPGP key (5594ADB8):

https://www.samba.org/ftp/ccache/ccache-3.1.12.tar.bz2.asc
https://www.samba.org/ftp/ccache/ccache-3.1.12.tar.gz.asc
https://www.samba.org/ftp/ccache/ccache-3.1.12.tar.xz.asc

You can run "gpg --verify ccache-3.1.12.tar.*.asc" to check the source
archive integrity after download.

Bug fixes:

- Fixed a bug where (due to ccache rewriting paths) the compiler could
  choose incorrect include files if `CCACHE_BASEDIR` is used and the
  source file path is absolute and is a symlink.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

[ccache] Moving to github

[Joel Rosdahl]

Hi,

Ever since Tridge handed over the project to me, the main ccache Git
repository has been hosted on git.samba.org and bugs handled in
bugzilla.samba.org. I also have kept an unofficial mirror on github.

Since some time now, almost all activity regarding new features and bug
reports have been circling around the github mirror repository, so I think
that it's time to move to github officially. (It will be located under
github.com/ccache, not under my personal github account.)

If anybody thinks this would be a bad idea, please make some noise.
Otherwise I'll do the move in a week or so.

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] preprocessor mode

[Joel Rosdahl]

Hi Yoav,

> I've dug a bit deeper, and found that when doing a diff between the output
> of the preprocessor the resolved include path is emitted as part of the
> output as a comment, in our case it's
>
> # 1 "a/a.h" 1
>
> There's a flag -P to the preprocessor to remove these comments.

It sounds good to not include names of included files in the hash, I think.

> [...]
> I think this should be default, do you think there's a safety issue with
> adding -P?

I see two problems:

1. -P is likely not supported by all compilers that can be used with ccache.
2. Using -P discards #line directives that may have been put there by code
generator tools, for instance yacc/bison. This would lead to incorrect file
name references when the compiler generates warnings when compiling the
preprocessed output.

However, it's possible to get more or less the same effect (not hashing
filenames) like this:

diff --git a/ccache.c b/ccache.c
index 23103ee..dd11927 100644
--- a/ccache.c
+++ b/ccache.c
@@ -863,7 +863,6 @@ process_preprocessed_file(struct mdfour *hash, const
char *path)
  has_absolute_include_headers = is_absolute_path(path);
  }
  path = make_relative_path(path);
- hash_string(hash, path);
  remember_include_file(path, hash, system);
  p = r;
  } else {

What do you think?

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache-3.2.4 and 3.1.11 fail to build

[Joel Rosdahl]

On 23 March 2016 at 20:34, Britton Kerin  wrote:
> I unpacked it and did ./configure then make I get this:
>
> $ ./configure
> [...]
> $ make
> make: *** No rule to make target `main.o', needed by `ccache'.  Stop.
>
> Same thing for 3.1.11

That's peculiar. What OS are you using? Which version of make?

-- Joel
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache