Re: [ccache] Migrating to ccache 4: a "drop-in replacement"?
Hi!
Thanks so much for all your detailed and helpful answers!
On Sat, Dec 5, 2020 at 3:30 PM Joel Rosdahl wrote:
> On Thu, 3 Dec 2020 at 11:28, Steffen Dettmer wrote:
> > a "plug-in replacement"?
>
> Yes to all instances of the question.
>
Wow, I didn't hope that it could be that great! Thanks so much!
> There is no difference in how cleanup works between old and new versions.
> In
> other words, both new and old ccache versions will clean up files
> generated by
> both new and old versions.
>
Ohh that is great news too!
> > [...] release notes tell that if ~/.ccache/ccache.conf exists, it will be
> > used (release notes are not clear for me if only as fallack or in any
> case
> > when it exists).
>
> See
> <
> https://ccache.dev/manual/4.1.html#_location_of_the_primary_configuration_file
> >.
>
ahh silly me, I had looked there for myself. Thank you, this is very clear.
> Also CCACHE_CONFIGPATH could be set to point to it if needed, but it is
> not
> > needed.
>
> There is almost never a reason to set CCACHE_CONFIGPATH. Instead, just set
> CCACHE_DIR to ~/.ccache if you want the old behavior.
>
> > $ grep -v '^#' ~/.ccache/ccache.conf
> > cache_dir=/...shared_local_path.../tmp/ccache
>
> This won't have any effect. cache_dir only has an effect if set in the
> secondary (system-wide read-only) configuration file (typically
> /etc/ccache.conf) or in the environment variable form ($CCACHE_DIR).
I just tested with ccache 3.7.12 and "ccache -s" that it seems to work as I
expect,
like:
$ $path/bin/ccache -s|grep "config"
primary config /nfshomes/sdettmer/.ccache/ccache.conf
secondary config (readonly)/usr/local/etc/ccache.conf
$ file /usr/local/etc/ccache.conf
/usr/local/etc/ccache.conf: cannot open `/usr/local/etc/ccache.conf' (No
such file or directory)
$ grep ^cache_dir ~/.ccache/ccache.conf
cache_dir=/tmp/sdettmer/ccache
$ $path/bin/ccache -s|grep "cache directory"
cache directory /tmp/sdettmer/ccache
$ grep ^cache_dir ~/.ccache/ccache.conf
cache_dir=/tmp/ccache-all/ccache
$ $path/bin/ccache -s|grep "cache directory"
cache directory /tmp/ccache-all/ccache
What do I miss?
(off topic)
> NB: Our reference system is "Devuan 2.0 ASCII", a Debian fork without
> > SystemD.
>
> Sorry to hear that (SCNR).
>
Thank you for sharing your expertise! Although not ccache specific,
I'm glad to get your input because your input usually is just great!
Do you think this is bad or could cause problems?
Do you have someone concrete in mind?
As far as I know, our main reason was, that our embedded systems don't
have SystemD but use init.d scripts, so we made the development ref sys
also don't having SystemD, assuming it would avoid trouble (SystemD is
said to boot faster, but boot time was no issue for us).
Much information about SystemD pros/cons in the web [1] seem to be
very polarized and was not found to be sufficient.
I'm afraid just because all major distros use it, avoiding it could
cause trouble[2].
What do you think, should we better use a mainstream distro
(Debian or any clone, Mint or whatever)?
Let me close with an old joke:
Finally SystemD now is almost feature complete!
Just an integrated kernel and a decent browser are missing. :-)
Steffen
[1] Good example is
http://0pointer.de/blog/projects/why.html which shows properties many
consider disadvantages ("Interfacing via D-Bus", "Shell-free bootup",
"fsck handling"...) as "features" -- but it contains no single feature
SystemD has not ("customisation without being C programmer",
"indepentent small building blocks leaving the choice", "deterministic",
"easy to trace", "simple to understand", "applications don't need to
depend on it", "no hardcoded DNS IPs"...).
[2] such as what we had:
that ssh-agent did not work anymore, because gnome-keyring
overwrote its env but did not work, because the xinitrc forcibly
loads gnome-keyring with shell code that was hacked to work for
Wayland and in short only works with SystemD, and all that because
Wayland with SystemD does not work with ssh-agent because user processes
are not forked from the xinit'd process.
I'm not sure but I think it somehow worked with systemd (someone else
had reported I think).
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Speeding up builds with clang-tidy, clang-analyzer and iwyu
Hi, On Sun, Aug 2, 2020 at 6:55 PM Christian Ledergerber via ccache < ccache@lists.samba.org> wrote: > for our builds currently clang-tidy is currently the limiting factor in > terms of speed. > > There is one utility out there which tries to cache the output of > clang-tidy, however it seems to be stale: > https://github.com/ejfitzgerald/clang-tidy-cache > > Yet, I have the impression that my case is much simpler: Do you have any new regarding caching clang-tidy? Here we also occasionally discuss things and we found the following opinions in our teams: 1) clang-tidy shall be in the IDE: it can apply fixes which needs interaction 2) clang-tidy shall be a Jenkins job: When you are using it for a while, number of warnings reduce a lot and it is not necessary to run it every time 3) clang-tidy shall be ran at every compilation: it is -Wall -Wextra -Wtidy made correctly For 3), which seems to be close to your use case, someone proposed in context of a legacy project to run clang-tidy along with the compiler. The discussed approach was to have a wrapper script to be used as a compiler, but the wrapper script also called clang-tidy before it ran the compiler. In this case, ccache would not call the compiler in case of a cache hit, which here means it would not call the wrapper script and thus not clang-tidy, so I think this could help you. Maybe. (the idea was quickly discarded because there was absolutely no way to fix the legacy code due to its ancient C++ and we joked about adding a --museum-mode to clang-tidy :)) Steffen ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Migrating to ccache 4: a "drop-in replacement"?
Hi,
in short:
Is upgrading to ccache 4 a "drop-in replacement" even if sharing
the cache directory with still running ccache 3 versions?
in detail:
I consider upgrading to ccache 4 with or without sharing the
cache directory with older ccache (~3.7.7). Sharing the directory
could ease migration for users of course.
The new ccache version would be automatically deployed to the
users and used by all recent software branches, but the old
ccache would still be used by older branches (toolchain tool
versions are "pinned" for better reproducibility). On some
servers, the cache directory is shared across users.
ccache version 4.0 introduces a new cache directory structure [1]:
> The cache directory structure has changed compared to previous
> versions (more details below). This means that ccache 4.0 will
> not share cache results with earlier versions. It is however safe
> to run ccache 4.0 and earlier versions against the same cache
> directory: cache bookkeeping, statistics and cleanup are backward
> compatible, with the minor exception that some statistics
> counters incremented by ccache 4.0 won’t be visible when
> running ccache -s with an older version.
About the cache directory structure, I didn't find "more details
below", but the description seems to be clear that sharing the
cache directory wouldn't be any problem, even cleanup should
work: it is backward compatible. If I understand correctly, this
means as long as the new ccache version runs from time to time,
it will cleanup both old and new files and avoid ccache directory
grow (much) beyond max_size. That sounds great! Does it really
make new versions a "plug-in replacement"? That would be so cool!
Also config file location has changed to match the expectations
on Poetterings laptop [2] (SCNR), but release notes tell that if
~/.ccache/ccache.conf exists, it will be used (release notes are
not clear for me if only as fallack or in any case when it
exists). Also CCACHE_CONFIGPATH could be set to point to it if
needed, but it is not needed.
So in short: I would not need to change anything at all. Just
deploy it.
Is this correct?
Sorry for the long mail, but using ccache in detail can be quite
complex and I try to avoid missing an important detail.
Steffen
In case it helps our configuration:
$ grep -v '^#' ~/.ccache/ccache.conf
max_size = 0
cache_dir=/...shared_local_path.../tmp/ccache
hard_link=false
umask=002
(only our Jenkins account runs with max_size = 50G to have a
central place to set the size)
cmake wrapper script example (it depends on the build):
#!/bin/bash
# Xcode generator doesn't include the compiler as the first arg:
[ "$1" = "/.../i586-pc-linux-gnu/gcc-4.4.3/usr/bin/i586-pc-linux-gnu-g++"
] && shift
# "Convert" cmake variable to an env (but keep present values) to allow
shared caches
: ${CCACHE_BASEDIR="/.../sdettmer/work/xyz-src"}
: ${CCACHE_CPP2="set"}
# : ${CCACHE_NODIRECT="set"}
export CCACHE_BASEDIR CCACHE_CPP2 CCACHE_NODIRECT
unset GCC_COLORS
{ "/.../bin/ccache" "/.../gcc-4.4.3/usr/bin/i586-pc-linux-gnu-g++" "$@"
2>&1 1>&3 3>&- | sed "s|^\(In file included from \)\?\.\./|\1$(pwd)/../|"
; exit ${PIPESTATUS[0]} ; } 3>&1 1>&2
NB: Our reference system is "Devuan 2.0 ASCII", a Debian fork without
SystemD.
[1] https://ccache.dev/releasenotes.html#_ccache_4_0
[2]
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
[ccache] Thank you
Hi, about half a year ago I started adding ccache to the proprietary toolchains at my workplace. I met several challenges (difficult environments) but I got so much help from Joel Rosdahl and others that they could be solved. I'm so glad that I added it, it works so fine now and saves so much! On my private machine I get around 50% cache hits and team mates have similar values. But most effective is the shared cache on a build server where Jenkins jobs and interactive shells share a ccache: stats updated Mon Nov 16 18:47:59 2020 stats zeroedThu Apr 2 19:57:58 2020 cache hit (direct) 3892380 cache hit (preprocessed) 650573 cache miss579045 cache hit rate 88.69 % compile failed 353 ccache internal error 10 preprocessor error 768 no input file 2 cleanups performed 995 files in cache450926 cache size 50.0 GB I hoped it would save, but that it would save that much I didn't expect. It is just great! This saves a lot of time, energy and even CO2. Thank you so much! Steffen ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using a shared ccache in cmake environment (linux)
Hi, thank you so much for your so valuable input!! On Sat, Mar 21, 2020 at 2:19 PM Joel Rosdahl wrote: > > On Tue, 17 Mar 2020 at 10:06, Steffen Dettmer via ccache > wrote: > > As workaround for a special unrelated issue currently we redefine > > __FILE__ (and try to remove that redefinition). I understand that > > ccache still works thanks to CCACHE_BASEDIR even for __FILE__ usage > > inside files. Is that correct? > > Yes, if basedir is a prefix of the the source file path then __FILE__ > will expand to a relative path since ccache passes a relative source > file path to the compiler. Thanks for confirmation. For us this now seems to work fine! Just BTW, isn't it common to build in some $builddir different from top $srcdir (e.g. automake, cmake) and in that case couldn't the common case need two base directories? (for me its no problem, I just build in a subdir below BASEDIR, but there are team mates who hate this and build in ramdisk or so). > > I understood that CCACHE_SLOPPINESS=file_macro means that cache > > results may be used even if __FILE__ is different, i.e. using a > > __FILE__ from another user (fine for our usecases), is this correct? > > That used to be the case, but the file_macro sloppiness was removed in > 3.7.6; see <https://ccache.dev/releasenotes.html#_ccache_3_7_6>. Ahh, thanks for the pointer. I think I now remember that someone posted about hunting some strange bug down to disassembly only to find something like that as a cause. Indeed, such case once cannot be ever saved by reduced compilation times. Good that you fixed it. > > How to find a reasonable max_size? For now I just arbitrarily picked > > 25 GB (approximately the build tree size) and I never saw it "full" > > according to ccache -s. > > "cache size" will never reach "max cache size", so that is not a good > indicator of whether "max cache size" is enough. See > <https://ccache.dev/manual/3.7.8.html#_automatic_cleanup> for details on > how automatic cache cleanup works. The TLDR is that "cache size" will > stay around 90% (assuming limit_multiple is the default 0.8) of "max > cache size" in steady state. This is because each of the 16 > subdirectories will be between 80% and 100% full with uniform > probability. Thanks for your great explanation! I read this (and almost understood it right, but just almost) and with "full" I meant more than 80% (I just saw slightly over 50%). > Instead you can have a look at "cleanups performed". Each time a > subdirectory gets full that counter will increase by 1. Ahh, this of course is a great idea, of course. I will watch. (Actually I wonder why I didn't had it in first place) > Especially with network caches it might be a good idea to disable > automatic cleanup and instead perform explicit cleanup periodically on > one server, preferably the server that hosts the filesystem. That way > the cleanup won't happen over slow network and several clients won't > compete to clean up. One way of doing this is to set an unlimited cache > size and then run something like "CCACHE_MAXSIZE=25G ccache -c" > periodically the server. This is again is a great idea. Will clean recover from corrupted caches, or should I add some script like "when each cache value is zero, clear it"? I think I could set a high "safety" value for the Jenkins user (ran locally) and have a smaller periodic cleanup after the nightly builds, later. > > Is sharing via CIFS possibly at all or could it have bad effects? > > Don't know, but I wouldn't be surprised if ccache's locking doesn't work > properly with SMB/CIFS. Locking is based on creating symlinks atomically > and I guess that doesn't translate well to Windows filesystems. Thanks for your clarification. I disabled the Samba share (just needed to reconfigure two repositories driving auto-updating docker containers, isn't it simple nowadays lol) and now it seems to run very well. I guess CIFS was the root of all our issues. > > Are cache and/or stats version dependent? > > The cache data and stats files are intended to be backward and forward > compatible from ccache 3.2. ahh that's good to know, so in case someone accidentally uses a wrong version, we shouldn't face issues. Great work! > > I'm also still facing scmake issues (using "physical" and "logical" in > > several mixed combinations). Complex topic. > > What is scmake? A typo! cmake what was I wanted to write :) Our issue somehow it that there are cases where cmake uses physical path instead of logical and then BASEDIR isn't matching. I didn't yet understand the whole topic, I need to read when I have a bit more time (https://discourse.cmake.org/t/when-does-cmake-current-binary-dir-resolve-symlinks/809) Thank you for your great support again!! Steffen ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using a shared ccache in cmake environment (linux)
Hi, * On Tue, Mar 17, 2020 at 3:54 PM Paul Smith wrote: > You don't say which compiler you're using but if you're using GCC you > can consider using the -ffile-prefix-map option to avoid these issues. Thanks for the tip! Your are right, this is for gcc, sorry that I didn't mention. Unfortunately only gcc-8, but also I have to support 4.4.3, 4.5.1 and others. These have only -fdebug-prefix-map (which I use). Steffen ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Using a shared ccache in cmake environment (linux)
Hi,
I setup ccache with shared cache for some of our projects and I would
like to learn how to do it correctly.
Projects here have 1-2 MLOC in 3-10k files (mostly C++) and are built
via cmake files. We have around 20 devs active typically plus Jenkins
and mostly they compile the same inputs (HEADs of active branches with
only their few changes) in various configurations (targets,
debug/release...), in total build output can be 5-30 GB. I made some
tests and found ccache be very efficient (reducing total build
duration by ten or so, depends on many factors of course). However, I
still have some issues.
Setup:
using cmake wrapper scripts that exports:
- CCACHE_BASEDIR
- CCACHE_SLOPPINESS=file_macro,time_macros
- CCACHE_CPP2=set
and havine ccache.conf like:
max_size = 25.0G
# find $CCACHE_DIR -type d | xargs chmod g+s
cache_dir=/local/users/zcone-pisint/tmp/ccache
hard_link=false
umask=002
The straight case ("ccache -Cz && make clean all && make clean all &&
ccache -s") works as expected, first build slow, second very fast, 50%
hits.
Is this so far reasonable?
As workaround for a special unrelated issue currently we redefine
__FILE__ (and try to remove that redefinition). I understand that
ccache still works thanks to CCACHE_BASEDIR even for __FILE__ usage
inside files. Is that correct?
I understood that CCACHE_SLOPPINESS=file_macro means that cache
results may be used even if __FILE__ is different, i.e. using a
__FILE__ from another user (fine for our usecases), is this correct?
NB: unfortunately cmake uses absolute paths, so __FILE__ contains user
specific information (currently we redefine it not to do so, but we
might drop this, because it harms other things).
How to find a reasonable max_size?
For now I just arbitrarily picked 25 GB (approximately the build tree
size) and I never saw it "full" according to ccache -s.
On build servers we usually run "make -j 25" (24 cores). Often,
several such jobs are running by different users (and Jenkins;
sometimes 400 compiler processes or even more). I assume ccache of
course safely handles parallel invocation, is this correct?
We have some team mates that have slow (old) laptops only. They
benefit from using a network shared ccache. Technically, they "mount
-t cifs" the cache_dir (NFS is firewalled unfortunately). We have
different Ubuntu/Mint/Debian/Devuan machines, but exactly the same
compilers (own toolchains).
Is sharing via CIFS possibly at all or could it have bad effects?
One issue that occures from time to time is that the ccache -s stats
become zero (all values expect max cache size are 0). I first didn't
notice because stats are shared so I assume someone zeroed the stats,
but with alternate directories we found that it sometimes happens
without ccache -s. "du -hs $CCACHE_DIR" still shows gigabytes used. We
didn't find a cause yet, but several candidates exist.
Can ccache be used on CIFS?
Are cache and/or stats version dependent?
I tried to deploy the same ccache everywhere (3.7.7, now
3.7.7+8_ge65d6c92), but maybe there is some host somewhere with an
older version, hard to say.
A few times we noticed that ccache -s reports few GB size but "du -hs"
reports 40 or 50 GB, although "max_size = 25.0G". Is this expected?
Could be a follow up problem from the one before.
I'm also still facing scmake issues (using "physical" and "logical" in
several mixed combinations). Complex topic.
Any information / hints / pointers are appreciated!
Best regards,
Steffen
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
