Tim Armstrong has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Add ccache support
Here we add ccache support, which speeds up building the entire
toolchain. This almost works out of the box, except for the following
complications:
* We bootstrap our own gcc, so CCACHE_COMPILERCHECK must be set to
'content', this is because by default ccache includes the modification
time of the compiler, which would prevent us to reuse the cache
between runs.
* We specifically set CC and CXX variables, the initial approach of
setting them to 'ccache $CC' breaks boost (and possibly others)
because they try to run `ccache $CC`. A second approach that i tried
was relying on PATH ordering to make ccache execute the right thing.
When using the system compiler, we want:
PATH=$CCACHE_PATH:/usr/bin:...
CC=$CCACHE_PATH/gcc
After bootstrapping we want:
PATH=$CCACHE_PATH:build/gcc-4.9.2:$PATH
CC=$CCACHE_PATH/gcc
I think complex modification of PATH is too britle and would cause
ccache to execute the system compiler when $CCACHE_PATH == /usr/bin.
Instead we create a wrapper around ccache that executes CC and place
it at the beginning of the path, we set CC to this wrapper script.
I ran `make -j DISTROS="debian8 redhat6 redhat7 sles12 ubuntu1604 ubuntu1804"`
on the following configs:
disabled ccache: 3.77user 1.80system 2:08:37elapsed 0%CPU
(0avgtext+0avgdata 52680maxresident)k
empty ccache: 3.97user 1.83system 2:15:51elapsed 0%CPU
(0avgtext+0avgdata 49956maxresident)k
empty ccache (compressed): 3.86user 2.05system 2:15:12elapsed 0%CPU
(0avgtext+0avgdata 53824maxresident)k
full ccache: 2.25user 1.79system 49:14.07elapsed 0%CPU
(0avgtext+0avgdata 47520maxresident)k
full ccache: (compressed): 2.40user 1.80system 49:45.01elapsed 0%CPU
(0avgtext+0avgdata 51484maxresident)k
Here's ccache -s after the full ccache build (statistics were cleared
before strting the build).
+ ccache -s
cache directory build_docker/ccache
primary config build_docker/ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 138821
cache hit (preprocessed) 17437
cache miss 225
cache hit rate 99.86 %
called for link 15551
called for preprocessing 9567
multiple source files 42
compile failed 7705
preprocessor error 2647
bad compiler arguments 6047
unsupported source language 42
autoconf compile/link 18651
unsupported compiler option 342
unsupported code directive 24
no input file 4323
cleanups performed 0
files in cache 289759
cache size 30.5 GB
max cache size 50.0 GB
Statistics using CCACHE_COMPRESS=1 are similar but cache size is ~7GB.
Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Reviewed-on: http://gerrit.cloudera.org:8080/12574
Reviewed-by: Philip Zeyliger <[email protected]>
Tested-by: Tim Armstrong <[email protected]>
---
M .gitignore
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
M docker/debian7.df
M docker/debian8.df
M docker/redhat6.df
M docker/redhat7.df
M docker/sles12.df
M docker/ubuntu1204.df
M docker/ubuntu1404.df
M docker/ubuntu1604.df
M docker/ubuntu1804.df
M functions.sh
M in-docker.py
M init-compiler.sh
M init.sh
16 files changed, 154 insertions(+), 113 deletions(-)
Approvals:
Philip Zeyliger: Looks good to me, approved
Tim Armstrong: Verified
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 3
Gerrit-Owner: Hector Acosta <[email protected]>
Gerrit-Reviewer: Hector Acosta <[email protected]>
Gerrit-Reviewer: Philip Zeyliger <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
