Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/7601 to look at the new patch set (#3). support ccache to speed up ttcn3 test compilation If ccache exists in PATH, use it to cache compiled C++ object files. Rebuilding the test suite from scratch with a populated cache is an order of magnitude faster than doing an initial build. In my case, compile time for BSC_Tests goes down from 3 minutes and 12 seconds to just 3 seconds, after 'make clean'. There is a small downside: We need to tweak the generated C++ files to prevent cache misses due to timestamps inserted into the code by ttcn3_compiler. This is done by regen-makefile.sh if ccache is used. This mechanism could break in case the output of ttcn3_makefilegen changes. However, the only consequence would be a slow build due to cache misses. Another thing to note is that builds using objects from ccache will contain timestamp strings from the build which populated the cache. ccache provides a way to repopulate the cache if needed though (set CCACHE_RECACHE=1 in the environment). These tradeoffs seem acceptable to me since in return we can achieve much faster turnaround, both during local test development and during test runs on jenkins. Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82 --- M regen-makefile.sh A strip-datetime-comments.sed 2 files changed, 63 insertions(+), 1 deletion(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/01/7601/3 diff --git a/regen-makefile.sh b/regen-makefile.sh index 57eebff..2d09e63 100755 --- a/regen-makefile.sh +++ b/regen-makefile.sh @@ -12,7 +12,22 @@ test -x "$(which ttcn3_makefilegen 2>/dev/null)" || { echo "ERROR: ttcn3_makefilegen not in PATH"; exit 1; } -ttcn3_makefilegen -p -l -f $* +# Enable ccache if it can be found in path. +# This speeds up repeated builds of the TTCN3 tests by an order of magnitude +# since most of the generated C++ source files don't change very often. +# Roughly, for an initial build which takes N minutes, a complete rebuild +# after 'make clean' will only take N seconds with ccache. +# Note that ccache cannot cache compilation of .o files to .so files, so do +# not pass the -l option to ttcn3_makefilegen if ccache is used. The time +# savings promised by the -l option are dwarfed by use of ccache anyway. +if which ccache 2>/dev/null; then + USE_CCACHE=1 + MAKEFILEGEN_FLAGS="-p -f" +else + MAKEFILEGEN_FLAGS="-p -l -f" +fi + +ttcn3_makefilegen $MAKEFILEGEN_FLAGS $* sed -i -e 's/# TTCN3_DIR = /TTCN3_DIR = \/usr/' Makefile sed -i -e 's/LDFLAGS = /LDFLAGS = -L \/usr\/lib\/titan /' Makefile #sed -i -e 's/TTCN3_LIB = ttcn3-parallel/TTCN3_LIB = ttcn3/' Makefile @@ -32,3 +47,11 @@ sed -i -e 's/TTCN3_DIR = $/TTCN3_DIR = \/usr/' Makefile fi sed -i -e 's/\/bin\/compiler/\/bin\/ttcn3_compiler/' Makefile + +if [ -n "$USE_CCACHE" ]; then + # enable ccache + sed -i -e 's/^CXX = g++ $/CXX = env CCACHE_SLOPPINESS=time_macros ccache g++/' Makefile + # The Makefile must tweak generated C++ files during the 'make compile' step. + # See the strip-datetime-comments sed script for details. + sed -i -f ../strip-datetime-comments.sed Makefile +fi diff --git a/strip-datetime-comments.sed b/strip-datetime-comments.sed new file mode 100644 index 0000000..217dedc --- /dev/null +++ b/strip-datetime-comments.sed @@ -0,0 +1,39 @@ +#!/usr/bin/sed +# This sed script edits a Makefile generated by ttcn3_makefilegen. +# We insert a sed command into the 'compile:' target in order to +# remove timestamp comments from generated C++ files. + +# The ttcn3_compiler will generate C++ comments with a timestamp, such as: +# // for Stefan Sperling (stsp@fintan) on Sat Mar 31 14:34:30 2018 +# We must remove such comments since they cause cache misses with ccache. +# +# A related problem is the use of __DATE__ and __TIME__ in generated C++ code, +# which can be worked around by setting "CCACHE_SLOPPINESS=time_macros". This +# workaround implies that we must get our cache hits from ccache's "direct" +# cache, since ccache's "preprocessor" caching method won't work. +# +# FIXME: This should really be addressed in ttcn3_compiler itself! +# It should provide an option to suppress use of timestamps in generated code. + +# The generated Makefile's compile target looks like this: +# +# compile: $(TTCN3_MODULES) $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES) +# $(TTCN3_DIR)/bin/ttcn3_compiler $(COMPILER_FLAGS) \ +# $(TTCN3_MODULES) $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES) - $? +# touch $@ +# +# Before the final 'touch $@' step, we now insert a `sed` command that strips +# offending C++ comments from the generated code. + +# match the line in the generated Makefile which starts with 'compile:' +/^compile:/ { + # go to the next line + n; + # go to the next line + n; + # go to the next line + n; + # on this line, try to match 'touch $@', and if there is a match preprend a + # sed command which deletes the offending comments from generated C++ files: + s!touch $@!sed -i -e '/\\/\\/ for .* on .*/d' $(GENERATED_HEADERS) $(GENERATED_SOURCES) \&\& &! +} -- To view, visit https://gerrit.osmocom.org/7601 To unsubscribe, visit https://gerrit.osmocom.org/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82 Gerrit-PatchSet: 3 Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Owner: Stefan Sperling <ssperl...@sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: Neels Hofmeyr <nhofm...@sysmocom.de> Gerrit-Reviewer: Stefan Sperling <ssperl...@sysmocom.de>