Code Analysis Tools

This page contain a guide to using various code profiling and analysis tools on the LHCb software. Please feel free to add any information to this page.

The "valgrind, callgrind and kcachegrind" Utilities

Valgrind is a general purpose utility for analyzing software. It contains various "tools" that perform tasks such as memory allocation checking, heap analysis and code profiling.

Building valgrind and callgrind

valgrind and callgrind can be built easily using the tar files available from the respective web pages (see above). However, it appears that the size of our LHCb applications is larger than can be handled by the default values coded into valgrind. This is seen by the error

--15591:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
--15591:0:aspacem    Increase it and rebuild.  Exiting now.

It can be easily fixed, by doing as it says ! The file is coregrind/m_aspacemgr/aspacemgr-linux.c in the valgrind build directory and I found increasing the value from 2000 to 10000 seems to do the trick.

Usage at CERN

The version of valgrind installed by default on lxplus has problems running with applications that require reasonably large amounts of memory due to bugs in its internal memory model. It also cannot run on the normal lxplus nodes due to the virtual memory limit in place - e.g. you will see

[lxplus066] ~ > valgrind --version
valgrind: mmap(0x8bf5000, -1488932864) failed during startup.
valgrind: is there a hard virtual memory limit set?

These bugs have been fixed in the more recent versions than that installed. Unfortunately such a version is not available by default, so I had to install it privately. To access this version run :-

 > source /afs/cern.ch/lhcb/group/rich/vol4/jonrob/scripts/new-valgrind.csh

 > source /afs/cern.ch/lhcb/group/rich/vol4/jonrob/scripts/new-valgrind.sh

for csh or bash like shells respectively.

which currently provides the latest versions of valgrind and callgrind, with a private patch applied to increase VG_N_SEGMENTS to 10000, which seems enough for the LHCb applications I have tried (If you find otherwise, please let me know).

Memory Tests

The main use of valgrind is perform a memory tests of your code for things like memory corruptions, memory leaks and uninitialised variables, using the memcheck valgrind tool.

Unfortunately, valgrind cannot be used directly with gaudirun.py, so you should first use gaudirun.py to generate "old style" Job Options:

 > gaudirun.py options.py -n -v --old-opts > options.opts

Depending on your options and the version of gaudirun.py used to generate them, options.opts may contain some invalid comment lines. If this happens simply edit the file by hand and remove these lines.

Then invoke valgrind with something like :-

 > valgrind --tool=memcheck Gaudi.exe options.opts

It is useful to add a few additional options, to improve the quality of the output. The options -v --error-limit=no --leak-check=yes --num-callers=9999 --leak-check=full increase the amount of information valgrind supplies.

In additional, there are some warnings which are well known about (for instance from the Gaudi framework or even thrid party libraries, like STL or boost) and it is best to suppress these, otherwise the amount of output to deal with is huge. This can be done with --suppressions=$STDOPTS/Gaudi.supp.

So the final full command is then :-

 > valgrind --tool=memcheck -v --error-limit=no --leak-check=yes --num-callers=9999 --leak-check=full --show-reachable=yes --suppressions=$STDOPTS/Gaudi.supp Gaudi.exe options.opt

The above command will send the output to the terminal. It is best to redirect this to file. One additional complication is the valgrind output is sent to STDERR, not STDOUT, so you will need to redirect both to file... This can be done (for bash like shells) by appending &>mem.log to the full valgrind command (for csh like shells use >& mem.log).

This will produce a large output file contain detailed information on any memory problems. You can either simply read this file directly, or if you prefer use the alleyoop application to help interpret the errors.

Code Profiling

A valgrind tool called "callgrind" also exists which does some detailed code profiling. For valgrind versions 3.1.1 and earlier callgrind is not included in the main valgrind package, it must be installed seperately. As of valgrind 3.2.0, callgrind was integrated into the mainstream valgrind package.

In addition, a nice GUI is available to view the output of this tool, called "kcachegrind" (kcachegrind can view the output of cachegrind, but despite its confusing name it is actually primarily designed for callgrind).

See here for more details on callgrind and kcachegrind.

In the simple case, usage is just

 > valgrind --tool=callgrind application

and where application is for example "Boole.exe options-file". This will produce an output file of the form callgrind.xxxxx, which can be read in by kcachegrind. More information on available command line options are given on the web page, or by running "callgrind --help or valgrind --help".

The options --dump-instr=yes --trace-jump=yes are also useful as they provide more information.

One very useful option is "--dump-before" which can be used for the creation of an output file before calling particular methods. Using this with for instance BooleInit::execute allows the creation of one dump per event, which can then be read in individually. I.e. a full command line could be

 > valgrind --tool=callgrind -v --dump-instr=yes --trace-jump=yes --dump-before="BooleInit*execute*" Gaudi.exe options.opts

for Boole, or for Brunel

 > valgrind --tool=callgrind -v --dump-instr=yes --trace-jump=yes --dump-before="RecInit*execute*" Gaudi.exe options.opts

( Note the *'s are needed with the --dump-before method, I think to cope with name mangling under -O2 )

Also note, valgrind generally requires a large amount of memory and CPU, and thus you may run into problems with the CPU time and virtual memory size limits in place on the general purpose lxplus nodes. If this happens, try using a private afs box (e.g. pclhcbXX.cern.ch) if such a machine is available to you. If not, you could try running on one of the lxbuildXXX machines (XXX = 035 or 042, although we should avoid all running at the same time on the same node ...).

kcachegrind is now available on lxplus - see here for details. Note, sourcing the above setup file (/afs/cern.ch/lhcb/group/rich/vol4/jonrob/scripts/new-valgrind.csh) adds the appropriate path for kcachegrind to your PATH.

Using kcachegrind takes some getting used to. One of the first things to do is add your code location (e.g. ~/cmtuser) to the list of known locations (settings -> configure kcachegrind). The kcachegrind web page contains more hints on getting started.

Debug and Optimised Libraries

callgrind can work on both optimised (CMTCONFIG) and un-optimised (CMTDEB) builds, although when running on optimised builds bear in mind you will not get all possible information. For instance inline functions, by their nature, are optimised away (those which are actually inlined by the compiler) and thus are not seen by callgrind and will not appear in the profiling. Similarly you will not get annotated source code listings. This does not happen in un-optimised builds, where all information is available, but of course you must then bear in mind the timing information you get will be different. A good approach is to first try running the normal optimised build. If you find you want finer grained information than that this method provides, then try the debug build.

Also, much more information is available if the "-g" option is used. Our un-optimised debug (CMTDEB) builds have this but not in our optimised (CMTCONFIG) builds. If you want to use this with your optimsed builds, this can be done by checking out a private version of GaudiPolicy, update the optimisation flags in the requirements file to include "-g" as well as "-O2" and rebuild the libraries you wish to profile. Note this will increase the size of the binaries so if you have many libraries and a small AFS user quota you might need to increase it (email Joel Closier).

Understanding the output

The Atlas valgrind TWiki contains hints on interpreting the valgrind output: https://twiki.cern.ch/twiki/bin/view/Atlas/UsingValgrind

Debugging gaudirun.py on Linux

We recommend that you use Emacs to start a debug session. The following works:

 > emacs
 M-X gdb  (or Esc-X on many keyboards)
 gdb python
 (gdb) run `which gaudirun.py` yourOptions.py

You can use the emacs toolbar to set break in lines, unset them and issue debugger commands, or you can pass them as command lines at the (gdb) prompt. In which case here are a couple of useful short-cuts:

(gdb) Ctrl- up-arrow/down-arrow allows to navigate through the commands you already typed

[linuxkernelnewbies] CodeAnalysisTools < LHCb < TWiki