Reformatted so it will look a little more readable as a text only email
(especially the table).
Thanks,
Marc
---------------------
Hi All,
I spent a little bit of time investigating how to reduce the build time of
gem5. I find that even when I don't edit a single file, it takes scons
around 1 min to tell me the build is up to date. I have identified several
build optimizations. On my system I was able to reduce an empty build (run
scons without any file edits) from 67 seconds to 24 seconds.
I build gem5 on an NFS filesystem. My general use case is to clone gem5
onto my mounted user directory (which creates an obvious IO bottleneck). In
addition, I am able to run experiments out of a local /tmp directory.
I ran experiments in both /tmp and the mounted network directory using the
time command in linux. I found the user and system time out of both the
local and mounted directories to be be the same. The total time (which
includes IO) was around 2 seconds slower than usr+system out of /tmp and
much slower from my mounted drive.
For brevity, I report the experiments ran out of my mounted directory with
usr/system distinguished from total time. The time on /tmp can be
extrapolated by adding ~2 seconds to the usr+system times. I did not use
the parallel scons option since no files are actually being compiled.
A table summarizing the data that I collected is below. Further down, I
explain each data point.
-----------------------------------------------------------------------------------------------
usr+system (s)
total (s)
1. normal ruby build 45.2 66.9
2. set NO_HTML to True 54.3 44.9
3. 1 makeEnv call in src/SConscript 41.5 29.4
4. decider/--implicit-cache 19.9 23.6
-----------------------------------------------------------------------------------------------
1. normal ruby build:
This is just an out the box build of gem5. All data points including this
one use the Ruby memory system.
cmd: time scons --default=X86 build/test/gem5.debug RUBY=True
PROTOCOL=MI_example NO_HTML=False
2. set NO_HTML to True:
Normal ruby build with NO_HTML set to True.
cmd: time scons --default=X86 build/test/gem5.debug RUBY=True
PROTOCOL=MI_example NO_HTML=True
question: Should NO_HTML default to true instead of False?
3. 1 makeEnv call in src/SConscript:
At the bottom of src/SConscript, there are 4 calls to the makeEnv function
(for debug, opt, fast, and profiling). In this experiment, I commented out
all of the calls to makeEnv except the one for debug. The bottleneck in
makeEnv is line 919: targets = new_env.Program(progname, main_objs +
static_objs). It is also worth noting that it takes ~10s to reach the first
makeEnv call. This 10 seconds seems to be distributed throughout the
src/SConscript file.
cmd: time scons --default=X86 build/test/gem5.debug RUBY=True
PROTOCOL=MI_example NO_HTML=True
*commented 3 makenv calls at bottom of src/Sconscript (lines
960-974)
question: Is it necessary to call makeEnv for all 4 binaries when a single
compile only generates a single binary?
4. decider/--implicit-cache:
a. By default, scons seems to run a checksum on every file to determine if
it has changed. Scons can be configured to only do a checksum if the time
stamp has changed via the decider function.
b. An --implicit-cache option can be passed through the command line to
reduce the number of dependency checks done on header files (at least this
is my understanding). I believe there is a way to make this the default
behavior as well.
I found that out of /tmp, the -implicit-cache option made a big difference
by itself and the decider function didn't do much. On my mounted directory,
neither of these optimizations did much alone, but together they made a big
difference.
cmd: time scons --implicit-cache --default=X86 build/test/gem5.debug
RUBY=True PROTOCOL=MI_example NO_HTML=True
*commented 3 makenv calls at bottom of src/Sconscript (lines
960-974)
*add line main.Decider('MD5-timestamp') to top-level SConstruct
file (line 183)
question: Should the decider be configured to default to MD5-timestamp?
Should the implicit-cache option be the default behavior?
5. ???
question: Can anything else be done to speed up the gem5 build?
Thanks,
Marc
On Thu, Mar 1, 2012 at 6:14 PM, Marc Orr <[email protected]> wrote:
> Hi All,
>
> I spent a little bit of time investigating how to reduce the build time of
> gem5. I find that even when I don't edit a single file, it takes scons
> around 1 min to tell me the build is up to date. I have identified several
> build optimizations. On my system I was able to reduce an empty build (run
> scons without any file edits) from 67 seconds to 24 seconds.
>
> I build gem5 on an NFS filesystem. My general use case is to clone gem5
> onto my mounted user directory (which creates an obvious IO bottleneck). In
> addition, I am able to run experiments out of a local /tmp directory.
>
> I ran experiments in both /tmp and the mounted network directory using the
> time command in linux. I found the user and system time out of both the
> local and mounted directories to be be the same. The total time (which
> includes IO) was around 2 seconds slower than usr+system out of /tmp and
> much slower from my mounted drive.
>
> For brevity, I report the experiments ran out of my mounted directory with
> usr/system distinguished from total time. The time on /tmp can be
> extrapolated by adding ~2 seconds to the usr+system times. I did not use
> the parallel scons option since no files are actually being compiled.
>
> A table summarizing the data that I collected is below. Further down, I
> explain each data point.
>
>
> usr+system (s) total (s) 1. normal ruby build 45.2 66.9 2. set NO_HTML
> to True 54.3 44.9 3. 1 makeEnv call in src/SConscript 41.5 29.4 4.
> decider/--implicit-cache 19.9 23.6
> *1. normal ruby build:*
> This is just an out the box build of gem5. All data points including this
> one use the Ruby memory system.
>
> *cmd: time scons --default=X86 build/test/gem5.debug RUBY=True
> PROTOCOL=MI_example NO_HTML=False*
>
> *2. set NO_HTML to True:
> * Normal ruby build with NO_HTML set to True.
>
> *cmd: **time scons --default=X86 build/test/gem5.debug RUBY=True
> PROTOCOL=MI_example NO_HTML=True*
> *
> question: Should NO_HTML default to true instead of False?*
>
> *3. 1 makeEnv call in src/SConscript**:*
> At the bottom of src/SConscript, there are 4 calls to the makeEnv function
> (for debug, opt, fast, and profiling). In this experiment, I commented out
> all of the calls to makeEnv except the one for debug. The bottleneck in
> makeEnv is line 919: targets = new_env.Program(progname, main_objs +
> static_objs). It is also worth noting that it takes ~10s to reach the first
> makeEnv call. This 10 seconds seems to be distributed throughout the
> src/SConscript file.
>
> *cmd: time scons --default=X86 build/test/gem5.debug RUBY=True
> PROTOCOL=MI_example NO_HTML=True*
> * *commented 3 makenv calls at bottom of src/Sconscript (lines
> 960-974)*
> *
> question: Is it necessary to call makeEnv for all 4 binaries when a single
> compile only generates a single binary?*
>
> *4. decider/--implicit-cache**:
> *a. By default, scons seems to run a checksum on every file to determine
> if it has changed. Scons can be configured to only do a checksum if the
> time stamp has changed via the decider function.
> b. An --implicit-cache option can be passed through the command line to
> reduce the number of dependency checks done on header files (at least this
> is my understanding). I believe there is a way to make this the default
> behavior as well.
>
> I found that out of /tmp, the -implicit-cache option made a big difference
> by itself and the decider function didn't do much. On my mounted directory,
> neither of these optimizations did much alone, but together they made a big
> difference.
>
> *cmd: time scons --implicit-cache --default=X86 build/test/gem5.debug
> RUBY=True PROTOCOL=MI_example NO_HTML=True
> ** *commented 3 makenv calls at bottom of src/Sconscript (lines
> 960-974)
> ** *add line main.Decider('MD5-timestamp') to top-level SConstruct
> file (line 183)
> **question: Should the decider be configured to default to MD5-timestamp?
> Should the implicit-cache option be the default behavior?*
>
> *5. ???*
> *question: Can anything else be done to speed up the gem5 build?*
>
> Thanks,
> Marc
>
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
