[Patch v2 3/4] perf-test: initial version of memory test infrastructure.

2012-12-24 Thread da...@tethera.net 
From: David Bremner 

The idea is run some code under valgrind --leak-check=full and report
a summary, leaving the user to peruse the log file if they want.

We go to some lengths to preserve the log files from accidental
overwriting; the full corpus takes about 3 hours to run under valgrind
on my machine.

The naming of the log directories may be slightly controversial; in
the unlikely event of two runs in less than a second, the log will be
overwritten. A previous version with mktemp+timestamp was dismissed as
overkill; just mktemp alone does not sort nicely.

One new test is included, to check notmuch new for memory leaks.
---
 performance-test/.gitignore   |1 +
 performance-test/M00-new  |   15 
 performance-test/Makefile.local   |   16 ++--
 performance-test/README   |   57 +++-
 performance-test/perf-test-lib.sh |   75 -
 5 files changed, 126 insertions(+), 38 deletions(-)
 create mode 100755 performance-test/M00-new

diff --git a/performance-test/.gitignore b/performance-test/.gitignore
index 6421a9a..f3f9be4 100644
--- a/performance-test/.gitignore
+++ b/performance-test/.gitignore
@@ -1,3 +1,4 @@
 tmp.*/
+log.*/
 corpus/
 notmuch.cache.*/
diff --git a/performance-test/M00-new b/performance-test/M00-new
new file mode 100755
index 000..99c3f52
--- /dev/null
+++ b/performance-test/M00-new
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+# ensure initial 'notmuch new' is run by memory_start
+uncache_database
+
+memory_start
+
+# run 'notmuch new' a second time, to test different code paths
+memory_run "notmuch new" "notmuch new"
+
+memory_done
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 57beb44..73aa963 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -4,14 +4,24 @@ dir := performance-test

 include $(dir)/version.sh

+TIME_TEST_SCRIPT := ${dir}/notmuch-time-test
+MEMORY_TEST_SCRIPT := ${dir}/notmuch-memory-test
+
 CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz
 TXZFILE := ${dir}/download/${CORPUS_NAME}
 SIGFILE := ${TXZFILE}.asc
-TEST_SCRIPT := ${dir}/notmuch-perf-test
 DEFAULT_URL :=  http://notmuchmail.org/releases/${CORPUS_NAME}

+perf-test: time-test memory-test
+
 time-test: setup-perf-test all
-   $(TEST_SCRIPT) $(OPTIONS)
+   @echo
+   $(TIME_TEST_SCRIPT) $(OPTIONS)
+
+memory-test: setup-perf-test all
+   @echo
+   $(MEMORY_TEST_SCRIPT) $(OPTIONS)
+

 .PHONY: download-corpus setup-perf-test

@@ -29,4 +39,4 @@ $(TXZFILE):
 download-corpus:
wget -O ${TXZFILE} ${DEFAULT_URL}

-CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.*
+CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/log.* $(dir)/corpus 
$(dir)/notmuch.cache.*
diff --git a/performance-test/README b/performance-test/README
index d1fb6de..996724c 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -1,3 +1,10 @@
+Performance Tests
+-
+
+This directory contains two kinds of performance tests: time tests,
+and memory tests. The former use gnu time, and the latter use
+valgrind.
+
 Pre-requisites
 --

@@ -5,9 +12,10 @@ In addition to having notmuch, you need:

 - gpg
 - gnu tar
-- gnu time
+- gnu time (for the time tests)
 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.
+- valgrind (for the memory tests)

 Getting set up to run tests:
 
@@ -36,34 +44,47 @@ for a list of mirrors.
 Running tests
 -

-The easiest way to run performance tests is to say "make time-test", (or
-simply run the notmuch-time-test script). Either command will run all
-available performance tests.
-
-Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./basic).
-Each test script supports the following arguments
+The easiest way to run performance tests is to say "make perf-test".
+This will run all time and memory tests.  Be aware that the memory
+tests are quite time consuming when run on the full corpus, and that
+depending on your interests it may be more sensible to run "make
+time-test" or "make memory-test".  You can also invoke one of the
+scripts notmuch-time-test or notmuch-memory-test or run a more
+specific subset of tests by simply invoking one of the executable
+scripts in this directory, (such as ./T00-new).  Each test script
+supports the following arguments

 --small / --medium / --large   Choose corpus size.
 --debugEnable debugging. In particular don't 
delete
temporary directories.

+When using the make targets, you can pass arguments to all test
+scripts by defining the make variable OPTIONS.
+
 Writing tests
 -

-Have a look at "T01-dump-restore" for an

[Patch v2 3/4] perf-test: initial version of memory test infrastructure.

2012-12-24 Thread david 
From: David Bremner brem...@debian.org

The idea is run some code under valgrind --leak-check=full and report
a summary, leaving the user to peruse the log file if they want.

We go to some lengths to preserve the log files from accidental
overwriting; the full corpus takes about 3 hours to run under valgrind
on my machine.

The naming of the log directories may be slightly controversial; in
the unlikely event of two runs in less than a second, the log will be
overwritten. A previous version with mktemp+timestamp was dismissed as
overkill; just mktemp alone does not sort nicely.

One new test is included, to check notmuch new for memory leaks.
---
 performance-test/.gitignore   |1 +
 performance-test/M00-new  |   15 
 performance-test/Makefile.local   |   16 ++--
 performance-test/README   |   57 +++-
 performance-test/perf-test-lib.sh |   75 -
 5 files changed, 126 insertions(+), 38 deletions(-)
 create mode 100755 performance-test/M00-new

diff --git a/performance-test/.gitignore b/performance-test/.gitignore
index 6421a9a..f3f9be4 100644
--- a/performance-test/.gitignore
+++ b/performance-test/.gitignore
@@ -1,3 +1,4 @@
 tmp.*/
+log.*/
 corpus/
 notmuch.cache.*/
diff --git a/performance-test/M00-new b/performance-test/M00-new
new file mode 100755
index 000..99c3f52
--- /dev/null
+++ b/performance-test/M00-new
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='notmuch new'
+
+. ./perf-test-lib.sh
+
+# ensure initial 'notmuch new' is run by memory_start
+uncache_database
+
+memory_start
+
+# run 'notmuch new' a second time, to test different code paths
+memory_run notmuch new notmuch new
+
+memory_done
diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local
index 57beb44..73aa963 100644
--- a/performance-test/Makefile.local
+++ b/performance-test/Makefile.local
@@ -4,14 +4,24 @@ dir := performance-test
 
 include $(dir)/version.sh
 
+TIME_TEST_SCRIPT := ${dir}/notmuch-time-test
+MEMORY_TEST_SCRIPT := ${dir}/notmuch-memory-test
+
 CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz
 TXZFILE := ${dir}/download/${CORPUS_NAME}
 SIGFILE := ${TXZFILE}.asc
-TEST_SCRIPT := ${dir}/notmuch-perf-test
 DEFAULT_URL :=  http://notmuchmail.org/releases/${CORPUS_NAME}
 
+perf-test: time-test memory-test
+
 time-test: setup-perf-test all
-   $(TEST_SCRIPT) $(OPTIONS)
+   @echo
+   $(TIME_TEST_SCRIPT) $(OPTIONS)
+
+memory-test: setup-perf-test all
+   @echo
+   $(MEMORY_TEST_SCRIPT) $(OPTIONS)
+
 
 .PHONY: download-corpus setup-perf-test
 
@@ -29,4 +39,4 @@ $(TXZFILE):
 download-corpus:
wget -O ${TXZFILE} ${DEFAULT_URL}
 
-CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.*
+CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/log.* $(dir)/corpus 
$(dir)/notmuch.cache.*
diff --git a/performance-test/README b/performance-test/README
index d1fb6de..996724c 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -1,3 +1,10 @@
+Performance Tests
+-
+
+This directory contains two kinds of performance tests: time tests,
+and memory tests. The former use gnu time, and the latter use
+valgrind.
+
 Pre-requisites
 --
 
@@ -5,9 +12,10 @@ In addition to having notmuch, you need:
 
 - gpg
 - gnu tar
-- gnu time
+- gnu time (for the time tests)
 - xz. Some speedup can be gotten by installing pixz, but this is
   probably only worthwhile if you are debugging the tests.
+- valgrind (for the memory tests)
 
 Getting set up to run tests:
 
@@ -36,34 +44,47 @@ for a list of mirrors.
 Running tests
 -
 
-The easiest way to run performance tests is to say make time-test, (or
-simply run the notmuch-time-test script). Either command will run all
-available performance tests.
-
-Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./basic).
-Each test script supports the following arguments
+The easiest way to run performance tests is to say make perf-test.
+This will run all time and memory tests.  Be aware that the memory
+tests are quite time consuming when run on the full corpus, and that
+depending on your interests it may be more sensible to run make
+time-test or make memory-test.  You can also invoke one of the
+scripts notmuch-time-test or notmuch-memory-test or run a more
+specific subset of tests by simply invoking one of the executable
+scripts in this directory, (such as ./T00-new).  Each test script
+supports the following arguments
 
 --small / --medium / --large   Choose corpus size.
 --debugEnable debugging. In particular don't 
delete
temporary directories.
 
+When using the make targets, you can pass arguments to all test
+scripts by defining the make variable OPTIONS.
+
 Writing tests
 -
 
-Have a look at T01-dump-restore for an