[PATCH 3/4] perf-test: initial version of memory test infrastructure.
From: David BremnerThe idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories is probably overkill; I find it nice to have them sequenced by time. Arguably the mktemp is then overkill, but I know people will be nervous if it looks like timestamps are being used for uniqueness. One new test is included, to check notmuch new for memory leaks. --- performance-test/.gitignore |1 + performance-test/M00-new | 14 + performance-test/Makefile.local | 17 +-- performance-test/README | 57 + performance-test/perf-test-lib.sh | 55 --- 5 files changed, 113 insertions(+), 31 deletions(-) create mode 100755 performance-test/M00-new diff --git a/performance-test/.gitignore b/performance-test/.gitignore index 6421a9a..f3f9be4 100644 --- a/performance-test/.gitignore +++ b/performance-test/.gitignore @@ -1,3 +1,4 @@ tmp.*/ +log.*/ corpus/ notmuch.cache.*/ diff --git a/performance-test/M00-new b/performance-test/M00-new new file mode 100755 index 000..733e9b0 --- /dev/null +++ b/performance-test/M00-new @@ -0,0 +1,14 @@ +#!/bin/bash + +test_description='notmuch new' + +. ./perf-test-lib.sh + +# ensure initial 'notmuch new' is run by memory_start +uncache_database + +memory_start + +memory_run "notmuch new" "notmuch new" + +memory_done diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local index 57beb44..357d800 100644 --- a/performance-test/Makefile.local +++ b/performance-test/Makefile.local @@ -4,14 +4,25 @@ dir := performance-test include $(dir)/version.sh +# these two are just make sure dir is expanded at the right time. +TIME_TEST_SCRIPT := ${dir}/notmuch-time-test +MEMORY_TEST_SCRIPT := ${dir}/notmuch-memory-test + CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz TXZFILE := ${dir}/download/${CORPUS_NAME} SIGFILE := ${TXZFILE}.asc -TEST_SCRIPT := ${dir}/notmuch-perf-test DEFAULT_URL := http://notmuchmail.org/releases/${CORPUS_NAME} +perf-test: time-test memory-test + time-test: setup-perf-test all - $(TEST_SCRIPT) $(OPTIONS) + @echo + $(TIME_TEST_SCRIPT) $(TEST_OPTIONS) + +memory-test: setup-perf-test all + @echo + $(MEMORY_TEST_SCRIPT) $(TEST_OPTIONS) + .PHONY: download-corpus setup-perf-test @@ -29,4 +40,4 @@ $(TXZFILE): download-corpus: wget -O ${TXZFILE} ${DEFAULT_URL} -CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.* +CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/log.* $(dir)/corpus $(dir)/notmuch.cache.* diff --git a/performance-test/README b/performance-test/README index d1fb6de..7eaf5f7 100644 --- a/performance-test/README +++ b/performance-test/README @@ -1,3 +1,10 @@ +Performance Tests +- + +This directory contains two kinds of performance tests, time tests, +and memory tests. The former use gnu time, and the latter use +valgrind. + Pre-requisites -- @@ -5,9 +12,10 @@ In addition to having notmuch, you need: - gpg - gnu tar -- gnu time +- gnu time (for the time tests). - xz. Some speedup can be gotten by installing "pixz", but this is probably only worthwhile if you are debugging the tests. +- valgrind (for the memory tests) Getting set up to run tests: @@ -36,34 +44,47 @@ for a list of mirrors. Running tests - -The easiest way to run performance tests is to say "make time-test", (or -simply run the notmuch-time-test script). Either command will run all -available performance tests. - -Alternately, you can run a specific subset of tests by simply invoking -one of the executable scripts in this directory, (such as ./basic). -Each test script supports the following arguments +The easiest way to run performance tests is to say "make perf-test". +This will run all time and memory tests. Be aware that the memory +tests are quite time consuming when run on the full corpus, and that +depending on your interests it may be more sensible to run "make +time-test" or "make memory-test". You can also invoke one of the +scripts notmuch-time-test or notmuch-memory-test or run a more +specific subset of tests by simply invoking one of the executable +scripts in this directory, (such as ./T00-new). Each test script +supports the following arguments --small / --medium / --large Choose corpus size. --debugEnable debugging. In particular don't delete temporary directories. +When using the make targets, you can pass arguments to all test +scripts by defining the make variable TEST_OPTIONS. + Writing tests - -Have a look at "T01-dump-restore"
[PATCH 3/4] perf-test: initial version of memory test infrastructure.
From: David Bremner brem...@debian.org The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories is probably overkill; I find it nice to have them sequenced by time. Arguably the mktemp is then overkill, but I know people will be nervous if it looks like timestamps are being used for uniqueness. One new test is included, to check notmuch new for memory leaks. --- performance-test/.gitignore |1 + performance-test/M00-new | 14 + performance-test/Makefile.local | 17 +-- performance-test/README | 57 + performance-test/perf-test-lib.sh | 55 --- 5 files changed, 113 insertions(+), 31 deletions(-) create mode 100755 performance-test/M00-new diff --git a/performance-test/.gitignore b/performance-test/.gitignore index 6421a9a..f3f9be4 100644 --- a/performance-test/.gitignore +++ b/performance-test/.gitignore @@ -1,3 +1,4 @@ tmp.*/ +log.*/ corpus/ notmuch.cache.*/ diff --git a/performance-test/M00-new b/performance-test/M00-new new file mode 100755 index 000..733e9b0 --- /dev/null +++ b/performance-test/M00-new @@ -0,0 +1,14 @@ +#!/bin/bash + +test_description='notmuch new' + +. ./perf-test-lib.sh + +# ensure initial 'notmuch new' is run by memory_start +uncache_database + +memory_start + +memory_run notmuch new notmuch new + +memory_done diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local index 57beb44..357d800 100644 --- a/performance-test/Makefile.local +++ b/performance-test/Makefile.local @@ -4,14 +4,25 @@ dir := performance-test include $(dir)/version.sh +# these two are just make sure dir is expanded at the right time. +TIME_TEST_SCRIPT := ${dir}/notmuch-time-test +MEMORY_TEST_SCRIPT := ${dir}/notmuch-memory-test + CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz TXZFILE := ${dir}/download/${CORPUS_NAME} SIGFILE := ${TXZFILE}.asc -TEST_SCRIPT := ${dir}/notmuch-perf-test DEFAULT_URL := http://notmuchmail.org/releases/${CORPUS_NAME} +perf-test: time-test memory-test + time-test: setup-perf-test all - $(TEST_SCRIPT) $(OPTIONS) + @echo + $(TIME_TEST_SCRIPT) $(TEST_OPTIONS) + +memory-test: setup-perf-test all + @echo + $(MEMORY_TEST_SCRIPT) $(TEST_OPTIONS) + .PHONY: download-corpus setup-perf-test @@ -29,4 +40,4 @@ $(TXZFILE): download-corpus: wget -O ${TXZFILE} ${DEFAULT_URL} -CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.* +CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/log.* $(dir)/corpus $(dir)/notmuch.cache.* diff --git a/performance-test/README b/performance-test/README index d1fb6de..7eaf5f7 100644 --- a/performance-test/README +++ b/performance-test/README @@ -1,3 +1,10 @@ +Performance Tests +- + +This directory contains two kinds of performance tests, time tests, +and memory tests. The former use gnu time, and the latter use +valgrind. + Pre-requisites -- @@ -5,9 +12,10 @@ In addition to having notmuch, you need: - gpg - gnu tar -- gnu time +- gnu time (for the time tests). - xz. Some speedup can be gotten by installing pixz, but this is probably only worthwhile if you are debugging the tests. +- valgrind (for the memory tests) Getting set up to run tests: @@ -36,34 +44,47 @@ for a list of mirrors. Running tests - -The easiest way to run performance tests is to say make time-test, (or -simply run the notmuch-time-test script). Either command will run all -available performance tests. - -Alternately, you can run a specific subset of tests by simply invoking -one of the executable scripts in this directory, (such as ./basic). -Each test script supports the following arguments +The easiest way to run performance tests is to say make perf-test. +This will run all time and memory tests. Be aware that the memory +tests are quite time consuming when run on the full corpus, and that +depending on your interests it may be more sensible to run make +time-test or make memory-test. You can also invoke one of the +scripts notmuch-time-test or notmuch-memory-test or run a more +specific subset of tests by simply invoking one of the executable +scripts in this directory, (such as ./T00-new). Each test script +supports the following arguments --small / --medium / --large Choose corpus size. --debugEnable debugging. In particular don't delete temporary directories. +When using the make targets, you can pass arguments to all test +scripts by defining the make variable TEST_OPTIONS. + Writing tests - -Have a look at T01-dump-restore for an
Re: [PATCH 3/4] perf-test: initial version of memory test infrastructure.
Quoth da...@tethera.net on Dec 16 at 2:23 pm: From: David Bremner brem...@debian.org The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories is probably overkill; I find it nice to have them sequenced by time. Arguably the mktemp is then overkill, but I know people will be nervous if it looks like timestamps are being used for uniqueness. One new test is included, to check notmuch new for memory leaks. --- performance-test/.gitignore |1 + performance-test/M00-new | 14 + performance-test/Makefile.local | 17 +-- performance-test/README | 57 + performance-test/perf-test-lib.sh | 55 --- 5 files changed, 113 insertions(+), 31 deletions(-) create mode 100755 performance-test/M00-new diff --git a/performance-test/.gitignore b/performance-test/.gitignore index 6421a9a..f3f9be4 100644 --- a/performance-test/.gitignore +++ b/performance-test/.gitignore @@ -1,3 +1,4 @@ tmp.*/ +log.*/ corpus/ notmuch.cache.*/ diff --git a/performance-test/M00-new b/performance-test/M00-new new file mode 100755 index 000..733e9b0 --- /dev/null +++ b/performance-test/M00-new @@ -0,0 +1,14 @@ +#!/bin/bash + +test_description='notmuch new' + +. ./perf-test-lib.sh + +# ensure initial 'notmuch new' is run by memory_start +uncache_database + +memory_start + +memory_run notmuch new notmuch new This will run notmuch new twice. Is that intentional? If so, it might be worth a comment. + +memory_done diff --git a/performance-test/Makefile.local b/performance-test/Makefile.local index 57beb44..357d800 100644 --- a/performance-test/Makefile.local +++ b/performance-test/Makefile.local @@ -4,14 +4,25 @@ dir := performance-test include $(dir)/version.sh +# these two are just make sure dir is expanded at the right time. +TIME_TEST_SCRIPT := ${dir}/notmuch-time-test +MEMORY_TEST_SCRIPT := ${dir}/notmuch-memory-test This is obviously fine, but I don't understand the comment. There are all sorts of places where we have to := assign to get ${dir} right (including TXZFILE file below). Is there something different here? + CORPUS_NAME := notmuch-email-corpus-$(PERFTEST_VERSION).tar.xz TXZFILE := ${dir}/download/${CORPUS_NAME} SIGFILE := ${TXZFILE}.asc -TEST_SCRIPT := ${dir}/notmuch-perf-test DEFAULT_URL := http://notmuchmail.org/releases/${CORPUS_NAME} +perf-test: time-test memory-test + time-test: setup-perf-test all - $(TEST_SCRIPT) $(OPTIONS) + @echo + $(TIME_TEST_SCRIPT) $(TEST_OPTIONS) Why not use the same OPTIONS variable name as the correctness tests? + +memory-test: setup-perf-test all + @echo + $(MEMORY_TEST_SCRIPT) $(TEST_OPTIONS) + .PHONY: download-corpus setup-perf-test @@ -29,4 +40,4 @@ $(TXZFILE): download-corpus: wget -O ${TXZFILE} ${DEFAULT_URL} -CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/corpus $(dir)/notmuch.cache.* +CLEAN := $(CLEAN) $(dir)/tmp.* $(dir)/log.* $(dir)/corpus $(dir)/notmuch.cache.* diff --git a/performance-test/README b/performance-test/README index d1fb6de..7eaf5f7 100644 --- a/performance-test/README +++ b/performance-test/README @@ -1,3 +1,10 @@ +Performance Tests +- + +This directory contains two kinds of performance tests, time tests, s/performance tests,/performance tests:/ +and memory tests. The former use gnu time, and the latter use +valgrind. + Pre-requisites -- @@ -5,9 +12,10 @@ In addition to having notmuch, you need: - gpg - gnu tar -- gnu time +- gnu time (for the time tests). No period? - xz. Some speedup can be gotten by installing pixz, but this is probably only worthwhile if you are debugging the tests. +- valgrind (for the memory tests) Getting set up to run tests: @@ -36,34 +44,47 @@ for a list of mirrors. Running tests - -The easiest way to run performance tests is to say make time-test, (or -simply run the notmuch-time-test script). Either command will run all -available performance tests. - -Alternately, you can run a specific subset of tests by simply invoking -one of the executable scripts in this directory, (such as ./basic). -Each test script supports the following arguments +The easiest way to run performance tests is to say make perf-test. +This will run all time and memory tests. Be aware that the memory +tests are quite time consuming when run on the full corpus, and that +depending on your interests it may be more sensible to run make +time-test or make memory-test. You can also invoke one of the +scripts