[notmuch] strange behavior of indexing of and searching for strings containing '[]'

2010-02-08 Thread Jed Brown
On Mon, 08 Feb 2010 12:24:06 -0500, Jameson Rollins  wrote:
> I don't think that this is exactly correct.  The quoting is interpreted
> by the shell in order to construct a single string that is then passed
> as an argument to the program.

The command line distinguishes, but the constructed query does not.
Look at query-string.c, the arguments are just concatenated.

Jed


[notmuch] [PATCH 3/3] Convert the actual tests to the new framework

2010-02-08 Thread Michal Sojka
The changes are:
- The notmuch-test was split into several files (t000?-*.sh).
- Removed helper functions which were moved to test-lib.sh
- Replaced every printf with test_expect_success.
- Replaced $NOTMUCH with notmuch (test-lib.sh sets $PATH appropriately)
- Test commands chained with && (test-lib.sh doesn't use "set -e" in
  order to complete the test suite even if something fails)
- Many variables such as ${MAIL_DIR} were properly quoted as they
  contain spaces.
- Changed quoting patterns in add_message and generate_message (single
  quotes are already used by the test framework).
- ${TEST_DIR} replaced by ./

QUICK HOWTO:

To run the whole test suite
make

To run only a single test
   ./t0001-new.sh

To stop on the first error
   ./t0001-new.sh -i
then mail store and database can be inspected in
"trash directory.t0001-new"

To see the output of tests
   ./t0001-new.sh -v

To not remove trash directory at the end:
   ./t0001-new.sh -d

To run all tests verbosely:
   make make GIT_TEST_OPTS="-v"

TODO: Convert GIT_* variables in test-lib.sh to NOTMUCH_*

Signed-off-by: Michal Sojka 
---
 test/notmuch-test|  542 --
 test/t-basic.sh  |2 +-
 test/t0001-new.sh|  135 +++
 test/t0002-search.sh |   62 +
 test/t0003-reply.sh  |  114 +
 test/t0004-uuencoded-data.sh |   35 +++
 test/t0005-dump-restore.sh   |   44 
 7 files changed, 391 insertions(+), 543 deletions(-)
 delete mode 100755 test/notmuch-test
 create mode 100755 test/t0001-new.sh
 create mode 100755 test/t0002-search.sh
 create mode 100755 test/t0003-reply.sh
 create mode 100755 test/t0004-uuencoded-data.sh
 create mode 100755 test/t0005-dump-restore.sh

diff --git a/test/notmuch-test b/test/notmuch-test
deleted file mode 100755
index 2e5eb24..000
--- a/test/notmuch-test
+++ /dev/null
@@ -1,542 +0,0 @@
-#!/bin/bash
-set -e
-
-find_notmuch_binary ()
-{
-dir=$1
-
-while [ -n "$dir" ]; do
-   bin=$dir/notmuch
-   if [ -x $bin ]; then
-   echo $bin
-   return
-   fi
-   dir=$(dirname $dir)
-   if [ "$dir" = "/" ]; then
-   break
-   fi
-done
-
-echo notmuch
-}
-
-increment_mtime_amount=0
-increment_mtime ()
-{
-dir=$1
-
-increment_mtime_amount=$((increment_mtime_amount + 1))
-touch -d "+${increment_mtime_amount} seconds" $dir
-}
-
-# Generate a new message in the mail directory, with a unique message
-# ID and subject. The message is not added to the index.
-#
-# After this function returns, the filename of the generated message
-# is available as $gen_msg_filename and the message ID is available as
-# $gen_msg_id .
-#
-# This function supports named parameters with the bash syntax for
-# assigning a value to an associative array ([name]=value). The
-# supported parameters are:
-#
-#  [dir]=directory/of/choice
-#
-#  Generate the message in directory 'directory/of/choice' within
-#  the mail store. The directory will be created if necessary.
-#
-#  [body]=text
-#
-#  Text to use as the body of the email message
-#
-#  '[from]="Some User "'
-#  '[to]="Some User "'
-#  '[subject]="Subject of email message"'
-#  '[date]="RFC 822 Date"'
-#
-#  Values for email headers. If not provided, default values will
-#  be generated instead.
-#
-#  '[cc]="Some User "'
-#  [reply-to]=some-address
-#  [in-reply-to]=
-#
-#  Additional values for email headers. If these are not provided
-#  then the relevant headers will simply not appear in the
-#  message.
-gen_msg_cnt=0
-gen_msg_filename=""
-gen_msg_id=""
-generate_message ()
-{
-# This is our (bash-specific) magic for doing named parameters
-local -A template="($@)"
-local additional_headers
-
-gen_msg_cnt=$((gen_msg_cnt + 1))
-gen_msg_name=msg-$(printf "%03d" $gen_msg_cnt)
-gen_msg_id="${gen_msg_name}@notmuch-test-suite"
-
-if [ -z "${template[dir]}" ]; then
-   gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
-else
-   gen_msg_filename="${MAIL_DIR}/${template[dir]}/$gen_msg_name"
-   mkdir -p $(dirname $gen_msg_filename)
-fi
-
-if [ -z "${template[body]}" ]; then
-   template[body]="This is just a test message at ${gen_msg_filename}"
-fi
-
-if [ -z "${template[from]}" ]; then
-   template[from]="Notmuch Test Suite "
-fi
-
-if [ -z "${template[to]}" ]; then
-   template[to]="Notmuch Test Suite "
-fi
-
-if [ -z "${template[subject]}" ]; then
-   template[subject]="Test message ${gen_msg_filename}"
-fi
-
-if [ -z "${template[date]}" ]; then
-   template[date]="Tue, 05 Jan 2010 15:43:57 -0800"
-fi
-
-additional_headers=""
-if [ ! -z "${template[reply-to]}" ]; then
-   additional_headers="Reply-To: ${template[reply-to]}
-${additional_headers}"
-fi
-
-if [ ! -z "${template[in-reply-to]}" ]; then
-   additional_headers="In-Reply-To: ${template[in-reply-to]}

[notmuch] [PATCH 2/3] Update test framework for use with notmuch

2010-02-08 Thread Michal Sojka
This removes Git specific things from the test-lib.sh and adds helper
functions for notmuch taken from Carl's notmuch-test script. README is
also slightly modified to reflect the current state.

Signed-off-by: Michal Sojka 
---
 test/Makefile   |1 +
 test/README |   12 +-
 test/t-basic.sh |  332 +++
 test/test-lib.sh|  320 +++--
 4 files changed, 204 insertions(+), 461 deletions(-)
 mode change 100644 => 100755 test/test-lib.sh

diff --git a/test/Makefile b/test/Makefile
index bd09390..e07c7fd 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -6,6 +6,7 @@
 -include ../config.mak

 #GIT_TEST_OPTS=--verbose --debug
+SHELL=/bin/bash
 SHELL_PATH ?= $(SHELL)
 TAR ?= $(TAR)
 RM ?= rm -f
diff --git a/test/README b/test/README
index dcd3ebb..0f5bddb 100644
--- a/test/README
+++ b/test/README
@@ -165,10 +165,10 @@ Writing Tests
 -

 The test script is written as a shell script.  It should start
-with the standard "#!/bin/sh" with copyright notices, and an
+with the standard "#!/bin/bash" with copyright notices, and an
 assignment to variable 'test_description', like this:

-   #!/bin/sh
+   #!/bin/bash
#
# Copyright (c) 2005 Junio C Hamano
#
@@ -192,9 +192,11 @@ This test harness library does the following things:
  - If the script is invoked with command line argument --help
(or -h), it shows the test_description and exits.

- - Creates an empty test directory with an empty .git/objects
-   database and chdir(2) into it.  This directory is 't/trash directory'
-   if you must know, but I do not think you care.
+ - Creates a test directory with default notmuch-config and empty mail
+   store. This directory is 't/trash directory.' (note
+   the space) if you must know, but I do not think you care. The path
+   to notmuch-config is exported in NOTMUCH_CONFIG environment
+   variable and mail store path is stored in MAIL_DIR variabl.

  - Defines standard test helper functions for your scripts to
use.  These functions are designed to make all scripts behave
diff --git a/test/t-basic.sh b/test/t-basic.sh
index f4ca4fc..cc2ca21 100755
--- a/test/t-basic.sh
+++ b/test/t-basic.sh
@@ -5,46 +5,39 @@

 test_description='Test the very basics part #1.

-The rest of the test suite does not check the basic operation of git
-plumbing commands to work very carefully.  Their job is to concentrate
-on tricky features that caused bugs in the past to detect regression.
-
-This test runs very basic features, like registering things in cache,
-writing tree, etc.
-
-Note that this test *deliberately* hard-codes many expected object
-IDs.  When object ID computation changes, like in the previous case of
-swapping compression and hashing order, the person who is making the
-modification *should* take notice and update the test vectors here.
+Tests the test framework itself.
 '
-
 
 # It appears that people try to run tests without building...

-../git >/dev/null
-if test $? != 1
+if ! test -x ../notmuch
 then
-   echo >&2 'You do not seem to have built git yet.'
+   echo >&2 'You do not seem to have built notmuch yet.'
exit 1
 fi

 . ./test-lib.sh

 
-# git init has been done in an empty repository.
-# make sure it is empty.
+# Test mail store prepared in test-lib.sh
+
+test_expect_success \
+'test that mail store was created' \
+'test -d "${MAIL_DIR}"'
+

-find .git/objects -type f -print >should-be-empty
+find "${MAIL_DIR}" -type f -print >should-be-empty
 test_expect_success \
-'.git/objects should be empty after git init in an empty repo.' \
+'mail store should be empty' \
 'cmp -s /dev/null should-be-empty'

-# also it should have 2 subdirectories; no fan-out anymore, pack, and info.
-# 3 is counting "objects" itself
-find .git/objects -type d -print >full-of-directories
 test_expect_success \
-'.git/objects should have 3 subdirectories.' \
-'test $(wc -l < full-of-directories) = 3'
+'NOTMUCH_CONFIG is set and points to an existing file' \
+'test -f "${NOTMUCH_CONFIG}"'
+
+test_expect_success \
+'PATH is set to this repository' \
+'test "`echo $PATH|cut -f1 -d:`" = "`dirname ${TEST_DIRECTORY}`"'

 
 # Test harness
@@ -73,296 +66,5 @@ then
exit 1
 fi

-
-# Basics of the basics
-
-# updating a new file without --add should fail.
-test_expect_success 'git update-index without --add should fail adding.' '
-test_must_fail git update-index should-be-empty
-'
-
-# and with --add it should succeed, even if it is empty (it used to fail).
-test_expect_success \
-'git update-index with --add should succeed.' \
-'git update-index --add 

[notmuch] [PATCH 1/3] Copy test framework from Git

2010-02-08 Thread Michal Sojka
Git uses a simple and yet powerfull test framework, written in shell.
The framework is easy to use for both users and developers so I thing
it would help if it is used in notmuch as well.

This is a copy of Git's test framework from commit
b8bba419250711a69e09e7648e5c991f4847a127.

Signed-off-by: Michal Sojka 
---
 test/Makefile |   46 +++
 test/README   |  297 +
 test/aggregate-results.sh |   34 ++
 test/t-basic.sh   |  368 +
 test/test-lib.sh  |  787 +
 5 files changed, 1532 insertions(+), 0 deletions(-)
 create mode 100644 test/Makefile
 create mode 100644 test/README
 create mode 100755 test/aggregate-results.sh
 create mode 100755 test/t-basic.sh
 create mode 100644 test/test-lib.sh

diff --git a/test/Makefile b/test/Makefile
new file mode 100644
index 000..bd09390
--- /dev/null
+++ b/test/Makefile
@@ -0,0 +1,46 @@
+# Run tests
+#
+# Copyright (c) 2005 Junio C Hamano
+#
+
+-include ../config.mak
+
+#GIT_TEST_OPTS=--verbose --debug
+SHELL_PATH ?= $(SHELL)
+TAR ?= $(TAR)
+RM ?= rm -f
+
+# Shell quote;
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
+
+T = $(wildcard t[0-9][0-9][0-9][0-9]-*.sh)
+TSVN = $(wildcard t91[0-9][0-9]-*.sh)
+
+all: pre-clean
+   $(MAKE) aggregate-results-and-cleanup
+
+$(T):
+   @echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ 
$(GIT_TEST_OPTS)
+
+pre-clean:
+   $(RM) -r test-results
+
+clean:
+   $(RM) -r 'trash directory'.* test-results
+
+aggregate-results-and-cleanup: $(T)
+   $(MAKE) aggregate-results
+   $(MAKE) clean
+
+aggregate-results:
+   '$(SHELL_PATH_SQ)' ./aggregate-results.sh test-results/t*-*
+
+# we can test NO_OPTIMIZE_COMMITS independently of LC_ALL
+full-svn-test:
+   $(MAKE) $(TSVN) GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C
+   $(MAKE) $(TSVN) GIT_SVN_NO_OPTIMIZE_COMMITS=0 LC_ALL=en_US.UTF-8
+
+valgrind:
+   GIT_TEST_OPTS=--valgrind $(MAKE)
+
+.PHONY: pre-clean $(T) aggregate-results clean valgrind
diff --git a/test/README b/test/README
new file mode 100644
index 000..dcd3ebb
--- /dev/null
+++ b/test/README
@@ -0,0 +1,297 @@
+Core GIT Tests
+==
+
+This directory holds many test scripts for core GIT tools.  The
+first part of this short document describes how to run the tests
+and read their output.
+
+When fixing the tools or adding enhancements, you are strongly
+encouraged to add tests in this directory to cover what you are
+trying to fix or enhance.  The later part of this short document
+describes how your test scripts should be organized.
+
+
+Running Tests
+-
+
+The easiest way to run tests is to say "make".  This runs all
+the tests.
+
+*** t-basic.sh ***
+*   ok 1: .git/objects should be empty after git-init in an empty repo.
+*   ok 2: .git/objects should have 256 subdirectories.
+*   ok 3: git-update-index without --add should fail adding.
+...
+*   ok 23: no diff after checkout and git-update-index --refresh.
+* passed all 23 test(s)
+*** t0100-environment-names.sh ***
+*   ok 1: using old names should issue warnings.
+*   ok 2: using old names but having new names should not issue warnings.
+...
+
+Or you can run each test individually from command line, like
+this:
+
+$ sh ./t3001-ls-files-killed.sh
+*   ok 1: git-update-index --add to add various paths.
+*   ok 2: git-ls-files -k to show killed files.
+*   ok 3: validate git-ls-files -k output.
+* passed all 3 test(s)
+
+You can pass --verbose (or -v), --debug (or -d), and --immediate
+(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
+appropriately before running "make".
+
+--verbose::
+   This makes the test more verbose.  Specifically, the
+   command being run and their output if any are also
+   output.
+
+--debug::
+   This may help the person who is developing a new test.
+   It causes the command defined with test_debug to run.
+
+--immediate::
+   This causes the test to immediately exit upon the first
+   failed test.
+
+--long-tests::
+   This causes additional long-running tests to be run (where
+   available), for more exhaustive testing.
+
+--valgrind::
+   Execute all Git binaries with valgrind and exit with status
+   126 on errors (just like regular tests, this will only stop
+   the test script when running under -i).  Valgrind errors
+   go to stderr, so you might want to pass the -v option, too.
+
+   Since it makes no sense to run the tests with --valgrind and
+   not see any output, this option implies --verbose.  For
+   convenience, it also implies --tee.
+
+--tee::
+   In addition to printing the test output to the terminal,
+   write it to files named 't/test-results/$TEST_NAME.out'.
+   As the names depend on the tests' file names, it is safe to
+   run the tests with this option in parallel.
+

[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-08 Thread Michal Sojka
On Thursday 04 of February 2010 21:50:18 Carl Worth wrote:
> The test suite is still extremely rudimentary. Here are some things I'd
> like to improve about it:

I converted the actual version of notmuch-test to git test framework.
The result is in the followup patches.

I'd like to know opinion of others. If Carl agrees that it could be
merged I can do additional work as I describe below.

The conversion was not as straightforward as I expected mainly because
of problems with quoting. There are several sources of quotation problems
one being Carl's hashed array parameters. I thing it would be
sufficient to use plain variables for passing additional parameters.
Instead of:
add_message [from]="\"Sender \"" \
[to]=test_suite at notmuchmail.org \
[subject]=notmuch-reply-test \
[date]="\"Tue, 05 Jan 2010 15:43:56 -0800\"" \
[body]="\"basic reply test\"" &&
I'd do:
(
msg_from="Sender "
msg_to=test_suite at notmuchmail.org 
msg_subject=notmuch-reply-test
msg_date="Tue, 05 Jan 2010 15:43:56 -0800"
msg_body="basic reply test"
add_message
)

A possible additional improvement is elimination of
execute_expecting(). Combination of action (running notmuch) and
testing of a result in a single function makes it hard to distinguish
whether the problem is in the action or in the output. For example, if
notmuch is killed because of segfault, it would look like that no
output was produced. So instead of:

execute_expecting new "No new mail. Removed 3 messages."

I'd write something like:

echo "No new mail. Removed 3 messages." > expected

  test_expect_success 'Run notmuch'
'notmuch_filter_output new > actual'
  test_expect_success 'Compare actual and correct output'
'test_cmp expected actual'

where test_cmp is basically diff -u. This has also the advantage that
longer output (e.g. as in t0003-reply.sh) is more readable if there is
a difference.

On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote:
> Carl, have you considered outputting the test suite in the same format
> as the test anything protocol? [1] I only mention this because it
> might be a nice way to easily do some reporting (or perhaps even
> continuous integration) notmuch, with trivial effort.

According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project 
Git test output does not conform to TAP, but from my quick look it could be 
easily changed to conform.


On Friday 05 of February 2010 00:29:27 Carl Worth wrote:
> Looking at TAP, one thing I don't like is that it prints the
> success/failure of the test first, before the description of the
> test. That's not so nice in the case of a long-running (perhaps
> infinitely running) test where you might need to interrupt it, but you'd
> still want to know *what* was running for so long.

Git test framework also outputs first the result and then the test. It can be 
easily changed by -v switch.

To conclude, the output of running make in test/ directory now looks like 
this:
*** t-basic.sh ***
*   ok 1: test that mail store was created
*   ok 2: mail store should be empty
*   ok 3: NOTMUCH_CONFIG is set and points to an existing file
*   ok 4: PATH is set to this repository
*   ok 5: success is reported like this
*   still broken 6: pretend we have a known breakage
*   FIXED 7: pretend we have fixed a known breakage
*   ok 8: test runs if prerequisite is satisfied
* skip 9: unmet prerequisite causes test to be skipped
* fixed 1 known breakage(s)
* still have 1 known breakage(s)
* passed all remaining 8 test(s)
*** t0001-new.sh ***
*   ok 1: No new messages
*   ok 2: Single new message
*   ok 3: Multiple new messages
*   ok 4: No new messages (non-empty DB)
*   ok 5: New directories
*   ok 6: Alternate inode order
*   ok 7: Message moved in
*   ok 8: Renamed message
*   ok 9: Deleted message
*   ok 10: Renamed directory
*   ok 11: Deleted directory
*   ok 12: New directory (at end of list)
*   ok 13: Deleted directory (end of list)
*   ok 14: New symlink to directory
*   ok 15: New symlink to a file
*   ok 16: New two-level directory
*   ok 17: Deleted two-level directory
* passed all 17 test(s)
*** t0002-search.sh ***
*   ok 1: Search body
*   ok 2: Search by from:
*   ok 3: Search by to:
*   ok 4: Search by subject:
*   ok 5: Search by id:
*   ok 6: Search by tag:
*   ok 7: Search by thread:
*   ok 8: Search body (phrase)
*   ok 9: Search by from: (address)
*   ok 10: Search by from: (name)
*   ok 11: Search by to: (address)
*   ok 12: Search by to: (name)
*   ok 13: Search by subject: (phrase)
* passed all 13 test(s)
*** t0003-reply.sh ***
*   ok 1: Basic reply
*   ok 2: Multiple recipients
*   ok 3: Reply with CC
*   ok 4: Reply from alternate address
*   ok 5: Support for Reply-To
*   ok 6: Un-munging Reply-To
* passed all 6 test(s)
*** t0004-uuencoded-data.sh ***
*   ok 1: Generate message
*   ok 2: Ensure content before uu data is indexed
*   ok 3: Ensure 

[notmuch] notmuch new: Memory problem (with uuencoded content)

2010-02-08 Thread Michal Sojka
On Saturday 06 of February 2010 22:45:32 Carl Worth wrote:
> On Sat, 6 Feb 2010 11:40:18 +0100, Michal Sojka  
> wrote:
> > It is straightforward to convert your current test script to Git's
> > framework. If you are interested I'll do it.
> 
> Yes, I'd be quite interested in seeing that. Thanks for your
> contributions, and sorry I missed (or haven't yet gotten to) the patch
> you sent earlier.

Hi Carl,

I did the conversion of the test script. I'll post it to thread 
id:87ljf8pvxx.fsf at yoom.home.cworth.org, where it is more appropriate.

Michal


[notmuch] notmuch.el and searching within citations

2010-02-08 Thread Carl Worth
On Sat, 9 Jan 2010 06:18:10 +1100, "Bart Bunting"  wrote:
> Hi all,

Hi Bart, welcome to notmuch!

> If however the word 'grandfather' appears in a citation then an incremental
> search through the buffer won't find it.
> 
> If the citation is expanded then things work as expected.
> 
> I don't think this is very logical behavior as it stands, having to expand
> each citation to be able to search in it appears to sort of defeat the
> search.

I agree 100%.

There's good support in emacs for doing what we want here. There are
variables to control what happens when a search terminates (or
temporarily matches) invisible text. And one of the possible behaviors
is for the invisible region to be made visible (and that perhaps
temporarily if the search doesn't terminate in the invisible region).

So we just need to play with things a bit to turn that behavior on.

I've made a note in the TODO file so that I won't (permanently) forget
about this feature request. Of course, I'd be quite glad if anyone had
the time and inclination to investigate.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100208/94bb5b2c/attachment.pgp>


[notmuch] [PATCH] Goto-Address mode hides the body toggle button on messages of zero depth.

2010-02-08 Thread Carl Worth
On Wed, 30 Dec 2009 13:47:45 +0100, Pablo Oliveira  wrote:
> Hello all,

Ol? Pablo! Bem-vindo a notmuch!

> I heard about notmuch some weeks ago and I'm already liking it
> a lot. At first I was a little disapointed by the slow tagging, 
> but that was fixed by Kan-Ru's great patch for Xapian. I have
> started using it daily, so many thanks for this mail reader !

I'm delighted to hear you're having a good experience with notmuch. And
I apologize that I wasn't able to reply any sooner. (Perhaps I can
excuse myself somewhat since the original message arrived on my wedding
anniversary? No I'm still *really* late.)

> One of the things I find counter-intuitive is that on closed messages
> of zero depth, the toggle body button is hidden by the email address
> button, so when working with the keyboard, it seems one cannot open
> the message, without moving further on the header button.

Yes, this is really annoying. I'd like to see a proper fix for this.

> I propose above fix because it is very simple, but I do not like that
> we lose one level of indentation on all the threads (screen space
> is precious). Another solution would be to change the name-email 
> order in the header... Or maybe we should deactivate somehow, the
> Goto-Address mode on headers ? What do you think ?

For me, I'm getting the sender's name before their email address, so I
only run into this problem in the case where there is no name displayed,
but only an email address. Is that the same for you?

I don't want to remove the goto-address-mode on the header, because it
is very convenient (for those people who use a mouse) to be able to just
click on an email address to begin composing a new message to that
address.

I think I might be willing to have the email address deactivated for
keypresses if that were possible (so that pressing Enter anywhere on the
row would reliably toggle the message).

In the meantime, I agree that incrementing the depth universally doesn't
make sense. How about simply displaying a single ' ' character when
there's no full name available? I think that might solve the problem for
most people.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100208/39ab6e5f/attachment.pgp>


[notmuch] [PATCH 3/3] Allow folders with no messages to be elided from list.

2010-02-08 Thread Carl Worth
On Sat, 26 Dec 2009 16:34:18 -0800, Keith Packard  wrote:
> This makes it easier to see folders with messages.
> Eliding empty folders is togged with the 'e' binding.
> 
> Signed-off-by: Keith Packard 

Thanks for the patches, Keith!

These are all very useful, and I've pushed all three...

>  lib/Makefile.local |1 +
>  lib/notmuch.h  |   11 +++
>  lib/query.cc   |   41 +
>  notmuch-new.c  |1 +
>  notmuch.el |   29 -
>  5 files changed, 78 insertions(+), 5 deletions(-)

...except that on this last one I only pushed changes to notmuch.el. ;-)

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100208/97c94017/attachment.pgp>


[notmuch] [PATCH 4/4] emacs: Use font-lock-comment-face to highlight citation button

2010-02-08 Thread Carl Worth
On Mon, 14 Dec 2009 13:41:35 +0800, Kan-Ru Chen  wrote:

[... no detailed commit message ...]

I've pushed this change now. It looked to me like the previous changes
in this series were made obsolete by the patches from David that I just
pushed. Let me know if I'm wrong about that and there's anything more I
should pick up from this series.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100208/d8357e03/attachment.pgp>


[notmuch] Update Patch Series. Only collapse part of citations.

2010-02-08 Thread Carl Worth
> On Fri, 25 Dec 2009 16:09:06 -0400, david at tethera.net wrote:
> > On Fri, 25 Dec 2009 09:47:02 +0800, Kan-Ru Chen  wrote:
> > > I like this idea, but this patch hides all citations larger than the
> > > threshold. I'd like to see limited lines of citations been displayed.
> > 
> > Since you ask nicely, OK :).

Very nice work, David!

I've just used this for a few minutes here, and it's very nice. (The
previous code was hiding far too much of the citations and I found
myself having to open them all the time just to be able to read things.)

On Sun, 27 Dec 2009 00:22:21 +0800, Kan-Ru Chen  wrote:
> Works great here.
>
> Reviewed-by: Kan-Ru Chen 

And thanks for the review. I've now committed and pushed these changes,
(with the reviewed-by added and some trailing whitespace removed).

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100208/ba86d6f2/attachment.pgp>


[notmuch] [PATCH] notmuch-test: add tests for counting within absolute time ranges and 'today'

2010-02-08 Thread Sebastian Spaeth
Signed-off-by: Sebastian Spaeth 
---
 test/notmuch-test |   13 +
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/test/notmuch-test b/test/notmuch-test
index 2e5eb24..adc9f24 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -534,6 +534,19 @@ printf " Restore with nothing to do...\t"
 $NOTMUCH restore dump.expected
 echo " PASS"

+# Test daterange with absolute month---
+printf "\nTesting \"notmuch count\" in several variations:\n"
+printf " absolute date range..."
+#Find all messages from January 2010
+execute_expecting "count date:2010-01..01" "8"
+
+# Test daterange with today keyword-
+printf " 'today' date range..."
+TODAY=`date -R`
+add_message '[date]="${TODAY}"'
+execute_expecting "count date:today..today" "1"
+# End Test daterange parser-
+
 cat <

[notmuch] [PATCH] Switch from random to sequential thread identifiers.

2010-02-08 Thread Carl Worth
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up half as much space in the "notmuch search" output (16 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).
---

Keith pointed out to me that there was obviously no benefit from
switching from hexadecimal to decimal here. So this second version of
the patch means 16-character identifiers rather than 20.

 lib/database-private.h |7 +-
 lib/database.cc|   52 ---
 lib/message.cc |   46 --
 test/notmuch-test  |2 +-
 4 files changed, 55 insertions(+), 52 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 5891584..5bb6e86 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -27,14 +27,19 @@

 struct _notmuch_database {
 notmuch_bool_t exception_reported;
+
 char *path;
+
+notmuch_bool_t needs_upgrade;
 notmuch_database_mode_t mode;
 Xapian::Database *xapian_db;
+
+uint64_t last_thread_id;
+
 Xapian::QueryParser *query_parser;
 Xapian::TermGenerator *term_gen;
 Xapian::ValueRangeProcessor *value_range_processor;

-notmuch_bool_t needs_upgrade;
 };

 /* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index cce7847..8641321 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -533,6 +533,8 @@ notmuch_database_open (const char *path,
 notmuch->needs_upgrade = FALSE;
 notmuch->mode = mode;
 try {
+   string last_thread_id;
+
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
notmuch->xapian_db = new Xapian::WritableDatabase (xapian_path,
   
Xapian::DB_CREATE_OR_OPEN);
@@ -567,6 +569,20 @@ notmuch_database_open (const char *path,
 notmuch_path, version, NOTMUCH_DATABASE_VERSION);
}
}
+
+   last_thread_id = notmuch->xapian_db->get_metadata ("last_thread_id");
+   if (last_thread_id.empty ()) {
+   notmuch->last_thread_id = 0;
+   } else {
+   const char *str;
+   char *end;
+
+   str = last_thread_id.c_str ();
+   notmuch->last_thread_id = strtoull (str, , 16);
+   if (*end != '\0')
+   INTERNAL_ERROR ("Malformed database last_thread_id: %s", str);
+   }
+
notmuch->query_parser = new Xapian::QueryParser;
notmuch->term_gen = new Xapian::TermGenerator;
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
@@ -1278,14 +1294,38 @@ _notmuch_database_link_message_to_children 
(notmuch_database_t *notmuch,
 return ret;
 }

+static const char *
+_notmuch_database_generate_thread_id (notmuch_database_t *notmuch)
+{
+/* 16 bytes (+ terminator) for hexadecimal representation of
+ * a 64-bit integer. */
+static char thread_id[17];
+Xapian::WritableDatabase *db;
+
+db = static_cast  (notmuch->xapian_db);
+
+notmuch->last_thread_id++;
+
+sprintf (thread_id, "%016llx", notmuch->last_thread_id);
+
+db->set_metadata ("last_thread_id", thread_id);
+
+return thread_id;
+}
+
 /* Given a (mostly empty) 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
  * We first look at 'message_file' and its link-relevant headers
  * (References and In-Reply-To) for message IDs. We also look in the
- * database for existing message that reference 'message'.
+ * database for existing message that reference 'message'. In either
+ * case, we will assign to the current message the first thread_id
+ * found (through either parent or child). We will also merge any
+ * existing, distinct threads where this message belongs to both,
+ * (which is not uncommon when mesages are processed out of order).
  *
- * The end result is to call _notmuch_message_ensure_thread_id which
+ * Finally, if not thread ID has been found through parent or child,
+ * we call _notmuch_message_generate_thread_id to generate a new
  * generates a new thread ID if the message doesn't connect to any
  * existing threads.
  */
@@ -1308,8 +1348,12 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
return status;

-if (thread_id == NULL)
-   _notmuch_message_ensure_thread_id (message);
+/* If not part of any existing thread, generate a new thread ID. */
+if (thread_id == NULL) {
+   thread_id = _notmuch_database_generate_thread_id (notmuch);
+
+   _notmuch_message_add_term (message, "thread", thread_id);
+}

 return 

[notmuch] strange behavior of indexing of and searching for strings containing '[]'

2010-02-08 Thread Jameson Rollins
On Mon, 08 Feb 2010 18:35:44 +0100, Jed Brown  wrote:
> On Mon, 08 Feb 2010 12:24:06 -0500, Jameson Rollins  finestructure.net> wrote:
> > I don't think that this is exactly correct.  The quoting is interpreted
> > by the shell in order to construct a single string that is then passed
> > as an argument to the program.
> 
> The command line distinguishes, but the constructed query does not.
> Look at query-string.c, the arguments are just concatenated.

Hi, Jed.  Yes, this is clear from the behavior, but I'm claiming it's a
bug that should be fixed.  It produces unexpected behavior with
confusing results.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100208/5168e42c/attachment.pgp>


[notmuch] strange behavior of indexing of and searching for strings containing '[]'

2010-02-08 Thread Jameson Rollins
On Fri, 5 Feb 2010 23:48:03 + (UTC), Olly Betts  wrote:
> On 2010-02-05, Jameson Rollins wrote:
> > Hey, folks.  I've been noticing some strange behavior of notmuch search
> > results for strings containing '[]'.  Here are some searches for some
> > exact strings in messages subjects:
> 
> The '[]' is a red herring.  Xapian's TermGenerator and QueryParser classes
> treat these two characters pretty much as if they were spaces.

Ah.  Thanks for the response, Olly.  This clears things up a lot.

> > servo:~ 0$ notmuch search subject:'emacs paned UI'
> 
> Note that the '' is quoting for the shell only here.  So Xapian sees:
> 
> subject:emacs paned UI
> 
> Assuming you are defaulting to an AND search, that's `emacs in the subject'
> AND `paned anywhere in the indexed text' AND `UI anywhere in the indexed 
> text'.

I don't think that this is exactly correct.  The quoting is interpreted
by the shell in order to construct a single string that is then passed
as an argument to the program.  Notmuch should then be seeing the single
string argument as the search parameter, and not breaking it up further.

Here's an example of what I mean:

servo:~/tmp/cdtemp.AYroUf 0$ cat parse 
#!/bin/bash
for arg; do echo "$arg"; done
servo:~/tmp/cdtemp.AYroUf 0$ ./parse subject:foo bar baz
subject:foo
bar
baz
servo:~/tmp/cdtemp.AYroUf 0$ ./parse subject:'foo bar' baz
subject:foo bar
baz
servo:~/tmp/cdtemp.AYroUf 0$ ./parse subject:"foo bar" baz
subject:foo bar
baz
servo:~/tmp/cdtemp.AYroUf 0$ 

As you can see in the last command, the argument "subject'foo bar'" is
passed as a single string by the shell, and should therefore be
interpreted as such by notmuch.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100208/a273d807/attachment-0001.pgp>


[notmuch] [PATCH 4/4] add documentation for the new daterange parser and remove TODO list item

2010-02-08 Thread Sebastian Spaeth
Signed-off-by: Sebastian Spaeth 
---
 TODO  |9 -
 notmuch.1 |   29 -
 notmuch.c |   24 +---
 3 files changed, 33 insertions(+), 29 deletions(-)

diff --git a/TODO b/TODO
index bdfe64c..86cbf74 100644
--- a/TODO
+++ b/TODO
@@ -114,15 +114,6 @@ notmuch library
 ---
 Index content from citations, please.

-Provide a sane syntax for date ranges. First, we don't want to require
-both endpoints to be specified. For example it would be nice to be
-able to say things like "since:2009-01-1" or "until:2009-01-1" and
-have the other endpoint be implicit. Second we'd like to support
-relative specifications of time such as "since:'2 months ago'". To do
-any of this we're probably going to need to break down an write our
-own parser for the query string rather than using Xapian's QueryParser
-class.
-
 Make failure to read a file (such as a permissions problem) a warning
 rather than an error (should be similar to the existing warning for a
 non-mail file).
diff --git a/notmuch.1 b/notmuch.1
index 282ad98..19628f0 100644
--- a/notmuch.1
+++ b/notmuch.1
@@ -413,17 +413,28 @@ expression).
 Finally, results can be restricted to only messages within a
 particular time range, (based on the Date: header) with a syntax of:

-   ..
+   date:..

-Each timestamp is a number representing the number of seconds since
-1970-01-01 00:00:00 UTC. This is not the most convenient means of
-expressing date ranges, but until notmuch is fixed to accept a more
-convenient form, one can use the date program to construct
-timestamps. For example, with the bash shell the folowing syntax would
-specify a date range to return messages from 2009-10-01 until the
-current time:
+A
+.B date
+can be specified in various formats. It parses the formats in this order and 
stops if one pattern matches:
+
+   Keywords:
+.B today,
+.B yesterday,
+.B thisweek,
+.B lastweek,
+.B thismonth,
+.B lastmonth.
+
+   Month-day: month[-day]] (month: "January", "Jan", 1)
+
+   ISO format: year[-month[-day]]
+
+   US format : month[/day[/year]]
+
+The date parser will try to fill in bits in the enddate from context if left 
out, e.g. a 'date:2004..01' will find from 2004-01-01 through 2004-01-31.

-   $(date +%s -d 2009-10-01)..$(date +%s)
 .SH SEE ALSO
 The emacs-based interface to notmuch (available as
 .B notmuch.el
diff --git a/notmuch.c b/notmuch.c
index 87479f8..b8fe01a 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -92,19 +92,21 @@ static const char search_terms_help[] =
 "\t\tmarks around any parenthesized expression).\n"
 "\n"
 "\t\tFinally, results can be restricted to only messages within a\n"
-"\t\tparticular time range, (based on the Date: header) with:\n"
+"\t\tparticular time range, (based on the Date: header) with a\n"
+"\t\tsyntax of: date:..\n"
 "\n"
-"\t\t\t..\n"
+"\t\tIt can be specified in the following formats, parsing will \n"
+"\t\tstop if the first pattern matches:\n"
+"\t\tKeywords: 'today','yesterday','thisweek','lastweek',\n"
+"\t\t'thismonth', 'lastmonth'. \n"
+"\t\tmonth-day : month[-day]] (month: January, Jan, or 1)\n"
+"\t\tISO format: year[-month[-day]] (month: January, Jan, or 1)\n"
+"\t\tUS format : month[/day[/year]]\n"
 "\n"
-"\t\tEach timestamp is a number representing the number of seconds\n"
-"\t\tsince 1970-01-01 00:00:00 UTC. This is not the most convenient\n"
-"\t\tmeans of expressing date ranges, but until notmuch is fixed to\n"
-"\t\taccept a more convenient form, one can use the date program to\n"
-"\t\tconstruct timestamps. For example, with the bash shell the\n"
-"\t\tfollowing syntax would specify a date range to return messages\n"
-"\t\tfrom 2009-10-01 until the current time:\n"
-"\n"
-"\t\t\t$(date +%%s -d 2009-10-01)..$(date +%%s)\n\n";
+"\t\tThe parser will fill in bits in the enddate from context if\n"
+"\t\tleft out, e.g. a 'date:2004..01' will find from 2004-01-01\n"
+"\t\tthrough 2004-01-31\n"
+"\n\n";

 command_t commands[] = {
 { "setup", notmuch_setup_command,
-- 
1.6.3.3



[notmuch] [PATCH 3/4] integrate date parser

2010-02-08 Thread Sebastian Spaeth
Integrate and make use of the notmuch_parse_date() function in date.c that is 
being called by the new MaildateValueRangeProcessor in lib/database.cc
Thanks to keithp for donating the date parser for achieving this in a nice way.

Signed-off-by: Sebastian Spaeth 
---
 lib/Makefile.local |1 +
 lib/database.cc|   33 -
 lib/notmuch.h  |   17 +
 3 files changed, 50 insertions(+), 1 deletions(-)

diff --git a/lib/Makefile.local b/lib/Makefile.local
index 70489e1..44deaf8 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -3,6 +3,7 @@ extra_cflags += -I$(dir)

 libnotmuch_c_srcs =\
$(dir)/libsha1.c\
+   $(dir)/date.c   \
$(dir)/message-file.c   \
$(dir)/messages.c   \
$(dir)/sha1.c   \
diff --git a/lib/database.cc b/lib/database.cc
index ddda933..78cd898 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -496,6 +496,37 @@ _notmuch_database_ensure_writable (notmuch_database_t 
*notmuch)
 return NOTMUCH_STATUS_SUCCESS;
 }

+struct MaildateValueRangeProcessor : public Xapian::ValueRangeProcessor {
+MaildateValueRangeProcessor() {}
+
+Xapian::valueno operator()(std::string , std::string ) {
+  time_t begin_first,begin_last, end_first, end_last;
+  int retval;
+
+  if (begin.substr(0, 5) != "date:")
+return Xapian::BAD_VALUENO;
+  begin.erase(0, 5);
+
+  retval = notmuch_parse_date(begin.c_str(), _first, _last, 0);
+
+  if (retval == NOTMUCH_STATUS_INVALID_DATE) {
+   fprintf(stderr,"Begin date failed to parse: %s",begin.c_str());
+   return Xapian::BAD_VALUENO;
+  }
+
+  retval = 
notmuch_parse_date(end.c_str(),_first,_last,begin_first);
+  if (retval == NOTMUCH_STATUS_INVALID_DATE) {
+   fprintf(stderr,"End date failed to parse: %s",end.c_str());
+   return Xapian::BAD_VALUENO;
+  }
+
+  begin.assign(Xapian::sortable_serialise(begin_first));
+  end.assign(Xapian::sortable_serialise(end_last));
+
+  return NOTMUCH_VALUE_TIMESTAMP;
+}
+};
+
 notmuch_database_t *
 notmuch_database_open (const char *path,
   notmuch_database_mode_t mode)
@@ -572,7 +603,7 @@ notmuch_database_open (const char *path,
notmuch->query_parser = new Xapian::QueryParser;
notmuch->term_gen = new Xapian::TermGenerator;
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
-   notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor 
(NOTMUCH_VALUE_TIMESTAMP);
+   notmuch->value_range_processor = new MaildateValueRangeProcessor();

notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
notmuch->query_parser->set_database (*notmuch->xapian_db);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 56a76d1..fa8f41a 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1089,6 +1089,23 @@ notmuch_filenames_advance (notmuch_filenames_t 
*filenames);
 void
 notmuch_filenames_destroy (notmuch_filenames_t *filenames);

+notmuch_status_t
+notmuch_parse_date(const char *text, time_t *first, time_t *last, time_t 
after);
+/* Parse a string into the first and last possible timestamps.
+ * It parses the possible formats and stops if one pattern matches.
+ * Keywords: 'today','yesterday','thisweek','lastweek','thismonth',
+ *   'lastmonth'
+ * Month-day : month[-day]] (month: January, Jan, or 1)\n"
+ * ISO format: year[-month[-day]]
+ * US format : month[/day[/year]]
+ *
+ * 'after' is used to fill in bits from context if left out, e.g. a
+ * 'date:2004..01' will find from 2004-01-01 through 2004-01-31
+ *
+ * Return values:
+ * NOTMUCH_STATUS_SUCCESS
+ * NOTMUCH_STATUS_INVALID_DATE: Error parsing the date string
+ */
 NOTMUCH_END_DECLS

 #endif
-- 
1.6.3.3



[notmuch] [PATCH 2/4] Introduce new notmuch return status NOTMUCH_STATUS_INVALID_DATE

2010-02-08 Thread Sebastian Spaeth
This is not used yet in this commit but will be the return value in case the 
date parser gets handed invalid dates.

Signed-off-by: Sebastian Spaeth 
---
 lib/database.cc |2 ++
 lib/notmuch.h   |3 +++
 notmuch-new.c   |1 +
 3 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index cce7847..ddda933 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -219,6 +219,8 @@ notmuch_status_to_string (notmuch_status_t status)
return "Erroneous NULL pointer";
 case NOTMUCH_STATUS_TAG_TOO_LONG:
return "Tag value is too long (exceeds NOTMUCH_TAG_MAX)";
+case NOTMUCH_STATUS_INVALID_DATE:
+   return "Date value did not parse to a valid date";
 case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
return "Unbalanced number of calls to notmuch_message_freeze/thaw";
 default:
diff --git a/lib/notmuch.h b/lib/notmuch.h
index d3e50a7..56a76d1 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -78,6 +78,8 @@ typedef int notmuch_bool_t;
  * NOTMUCH_STATUS_TAG_TOO_LONG: A tag value is too long (exceeds
  * NOTMUCH_TAG_MAX)
  *
+ * NOTMUCH_STATUS_INVALID_DATE: Date parsing failed
+ *
  * NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW: The notmuch_message_thaw
  * function has been called more times than notmuch_message_freeze.
  *
@@ -96,6 +98,7 @@ typedef enum _notmuch_status {
 NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID,
 NOTMUCH_STATUS_NULL_POINTER,
 NOTMUCH_STATUS_TAG_TOO_LONG,
+NOTMUCH_STATUS_INVALID_DATE,
 NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW,

 NOTMUCH_STATUS_LAST_STATUS
diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..5da31c1 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -431,6 +431,7 @@ add_files_recursive (notmuch_database_t *notmuch,
ret = status;
goto DONE;
default:
+   case NOTMUCH_STATUS_INVALID_DATE:
case NOTMUCH_STATUS_FILE_ERROR:
case NOTMUCH_STATUS_NULL_POINTER:
case NOTMUCH_STATUS_TAG_TOO_LONG:
-- 
1.6.3.3



[notmuch] [PATCH 1/4] add date parser file from Keith

2010-02-08 Thread Sebastian Spaeth
From: Keith Packard 

Here's some code which further improves date parsing by allowing lots of
date formats, including things like "today", "thisweek", ISO and US date
formats and month names. You can separate two dates with .. to make a
range, or you can just use the default range ("lastmonth" is everything
>From the 1st of the previous month to the 1st of the current month).

Sebastian Spaeth removed 2 unneeded functions and silenced some unused variable 
warnings, but left the code otherwise as-is.
---
 lib/date.c |  414 
 1 files changed, 414 insertions(+), 0 deletions(-)
 create mode 100644 lib/date.c

diff --git a/lib/date.c b/lib/date.c
new file mode 100644
index 000..805a1d9
--- /dev/null
+++ b/lib/date.c
@@ -0,0 +1,414 @@
+/*
+ * Copyright ?? 2009 Keith Packard 
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
+ */
+
+#include "notmuch.h"
+#include 
+#include 
+#include 
+#include 
+
+#define DAY(24 * 60 * 60)
+
+static void
+today(struct tm *result, time_t after) {
+time_t t;
+
+if (after)
+   t = after;
+else
+   time();
+localtime_r(, result);
+result->tm_sec = result->tm_min = result->tm_hour = 0;
+}
+
+static int parse_today(const char *text, time_t *first, time_t *last, time_t 
after) {
+(void)after; /*disable unused paramter warning*/
+if (strcasecmp(text, "today") == 0) {
+   struct tm n;
+   today(, 0);
+   *first = mktime();
+   *last = *first + DAY;
+   return 0;
+}
+return 1;
+}
+
+static int parse_yesterday(const char *text, time_t *first, time_t *last, 
time_t after) {
+if (strcasecmp(text, "yesterday") == 0) {
+   struct tm n;
+   today(, 0);
+   *last = mktime();
+   *first = *last - DAY;
+   return 0;
+}
+return 1;
+(void)after; /*disable unused paramter warning*/
+}
+
+static int parse_thisweek(const char *text, time_t *first, time_t *last, 
time_t after) {
+if (strcasecmp(text, "thisweek") == 0) {
+   struct tm n;
+   today(, 0);
+   *first = mktime() - (n.tm_wday * DAY);
+   *last = *first + DAY * 7;
+   return 0;
+}
+return 1;
+(void)after; /*disable unused paramter warning*/
+}
+
+static int parse_lastweek(const char *text, time_t *first, time_t *last, 
time_t after) {
+if (strcasecmp(text, "lastweek") == 0) {
+   struct tm n;
+   today(, 0);
+   *last = mktime() - (n.tm_wday * DAY);
+   *first = *last - DAY * 7;
+   return 0;
+}
+return 1;
+(void)after; /*disable unused paramter warning*/
+}
+
+static int parse_thismonth(const char *text, time_t *first, time_t *last, 
time_t after) {
+if (strcasecmp(text, "thismonth") == 0) {
+   struct tm n;
+   today(, 0);
+   n.tm_mday = 1;
+   *first = mktime();
+   if (n.tm_mon++ == 12) {
+   n.tm_mon = 0;
+   n.tm_year++;
+   }
+   *last = mktime();
+   return 0;
+}
+return 1;
+(void)after; /*disable unused paramter warning*/
+}
+
+static int parse_lastmonth(const char *text, time_t *first, time_t *last, 
time_t after) {
+if (strcasecmp(text, "lastmonth") == 0) {
+   struct tm n;
+   today(, 0);
+   n.tm_mday = 1;
+   if (n.tm_mon == 0) {
+   n.tm_year--;
+   n.tm_mon = 11;
+   } else
+   n.tm_mon--;
+   *first = mktime();
+   if (n.tm_mon++ == 12) {
+   n.tm_mon = 0;
+   n.tm_year++;
+   }
+   *last = mktime();
+   return 0;
+}
+return 1;
+(void)after; /*disable unused paramter warning*/
+}
+
+static const char *months[12][2] = {
+{ "January", "Jan" },
+{ "February", "Feb" },
+{ "March", "Mar" },
+{ "April", "Apr" },
+{ "May", "May" },
+{ "June", "Jun" },
+{ "July", "Jul" },
+{ "August", "Aug" },
+{ "September", "Sep" },
+{ "October", "Oct" },
+{ "November", "Nov" },
+{ "December", "Dec" },
+};
+
+static int year(const char *text, int *y) {
+char *end;
+*y = strtol(text, , 10);
+if (end == text)
+   return 1;
+if (*end != '\0')
+   return 1;
+if (*y < 1970 || *y > 2038)
+   return 1;
+*y -= 1900;
+return 0;
+}
+
+static int month(const char *text, int *m) {
+

[notmuch] [PATCH] Switch from random to sequential thread identifiers.

2010-02-08 Thread Carl Worth
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up slightly less space in the "notmuch search" output (20 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).

---

I'm sending this patch to the mailing-list rather than pushing it
directly so that any authors of user interfaces can ensure that they are
ready for the length of thread identifiers in "notmuch search" output to
change.

I tested that the emacs client doesn't need any change in this regard.

And the change doesn't introduce any compatibility with an existing
database, so no rebuild required (hurrah!).

-Carl

 lib/database-private.h |7 +-
 lib/database.cc|   52 ---
 lib/message.cc |   46 --
 test/notmuch-test  |2 +-
 4 files changed, 55 insertions(+), 52 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 5891584..5bb6e86 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -27,14 +27,19 @@

 struct _notmuch_database {
 notmuch_bool_t exception_reported;
+
 char *path;
+
+notmuch_bool_t needs_upgrade;
 notmuch_database_mode_t mode;
 Xapian::Database *xapian_db;
+
+uint64_t last_thread_id;
+
 Xapian::QueryParser *query_parser;
 Xapian::TermGenerator *term_gen;
 Xapian::ValueRangeProcessor *value_range_processor;

-notmuch_bool_t needs_upgrade;
 };

 /* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index cce7847..65ff12e 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -533,6 +533,8 @@ notmuch_database_open (const char *path,
 notmuch->needs_upgrade = FALSE;
 notmuch->mode = mode;
 try {
+   string last_thread_id;
+
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
notmuch->xapian_db = new Xapian::WritableDatabase (xapian_path,
   
Xapian::DB_CREATE_OR_OPEN);
@@ -567,6 +569,20 @@ notmuch_database_open (const char *path,
 notmuch_path, version, NOTMUCH_DATABASE_VERSION);
}
}
+
+   last_thread_id = notmuch->xapian_db->get_metadata ("last_thread_id");
+   if (last_thread_id.empty ()) {
+   notmuch->last_thread_id = 0;
+   } else {
+   const char *str;
+   char *end;
+
+   str = last_thread_id.c_str ();
+   notmuch->last_thread_id = strtoull (str, , 10);
+   if (*end != '\0')
+   INTERNAL_ERROR ("Malformed database last_thread_id: %s", str);
+   }
+
notmuch->query_parser = new Xapian::QueryParser;
notmuch->term_gen = new Xapian::TermGenerator;
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
@@ -1278,14 +1294,38 @@ _notmuch_database_link_message_to_children 
(notmuch_database_t *notmuch,
 return ret;
 }

+static const char *
+_notmuch_database_generate_thread_id (notmuch_database_t *notmuch)
+{
+/* 20 bytes (+ terminator) for ASCII decimal representation of
+ * a 64-bit integer. */
+static char thread_id[21];
+Xapian::WritableDatabase *db;
+
+db = static_cast  (notmuch->xapian_db);
+
+notmuch->last_thread_id++;
+
+sprintf (thread_id, "%020llu", notmuch->last_thread_id);
+
+db->set_metadata ("last_thread_id", thread_id);
+
+return thread_id;
+}
+
 /* Given a (mostly empty) 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
  * We first look at 'message_file' and its link-relevant headers
  * (References and In-Reply-To) for message IDs. We also look in the
- * database for existing message that reference 'message'.
+ * database for existing message that reference 'message'. In either
+ * case, we will assign to the current message the first thread_id
+ * found (through either parent or child). We will also merge any
+ * existing, distinct threads where this message belongs to both,
+ * (which is not uncommon when mesages are processed out of order).
  *
- * The end result is to call _notmuch_message_ensure_thread_id which
+ * Finally, if not thread ID has been found through parent or child,
+ * we call _notmuch_message_generate_thread_id to generate a new
  * generates a new thread ID if the message doesn't connect to any
  * existing threads.
  */
@@ -1308,8 +1348,12 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
return status;

-if (thread_id == NULL)
-   _notmuch_message_ensure_thread_id (message);
+/* If not part of any existing thread, 

[notmuch] date parser branch updated and rebased

2010-02-08 Thread Sebastian Spaeth
In order to not let this bitrot, I just rebased my dateparser branch to
current cworth master and split up the patches in a nicer way.
I made sure that every commit compiles. Let me know if there are things
that should be handled in a different way.

find the git branch here:
http://github.com/spaetz/notmuch-all-feature/commits/dateparser5

All 4 patches are going to be send as a reply to this mail in a second.

Sebastian


[notmuch] date parser branch updated and rebased

2010-02-08 Thread Sebastian Spaeth
In order to not let this bitrot, I just rebased my dateparser branch to
current cworth master and split up the patches in a nicer way.
I made sure that every commit compiles. Let me know if there are things
that should be handled in a different way.

find the git branch here:
http://github.com/spaetz/notmuch-all-feature/commits/dateparser5

All 4 patches are going to be send as a reply to this mail in a second.

Sebastian
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] [PATCH 2/4] Introduce new notmuch return status NOTMUCH_STATUS_INVALID_DATE

2010-02-08 Thread Sebastian Spaeth
This is not used yet in this commit but will be the return value in case the 
date parser gets handed invalid dates.

Signed-off-by: Sebastian Spaeth sebast...@sspaeth.de
---
 lib/database.cc |2 ++
 lib/notmuch.h   |3 +++
 notmuch-new.c   |1 +
 3 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index cce7847..ddda933 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -219,6 +219,8 @@ notmuch_status_to_string (notmuch_status_t status)
return Erroneous NULL pointer;
 case NOTMUCH_STATUS_TAG_TOO_LONG:
return Tag value is too long (exceeds NOTMUCH_TAG_MAX);
+case NOTMUCH_STATUS_INVALID_DATE:
+   return Date value did not parse to a valid date;
 case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
return Unbalanced number of calls to notmuch_message_freeze/thaw;
 default:
diff --git a/lib/notmuch.h b/lib/notmuch.h
index d3e50a7..56a76d1 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -78,6 +78,8 @@ typedef int notmuch_bool_t;
  * NOTMUCH_STATUS_TAG_TOO_LONG: A tag value is too long (exceeds
  * NOTMUCH_TAG_MAX)
  *
+ * NOTMUCH_STATUS_INVALID_DATE: Date parsing failed
+ *
  * NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW: The notmuch_message_thaw
  * function has been called more times than notmuch_message_freeze.
  *
@@ -96,6 +98,7 @@ typedef enum _notmuch_status {
 NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID,
 NOTMUCH_STATUS_NULL_POINTER,
 NOTMUCH_STATUS_TAG_TOO_LONG,
+NOTMUCH_STATUS_INVALID_DATE,
 NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW,
 
 NOTMUCH_STATUS_LAST_STATUS
diff --git a/notmuch-new.c b/notmuch-new.c
index f25c71f..5da31c1 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -431,6 +431,7 @@ add_files_recursive (notmuch_database_t *notmuch,
ret = status;
goto DONE;
default:
+   case NOTMUCH_STATUS_INVALID_DATE:
case NOTMUCH_STATUS_FILE_ERROR:
case NOTMUCH_STATUS_NULL_POINTER:
case NOTMUCH_STATUS_TAG_TOO_LONG:
-- 
1.6.3.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] [PATCH 3/4] integrate date parser

2010-02-08 Thread Sebastian Spaeth
Integrate and make use of the notmuch_parse_date() function in date.c that is 
being called by the new MaildateValueRangeProcessor in lib/database.cc
Thanks to keithp for donating the date parser for achieving this in a nice way.

Signed-off-by: Sebastian Spaeth sebast...@sspaeth.de
---
 lib/Makefile.local |1 +
 lib/database.cc|   33 -
 lib/notmuch.h  |   17 +
 3 files changed, 50 insertions(+), 1 deletions(-)

diff --git a/lib/Makefile.local b/lib/Makefile.local
index 70489e1..44deaf8 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -3,6 +3,7 @@ extra_cflags += -I$(dir)
 
 libnotmuch_c_srcs =\
$(dir)/libsha1.c\
+   $(dir)/date.c   \
$(dir)/message-file.c   \
$(dir)/messages.c   \
$(dir)/sha1.c   \
diff --git a/lib/database.cc b/lib/database.cc
index ddda933..78cd898 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -496,6 +496,37 @@ _notmuch_database_ensure_writable (notmuch_database_t 
*notmuch)
 return NOTMUCH_STATUS_SUCCESS;
 }
 
+struct MaildateValueRangeProcessor : public Xapian::ValueRangeProcessor {
+MaildateValueRangeProcessor() {}
+
+Xapian::valueno operator()(std::string begin, std::string end) {
+  time_t begin_first,begin_last, end_first, end_last;
+  int retval;
+
+  if (begin.substr(0, 5) != date:)
+return Xapian::BAD_VALUENO;
+  begin.erase(0, 5);
+
+  retval = notmuch_parse_date(begin.c_str(), begin_first, begin_last, 0);
+
+  if (retval == NOTMUCH_STATUS_INVALID_DATE) {
+   fprintf(stderr,Begin date failed to parse: %s,begin.c_str());
+   return Xapian::BAD_VALUENO;
+  }
+
+  retval = 
notmuch_parse_date(end.c_str(),end_first,end_last,begin_first);
+  if (retval == NOTMUCH_STATUS_INVALID_DATE) {
+   fprintf(stderr,End date failed to parse: %s,end.c_str());
+   return Xapian::BAD_VALUENO;
+  }
+
+  begin.assign(Xapian::sortable_serialise(begin_first));
+  end.assign(Xapian::sortable_serialise(end_last));
+
+  return NOTMUCH_VALUE_TIMESTAMP;
+}
+};
+
 notmuch_database_t *
 notmuch_database_open (const char *path,
   notmuch_database_mode_t mode)
@@ -572,7 +603,7 @@ notmuch_database_open (const char *path,
notmuch-query_parser = new Xapian::QueryParser;
notmuch-term_gen = new Xapian::TermGenerator;
notmuch-term_gen-set_stemmer (Xapian::Stem (english));
-   notmuch-value_range_processor = new Xapian::NumberValueRangeProcessor 
(NOTMUCH_VALUE_TIMESTAMP);
+   notmuch-value_range_processor = new MaildateValueRangeProcessor();
 
notmuch-query_parser-set_default_op (Xapian::Query::OP_AND);
notmuch-query_parser-set_database (*notmuch-xapian_db);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 56a76d1..fa8f41a 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1089,6 +1089,23 @@ notmuch_filenames_advance (notmuch_filenames_t 
*filenames);
 void
 notmuch_filenames_destroy (notmuch_filenames_t *filenames);
 
+notmuch_status_t
+notmuch_parse_date(const char *text, time_t *first, time_t *last, time_t 
after);
+/* Parse a string into the first and last possible timestamps.
+ * It parses the possible formats and stops if one pattern matches.
+ * Keywords: 'today','yesterday','thisweek','lastweek','thismonth',
+ *   'lastmonth'
+ * Month-day : month[-day]] (month: January, Jan, or 1)\n
+ * ISO format: year[-month[-day]]
+ * US format : month[/day[/year]]
+ *
+ * 'after' is used to fill in bits from context if left out, e.g. a
+ * 'date:2004..01' will find from 2004-01-01 through 2004-01-31
+ *
+ * Return values:
+ * NOTMUCH_STATUS_SUCCESS
+ * NOTMUCH_STATUS_INVALID_DATE: Error parsing the date string
+ */
 NOTMUCH_END_DECLS
 
 #endif
-- 
1.6.3.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] [PATCH 4/4] add documentation for the new daterange parser and remove TODO list item

2010-02-08 Thread Sebastian Spaeth
Signed-off-by: Sebastian Spaeth sebast...@sspaeth.de
---
 TODO  |9 -
 notmuch.1 |   29 -
 notmuch.c |   24 +---
 3 files changed, 33 insertions(+), 29 deletions(-)

diff --git a/TODO b/TODO
index bdfe64c..86cbf74 100644
--- a/TODO
+++ b/TODO
@@ -114,15 +114,6 @@ notmuch library
 ---
 Index content from citations, please.
 
-Provide a sane syntax for date ranges. First, we don't want to require
-both endpoints to be specified. For example it would be nice to be
-able to say things like since:2009-01-1 or until:2009-01-1 and
-have the other endpoint be implicit. Second we'd like to support
-relative specifications of time such as since:'2 months ago'. To do
-any of this we're probably going to need to break down an write our
-own parser for the query string rather than using Xapian's QueryParser
-class.
-
 Make failure to read a file (such as a permissions problem) a warning
 rather than an error (should be similar to the existing warning for a
 non-mail file).
diff --git a/notmuch.1 b/notmuch.1
index 282ad98..19628f0 100644
--- a/notmuch.1
+++ b/notmuch.1
@@ -413,17 +413,28 @@ expression).
 Finally, results can be restricted to only messages within a
 particular time range, (based on the Date: header) with a syntax of:
 
-   intial-timestamp..final-timestamp
+   date:startdate..enddate
 
-Each timestamp is a number representing the number of seconds since
-1970-01-01 00:00:00 UTC. This is not the most convenient means of
-expressing date ranges, but until notmuch is fixed to accept a more
-convenient form, one can use the date program to construct
-timestamps. For example, with the bash shell the folowing syntax would
-specify a date range to return messages from 2009-10-01 until the
-current time:
+A
+.B date
+can be specified in various formats. It parses the formats in this order and 
stops if one pattern matches:
+
+   Keywords:
+.B today,
+.B yesterday,
+.B thisweek,
+.B lastweek,
+.B thismonth,
+.B lastmonth.
+
+   Month-day: month[-day]] (month: January, Jan, 1)
+
+   ISO format: year[-month[-day]]
+
+   US format : month[/day[/year]]
+
+The date parser will try to fill in bits in the enddate from context if left 
out, e.g. a 'date:2004..01' will find from 2004-01-01 through 2004-01-31.
 
-   $(date +%s -d 2009-10-01)..$(date +%s)
 .SH SEE ALSO
 The emacs-based interface to notmuch (available as
 .B notmuch.el
diff --git a/notmuch.c b/notmuch.c
index 87479f8..b8fe01a 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -92,19 +92,21 @@ static const char search_terms_help[] =
 \t\tmarks around any parenthesized expression).\n
 \n
 \t\tFinally, results can be restricted to only messages within a\n
-\t\tparticular time range, (based on the Date: header) with:\n
+\t\tparticular time range, (based on the Date: header) with a\n
+\t\tsyntax of: date:startdate..enddate\n
 \n
-\t\t\tintial-timestamp..final-timestamp\n
+\t\tIt can be specified in the following formats, parsing will \n
+\t\tstop if the first pattern matches:\n
+\t\tKeywords: 'today','yesterday','thisweek','lastweek',\n
+\t\t'thismonth', 'lastmonth'. \n
+\t\tmonth-day : month[-day]] (month: January, Jan, or 1)\n
+\t\tISO format: year[-month[-day]] (month: January, Jan, or 1)\n
+\t\tUS format : month[/day[/year]]\n
 \n
-\t\tEach timestamp is a number representing the number of seconds\n
-\t\tsince 1970-01-01 00:00:00 UTC. This is not the most convenient\n
-\t\tmeans of expressing date ranges, but until notmuch is fixed to\n
-\t\taccept a more convenient form, one can use the date program to\n
-\t\tconstruct timestamps. For example, with the bash shell the\n
-\t\tfollowing syntax would specify a date range to return messages\n
-\t\tfrom 2009-10-01 until the current time:\n
-\n
-\t\t\t$(date +%%s -d 2009-10-01)..$(date +%%s)\n\n;
+\t\tThe parser will fill in bits in the enddate from context if\n
+\t\tleft out, e.g. a 'date:2004..01' will find from 2004-01-01\n
+\t\tthrough 2004-01-31\n
+\n\n;
 
 command_t commands[] = {
 { setup, notmuch_setup_command,
-- 
1.6.3.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] notmuch new: Memory problem (with uuencoded content)

2010-02-08 Thread Michal Sojka
On Saturday 06 of February 2010 22:45:32 Carl Worth wrote:
 On Sat, 6 Feb 2010 11:40:18 +0100, Michal Sojka sojk...@fel.cvut.cz wrote:
  It is straightforward to convert your current test script to Git's
  framework. If you are interested I'll do it.
 
 Yes, I'd be quite interested in seeing that. Thanks for your
 contributions, and sorry I missed (or haven't yet gotten to) the patch
 you sent earlier.

Hi Carl,

I did the conversion of the test script. I'll post it to thread 
id:87ljf8pvxx@yoom.home.cworth.org, where it is more appropriate.

Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-08 Thread Michal Sojka
On Thursday 04 of February 2010 21:50:18 Carl Worth wrote:
 The test suite is still extremely rudimentary. Here are some things I'd
 like to improve about it:

I converted the actual version of notmuch-test to git test framework.
The result is in the followup patches.

I'd like to know opinion of others. If Carl agrees that it could be
merged I can do additional work as I describe below.

The conversion was not as straightforward as I expected mainly because
of problems with quoting. There are several sources of quotation problems
one being Carl's hashed array parameters. I thing it would be
sufficient to use plain variables for passing additional parameters.
Instead of:
add_message [from]=\Sender sen...@example.com\ \
[to]=test_su...@notmuchmail.org \
[subject]=notmuch-reply-test \
[date]=\Tue, 05 Jan 2010 15:43:56 -0800\ \
[body]=\basic reply test\ 
I'd do:
(
msg_from=Sender sen...@example.com
msg_to=test_su...@notmuchmail.org 
msg_subject=notmuch-reply-test
msg_date=Tue, 05 Jan 2010 15:43:56 -0800
msg_body=basic reply test
add_message
)

A possible additional improvement is elimination of
execute_expecting(). Combination of action (running notmuch) and
testing of a result in a single function makes it hard to distinguish
whether the problem is in the action or in the output. For example, if
notmuch is killed because of segfault, it would look like that no
output was produced. So instead of:

execute_expecting new No new mail. Removed 3 messages.

I'd write something like:

echo No new mail. Removed 3 messages.  expected

  test_expect_success 'Run notmuch'
'notmuch_filter_output new  actual'
  test_expect_success 'Compare actual and correct output'
'test_cmp expected actual'

where test_cmp is basically diff -u. This has also the advantage that
longer output (e.g. as in t0003-reply.sh) is more readable if there is
a difference.

On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote:
 Carl, have you considered outputting the test suite in the same format
 as the test anything protocol? [1] I only mention this because it
 might be a nice way to easily do some reporting (or perhaps even
 continuous integration) notmuch, with trivial effort.

According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project 
Git test output does not conform to TAP, but from my quick look it could be 
easily changed to conform.


On Friday 05 of February 2010 00:29:27 Carl Worth wrote:
 Looking at TAP, one thing I don't like is that it prints the
 success/failure of the test first, before the description of the
 test. That's not so nice in the case of a long-running (perhaps
 infinitely running) test where you might need to interrupt it, but you'd
 still want to know *what* was running for so long.

Git test framework also outputs first the result and then the test. It can be 
easily changed by -v switch.

To conclude, the output of running make in test/ directory now looks like 
this:
*** t-basic.sh ***
*   ok 1: test that mail store was created
*   ok 2: mail store should be empty
*   ok 3: NOTMUCH_CONFIG is set and points to an existing file
*   ok 4: PATH is set to this repository
*   ok 5: success is reported like this
*   still broken 6: pretend we have a known breakage
*   FIXED 7: pretend we have fixed a known breakage
*   ok 8: test runs if prerequisite is satisfied
* skip 9: unmet prerequisite causes test to be skipped
* fixed 1 known breakage(s)
* still have 1 known breakage(s)
* passed all remaining 8 test(s)
*** t0001-new.sh ***
*   ok 1: No new messages
*   ok 2: Single new message
*   ok 3: Multiple new messages
*   ok 4: No new messages (non-empty DB)
*   ok 5: New directories
*   ok 6: Alternate inode order
*   ok 7: Message moved in
*   ok 8: Renamed message
*   ok 9: Deleted message
*   ok 10: Renamed directory
*   ok 11: Deleted directory
*   ok 12: New directory (at end of list)
*   ok 13: Deleted directory (end of list)
*   ok 14: New symlink to directory
*   ok 15: New symlink to a file
*   ok 16: New two-level directory
*   ok 17: Deleted two-level directory
* passed all 17 test(s)
*** t0002-search.sh ***
*   ok 1: Search body
*   ok 2: Search by from:
*   ok 3: Search by to:
*   ok 4: Search by subject:
*   ok 5: Search by id:
*   ok 6: Search by tag:
*   ok 7: Search by thread:
*   ok 8: Search body (phrase)
*   ok 9: Search by from: (address)
*   ok 10: Search by from: (name)
*   ok 11: Search by to: (address)
*   ok 12: Search by to: (name)
*   ok 13: Search by subject: (phrase)
* passed all 13 test(s)
*** t0003-reply.sh ***
*   ok 1: Basic reply
*   ok 2: Multiple recipients
*   ok 3: Reply with CC
*   ok 4: Reply from alternate address
*   ok 5: Support for Reply-To
*   ok 6: Un-munging Reply-To
* passed all 6 test(s)
*** t0004-uuencoded-data.sh ***
*   ok 1: Generate message
*   ok 2: Ensure content before uu data is indexed
*   ok 3: Ensure uu 

[notmuch] [PATCH 1/3] Copy test framework from Git

2010-02-08 Thread Michal Sojka
Git uses a simple and yet powerfull test framework, written in shell.
The framework is easy to use for both users and developers so I thing
it would help if it is used in notmuch as well.

This is a copy of Git's test framework from commit
b8bba419250711a69e09e7648e5c991f4847a127.

Signed-off-by: Michal Sojka sojk...@fel.cvut.cz
---
 test/Makefile |   46 +++
 test/README   |  297 +
 test/aggregate-results.sh |   34 ++
 test/t-basic.sh   |  368 +
 test/test-lib.sh  |  787 +
 5 files changed, 1532 insertions(+), 0 deletions(-)
 create mode 100644 test/Makefile
 create mode 100644 test/README
 create mode 100755 test/aggregate-results.sh
 create mode 100755 test/t-basic.sh
 create mode 100644 test/test-lib.sh

diff --git a/test/Makefile b/test/Makefile
new file mode 100644
index 000..bd09390
--- /dev/null
+++ b/test/Makefile
@@ -0,0 +1,46 @@
+# Run tests
+#
+# Copyright (c) 2005 Junio C Hamano
+#
+
+-include ../config.mak
+
+#GIT_TEST_OPTS=--verbose --debug
+SHELL_PATH ?= $(SHELL)
+TAR ?= $(TAR)
+RM ?= rm -f
+
+# Shell quote;
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
+
+T = $(wildcard t[0-9][0-9][0-9][0-9]-*.sh)
+TSVN = $(wildcard t91[0-9][0-9]-*.sh)
+
+all: pre-clean
+   $(MAKE) aggregate-results-and-cleanup
+
+$(T):
+   @echo *** $@ ***; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ 
$(GIT_TEST_OPTS)
+
+pre-clean:
+   $(RM) -r test-results
+
+clean:
+   $(RM) -r 'trash directory'.* test-results
+
+aggregate-results-and-cleanup: $(T)
+   $(MAKE) aggregate-results
+   $(MAKE) clean
+
+aggregate-results:
+   '$(SHELL_PATH_SQ)' ./aggregate-results.sh test-results/t*-*
+
+# we can test NO_OPTIMIZE_COMMITS independently of LC_ALL
+full-svn-test:
+   $(MAKE) $(TSVN) GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C
+   $(MAKE) $(TSVN) GIT_SVN_NO_OPTIMIZE_COMMITS=0 LC_ALL=en_US.UTF-8
+
+valgrind:
+   GIT_TEST_OPTS=--valgrind $(MAKE)
+
+.PHONY: pre-clean $(T) aggregate-results clean valgrind
diff --git a/test/README b/test/README
new file mode 100644
index 000..dcd3ebb
--- /dev/null
+++ b/test/README
@@ -0,0 +1,297 @@
+Core GIT Tests
+==
+
+This directory holds many test scripts for core GIT tools.  The
+first part of this short document describes how to run the tests
+and read their output.
+
+When fixing the tools or adding enhancements, you are strongly
+encouraged to add tests in this directory to cover what you are
+trying to fix or enhance.  The later part of this short document
+describes how your test scripts should be organized.
+
+
+Running Tests
+-
+
+The easiest way to run tests is to say make.  This runs all
+the tests.
+
+*** t-basic.sh ***
+*   ok 1: .git/objects should be empty after git-init in an empty repo.
+*   ok 2: .git/objects should have 256 subdirectories.
+*   ok 3: git-update-index without --add should fail adding.
+...
+*   ok 23: no diff after checkout and git-update-index --refresh.
+* passed all 23 test(s)
+*** t0100-environment-names.sh ***
+*   ok 1: using old names should issue warnings.
+*   ok 2: using old names but having new names should not issue warnings.
+...
+
+Or you can run each test individually from command line, like
+this:
+
+$ sh ./t3001-ls-files-killed.sh
+*   ok 1: git-update-index --add to add various paths.
+*   ok 2: git-ls-files -k to show killed files.
+*   ok 3: validate git-ls-files -k output.
+* passed all 3 test(s)
+
+You can pass --verbose (or -v), --debug (or -d), and --immediate
+(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
+appropriately before running make.
+
+--verbose::
+   This makes the test more verbose.  Specifically, the
+   command being run and their output if any are also
+   output.
+
+--debug::
+   This may help the person who is developing a new test.
+   It causes the command defined with test_debug to run.
+
+--immediate::
+   This causes the test to immediately exit upon the first
+   failed test.
+
+--long-tests::
+   This causes additional long-running tests to be run (where
+   available), for more exhaustive testing.
+
+--valgrind::
+   Execute all Git binaries with valgrind and exit with status
+   126 on errors (just like regular tests, this will only stop
+   the test script when running under -i).  Valgrind errors
+   go to stderr, so you might want to pass the -v option, too.
+
+   Since it makes no sense to run the tests with --valgrind and
+   not see any output, this option implies --verbose.  For
+   convenience, it also implies --tee.
+
+--tee::
+   In addition to printing the test output to the terminal,
+   write it to files named 't/test-results/$TEST_NAME.out'.
+   As the names depend on the tests' file names, it is safe to
+   run the tests with this option in 

[notmuch] [PATCH 3/3] Convert the actual tests to the new framework

2010-02-08 Thread Michal Sojka
The changes are:
- The notmuch-test was split into several files (t000?-*.sh).
- Removed helper functions which were moved to test-lib.sh
- Replaced every printf with test_expect_success.
- Replaced $NOTMUCH with notmuch (test-lib.sh sets $PATH appropriately)
- Test commands chained with  (test-lib.sh doesn't use set -e in
  order to complete the test suite even if something fails)
- Many variables such as ${MAIL_DIR} were properly quoted as they
  contain spaces.
- Changed quoting patterns in add_message and generate_message (single
  quotes are already used by the test framework).
- ${TEST_DIR} replaced by ./

QUICK HOWTO:

To run the whole test suite
make

To run only a single test
   ./t0001-new.sh

To stop on the first error
   ./t0001-new.sh -i
then mail store and database can be inspected in
trash directory.t0001-new

To see the output of tests
   ./t0001-new.sh -v

To not remove trash directory at the end:
   ./t0001-new.sh -d

To run all tests verbosely:
   make make GIT_TEST_OPTS=-v

TODO: Convert GIT_* variables in test-lib.sh to NOTMUCH_*

Signed-off-by: Michal Sojka sojk...@fel.cvut.cz
---
 test/notmuch-test|  542 --
 test/t-basic.sh  |2 +-
 test/t0001-new.sh|  135 +++
 test/t0002-search.sh |   62 +
 test/t0003-reply.sh  |  114 +
 test/t0004-uuencoded-data.sh |   35 +++
 test/t0005-dump-restore.sh   |   44 
 7 files changed, 391 insertions(+), 543 deletions(-)
 delete mode 100755 test/notmuch-test
 create mode 100755 test/t0001-new.sh
 create mode 100755 test/t0002-search.sh
 create mode 100755 test/t0003-reply.sh
 create mode 100755 test/t0004-uuencoded-data.sh
 create mode 100755 test/t0005-dump-restore.sh

diff --git a/test/notmuch-test b/test/notmuch-test
deleted file mode 100755
index 2e5eb24..000
--- a/test/notmuch-test
+++ /dev/null
@@ -1,542 +0,0 @@
-#!/bin/bash
-set -e
-
-find_notmuch_binary ()
-{
-dir=$1
-
-while [ -n $dir ]; do
-   bin=$dir/notmuch
-   if [ -x $bin ]; then
-   echo $bin
-   return
-   fi
-   dir=$(dirname $dir)
-   if [ $dir = / ]; then
-   break
-   fi
-done
-
-echo notmuch
-}
-
-increment_mtime_amount=0
-increment_mtime ()
-{
-dir=$1
-
-increment_mtime_amount=$((increment_mtime_amount + 1))
-touch -d +${increment_mtime_amount} seconds $dir
-}
-
-# Generate a new message in the mail directory, with a unique message
-# ID and subject. The message is not added to the index.
-#
-# After this function returns, the filename of the generated message
-# is available as $gen_msg_filename and the message ID is available as
-# $gen_msg_id .
-#
-# This function supports named parameters with the bash syntax for
-# assigning a value to an associative array ([name]=value). The
-# supported parameters are:
-#
-#  [dir]=directory/of/choice
-#
-#  Generate the message in directory 'directory/of/choice' within
-#  the mail store. The directory will be created if necessary.
-#
-#  [body]=text
-#
-#  Text to use as the body of the email message
-#
-#  '[from]=Some User u...@example.com'
-#  '[to]=Some User u...@example.com'
-#  '[subject]=Subject of email message'
-#  '[date]=RFC 822 Date'
-#
-#  Values for email headers. If not provided, default values will
-#  be generated instead.
-#
-#  '[cc]=Some User u...@example.com'
-#  [reply-to]=some-address
-#  [in-reply-to]=message-id
-#
-#  Additional values for email headers. If these are not provided
-#  then the relevant headers will simply not appear in the
-#  message.
-gen_msg_cnt=0
-gen_msg_filename=
-gen_msg_id=
-generate_message ()
-{
-# This is our (bash-specific) magic for doing named parameters
-local -A template=($@)
-local additional_headers
-
-gen_msg_cnt=$((gen_msg_cnt + 1))
-gen_msg_name=msg-$(printf %03d $gen_msg_cnt)
-gen_msg_id=${gen_msg_na...@notmuch-test-suite
-
-if [ -z ${template[dir]} ]; then
-   gen_msg_filename=${MAIL_DIR}/$gen_msg_name
-else
-   gen_msg_filename=${MAIL_DIR}/${template[dir]}/$gen_msg_name
-   mkdir -p $(dirname $gen_msg_filename)
-fi
-
-if [ -z ${template[body]} ]; then
-   template[body]=This is just a test message at ${gen_msg_filename}
-fi
-
-if [ -z ${template[from]} ]; then
-   template[from]=Notmuch Test Suite test_su...@notmuchmail.org
-fi
-
-if [ -z ${template[to]} ]; then
-   template[to]=Notmuch Test Suite test_su...@notmuchmail.org
-fi
-
-if [ -z ${template[subject]} ]; then
-   template[subject]=Test message ${gen_msg_filename}
-fi
-
-if [ -z ${template[date]} ]; then
-   template[date]=Tue, 05 Jan 2010 15:43:57 -0800
-fi
-
-additional_headers=
-if [ ! -z ${template[reply-to]} ]; then
-   additional_headers=Reply-To: ${template[reply-to]}
-${additional_headers}
-fi
-
-if [ ! -z ${template[in-reply-to]} ]; then
-   

[notmuch] [PATCH 2/3] Update test framework for use with notmuch

2010-02-08 Thread Michal Sojka
This removes Git specific things from the test-lib.sh and adds helper
functions for notmuch taken from Carl's notmuch-test script. README is
also slightly modified to reflect the current state.

Signed-off-by: Michal Sojka sojk...@fel.cvut.cz
---
 test/Makefile   |1 +
 test/README |   12 +-
 test/t-basic.sh |  332 +++
 test/test-lib.sh|  320 +++--
 4 files changed, 204 insertions(+), 461 deletions(-)
 mode change 100644 = 100755 test/test-lib.sh

diff --git a/test/Makefile b/test/Makefile
index bd09390..e07c7fd 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -6,6 +6,7 @@
 -include ../config.mak
 
 #GIT_TEST_OPTS=--verbose --debug
+SHELL=/bin/bash
 SHELL_PATH ?= $(SHELL)
 TAR ?= $(TAR)
 RM ?= rm -f
diff --git a/test/README b/test/README
index dcd3ebb..0f5bddb 100644
--- a/test/README
+++ b/test/README
@@ -165,10 +165,10 @@ Writing Tests
 -
 
 The test script is written as a shell script.  It should start
-with the standard #!/bin/sh with copyright notices, and an
+with the standard #!/bin/bash with copyright notices, and an
 assignment to variable 'test_description', like this:
 
-   #!/bin/sh
+   #!/bin/bash
#
# Copyright (c) 2005 Junio C Hamano
#
@@ -192,9 +192,11 @@ This test harness library does the following things:
  - If the script is invoked with command line argument --help
(or -h), it shows the test_description and exits.
 
- - Creates an empty test directory with an empty .git/objects
-   database and chdir(2) into it.  This directory is 't/trash directory'
-   if you must know, but I do not think you care.
+ - Creates a test directory with default notmuch-config and empty mail
+   store. This directory is 't/trash directory.test-basename' (note
+   the space) if you must know, but I do not think you care. The path
+   to notmuch-config is exported in NOTMUCH_CONFIG environment
+   variable and mail store path is stored in MAIL_DIR variabl.
 
  - Defines standard test helper functions for your scripts to
use.  These functions are designed to make all scripts behave
diff --git a/test/t-basic.sh b/test/t-basic.sh
index f4ca4fc..cc2ca21 100755
--- a/test/t-basic.sh
+++ b/test/t-basic.sh
@@ -5,46 +5,39 @@
 
 test_description='Test the very basics part #1.
 
-The rest of the test suite does not check the basic operation of git
-plumbing commands to work very carefully.  Their job is to concentrate
-on tricky features that caused bugs in the past to detect regression.
-
-This test runs very basic features, like registering things in cache,
-writing tree, etc.
-
-Note that this test *deliberately* hard-codes many expected object
-IDs.  When object ID computation changes, like in the previous case of
-swapping compression and hashing order, the person who is making the
-modification *should* take notice and update the test vectors here.
+Tests the test framework itself.
 '
-
 
 # It appears that people try to run tests without building...
 
-../git /dev/null
-if test $? != 1
+if ! test -x ../notmuch
 then
-   echo 2 'You do not seem to have built git yet.'
+   echo 2 'You do not seem to have built notmuch yet.'
exit 1
 fi
 
 . ./test-lib.sh
 
 
-# git init has been done in an empty repository.
-# make sure it is empty.
+# Test mail store prepared in test-lib.sh
+
+test_expect_success \
+'test that mail store was created' \
+'test -d ${MAIL_DIR}'
+
 
-find .git/objects -type f -print should-be-empty
+find ${MAIL_DIR} -type f -print should-be-empty
 test_expect_success \
-'.git/objects should be empty after git init in an empty repo.' \
+'mail store should be empty' \
 'cmp -s /dev/null should-be-empty'
 
-# also it should have 2 subdirectories; no fan-out anymore, pack, and info.
-# 3 is counting objects itself
-find .git/objects -type d -print full-of-directories
 test_expect_success \
-'.git/objects should have 3 subdirectories.' \
-'test $(wc -l  full-of-directories) = 3'
+'NOTMUCH_CONFIG is set and points to an existing file' \
+'test -f ${NOTMUCH_CONFIG}'
+
+test_expect_success \
+'PATH is set to this repository' \
+'test `echo $PATH|cut -f1 -d:` = `dirname ${TEST_DIRECTORY}`'
 
 
 # Test harness
@@ -73,296 +66,5 @@ then
exit 1
 fi
 
-
-# Basics of the basics
-
-# updating a new file without --add should fail.
-test_expect_success 'git update-index without --add should fail adding.' '
-test_must_fail git update-index should-be-empty
-'
-
-# and with --add it should succeed, even if it is empty (it used to fail).
-test_expect_success \
-'git update-index with --add should succeed.' \
-

Re: [notmuch] strange behavior of indexing of and searching for strings containing '[]'

2010-02-08 Thread Jameson Rollins
On Fri, 5 Feb 2010 23:48:03 + (UTC), Olly Betts o...@survex.com wrote:
 On 2010-02-05, Jameson Rollins wrote:
  Hey, folks.  I've been noticing some strange behavior of notmuch search
  results for strings containing '[]'.  Here are some searches for some
  exact strings in messages subjects:
 
 The '[]' is a red herring.  Xapian's TermGenerator and QueryParser classes
 treat these two characters pretty much as if they were spaces.

Ah.  Thanks for the response, Olly.  This clears things up a lot.

  servo:~ 0$ notmuch search subject:'emacs paned UI'
 
 Note that the '' is quoting for the shell only here.  So Xapian sees:
 
 subject:emacs paned UI
 
 Assuming you are defaulting to an AND search, that's `emacs in the subject'
 AND `paned anywhere in the indexed text' AND `UI anywhere in the indexed 
 text'.

I don't think that this is exactly correct.  The quoting is interpreted
by the shell in order to construct a single string that is then passed
as an argument to the program.  Notmuch should then be seeing the single
string argument as the search parameter, and not breaking it up further.

Here's an example of what I mean:

servo:~/tmp/cdtemp.AYroUf 0$ cat parse 
#!/bin/bash
for arg; do echo $arg; done
servo:~/tmp/cdtemp.AYroUf 0$ ./parse subject:foo bar baz
subject:foo
bar
baz
servo:~/tmp/cdtemp.AYroUf 0$ ./parse subject:'foo bar' baz
subject:foo bar
baz
servo:~/tmp/cdtemp.AYroUf 0$ ./parse subject:foo bar baz
subject:foo bar
baz
servo:~/tmp/cdtemp.AYroUf 0$ 

As you can see in the last command, the argument subject'foo bar' is
passed as a single string by the shell, and should therefore be
interpreted as such by notmuch.

jamie.


pgpBHd8c9bE0Q.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] [PATCH] Switch from random to sequential thread identifiers.

2010-02-08 Thread Carl Worth
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up slightly less space in the notmuch search output (20 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).

---

I'm sending this patch to the mailing-list rather than pushing it
directly so that any authors of user interfaces can ensure that they are
ready for the length of thread identifiers in notmuch search output to
change.

I tested that the emacs client doesn't need any change in this regard.

And the change doesn't introduce any compatibility with an existing
database, so no rebuild required (hurrah!).

-Carl

 lib/database-private.h |7 +-
 lib/database.cc|   52 ---
 lib/message.cc |   46 --
 test/notmuch-test  |2 +-
 4 files changed, 55 insertions(+), 52 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 5891584..5bb6e86 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -27,14 +27,19 @@
 
 struct _notmuch_database {
 notmuch_bool_t exception_reported;
+
 char *path;
+
+notmuch_bool_t needs_upgrade;
 notmuch_database_mode_t mode;
 Xapian::Database *xapian_db;
+
+uint64_t last_thread_id;
+
 Xapian::QueryParser *query_parser;
 Xapian::TermGenerator *term_gen;
 Xapian::ValueRangeProcessor *value_range_processor;
 
-notmuch_bool_t needs_upgrade;
 };
 
 /* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index cce7847..65ff12e 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -533,6 +533,8 @@ notmuch_database_open (const char *path,
 notmuch-needs_upgrade = FALSE;
 notmuch-mode = mode;
 try {
+   string last_thread_id;
+
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
notmuch-xapian_db = new Xapian::WritableDatabase (xapian_path,
   
Xapian::DB_CREATE_OR_OPEN);
@@ -567,6 +569,20 @@ notmuch_database_open (const char *path,
 notmuch_path, version, NOTMUCH_DATABASE_VERSION);
}
}
+
+   last_thread_id = notmuch-xapian_db-get_metadata (last_thread_id);
+   if (last_thread_id.empty ()) {
+   notmuch-last_thread_id = 0;
+   } else {
+   const char *str;
+   char *end;
+
+   str = last_thread_id.c_str ();
+   notmuch-last_thread_id = strtoull (str, end, 10);
+   if (*end != '\0')
+   INTERNAL_ERROR (Malformed database last_thread_id: %s, str);
+   }
+
notmuch-query_parser = new Xapian::QueryParser;
notmuch-term_gen = new Xapian::TermGenerator;
notmuch-term_gen-set_stemmer (Xapian::Stem (english));
@@ -1278,14 +1294,38 @@ _notmuch_database_link_message_to_children 
(notmuch_database_t *notmuch,
 return ret;
 }
 
+static const char *
+_notmuch_database_generate_thread_id (notmuch_database_t *notmuch)
+{
+/* 20 bytes (+ terminator) for ASCII decimal representation of
+ * a 64-bit integer. */
+static char thread_id[21];
+Xapian::WritableDatabase *db;
+
+db = static_cast Xapian::WritableDatabase * (notmuch-xapian_db);
+
+notmuch-last_thread_id++;
+
+sprintf (thread_id, %020llu, notmuch-last_thread_id);
+
+db-set_metadata (last_thread_id, thread_id);
+
+return thread_id;
+}
+
 /* Given a (mostly empty) 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
  * We first look at 'message_file' and its link-relevant headers
  * (References and In-Reply-To) for message IDs. We also look in the
- * database for existing message that reference 'message'.
+ * database for existing message that reference 'message'. In either
+ * case, we will assign to the current message the first thread_id
+ * found (through either parent or child). We will also merge any
+ * existing, distinct threads where this message belongs to both,
+ * (which is not uncommon when mesages are processed out of order).
  *
- * The end result is to call _notmuch_message_ensure_thread_id which
+ * Finally, if not thread ID has been found through parent or child,
+ * we call _notmuch_message_generate_thread_id to generate a new
  * generates a new thread ID if the message doesn't connect to any
  * existing threads.
  */
@@ -1308,8 +1348,12 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
return status;
 
-if (thread_id == NULL)
-   _notmuch_message_ensure_thread_id (message);
+/* If not part of any existing thread, 

Re: [notmuch] [PATCH] Switch from random to sequential thread identifiers.

2010-02-08 Thread Carl Worth
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up half as much space in the notmuch search output (16 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).
---

Keith pointed out to me that there was obviously no benefit from
switching from hexadecimal to decimal here. So this second version of
the patch means 16-character identifiers rather than 20.

 lib/database-private.h |7 +-
 lib/database.cc|   52 ---
 lib/message.cc |   46 --
 test/notmuch-test  |2 +-
 4 files changed, 55 insertions(+), 52 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index 5891584..5bb6e86 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -27,14 +27,19 @@
 
 struct _notmuch_database {
 notmuch_bool_t exception_reported;
+
 char *path;
+
+notmuch_bool_t needs_upgrade;
 notmuch_database_mode_t mode;
 Xapian::Database *xapian_db;
+
+uint64_t last_thread_id;
+
 Xapian::QueryParser *query_parser;
 Xapian::TermGenerator *term_gen;
 Xapian::ValueRangeProcessor *value_range_processor;
 
-notmuch_bool_t needs_upgrade;
 };
 
 /* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index cce7847..8641321 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -533,6 +533,8 @@ notmuch_database_open (const char *path,
 notmuch-needs_upgrade = FALSE;
 notmuch-mode = mode;
 try {
+   string last_thread_id;
+
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
notmuch-xapian_db = new Xapian::WritableDatabase (xapian_path,
   
Xapian::DB_CREATE_OR_OPEN);
@@ -567,6 +569,20 @@ notmuch_database_open (const char *path,
 notmuch_path, version, NOTMUCH_DATABASE_VERSION);
}
}
+
+   last_thread_id = notmuch-xapian_db-get_metadata (last_thread_id);
+   if (last_thread_id.empty ()) {
+   notmuch-last_thread_id = 0;
+   } else {
+   const char *str;
+   char *end;
+
+   str = last_thread_id.c_str ();
+   notmuch-last_thread_id = strtoull (str, end, 16);
+   if (*end != '\0')
+   INTERNAL_ERROR (Malformed database last_thread_id: %s, str);
+   }
+
notmuch-query_parser = new Xapian::QueryParser;
notmuch-term_gen = new Xapian::TermGenerator;
notmuch-term_gen-set_stemmer (Xapian::Stem (english));
@@ -1278,14 +1294,38 @@ _notmuch_database_link_message_to_children 
(notmuch_database_t *notmuch,
 return ret;
 }
 
+static const char *
+_notmuch_database_generate_thread_id (notmuch_database_t *notmuch)
+{
+/* 16 bytes (+ terminator) for hexadecimal representation of
+ * a 64-bit integer. */
+static char thread_id[17];
+Xapian::WritableDatabase *db;
+
+db = static_cast Xapian::WritableDatabase * (notmuch-xapian_db);
+
+notmuch-last_thread_id++;
+
+sprintf (thread_id, %016llx, notmuch-last_thread_id);
+
+db-set_metadata (last_thread_id, thread_id);
+
+return thread_id;
+}
+
 /* Given a (mostly empty) 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
  * We first look at 'message_file' and its link-relevant headers
  * (References and In-Reply-To) for message IDs. We also look in the
- * database for existing message that reference 'message'.
+ * database for existing message that reference 'message'. In either
+ * case, we will assign to the current message the first thread_id
+ * found (through either parent or child). We will also merge any
+ * existing, distinct threads where this message belongs to both,
+ * (which is not uncommon when mesages are processed out of order).
  *
- * The end result is to call _notmuch_message_ensure_thread_id which
+ * Finally, if not thread ID has been found through parent or child,
+ * we call _notmuch_message_generate_thread_id to generate a new
  * generates a new thread ID if the message doesn't connect to any
  * existing threads.
  */
@@ -1308,8 +1348,12 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
return status;
 
-if (thread_id == NULL)
-   _notmuch_message_ensure_thread_id (message);
+/* If not part of any existing thread, generate a new thread ID. */
+if (thread_id == NULL) {
+   thread_id = _notmuch_database_generate_thread_id (notmuch);
+
+   _notmuch_message_add_term (message, thread, thread_id);
+}
 
 return 

Re: [notmuch] Update Patch Series. Only collapse part of citations.

2010-02-08 Thread Carl Worth
 On Fri, 25 Dec 2009 16:09:06 -0400, da...@tethera.net wrote:
  On Fri, 25 Dec 2009 09:47:02 +0800, Kan-Ru Chen ka...@kanru.info wrote:
   I like this idea, but this patch hides all citations larger than the
   threshold. I'd like to see limited lines of citations been displayed.
  
  Since you ask nicely, OK :).

Very nice work, David!

I've just used this for a few minutes here, and it's very nice. (The
previous code was hiding far too much of the citations and I found
myself having to open them all the time just to be able to read things.)

On Sun, 27 Dec 2009 00:22:21 +0800, Kan-Ru Chen ka...@kanru.info wrote:
 Works great here.

 Reviewed-by: Kan-Ru Chen ka...@kanru.info

And thanks for the review. I've now committed and pushed these changes,
(with the reviewed-by added and some trailing whitespace removed).

-Carl


pgpVqlL6rmyso.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] [PATCH 4/4] emacs: Use font-lock-comment-face to highlight citation button

2010-02-08 Thread Carl Worth
On Mon, 14 Dec 2009 13:41:35 +0800, Kan-Ru Chen ka...@kanru.info wrote:

[... no detailed commit message ...]

I've pushed this change now. It looked to me like the previous changes
in this series were made obsolete by the patches from David that I just
pushed. Let me know if I'm wrong about that and there's anything more I
should pick up from this series.

-Carl


pgpmdK0CWBLFE.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] [PATCH 3/3] Allow folders with no messages to be elided from list.

2010-02-08 Thread Carl Worth
On Sat, 26 Dec 2009 16:34:18 -0800, Keith Packard kei...@keithp.com wrote:
 This makes it easier to see folders with messages.
 Eliding empty folders is togged with the 'e' binding.
 
 Signed-off-by: Keith Packard kei...@keithp.com

Thanks for the patches, Keith!

These are all very useful, and I've pushed all three...

  lib/Makefile.local |1 +
  lib/notmuch.h  |   11 +++
  lib/query.cc   |   41 +
  notmuch-new.c  |1 +
  notmuch.el |   29 -
  5 files changed, 78 insertions(+), 5 deletions(-)

...except that on this last one I only pushed changes to notmuch.el. ;-)

-Carl


pgpjZZUyMWBcB.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] [PATCH] Goto-Address mode hides the body toggle button on messages of zero depth.

2010-02-08 Thread Carl Worth
On Wed, 30 Dec 2009 13:47:45 +0100, Pablo Oliveira pa...@sifflez.org wrote:
 Hello all,

Olá Pablo! Bem-vindo a notmuch!

 I heard about notmuch some weeks ago and I'm already liking it
 a lot. At first I was a little disapointed by the slow tagging, 
 but that was fixed by Kan-Ru's great patch for Xapian. I have
 started using it daily, so many thanks for this mail reader !

I'm delighted to hear you're having a good experience with notmuch. And
I apologize that I wasn't able to reply any sooner. (Perhaps I can
excuse myself somewhat since the original message arrived on my wedding
anniversary? No I'm still *really* late.)

 One of the things I find counter-intuitive is that on closed messages
 of zero depth, the toggle body button is hidden by the email address
 button, so when working with the keyboard, it seems one cannot open
 the message, without moving further on the header button.

Yes, this is really annoying. I'd like to see a proper fix for this.

 I propose above fix because it is very simple, but I do not like that
 we lose one level of indentation on all the threads (screen space
 is precious). Another solution would be to change the name-email 
 order in the header... Or maybe we should deactivate somehow, the
 Goto-Address mode on headers ? What do you think ?

For me, I'm getting the sender's name before their email address, so I
only run into this problem in the case where there is no name displayed,
but only an email address. Is that the same for you?

I don't want to remove the goto-address-mode on the header, because it
is very convenient (for those people who use a mouse) to be able to just
click on an email address to begin composing a new message to that
address.

I think I might be willing to have the email address deactivated for
keypresses if that were possible (so that pressing Enter anywhere on the
row would reliably toggle the message).

In the meantime, I agree that incrementing the depth universally doesn't
make sense. How about simply displaying a single ' ' character when
there's no full name available? I think that might solve the problem for
most people.

-Carl


pgp7ChbbqK8YY.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] notmuch.el and searching within citations

2010-02-08 Thread Carl Worth
On Sat, 9 Jan 2010 06:18:10 +1100, Bart Bunting b...@ursys.com.au wrote:
 Hi all,

Hi Bart, welcome to notmuch!

 If however the word 'grandfather' appears in a citation then an incremental
 search through the buffer won't find it.
 
 If the citation is expanded then things work as expected.
 
 I don't think this is very logical behavior as it stands, having to expand
 each citation to be able to search in it appears to sort of defeat the
 search.

I agree 100%.

There's good support in emacs for doing what we want here. There are
variables to control what happens when a search terminates (or
temporarily matches) invisible text. And one of the possible behaviors
is for the invisible region to be made visible (and that perhaps
temporarily if the search doesn't terminate in the invisible region).

So we just need to play with things a bit to turn that behavior on.

I've made a note in the TODO file so that I won't (permanently) forget
about this feature request. Of course, I'd be quite glad if anyone had
the time and inclination to investigate.

-Carl


pgpkY1E5mIJub.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch