[PATCH 2/2] configure: add debug flags by default.

[David Bremner]

This makes development (in particular the test suite) easier. Those
concerned about the extra diskspace can override the default or use
strip.
---
 configure  | 2 +-
 test/T000-basic.sh | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/configure b/configure
index 86ba2f7..331f29b 100755
--- a/configure
+++ b/configure
@@ -45,7 +45,7 @@ fi
 # environment variables)
 CC=${CC:-cc}
 CXX=${CXX:-c++}
-CFLAGS=${CFLAGS:--O2}
+CFLAGS=${CFLAGS:--g -O2}
 CPPFLAGS=${CPPFLAGS:-}
 CXXFLAGS=${CXXFLAGS:-\$(CFLAGS)}
 LDFLAGS=${LDFLAGS:-}
diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index bf08f3e..ef64245 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -92,7 +92,6 @@ test_expect_equal \
 "$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"

 test_begin_subtest 'notmuch is compiled with debugging symbols'
-test_subtest_known_broken
 readelf --sections $(which notmuch) | grep \.debug
 test_expect_equal 0 $?

-- 
2.1.0

[PATCH 1/2] test: check for debug symbols in notmuch

[David Bremner]

In the future, tests may rely on debug symbols being present in
notmuch, so we plan to switch the default flags.

The main purpose of this test is to help explain the perhaps
mysterious failures of other tests which rely on symbols being
present.
---
 test/T000-basic.sh | 5 +
 1 file changed, 5 insertions(+)

diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index ebbb6d2..bf08f3e 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -91,4 +91,9 @@ test_expect_equal \
 "$(dirname ${TEST_DIRECTORY})" \
 "$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"

+test_begin_subtest 'notmuch is compiled with debugging symbols'
+test_subtest_known_broken
+readelf --sections $(which notmuch) | grep \.debug
+test_expect_equal 0 $?
+
 test_done
-- 
2.1.0

[PATCH] debian: append elinks to notmuch-vim dependencies

[Sergei Shilovsky]

notmuch-vim uses elinks to convert HTML messages to text.
---
 debian/control | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/debian/control b/debian/control
index 6298deb..146dfea 100644
--- a/debian/control
+++ b/debian/control
@@ -128,7 +128,7 @@ Architecture: all
 Section: mail
 Breaks: notmuch (<<0.6~254~)
 Replaces: notmuch (<<0.6~254~)
-Depends: ${misc:Depends}, notmuch, vim-addon-manager, vim-ruby, ruby-notmuch
+Depends: ${misc:Depends}, notmuch, vim-addon-manager, vim-ruby, ruby-notmuch, 
elinks
 Recommends: ruby-mail
 Description: thread-based email index, search and tagging (vim interface)
  Notmuch is a system for indexing, searching, reading, and tagging
-- 
2.1.0

[PATCH v2] VIM: Use notmuch CLI for config

[Sergei Shilovsky]

The patch works fine for me.

Ian, though I could not test whether result='' matters or not, cause
'notmuch get' looks to always return a string.

It also fixes id:CAHc2pO19AZabAiJhdfQR1rL5t0GufLAq5cKb_7w-Z8y+JXT5+g at 
mail.gmail.com

On Fri, Oct 3, 2014 at 10:54 AM, David Bremner  wrote:
> Ian Main  writes:
>
>> This patch switches from reading .notmuch-config directly to using
>> the CLI the same way that emacs does it.  It actually uses less code
>> and is probably less error prone.
>>
>> Ian
>
> The general approach seems sane; it seems quite brittle to read the
> config file directly. I notice there is not really any error handling;
> OTOH, as far as I can read Ruby, there is not any in the previous
> version either.  Technically, this does add a dependency of the vim
> client on the CLI that did not exist before. Personally I don't find
> this onerous (even notmuch-vim users need "notmuch new", except in
> rather unusual circumstances.).
>
> I'd like feedback/testing from actual vim interface users before
> merging.
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch



-- 
? ?,
?? ?
Sergei Shilovsky

[PATCH v5] nmbug: Translate to Python

[Tomi Ollila]

On Fri, Oct 03 2014, David Bremner  wrote:

> "W. Trevor King"  writes:
>
>> This allows us to capture stdout and stderr separately, and do other
>> explicit subprocess manipulation without resorting to external
>> packages.  It should be compatible with Python 2.6 and later
>> (including the 3.x series), although with 2.6 you'll need the external
>> argparse package.
>
> As reported on IRC, this version has been working ok for me for a week
> or so. I haven't marked it ready to push because iirc, the python 2.6
> support doesn't really work and is doomed to be removed.

Maybe just amend to "It is compatible with Python 2.7 and later...", with
dropping the comment about external argparse package?

Currently also tests don't work with Python 2.6 -- I have a patch that
would "fix" this but probably it is just noise at this time (and I'd still 
need to do the harderst part -- write the commit message).

> d

Tomi

[PATCH] test: Port atomicity test to Python

[Austin Clements]

On Fri, 03 Oct 2014, Austin Clements  wrote:
> Previously, this was implemented using a horrible GDB script (because
> there is no such thing as a non-horrible GDB script).  This GDB script
> often broke with newer versions of GDB for mysterious reasons.  Port
> the test script to GDB's Python API, which makes the code much cleaner
> and, hopefully, more stable.
> ---
>
> Hi Amadeusz.  Does this patch fix the problem for you?  I don't have
> GDB 7.8, so I can't test against it, but the Python interface is less
> fragile than the GDB scripting language.  (Even if this doesn't fix
> your problem, I think we should switch to the Python interface.)

Oops.  I meant this to be a reply to
id:87sijovhuf.fsf at freja.aidecoe.name.  Sorry for any confusion.

[PATCH] test: Port atomicity test to Python

[Austin Clements]

Previously, this was implemented using a horrible GDB script (because
there is no such thing as a non-horrible GDB script).  This GDB script
often broke with newer versions of GDB for mysterious reasons.  Port
the test script to GDB's Python API, which makes the code much cleaner
and, hopefully, more stable.
---

Hi Amadeusz.  Does this patch fix the problem for you?  I don't have
GDB 7.8, so I can't test against it, but the Python interface is less
fragile than the GDB scripting language.  (Even if this doesn't fix
your problem, I think we should switch to the Python interface.)

 test/T380-atomicity.sh |  2 +-
 test/atomicity.gdb | 54 --
 test/atomicity.py  | 71 ++
 3 files changed, 72 insertions(+), 55 deletions(-)
 delete mode 100644 test/atomicity.gdb
 create mode 100644 test/atomicity.py

diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh
index 2daef90..ee1e2f4 100755
--- a/test/T380-atomicity.sh
+++ b/test/T380-atomicity.sh
@@ -64,7 +64,7 @@ if test_require_external_prereq gdb; then
 # -tty /dev/null works around a conflict between the 'timeout' wrapper
 # and gdb's attempt to control the TTY.
 export MAIL_DIR
-gdb -tty /dev/null -batch -x $TEST_DIRECTORY/atomicity.gdb notmuch 
1>gdb.out 2>&1
+gdb -tty /dev/null -batch -x $TEST_DIRECTORY/atomicity.py notmuch 
1>gdb.out 2>&1

 # Get the final, golden output
 notmuch search '*' > expected
diff --git a/test/atomicity.gdb b/test/atomicity.gdb
deleted file mode 100644
index 15adb16..000
--- a/test/atomicity.gdb
+++ /dev/null
@@ -1,54 +0,0 @@
-# This gdb script runs notmuch new and simulates killing and
-# restarting notmuch new after every Xapian commit.  To simulate this
-# more efficiently, this script runs notmuch new and, immediately
-# after every Xapian commit, it *pauses* the running notmuch new,
-# copies the entire database and maildir to a snapshot directory, and
-# executes a full notmuch new on that snapshot, comparing the final
-# results with the expected output.  It can then resume the paused
-# notmuch new, which is still running on the original maildir, and
-# repeat this process.
-
-set args new
-
-# Make Xapian commit after every operation instead of batching
-set environment XAPIAN_FLUSH_THRESHOLD = 1
-
-# gdb can't keep track of a simple integer.  This is me weeping.
-shell echo 0 > outcount
-
-shell touch inodes
-
-# work around apparent issue with lazy library loading on some
-# platforms
-set breakpoint pending on
-
-break rename
-commands
-# As an optimization, only consider snapshots after a Xapian commit.
-# Xapian overwrites record.base? as the last step in the commit.
-shell echo > gdbcmd
-shell stat -c %i $MAIL_DIR/.notmuch/xapian/record.base* > inodes.new
-shell if cmp inodes inodes.new; then echo cont > gdbcmd; fi
-shell mv inodes.new inodes
-source gdbcmd
-
-# Save a backtrace in case the test does fail
-set logging file backtrace
-set logging on
-backtrace
-set logging off
-shell mv backtrace backtrace.`cat outcount`
-
-# Snapshot the database
-shell rm -r $MAIL_DIR.snap/.notmuch
-shell cp -r $MAIL_DIR/.notmuch $MAIL_DIR.snap/.notmuch
-# Restore the mtime of $MAIL_DIR.snap, which we just changed
-shell touch -r $MAIL_DIR $MAIL_DIR.snap
-# Run notmuch new to completion on the snapshot
-shell NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap XAPIAN_FLUSH_THRESHOLD=1000 
notmuch new > /dev/null
-shell NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap notmuch search '*' > search.`cat 
outcount` 2>&1
-shell echo $(expr $(cat outcount) + 1) > outcount
-cont
-end
-
-run
diff --git a/test/atomicity.py b/test/atomicity.py
new file mode 100644
index 000..01a4205
--- /dev/null
+++ b/test/atomicity.py
@@ -0,0 +1,71 @@
+# This gdb Python script runs notmuch new and simulates killing and
+# restarting notmuch new after every Xapian commit.  To simulate this
+# more efficiently, this script runs notmuch new and, immediately
+# after every Xapian commit, it *pauses* the running notmuch new,
+# copies the entire database and maildir to a snapshot directory, and
+# executes a full notmuch new on that snapshot, comparing the final
+# results with the expected output.  It can then resume the paused
+# notmuch new, which is still running on the original maildir, and
+# repeat this process.
+
+import gdb
+import os
+import glob
+import shutil
+import subprocess
+
+gdb.execute('set args new')
+
+# Make Xapian commit after every operation instead of batching
+gdb.execute('set environment XAPIAN_FLUSH_THRESHOLD = 1')
+
+maildir = os.environ['MAIL_DIR']
+
+# Trap calls to rename, which happens just before Xapian commits
+class RenameBreakpoint(gdb.Breakpoint):
+def __init__(self, *args, **kwargs):
+super(RenameBreakpoint, self).__init__(*args, **kwargs)
+self.last_inodes = {}
+self.n = 0
+
+def stop(self):
+# As an optimization, only consider snapshots after a Xapian
+# has really 

[PATCH v2] VIM: Use notmuch CLI for config

[Ian Main]

David Bremner wrote:
> Ian Main  writes:
> 
> > This patch switches from reading .notmuch-config directly to using
> > the CLI the same way that emacs does it.  It actually uses less code
> > and is probably less error prone.
> >
> > Ian
> 
> The general approach seems sane; it seems quite brittle to read the
> config file directly. I notice there is not really any error handling;
> OTOH, as far as I can read Ruby, there is not any in the previous
> version either.  Technically, this does add a dependency of the vim
> client on the CLI that did not exist before. Personally I don't find
> this onerous (even notmuch-vim users need "notmuch new", except in
> rather unusual circumstances.).
> 
> I'd like feedback/testing from actual vim interface users before
> merging.

I am actually just following suit on what was already being done.
The Vim client was already calling out to notmuch CLI for other things,
eg:

   system "notmuch show --format=mbox id:#{m.message_id} > #{mbox} && #{cmd}"

Is used to save the email for display in another program.  Also with no
error checking.  I think basically we are relying on rubys exception handling
to display errors to the user.. not the best idea but it is functional.

I could add a check for 'notmuch' binary.. especially there because loading
the config is the first thing that is done on startup.

Ian

[PATCH v6 2/2] nmbug: Add a 'help' command for folks who don't like --help

[W. Trevor King]

The 'if args.func == help' block at the end avoids:

AttributeError: 'functools.partial' object has no attribute '__code__'
---
 devel/nmbug/nmbug | 31 ++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug
index 9402ead..932ec12 100755
--- a/devel/nmbug/nmbug
+++ b/devel/nmbug/nmbug
@@ -32,6 +32,7 @@ from __future__ import unicode_literals

 import codecs as _codecs
 import collections as _collections
+import functools as _functools
 import inspect as _inspect
 import locale as _locale
 import logging as _logging
@@ -677,6 +678,24 @@ def _unpack_diff_lines(stream):
 yield (id, tag)


+def _help(parser, command=None):
+"""
+Show help for an nmbug command.
+
+Because some folks prefer:
+
+  $ nmbug help COMMAND
+
+to
+
+  $ nmbug COMMAND --help
+"""
+if command:
+parser.parse_args([command, '--help'])
+else:
+parser.parse_args(['--help'])
+
+
 if __name__ == '__main__':
 import argparse

@@ -692,6 +711,8 @@ if __name__ == '__main__':
 help='Log verbosity.  Defaults to {!r}.'.format(
 _logging.getLevelName(_LOG.level).lower()))

+help = _functools.partial(_help, parser=parser)
+help.__doc__ = _help.__doc__
 subparsers = parser.add_subparsers(
 title='commands',
 description=(
@@ -703,6 +724,7 @@ if __name__ == '__main__':
 'clone',
 'commit',
 'fetch',
+'help',
 'log',
 'merge',
 'pull',
@@ -746,6 +768,10 @@ if __name__ == '__main__':
 'Override the default configured in branch..remote '
 'to fetch from a particular remote repository (e.g. '
 "'origin')."))
+elif command == 'help':
+subparser.add_argument(
+'command', metavar='COMMAND', nargs='?',
+help='The command to show help for.')
 elif command == 'log':
 subparser.add_argument(
 'args', metavar='ARG', nargs='*',
@@ -796,7 +822,10 @@ if __name__ == '__main__':
 parser.print_usage()
 _sys.exit(1)

-(arg_names, varargs, varkw) = _inspect.getargs(args.func.__code__)
+if args.func == help:
+arg_names = ['command']
+else:
+(arg_names, varargs, varkw) = _inspect.getargs(args.func.__code__)
 kwargs = {key: getattr(args, key) for key in arg_names if key in args}
 try:
 args.func(**kwargs)
-- 
2.1.0.60.g85f0837

[PATCH v6 1/2] nmbug: Translate to Python

[W. Trevor King]

This allows us to capture stdout and stderr separately, and do other
explicit subprocess manipulation without resorting to external
packages.  It should be compatible with Python 2.7 and later
(including the 3.x series).

Most of the user-facing interface is the same, but there are a few
changes, where reproducing the original interface was too difficult or
I saw a change to make the underlying Git UI accessible:

* 'nmbug help' has been split between the general 'nmbug --help' and
  the command-specific 'nmbug COMMAND --help'.

* Commands are no longer split into "most common", "other useful", and
  "less common" sets.  If we need something like this, I'd prefer
  workflow examples highlighting common commands in the module
  docstring (available with 'nmbug --help').

* 'nmbug commit' now only uses a single argument for the optional
  commit-message text.  I wanted to expose more of the underlying 'git
  commit' UI, since I personally like to write my commit messages in
  an editor with the notes added by 'git commit -v' to jog my memory.
  Unfortunately, we're using 'git commit-tree' instead of 'git
  commit', and commit-tree is too low-level for editor-launching.  I'd
  be interested in rewriting commit() to use 'git commit', but that
  seemed like it was outside the scope of this rewrite.  So I'm not
  supporting all of Git's commit syntax in this patch, but I can at
  least match 'git commit -m MESSAGE' in requiring command-line commit
  messages to be a single argument.

* The default repository for 'nmbug push' and 'nmbug fetch' is now the
  current branch's upstream (branch..remote) instead of
  'origin'.  When we have to, we extract this remote by hand, but
  where possible we just call the Git command without a repository
  argument, and leave it to Git to figure out the default.

* 'nmbug push' accepts multiple refspecs if you want to explicitly
  specify what to push.  Otherwise, the refspec(s) pushed depend on
  push.default.  The Perl version hardcoded 'master' as the pushed
  refspec.

* 'nmbug pull' defaults to the current branch's upstream
  (branch..remote and branch..merge) instead of hardcoding
  'origin' and 'master'.  It also supports multiple refspecs if for
  some crazy reason you need an octopus merge (but mostly to avoid
  breaking consistency with 'git pull').

* 'nmbug log' now execs 'git log', as there's no need to keep the
  Python process around once we've launched Git there.

* 'nmbug status' now catches stderr, and doesn't print errors like:

No upstream configured for branch 'master'

  The Perl implementation had just learned to avoid crashing on that
  case, but wasn't yet catching the dying subprocess's stderr.

* 'nmbug archive' now accepts positional arguments for the tree-ish
  and additional 'git archive' options.  For example, you can run:

$ nmbug archive HEAD -- --format tar.gz

  I wish I could have preserved the argument order from 'git archive'
  (with the tree-ish at the end), but I'm not sure how to make
  argparse accept arbitrary possitional arguments (some of which take
  arguments).  Flipping the order to put the tree-ish first seemed
  easiest.

* 'nmbug merge' and 'pull' no longer checkout HEAD before running
  their command, because blindly clobbering the index seems overly
  risky.

* In order to avoid creating a dirty index, 'nmbug commit' now uses
  the default index (instead of nmbug.index) for composing the commit.
  That way the index matches the committed tree.  To avoid leaving a
  broken index after a failed commit, I've wrapped the whole thing in
  a try/except block that resets the index to match the pre-commit
  treeish on errors.  That means that 'nmbug commit' will ignore
  anything you've cached in the index via direct Git calls, and you'll
  either end up with an index matching your notmuch tags and the new
  HEAD (after a successful commit) or an index matching the original
  HEAD (after a failed commit).
---
 devel/nmbug/nmbug | 1515 -
 1 file changed, 807 insertions(+), 708 deletions(-)

diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug
index 998ee6b..9402ead 100755
--- a/devel/nmbug/nmbug
+++ b/devel/nmbug/nmbug
@@ -1,708 +1,807 @@
-#!/usr/bin/env perl
-# Copyright (c) 2011 David Bremner
-# License: same as notmuch
-
-use strict;
-use warnings;
-use File::Temp qw(tempdir);
-use Pod::Usage;
-
-no encoding;
-
-my $NMBGIT = $ENV{NMBGIT} || $ENV{HOME}.'/.nmbug';
-
-$NMBGIT .= '/.git' if (-d $NMBGIT.'/.git');
-
-my $TAGPREFIX = defined($ENV{NMBPREFIX}) ? $ENV{NMBPREFIX} : 'notmuch::';
-
-# for encoding
-
-my $ESCAPE_CHAR =  '%';
-my $NO_ESCAPE =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'.
-   '0123456789+-_@=.:,';
-my $MUST_ENCODE =  qr{[^\Q$NO_ESCAPE\E]};
-my $ESCAPED_RX =   qr{$ESCAPE_CHAR([A-Fa-f0-9]{2})};
-
-my %command = (
-archive=> \_archive,
-checkout   => \_checkout,
-

[PATCH v6 0/2] nmbug: Translate to Python

[W. Trevor King]

Hopefully the last round :).  Changes since v5 [1]:

* Dropped 2.6 compatibility claim from the first patch's commit
  message, since Tomi pointed out that at least the '{}'.format()
  syntax is just since 2.7.
* Added a new 'help' command as a separate patch at David's request.

[1]: id:d44bb6ad59ee0a30ac4a8d2e9fe50e3b98d1c408.1411572592.git.wking at 
tremily.us
 http://thread.gmane.org/gmane.mail.notmuch.general/19108

W. Trevor King (2):
  nmbug: Translate to Python
  nmbug: Add a 'help' command for folks who don't like --help

 devel/nmbug/nmbug | 1544 +
 1 file changed, 836 insertions(+), 708 deletions(-)

-- 
2.1.0.60.g85f0837

[PATCH v5] nmbug: Translate to Python

[David Bremner]

"W. Trevor King"  writes:

> This allows us to capture stdout and stderr separately, and do other
> explicit subprocess manipulation without resorting to external
> packages.  It should be compatible with Python 2.6 and later
> (including the 3.x series), although with 2.6 you'll need the external
> argparse package.

As reported on IRC, this version has been working ok for me for a week
or so. I haven't marked it ready to push because iirc, the python 2.6
support doesn't really work and is doomed to be removed.

d

[PATCH v4] lib: Simplify close and codify aborting atomic section

[David Bremner]

Austin Clements  writes:

> This patch simplifies notmuch_database_close to explicitly abort any
> outstanding transaction and then just call Database::close.  This
> works for both read-only and read/write databases, takes care of
> committing changes, unifies the exception handling path, and codifies
> aborting outstanding transactions.  This is currently the only way to
> abort an atomic section (and may remain so, since it would be
> difficult to roll back things we may have cached from rolled-back
> modifications).

pushed

d

[PATCH] doc: Emacs manual improvements and expansions

[Austin Clements]

Fix several typos, improve general wording and flow, and add some
information on notmuch-jump.
---

Sorry I didn't tease these changes apart.  I went through it linearly
and it didn't seem worth the trouble of separating them after the fact.

 doc/notmuch-emacs.rst | 50 +-
 1 file changed, 37 insertions(+), 13 deletions(-)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 09579bf..6f2f61e 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -6,17 +6,17 @@ About this Manual
 =

 This manual covers only the Emacs interface to Notmuch. For information
-on the command line interface, see See section ?Description? in Notmuch
-Manual Pager. To save typing, we will sometimes use *notmuch* in this
-manual to refer to the Emacs interface to Notmuch. If the distinction
-should every be important, we?ll refer to the Emacs interface as
+on the command line interface, see section ?Description? in the Notmuch
+Manual Pages. To save typing, we will sometimes use *notmuch* in this
+manual to refer to the Emacs interface to Notmuch. When this distinction
+is important, we?ll refer to the Emacs interface as
 *notmuch-emacs*.

 Notmuch-emacs is highly customizable via the the Emacs customization
 framework (or just by setting the appropriate variables). We try to
 point out relevant variables in this manual, but in order to avoid
-duplication of information, but you can usually find the most detailed
-description in the variables docstring.
+duplication of information, you can usually find the most detailed
+description in the variables' docstring.

 notmuch-hello
 =
@@ -89,15 +89,19 @@ notmuch-hello key bindings
 Saved Searches
 --

-Notmuch replaces the static assignment of messages with the more dynamic
-notion of searching. Notmuch-hello presents the user with a customizable
-set of saved searches. The initial defaults are ``tag:inbox`` and
-``tag:unread``, but you can customize the following variables
+Since notmuch is entirely search-based, it's often useful to organize
+mail around common searches.  To facilitate this, the first section of
+notmuch-hello presents a customizable set of saved searches.  Saved
+searches can also be accessed from anywhere in notmuch by pressing
+``j`` to access :ref:`notmuch-jump`.
+
+The saved searches default to various common searches such as
+``tag:inbox`` to access the inbox and ``tag:unread`` to access all
+unread mail, but there are several options for customization:

 :index:`notmuch-saved-searches`
-A list of cons pairs, the first being the name to display, the
-second being a query string for Notmuch. See section ?Description?
-in Notmuch Query Syntax.
+The list of saved searches, including names, queries, and
+additional per-query options.

 :index:`notmuch-saved-searches-sort-function`
 This variable controls how saved searches should be sorted. A value
@@ -179,6 +183,26 @@ notmuch-show
 notmuch-tree
 

+Global key bindings
+===
+
+Several features are accessible from anywhere in notmuch through the
+following key bindings:
+
+``j``
+Jump to saved searches using :ref:`notmuch-jump`.
+
+notmuch-jump
+
+
+Saved searches configured through :ref:`notmuch-saved-searches` can
+include a "shortcut key" that's accessible through notmuch-jump.
+Pressing ``j`` anywhere in notmuch followed by the configured shortcut
+key of a saved search will immediately jump to that saved search.  For
+example, in the default configuration ``j i`` jumps immediately to the
+inbox search.  When you press ``j``, notmuch-jump shows the saved
+searches and their shortcut keys in the mini-buffer.
+
 Configuration
 =

-- 
2.1.0

[PATCH] NEWS: News for notmuch-jump

[Austin Clements]

---
 NEWS | 15 +++
 1 file changed, 15 insertions(+)

diff --git a/NEWS b/NEWS
index 2d85090..fa57e5d 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,21 @@
 Notmuch 0.19 (UNRELEASED)
 =

+Emacs Interface
+---
+
+Use the `j` key to access saved searches from anywhere in notmuch
+
+  `j` is now globally bound to `notmuch-jump`, which provides fast,
+  interactive keyboard shortcuts to saved searches.  For example,
+  with the default saved searches `j i` from anywhere in notmuch will
+  bring up the inbox.
+
+Expanded default saved search settings
+
+  The default saved searches now include several more common searches,
+  as well as shortcut keys for `notmuch-jump`.
+
 Library changes
 ---

-- 
2.1.0

[PATCH 11/11] lib: Remove unnecessary thread linking steps when using ghost messages

[Austin Clements]

From: Austin Clements 

Previously, it was necessary to link new messages to children to work
around some (though not all) problems with the old metadata-based
approach to stored thread IDs.  With ghost messages, this is no longer
necessary, so don't bother with child linking when ghost messages are
in use.
---
 lib/database.cc | 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index ff6a7f6..4655f59 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -2167,10 +2167,23 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
goto DONE;

-status = _notmuch_database_link_message_to_children (notmuch, message,
-_id);
-if (status)
-   goto DONE;
+if (! (notmuch->features & NOTMUCH_FEATURE_GHOSTS)) {
+   /* In general, it shouldn't be necessary to link children,
+* since the earlier indexing of those children will have
+* stored a thread ID for the missing parent.  However, prior
+* to ghost messages, these stored thread IDs were NOT
+* rewritten during thread merging (and there was no
+* performant way to do so), so if indexed children were
+* pulled into a different thread ID by a merge, it was
+* necessary to pull them *back* into the stored thread ID of
+* the parent.  With ghost messages, we just rewrite the
+* stored thread IDs during merging, so this workaround isn't
+* necessary. */
+   status = _notmuch_database_link_message_to_children (notmuch, message,
+_id);
+   if (status)
+   goto DONE;
+}

 /* If not part of any existing thread, generate a new thread ID. */
 if (thread_id == NULL) {
-- 
2.1.0

[PATCH 10/11] test: Test upgrade to ghost messages feature

[Austin Clements]

---
 test/T530-upgrade.sh | 21 +
 1 file changed, 21 insertions(+)

diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
index c4c4ac8..6b42a69 100755
--- a/test/T530-upgrade.sh
+++ b/test/T530-upgrade.sh
@@ -116,4 +116,25 @@ MAIL_DIR/bar/new/21:2,
 MAIL_DIR/bar/new/22:2,
 MAIL_DIR/cur/51:2,"

+# Ghost messages are difficult to test since they're nearly invisible.
+# However, if the upgrade works correctly, the ghost message should
+# retain the right thread ID even if all of the original messages in
+# the thread are deleted.  That's what we test.  This won't detect if
+# the upgrade just plain didn't happen, but it should detect if
+# something went wrong.
+test_begin_subtest "ghost message retains thread ID"
+# Upgrade database
+notmuch new
+# Get thread ID of real message
+thread=$(notmuch search --output=threads id:4EFC743A.3060609 at april.org)
+# Delete all real messages in that thread
+rm $(notmuch search --output=files $thread)
+notmuch new
+# "Deliver" ghost message
+add_message '[subject]=Ghost' '[id]=4EFC3931.6030007 at april.org'
+# If the ghost upgrade worked, the new message should be attached to
+# the existing thread ID.
+nthread=$(notmuch search --output=threads id:4EFC3931.6030007 at april.org)
+test_expect_equal "$thread" "$nthread"
+
 test_done
-- 
2.1.0

[PATCH 09/11] lib: Enable ghost messages feature

[Austin Clements]

From: Austin Clements 

This fixes the broken thread order test.
---
 lib/database-private.h| 2 +-
 test/T260-thread-order.sh | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index e2e4bc8..15e03cc 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -166,7 +166,7 @@ struct _notmuch_database {
  * databases will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
 (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
- NOTMUCH_FEATURE_BOOL_FOLDER)
+ NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)

 /* Return the list of terms from the given iterator matching a prefix.
  * The prefix will be stripped from the strings in the returned list.
diff --git a/test/T260-thread-order.sh b/test/T260-thread-order.sh
index b435d79..99f5833 100755
--- a/test/T260-thread-order.sh
+++ b/test/T260-thread-order.sh
@@ -30,7 +30,6 @@ expected=$(for ((i = 0; i < $nthreads; i++)); do
 test_expect_equal "$output" "$expected"

 test_begin_subtest "Messages with all parents get linked in all delivery 
orders"
-test_subtest_known_broken
 # Here we do the same thing as the previous test, but each message
 # references all of its parents.  Since every message references the
 # root of the thread, each thread should always be fully joined.  This
-- 
2.1.0

[PATCH 08/11] lib: Implement upgrade to ghost messages feature

[Austin Clements]

From: Austin Clements 

Somehow this is the first upgrade pass that actually does *any* error
checking, so this also adds the bit of necessary infrastructure to
handle that.
---
 lib/database.cc | 64 +++--
 1 file changed, 62 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index fdcc526..ff6a7f6 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1231,6 +1231,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 notmuch_bool_t timer_is_active = FALSE;
 enum _notmuch_features target_features, new_features;
 notmuch_status_t status;
+notmuch_private_status_t private_status;
 unsigned int count = 0, total = 0;

 status = _notmuch_database_ensure_writable (notmuch);
@@ -1275,6 +1276,11 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
for (t = db->allterms_begin ("XTIMESTAMP"); t != t_end; t++)
++total;
 }
+if (new_features & NOTMUCH_FEATURE_GHOSTS) {
+   t_end = db->metadata_keys_end ("thread_id_");
+   for (t = db->metadata_keys_begin ("thread_id_"); t != t_end; ++t)
+   ++total;
+}

 /* Perform the upgrade in a transaction. */
 db->begin_transaction (true);
@@ -1378,10 +1384,64 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
}
 }

+/* Perform metadata upgrades. */
+
+/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
+ * messages were stored as database metadata. Change these to
+ * ghost messages.
+ */
+if (new_features & NOTMUCH_FEATURE_GHOSTS) {
+   notmuch_message_t *message;
+   std::string message_id, thread_id;
+
+   t_end = db->metadata_keys_end (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+   for (t = db->metadata_keys_begin (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+t != t_end; ++t) {
+   if (do_progress_notify) {
+   progress_notify (closure, (double) count / total);
+   do_progress_notify = 0;
+   }
+
+   message_id = (*t).substr (
+   strlen (NOTMUCH_METADATA_THREAD_ID_PREFIX));
+   thread_id = db->get_metadata (*t);
+
+   /* Create ghost message */
+   message = _notmuch_message_create_for_message_id (
+   notmuch, message_id.c_str (), _status);
+   if (private_status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Document already exists; ignore the stored thread ID */
+   } else if (private_status ==
+  NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   private_status = _notmuch_message_initialize_ghost (
+   message, thread_id.c_str ());
+   if (! private_status)
+   _notmuch_message_sync (message);
+   }
+
+   if (private_status) {
+   fprintf (stderr,
+"Upgrade failed while creating ghost messages.\n");
+   status = COERCE_STATUS (private_status, "Unexpected status from 
_notmuch_message_initialize_ghost");
+   goto DONE;
+   }
+
+   /* Clear saved metadata thread ID */
+   db->set_metadata (*t, "");
+
+   ++count;
+   }
+}
+
+status = NOTMUCH_STATUS_SUCCESS;
 db->set_metadata ("features", _print_features (local, notmuch->features));
 db->set_metadata ("version", STRINGIFY (NOTMUCH_DATABASE_VERSION));

-db->commit_transaction ();
+ DONE:
+if (status == NOTMUCH_STATUS_SUCCESS)
+   db->commit_transaction ();
+else
+   db->cancel_transaction ();

 if (timer_is_active) {
/* Now stop the timer. */
@@ -1397,7 +1457,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 }

 talloc_free (local);
-return NOTMUCH_STATUS_SUCCESS;
+return status;
 }

 notmuch_status_t
-- 
2.1.0

[PATCH 07/11] lib: Implement ghost-based thread linking

[Austin Clements]

From: Austin Clements 

This updates the thread linking code to use ghost messages instead of
user metadata to link messages into threads.

In contrast with the old approach, this is actually correct.
Previously, thread merging updated only the thread IDs of message
documents, not thread IDs stored in user metadata.  As originally
diagnosed by Mark Walters [1] and as demonstrated by the broken
T260-thread-order test, this can cause notmuch to fail to link
messages even though they're in the same thread.  In principle the old
approach could have been fixed by updating the user metadata thread
IDs as well, but these are not indexed and hence this would have
required a full scan of all stored thread IDs.  Ghost messages solve
this problem naturally by reusing the exact same thread ID and message
ID representation and indexing as regular messages.

Furthermore, thanks to this greater symmetry, ghost messages are also
algorithmically simpler.  We continue to support the old user metadata
format, so this patch can't delete any code, but when we do remove
support for the old format, several functions can simply be deleted.

[1] id:8738h7kv2q.fsf at qmul.ac.uk
---
 lib/database.cc | 86 +
 1 file changed, 75 insertions(+), 11 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index c641bcd..fdcc526 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1752,6 +1752,12 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
message_id);
 }

+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret);
+
 /* Find the thread ID to which the message with 'message_id' belongs.
  *
  * Note: 'thread_id_ret' must not be NULL!
@@ -1760,9 +1766,9 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
  *
  * Note: If there is no message in the database with the given
  * 'message_id' then a new thread_id will be allocated for this
- * message and stored in the database metadata, (where this same
+ * message ID and stored in the database metadata so that the
  * thread ID can be looked up if the message is added to the database
- * later).
+ * later.
  */
 static notmuch_status_t
 _resolve_message_id_to_thread_id (notmuch_database_t *notmuch,
@@ -1770,6 +1776,49 @@ _resolve_message_id_to_thread_id (notmuch_database_t 
*notmuch,
  const char *message_id,
  const char **thread_id_ret)
 {
+notmuch_private_status_t status;
+notmuch_message_t *message;
+
+if (! (notmuch->features & NOTMUCH_FEATURE_GHOSTS))
+   return _resolve_message_id_to_thread_id_old (notmuch, ctx, message_id,
+thread_id_ret);
+
+/* Look for this message (regular or ghost) */
+message = _notmuch_message_create_for_message_id (
+   notmuch, message_id, );
+if (status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Message exists */
+   *thread_id_ret = talloc_steal (
+   ctx, notmuch_message_get_thread_id (message));
+} else if (status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   /* Message did not exist.  Give it a fresh thread ID and
+* populate this message as a ghost message. */
+   *thread_id_ret = talloc_strdup (
+   ctx, _notmuch_database_generate_thread_id (notmuch));
+   if (! *thread_id_ret) {
+   status = NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+   } else {
+   status = _notmuch_message_initialize_ghost (message, 
*thread_id_ret);
+   if (status == 0)
+   /* Commit the new ghost message */
+   _notmuch_message_sync (message);
+   }
+} else {
+   /* Create failed. Fall through. */
+}
+
+notmuch_message_destroy (message);
+
+return COERCE_STATUS (status, "Error creating ghost message");
+}
+
+/* Pre-ghost messages _resolve_message_id_to_thread_id */
+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret)
+{
 notmuch_status_t status;
 notmuch_message_t *message;
 string thread_id_string;
@@ -2007,7 +2056,7 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
 }
 }

-/* Given a (mostly empty) 'message' and its corresponding
+/* Given a blank or ghost 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
  * The first check is in the metadata of the database to see if we
@@ -2035,16 +2084,22 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t 

[PATCH 06/11] lib: Internal support for querying and creating ghost messages

[Austin Clements]

From: Austin Clements 

This updates the message abstraction to support ghost messages: it
adds a message flag that distinguishes regular messages from ghost
messages, and an internal function for initializing a newly created
(blank) message as a ghost message.
---
 lib/message.cc| 50 --
 lib/notmuch-private.h |  4 
 lib/notmuch.h |  9 -
 3 files changed, 60 insertions(+), 3 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 38bc929..ad832cf 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -39,6 +39,9 @@ struct visible _notmuch_message {
 notmuch_message_file_t *message_file;
 notmuch_message_list_t *replies;
 unsigned long flags;
+/* For flags that are initialized on-demand, lazy_flags indicates
+ * if each flag has been initialized. */
+unsigned long lazy_flags;

 Xapian::Document doc;
 Xapian::termcount termpos;
@@ -99,6 +102,7 @@ _notmuch_message_create_for_document (const void 
*talloc_owner,

 message->frozen = 0;
 message->flags = 0;
+message->lazy_flags = 0;

 /* Each of these will be lazily created as needed. */
 message->message_id = NULL;
@@ -192,7 +196,7 @@ _notmuch_message_create (const void *talloc_owner,
  *
  * There is already a document with message ID 'message_id' in the
  * database. The returned message can be used to query/modify the
- * document.
+ * document. The message may be a ghost message.
  *
  *   NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND:
  *
@@ -305,6 +309,7 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
 const char *thread_prefix = _find_prefix ("thread"),
*tag_prefix = _find_prefix ("tag"),
*id_prefix = _find_prefix ("id"),
+   *type_prefix = _find_prefix ("type"),
*filename_prefix = _find_prefix ("file-direntry"),
*replyto_prefix = _find_prefix ("replyto");

@@ -337,10 +342,23 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
message->message_id =
_notmuch_message_get_term (message, i, end, id_prefix);

+/* Get document type */
+assert (strcmp (id_prefix, type_prefix) < 0);
+if (! (message->lazy_flags & (1 << NOTMUCH_MESSAGE_FLAG_GHOST))) {
+   i.skip_to (type_prefix);
+   if (*i == "Tmail")
+   message->flags &= ~(1 << NOTMUCH_MESSAGE_FLAG_GHOST);
+   else if (*i == "Tghost")
+   message->flags |= (1 << NOTMUCH_MESSAGE_FLAG_GHOST);
+   else
+   INTERNAL_ERROR ("Message without type term");
+   message->lazy_flags |= (1 << NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 /* Get filename list.  Here we get only the terms.  We lazily
  * expand them to full file names when needed in
  * _notmuch_message_ensure_filename_list. */
-assert (strcmp (id_prefix, filename_prefix) < 0);
+assert (strcmp (type_prefix, filename_prefix) < 0);
 if (!message->filename_term_list && !message->filename_list)
message->filename_term_list =
_notmuch_database_get_terms_with_prefix (message, i, end,
@@ -371,6 +389,11 @@ _notmuch_message_invalidate_metadata (notmuch_message_t 
*message,
message->tag_list = NULL;
 }

+if (strcmp ("type", prefix_name) == 0) {
+   message->flags &= ~(1 << NOTMUCH_MESSAGE_FLAG_GHOST);
+   message->lazy_flags &= ~(1 << NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 if (strcmp ("file-direntry", prefix_name) == 0) {
talloc_free (message->filename_term_list);
talloc_free (message->filename_list);
@@ -869,6 +892,10 @@ notmuch_bool_t
 notmuch_message_get_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag)
 {
+if (flag == NOTMUCH_MESSAGE_FLAG_GHOST &&
+   ! (message->lazy_flags & (1 << flag)))
+   _notmuch_message_ensure_metadata (message);
+
 return message->flags & (1 << flag);
 }

@@ -880,6 +907,7 @@ notmuch_message_set_flag (notmuch_message_t *message,
message->flags |= (1 << flag);
 else
message->flags &= ~(1 << flag);
+message->lazy_flags |= (1 << flag);
 }

 time_t
@@ -989,6 +1017,24 @@ _notmuch_message_delete (notmuch_message_t *message)
 return NOTMUCH_STATUS_SUCCESS;
 }

+/* Transform a blank message into a ghost message.  The caller must
+ * _notmuch_message_sync the message. */
+notmuch_private_status_t
+_notmuch_message_initialize_ghost (notmuch_message_t *message,
+  const char *thread_id)
+{
+notmuch_private_status_t status;
+
+status = _notmuch_message_add_term (message, "type", "ghost");
+if (status)
+   return status;
+status = _notmuch_message_add_term (message, "thread", thread_id);
+if (status)
+   return status;
+
+return NOTMUCH_PRIVATE_STATUS_SUCCESS;
+}
+
 /* Ensure that 'message' is not holding any file object open. Future
  * calls to various functions will still automatically open the
  * message file as needed.

[PATCH 05/11] lib: Update database schema doc for ghost messages

[Austin Clements]

From: Austin Clements 

This describes the structure of ghost mail documents.  Ghost messages
are not yet implemented.
---
 lib/database.cc | 20 ++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 8fd7fad..c641bcd 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -50,8 +50,8 @@ typedef struct {

 /* Here's the current schema for our database (for NOTMUCH_DATABASE_VERSION):
  *
- * We currently have two different types of documents (mail and
- * directory) and also some metadata.
+ * We currently have three different types of documents (mail, ghost,
+ * and directory) and also some metadata.
  *
  * Mail document
  * -
@@ -109,6 +109,15 @@ typedef struct {
  *
  * The data portion of a mail document is empty.
  *
+ * Ghost mail document [if NOTMUCH_FEATURE_GHOSTS]
+ * ---
+ * A ghost mail document is like a mail document, but where we don't
+ * have the message content.  These are used to track thread reference
+ * information for messages we haven't received.
+ *
+ * A ghost mail document has type: ghost; id and thread fields that
+ * are identical to the mail document fields; and a MESSAGE_ID value.
+ *
  * Directory document
  * --
  * A directory document is used by a client of the notmuch library to
@@ -172,6 +181,13 @@ typedef struct {
  * generated is 1 and the value will be
  * incremented for each thread ID.
  *
+ * Obsolete metadata
+ * -
+ *
+ * If ! NOTMUCH_FEATURE_GHOSTS, there are no ghost mail documents.
+ * Instead, the database has the following additional database
+ * metadata:
+ *
  * thread_id_* A pre-allocated thread ID for a particular
  * message. This is actually an arbitrarily large
  * family of metadata name. Any particular name is
-- 
2.1.0

[PATCH 04/11] lib: Add a ghost messages database feature

[Austin Clements]

From: Austin Clements 

This will be implemented over the next several patches.  The feature
is not yet "enabled" (this does not add it to
NOTMUCH_FEATURES_CURRENT).
---
 lib/database-private.h | 7 +++
 lib/database.cc| 2 ++
 2 files changed, 9 insertions(+)

diff --git a/lib/database-private.h b/lib/database-private.h
index ca0751c..e2e4bc8 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -85,6 +85,13 @@ enum _notmuch_features {
  *
  * Introduced: version 2. */
 NOTMUCH_FEATURE_BOOL_FOLDER = 1 << 3,
+
+/* If set, missing messages are stored in ghost mail documents.
+ * If unset, thread IDs of ghost messages are stored as database
+ * metadata instead of in ghost documents.
+ *
+ * Introduced: version 3. */
+NOTMUCH_FEATURE_GHOSTS = 1 << 4,
 };

 /* In C++, a named enum is its own type, so define bitwise operators
diff --git a/lib/database.cc b/lib/database.cc
index 1c6ffc5..8fd7fad 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -286,6 +286,8 @@ static const struct {
   "from/subject/message-ID in database", "w" },
 { NOTMUCH_FEATURE_BOOL_FOLDER,
   "exact folder:/path: search", "rw" },
+{ NOTMUCH_FEATURE_GHOSTS,
+  "mail documents for missing messages", "w"},
 };

 const char *
-- 
2.1.0

[PATCH 03/11] lib: Handle empty date value

[Austin Clements]

From: Austin Clements 

In the interest of robustness, avoid undefined behavior of
sortable_unserialise if the date value is missing.  This shouldn't
happen now, but ghost messages will have blank date values.
---
 lib/message.cc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/message.cc b/lib/message.cc
index bbfc250..38bc929 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -896,6 +896,9 @@ notmuch_message_get_date (notmuch_message_t *message)
return 0;
 }

+if (value.empty ())
+   /* sortable_unserialise is undefined on empty string */
+   return 0;
 return Xapian::sortable_unserialise (value);
 }

-- 
2.1.0

[PATCH 02/11] lib: Refactor _notmuch_database_link_message

[Austin Clements]

From: Austin Clements 

This moves the code to retrieve and clear the metadata thread ID out
of _notmuch_database_link_message into its own function.  This will
simplify future changes.
---
 lib/database.cc | 69 +++--
 1 file changed, 43 insertions(+), 26 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 4e68706..1c6ffc5 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1958,6 +1958,37 @@ _notmuch_database_link_message_to_children 
(notmuch_database_t *notmuch,
 return ret;
 }

+/* Fetch and clear the stored thread_id for message, or NULL if none. */
+static char *
+_consume_metadata_thread_id (void *ctx, notmuch_database_t *notmuch,
+notmuch_message_t *message)
+{
+const char *message_id;
+string stored_id;
+char *metadata_key;
+
+message_id = notmuch_message_get_message_id (message);
+metadata_key = _get_metadata_thread_id_key (ctx, message_id);
+
+/* Check if we have already seen related messages to this one.
+ * If we have then use the thread_id that we stored at that time.
+ */
+stored_id = notmuch->xapian_db->get_metadata (metadata_key);
+if (stored_id.empty ()) {
+   return NULL;
+} else {
+Xapian::WritableDatabase *db;
+
+   db = static_cast  (notmuch->xapian_db);
+
+   /* Clear the metadata for this message ID. We don't need it
+* anymore. */
+db->set_metadata (metadata_key, "");
+
+return talloc_strdup (ctx, stored_id.c_str ());
+}
+}
+
 /* Given a (mostly empty) 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
@@ -1988,42 +2019,25 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
notmuch_message_t *message,
notmuch_message_file_t *message_file)
 {
+void *local = talloc_new (NULL);
 notmuch_status_t status;
-const char *message_id, *thread_id = NULL;
-char *metadata_key;
-string stored_id;
-
-message_id = notmuch_message_get_message_id (message);
-metadata_key = _get_metadata_thread_id_key (message, message_id);
-
-/* Check if we have already seen related messages to this one.
- * If we have then use the thread_id that we stored at that time.
- */
-stored_id = notmuch->xapian_db->get_metadata (metadata_key);
-if (! stored_id.empty()) {
-Xapian::WritableDatabase *db;
-
-   db = static_cast  (notmuch->xapian_db);
-
-   /* Clear the metadata for this message ID. We don't need it
-* anymore. */
-db->set_metadata (metadata_key, "");
-thread_id = stored_id.c_str();
+const char *thread_id;

-_notmuch_message_add_term (message, "thread", thread_id);
-}
-talloc_free (metadata_key);
+/* Check if the message already had a thread ID */
+thread_id = _consume_metadata_thread_id (local, notmuch, message);
+if (thread_id)
+   _notmuch_message_add_term (message, "thread", thread_id);

 status = _notmuch_database_link_message_to_parents (notmuch, message,
message_file,
_id);
 if (status)
-   return status;
+   goto DONE;

 status = _notmuch_database_link_message_to_children (notmuch, message,
 _id);
 if (status)
-   return status;
+   goto DONE;

 /* If not part of any existing thread, generate a new thread ID. */
 if (thread_id == NULL) {
@@ -2032,7 +2046,10 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
_notmuch_message_add_term (message, "thread", thread_id);
 }

-return NOTMUCH_STATUS_SUCCESS;
+ DONE:
+talloc_free (local);
+
+return status;
 }

 notmuch_status_t
-- 
2.1.0

[PATCH 01/11] lib: Move message ID compression to _notmuch_message_create_for_message_id

[Austin Clements]

From: Austin Clements 

Previously, this was performed by notmuch_database_add_message.  This
happens to be the only caller currently (which is why this was safe),
but we're about to introduce more callers, and it makes more sense to
put responsibility for ID compression in the lower-level function
rather than requiring each caller to handle it.
---
 lib/database.cc   | 16 
 lib/message.cc|  4 
 lib/notmuch-private.h |  3 +++
 3 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index a47a71d..4e68706 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -390,8 +390,8 @@ find_document_for_doc_id (notmuch_database_t *notmuch, 
unsigned doc_id)
  *
  * notmuch-sha1-
  */
-static char *
-_message_id_compressed (void *ctx, const char *message_id)
+char *
+_notmuch_message_id_compressed (void *ctx, const char *message_id)
 {
 char *sha1, *compressed;

@@ -415,7 +415,7 @@ notmuch_database_find_message (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_NULL_POINTER;

 if (strlen (message_id) > NOTMUCH_MESSAGE_ID_MAX)
-   message_id = _message_id_compressed (notmuch, message_id);
+   message_id = _notmuch_message_id_compressed (notmuch, message_id);

 try {
status = _notmuch_database_find_unique_doc_id (notmuch, "id",
@@ -1728,7 +1728,7 @@ static char *
 _get_metadata_thread_id_key (void *ctx, const char *message_id)
 {
 if (strlen (message_id) > NOTMUCH_MESSAGE_ID_MAX)
-   message_id = _message_id_compressed (ctx, message_id);
+   message_id = _notmuch_message_id_compressed (ctx, message_id);

 return talloc_asprintf (ctx, NOTMUCH_METADATA_THREAD_ID_PREFIX "%s",
message_id);
@@ -2100,14 +2100,6 @@ notmuch_database_add_message (notmuch_database_t 
*notmuch,
 * better than no message-id at all. */
if (message_id == NULL)
message_id = talloc_strdup (message_file, header);
-
-   /* If a message ID is too long, substitute its sha1 instead. */
-   if (message_id && strlen (message_id) > NOTMUCH_MESSAGE_ID_MAX) {
-   char *compressed = _message_id_compressed (message_file,
-  message_id);
-   talloc_free (message_id);
-   message_id = compressed;
-   }
}

if (message_id == NULL ) {
diff --git a/lib/message.cc b/lib/message.cc
index 7e82548..bbfc250 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -226,6 +226,10 @@ _notmuch_message_create_for_message_id (notmuch_database_t 
*notmuch,
 else if (*status_ret)
return NULL;

+/* If the message ID is too long, substitute its sha1 instead. */
+if (strlen (message_id) > NOTMUCH_MESSAGE_ID_MAX)
+   message_id = _notmuch_message_id_compressed (message, message_id);
+
 term = talloc_asprintf (NULL, "%s%s",
_find_prefix ("id"), message_id);
 if (term == NULL) {
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 17f3061..36cc12b 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -174,6 +174,9 @@ typedef struct _notmuch_doc_id_set notmuch_doc_id_set_t;
 const char *
 _find_prefix (const char *name);

+char *
+_notmuch_message_id_compressed (void *ctx, const char *message_id);
+
 notmuch_status_t
 _notmuch_database_ensure_writable (notmuch_database_t *notmuch);

-- 
2.1.0

[PATCH 00/11] Add ghost messages and fix thread linking

[Austin Clements]

This series modifies our database representation of messages that have
been referenced by other messages, but for which we don't have the
message itself.  Currently, we store this information as Xapian
metadata, but this has several downsides for performance and
complexity and results in hard-to-fix thread linking bugs.  This patch
series implements "ghost messages", which replace this Xapian metadata
with Xapian documents that look and act very much like regular message
documents, but simply have no content.  This simplifies and speeds up
our thread linking algorithm and fixes the currently broken thread
linking test.

Ghost messages also open up interesting future possibilities, such as
"pre-seeding" tags for messages that are not yet indexed.  This could
be used to make notmuch insert simpler and more robust, as part of tag
synchronization, and to improve nmbug's behavior when tags arrive
before messages.

[PATCH v2] VIM: Use notmuch CLI for config

[David Bremner]

Ian Main  writes:

> This patch switches from reading .notmuch-config directly to using
> the CLI the same way that emacs does it.  It actually uses less code
> and is probably less error prone.
>
> Ian

The general approach seems sane; it seems quite brittle to read the
config file directly. I notice there is not really any error handling;
OTOH, as far as I can read Ruby, there is not any in the previous
version either.  Technically, this does add a dependency of the vim
client on the CLI that did not exist before. Personally I don't find
this onerous (even notmuch-vim users need "notmuch new", except in
rather unusual circumstances.).

I'd like feedback/testing from actual vim interface users before
merging.

Re: [PATCH v2] VIM: Use notmuch CLI for config

[David Bremner]

Ian Main im...@stemwinder.org writes:

 This patch switches from reading .notmuch-config directly to using
 the CLI the same way that emacs does it.  It actually uses less code
 and is probably less error prone.

 Ian

The general approach seems sane; it seems quite brittle to read the
config file directly. I notice there is not really any error handling;
OTOH, as far as I can read Ruby, there is not any in the previous
version either.  Technically, this does add a dependency of the vim
client on the CLI that did not exist before. Personally I don't find
this onerous (even notmuch-vim users need notmuch new, except in
rather unusual circumstances.).

I'd like feedback/testing from actual vim interface users before
merging.

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v5] nmbug: Translate to Python

[David Bremner]

W. Trevor King wk...@tremily.us writes:

 This allows us to capture stdout and stderr separately, and do other
 explicit subprocess manipulation without resorting to external
 packages.  It should be compatible with Python 2.6 and later
 (including the 3.x series), although with 2.6 you'll need the external
 argparse package.

As reported on IRC, this version has been working ok for me for a week
or so. I haven't marked it ready to push because iirc, the python 2.6
support doesn't really work and is doomed to be removed.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v2] VIM: Use notmuch CLI for config

[Sergei Shilovsky]

The patch works fine for me.

Ian, though I could not test whether result='' matters or not, cause
'notmuch get' looks to always return a string.

It also fixes 
id:cahc2po19azabaijhdfqr1rl5t0guflaq5ckb_7w-z8y+jxt...@mail.gmail.com

On Fri, Oct 3, 2014 at 10:54 AM, David Bremner da...@tethera.net wrote:
 Ian Main im...@stemwinder.org writes:

 This patch switches from reading .notmuch-config directly to using
 the CLI the same way that emacs does it.  It actually uses less code
 and is probably less error prone.

 Ian

 The general approach seems sane; it seems quite brittle to read the
 config file directly. I notice there is not really any error handling;
 OTOH, as far as I can read Ruby, there is not any in the previous
 version either.  Technically, this does add a dependency of the vim
 client on the CLI that did not exist before. Personally I don't find
 this onerous (even notmuch-vim users need notmuch new, except in
 rather unusual circumstances.).

 I'd like feedback/testing from actual vim interface users before
 merging.

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch



-- 
С уважением,
Сергей Шиловский
Sergei Shilovsky
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v5] nmbug: Translate to Python

[Tomi Ollila]

On Fri, Oct 03 2014, David Bremner brem...@debian.org wrote:

 W. Trevor King wk...@tremily.us writes:

 This allows us to capture stdout and stderr separately, and do other
 explicit subprocess manipulation without resorting to external
 packages.  It should be compatible with Python 2.6 and later
 (including the 3.x series), although with 2.6 you'll need the external
 argparse package.

 As reported on IRC, this version has been working ok for me for a week
 or so. I haven't marked it ready to push because iirc, the python 2.6
 support doesn't really work and is doomed to be removed.

Maybe just amend to It is compatible with Python 2.7 and later..., with
dropping the comment about external argparse package?

Currently also tests don't work with Python 2.6 -- I have a patch that
would fix this but probably it is just noise at this time (and I'd still 
need to do the harderst part -- write the commit message).

 d

Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 11/11] lib: Remove unnecessary thread linking steps when using ghost messages

[Austin Clements]

From: Austin Clements amdra...@mit.edu

Previously, it was necessary to link new messages to children to work
around some (though not all) problems with the old metadata-based
approach to stored thread IDs.  With ghost messages, this is no longer
necessary, so don't bother with child linking when ghost messages are
in use.
---
 lib/database.cc | 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index ff6a7f6..4655f59 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -2167,10 +2167,23 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
goto DONE;
 
-status = _notmuch_database_link_message_to_children (notmuch, message,
-thread_id);
-if (status)
-   goto DONE;
+if (! (notmuch-features  NOTMUCH_FEATURE_GHOSTS)) {
+   /* In general, it shouldn't be necessary to link children,
+* since the earlier indexing of those children will have
+* stored a thread ID for the missing parent.  However, prior
+* to ghost messages, these stored thread IDs were NOT
+* rewritten during thread merging (and there was no
+* performant way to do so), so if indexed children were
+* pulled into a different thread ID by a merge, it was
+* necessary to pull them *back* into the stored thread ID of
+* the parent.  With ghost messages, we just rewrite the
+* stored thread IDs during merging, so this workaround isn't
+* necessary. */
+   status = _notmuch_database_link_message_to_children (notmuch, message,
+thread_id);
+   if (status)
+   goto DONE;
+}
 
 /* If not part of any existing thread, generate a new thread ID. */
 if (thread_id == NULL) {
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 05/11] lib: Update database schema doc for ghost messages

[Austin Clements]

From: Austin Clements amdra...@mit.edu

This describes the structure of ghost mail documents.  Ghost messages
are not yet implemented.
---
 lib/database.cc | 20 ++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 8fd7fad..c641bcd 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -50,8 +50,8 @@ typedef struct {
 
 /* Here's the current schema for our database (for NOTMUCH_DATABASE_VERSION):
  *
- * We currently have two different types of documents (mail and
- * directory) and also some metadata.
+ * We currently have three different types of documents (mail, ghost,
+ * and directory) and also some metadata.
  *
  * Mail document
  * -
@@ -109,6 +109,15 @@ typedef struct {
  *
  * The data portion of a mail document is empty.
  *
+ * Ghost mail document [if NOTMUCH_FEATURE_GHOSTS]
+ * ---
+ * A ghost mail document is like a mail document, but where we don't
+ * have the message content.  These are used to track thread reference
+ * information for messages we haven't received.
+ *
+ * A ghost mail document has type: ghost; id and thread fields that
+ * are identical to the mail document fields; and a MESSAGE_ID value.
+ *
  * Directory document
  * --
  * A directory document is used by a client of the notmuch library to
@@ -172,6 +181,13 @@ typedef struct {
  * generated is 1 and the value will be
  * incremented for each thread ID.
  *
+ * Obsolete metadata
+ * -
+ *
+ * If ! NOTMUCH_FEATURE_GHOSTS, there are no ghost mail documents.
+ * Instead, the database has the following additional database
+ * metadata:
+ *
  * thread_id_* A pre-allocated thread ID for a particular
  * message. This is actually an arbitrarily large
  * family of metadata name. Any particular name is
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 00/11] Add ghost messages and fix thread linking

[Austin Clements]

This series modifies our database representation of messages that have
been referenced by other messages, but for which we don't have the
message itself.  Currently, we store this information as Xapian
metadata, but this has several downsides for performance and
complexity and results in hard-to-fix thread linking bugs.  This patch
series implements ghost messages, which replace this Xapian metadata
with Xapian documents that look and act very much like regular message
documents, but simply have no content.  This simplifies and speeds up
our thread linking algorithm and fixes the currently broken thread
linking test.

Ghost messages also open up interesting future possibilities, such as
pre-seeding tags for messages that are not yet indexed.  This could
be used to make notmuch insert simpler and more robust, as part of tag
synchronization, and to improve nmbug's behavior when tags arrive
before messages.

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 06/11] lib: Internal support for querying and creating ghost messages

[Austin Clements]

From: Austin Clements amdra...@mit.edu

This updates the message abstraction to support ghost messages: it
adds a message flag that distinguishes regular messages from ghost
messages, and an internal function for initializing a newly created
(blank) message as a ghost message.
---
 lib/message.cc| 50 --
 lib/notmuch-private.h |  4 
 lib/notmuch.h |  9 -
 3 files changed, 60 insertions(+), 3 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 38bc929..ad832cf 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -39,6 +39,9 @@ struct visible _notmuch_message {
 notmuch_message_file_t *message_file;
 notmuch_message_list_t *replies;
 unsigned long flags;
+/* For flags that are initialized on-demand, lazy_flags indicates
+ * if each flag has been initialized. */
+unsigned long lazy_flags;
 
 Xapian::Document doc;
 Xapian::termcount termpos;
@@ -99,6 +102,7 @@ _notmuch_message_create_for_document (const void 
*talloc_owner,
 
 message-frozen = 0;
 message-flags = 0;
+message-lazy_flags = 0;
 
 /* Each of these will be lazily created as needed. */
 message-message_id = NULL;
@@ -192,7 +196,7 @@ _notmuch_message_create (const void *talloc_owner,
  *
  * There is already a document with message ID 'message_id' in the
  * database. The returned message can be used to query/modify the
- * document.
+ * document. The message may be a ghost message.
  *
  *   NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND:
  *
@@ -305,6 +309,7 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
 const char *thread_prefix = _find_prefix (thread),
*tag_prefix = _find_prefix (tag),
*id_prefix = _find_prefix (id),
+   *type_prefix = _find_prefix (type),
*filename_prefix = _find_prefix (file-direntry),
*replyto_prefix = _find_prefix (replyto);
 
@@ -337,10 +342,23 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
message-message_id =
_notmuch_message_get_term (message, i, end, id_prefix);
 
+/* Get document type */
+assert (strcmp (id_prefix, type_prefix)  0);
+if (! (message-lazy_flags  (1  NOTMUCH_MESSAGE_FLAG_GHOST))) {
+   i.skip_to (type_prefix);
+   if (*i == Tmail)
+   message-flags = ~(1  NOTMUCH_MESSAGE_FLAG_GHOST);
+   else if (*i == Tghost)
+   message-flags |= (1  NOTMUCH_MESSAGE_FLAG_GHOST);
+   else
+   INTERNAL_ERROR (Message without type term);
+   message-lazy_flags |= (1  NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 /* Get filename list.  Here we get only the terms.  We lazily
  * expand them to full file names when needed in
  * _notmuch_message_ensure_filename_list. */
-assert (strcmp (id_prefix, filename_prefix)  0);
+assert (strcmp (type_prefix, filename_prefix)  0);
 if (!message-filename_term_list  !message-filename_list)
message-filename_term_list =
_notmuch_database_get_terms_with_prefix (message, i, end,
@@ -371,6 +389,11 @@ _notmuch_message_invalidate_metadata (notmuch_message_t 
*message,
message-tag_list = NULL;
 }
 
+if (strcmp (type, prefix_name) == 0) {
+   message-flags = ~(1  NOTMUCH_MESSAGE_FLAG_GHOST);
+   message-lazy_flags = ~(1  NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 if (strcmp (file-direntry, prefix_name) == 0) {
talloc_free (message-filename_term_list);
talloc_free (message-filename_list);
@@ -869,6 +892,10 @@ notmuch_bool_t
 notmuch_message_get_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag)
 {
+if (flag == NOTMUCH_MESSAGE_FLAG_GHOST 
+   ! (message-lazy_flags  (1  flag)))
+   _notmuch_message_ensure_metadata (message);
+
 return message-flags  (1  flag);
 }
 
@@ -880,6 +907,7 @@ notmuch_message_set_flag (notmuch_message_t *message,
message-flags |= (1  flag);
 else
message-flags = ~(1  flag);
+message-lazy_flags |= (1  flag);
 }
 
 time_t
@@ -989,6 +1017,24 @@ _notmuch_message_delete (notmuch_message_t *message)
 return NOTMUCH_STATUS_SUCCESS;
 }
 
+/* Transform a blank message into a ghost message.  The caller must
+ * _notmuch_message_sync the message. */
+notmuch_private_status_t
+_notmuch_message_initialize_ghost (notmuch_message_t *message,
+  const char *thread_id)
+{
+notmuch_private_status_t status;
+
+status = _notmuch_message_add_term (message, type, ghost);
+if (status)
+   return status;
+status = _notmuch_message_add_term (message, thread, thread_id);
+if (status)
+   return status;
+
+return NOTMUCH_PRIVATE_STATUS_SUCCESS;
+}
+
 /* Ensure that 'message' is not holding any file object open. Future
  * calls to various functions will still automatically open the
  * message file as needed.
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 

[PATCH 08/11] lib: Implement upgrade to ghost messages feature

[Austin Clements]

From: Austin Clements amdra...@mit.edu

Somehow this is the first upgrade pass that actually does *any* error
checking, so this also adds the bit of necessary infrastructure to
handle that.
---
 lib/database.cc | 64 +++--
 1 file changed, 62 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index fdcc526..ff6a7f6 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1231,6 +1231,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 notmuch_bool_t timer_is_active = FALSE;
 enum _notmuch_features target_features, new_features;
 notmuch_status_t status;
+notmuch_private_status_t private_status;
 unsigned int count = 0, total = 0;
 
 status = _notmuch_database_ensure_writable (notmuch);
@@ -1275,6 +1276,11 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
for (t = db-allterms_begin (XTIMESTAMP); t != t_end; t++)
++total;
 }
+if (new_features  NOTMUCH_FEATURE_GHOSTS) {
+   t_end = db-metadata_keys_end (thread_id_);
+   for (t = db-metadata_keys_begin (thread_id_); t != t_end; ++t)
+   ++total;
+}
 
 /* Perform the upgrade in a transaction. */
 db-begin_transaction (true);
@@ -1378,10 +1384,64 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
}
 }
 
+/* Perform metadata upgrades. */
+
+/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
+ * messages were stored as database metadata. Change these to
+ * ghost messages.
+ */
+if (new_features  NOTMUCH_FEATURE_GHOSTS) {
+   notmuch_message_t *message;
+   std::string message_id, thread_id;
+
+   t_end = db-metadata_keys_end (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+   for (t = db-metadata_keys_begin (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+t != t_end; ++t) {
+   if (do_progress_notify) {
+   progress_notify (closure, (double) count / total);
+   do_progress_notify = 0;
+   }
+
+   message_id = (*t).substr (
+   strlen (NOTMUCH_METADATA_THREAD_ID_PREFIX));
+   thread_id = db-get_metadata (*t);
+
+   /* Create ghost message */
+   message = _notmuch_message_create_for_message_id (
+   notmuch, message_id.c_str (), private_status);
+   if (private_status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Document already exists; ignore the stored thread ID */
+   } else if (private_status ==
+  NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   private_status = _notmuch_message_initialize_ghost (
+   message, thread_id.c_str ());
+   if (! private_status)
+   _notmuch_message_sync (message);
+   }
+
+   if (private_status) {
+   fprintf (stderr,
+Upgrade failed while creating ghost messages.\n);
+   status = COERCE_STATUS (private_status, Unexpected status from 
_notmuch_message_initialize_ghost);
+   goto DONE;
+   }
+
+   /* Clear saved metadata thread ID */
+   db-set_metadata (*t, );
+
+   ++count;
+   }
+}
+
+status = NOTMUCH_STATUS_SUCCESS;
 db-set_metadata (features, _print_features (local, notmuch-features));
 db-set_metadata (version, STRINGIFY (NOTMUCH_DATABASE_VERSION));
 
-db-commit_transaction ();
+ DONE:
+if (status == NOTMUCH_STATUS_SUCCESS)
+   db-commit_transaction ();
+else
+   db-cancel_transaction ();
 
 if (timer_is_active) {
/* Now stop the timer. */
@@ -1397,7 +1457,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 }
 
 talloc_free (local);
-return NOTMUCH_STATUS_SUCCESS;
+return status;
 }
 
 notmuch_status_t
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 02/11] lib: Refactor _notmuch_database_link_message

[Austin Clements]

From: Austin Clements amdra...@mit.edu

This moves the code to retrieve and clear the metadata thread ID out
of _notmuch_database_link_message into its own function.  This will
simplify future changes.
---
 lib/database.cc | 69 +++--
 1 file changed, 43 insertions(+), 26 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 4e68706..1c6ffc5 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1958,6 +1958,37 @@ _notmuch_database_link_message_to_children 
(notmuch_database_t *notmuch,
 return ret;
 }
 
+/* Fetch and clear the stored thread_id for message, or NULL if none. */
+static char *
+_consume_metadata_thread_id (void *ctx, notmuch_database_t *notmuch,
+notmuch_message_t *message)
+{
+const char *message_id;
+string stored_id;
+char *metadata_key;
+
+message_id = notmuch_message_get_message_id (message);
+metadata_key = _get_metadata_thread_id_key (ctx, message_id);
+
+/* Check if we have already seen related messages to this one.
+ * If we have then use the thread_id that we stored at that time.
+ */
+stored_id = notmuch-xapian_db-get_metadata (metadata_key);
+if (stored_id.empty ()) {
+   return NULL;
+} else {
+Xapian::WritableDatabase *db;
+
+   db = static_cast Xapian::WritableDatabase * (notmuch-xapian_db);
+
+   /* Clear the metadata for this message ID. We don't need it
+* anymore. */
+db-set_metadata (metadata_key, );
+
+return talloc_strdup (ctx, stored_id.c_str ());
+}
+}
+
 /* Given a (mostly empty) 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
@@ -1988,42 +2019,25 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
notmuch_message_t *message,
notmuch_message_file_t *message_file)
 {
+void *local = talloc_new (NULL);
 notmuch_status_t status;
-const char *message_id, *thread_id = NULL;
-char *metadata_key;
-string stored_id;
-
-message_id = notmuch_message_get_message_id (message);
-metadata_key = _get_metadata_thread_id_key (message, message_id);
-
-/* Check if we have already seen related messages to this one.
- * If we have then use the thread_id that we stored at that time.
- */
-stored_id = notmuch-xapian_db-get_metadata (metadata_key);
-if (! stored_id.empty()) {
-Xapian::WritableDatabase *db;
-
-   db = static_cast Xapian::WritableDatabase * (notmuch-xapian_db);
-
-   /* Clear the metadata for this message ID. We don't need it
-* anymore. */
-db-set_metadata (metadata_key, );
-thread_id = stored_id.c_str();
+const char *thread_id;
 
-_notmuch_message_add_term (message, thread, thread_id);
-}
-talloc_free (metadata_key);
+/* Check if the message already had a thread ID */
+thread_id = _consume_metadata_thread_id (local, notmuch, message);
+if (thread_id)
+   _notmuch_message_add_term (message, thread, thread_id);
 
 status = _notmuch_database_link_message_to_parents (notmuch, message,
message_file,
thread_id);
 if (status)
-   return status;
+   goto DONE;
 
 status = _notmuch_database_link_message_to_children (notmuch, message,
 thread_id);
 if (status)
-   return status;
+   goto DONE;
 
 /* If not part of any existing thread, generate a new thread ID. */
 if (thread_id == NULL) {
@@ -2032,7 +2046,10 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
_notmuch_message_add_term (message, thread, thread_id);
 }
 
-return NOTMUCH_STATUS_SUCCESS;
+ DONE:
+talloc_free (local);
+
+return status;
 }
 
 notmuch_status_t
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 09/11] lib: Enable ghost messages feature

[Austin Clements]

From: Austin Clements amdra...@mit.edu

This fixes the broken thread order test.
---
 lib/database-private.h| 2 +-
 test/T260-thread-order.sh | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index e2e4bc8..15e03cc 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -166,7 +166,7 @@ struct _notmuch_database {
  * databases will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
 (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
- NOTMUCH_FEATURE_BOOL_FOLDER)
+ NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
 
 /* Return the list of terms from the given iterator matching a prefix.
  * The prefix will be stripped from the strings in the returned list.
diff --git a/test/T260-thread-order.sh b/test/T260-thread-order.sh
index b435d79..99f5833 100755
--- a/test/T260-thread-order.sh
+++ b/test/T260-thread-order.sh
@@ -30,7 +30,6 @@ expected=$(for ((i = 0; i  $nthreads; i++)); do
 test_expect_equal $output $expected
 
 test_begin_subtest Messages with all parents get linked in all delivery 
orders
-test_subtest_known_broken
 # Here we do the same thing as the previous test, but each message
 # references all of its parents.  Since every message references the
 # root of the thread, each thread should always be fully joined.  This
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 10/11] test: Test upgrade to ghost messages feature

[Austin Clements]

---
 test/T530-upgrade.sh | 21 +
 1 file changed, 21 insertions(+)

diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
index c4c4ac8..6b42a69 100755
--- a/test/T530-upgrade.sh
+++ b/test/T530-upgrade.sh
@@ -116,4 +116,25 @@ MAIL_DIR/bar/new/21:2,
 MAIL_DIR/bar/new/22:2,
 MAIL_DIR/cur/51:2,
 
+# Ghost messages are difficult to test since they're nearly invisible.
+# However, if the upgrade works correctly, the ghost message should
+# retain the right thread ID even if all of the original messages in
+# the thread are deleted.  That's what we test.  This won't detect if
+# the upgrade just plain didn't happen, but it should detect if
+# something went wrong.
+test_begin_subtest ghost message retains thread ID
+# Upgrade database
+notmuch new
+# Get thread ID of real message
+thread=$(notmuch search --output=threads id:4efc743a.3060...@april.org)
+# Delete all real messages in that thread
+rm $(notmuch search --output=files $thread)
+notmuch new
+# Deliver ghost message
+add_message '[subject]=Ghost' '[id]=4efc3931.6030...@april.org'
+# If the ghost upgrade worked, the new message should be attached to
+# the existing thread ID.
+nthread=$(notmuch search --output=threads id:4efc3931.6030...@april.org)
+test_expect_equal $thread $nthread
+
 test_done
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 04/11] lib: Add a ghost messages database feature

[Austin Clements]

From: Austin Clements amdra...@mit.edu

This will be implemented over the next several patches.  The feature
is not yet enabled (this does not add it to
NOTMUCH_FEATURES_CURRENT).
---
 lib/database-private.h | 7 +++
 lib/database.cc| 2 ++
 2 files changed, 9 insertions(+)

diff --git a/lib/database-private.h b/lib/database-private.h
index ca0751c..e2e4bc8 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -85,6 +85,13 @@ enum _notmuch_features {
  *
  * Introduced: version 2. */
 NOTMUCH_FEATURE_BOOL_FOLDER = 1  3,
+
+/* If set, missing messages are stored in ghost mail documents.
+ * If unset, thread IDs of ghost messages are stored as database
+ * metadata instead of in ghost documents.
+ *
+ * Introduced: version 3. */
+NOTMUCH_FEATURE_GHOSTS = 1  4,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
diff --git a/lib/database.cc b/lib/database.cc
index 1c6ffc5..8fd7fad 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -286,6 +286,8 @@ static const struct {
   from/subject/message-ID in database, w },
 { NOTMUCH_FEATURE_BOOL_FOLDER,
   exact folder:/path: search, rw },
+{ NOTMUCH_FEATURE_GHOSTS,
+  mail documents for missing messages, w},
 };
 
 const char *
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 03/11] lib: Handle empty date value

[Austin Clements]

From: Austin Clements amdra...@mit.edu

In the interest of robustness, avoid undefined behavior of
sortable_unserialise if the date value is missing.  This shouldn't
happen now, but ghost messages will have blank date values.
---
 lib/message.cc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/message.cc b/lib/message.cc
index bbfc250..38bc929 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -896,6 +896,9 @@ notmuch_message_get_date (notmuch_message_t *message)
return 0;
 }
 
+if (value.empty ())
+   /* sortable_unserialise is undefined on empty string */
+   return 0;
 return Xapian::sortable_unserialise (value);
 }
 
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 07/11] lib: Implement ghost-based thread linking

[Austin Clements]

From: Austin Clements amdra...@mit.edu

This updates the thread linking code to use ghost messages instead of
user metadata to link messages into threads.

In contrast with the old approach, this is actually correct.
Previously, thread merging updated only the thread IDs of message
documents, not thread IDs stored in user metadata.  As originally
diagnosed by Mark Walters [1] and as demonstrated by the broken
T260-thread-order test, this can cause notmuch to fail to link
messages even though they're in the same thread.  In principle the old
approach could have been fixed by updating the user metadata thread
IDs as well, but these are not indexed and hence this would have
required a full scan of all stored thread IDs.  Ghost messages solve
this problem naturally by reusing the exact same thread ID and message
ID representation and indexing as regular messages.

Furthermore, thanks to this greater symmetry, ghost messages are also
algorithmically simpler.  We continue to support the old user metadata
format, so this patch can't delete any code, but when we do remove
support for the old format, several functions can simply be deleted.

[1] id:8738h7kv2q@qmul.ac.uk
---
 lib/database.cc | 86 +
 1 file changed, 75 insertions(+), 11 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index c641bcd..fdcc526 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1752,6 +1752,12 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
message_id);
 }
 
+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret);
+
 /* Find the thread ID to which the message with 'message_id' belongs.
  *
  * Note: 'thread_id_ret' must not be NULL!
@@ -1760,9 +1766,9 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
  *
  * Note: If there is no message in the database with the given
  * 'message_id' then a new thread_id will be allocated for this
- * message and stored in the database metadata, (where this same
+ * message ID and stored in the database metadata so that the
  * thread ID can be looked up if the message is added to the database
- * later).
+ * later.
  */
 static notmuch_status_t
 _resolve_message_id_to_thread_id (notmuch_database_t *notmuch,
@@ -1770,6 +1776,49 @@ _resolve_message_id_to_thread_id (notmuch_database_t 
*notmuch,
  const char *message_id,
  const char **thread_id_ret)
 {
+notmuch_private_status_t status;
+notmuch_message_t *message;
+
+if (! (notmuch-features  NOTMUCH_FEATURE_GHOSTS))
+   return _resolve_message_id_to_thread_id_old (notmuch, ctx, message_id,
+thread_id_ret);
+
+/* Look for this message (regular or ghost) */
+message = _notmuch_message_create_for_message_id (
+   notmuch, message_id, status);
+if (status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Message exists */
+   *thread_id_ret = talloc_steal (
+   ctx, notmuch_message_get_thread_id (message));
+} else if (status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   /* Message did not exist.  Give it a fresh thread ID and
+* populate this message as a ghost message. */
+   *thread_id_ret = talloc_strdup (
+   ctx, _notmuch_database_generate_thread_id (notmuch));
+   if (! *thread_id_ret) {
+   status = NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+   } else {
+   status = _notmuch_message_initialize_ghost (message, 
*thread_id_ret);
+   if (status == 0)
+   /* Commit the new ghost message */
+   _notmuch_message_sync (message);
+   }
+} else {
+   /* Create failed. Fall through. */
+}
+
+notmuch_message_destroy (message);
+
+return COERCE_STATUS (status, Error creating ghost message);
+}
+
+/* Pre-ghost messages _resolve_message_id_to_thread_id */
+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret)
+{
 notmuch_status_t status;
 notmuch_message_t *message;
 string thread_id_string;
@@ -2007,7 +2056,7 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
 }
 }
 
-/* Given a (mostly empty) 'message' and its corresponding
+/* Given a blank or ghost 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
  * The first check is in the metadata of the database to see if we
@@ -2035,16 +2084,22 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t 

[PATCH 01/11] lib: Move message ID compression to _notmuch_message_create_for_message_id

[Austin Clements]

From: Austin Clements amdra...@mit.edu

Previously, this was performed by notmuch_database_add_message.  This
happens to be the only caller currently (which is why this was safe),
but we're about to introduce more callers, and it makes more sense to
put responsibility for ID compression in the lower-level function
rather than requiring each caller to handle it.
---
 lib/database.cc   | 16 
 lib/message.cc|  4 
 lib/notmuch-private.h |  3 +++
 3 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index a47a71d..4e68706 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -390,8 +390,8 @@ find_document_for_doc_id (notmuch_database_t *notmuch, 
unsigned doc_id)
  *
  * notmuch-sha1-sha1_sum_of_message_id
  */
-static char *
-_message_id_compressed (void *ctx, const char *message_id)
+char *
+_notmuch_message_id_compressed (void *ctx, const char *message_id)
 {
 char *sha1, *compressed;
 
@@ -415,7 +415,7 @@ notmuch_database_find_message (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_NULL_POINTER;
 
 if (strlen (message_id)  NOTMUCH_MESSAGE_ID_MAX)
-   message_id = _message_id_compressed (notmuch, message_id);
+   message_id = _notmuch_message_id_compressed (notmuch, message_id);
 
 try {
status = _notmuch_database_find_unique_doc_id (notmuch, id,
@@ -1728,7 +1728,7 @@ static char *
 _get_metadata_thread_id_key (void *ctx, const char *message_id)
 {
 if (strlen (message_id)  NOTMUCH_MESSAGE_ID_MAX)
-   message_id = _message_id_compressed (ctx, message_id);
+   message_id = _notmuch_message_id_compressed (ctx, message_id);
 
 return talloc_asprintf (ctx, NOTMUCH_METADATA_THREAD_ID_PREFIX %s,
message_id);
@@ -2100,14 +2100,6 @@ notmuch_database_add_message (notmuch_database_t 
*notmuch,
 * better than no message-id at all. */
if (message_id == NULL)
message_id = talloc_strdup (message_file, header);
-
-   /* If a message ID is too long, substitute its sha1 instead. */
-   if (message_id  strlen (message_id)  NOTMUCH_MESSAGE_ID_MAX) {
-   char *compressed = _message_id_compressed (message_file,
-  message_id);
-   talloc_free (message_id);
-   message_id = compressed;
-   }
}
 
if (message_id == NULL ) {
diff --git a/lib/message.cc b/lib/message.cc
index 7e82548..bbfc250 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -226,6 +226,10 @@ _notmuch_message_create_for_message_id (notmuch_database_t 
*notmuch,
 else if (*status_ret)
return NULL;
 
+/* If the message ID is too long, substitute its sha1 instead. */
+if (strlen (message_id)  NOTMUCH_MESSAGE_ID_MAX)
+   message_id = _notmuch_message_id_compressed (message, message_id);
+
 term = talloc_asprintf (NULL, %s%s,
_find_prefix (id), message_id);
 if (term == NULL) {
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 17f3061..36cc12b 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -174,6 +174,9 @@ typedef struct _notmuch_doc_id_set notmuch_doc_id_set_t;
 const char *
 _find_prefix (const char *name);
 
+char *
+_notmuch_message_id_compressed (void *ctx, const char *message_id);
+
 notmuch_status_t
 _notmuch_database_ensure_writable (notmuch_database_t *notmuch);
 
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] NEWS: News for notmuch-jump

[Austin Clements]

---
 NEWS | 15 +++
 1 file changed, 15 insertions(+)

diff --git a/NEWS b/NEWS
index 2d85090..fa57e5d 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,21 @@
 Notmuch 0.19 (UNRELEASED)
 =
 
+Emacs Interface
+---
+
+Use the `j` key to access saved searches from anywhere in notmuch
+
+  `j` is now globally bound to `notmuch-jump`, which provides fast,
+  interactive keyboard shortcuts to saved searches.  For example,
+  with the default saved searches `j i` from anywhere in notmuch will
+  bring up the inbox.
+
+Expanded default saved search settings
+
+  The default saved searches now include several more common searches,
+  as well as shortcut keys for `notmuch-jump`.
+
 Library changes
 ---
 
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] doc: Emacs manual improvements and expansions

[Austin Clements]

Fix several typos, improve general wording and flow, and add some
information on notmuch-jump.
---

Sorry I didn't tease these changes apart.  I went through it linearly
and it didn't seem worth the trouble of separating them after the fact.

 doc/notmuch-emacs.rst | 50 +-
 1 file changed, 37 insertions(+), 13 deletions(-)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 09579bf..6f2f61e 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -6,17 +6,17 @@ About this Manual
 =
 
 This manual covers only the Emacs interface to Notmuch. For information
-on the command line interface, see See section “Description” in Notmuch
-Manual Pager. To save typing, we will sometimes use *notmuch* in this
-manual to refer to the Emacs interface to Notmuch. If the distinction
-should every be important, we’ll refer to the Emacs interface as
+on the command line interface, see section “Description” in the Notmuch
+Manual Pages. To save typing, we will sometimes use *notmuch* in this
+manual to refer to the Emacs interface to Notmuch. When this distinction
+is important, we’ll refer to the Emacs interface as
 *notmuch-emacs*.
 
 Notmuch-emacs is highly customizable via the the Emacs customization
 framework (or just by setting the appropriate variables). We try to
 point out relevant variables in this manual, but in order to avoid
-duplication of information, but you can usually find the most detailed
-description in the variables docstring.
+duplication of information, you can usually find the most detailed
+description in the variables' docstring.
 
 notmuch-hello
 =
@@ -89,15 +89,19 @@ notmuch-hello key bindings
 Saved Searches
 --
 
-Notmuch replaces the static assignment of messages with the more dynamic
-notion of searching. Notmuch-hello presents the user with a customizable
-set of saved searches. The initial defaults are ``tag:inbox`` and
-``tag:unread``, but you can customize the following variables
+Since notmuch is entirely search-based, it's often useful to organize
+mail around common searches.  To facilitate this, the first section of
+notmuch-hello presents a customizable set of saved searches.  Saved
+searches can also be accessed from anywhere in notmuch by pressing
+``j`` to access :ref:`notmuch-jump`.
+
+The saved searches default to various common searches such as
+``tag:inbox`` to access the inbox and ``tag:unread`` to access all
+unread mail, but there are several options for customization:
 
 :index:`notmuch-saved-searches`
-A list of cons pairs, the first being the name to display, the
-second being a query string for Notmuch. See section “Description”
-in Notmuch Query Syntax.
+The list of saved searches, including names, queries, and
+additional per-query options.
 
 :index:`notmuch-saved-searches-sort-function`
 This variable controls how saved searches should be sorted. A value
@@ -179,6 +183,26 @@ notmuch-show
 notmuch-tree
 
 
+Global key bindings
+===
+
+Several features are accessible from anywhere in notmuch through the
+following key bindings:
+
+``j``
+Jump to saved searches using :ref:`notmuch-jump`.
+
+notmuch-jump
+
+
+Saved searches configured through :ref:`notmuch-saved-searches` can
+include a shortcut key that's accessible through notmuch-jump.
+Pressing ``j`` anywhere in notmuch followed by the configured shortcut
+key of a saved search will immediately jump to that saved search.  For
+example, in the default configuration ``j i`` jumps immediately to the
+inbox search.  When you press ``j``, notmuch-jump shows the saved
+searches and their shortcut keys in the mini-buffer.
+
 Configuration
 =
 
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] debian: append elinks to notmuch-vim dependencies

[Sergei Shilovsky]

notmuch-vim uses elinks to convert HTML messages to text.
---
 debian/control | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/debian/control b/debian/control
index 6298deb..146dfea 100644
--- a/debian/control
+++ b/debian/control
@@ -128,7 +128,7 @@ Architecture: all
 Section: mail
 Breaks: notmuch (0.6~254~)
 Replaces: notmuch (0.6~254~)
-Depends: ${misc:Depends}, notmuch, vim-addon-manager, vim-ruby, ruby-notmuch
+Depends: ${misc:Depends}, notmuch, vim-addon-manager, vim-ruby, ruby-notmuch, 
elinks
 Recommends: ruby-mail
 Description: thread-based email index, search and tagging (vim interface)
  Notmuch is a system for indexing, searching, reading, and tagging
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] test: Port atomicity test to Python

[Austin Clements]

Previously, this was implemented using a horrible GDB script (because
there is no such thing as a non-horrible GDB script).  This GDB script
often broke with newer versions of GDB for mysterious reasons.  Port
the test script to GDB's Python API, which makes the code much cleaner
and, hopefully, more stable.
---

Hi Amadeusz.  Does this patch fix the problem for you?  I don't have
GDB 7.8, so I can't test against it, but the Python interface is less
fragile than the GDB scripting language.  (Even if this doesn't fix
your problem, I think we should switch to the Python interface.)

 test/T380-atomicity.sh |  2 +-
 test/atomicity.gdb | 54 --
 test/atomicity.py  | 71 ++
 3 files changed, 72 insertions(+), 55 deletions(-)
 delete mode 100644 test/atomicity.gdb
 create mode 100644 test/atomicity.py

diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh
index 2daef90..ee1e2f4 100755
--- a/test/T380-atomicity.sh
+++ b/test/T380-atomicity.sh
@@ -64,7 +64,7 @@ if test_require_external_prereq gdb; then
 # -tty /dev/null works around a conflict between the 'timeout' wrapper
 # and gdb's attempt to control the TTY.
 export MAIL_DIR
-gdb -tty /dev/null -batch -x $TEST_DIRECTORY/atomicity.gdb notmuch 
1gdb.out 21
+gdb -tty /dev/null -batch -x $TEST_DIRECTORY/atomicity.py notmuch 
1gdb.out 21
 
 # Get the final, golden output
 notmuch search '*'  expected
diff --git a/test/atomicity.gdb b/test/atomicity.gdb
deleted file mode 100644
index 15adb16..000
--- a/test/atomicity.gdb
+++ /dev/null
@@ -1,54 +0,0 @@
-# This gdb script runs notmuch new and simulates killing and
-# restarting notmuch new after every Xapian commit.  To simulate this
-# more efficiently, this script runs notmuch new and, immediately
-# after every Xapian commit, it *pauses* the running notmuch new,
-# copies the entire database and maildir to a snapshot directory, and
-# executes a full notmuch new on that snapshot, comparing the final
-# results with the expected output.  It can then resume the paused
-# notmuch new, which is still running on the original maildir, and
-# repeat this process.
-
-set args new
-
-# Make Xapian commit after every operation instead of batching
-set environment XAPIAN_FLUSH_THRESHOLD = 1
-
-# gdb can't keep track of a simple integer.  This is me weeping.
-shell echo 0  outcount
-
-shell touch inodes
-
-# work around apparent issue with lazy library loading on some
-# platforms
-set breakpoint pending on
-
-break rename
-commands
-# As an optimization, only consider snapshots after a Xapian commit.
-# Xapian overwrites record.base? as the last step in the commit.
-shell echo  gdbcmd
-shell stat -c %i $MAIL_DIR/.notmuch/xapian/record.base*  inodes.new
-shell if cmp inodes inodes.new; then echo cont  gdbcmd; fi
-shell mv inodes.new inodes
-source gdbcmd
-
-# Save a backtrace in case the test does fail
-set logging file backtrace
-set logging on
-backtrace
-set logging off
-shell mv backtrace backtrace.`cat outcount`
-
-# Snapshot the database
-shell rm -r $MAIL_DIR.snap/.notmuch
-shell cp -r $MAIL_DIR/.notmuch $MAIL_DIR.snap/.notmuch
-# Restore the mtime of $MAIL_DIR.snap, which we just changed
-shell touch -r $MAIL_DIR $MAIL_DIR.snap
-# Run notmuch new to completion on the snapshot
-shell NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap XAPIAN_FLUSH_THRESHOLD=1000 
notmuch new  /dev/null
-shell NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap notmuch search '*'  search.`cat 
outcount` 21
-shell echo $(expr $(cat outcount) + 1)  outcount
-cont
-end
-
-run
diff --git a/test/atomicity.py b/test/atomicity.py
new file mode 100644
index 000..01a4205
--- /dev/null
+++ b/test/atomicity.py
@@ -0,0 +1,71 @@
+# This gdb Python script runs notmuch new and simulates killing and
+# restarting notmuch new after every Xapian commit.  To simulate this
+# more efficiently, this script runs notmuch new and, immediately
+# after every Xapian commit, it *pauses* the running notmuch new,
+# copies the entire database and maildir to a snapshot directory, and
+# executes a full notmuch new on that snapshot, comparing the final
+# results with the expected output.  It can then resume the paused
+# notmuch new, which is still running on the original maildir, and
+# repeat this process.
+
+import gdb
+import os
+import glob
+import shutil
+import subprocess
+
+gdb.execute('set args new')
+
+# Make Xapian commit after every operation instead of batching
+gdb.execute('set environment XAPIAN_FLUSH_THRESHOLD = 1')
+
+maildir = os.environ['MAIL_DIR']
+
+# Trap calls to rename, which happens just before Xapian commits
+class RenameBreakpoint(gdb.Breakpoint):
+def __init__(self, *args, **kwargs):
+super(RenameBreakpoint, self).__init__(*args, **kwargs)
+self.last_inodes = {}
+self.n = 0
+
+def stop(self):
+# As an optimization, only consider snapshots after a Xapian
+# has really committed.  Xapian 

Re: [PATCH] test: Port atomicity test to Python

[Austin Clements]

On Fri, 03 Oct 2014, Austin Clements acleme...@csail.mit.edu wrote:
 Previously, this was implemented using a horrible GDB script (because
 there is no such thing as a non-horrible GDB script).  This GDB script
 often broke with newer versions of GDB for mysterious reasons.  Port
 the test script to GDB's Python API, which makes the code much cleaner
 and, hopefully, more stable.
 ---

 Hi Amadeusz.  Does this patch fix the problem for you?  I don't have
 GDB 7.8, so I can't test against it, but the Python interface is less
 fragile than the GDB scripting language.  (Even if this doesn't fix
 your problem, I think we should switch to the Python interface.)

Oops.  I meant this to be a reply to
id:87sijovhuf@freja.aidecoe.name.  Sorry for any confusion.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v6 2/2] nmbug: Add a 'help' command for folks who don't like --help

[W. Trevor King]

The 'if args.func == help' block at the end avoids:

AttributeError: 'functools.partial' object has no attribute '__code__'
---
 devel/nmbug/nmbug | 31 ++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug
index 9402ead..932ec12 100755
--- a/devel/nmbug/nmbug
+++ b/devel/nmbug/nmbug
@@ -32,6 +32,7 @@ from __future__ import unicode_literals
 
 import codecs as _codecs
 import collections as _collections
+import functools as _functools
 import inspect as _inspect
 import locale as _locale
 import logging as _logging
@@ -677,6 +678,24 @@ def _unpack_diff_lines(stream):
 yield (id, tag)
 
 
+def _help(parser, command=None):
+
+Show help for an nmbug command.
+
+Because some folks prefer:
+
+  $ nmbug help COMMAND
+
+to
+
+  $ nmbug COMMAND --help
+
+if command:
+parser.parse_args([command, '--help'])
+else:
+parser.parse_args(['--help'])
+
+
 if __name__ == '__main__':
 import argparse
 
@@ -692,6 +711,8 @@ if __name__ == '__main__':
 help='Log verbosity.  Defaults to {!r}.'.format(
 _logging.getLevelName(_LOG.level).lower()))
 
+help = _functools.partial(_help, parser=parser)
+help.__doc__ = _help.__doc__
 subparsers = parser.add_subparsers(
 title='commands',
 description=(
@@ -703,6 +724,7 @@ if __name__ == '__main__':
 'clone',
 'commit',
 'fetch',
+'help',
 'log',
 'merge',
 'pull',
@@ -746,6 +768,10 @@ if __name__ == '__main__':
 'Override the default configured in branch.name.remote '
 'to fetch from a particular remote repository (e.g. '
 'origin').))
+elif command == 'help':
+subparser.add_argument(
+'command', metavar='COMMAND', nargs='?',
+help='The command to show help for.')
 elif command == 'log':
 subparser.add_argument(
 'args', metavar='ARG', nargs='*',
@@ -796,7 +822,10 @@ if __name__ == '__main__':
 parser.print_usage()
 _sys.exit(1)
 
-(arg_names, varargs, varkw) = _inspect.getargs(args.func.__code__)
+if args.func == help:
+arg_names = ['command']
+else:
+(arg_names, varargs, varkw) = _inspect.getargs(args.func.__code__)
 kwargs = {key: getattr(args, key) for key in arg_names if key in args}
 try:
 args.func(**kwargs)
-- 
2.1.0.60.g85f0837

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v6 0/2] nmbug: Translate to Python

[W. Trevor King]

Hopefully the last round :).  Changes since v5 [1]:

* Dropped 2.6 compatibility claim from the first patch's commit
  message, since Tomi pointed out that at least the '{}'.format()
  syntax is just since 2.7.
* Added a new 'help' command as a separate patch at David's request.

[1]: id:d44bb6ad59ee0a30ac4a8d2e9fe50e3b98d1c408.1411572592.git.wk...@tremily.us
 http://thread.gmane.org/gmane.mail.notmuch.general/19108

W. Trevor King (2):
  nmbug: Translate to Python
  nmbug: Add a 'help' command for folks who don't like --help

 devel/nmbug/nmbug | 1544 +
 1 file changed, 836 insertions(+), 708 deletions(-)

-- 
2.1.0.60.g85f0837

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v6 1/2] nmbug: Translate to Python

[W. Trevor King]

This allows us to capture stdout and stderr separately, and do other
explicit subprocess manipulation without resorting to external
packages.  It should be compatible with Python 2.7 and later
(including the 3.x series).

Most of the user-facing interface is the same, but there are a few
changes, where reproducing the original interface was too difficult or
I saw a change to make the underlying Git UI accessible:

* 'nmbug help' has been split between the general 'nmbug --help' and
  the command-specific 'nmbug COMMAND --help'.

* Commands are no longer split into most common, other useful, and
  less common sets.  If we need something like this, I'd prefer
  workflow examples highlighting common commands in the module
  docstring (available with 'nmbug --help').

* 'nmbug commit' now only uses a single argument for the optional
  commit-message text.  I wanted to expose more of the underlying 'git
  commit' UI, since I personally like to write my commit messages in
  an editor with the notes added by 'git commit -v' to jog my memory.
  Unfortunately, we're using 'git commit-tree' instead of 'git
  commit', and commit-tree is too low-level for editor-launching.  I'd
  be interested in rewriting commit() to use 'git commit', but that
  seemed like it was outside the scope of this rewrite.  So I'm not
  supporting all of Git's commit syntax in this patch, but I can at
  least match 'git commit -m MESSAGE' in requiring command-line commit
  messages to be a single argument.

* The default repository for 'nmbug push' and 'nmbug fetch' is now the
  current branch's upstream (branch.name.remote) instead of
  'origin'.  When we have to, we extract this remote by hand, but
  where possible we just call the Git command without a repository
  argument, and leave it to Git to figure out the default.

* 'nmbug push' accepts multiple refspecs if you want to explicitly
  specify what to push.  Otherwise, the refspec(s) pushed depend on
  push.default.  The Perl version hardcoded 'master' as the pushed
  refspec.

* 'nmbug pull' defaults to the current branch's upstream
  (branch.name.remote and branch.name.merge) instead of hardcoding
  'origin' and 'master'.  It also supports multiple refspecs if for
  some crazy reason you need an octopus merge (but mostly to avoid
  breaking consistency with 'git pull').

* 'nmbug log' now execs 'git log', as there's no need to keep the
  Python process around once we've launched Git there.

* 'nmbug status' now catches stderr, and doesn't print errors like:

No upstream configured for branch 'master'

  The Perl implementation had just learned to avoid crashing on that
  case, but wasn't yet catching the dying subprocess's stderr.

* 'nmbug archive' now accepts positional arguments for the tree-ish
  and additional 'git archive' options.  For example, you can run:

$ nmbug archive HEAD -- --format tar.gz

  I wish I could have preserved the argument order from 'git archive'
  (with the tree-ish at the end), but I'm not sure how to make
  argparse accept arbitrary possitional arguments (some of which take
  arguments).  Flipping the order to put the tree-ish first seemed
  easiest.

* 'nmbug merge' and 'pull' no longer checkout HEAD before running
  their command, because blindly clobbering the index seems overly
  risky.

* In order to avoid creating a dirty index, 'nmbug commit' now uses
  the default index (instead of nmbug.index) for composing the commit.
  That way the index matches the committed tree.  To avoid leaving a
  broken index after a failed commit, I've wrapped the whole thing in
  a try/except block that resets the index to match the pre-commit
  treeish on errors.  That means that 'nmbug commit' will ignore
  anything you've cached in the index via direct Git calls, and you'll
  either end up with an index matching your notmuch tags and the new
  HEAD (after a successful commit) or an index matching the original
  HEAD (after a failed commit).
---
 devel/nmbug/nmbug | 1515 -
 1 file changed, 807 insertions(+), 708 deletions(-)

diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug
index 998ee6b..9402ead 100755
--- a/devel/nmbug/nmbug
+++ b/devel/nmbug/nmbug
@@ -1,708 +1,807 @@
-#!/usr/bin/env perl
-# Copyright (c) 2011 David Bremner
-# License: same as notmuch
-
-use strict;
-use warnings;
-use File::Temp qw(tempdir);
-use Pod::Usage;
-
-no encoding;
-
-my $NMBGIT = $ENV{NMBGIT} || $ENV{HOME}.'/.nmbug';
-
-$NMBGIT .= '/.git' if (-d $NMBGIT.'/.git');
-
-my $TAGPREFIX = defined($ENV{NMBPREFIX}) ? $ENV{NMBPREFIX} : 'notmuch::';
-
-# for encoding
-
-my $ESCAPE_CHAR =  '%';
-my $NO_ESCAPE =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'.
-   '0123456789+-_@=.:,';
-my $MUST_ENCODE =  qr{[^\Q$NO_ESCAPE\E]};
-my $ESCAPED_RX =   qr{$ESCAPE_CHAR([A-Fa-f0-9]{2})};
-
-my %command = (
-archive= \do_archive,
-checkout   = \do_checkout,
-

Re: [PATCH v2] VIM: Use notmuch CLI for config

[Ian Main]

David Bremner wrote:
 Ian Main im...@stemwinder.org writes:
 
  This patch switches from reading .notmuch-config directly to using
  the CLI the same way that emacs does it.  It actually uses less code
  and is probably less error prone.
 
  Ian
 
 The general approach seems sane; it seems quite brittle to read the
 config file directly. I notice there is not really any error handling;
 OTOH, as far as I can read Ruby, there is not any in the previous
 version either.  Technically, this does add a dependency of the vim
 client on the CLI that did not exist before. Personally I don't find
 this onerous (even notmuch-vim users need notmuch new, except in
 rather unusual circumstances.).
 
 I'd like feedback/testing from actual vim interface users before
 merging.

I am actually just following suit on what was already being done.
The Vim client was already calling out to notmuch CLI for other things,
eg:

   system notmuch show --format=mbox id:#{m.message_id}  #{mbox}  #{cmd}

Is used to save the email for display in another program.  Also with no
error checking.  I think basically we are relying on rubys exception handling
to display errors to the user.. not the best idea but it is functional.

I could add a check for 'notmuch' binary.. especially there because loading
the config is the first thing that is done on startup.

Ian
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/2] test: check for debug symbols in notmuch

[David Bremner]

In the future, tests may rely on debug symbols being present in
notmuch, so we plan to switch the default flags.

The main purpose of this test is to help explain the perhaps
mysterious failures of other tests which rely on symbols being
present.
---
 test/T000-basic.sh | 5 +
 1 file changed, 5 insertions(+)

diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index ebbb6d2..bf08f3e 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -91,4 +91,9 @@ test_expect_equal \
 $(dirname ${TEST_DIRECTORY}) \
 $(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')
 
+test_begin_subtest 'notmuch is compiled with debugging symbols'
+test_subtest_known_broken
+readelf --sections $(which notmuch) | grep \.debug
+test_expect_equal 0 $?
+
 test_done
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/2] configure: add debug flags by default.

[David Bremner]

This makes development (in particular the test suite) easier. Those
concerned about the extra diskspace can override the default or use
strip.
---
 configure  | 2 +-
 test/T000-basic.sh | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/configure b/configure
index 86ba2f7..331f29b 100755
--- a/configure
+++ b/configure
@@ -45,7 +45,7 @@ fi
 # environment variables)
 CC=${CC:-cc}
 CXX=${CXX:-c++}
-CFLAGS=${CFLAGS:--O2}
+CFLAGS=${CFLAGS:--g -O2}
 CPPFLAGS=${CPPFLAGS:-}
 CXXFLAGS=${CXXFLAGS:-\$(CFLAGS)}
 LDFLAGS=${LDFLAGS:-}
diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index bf08f3e..ef64245 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -92,7 +92,6 @@ test_expect_equal \
 $(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')
 
 test_begin_subtest 'notmuch is compiled with debugging symbols'
-test_subtest_known_broken
 readelf --sections $(which notmuch) | grep \.debug
 test_expect_equal 0 $?
 
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/2] test: check for debug symbols in notmuch

[Tomi Ollila]

On Fri, Oct 03 2014, David Bremner da...@tethera.net wrote:

 In the future, tests may rely on debug symbols being present in
 notmuch, so we plan to switch the default flags.

 The main purpose of this test is to help explain the perhaps
 mysterious failures of other tests which rely on symbols being
 present.
 ---

series LGTM.

  test/T000-basic.sh | 5 +
  1 file changed, 5 insertions(+)

 diff --git a/test/T000-basic.sh b/test/T000-basic.sh
 index ebbb6d2..bf08f3e 100755
 --- a/test/T000-basic.sh
 +++ b/test/T000-basic.sh
 @@ -91,4 +91,9 @@ test_expect_equal \
  $(dirname ${TEST_DIRECTORY}) \
  $(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')
  
 +test_begin_subtest 'notmuch is compiled with debugging symbols'
 +test_subtest_known_broken
 +readelf --sections $(which notmuch) | grep \.debug
 +test_expect_equal 0 $?
 +
  test_done
 -- 
 2.1.0

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

RE: notmuch-vim doesn't respect notmuch config defaults

[Ian Main]

Sergei Shilovsky wrote:
 I would suggest to use `notmuch config list` to get configuration
 values in vim if possible
 
 Reproduction example follows:
 
  /home/sh cat .notmuch-config
  [new]
  tags=new;inbox;unread
  ignore=
 
  [search]
  exclude_tags=deleted;spam
 
  [maildir]
  synchronize_flags=false
 
  /home/sh notmuch config list
  new.tags=new;inbox;unread
  new.ignore=
  search.exclude_tags=deleted;spam
  maildir.synchronize_flags=false
  database.path=/home/sh/mail
  user.name=Sergei Shilovsky
  user.primary_email=sshilov...@gmail.com
 
  /home/sh vim -c :NotMuch
  Error detected while processing function
 SNR13_NotMuch..SNR13_folders..SNR
  13_new_buffer:
  line6:
  TypeError: no implicit conversion of nil into String
  Error detected while processing function SNR13_NotMuch..SNR13_folders:
  line2:
  NoMethodError: undefined method `query' for nil:NilClass
 
 Having explicit database.path=/home/sh/mail in the configuration file
 makes it work

Right, so you didn't have database.path in your config file before right?
Doesn't the default notmuch setup set that for you?

It wouldn't be too hard to change vim to use 'notmuch config list' to get
its configuration information, but as mentioned a library would be much
nicer.

Anyone have any thoughts on this?  How does the emacs client do it?  If
the config list method is superior I'll write up a patch for it.

   Ian
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[no subject]

[David Bremner]

This is in some sense a successor to 

 id:cover.1411914914.git.j...@nikula.org

It includes the first two patches of that series verbatim, and adds
some tests.



___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v2.5 3/4] cli/insert: require succesful message indexing for success status

[David Bremner]

From: Jani Nikula j...@nikula.org

Add --keep option to keep any remaining stuff in index or file. We
could distinguish between failures to index and failures to apply tags
or maildir sync, but for simplicity just have one.
---
 doc/man1/notmuch-insert.rst | 19 ---
 notmuch-insert.c| 36 ++--
 test/T070-insert.sh |  3 +--
 3 files changed, 43 insertions(+), 15 deletions(-)

diff --git a/doc/man1/notmuch-insert.rst b/doc/man1/notmuch-insert.rst
index 2be1a7b..e396f6c 100644
--- a/doc/man1/notmuch-insert.rst
+++ b/doc/man1/notmuch-insert.rst
@@ -38,16 +38,21 @@ Supported options for **insert** include
 does not exist. Otherwise the folder must already exist for mail
 delivery to succeed.
 
+``--keep``
+Keep the message file if indexing fails, and keep the message
+indexed if applying tags or maildir flag synchronization
+fails. Ignore these errors and return exit status 0 to
+indicate succesful mail delivery.
+
 EXIT STATUS
 ===
 
-This command returns exit status 0 if the message was successfully added
-to the mail directory, even if the message could not be indexed and
-added to the notmuch database. In the latter case, a warning will be
-printed to standard error but the message file will be left on disk.
-
-If the message could not be written to disk then a non-zero exit status
-is returned.
+This command returns exit status 0 on succesful mail delivery,
+non-zero otherwise. The default is to indicate failed mail delivery on
+any errors, including message file delivery to the filesystem, message
+indexing to Notmuch database, changing tags, and synchronizing tags to
+maildir flags. The ``--keep`` option may be used to settle for
+successful message file delivery.
 
 SEE ALSO
 
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 0ea4380..7074077 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -443,6 +443,7 @@ add_file (notmuch_database_t *notmuch, const char *path, 
tag_op_list_t *tag_ops,
 int
 notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
 {
+notmuch_status_t status, close_status;
 notmuch_database_t *notmuch;
 struct sigaction action;
 const char *db_path;
@@ -452,6 +453,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 char *query_string = NULL;
 const char *folder = NULL;
 notmuch_bool_t create_folder = FALSE;
+notmuch_bool_t keep = FALSE;
 notmuch_bool_t synchronize_flags;
 const char *maildir;
 char *newpath;
@@ -461,6 +463,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc, 
char *argv[])
 notmuch_opt_desc_t options[] = {
{ NOTMUCH_OPT_STRING, folder, folder, 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, create_folder, create-folder, 0, 0 },
+   { NOTMUCH_OPT_BOOLEAN, keep, keep, 0, 0 },
{ NOTMUCH_OPT_END, 0, 0, 0, 0 }
 };
 
@@ -535,11 +538,32 @@ notmuch_insert_command (notmuch_config_t *config, int 
argc, char *argv[])
return EXIT_FAILURE;
 }
 
-/* Add the message to the index.
- * Even if adding the message to the notmuch database fails,
- * the message is on disk and we consider the delivery completed. */
-add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
+/* Index the message. */
+status = add_file (notmuch, newpath, tag_ops, synchronize_flags, keep);
+
+/* Commit changes. */
+close_status = notmuch_database_destroy (notmuch);
+if (close_status) {
+   /* Hold on to the first error, if any. */
+   if (! status)
+   status = close_status;
+   fprintf (stderr, %s: failed to commit database changes: %s\n,
+keep ? Warning : Error,
+notmuch_status_to_string (close_status));
+}
+
+if (status) {
+   if (keep) {
+   status = NOTMUCH_STATUS_SUCCESS;
+   } else {
+   /* If maildir flag sync failed, this might fail. */
+   if (unlink (newpath)) {
+   fprintf (stderr, Warning: failed to remove '%s' from maildir 
+after errors: %s. Please run 'notmuch new' to fix.\n,
+newpath, strerror (errno));
+   }
+   }
+}
 
-notmuch_database_destroy (notmuch);
-return EXIT_SUCCESS;
+return status ? EXIT_FAILURE : EXIT_SUCCESS;
 }
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index 40a7636..be8060e 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -23,7 +23,7 @@ test_expect_code 1 Insert zero-length file \
 
 # This test is a proxy for other errors that may occur while trying to
 # add a message to the notmuch database, e.g. database locked.
-test_expect_code 0 Insert non-message \
+test_expect_code 1 Insert non-message \
 echo bad_message | notmuch insert
 
 test_begin_subtest Database empty so far
@@ -199,7 +199,6 @@ end
 run
 EOF
 test_begin_subtest error exit when add_message 

[Patch v2.5 4/4] test/insert: check that indexing errors are accepted with --keep

[David Bremner]

This is overkill for the current code path, but should provide some
robustness for future changes in error handling.
---
 test/T070-insert.sh | 5 +
 1 file changed, 5 insertions(+)

diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index be8060e..80a22c1 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -202,6 +202,11 @@ test_begin_subtest error exit when add_message returns 
$code
 gdb --batch-silent --return-child-result -x index-file-$code.gdb \
 --args notmuch insert   $gen_msg_filename
 test_expect_equal $? 1
+
+test_begin_subtest success exit with --keep when add_message returns $code
+gdb --batch-silent --return-child-result -x index-file-$code.gdb \
+--args notmuch insert --keep   $gen_msg_filename
+test_expect_equal $? 0
 done
 
 test_done
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v2.5 1/4] test/insert: add known broken tests for indexing failures

[David Bremner]

These tests are written with the assumption that we want all indexing
failures to be considered as failures by notmuch insert.
---
 test/T070-insert.sh | 22 ++
 1 file changed, 22 insertions(+)

diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index ea9db07..40a7636 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -183,4 +183,26 @@ test_expect_code 1 Invalid tags set exit code \
 
 notmuch config set new.tags $OLDCONFIG
 
+# DUPLICATE_MESSAGE_ID is not tested here, because it should actually pass.
+
+for code in OUT_OF_MEMORY XAPIAN_EXCEPTION FILE_NOT_EMAIL \
+READ_ONLY_DATABASE UPGRADE_REQUIRED; do
+gen_insert_msg
+cat EOF  index-file-$code.gdb
+file notmuch
+set breakpoint pending on
+break notmuch_database_add_message
+commands
+return NOTMUCH_STATUS_$code
+continue
+end
+run
+EOF
+test_begin_subtest error exit when add_message returns $code
+test_subtest_known_broken
+gdb --batch-silent --return-child-result -x index-file-$code.gdb \
+--args notmuch insert   $gen_msg_filename
+test_expect_equal $? 1
+done
+
 test_done
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[Patch v2.5 2/4] cli/insert: add fail path to add_file_to_database

[David Bremner]

From: Jani Nikula j...@nikula.org

Handle failures gracefully in add_file_to_database, renamed simply
add_file while at it. Add keep option to not remove the message from
database if tagging or tag syncing to maildir flags fails. Expand the
function documentation to cover the changes.
---
 notmuch-insert.c | 99 
 1 file changed, 64 insertions(+), 35 deletions(-)

diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5ef6e66..0ea4380 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -364,50 +364,80 @@ FAIL:
 return NULL;
 }
 
-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+/*
+ * Add the specified message file to the notmuch database, applying
+ * tags in tag_ops. If synchronize_flags is TRUE, the tags are
+ * synchronized to maildir flags (which may result in message file
+ * rename).
+ *
+ * Return NOTMUCH_STATUS_SUCCESS on success, errors otherwise. If keep
+ * is TRUE, errors in tag changes and flag syncing are ignored and
+ * success status is returned; otherwise such errors cause the message
+ * to be removed from the database. Failure to add the message to the
+ * database results in error status regardless of keep.
+ */
+static notmuch_status_t
+add_file (notmuch_database_t *notmuch, const char *path, tag_op_list_t 
*tag_ops,
+ notmuch_bool_t synchronize_flags, notmuch_bool_t keep)
 {
 notmuch_message_t *message;
 notmuch_status_t status;
 
 status = notmuch_database_add_message (notmuch, path, message);
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
-   break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
-   fprintf (stderr, Error: failed to add `%s' to notmuch database: %s\n,
-path, notmuch_status_to_string (status));
-   return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
-   /* Don't change tags of an existing message. */
-   if (synchronize_flags) {
-   status = notmuch_message_tags_to_maildir_flags (message);
-   if (status != NOTMUCH_STATUS_SUCCESS)
-   fprintf (stderr, Error: failed to sync tags to maildir 
flags\n);
+if (status == NOTMUCH_STATUS_SUCCESS) {
+   status = tag_op_list_apply (message, tag_ops, 0);
+   if (status) {
+   fprintf (stderr, %s: failed to apply tags to file '%s': %s\n,
+keep ? Warning : Error,
+path, notmuch_status_to_string (status));
+   goto DONE;
}
+} else if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+   status = NOTMUCH_STATUS_SUCCESS;
+} else if (status == NOTMUCH_STATUS_FILE_NOT_EMAIL) {
+   fprintf (stderr, Error: delivery of non-mail file: '%s'\n, path);
+   goto FAIL;
 } else {
-   tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+   fprintf (stderr, Error: failed to add '%s' to notmuch database: %s\n,
+path, notmuch_status_to_string (status));
+   goto FAIL;
+}
 
-   tag_op_list_apply (message, tag_ops, flags);
+if (synchronize_flags) {
+   status = notmuch_message_tags_to_maildir_flags (message);
+   if (status != NOTMUCH_STATUS_SUCCESS)
+   fprintf (stderr, %s: failed to sync tags to maildir flags for 
'%s': %s\n,
+keep ? Warning : Error,
+path, notmuch_status_to_string (status));
+
+   /*
+* Note: Unfortunately a failed maildir flag sync might
+* already have renamed the file, in which case the cleanup
+* path may fail.
+*/
 }
 
+  DONE:
 notmuch_message_destroy (message);
+
+if (status) {
+   if (keep) {
+   status = NOTMUCH_STATUS_SUCCESS;
+   } else {
+   notmuch_status_t cleanup_status;
+
+   cleanup_status = notmuch_database_remove_message (notmuch, path);
+   if (cleanup_status != NOTMUCH_STATUS_SUCCESS 
+   cleanup_status != NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+   fprintf (stderr, Warning: failed to remove '%s' from database 
+after errors: %s. Please run 'notmuch new' to fix.\n,
+path, notmuch_status_to_string (cleanup_status));
+   }
+   }
+}
+
+  FAIL:
+return status;
 }
 
 int
@@ -508,8 

Re:

[David Bremner]

David Bremner da...@tethera.net writes:

 This is in some sense a successor to 

  id:cover.1411914914.git.j...@nikula.org

 It includes the first two patches of that series verbatim, and adds
 some tests.

I should have said _almost_ verbatim; it marks some tests non-broken.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 07/11] lib: Implement ghost-based thread linking

[Carl Worth]

On Fri, Oct 03 2014, Austin Clements wrote:
 This updates the thread linking code to use ghost messages instead of
 user metadata to link messages into threads.

 In contrast with the old approach, this is actually correct.

Hi Austin,

I've read through your commit messages, (not the code), and I just
wanted to say a big thank you to you!

I'm really looking forward to the improved robustness in notmuch with
this support.

I really appreciate your diligence in working through a fairly large,
low-level change here.

-Carl


pgpavnlvD49KS.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Add default configuration values to the man page

[David Bremner]

Sergei Shilovsky sshilov...@gmail.com writes:

  **new.tags**
  A list of tags that will be added to all messages incorporated
  by **notmuch new**.
  
 +Default: ``unread;inbox``.
 +

There is actually something a bit subtle about some of these defaults.
Some of them are applied only to newly created configuration files,
while others are applied if there is now config value in the file.

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] NEWS: News for notmuch-jump

[David Bremner]

Austin Clements acleme...@csail.mit.edu writes:

 ---
  NEWS | 15 +++
  1 file changed, 15 insertions(+)

pushed, with thanks.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch