Do path: searches handle spaces correctly?
On Mon, 2014-09-22 at 15:06 +, Austin Clements wrote: > I assume you're doing this from the command line? Does the following > work? > > notmuch search --output=files 'path:"dir/INBOX/INBOX/Sent Items/**"' > > Shell quoting and Xapian quoting interact in often confusing ways. In > your original command line, the single quotes suppress shell argument > splitting, but never make it to notmuch, so notmuch sees two search > terms "path:dir/INBOX/INBOX/Sent" and "Items/**". In the command line > I suggested, the single quotes play the same role, but for the entire > query. Because of the quoting, notmuch *does* see the inner double > quotes, which makes the path a single term (note that Xapian only > accepts double quotes, not single quotes). In general, it's good to > enclose the entire command line query in single quotes so shell > parsing doesn't get in the way of query parsing. Bingo. That was it. Thanks for clarifying how the quoting interacts. What do you think about adding a section to the notmuch-search-terms web page that talks in more detail about the quoting interactions? If there were on obvious section about this in the man page it might have helped me find the issue myself. If this seems like a good idea I'll take a crack at it later this week when I get time to research the Xapian quoting in a bit more detail. --- Keith
[PATCH] emacs: jump: fix compile warning on emacs 23
LGTM. I'm a little surprised this is necessary, but whatever. I think the eval-and-compile has to be top-level; it's certainly not wrong for it to be top-level. (I like the comment in the eval-and-compile implementation: ";; Remember, it's magic.") On Thu, 04 Sep 2014, Mark Walters wrote: > notmuch-jump uses window-body-width which is not defined in emacs > 23. To get around this it does > > (unless (fboundp 'window-body-width) > ;; Compatibility for Emacs pre-24 > (defalias 'window-body-width 'window-width)) > > This makes sure window-body-width is defined and all should be > well. But it seems that the byte compiler does not realise that this > guarantees that window-body-width will be defined and so, when > compiling with emacs 23, it gives an error > > In end of data: > notmuch-jump.el:172:1:Warning: the function `window-body-width' is not known > to be defined. > > Domo and I came to following on irc: wrap the (unless (fboundp ...)) > inside eval-and-compile which ensures that both the test and the > defalias (if needed) happen at both compile and load time. This fixes > the warning. > --- > I think Domo and I were both not completely sure whether the > eval-and-compile should be inside or outside the (unless fboundp ..) > or not (ie should it wrap the unless fboundp or just the defalias) nor > whether it should be eval-and-compile or eval-when-compile. > > We think this is the right version but it would be good to have confirmation. > > I tested notmuch jump inside emacs 23 and 24 with notmuch-emacs > compiled with emacs 23 or 24 (ie all four combinations) and it seemed to work. > > Best wishes > > Mark > > > > emacs/notmuch-jump.el |7 --- > 1 file changed, 4 insertions(+), 3 deletions(-) > > diff --git a/emacs/notmuch-jump.el b/emacs/notmuch-jump.el > index 5eb0949..2706b6c 100644 > --- a/emacs/notmuch-jump.el > +++ b/emacs/notmuch-jump.el > @@ -25,9 +25,10 @@ > (require 'notmuch-lib) > (require 'notmuch-hello) > > -(unless (fboundp 'window-body-width) > - ;; Compatibility for Emacs pre-24 > - (defalias 'window-body-width 'window-width)) > +(eval-and-compile > + (unless (fboundp 'window-body-width) > +;; Compatibility for Emacs pre-24 > +(defalias 'window-body-width 'window-width))) > > ;;;###autoload > (defun notmuch-jump-search () > -- > 1.7.10.4 > > ___ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] lib: Simplify close and codify aborting atomic section
Quoth W. Trevor King on Sep 22 at 9:59 am: > On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote: > > This patch simplifies notmuch_database_close to just call > > Database::close. This works for both read-only and read/write > > databases, takes care of committing changes, unifies the exception > > handling path, and codifies aborting outstanding transactions. > > If we're dropping the flush call here, where will it be triggered > instead? We'll need to flush/commit our changes to the database at > some point before closing. Do clients now need an explicit > flush/commit command (explicit, client-initiated flushes sound like a > good idea to me). The call to Database::close implicitly flushes/commits, as mentioned in the comment in the patch, so there's no need for any new APIs or client changes. The call to Database::flush in notmuch_database_close was entirely redundant with the call to Database::close.
[PATCH] Add configurable changed tag to messages that have been changed on disk
On Mon, Sep 22 2014, Gaute Hope wrote: > Excerpts from Gaute Hope's message of August 6, 2014 10:29: >> Austin Clements wrote on Fri, 01 Aug 2014 14:55:05 >> -0400: >>> I have a prototype implementation of message modification times on my >>> lastmod-v1 branch at >>> >>> https://github.com/aclements/notmuch/tree/lastmod-v1 >>> >>> It builds on my database features series that's currently awaiting >>> review [1]. >>> >>> The series uses a monotonic revision number, rather than wall-clock >>> time, for reasons related to Xapian's concurrent control and detailed >>> in the main commit's commit message. The implementation isn't quite >>> useful from the CLI yet because I haven't added any way to query the >>> database's current revision number. (I'm still thinking about how I >>> want to do this, since search/show don't have a good way to deliver >>> "additional" information right now. I might just add the last >>> modification for each individual message/max of all messages in a >>> thread, similar to what Thomas Jost's patch did long ago.) >>> >>> [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu > >> this should allow me to do what I wish to accomplish. The message >> deletion is still a problem though, I can see two options at the moment: > > Hi list, > > While exploring the possibility of syncing maildir/X-keywords with tags > I had some thoughts about lastmod and message modification: > > As briefly discussed on #notmuch, I noticed that it seems that 'notmuch > new' does not detect that a message source has been changed, unless the > file is also re-named. > > This means that for instance if the X-Keywords fields have been updated > in a message (from GMail with offlineimap, synclabels = yes) the lastmod > field will remain unchanged, and a source modification will be > undetectable to a client program using this value. > > Would it not make sense that if a message has a more recent mtime than > at index time it is re-indexed? That would require notmuch to scan the contents of a directory for changed mtimes of the files -- now notmuch skips looking for the files unless directory mtime has changed. Directory mtime changes when files are added/deleted/renamed (as that is what directory needs to know) -- file mtime change are stored in file information and therefore change there does not probagate to parent directory (and, if such happened, to it's parent and so on...) That would mean the scanning would be slower than it is now. Tomi > > Also, for the lastmod branch I would wish for a notmuch_message_touch() > method where the lastmod time is updated to the last. As well as a > notmuch_database_reindex_message () - possibly defined/documented > behaviour for notmuch_database_add_message () when the filename is > already added (in which case I would expect notmuch to re-index the > message). > > Doing notmuch_database_remove_message followed by _add_message could > risk deleting the entry if this file is the only on-disk-representation. > > Cheers, Gaute
Do path: searches handle spaces correctly?
Quoth Keith Amidon on Sep 22 at 7:42 am: > On Mon, 2014-09-22 at 11:20 +0200, David Bremner wrote: > > Keith Amidon writes: > > > > > > notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items' > > > > > > I don't get any results, but it seems like the two results above should > > > be shown, right? Am I doing something wrong here? If it looks like I'm > > > doing it correctly, what can I do to help troubleshoot the issue? > > > > Note that path:, unlike folder:, does not add the maildir subdirs. Dunno > > if that's the only issue you are having, but I guess it's one. > > Darn it! I made a mistake in my original email. In the test I was > doing I actually had: > > notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items/**' > > which I believe should have picked up all the subdirectory paths. I > just retested to make sure that it still didn't work, and it doesn't. > > Am I still missing something? Thanks for the help, Keith I assume you're doing this from the command line? Does the following work? notmuch search --output=files 'path:"dir/INBOX/INBOX/Sent Items/**"' Shell quoting and Xapian quoting interact in often confusing ways. In your original command line, the single quotes suppress shell argument splitting, but never make it to notmuch, so notmuch sees two search terms "path:dir/INBOX/INBOX/Sent" and "Items/**". In the command line I suggested, the single quotes play the same role, but for the entire query. Because of the quoting, notmuch *does* see the inner double quotes, which makes the path a single term (note that Xapian only accepts double quotes, not single quotes). In general, it's good to enclose the entire command line query in single quotes so shell parsing doesn't get in the way of query parsing.
[PATCH] Add configurable changed tag to messages that have been changed on disk
Excerpts from Gaute Hope's message of August 6, 2014 10:29: > Austin Clements wrote on Fri, 01 Aug 2014 14:55:05 > -0400: >> I have a prototype implementation of message modification times on my >> lastmod-v1 branch at >> >> https://github.com/aclements/notmuch/tree/lastmod-v1 >> >> It builds on my database features series that's currently awaiting >> review [1]. >> >> The series uses a monotonic revision number, rather than wall-clock >> time, for reasons related to Xapian's concurrent control and detailed >> in the main commit's commit message. The implementation isn't quite >> useful from the CLI yet because I haven't added any way to query the >> database's current revision number. (I'm still thinking about how I >> want to do this, since search/show don't have a good way to deliver >> "additional" information right now. I might just add the last >> modification for each individual message/max of all messages in a >> thread, similar to what Thomas Jost's patch did long ago.) >> >> [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu > this should allow me to do what I wish to accomplish. The message > deletion is still a problem though, I can see two options at the moment: Hi list, While exploring the possibility of syncing maildir/X-keywords with tags I had some thoughts about lastmod and message modification: As briefly discussed on #notmuch, I noticed that it seems that 'notmuch new' does not detect that a message source has been changed, unless the file is also re-named. This means that for instance if the X-Keywords fields have been updated in a message (from GMail with offlineimap, synclabels = yes) the lastmod field will remain unchanged, and a source modification will be undetectable to a client program using this value. Would it not make sense that if a message has a more recent mtime than at index time it is re-indexed? Also, for the lastmod branch I would wish for a notmuch_message_touch() method where the lastmod time is updated to the last. As well as a notmuch_database_reindex_message () - possibly defined/documented behaviour for notmuch_database_add_message () when the filename is already added (in which case I would expect notmuch to re-index the message). Doing notmuch_database_remove_message followed by _add_message could risk deleting the entry if this file is the only on-disk-representation. Cheers, Gaute
[PATCH] lib: Simplify close and codify aborting atomic section
On Mon, Sep 22, 2014 at 06:50:50PM +, Austin Clements wrote: > Quoth W. Trevor King on Sep 22 at 9:59 am: > > On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote: > > > This patch simplifies notmuch_database_close to just call > > > Database::close. This works for both read-only and read/write > > > databases, takes care of committing changes, unifies the > > > exception handling path, and codifies aborting outstanding > > > transactions. > > > > If we're dropping the flush call here, where will it be triggered > > instead? We'll need to flush/commit our changes to the database > > at some point before closing. Do clients now need an explicit > > flush/commit command (explicit, client-initiated flushes sound > > like a good idea to me). > > The call to Database::close implicitly flushes/commits, as mentioned > in the comment in the patch, so there's no need for any new APIs or > client changes. The call to Database::flush in > notmuch_database_close was entirely redundant with the call to > Database::close. Ah, I thought the implicit flush/commit was just in our code. Since it's also in the underlying Xapian close, then this patch looks pretty good to me. I'd mention Xapian's explicit close in the notmuch.h message. Xapain's docs say [1]: For a WritableDatabase, if a transaction is active it will be aborted, while if no transaction is active commit() will be implicitly called. Cheers, Trevor [1]: http://xapian.org/docs/apidoc/html/classXapian_1_1Database.html#a59f5f8b137723dcaaabdbdccbc0cf1eb -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140922/4cb115a4/attachment.pgp>
[PATCH 11/11] cli/insert: add post-insert hook
The post-new hook might no longer be needed or run very often if
notmuch insert is being used. Therefore a post-insert hook is needed
(arguably pre-insert not so much, so don't add one). Also add the
--no-hooks option to skip hooks.
---
notmuch-insert.c |7 +++
1 files changed, 7 insertions(+), 0 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index f27b9cb..adadd12 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -444,6 +444,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
const char *folder = NULL;
notmuch_bool_t create_folder = FALSE;
notmuch_bool_t keep = FALSE;
+notmuch_bool_t no_hooks = FALSE;
notmuch_bool_t synchronize_flags;
const char *maildir;
char *newpath;
@@ -454,6 +455,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
{ NOTMUCH_OPT_STRING, , "folder", 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, _folder, "create-folder", 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, , "keep", 0, 0 },
+ { NOTMUCH_OPT_BOOLEAN, _hooks, "no-hooks", 'n', 0 },
{ NOTMUCH_OPT_END, 0, 0, 0, 0 }
};
@@ -541,5 +543,10 @@ notmuch_insert_command (notmuch_config_t *config, int
argc, char *argv[])
notmuch_database_destroy (notmuch);
+if (! no_hooks && status == NOTMUCH_STATUS_SUCCESS) {
+ /* Ignore hook failures. */
+ notmuch_run_hook (db_path, "post-insert");
+}
+
return status ? EXIT_FAILURE : EXIT_SUCCESS;
}
--
1.7.2.5
[PATCH 10/11] cli/insert: require succesful message indexing for success status
Add --keep option to keep any remaining stuff in index or file. We
could distinguish between failures to index and failures to apply tags
or maildir sync, but for simplicity just have one.
---
notmuch-insert.c| 20 +++-
test/T070-insert.sh |2 +-
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 80f52d4..f27b9cb 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -433,6 +433,7 @@ DONE:
int
notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
{
+notmuch_status_t status;
notmuch_database_t *notmuch;
struct sigaction action;
const char *db_path;
@@ -442,6 +443,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
char *query_string = NULL;
const char *folder = NULL;
notmuch_bool_t create_folder = FALSE;
+notmuch_bool_t keep = FALSE;
notmuch_bool_t synchronize_flags;
const char *maildir;
char *newpath;
@@ -451,6 +453,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
notmuch_opt_desc_t options[] = {
{ NOTMUCH_OPT_STRING, , "folder", 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, _folder, "create-folder", 0, 0 },
+ { NOTMUCH_OPT_BOOLEAN, , "keep", 0, 0 },
{ NOTMUCH_OPT_END, 0, 0, 0, 0 }
};
@@ -525,11 +528,18 @@ notmuch_insert_command (notmuch_config_t *config, int
argc, char *argv[])
return EXIT_FAILURE;
}
-/* Add the message to the index.
- * Even if adding the message to the notmuch database fails,
- * the message is on disk and we consider the delivery completed. */
-add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
+/* Index the message. */
+status = add_file (notmuch, newpath, tag_ops, synchronize_flags, keep);
+if (status) {
+ if (keep) {
+ status = NOTMUCH_STATUS_SUCCESS;
+ } else {
+ /* If maildir flag sync failed, this might fail. */
+ unlink (newpath);
+ }
+}
notmuch_database_destroy (notmuch);
-return EXIT_SUCCESS;
+
+return status ? EXIT_FAILURE : EXIT_SUCCESS;
}
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index ea9db07..aacc643 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -23,7 +23,7 @@ test_expect_code 1 "Insert zero-length file" \
# This test is a proxy for other errors that may occur while trying to
# add a message to the notmuch database, e.g. database locked.
-test_expect_code 0 "Insert non-message" \
+test_expect_code 1 "Insert non-message" \
"echo bad_message | notmuch insert"
test_begin_subtest "Database empty so far"
--
1.7.2.5
[PATCH 09/11] cli/insert: add fail path to add_file_to_database
Handle failures gracefully in add_file_to_database, renamed simply
add_file while at it. Add keep option to not remove the message from
database if tagging or tag syncing to maildir flags fails. Expand the
function documentation to cover the changes.
---
notmuch-insert.c | 89 -
1 files changed, 54 insertions(+), 35 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5ef6e66..80f52d4 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -364,50 +364,70 @@ FAIL:
return NULL;
}
-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+/*
+ * Add the specified message file to the notmuch database, applying
+ * tags in tag_ops. If synchronize_flags is TRUE, the tags are
+ * synchronized to maildir flags (which may result in message file
+ * rename).
+ *
+ * Return NOTMUCH_STATUS_SUCCESS on success, errors otherwise. If keep
+ * is TRUE, errors in tag changes and flag syncing are ignored and
+ * success status is returned; otherwise such errors cause the message
+ * to be removed from the database. Failure to add the message to the
+ * database results in error status regardless of keep.
+ */
+static notmuch_status_t
+add_file (notmuch_database_t *notmuch, const char *path, tag_op_list_t
*tag_ops,
+ notmuch_bool_t synchronize_flags, notmuch_bool_t keep)
{
notmuch_message_t *message;
notmuch_status_t status;
status = notmuch_database_add_message (notmuch, path, );
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
- fprintf (stderr, "Error: failed to add `%s' to notmuch database: %s\n",
-path, notmuch_status_to_string (status));
- return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
- /* Don't change tags of an existing message. */
- if (synchronize_flags) {
- status = notmuch_message_tags_to_maildir_flags (message);
- if (status != NOTMUCH_STATUS_SUCCESS)
- fprintf (stderr, "Error: failed to sync tags to maildir
flags\n");
+if (status == NOTMUCH_STATUS_SUCCESS) {
+ status = tag_op_list_apply (message, tag_ops, 0);
+ if (status) {
+ fprintf (stderr, "%s: failed to apply tags to file '%s': %s\n",
+keep ? "Warning" : "Error",
+path, notmuch_status_to_string (status));
+ goto DONE;
}
+} else if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+ status = NOTMUCH_STATUS_SUCCESS;
+} else if (status == NOTMUCH_STATUS_FILE_NOT_EMAIL) {
+ fprintf (stderr, "Error: delivery of non-mail file: '%s'\n", path);
+ return status;
} else {
- tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+ fprintf (stderr, "Error: failed to add '%s' to notmuch database: %s\n",
+path, notmuch_status_to_string (status));
+ return status;
+}
- tag_op_list_apply (message, tag_ops, flags);
+if (synchronize_flags) {
+ status = notmuch_message_tags_to_maildir_flags (message);
+ if (status != NOTMUCH_STATUS_SUCCESS)
+ fprintf (stderr, "%s: failed to sync tags to maildir flags for
'%s': %s\n",
+keep ? "Warning" : "Error",
+path, notmuch_status_to_string (status));
+
+ /*
+* Note: Unfortunately a failed maildir flag sync might
+* already have renamed the file, in which case the cleanup
+* path will fail.
+*/
}
+DONE:
notmuch_message_destroy (message);
+
+if (status) {
+ if (keep)
+ status = NOTMUCH_STATUS_SUCCESS;
+ else
+ notmuch_database_remove_message (notmuch, path);
+}
+
+return status;
}
int
@@ -508,8 +528,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
/* Add the message to the index.
* Even if adding the message to the notmuch database fails,
* the message is on disk and we consider the delivery completed. */
-add_file_to_database (notmuch, newpath, tag_ops,
- synchronize_flags);
+add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
notmuch_database_destroy (notmuch);
[PATCH 08/11] cli/insert: rehash file writing functions
Make the function calls make more sense as independent building blocks
of the big picture, with clear inputs and outputs. Split up
write_message into two. Improve function documentation. Cleanup and
clarify the error paths.
---
notmuch-insert.c | 127 --
1 files changed, 75 insertions(+), 52 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index a1d564c..5ef6e66 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -204,47 +204,37 @@ tempfilename (const void *ctx)
return filename;
}
-/* Open a unique file in the 'tmp' sub-directory of dir.
- * Returns the file descriptor on success, or -1 on failure.
- * On success, file paths for the message in the 'tmp' and 'new'
- * directories are returned via tmppath and newpath,
- * and the path of the 'new' directory itself in newdir. */
+/*
+ * Create a unique temporary file in maildir/tmp, return fd and full
+ * path to file in *path_out, or -1 on errors (in which case *path_out
+ * is not touched).
+ */
static int
-maildir_open_tmp_file (void *ctx, const char *dir,
- char **tmppath, char **newpath, char **newdir)
+maildir_mktemp (const void *ctx, const char *maildir, char **path_out)
{
-char *filename;
-int fd = -1;
+char *filename, *path;
+int fd;
do {
filename = tempfilename (ctx);
if (! filename)
return -1;
- *tmppath = talloc_asprintf (ctx, "%s/tmp/%s", dir, filename);
- if (! *tmppath) {
- fprintf (stderr, "Out of memory\n");
+ path = talloc_asprintf (ctx, "%s/tmp/%s", maildir, filename);
+ if (! path) {
+ fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
return -1;
}
- fd = open (*tmppath, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0600);
+ fd = open (path, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0600);
} while (fd == -1 && errno == EEXIST);
if (fd == -1) {
- fprintf (stderr, "Error: opening %s: %s\n", *tmppath, strerror (errno));
+ fprintf (stderr, "Error: open '%s': %s\n", path, strerror (errno));
return -1;
}
-*newdir = talloc_asprintf (ctx, "%s/new", dir);
-*newpath = talloc_asprintf (ctx, "%s/new/%s", dir, filename);
-if (! *newdir || ! *newpath) {
- fprintf (stderr, "Out of memory\n");
- close (fd);
- unlink (*tmppath);
- return -1;
-}
-
-talloc_free (filename);
+*path_out = path;
return fd;
}
@@ -293,53 +283,85 @@ copy_fd (int fdout, int fdin)
return (!interrupted && !empty);
}
-static notmuch_bool_t
-write_message (void *ctx, int fdin, const char *dir, char **newpath)
+/*
+ * Write fdin to a new temp file in maildir/tmp, return full path to
+ * the file, or NULL on errors.
+ */
+static char *
+maildir_write_tmp (const void *ctx, int fdin, const char *maildir)
{
-char *tmppath;
-char *newdir;
-char *cleanup_path;
+char *path;
int fdout;
-fdout = maildir_open_tmp_file (ctx, dir, , newpath, );
+fdout = maildir_mktemp (ctx, maildir, );
if (fdout < 0)
- return FALSE;
-
-cleanup_path = tmppath;
+ return NULL;
if (! copy_fd (fdout, fdin))
goto FAIL;
-if (fsync (fdout) != 0) {
- fprintf (stderr, "Error: fsync failed: %s\n", strerror (errno));
+if (fsync (fdout)) {
+ fprintf (stderr, "Error: fsync '%s': %s\n", path, strerror (errno));
goto FAIL;
}
close (fdout);
-fdout = -1;
-
-/* Atomically move the new message file from the Maildir 'tmp' directory
- * to the 'new' directory. We follow the Dovecot recommendation to
- * simply use rename() instead of link() and unlink().
- * See also: http://wiki.dovecot.org/MailboxFormat/Maildir#Mail_delivery
- */
-if (rename (tmppath, *newpath) != 0) {
- fprintf (stderr, "Error: rename() failed: %s\n", strerror (errno));
+
+return path;
+
+FAIL:
+close (fdout);
+unlink (path);
+
+return NULL;
+}
+
+/*
+ * Write fdin to a new file in maildir/new, using an intermediate temp
+ * file in maildir/tmp, return full path to the new file, or NULL on
+ * errors.
+ */
+static char *
+maildir_write_new (const void *ctx, int fdin, const char *maildir)
+{
+char *cleanpath, *tmppath, *newpath, *newdir;
+
+tmppath = maildir_write_tmp (ctx, fdin, maildir);
+if (! tmppath)
+ return NULL;
+cleanpath = tmppath;
+
+newpath = talloc_strdup (ctx, tmppath);
+if (! newpath) {
+ fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
goto FAIL;
}
-cleanup_path = *newpath;
+/* sanity checks needed? */
+memcpy (newpath + strlen (maildir) + 1, "new", 3);
+
+if (rename (tmppath, newpath)) {
+ fprintf (stderr, "Error: rename '%s' '%s': %s\n",
+tmppath, newpath, strerror (errno));
+ goto FAIL;
+}
+cleanpath = newpath;
+
+newdir = talloc_asprintf (ctx, "%s/%s", maildir, "new");
+
[PATCH 06/11] cli/insert: use a single recursive mkdir function
Combine make_directory() and make_directory_and_parents() into a
single recursive mkdir_recursive() function. Clarify the code and
improve error handling. Improve error messages. Switch to using the
new function in maildir_create_folder(). Constify talloc context.
---
notmuch-insert.c | 131 +++---
1 files changed, 55 insertions(+), 76 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 7375c54..cdeeb41 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -104,96 +104,78 @@ is_valid_folder_name (const char *folder)
}
}
-/* Make the given directory, succeeding if it already exists. */
+/*
+ * Make the given directory and its parents as necessary, using the
+ * given mode. Return TRUE on success, FALSE otherwise. Partial
+ * results are not cleaned up on errors.
+ */
static notmuch_bool_t
-make_directory (char *path, int mode)
+mkdir_recursive (const void *ctx, const char *path, int mode)
{
-notmuch_bool_t ret;
-char *slash;
+struct stat st;
+int r;
+char *parent = NULL, *slash;
-if (mkdir (path, mode) != 0)
- return (errno == EEXIST);
+/* First check the common case: directory already exists. */
+r = stat (path, );
+if (r == 0) {
+if (! S_ISDIR (st.st_mode)) {
+ fprintf (stderr, "Error: '%s' is not a directory: %s\n",
+path, strerror (EEXIST));
+ return FALSE;
+ }
-/* Sync the parent directory for durability. */
-ret = TRUE;
-slash = strrchr (path, '/');
-if (slash) {
- *slash = '\0';
- ret = sync_dir (path);
- *slash = '/';
+ return TRUE;
+} else if (errno != ENOENT) {
+ fprintf (stderr, "Error: stat '%s': %s\n", path, strerror (errno));
+ return FALSE;
}
-return ret;
-}
-
-/* Make the given directory including its parent directories as necessary.
- * Return TRUE on success, FALSE on error. */
-static notmuch_bool_t
-make_directory_and_parents (char *path, int mode)
-{
-struct stat st;
-char *start;
-char *end;
-notmuch_bool_t ret;
-/* First check the common case: directory already exists. */
-if (stat (path, ) == 0)
- return S_ISDIR (st.st_mode) ? TRUE : FALSE;
-
-for (start = path; *start != '\0'; start = end + 1) {
- /* start points to the first unprocessed character.
-* Find the next slash from start onwards. */
- end = strchr (start, '/');
-
- /* If there are no more slashes then all the parent directories
-* have been made. Now attempt to make the whole path. */
- if (end == NULL)
- return make_directory (path, mode);
-
- /* Make the path up to the next slash, unless the current
-* directory component is actually empty. */
- if (end > start) {
- *end = '\0';
- ret = make_directory (path, mode);
- *end = '/';
- if (! ret)
- return FALSE;
+/* mkdir parents, if any */
+slash = strrchr (path, '/');
+if (slash && slash != path) {
+ parent = talloc_strndup (ctx, path, slash - path);
+ if (! parent) {
+ fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
+ return FALSE;
}
+
+ if (! mkdir_recursive (ctx, parent, mode))
+ return FALSE;
}
-return TRUE;
+if (mkdir (path, mode)) {
+ fprintf (stderr, "Error: mkdir '%s': %s\n", path, strerror (errno));
+ return FALSE;
+}
+
+return parent ? sync_dir (parent) : TRUE;
}
-/* Create the given maildir folder, i.e. dir and its subdirectories
- * 'cur', 'new', 'tmp'. */
+/*
+ * Create the given maildir folder, i.e. maildir and its
+ * subdirectories cur/new/tmp. Return TRUE on success, FALSE
+ * otherwise. Partial results are not cleaned up on errors.
+ */
static notmuch_bool_t
-maildir_create_folder (void *ctx, const char *dir)
+maildir_create_folder (const void *ctx, const char *maildir)
{
+const char *subdirs[] = { "cur", "new", "tmp" };
const int mode = 0700;
char *subdir;
-char *tail;
-
-/* Create 'cur' directory, including parent directories. */
-subdir = talloc_asprintf (ctx, "%s/cur", dir);
-if (! subdir) {
- fprintf (stderr, "Out of memory.\n");
- return FALSE;
-}
-if (! make_directory_and_parents (subdir, mode))
- return FALSE;
-
-tail = subdir + strlen (subdir) - 3;
+unsigned int i;
-/* Create 'new' directory. */
-strcpy (tail, "new");
-if (! make_directory (subdir, mode))
- return FALSE;
+for (i = 0; i < ARRAY_SIZE (subdirs); i++) {
+ subdir = talloc_asprintf (ctx, "%s/%s", maildir, subdirs[i]);
+ if (! subdir) {
+ fprintf (stderr, "Error: %s\n", strerror (ENOMEM));
+ return FALSE;
+ }
-/* Create 'tmp' directory. */
-strcpy (tail, "tmp");
-if (! make_directory (subdir, mode))
- return FALSE;
+ if (!
[PATCH 05/11] cli/insert: clean up sync_dir
Clarify the code slightly, improve error messages. Apart from the
error message changes, no functional changes.
---
notmuch-insert.c | 17 +
1 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5d47806..7375c54 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -67,20 +67,21 @@ safe_gethostname (char *hostname, size_t len)
static notmuch_bool_t
sync_dir (const char *dir)
{
-notmuch_bool_t ret;
-int fd;
+int fd, r;
fd = open (dir, O_RDONLY);
if (fd == -1) {
- fprintf (stderr, "Error: open() dir failed: %s\n", strerror (errno));
+ fprintf (stderr, "Error: open %s: %s\n", dir, strerror (errno));
return FALSE;
}
-ret = (fsync (fd) == 0);
-if (! ret) {
- fprintf (stderr, "Error: fsync() dir failed: %s\n", strerror (errno));
-}
+
+r = fsync (fd);
+if (r)
+ fprintf (stderr, "Error: fsync %s: %s\n", dir, strerror (errno));
+
close (fd);
-return ret;
+
+return r == 0;
}
/*
--
1.7.2.5
[PATCH 04/11] cli/insert: rename file copy function
The copying has nothing to do with stdin, so call it copy_fd
instead. While at it, improve documentation and reverse the
parameters, as destination is traditionally the first parameter.
---
notmuch-insert.c | 11 ++-
1 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index ccb091a..5d47806 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -251,11 +251,12 @@ maildir_open_tmp_file (void *ctx, const char *dir,
return fd;
}
-/* Copy the contents of standard input (fdin) into fdout.
- * Returns TRUE if a non-empty file was written successfully.
- * Otherwise, return FALSE. */
+/*
+ * Copy fdin to fdout, return TRUE on success, and FALSE on errors and
+ * empty input.
+ */
static notmuch_bool_t
-copy_stdin (int fdin, int fdout)
+copy_fd (int fdout, int fdin)
{
notmuch_bool_t empty = TRUE;
@@ -308,7 +309,7 @@ write_message (void *ctx, int fdin, const char *dir, char
**newpath)
cleanup_path = tmppath;
-if (! copy_stdin (fdin, fdout))
+if (! copy_fd (fdout, fdin))
goto FAIL;
if (fsync (fdout) != 0) {
--
1.7.2.5
[PATCH 03/11] cli/insert: move add_file_to_database to a better place
Move add_file_to_database around to keep the filesystem related
functions grouped together, improving readability. No functional
changes.
---
notmuch-insert.c | 92 +++---
1 files changed, 46 insertions(+), 46 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 770275b..ccb091a 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -294,52 +294,6 @@ copy_stdin (int fdin, int fdout)
return (!interrupted && !empty);
}
-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
-{
-notmuch_message_t *message;
-notmuch_status_t status;
-
-status = notmuch_database_add_message (notmuch, path, );
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
- fprintf (stderr, "Error: failed to add `%s' to notmuch database: %s\n",
-path, notmuch_status_to_string (status));
- return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
- /* Don't change tags of an existing message. */
- if (synchronize_flags) {
- status = notmuch_message_tags_to_maildir_flags (message);
- if (status != NOTMUCH_STATUS_SUCCESS)
- fprintf (stderr, "Error: failed to sync tags to maildir
flags\n");
- }
-} else {
- tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
-
- tag_op_list_apply (message, tag_ops, flags);
-}
-
-notmuch_message_destroy (message);
-}
-
static notmuch_bool_t
write_message (void *ctx, int fdin, const char *dir, char **newpath)
{
@@ -389,6 +343,52 @@ write_message (void *ctx, int fdin, const char *dir, char
**newpath)
return FALSE;
}
+/* Add the specified message file to the notmuch database, applying tags.
+ * The file is renamed to encode notmuch tags as maildir flags. */
+static void
+add_file_to_database (notmuch_database_t *notmuch, const char *path,
+ tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+{
+notmuch_message_t *message;
+notmuch_status_t status;
+
+status = notmuch_database_add_message (notmuch, path, );
+switch (status) {
+case NOTMUCH_STATUS_SUCCESS:
+case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ break;
+default:
+case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+case NOTMUCH_STATUS_OUT_OF_MEMORY:
+case NOTMUCH_STATUS_FILE_ERROR:
+case NOTMUCH_STATUS_NULL_POINTER:
+case NOTMUCH_STATUS_TAG_TOO_LONG:
+case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
+case NOTMUCH_STATUS_LAST_STATUS:
+ fprintf (stderr, "Error: failed to add `%s' to notmuch database: %s\n",
+path, notmuch_status_to_string (status));
+ return;
+}
+
+if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+ /* Don't change tags of an existing message. */
+ if (synchronize_flags) {
+ status = notmuch_message_tags_to_maildir_flags (message);
+ if (status != NOTMUCH_STATUS_SUCCESS)
+ fprintf (stderr, "Error: failed to sync tags to maildir
flags\n");
+ }
+} else {
+ tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+
+ tag_op_list_apply (message, tag_ops, flags);
+}
+
+notmuch_message_destroy (message);
+}
+
int
notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
{
--
1.7.2.5
[PATCH 02/11] cli/insert: rename check_folder_name to is_valid_folder_name
An "is something" predicate conveys the meaning better. While at it,
improve the function documentation and error message. Besides the
error message change, no functional changes.
---
notmuch-insert.c | 13 -
1 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 8dfc8bb..770275b 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -83,10 +83,13 @@ sync_dir (const char *dir)
return ret;
}
-/* Check the specified folder name does not contain a directory
- * component ".." to prevent writes outside of the Maildir hierarchy. */
+/*
+ * Check the specified folder name does not contain a directory
+ * component ".." to prevent writes outside of the Maildir
+ * hierarchy. Return TRUE on valid folder name, FALSE otherwise.
+ */
static notmuch_bool_t
-check_folder_name (const char *folder)
+is_valid_folder_name (const char *folder)
{
const char *p = folder;
@@ -449,8 +452,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
if (folder == NULL) {
maildir = db_path;
} else {
- if (! check_folder_name (folder)) {
- fprintf (stderr, "Error: bad folder name: %s\n", folder);
+ if (! is_valid_folder_name (folder)) {
+ fprintf (stderr, "Error: invalid folder name: '%s'\n", folder);
return EXIT_FAILURE;
}
maildir = talloc_asprintf (config, "%s/%s", db_path, folder);
--
1.7.2.5
[PATCH 01/11] lib: actually return failures from notmuch_message_tags_to_maildir_flags
The function takes great care to preserve the first error status it encounters, yet fails to return that status to the caller. Fix it. --- lib/message.cc |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/lib/message.cc b/lib/message.cc index 68f7e68..7e82548 100644 --- a/lib/message.cc +++ b/lib/message.cc @@ -1497,7 +1497,7 @@ notmuch_message_tags_to_maildir_flags (notmuch_message_t *message) talloc_free (to_set); talloc_free (to_clear); -return NOTMUCH_STATUS_SUCCESS; +return status; } notmuch_status_t -- 1.7.2.5
[PATCH 00/11] notmuch insert updates
This series refactors and cleans up insert, improves error handling and reporting, and adds post-insert hook. I intend to add documentation and more tests, but the code is ready for review. Also, at least some of the cleanups and fixes in the beginning of the series could go in without additional tests or documentation. BR, Jani. Jani Nikula (11): lib: actually return failures from notmuch_message_tags_to_maildir_flags cli/insert: rename check_folder_name to is_valid_folder_name cli/insert: move add_file_to_database to a better place cli/insert: rename file copy function cli/insert: clean up sync_dir cli/insert: use a single recursive mkdir function cli/insert: abstract temporary filename generation cli/insert: rehash file writing functions cli/insert: add fail path to add_file_to_database cli/insert: require succesful message indexing for success status cli/insert: add post-insert hook lib/message.cc |2 +- notmuch-insert.c| 462 +-- test/T070-insert.sh |2 +- 3 files changed, 262 insertions(+), 204 deletions(-) -- 1.7.2.5
[PATCH] lib: Simplify close and codify aborting atomic section
In Xapian, closing a database implicitly aborts any outstanding
transaction and commits changes. For historical reasons,
notmuch_database_close had grown to almost, but not quite duplicate
this behavior. Before closing the database, it would explicitly (and
unnecessarily) commit it. However, if there was an outstanding
transaction (ie atomic section), commit would throw a Xapian
exception, which notmuch_database_close would unnecessarily print to
stderr, even though notmuch_database_close would ultimately abort the
transaction anyway when it called close.
This patch simplifies notmuch_database_close to just call
Database::close. This works for both read-only and read/write
databases, takes care of committing changes, unifies the exception
handling path, and codifies aborting outstanding transactions. This
is currently the only way to abort an atomic section (and may remain
so, since it would be difficult to roll back things we may have cached
from rolled-back modifications).
---
lib/database.cc | 23 +++
lib/notmuch.h | 5 +
2 files changed, 12 insertions(+), 16 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index a3a7cd3..1f7ff2a 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -903,28 +903,19 @@ notmuch_database_close (notmuch_database_t *notmuch)
{
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
-try {
- if (notmuch->xapian_db != NULL &&
- notmuch->mode == NOTMUCH_DATABASE_MODE_READ_WRITE)
- (static_cast
(notmuch->xapian_db))->flush ();
-} catch (const Xapian::Error ) {
- status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
- if (! notmuch->exception_reported) {
- fprintf (stderr, "Error: A Xapian exception occurred flushing
database: %s\n",
-error.get_msg().c_str());
- }
-}
-
/* Many Xapian objects (and thus notmuch objects) hold references to
* the database, so merely deleting the database may not suffice to
- * close it. Thus, we explicitly close it here. */
+ * close it. Thus, we explicitly close it here. This will
+ * implicitly abort any outstanding transaction and commit changes. */
if (notmuch->xapian_db != NULL) {
try {
notmuch->xapian_db->close();
} catch (const Xapian::Error ) {
- /* don't clobber previous error status */
- if (status == NOTMUCH_STATUS_SUCCESS)
- status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ if (! notmuch->exception_reported) {
+ fprintf (stderr, "Error: A Xapian exception occurred closing
database: %s\n",
+error.get_msg().c_str());
+ }
}
}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index fe2340b..5c40c67 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -292,6 +292,11 @@ notmuch_database_open (const char *path,
* notmuch_database_close can be called multiple times. Later calls
* have no effect.
*
+ * If the caller is currently in an atomic section (there was a
+ * notmuch_database_begin_atomic without a matching
+ * notmuch_database_end_atomic), this will abort the atomic section,
+ * discarding any modifications made in the atomic section.
+ *
* Return value:
*
* NOTMUCH_STATUS_SUCCESS: Successfully closed the database.
--
2.1.0
[PATCH] Add configurable changed tag to messages that have been changed on disk
On Mon, 22 Sep 2014, Gaute Hope wrote: > Excerpts from Gaute Hope's message of August 6, 2014 10:29: >> Austin Clements wrote on Fri, 01 Aug 2014 14:55:05 >> -0400: >>> I have a prototype implementation of message modification times on my >>> lastmod-v1 branch at >>> >>> https://github.com/aclements/notmuch/tree/lastmod-v1 >>> >>> It builds on my database features series that's currently awaiting >>> review [1]. >>> >>> The series uses a monotonic revision number, rather than wall-clock >>> time, for reasons related to Xapian's concurrent control and detailed >>> in the main commit's commit message. The implementation isn't quite >>> useful from the CLI yet because I haven't added any way to query the >>> database's current revision number. (I'm still thinking about how I >>> want to do this, since search/show don't have a good way to deliver >>> "additional" information right now. I might just add the last >>> modification for each individual message/max of all messages in a >>> thread, similar to what Thomas Jost's patch did long ago.) >>> >>> [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu > >> this should allow me to do what I wish to accomplish. The message >> deletion is still a problem though, I can see two options at the moment: > > Hi list, > > While exploring the possibility of syncing maildir/X-keywords with tags > I had some thoughts about lastmod and message modification: > > As briefly discussed on #notmuch, I noticed that it seems that 'notmuch > new' does not detect that a message source has been changed, unless the > file is also re-named. > > This means that for instance if the X-Keywords fields have been updated > in a message (from GMail with offlineimap, synclabels = yes) the lastmod > field will remain unchanged, and a source modification will be > undetectable to a client program using this value. > > Would it not make sense that if a message has a more recent mtime than > at index time it is re-indexed? This has the potential to make notmuch new substantially more expensive. Currently, if there are no changes, it only has to stat each directory in your maildir (in fact, some restructuring of new would let us eliminate almost all database access during a no-op notmuch new as well). Checking for changes to individual messages would require stat'ing every single message file as well as accessing the database to check the paths and mtimes of every message, increasing the number of stat calls and disk accesses by several orders of magnitude. It may be that this is fast enough that it's okay, but it would be good to gather some evidence first. That includes hot and cold caches, and maildir over NFS. With respect to X-Keywords specifically, note that it's a fairly basic design decision that notmuch never modifies message files. This gives us strong robustness guarantees we would be loathe to part with. It has puzzled me ever since offlineimap added X-Keywords why they didn't just translate these keywords into folders and create hard links of message files. Anything could interact smoothly with that. > Also, for the lastmod branch I would wish for a notmuch_message_touch() > method where the lastmod time is updated to the last. As well as a > notmuch_database_reindex_message () - possibly defined/documented > behaviour for notmuch_database_add_message () when the filename is > already added (in which case I would expect notmuch to re-index the > message). What's the use case for these? > Doing notmuch_database_remove_message followed by _add_message could > risk deleting the entry if this file is the only on-disk-representation. > > Cheers, Gaute
[PATCH 5/5] cli: Add tests for 'search --output=addresses' and similar
--- test/T090-search-output.sh | 59 +++ test/T095-search-unique.sh | 63 ++ 2 files changed, 122 insertions(+) create mode 100755 test/T095-search-unique.sh diff --git a/test/T090-search-output.sh b/test/T090-search-output.sh index 947d572..ebc8c37 100755 --- a/test/T090-search-output.sh +++ b/test/T090-search-output.sh @@ -387,6 +387,65 @@ catOUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat
[PATCH 4/5] cli: Add configurable address deduplication for --output=addresses
The code here is an extended version of a path from Jani Nikula.
---
completion/notmuch-completion.bash | 6 ++-
completion/notmuch-completion.zsh | 3 +-
doc/man1/notmuch-search.rst| 33
notmuch-search.c | 101 ++---
4 files changed, 135 insertions(+), 8 deletions(-)
diff --git a/completion/notmuch-completion.bash
b/completion/notmuch-completion.bash
index c37ddf5..8bc7874 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -305,12 +305,16 @@ _notmuch_search()
COMPREPLY=( $( compgen -W "true false flag all" -- "${cur}" ) )
return
;;
+ --unique)
+ COMPREPLY=( $( compgen -W "none addr addrfold name" -- "${cur}" ) )
+ return
+ ;;
esac
! $split &&
case "${cur}" in
-*)
- local options="--format= --output= --sort= --offset= --limit=
--exclude= --duplicate="
+ local options="--format= --output= --sort= --offset= --limit=
--exclude= --duplicate= --unique="
compopt -o nospace
COMPREPLY=( $(compgen -W "$options" -- ${cur}) )
;;
diff --git a/completion/notmuch-completion.zsh
b/completion/notmuch-completion.zsh
index bff8fd5..cf4968c 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -53,7 +53,8 @@ _notmuch_search()
'--max-threads=[display only the first x threads from the search
results]:number of threads to show: ' \
'--first=[omit the first x threads from the search results]:number of
threads to omit: ' \
'--sort=[sort results]:sorting:((newest-first\:"reverse chronological
order" oldest-first\:"chronological order"))' \
-'--output=[select what to output]:output:((summary threads messages files
tags sender recipients addresses))'
+'--output=[select what to output]:output:((summary threads messages files
tags sender recipients addresses))' \
+'--unique=[address deduplication]:unique:((none\:"no deduplication"
addr\:"deduplicate by address" addrfold\:"deduplicate by case-insensitive
address" name\:"deduplicate by name"))'
}
_notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 6094906..a92779a 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -85,6 +85,9 @@ Supported options for **search** include
(--format=text0), as a JSON array (--format=json), or as
an S-Expression list (--format=sexp).
+Handling of duplicate addresses and/or names can be
+controlled with the --unique option.
+
Note: Searching for **sender** should much be faster than
searching for **recipients** or **addresses**, because
sender addresses are cached directly in the database
@@ -151,6 +154,36 @@ Supported options for **search** include
prefix. The prefix matches messages based on filenames. This
option filters filenames of the matching messages.
+``--unique=``\ (**none**\ \|\ **addr**\ \|\ **addrfold**\ \|\ **name**)[,\
...]
+
+Can be used with ``--output=addresses``, ``--output=sender``
+or ``--output=recipients`` to control the address
+deduplication algorithm.
+
+ **none** means that no deduplication is performed. The same
+ address can appear multiple times in the output.
+
+ **addr** means that case-sensitive deduplication is performed
+ on the address part. For example, given the addresses "John
+ Doe " and "Dr. John Doe ",
+ only one will be printed.
+
+ **addrfold** means that case-insensitive deduplication is
+ performed on the address part. For example, given the
+ addresses "John Doe " and "John Doe
+ ", only one will be printed. This is the
+ default.
+
+ **name** means that case-sensitive deduplication is performed
+ on the name part. For example, given the addresses "John Doe
+ " and "John Doe ", only one
+ will be printed.
+
+ It is possible to combine the above flags (except **none**) by
+ separating them with comma. For example,
+ ``--unique=name,addr`` will print unique case-sensitive
+ combinations of name and address.
+
EXIT STATUS
===
diff --git a/notmuch-search.c b/notmuch-search.c
index 0614f10..00d6771 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -33,6 +33,15 @@ typedef enum {
OUTPUT_ADDRESSES = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
} output_t;
+typedef enum {
+UNIQUE_NONE = 1 << 0,
+UNIQUE_ADDR = 1 << 1,
+UNIQUE_NAME = 1 << 2,
+UNIQUE_ADDR_CASEFOLD = 1 << 3,
+
+UNIQUE_BOTH = UNIQUE_NAME | UNIQUE_ADDR,
+} unique_t;
+
typedef struct {
sprinter_t *format;
notmuch_query_t *query;
@@ -41,6 +50,7 @@ typedef struct {
int offset;
int limit;
int dupe;
+unique_t unique;
}
[PATCH 3/5] cli: Add support for parsing command line "flag" options
This allows having multiple flags as a value of a command line
option (e.g. --foo=bar,baz). The values of the given flags are OR'd
together.
This was inspired by a similar patch from Jani Nikula.
---
command-line-arguments.c | 40
command-line-arguments.h | 1 +
test/Makefile.local | 2 +-
test/T410-argument-parsing.sh | 3 ++-
test/arg-test.c | 8
5 files changed, 52 insertions(+), 2 deletions(-)
diff --git a/command-line-arguments.c b/command-line-arguments.c
index 844d6c3..50723e4 100644
--- a/command-line-arguments.c
+++ b/command-line-arguments.c
@@ -3,6 +3,7 @@
#include
#include "error_util.h"
#include "command-line-arguments.h"
+#include "string-util.h"
/*
Search the array of keywords for a given argument, assigning the
@@ -36,6 +37,43 @@ _process_keyword_arg (const notmuch_opt_desc_t *arg_desc,
char next, const char
return FALSE;
}
+/*
+ An arguments composed of comma-separated flags is parsed and the
+ output variable is assigned the ORed flag values. Return FALSE in
+ case of error.
+*/
+static notmuch_bool_t
+_process_flags_arg (const notmuch_opt_desc_t *arg_desc, char next, const char
*arg_str) {
+
+const notmuch_keyword_t *keyword;
+const char *flag = arg_str;
+size_t flen = 0;
+notmuch_bool_t match;
+
+if (next == '\0' || *arg_str == '\0') {
+ /* No flag given */
+ fprintf (stderr, "Option \"%s\" needs an flags argument.\n",
arg_desc->name);
+ return FALSE;
+}
+
+while ((flag = strtok_len_c (flag + flen, ",", ))) {
+ for (keyword = arg_desc->keywords, match = FALSE; keyword->name;
keyword++) {
+ if (strncmp (flag, keyword->name, flen) == 0 &&
+ flen == strlen (keyword->name)) {
+ *((int *)arg_desc->output_var) |= keyword->value;
+ match = TRUE;
+ break;
+ }
+ }
+ if (! match) {
+ fprintf (stderr, "Unknown flag argument \"%.*s\" for option
\"%s\".\n", (int)flen, flag, arg_desc->name);
+ return FALSE;
+ }
+}
+return TRUE;
+}
+
+
static notmuch_bool_t
_process_boolean_arg (const notmuch_opt_desc_t *arg_desc, char next, const
char *arg_str) {
@@ -153,6 +191,8 @@ parse_option (const char *arg,
switch (try->opt_type) {
case NOTMUCH_OPT_KEYWORD:
return _process_keyword_arg (try, next, value);
+ case NOTMUCH_OPT_FLAGS:
+ return _process_flags_arg (try, next, value);
case NOTMUCH_OPT_BOOLEAN:
return _process_boolean_arg (try, next, value);
case NOTMUCH_OPT_INT:
diff --git a/command-line-arguments.h b/command-line-arguments.h
index de1734a..192ce06 100644
--- a/command-line-arguments.h
+++ b/command-line-arguments.h
@@ -8,6 +8,7 @@ enum notmuch_opt_type {
NOTMUCH_OPT_BOOLEAN, /* --verbose */
NOTMUCH_OPT_INT, /* --frob=8 */
NOTMUCH_OPT_KEYWORD, /* --format=raw|json|text */
+NOTMUCH_OPT_FLAGS, /* --flags=name,addr,casefold */
NOTMUCH_OPT_STRING,/* --file=/tmp/gnarf.txt */
NOTMUCH_OPT_POSITION /* notmuch dump pos_arg */
};
diff --git a/test/Makefile.local b/test/Makefile.local
index a2d58fc..efa3410 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -13,7 +13,7 @@ smtp_dummy_srcs = \
smtp_dummy_modules = $(smtp_dummy_srcs:.c=.o)
$(dir)/arg-test: $(dir)/arg-test.o command-line-arguments.o util/libutil.a
- $(call quiet,CC) $^ -o $@
+ $(call quiet,CC) $^ -o $@ -ltalloc
$(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libutil.a
$(call quiet,CC) $^ $(TALLOC_LDFLAGS) -o $@
diff --git a/test/T410-argument-parsing.sh b/test/T410-argument-parsing.sh
index 94e9087..9c4cc84 100755
--- a/test/T410-argument-parsing.sh
+++ b/test/T410-argument-parsing.sh
@@ -3,9 +3,10 @@ test_description="argument parsing"
. ./test-lib.sh
test_begin_subtest "sanity check"
-$TEST_DIRECTORY/arg-test pos1 --keyword=one --string=foo pos2 --int=7 >
OUTPUT
+$TEST_DIRECTORY/arg-test pos1 --keyword=one --flags=one,two --string=foo
pos2 --int=7 > OUTPUT
cat < EXPECTED
keyword 1
+flags 3
int 7
string foo
positional arg 1 pos1
diff --git a/test/arg-test.c b/test/arg-test.c
index 6c49eac..0d75fe7 100644
--- a/test/arg-test.c
+++ b/test/arg-test.c
@@ -7,6 +7,7 @@ int main(int argc, char **argv){
int opt_index=1;
int kw_val=0;
+int fl_val=0;
int int_val=0;
char *pos_arg1=NULL;
char *pos_arg2=NULL;
@@ -17,6 +18,10 @@ int main(int argc, char **argv){
(notmuch_keyword_t []){ { "one", 1 },
{ "two", 2 },
{ 0, 0 } } },
+ { NOTMUCH_OPT_FLAGS, _val, "flags", 'f',
+ (notmuch_keyword_t []){ { "one", 1 << 0},
+ { "two", 1 << 1 },
+
[PATCH 2/5] cli: Extend the search command for --output=addresses and similar
The new outputs allow printing senders, recipients or both of matching
messages.
This code based on a patch from Jani Nikula.
---
completion/notmuch-completion.bash | 2 +-
completion/notmuch-completion.zsh | 3 +-
doc/man1/notmuch-search.rst| 22 +++-
notmuch-search.c | 100 ++---
4 files changed, 118 insertions(+), 9 deletions(-)
diff --git a/completion/notmuch-completion.bash
b/completion/notmuch-completion.bash
index 0571dc9..c37ddf5 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -294,7 +294,7 @@ _notmuch_search()
return
;;
--output)
- COMPREPLY=( $( compgen -W "summary threads messages files tags" --
"${cur}" ) )
+ COMPREPLY=( $( compgen -W "summary threads messages files tags
sender recipients addresses" -- "${cur}" ) )
return
;;
--sort)
diff --git a/completion/notmuch-completion.zsh
b/completion/notmuch-completion.zsh
index 67a9aba..bff8fd5 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -52,7 +52,8 @@ _notmuch_search()
_arguments -s : \
'--max-threads=[display only the first x threads from the search
results]:number of threads to show: ' \
'--first=[omit the first x threads from the search results]:number of
threads to omit: ' \
-'--sort=[sort results]:sorting:((newest-first\:"reverse chronological
order" oldest-first\:"chronological order"))'
+'--sort=[sort results]:sorting:((newest-first\:"reverse chronological
order" oldest-first\:"chronological order"))' \
+'--output=[select what to output]:output:((summary threads messages files
tags sender recipients addresses))'
}
_notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 90160f2..6094906 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -35,7 +35,7 @@ Supported options for **search** include
intended for programs that invoke **notmuch(1)** internally. If
omitted, the latest supported version will be used.
-``--output=(summary|threads|messages|files|tags)``
+
``--output=(summary|threads|messages|files|tags|sender|recipients|addresses)``
**summary**
Output a summary of each thread with any message matching
@@ -78,6 +78,26 @@ Supported options for **search** include
by null characters (--format=text0), as a JSON array
(--format=json), or as an S-Expression list (--format=sexp).
+ **sender**
+Output all addresses from the *From* header that appear on
+any message matching the search terms, either one per line
+(--format=text), separated by null characters
+(--format=text0), as a JSON array (--format=json), or as
+an S-Expression list (--format=sexp).
+
+ Note: Searching for **sender** should much be faster than
+ searching for **recipients** or **addresses**, because
+ sender addresses are cached directly in the database
+ whereas other addresses need to be fetched from the
+ message file by parsing it.
+
+ **recipients**
+Like **sender** but for addresses from *To*, *Cc* and
+ *Bcc* headers.
+
+ **addresses**
+ Like **sender** and **recipients** together.
+
``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
This option can be used to present results in either
chronological order (**oldest-first**) or reverse chronological
diff --git a/notmuch-search.c b/notmuch-search.c
index 5ac2a26..0614f10 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -23,11 +23,14 @@
#include "string-util.h"
typedef enum {
-OUTPUT_SUMMARY,
-OUTPUT_THREADS,
-OUTPUT_MESSAGES,
-OUTPUT_FILES,
-OUTPUT_TAGS
+OUTPUT_SUMMARY = 1 << 0,
+OUTPUT_THREADS = 1 << 1,
+OUTPUT_MESSAGES= 1 << 2,
+OUTPUT_FILES = 1 << 3,
+OUTPUT_TAGS= 1 << 4,
+OUTPUT_SENDER = 1 << 5,
+OUTPUT_RECIPIENTS = 1 << 6,
+OUTPUT_ADDRESSES = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
} output_t;
typedef struct {
@@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
return 0;
}
+static void
+print_address_list (const search_options_t *o, InternetAddressList *list)
+{
+InternetAddress *address;
+int i;
+
+for (i = 0; i < internet_address_list_length (list); i++) {
+ address = internet_address_list_get_address (list, i);
+ if (INTERNET_ADDRESS_IS_GROUP (address)) {
+ InternetAddressGroup *group;
+ InternetAddressList *group_list;
+
+ group = INTERNET_ADDRESS_GROUP (address);
+ group_list = internet_address_group_get_members (group);
+ if (group_list == NULL)
+ continue;
+
+ print_address_list (o, group_list);
+ } else
[PATCH 1/5] cli: Refactor option passing in the search command
Many functions that implement the search command need to access command
line options. Instead of passing each option in a separate variable, put
them in a structure and pass only this structure.
This will become handy in the following patches.
---
notmuch-search.c | 122 ---
1 file changed, 62 insertions(+), 60 deletions(-)
diff --git a/notmuch-search.c b/notmuch-search.c
index bc9be45..5ac2a26 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -30,6 +30,16 @@ typedef enum {
OUTPUT_TAGS
} output_t;
+typedef struct {
+sprinter_t *format;
+notmuch_query_t *query;
+notmuch_sort_t sort;
+output_t output;
+int offset;
+int limit;
+int dupe;
+} search_options_t;
+
/* Return two stable query strings that identify exactly the matched
* and unmatched messages currently in thread. If there are no
* matched or unmatched messages, the returned buffers will be
@@ -70,46 +80,42 @@ get_thread_query (notmuch_thread_t *thread,
}
static int
-do_search_threads (sprinter_t *format,
- notmuch_query_t *query,
- notmuch_sort_t sort,
- output_t output,
- int offset,
- int limit)
+do_search_threads (search_options_t *o)
{
notmuch_thread_t *thread;
notmuch_threads_t *threads;
notmuch_tags_t *tags;
+sprinter_t *format = o->format;
time_t date;
int i;
-if (offset < 0) {
- offset += notmuch_query_count_threads (query);
- if (offset < 0)
- offset = 0;
+if (o->offset < 0) {
+ o->offset += notmuch_query_count_threads (o->query);
+ if (o->offset < 0)
+ o->offset = 0;
}
-threads = notmuch_query_search_threads (query);
+threads = notmuch_query_search_threads (o->query);
if (threads == NULL)
return 1;
format->begin_list (format);
for (i = 0;
-notmuch_threads_valid (threads) && (limit < 0 || i < offset + limit);
+notmuch_threads_valid (threads) && (o->limit < 0 || i < o->offset +
o->limit);
notmuch_threads_move_to_next (threads), i++)
{
thread = notmuch_threads_get (threads);
- if (i < offset) {
+ if (i < o->offset) {
notmuch_thread_destroy (thread);
continue;
}
- if (output == OUTPUT_THREADS) {
+ if (o->output == OUTPUT_THREADS) {
format->set_prefix (format, "thread");
format->string (format,
- notmuch_thread_get_thread_id (thread));
+ notmuch_thread_get_thread_id (thread));
format->separator (format);
} else { /* output == OUTPUT_SUMMARY */
void *ctx_quote = talloc_new (thread);
@@ -123,7 +129,7 @@ do_search_threads (sprinter_t *format,
format->begin_map (format);
- if (sort == NOTMUCH_SORT_OLDEST_FIRST)
+ if (o->sort == NOTMUCH_SORT_OLDEST_FIRST)
date = notmuch_thread_get_oldest_date (thread);
else
date = notmuch_thread_get_newest_date (thread);
@@ -215,40 +221,36 @@ do_search_threads (sprinter_t *format,
}
static int
-do_search_messages (sprinter_t *format,
- notmuch_query_t *query,
- output_t output,
- int offset,
- int limit,
- int dupe)
+do_search_messages (search_options_t *o)
{
notmuch_message_t *message;
notmuch_messages_t *messages;
notmuch_filenames_t *filenames;
+sprinter_t *format = o->format;
int i;
-if (offset < 0) {
- offset += notmuch_query_count_messages (query);
- if (offset < 0)
- offset = 0;
+if (o->offset < 0) {
+ o->offset += notmuch_query_count_messages (o->query);
+ if (o->offset < 0)
+ o->offset = 0;
}
-messages = notmuch_query_search_messages (query);
+messages = notmuch_query_search_messages (o->query);
if (messages == NULL)
return 1;
format->begin_list (format);
for (i = 0;
-notmuch_messages_valid (messages) && (limit < 0 || i < offset + limit);
+notmuch_messages_valid (messages) && (o->limit < 0 || i < o->offset +
o->limit);
notmuch_messages_move_to_next (messages), i++)
{
- if (i < offset)
+ if (i < o->offset)
continue;
message = notmuch_messages_get (messages);
- if (output == OUTPUT_FILES) {
+ if (o->output == OUTPUT_FILES) {
int j;
filenames = notmuch_message_get_filenames (message);
@@ -256,7 +258,7 @@ do_search_messages (sprinter_t *format,
notmuch_filenames_valid (filenames);
notmuch_filenames_move_to_next (filenames), j++)
{
- if (dupe < 0 || dupe == j) {
+ if (o->dupe < 0 || o->dupe == j) {
format->string (format,
[PATCH 0/5] notmuch search --output=addresses
Hello all, this patch series adds support for --output=addresses and similar arguments in notmuch search. It is a reworked and extended version of id:1410021689-15901-1-git-send-email-jani at nikula.org. Compared to Jani's patches, I do not use keyword-flag command line parser for the --output option, because this would not be useful for most but one combination of keywords. Instead, this one combination is explicitely mentioned as another keyword. The flag command-line parser is introduced later, with (IMHO) better syntax: --flags=one,two,... This feature is used to control address deduplication via the --unique option. Other extensions are added documentation, tests and shell completions. The whole test suite passes with these patches. The functionality presented here will be useful for improving Emacs address completion (id:1411150602-21892-1-git-send-email-sojkam1 at fel.cvut.cz). Another nice use case is sending patched with git: git send-email --cc-cmd='nmsenders id:' ... where nmsenders script contains: notmuch search --output=sender $(notmuch search --output=threads $1) Michal Sojka (5): cli: Refactor option passing in the search command cli: Extend the search command for --output=addresses and similar cli: Add support for parsing command line "flag" options cli: Add configurable address deduplication for --output=addresses cli: Add tests for 'search --output=addresses' and similar command-line-arguments.c | 40 + command-line-arguments.h | 1 + completion/notmuch-completion.bash | 8 +- completion/notmuch-completion.zsh | 4 +- doc/man1/notmuch-search.rst| 55 ++- notmuch-search.c | 311 + test/Makefile.local| 2 +- test/T090-search-output.sh | 59 +++ test/T095-search-unique.sh | 63 test/T410-argument-parsing.sh | 3 +- test/arg-test.c| 8 + 11 files changed, 482 insertions(+), 72 deletions(-) create mode 100755 test/T095-search-unique.sh -- 2.1.0
Do path: searches handle spaces correctly?
Keith Amidon writes: > > notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items' > > I don't get any results, but it seems like the two results above should > be shown, right? Am I doing something wrong here? If it looks like I'm > doing it correctly, what can I do to help troubleshoot the issue? Note that path:, unlike folder:, does not add the maildir subdirs. Dunno if that's the only issue you are having, but I guess it's one. d
[announce] nmhive v0.1.0, a bookmarklet/server for nmbug tags
On Mon, Sep 22, 2014 at 10:19:35AM -0700, W. Trevor King wrote: > I like nmbug's distributed tag maintenance, but not everyone has > notmuch/nmbug installed locally (yet ;). However, everyone that I > know does have a browser and a mail client. They can submit > messages with their mail client already, but we've been missing a > way for them to help tag messages. Ah, and the other piece to this workflow is the existing nmbug-status, which collects the results of canned searches so folks without a local notmuch can use the tags [1]. Folks using a local nmhive will probably want to run their own status-genertion via a post-commit hook in their nmhive repository. Then their users will have their search-results updated after each web-initiated change. If you also wanted them to see updates from changes to tethera's nmbug repository, you'd probably also want a cron job that tried to fetch and merge tethera's changes with the nmhive changes: -o---o---oo tethera/master \\ \o nmhive/status (auto-generated merge for nmbug-status) \ / o---oo nmhive/master (with web-initiated changes) You'd want to resolve conflicts somehow, but any resolution strategy is probably fine, since it's unlikely that we get conflicts very often. Cheers, Trevor [1]: http://nmbug.tethera.net/status/ -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140922/de3e518b/attachment.pgp>
[announce] nmhive v0.1.0
I like nmbug's distributed tag maintenance, but not everyone has notmuch/nmbug installed locally (yet ;). However, everyone that I know does have a browser and a mail client. They can submit messages with their mail client already, but we've been missing a way for them to help tag messages. Nmhive is a lightweight server that allows clients to read and write notmuch tags using a JSON API [1]. It uses nmbug locally to commit after each write, and the admin can then pull tag changes made by nmhive and push them into the native nmbug ecosystem: web client ? nmhive ? nmhive's nmbug ? admin's nmbug ? tethera's nmbug To make it easy for folks to drive nmhive, the repository also contains a bookmarklet [2] that you can use to interactively manage tags from a message's Gmane page (e.g., [3,4]). The setup currently uses my ported-to-Python nmbug [5] if you want to play with it locally. I haven't added user authentication yet, so it's probably best to just run your own nmhive for now. If anyone has preferences for authentication, send a patch :). Or at least let me know, and I'll see what I can do ;). I'd appreciate any other feedback folks have as well. Cheers, Trevor [1]: https://github.com/wking/nmhive [2]: https://github.com/wking/nmhive/blob/v0.1.0/nmbug.js [3]: http://thread.gmane.org/gmane.mail.notmuch.general/19091/focus=19092 [4]: http://article.gmane.org/gmane.mail.notmuch.general/19092 [5]: http://thread.gmane.org/gmane.mail.notmuch.general/19007 id:e630b6763e9d0771718afee41ea15b29bb4a1de8.1409935538.git.wking at tremily.us -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140922/b07e5322/attachment.pgp>
[PATCH] lib: Simplify close and codify aborting atomic section
On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote: > This patch simplifies notmuch_database_close to just call > Database::close. This works for both read-only and read/write > databases, takes care of committing changes, unifies the exception > handling path, and codifies aborting outstanding transactions. If we're dropping the flush call here, where will it be triggered instead? We'll need to flush/commit our changes to the database at some point before closing. Do clients now need an explicit flush/commit command (explicit, client-initiated flushes sound like a good idea to me). Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140922/e44bf0ac/attachment.pgp>
Do path: searches handle spaces correctly?
On Mon, 2014-09-22 at 11:20 +0200, David Bremner wrote: > Keith Amidon writes: > > > > notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items' > > > > I don't get any results, but it seems like the two results above should > > be shown, right? Am I doing something wrong here? If it looks like I'm > > doing it correctly, what can I do to help troubleshoot the issue? > > Note that path:, unlike folder:, does not add the maildir subdirs. Dunno > if that's the only issue you are having, but I guess it's one. Darn it! I made a mistake in my original email. In the test I was doing I actually had: notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items/**' which I believe should have picked up all the subdirectory paths. I just retested to make sure that it still didn't work, and it doesn't. Am I still missing something? Thanks for the help, Keith
Do path: searches handle spaces correctly?
I'm trying to update an archiving tool that used the old folder: search terms to use the newer folder: or path: terms. From playing around with it, I think that the path: term is what I want to use. However, I am getting some behavior I don't understand unless path: searches don't properly handle spaces or I don't know how to quote them properly. If I do: notmuch search --output=files path:'dir/INBOX/**' I get: base/dir/INBOX/cur/1411244319_3.18990.kea-tablet,U=24,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S base/dir/INBOX/cur/1411244319_2.18990.kea-tablet,U=23,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S base/dir/INBOX/INBOX/Sent Items/cur/1411244324_2.18990.kea-tablet,U=3,FMD5=e99e6ef6cc1489bb6d8b47a4c49bb989:2,S base/dir/INBOX/cur/1411244319_1.18990.kea-tablet,U=22,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,RS base/dir/INBOX/cur/1411244319_0.18990.kea-tablet,U=21,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S base/dir/INBOX/cur/1411244318_3.18990.kea-tablet,U=20,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S base/dir/INBOX/cur/1411244318_2.18990.kea-tablet,U=19,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S base/dir/INBOX/cur/1411244318_1.18990.kea-tablet,U=18,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S base/dir/INBOX/INBOX/Sent Items/cur/1411244324_1.18990.kea-tablet,U=2,FMD5=e99e6ef6cc1489bb6d8b47a4c49bb989:2,S base/dir/INBOX/cur/1411244318_0.18990.kea-tablet,U=17,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S Notice that this contains results for dir/INBOX/INBOX/Sent Items, which has a space in it. If I do: notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items' I don't get any results, but it seems like the two results above should be shown, right? Am I doing something wrong here? If it looks like I'm doing it correctly, what can I do to help troubleshoot the issue? Thanks, Keith ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/5] cli: Refactor option passing in the search command
Many functions that implement the search command need to access command
line options. Instead of passing each option in a separate variable, put
them in a structure and pass only this structure.
This will become handy in the following patches.
---
notmuch-search.c | 122 ---
1 file changed, 62 insertions(+), 60 deletions(-)
diff --git a/notmuch-search.c b/notmuch-search.c
index bc9be45..5ac2a26 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -30,6 +30,16 @@ typedef enum {
OUTPUT_TAGS
} output_t;
+typedef struct {
+sprinter_t *format;
+notmuch_query_t *query;
+notmuch_sort_t sort;
+output_t output;
+int offset;
+int limit;
+int dupe;
+} search_options_t;
+
/* Return two stable query strings that identify exactly the matched
* and unmatched messages currently in thread. If there are no
* matched or unmatched messages, the returned buffers will be
@@ -70,46 +80,42 @@ get_thread_query (notmuch_thread_t *thread,
}
static int
-do_search_threads (sprinter_t *format,
- notmuch_query_t *query,
- notmuch_sort_t sort,
- output_t output,
- int offset,
- int limit)
+do_search_threads (search_options_t *o)
{
notmuch_thread_t *thread;
notmuch_threads_t *threads;
notmuch_tags_t *tags;
+sprinter_t *format = o-format;
time_t date;
int i;
-if (offset 0) {
- offset += notmuch_query_count_threads (query);
- if (offset 0)
- offset = 0;
+if (o-offset 0) {
+ o-offset += notmuch_query_count_threads (o-query);
+ if (o-offset 0)
+ o-offset = 0;
}
-threads = notmuch_query_search_threads (query);
+threads = notmuch_query_search_threads (o-query);
if (threads == NULL)
return 1;
format-begin_list (format);
for (i = 0;
-notmuch_threads_valid (threads) (limit 0 || i offset + limit);
+notmuch_threads_valid (threads) (o-limit 0 || i o-offset +
o-limit);
notmuch_threads_move_to_next (threads), i++)
{
thread = notmuch_threads_get (threads);
- if (i offset) {
+ if (i o-offset) {
notmuch_thread_destroy (thread);
continue;
}
- if (output == OUTPUT_THREADS) {
+ if (o-output == OUTPUT_THREADS) {
format-set_prefix (format, thread);
format-string (format,
- notmuch_thread_get_thread_id (thread));
+ notmuch_thread_get_thread_id (thread));
format-separator (format);
} else { /* output == OUTPUT_SUMMARY */
void *ctx_quote = talloc_new (thread);
@@ -123,7 +129,7 @@ do_search_threads (sprinter_t *format,
format-begin_map (format);
- if (sort == NOTMUCH_SORT_OLDEST_FIRST)
+ if (o-sort == NOTMUCH_SORT_OLDEST_FIRST)
date = notmuch_thread_get_oldest_date (thread);
else
date = notmuch_thread_get_newest_date (thread);
@@ -215,40 +221,36 @@ do_search_threads (sprinter_t *format,
}
static int
-do_search_messages (sprinter_t *format,
- notmuch_query_t *query,
- output_t output,
- int offset,
- int limit,
- int dupe)
+do_search_messages (search_options_t *o)
{
notmuch_message_t *message;
notmuch_messages_t *messages;
notmuch_filenames_t *filenames;
+sprinter_t *format = o-format;
int i;
-if (offset 0) {
- offset += notmuch_query_count_messages (query);
- if (offset 0)
- offset = 0;
+if (o-offset 0) {
+ o-offset += notmuch_query_count_messages (o-query);
+ if (o-offset 0)
+ o-offset = 0;
}
-messages = notmuch_query_search_messages (query);
+messages = notmuch_query_search_messages (o-query);
if (messages == NULL)
return 1;
format-begin_list (format);
for (i = 0;
-notmuch_messages_valid (messages) (limit 0 || i offset + limit);
+notmuch_messages_valid (messages) (o-limit 0 || i o-offset +
o-limit);
notmuch_messages_move_to_next (messages), i++)
{
- if (i offset)
+ if (i o-offset)
continue;
message = notmuch_messages_get (messages);
- if (output == OUTPUT_FILES) {
+ if (o-output == OUTPUT_FILES) {
int j;
filenames = notmuch_message_get_filenames (message);
@@ -256,7 +258,7 @@ do_search_messages (sprinter_t *format,
notmuch_filenames_valid (filenames);
notmuch_filenames_move_to_next (filenames), j++)
{
- if (dupe 0 || dupe == j) {
+ if (o-dupe 0 || o-dupe == j) {
format-string (format, notmuch_filenames_get (filenames));
[PATCH 5/5] cli: Add tests for 'search --output=addresses' and similar
--- test/T090-search-output.sh | 59 +++ test/T095-search-unique.sh | 63 ++ 2 files changed, 122 insertions(+) create mode 100755 test/T095-search-unique.sh diff --git a/test/T090-search-output.sh b/test/T090-search-output.sh index 947d572..ebc8c37 100755 --- a/test/T090-search-output.sh +++ b/test/T090-search-output.sh @@ -387,6 +387,65 @@ cat EOF EXPECTED EOF test_expect_equal_file OUTPUT EXPECTED +test_begin_subtest --output=sender +notmuch search --output=sender '*' | sort OUTPUT +cat EOF EXPECTED +Adrian Perez de Castro ape...@igalia.com +Alexander Botero-Lowry alex.boterolo...@gmail.com +Aron Griffis agrif...@n01se.net +Carl Worth cwo...@cworth.org +Chris Wilson ch...@chris-wilson.co.uk +François Boulogne boulogn...@gmail.com +Ingmar Vanhassel ing...@exherbo.org +Israel Herraiz i...@herraiz.org +Jan Janak j...@ryngle.com +Jjgod Jiang gzjj...@gmail.com +Keith Packard kei...@keithp.com +Lars Kellogg-Stedman l...@seas.harvard.edu +Mikhail Gusarov dotted...@dottedmag.net +Olivier Berger olivier.ber...@it-sudparis.eu +Rolland Santimano rollandsantim...@yahoo.com +Stewart Smith stew...@flamingspork.com +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_begin_subtest --output=recipients +notmuch search --output=recipients '*' | sort OUTPUT +cat EOF EXPECTED +Allan McRae al...@archlinux.org +Discussion about the Arch User Repository (AUR) aur-gene...@archlinux.org +Keith Packard kei...@keithp.com +Mikhail Gusarov dotted...@dottedmag.net +notmuch@notmuchmail.org +olivier.ber...@it-sudparis.eu +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_begin_subtest --output=addresses +notmuch search --output=addresses '*' | sort OUTPUT +cat EOF EXPECTED +Adrian Perez de Castro ape...@igalia.com +Alexander Botero-Lowry alex.boterolo...@gmail.com +Allan McRae al...@archlinux.org +Aron Griffis agrif...@n01se.net +Carl Worth cwo...@cworth.org +Chris Wilson ch...@chris-wilson.co.uk +Discussion about the Arch User Repository (AUR) aur-gene...@archlinux.org +François Boulogne boulogn...@gmail.com +Ingmar Vanhassel ing...@exherbo.org +Israel Herraiz i...@herraiz.org +Jan Janak j...@ryngle.com +Jjgod Jiang gzjj...@gmail.com +Keith Packard kei...@keithp.com +Lars Kellogg-Stedman l...@seas.harvard.edu +Mikhail Gusarov dotted...@dottedmag.net +Olivier Berger olivier.ber...@it-sudparis.eu +Rolland Santimano rollandsantim...@yahoo.com +Stewart Smith stew...@flamingspork.com +notmuch@notmuchmail.org +EOF +test_expect_equal_file OUTPUT EXPECTED + test_begin_subtest sanitize output for quoted-printable line-breaks in author and subject add_message [subject]='two =?ISO-8859-1?Q?line=0A_subject?= headers' diff --git a/test/T095-search-unique.sh b/test/T095-search-unique.sh new file mode 100755 index 000..8fd8fc0 --- /dev/null +++ b/test/T095-search-unique.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +test_description='address deduplication in notmuch search --output=addresses' +. ./test-lib.sh + +add_message '[to]=Real Name f...@example.com, Real Name b...@example.com' +add_message '[to]=Nickname f...@example.com' '[cc]=Real Name b...@example.com' +add_message '[to]=Nickname f...@example.com' '[bcc]=Real Name b...@example.com' + +test_begin_subtest --output=recipients +notmuch search --output=recipients * OUTPUT +cat EOF EXPECTED +Real Name f...@example.com +Real Name b...@example.com +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_begin_subtest --output=recipients --unique=none +notmuch search --output=recipients --unique=none * OUTPUT +cat EOF EXPECTED +Real Name f...@example.com +Real Name b...@example.com +Nickname f...@example.com +Real Name b...@example.com +Nickname f...@example.com +Real Name b...@example.com +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_begin_subtest --output=recipients --unique=addr +notmuch search --output=recipients --unique=addr * OUTPUT +cat EOF EXPECTED +Real Name f...@example.com +Real Name b...@example.com +Real Name b...@example.com +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_begin_subtest --output=recipients --unique=addrfold +notmuch search --output=recipients --unique=addrfold * OUTPUT +cat EOF EXPECTED +Real Name f...@example.com +Real Name b...@example.com +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_begin_subtest --output=recipients --unique=name +notmuch search --output=recipients --unique=name * OUTPUT +cat EOF EXPECTED +Real Name f...@example.com +Nickname f...@example.com +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_begin_subtest --output=recipients --unique=name,addrfold +notmuch search --output=recipients --unique=name,addrfold * OUTPUT +cat EOF EXPECTED +Real Name f...@example.com +Real Name b...@example.com +Nickname f...@example.com +EOF +test_expect_equal_file OUTPUT EXPECTED + +test_done -- 2.1.0 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/5] cli: Extend the search command for --output=addresses and similar
The new outputs allow printing senders, recipients or both of matching
messages.
This code based on a patch from Jani Nikula.
---
completion/notmuch-completion.bash | 2 +-
completion/notmuch-completion.zsh | 3 +-
doc/man1/notmuch-search.rst| 22 +++-
notmuch-search.c | 100 ++---
4 files changed, 118 insertions(+), 9 deletions(-)
diff --git a/completion/notmuch-completion.bash
b/completion/notmuch-completion.bash
index 0571dc9..c37ddf5 100644
--- a/completion/notmuch-completion.bash
+++ b/completion/notmuch-completion.bash
@@ -294,7 +294,7 @@ _notmuch_search()
return
;;
--output)
- COMPREPLY=( $( compgen -W summary threads messages files tags --
${cur} ) )
+ COMPREPLY=( $( compgen -W summary threads messages files tags
sender recipients addresses -- ${cur} ) )
return
;;
--sort)
diff --git a/completion/notmuch-completion.zsh
b/completion/notmuch-completion.zsh
index 67a9aba..bff8fd5 100644
--- a/completion/notmuch-completion.zsh
+++ b/completion/notmuch-completion.zsh
@@ -52,7 +52,8 @@ _notmuch_search()
_arguments -s : \
'--max-threads=[display only the first x threads from the search
results]:number of threads to show: ' \
'--first=[omit the first x threads from the search results]:number of
threads to omit: ' \
-'--sort=[sort results]:sorting:((newest-first\:reverse chronological
order oldest-first\:chronological order))'
+'--sort=[sort results]:sorting:((newest-first\:reverse chronological
order oldest-first\:chronological order))' \
+'--output=[select what to output]:output:((summary threads messages files
tags sender recipients addresses))'
}
_notmuch()
diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
index 90160f2..6094906 100644
--- a/doc/man1/notmuch-search.rst
+++ b/doc/man1/notmuch-search.rst
@@ -35,7 +35,7 @@ Supported options for **search** include
intended for programs that invoke **notmuch(1)** internally. If
omitted, the latest supported version will be used.
-``--output=(summary|threads|messages|files|tags)``
+
``--output=(summary|threads|messages|files|tags|sender|recipients|addresses)``
**summary**
Output a summary of each thread with any message matching
@@ -78,6 +78,26 @@ Supported options for **search** include
by null characters (--format=text0), as a JSON array
(--format=json), or as an S-Expression list (--format=sexp).
+ **sender**
+Output all addresses from the *From* header that appear on
+any message matching the search terms, either one per line
+(--format=text), separated by null characters
+(--format=text0), as a JSON array (--format=json), or as
+an S-Expression list (--format=sexp).
+
+ Note: Searching for **sender** should much be faster than
+ searching for **recipients** or **addresses**, because
+ sender addresses are cached directly in the database
+ whereas other addresses need to be fetched from the
+ message file by parsing it.
+
+ **recipients**
+Like **sender** but for addresses from *To*, *Cc* and
+ *Bcc* headers.
+
+ **addresses**
+ Like **sender** and **recipients** together.
+
``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
This option can be used to present results in either
chronological order (**oldest-first**) or reverse chronological
diff --git a/notmuch-search.c b/notmuch-search.c
index 5ac2a26..0614f10 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -23,11 +23,14 @@
#include string-util.h
typedef enum {
-OUTPUT_SUMMARY,
-OUTPUT_THREADS,
-OUTPUT_MESSAGES,
-OUTPUT_FILES,
-OUTPUT_TAGS
+OUTPUT_SUMMARY = 1 0,
+OUTPUT_THREADS = 1 1,
+OUTPUT_MESSAGES= 1 2,
+OUTPUT_FILES = 1 3,
+OUTPUT_TAGS= 1 4,
+OUTPUT_SENDER = 1 5,
+OUTPUT_RECIPIENTS = 1 6,
+OUTPUT_ADDRESSES = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
} output_t;
typedef struct {
@@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
return 0;
}
+static void
+print_address_list (const search_options_t *o, InternetAddressList *list)
+{
+InternetAddress *address;
+int i;
+
+for (i = 0; i internet_address_list_length (list); i++) {
+ address = internet_address_list_get_address (list, i);
+ if (INTERNET_ADDRESS_IS_GROUP (address)) {
+ InternetAddressGroup *group;
+ InternetAddressList *group_list;
+
+ group = INTERNET_ADDRESS_GROUP (address);
+ group_list = internet_address_group_get_members (group);
+ if (group_list == NULL)
+ continue;
+
+ print_address_list (o, group_list);
+ } else {
+
[PATCH 0/5] notmuch search --output=addresses
Hello all, this patch series adds support for --output=addresses and similar arguments in notmuch search. It is a reworked and extended version of id:1410021689-15901-1-git-send-email-j...@nikula.org. Compared to Jani's patches, I do not use keyword-flag command line parser for the --output option, because this would not be useful for most but one combination of keywords. Instead, this one combination is explicitely mentioned as another keyword. The flag command-line parser is introduced later, with (IMHO) better syntax: --flags=one,two,... This feature is used to control address deduplication via the --unique option. Other extensions are added documentation, tests and shell completions. The whole test suite passes with these patches. The functionality presented here will be useful for improving Emacs address completion (id:1411150602-21892-1-git-send-email-sojk...@fel.cvut.cz). Another nice use case is sending patched with git: git send-email --cc-cmd='nmsenders id:' ... where nmsenders script contains: notmuch search --output=sender $(notmuch search --output=threads $1) Michal Sojka (5): cli: Refactor option passing in the search command cli: Extend the search command for --output=addresses and similar cli: Add support for parsing command line flag options cli: Add configurable address deduplication for --output=addresses cli: Add tests for 'search --output=addresses' and similar command-line-arguments.c | 40 + command-line-arguments.h | 1 + completion/notmuch-completion.bash | 8 +- completion/notmuch-completion.zsh | 4 +- doc/man1/notmuch-search.rst| 55 ++- notmuch-search.c | 311 + test/Makefile.local| 2 +- test/T090-search-output.sh | 59 +++ test/T095-search-unique.sh | 63 test/T410-argument-parsing.sh | 3 +- test/arg-test.c| 8 + 11 files changed, 482 insertions(+), 72 deletions(-) create mode 100755 test/T095-search-unique.sh -- 2.1.0 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 03/11] cli/insert: move add_file_to_database to a better place
Move add_file_to_database around to keep the filesystem related
functions grouped together, improving readability. No functional
changes.
---
notmuch-insert.c | 92 +++---
1 files changed, 46 insertions(+), 46 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 770275b..ccb091a 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -294,52 +294,6 @@ copy_stdin (int fdin, int fdout)
return (!interrupted !empty);
}
-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
-{
-notmuch_message_t *message;
-notmuch_status_t status;
-
-status = notmuch_database_add_message (notmuch, path, message);
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
- fprintf (stderr, Error: failed to add `%s' to notmuch database: %s\n,
-path, notmuch_status_to_string (status));
- return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
- /* Don't change tags of an existing message. */
- if (synchronize_flags) {
- status = notmuch_message_tags_to_maildir_flags (message);
- if (status != NOTMUCH_STATUS_SUCCESS)
- fprintf (stderr, Error: failed to sync tags to maildir
flags\n);
- }
-} else {
- tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
-
- tag_op_list_apply (message, tag_ops, flags);
-}
-
-notmuch_message_destroy (message);
-}
-
static notmuch_bool_t
write_message (void *ctx, int fdin, const char *dir, char **newpath)
{
@@ -389,6 +343,52 @@ write_message (void *ctx, int fdin, const char *dir, char
**newpath)
return FALSE;
}
+/* Add the specified message file to the notmuch database, applying tags.
+ * The file is renamed to encode notmuch tags as maildir flags. */
+static void
+add_file_to_database (notmuch_database_t *notmuch, const char *path,
+ tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+{
+notmuch_message_t *message;
+notmuch_status_t status;
+
+status = notmuch_database_add_message (notmuch, path, message);
+switch (status) {
+case NOTMUCH_STATUS_SUCCESS:
+case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ break;
+default:
+case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+case NOTMUCH_STATUS_OUT_OF_MEMORY:
+case NOTMUCH_STATUS_FILE_ERROR:
+case NOTMUCH_STATUS_NULL_POINTER:
+case NOTMUCH_STATUS_TAG_TOO_LONG:
+case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
+case NOTMUCH_STATUS_LAST_STATUS:
+ fprintf (stderr, Error: failed to add `%s' to notmuch database: %s\n,
+path, notmuch_status_to_string (status));
+ return;
+}
+
+if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+ /* Don't change tags of an existing message. */
+ if (synchronize_flags) {
+ status = notmuch_message_tags_to_maildir_flags (message);
+ if (status != NOTMUCH_STATUS_SUCCESS)
+ fprintf (stderr, Error: failed to sync tags to maildir
flags\n);
+ }
+} else {
+ tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+
+ tag_op_list_apply (message, tag_ops, flags);
+}
+
+notmuch_message_destroy (message);
+}
+
int
notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
{
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 01/11] lib: actually return failures from notmuch_message_tags_to_maildir_flags
The function takes great care to preserve the first error status it encounters, yet fails to return that status to the caller. Fix it. --- lib/message.cc |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/lib/message.cc b/lib/message.cc index 68f7e68..7e82548 100644 --- a/lib/message.cc +++ b/lib/message.cc @@ -1497,7 +1497,7 @@ notmuch_message_tags_to_maildir_flags (notmuch_message_t *message) talloc_free (to_set); talloc_free (to_clear); -return NOTMUCH_STATUS_SUCCESS; +return status; } notmuch_status_t -- 1.7.2.5 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 02/11] cli/insert: rename check_folder_name to is_valid_folder_name
An is something predicate conveys the meaning better. While at it,
improve the function documentation and error message. Besides the
error message change, no functional changes.
---
notmuch-insert.c | 13 -
1 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 8dfc8bb..770275b 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -83,10 +83,13 @@ sync_dir (const char *dir)
return ret;
}
-/* Check the specified folder name does not contain a directory
- * component .. to prevent writes outside of the Maildir hierarchy. */
+/*
+ * Check the specified folder name does not contain a directory
+ * component .. to prevent writes outside of the Maildir
+ * hierarchy. Return TRUE on valid folder name, FALSE otherwise.
+ */
static notmuch_bool_t
-check_folder_name (const char *folder)
+is_valid_folder_name (const char *folder)
{
const char *p = folder;
@@ -449,8 +452,8 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
if (folder == NULL) {
maildir = db_path;
} else {
- if (! check_folder_name (folder)) {
- fprintf (stderr, Error: bad folder name: %s\n, folder);
+ if (! is_valid_folder_name (folder)) {
+ fprintf (stderr, Error: invalid folder name: '%s'\n, folder);
return EXIT_FAILURE;
}
maildir = talloc_asprintf (config, %s/%s, db_path, folder);
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 06/11] cli/insert: use a single recursive mkdir function
Combine make_directory() and make_directory_and_parents() into a
single recursive mkdir_recursive() function. Clarify the code and
improve error handling. Improve error messages. Switch to using the
new function in maildir_create_folder(). Constify talloc context.
---
notmuch-insert.c | 131 +++---
1 files changed, 55 insertions(+), 76 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 7375c54..cdeeb41 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -104,96 +104,78 @@ is_valid_folder_name (const char *folder)
}
}
-/* Make the given directory, succeeding if it already exists. */
+/*
+ * Make the given directory and its parents as necessary, using the
+ * given mode. Return TRUE on success, FALSE otherwise. Partial
+ * results are not cleaned up on errors.
+ */
static notmuch_bool_t
-make_directory (char *path, int mode)
+mkdir_recursive (const void *ctx, const char *path, int mode)
{
-notmuch_bool_t ret;
-char *slash;
+struct stat st;
+int r;
+char *parent = NULL, *slash;
-if (mkdir (path, mode) != 0)
- return (errno == EEXIST);
+/* First check the common case: directory already exists. */
+r = stat (path, st);
+if (r == 0) {
+if (! S_ISDIR (st.st_mode)) {
+ fprintf (stderr, Error: '%s' is not a directory: %s\n,
+path, strerror (EEXIST));
+ return FALSE;
+ }
-/* Sync the parent directory for durability. */
-ret = TRUE;
-slash = strrchr (path, '/');
-if (slash) {
- *slash = '\0';
- ret = sync_dir (path);
- *slash = '/';
+ return TRUE;
+} else if (errno != ENOENT) {
+ fprintf (stderr, Error: stat '%s': %s\n, path, strerror (errno));
+ return FALSE;
}
-return ret;
-}
-
-/* Make the given directory including its parent directories as necessary.
- * Return TRUE on success, FALSE on error. */
-static notmuch_bool_t
-make_directory_and_parents (char *path, int mode)
-{
-struct stat st;
-char *start;
-char *end;
-notmuch_bool_t ret;
-/* First check the common case: directory already exists. */
-if (stat (path, st) == 0)
- return S_ISDIR (st.st_mode) ? TRUE : FALSE;
-
-for (start = path; *start != '\0'; start = end + 1) {
- /* start points to the first unprocessed character.
-* Find the next slash from start onwards. */
- end = strchr (start, '/');
-
- /* If there are no more slashes then all the parent directories
-* have been made. Now attempt to make the whole path. */
- if (end == NULL)
- return make_directory (path, mode);
-
- /* Make the path up to the next slash, unless the current
-* directory component is actually empty. */
- if (end start) {
- *end = '\0';
- ret = make_directory (path, mode);
- *end = '/';
- if (! ret)
- return FALSE;
+/* mkdir parents, if any */
+slash = strrchr (path, '/');
+if (slash slash != path) {
+ parent = talloc_strndup (ctx, path, slash - path);
+ if (! parent) {
+ fprintf (stderr, Error: %s\n, strerror (ENOMEM));
+ return FALSE;
}
+
+ if (! mkdir_recursive (ctx, parent, mode))
+ return FALSE;
}
-return TRUE;
+if (mkdir (path, mode)) {
+ fprintf (stderr, Error: mkdir '%s': %s\n, path, strerror (errno));
+ return FALSE;
+}
+
+return parent ? sync_dir (parent) : TRUE;
}
-/* Create the given maildir folder, i.e. dir and its subdirectories
- * 'cur', 'new', 'tmp'. */
+/*
+ * Create the given maildir folder, i.e. maildir and its
+ * subdirectories cur/new/tmp. Return TRUE on success, FALSE
+ * otherwise. Partial results are not cleaned up on errors.
+ */
static notmuch_bool_t
-maildir_create_folder (void *ctx, const char *dir)
+maildir_create_folder (const void *ctx, const char *maildir)
{
+const char *subdirs[] = { cur, new, tmp };
const int mode = 0700;
char *subdir;
-char *tail;
-
-/* Create 'cur' directory, including parent directories. */
-subdir = talloc_asprintf (ctx, %s/cur, dir);
-if (! subdir) {
- fprintf (stderr, Out of memory.\n);
- return FALSE;
-}
-if (! make_directory_and_parents (subdir, mode))
- return FALSE;
-
-tail = subdir + strlen (subdir) - 3;
+unsigned int i;
-/* Create 'new' directory. */
-strcpy (tail, new);
-if (! make_directory (subdir, mode))
- return FALSE;
+for (i = 0; i ARRAY_SIZE (subdirs); i++) {
+ subdir = talloc_asprintf (ctx, %s/%s, maildir, subdirs[i]);
+ if (! subdir) {
+ fprintf (stderr, Error: %s\n, strerror (ENOMEM));
+ return FALSE;
+ }
-/* Create 'tmp' directory. */
-strcpy (tail, tmp);
-if (! make_directory (subdir, mode))
- return FALSE;
+ if (! mkdir_recursive (ctx, subdir,
[PATCH 04/11] cli/insert: rename file copy function
The copying has nothing to do with stdin, so call it copy_fd
instead. While at it, improve documentation and reverse the
parameters, as destination is traditionally the first parameter.
---
notmuch-insert.c | 11 ++-
1 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index ccb091a..5d47806 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -251,11 +251,12 @@ maildir_open_tmp_file (void *ctx, const char *dir,
return fd;
}
-/* Copy the contents of standard input (fdin) into fdout.
- * Returns TRUE if a non-empty file was written successfully.
- * Otherwise, return FALSE. */
+/*
+ * Copy fdin to fdout, return TRUE on success, and FALSE on errors and
+ * empty input.
+ */
static notmuch_bool_t
-copy_stdin (int fdin, int fdout)
+copy_fd (int fdout, int fdin)
{
notmuch_bool_t empty = TRUE;
@@ -308,7 +309,7 @@ write_message (void *ctx, int fdin, const char *dir, char
**newpath)
cleanup_path = tmppath;
-if (! copy_stdin (fdin, fdout))
+if (! copy_fd (fdout, fdin))
goto FAIL;
if (fsync (fdout) != 0) {
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 05/11] cli/insert: clean up sync_dir
Clarify the code slightly, improve error messages. Apart from the
error message changes, no functional changes.
---
notmuch-insert.c | 17 +
1 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5d47806..7375c54 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -67,20 +67,21 @@ safe_gethostname (char *hostname, size_t len)
static notmuch_bool_t
sync_dir (const char *dir)
{
-notmuch_bool_t ret;
-int fd;
+int fd, r;
fd = open (dir, O_RDONLY);
if (fd == -1) {
- fprintf (stderr, Error: open() dir failed: %s\n, strerror (errno));
+ fprintf (stderr, Error: open %s: %s\n, dir, strerror (errno));
return FALSE;
}
-ret = (fsync (fd) == 0);
-if (! ret) {
- fprintf (stderr, Error: fsync() dir failed: %s\n, strerror (errno));
-}
+
+r = fsync (fd);
+if (r)
+ fprintf (stderr, Error: fsync %s: %s\n, dir, strerror (errno));
+
close (fd);
-return ret;
+
+return r == 0;
}
/*
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 07/11] cli/insert: abstract temporary filename generation
This will clean up the usage. There's the slight functional change of
potentially ending up doing extra gethostname and getpid calls, but
this is neglible.
---
notmuch-insert.c | 39 +++
1 files changed, 27 insertions(+), 12 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index cdeeb41..a1d564c 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -179,6 +179,31 @@ maildir_create_folder (const void *ctx, const char
*maildir)
return TRUE;
}
+/*
+ * Generate a temporary file basename, no path, do not create an
+ * actual file. Return the basename, or NULL on errors.
+ */
+static char *
+tempfilename (const void *ctx)
+{
+char *filename;
+char hostname[256];
+struct timeval tv;
+pid_t pid;
+
+/* We follow the Dovecot file name generation algorithm. */
+pid = getpid ();
+safe_gethostname (hostname, sizeof (hostname));
+gettimeofday (tv, NULL);
+
+filename = talloc_asprintf (ctx, %ld.M%ldP%d.%s,
+ tv.tv_sec, tv.tv_usec, pid, hostname);
+if (! filename)
+ fprintf (stderr, Error: %s\n, strerror (ENOMEM));
+
+return filename;
+}
+
/* Open a unique file in the 'tmp' sub-directory of dir.
* Returns the file descriptor on success, or -1 on failure.
* On success, file paths for the message in the 'tmp' and 'new'
@@ -188,23 +213,13 @@ static int
maildir_open_tmp_file (void *ctx, const char *dir,
char **tmppath, char **newpath, char **newdir)
{
-pid_t pid;
-char hostname[256];
-struct timeval tv;
char *filename;
int fd = -1;
-/* We follow the Dovecot file name generation algorithm. */
-pid = getpid ();
-safe_gethostname (hostname, sizeof (hostname));
do {
- gettimeofday (tv, NULL);
- filename = talloc_asprintf (ctx, %ld.M%ldP%d.%s,
- tv.tv_sec, tv.tv_usec, pid, hostname);
- if (! filename) {
- fprintf (stderr, Out of memory\n);
+ filename = tempfilename (ctx);
+ if (! filename)
return -1;
- }
*tmppath = talloc_asprintf (ctx, %s/tmp/%s, dir, filename);
if (! *tmppath) {
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 09/11] cli/insert: add fail path to add_file_to_database
Handle failures gracefully in add_file_to_database, renamed simply
add_file while at it. Add keep option to not remove the message from
database if tagging or tag syncing to maildir flags fails. Expand the
function documentation to cover the changes.
---
notmuch-insert.c | 89 -
1 files changed, 54 insertions(+), 35 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 5ef6e66..80f52d4 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -364,50 +364,70 @@ FAIL:
return NULL;
}
-/* Add the specified message file to the notmuch database, applying tags.
- * The file is renamed to encode notmuch tags as maildir flags. */
-static void
-add_file_to_database (notmuch_database_t *notmuch, const char *path,
- tag_op_list_t *tag_ops, notmuch_bool_t synchronize_flags)
+/*
+ * Add the specified message file to the notmuch database, applying
+ * tags in tag_ops. If synchronize_flags is TRUE, the tags are
+ * synchronized to maildir flags (which may result in message file
+ * rename).
+ *
+ * Return NOTMUCH_STATUS_SUCCESS on success, errors otherwise. If keep
+ * is TRUE, errors in tag changes and flag syncing are ignored and
+ * success status is returned; otherwise such errors cause the message
+ * to be removed from the database. Failure to add the message to the
+ * database results in error status regardless of keep.
+ */
+static notmuch_status_t
+add_file (notmuch_database_t *notmuch, const char *path, tag_op_list_t
*tag_ops,
+ notmuch_bool_t synchronize_flags, notmuch_bool_t keep)
{
notmuch_message_t *message;
notmuch_status_t status;
status = notmuch_database_add_message (notmuch, path, message);
-switch (status) {
-case NOTMUCH_STATUS_SUCCESS:
-case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- break;
-default:
-case NOTMUCH_STATUS_FILE_NOT_EMAIL:
-case NOTMUCH_STATUS_READ_ONLY_DATABASE:
-case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
-case NOTMUCH_STATUS_OUT_OF_MEMORY:
-case NOTMUCH_STATUS_FILE_ERROR:
-case NOTMUCH_STATUS_NULL_POINTER:
-case NOTMUCH_STATUS_TAG_TOO_LONG:
-case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
-case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
-case NOTMUCH_STATUS_LAST_STATUS:
- fprintf (stderr, Error: failed to add `%s' to notmuch database: %s\n,
-path, notmuch_status_to_string (status));
- return;
-}
-
-if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
- /* Don't change tags of an existing message. */
- if (synchronize_flags) {
- status = notmuch_message_tags_to_maildir_flags (message);
- if (status != NOTMUCH_STATUS_SUCCESS)
- fprintf (stderr, Error: failed to sync tags to maildir
flags\n);
+if (status == NOTMUCH_STATUS_SUCCESS) {
+ status = tag_op_list_apply (message, tag_ops, 0);
+ if (status) {
+ fprintf (stderr, %s: failed to apply tags to file '%s': %s\n,
+keep ? Warning : Error,
+path, notmuch_status_to_string (status));
+ goto DONE;
}
+} else if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+ status = NOTMUCH_STATUS_SUCCESS;
+} else if (status == NOTMUCH_STATUS_FILE_NOT_EMAIL) {
+ fprintf (stderr, Error: delivery of non-mail file: '%s'\n, path);
+ return status;
} else {
- tag_op_flag_t flags = synchronize_flags ? TAG_FLAG_MAILDIR_SYNC : 0;
+ fprintf (stderr, Error: failed to add '%s' to notmuch database: %s\n,
+path, notmuch_status_to_string (status));
+ return status;
+}
- tag_op_list_apply (message, tag_ops, flags);
+if (synchronize_flags) {
+ status = notmuch_message_tags_to_maildir_flags (message);
+ if (status != NOTMUCH_STATUS_SUCCESS)
+ fprintf (stderr, %s: failed to sync tags to maildir flags for
'%s': %s\n,
+keep ? Warning : Error,
+path, notmuch_status_to_string (status));
+
+ /*
+* Note: Unfortunately a failed maildir flag sync might
+* already have renamed the file, in which case the cleanup
+* path will fail.
+*/
}
+DONE:
notmuch_message_destroy (message);
+
+if (status) {
+ if (keep)
+ status = NOTMUCH_STATUS_SUCCESS;
+ else
+ notmuch_database_remove_message (notmuch, path);
+}
+
+return status;
}
int
@@ -508,8 +528,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
/* Add the message to the index.
* Even if adding the message to the notmuch database fails,
* the message is on disk and we consider the delivery completed. */
-add_file_to_database (notmuch, newpath, tag_ops,
- synchronize_flags);
+add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
notmuch_database_destroy (notmuch);
[PATCH 10/11] cli/insert: require succesful message indexing for success status
Add --keep option to keep any remaining stuff in index or file. We
could distinguish between failures to index and failures to apply tags
or maildir sync, but for simplicity just have one.
---
notmuch-insert.c| 20 +++-
test/T070-insert.sh |2 +-
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/notmuch-insert.c b/notmuch-insert.c
index 80f52d4..f27b9cb 100644
--- a/notmuch-insert.c
+++ b/notmuch-insert.c
@@ -433,6 +433,7 @@ DONE:
int
notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[])
{
+notmuch_status_t status;
notmuch_database_t *notmuch;
struct sigaction action;
const char *db_path;
@@ -442,6 +443,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
char *query_string = NULL;
const char *folder = NULL;
notmuch_bool_t create_folder = FALSE;
+notmuch_bool_t keep = FALSE;
notmuch_bool_t synchronize_flags;
const char *maildir;
char *newpath;
@@ -451,6 +453,7 @@ notmuch_insert_command (notmuch_config_t *config, int argc,
char *argv[])
notmuch_opt_desc_t options[] = {
{ NOTMUCH_OPT_STRING, folder, folder, 0, 0 },
{ NOTMUCH_OPT_BOOLEAN, create_folder, create-folder, 0, 0 },
+ { NOTMUCH_OPT_BOOLEAN, keep, keep, 0, 0 },
{ NOTMUCH_OPT_END, 0, 0, 0, 0 }
};
@@ -525,11 +528,18 @@ notmuch_insert_command (notmuch_config_t *config, int
argc, char *argv[])
return EXIT_FAILURE;
}
-/* Add the message to the index.
- * Even if adding the message to the notmuch database fails,
- * the message is on disk and we consider the delivery completed. */
-add_file (notmuch, newpath, tag_ops, synchronize_flags, TRUE);
+/* Index the message. */
+status = add_file (notmuch, newpath, tag_ops, synchronize_flags, keep);
+if (status) {
+ if (keep) {
+ status = NOTMUCH_STATUS_SUCCESS;
+ } else {
+ /* If maildir flag sync failed, this might fail. */
+ unlink (newpath);
+ }
+}
notmuch_database_destroy (notmuch);
-return EXIT_SUCCESS;
+
+return status ? EXIT_FAILURE : EXIT_SUCCESS;
}
diff --git a/test/T070-insert.sh b/test/T070-insert.sh
index ea9db07..aacc643 100755
--- a/test/T070-insert.sh
+++ b/test/T070-insert.sh
@@ -23,7 +23,7 @@ test_expect_code 1 Insert zero-length file \
# This test is a proxy for other errors that may occur while trying to
# add a message to the notmuch database, e.g. database locked.
-test_expect_code 0 Insert non-message \
+test_expect_code 1 Insert non-message \
echo bad_message | notmuch insert
test_begin_subtest Database empty so far
--
1.7.2.5
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
Excerpts from Gaute Hope's message of August 6, 2014 10:29: Austin Clements amdra...@mit.edu wrote on Fri, 01 Aug 2014 14:55:05 -0400: I have a prototype implementation of message modification times on my lastmod-v1 branch at https://github.com/aclements/notmuch/tree/lastmod-v1 It builds on my database features series that's currently awaiting review [1]. The series uses a monotonic revision number, rather than wall-clock time, for reasons related to Xapian's concurrent control and detailed in the main commit's commit message. The implementation isn't quite useful from the CLI yet because I haven't added any way to query the database's current revision number. (I'm still thinking about how I want to do this, since search/show don't have a good way to deliver additional information right now. I might just add the last modification for each individual message/max of all messages in a thread, similar to what Thomas Jost's patch did long ago.) [1] id:1406859003-11561-1-git-send-email-amdra...@mit.edu this should allow me to do what I wish to accomplish. The message deletion is still a problem though, I can see two options at the moment: Hi list, While exploring the possibility of syncing maildir/X-keywords with tags I had some thoughts about lastmod and message modification: As briefly discussed on #notmuch, I noticed that it seems that 'notmuch new' does not detect that a message source has been changed, unless the file is also re-named. This means that for instance if the X-Keywords fields have been updated in a message (from GMail with offlineimap, synclabels = yes) the lastmod field will remain unchanged, and a source modification will be undetectable to a client program using this value. Would it not make sense that if a message has a more recent mtime than at index time it is re-indexed? Also, for the lastmod branch I would wish for a notmuch_message_touch() method where the lastmod time is updated to the last. As well as a notmuch_database_reindex_message () - possibly defined/documented behaviour for notmuch_database_add_message () when the filename is already added (in which case I would expect notmuch to re-index the message). Doing notmuch_database_remove_message followed by _add_message could risk deleting the entry if this file is the only on-disk-representation. Cheers, Gaute ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Do path: searches handle spaces correctly?
Quoth Keith Amidon on Sep 22 at 7:42 am: On Mon, 2014-09-22 at 11:20 +0200, David Bremner wrote: Keith Amidon ke...@awakenetworks.com writes: notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items' I don't get any results, but it seems like the two results above should be shown, right? Am I doing something wrong here? If it looks like I'm doing it correctly, what can I do to help troubleshoot the issue? Note that path:, unlike folder:, does not add the maildir subdirs. Dunno if that's the only issue you are having, but I guess it's one. Darn it! I made a mistake in my original email. In the test I was doing I actually had: notmuch search --output=files path:'dir/INBOX/INBOX/Sent Items/**' which I believe should have picked up all the subdirectory paths. I just retested to make sure that it still didn't work, and it doesn't. Am I still missing something? Thanks for the help, Keith I assume you're doing this from the command line? Does the following work? notmuch search --output=files 'path:dir/INBOX/INBOX/Sent Items/**' Shell quoting and Xapian quoting interact in often confusing ways. In your original command line, the single quotes suppress shell argument splitting, but never make it to notmuch, so notmuch sees two search terms path:dir/INBOX/INBOX/Sent and Items/**. In the command line I suggested, the single quotes play the same role, but for the entire query. Because of the quoting, notmuch *does* see the inner double quotes, which makes the path a single term (note that Xapian only accepts double quotes, not single quotes). In general, it's good to enclose the entire command line query in single quotes so shell parsing doesn't get in the way of query parsing. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
On Mon, Sep 22 2014, Gaute Hope e...@gaute.vetsj.com wrote: Excerpts from Gaute Hope's message of August 6, 2014 10:29: Austin Clements amdra...@mit.edu wrote on Fri, 01 Aug 2014 14:55:05 -0400: I have a prototype implementation of message modification times on my lastmod-v1 branch at https://github.com/aclements/notmuch/tree/lastmod-v1 It builds on my database features series that's currently awaiting review [1]. The series uses a monotonic revision number, rather than wall-clock time, for reasons related to Xapian's concurrent control and detailed in the main commit's commit message. The implementation isn't quite useful from the CLI yet because I haven't added any way to query the database's current revision number. (I'm still thinking about how I want to do this, since search/show don't have a good way to deliver additional information right now. I might just add the last modification for each individual message/max of all messages in a thread, similar to what Thomas Jost's patch did long ago.) [1] id:1406859003-11561-1-git-send-email-amdra...@mit.edu this should allow me to do what I wish to accomplish. The message deletion is still a problem though, I can see two options at the moment: Hi list, While exploring the possibility of syncing maildir/X-keywords with tags I had some thoughts about lastmod and message modification: As briefly discussed on #notmuch, I noticed that it seems that 'notmuch new' does not detect that a message source has been changed, unless the file is also re-named. This means that for instance if the X-Keywords fields have been updated in a message (from GMail with offlineimap, synclabels = yes) the lastmod field will remain unchanged, and a source modification will be undetectable to a client program using this value. Would it not make sense that if a message has a more recent mtime than at index time it is re-indexed? That would require notmuch to scan the contents of a directory for changed mtimes of the files -- now notmuch skips looking for the files unless directory mtime has changed. Directory mtime changes when files are added/deleted/renamed (as that is what directory needs to know) -- file mtime change are stored in file information and therefore change there does not probagate to parent directory (and, if such happened, to it's parent and so on...) That would mean the scanning would be slower than it is now. Tomi Also, for the lastmod branch I would wish for a notmuch_message_touch() method where the lastmod time is updated to the last. As well as a notmuch_database_reindex_message () - possibly defined/documented behaviour for notmuch_database_add_message () when the filename is already added (in which case I would expect notmuch to re-index the message). Doing notmuch_database_remove_message followed by _add_message could risk deleting the entry if this file is the only on-disk-representation. Cheers, Gaute ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
On Mon, 22 Sep 2014, Gaute Hope e...@gaute.vetsj.com wrote: Excerpts from Gaute Hope's message of August 6, 2014 10:29: Austin Clements amdra...@mit.edu wrote on Fri, 01 Aug 2014 14:55:05 -0400: I have a prototype implementation of message modification times on my lastmod-v1 branch at https://github.com/aclements/notmuch/tree/lastmod-v1 It builds on my database features series that's currently awaiting review [1]. The series uses a monotonic revision number, rather than wall-clock time, for reasons related to Xapian's concurrent control and detailed in the main commit's commit message. The implementation isn't quite useful from the CLI yet because I haven't added any way to query the database's current revision number. (I'm still thinking about how I want to do this, since search/show don't have a good way to deliver additional information right now. I might just add the last modification for each individual message/max of all messages in a thread, similar to what Thomas Jost's patch did long ago.) [1] id:1406859003-11561-1-git-send-email-amdra...@mit.edu this should allow me to do what I wish to accomplish. The message deletion is still a problem though, I can see two options at the moment: Hi list, While exploring the possibility of syncing maildir/X-keywords with tags I had some thoughts about lastmod and message modification: As briefly discussed on #notmuch, I noticed that it seems that 'notmuch new' does not detect that a message source has been changed, unless the file is also re-named. This means that for instance if the X-Keywords fields have been updated in a message (from GMail with offlineimap, synclabels = yes) the lastmod field will remain unchanged, and a source modification will be undetectable to a client program using this value. Would it not make sense that if a message has a more recent mtime than at index time it is re-indexed? This has the potential to make notmuch new substantially more expensive. Currently, if there are no changes, it only has to stat each directory in your maildir (in fact, some restructuring of new would let us eliminate almost all database access during a no-op notmuch new as well). Checking for changes to individual messages would require stat'ing every single message file as well as accessing the database to check the paths and mtimes of every message, increasing the number of stat calls and disk accesses by several orders of magnitude. It may be that this is fast enough that it's okay, but it would be good to gather some evidence first. That includes hot and cold caches, and maildir over NFS. With respect to X-Keywords specifically, note that it's a fairly basic design decision that notmuch never modifies message files. This gives us strong robustness guarantees we would be loathe to part with. It has puzzled me ever since offlineimap added X-Keywords why they didn't just translate these keywords into folders and create hard links of message files. Anything could interact smoothly with that. Also, for the lastmod branch I would wish for a notmuch_message_touch() method where the lastmod time is updated to the last. As well as a notmuch_database_reindex_message () - possibly defined/documented behaviour for notmuch_database_add_message () when the filename is already added (in which case I would expect notmuch to re-index the message). What's the use case for these? Doing notmuch_database_remove_message followed by _add_message could risk deleting the entry if this file is the only on-disk-representation. Cheers, Gaute ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] lib: Simplify close and codify aborting atomic section
In Xapian, closing a database implicitly aborts any outstanding
transaction and commits changes. For historical reasons,
notmuch_database_close had grown to almost, but not quite duplicate
this behavior. Before closing the database, it would explicitly (and
unnecessarily) commit it. However, if there was an outstanding
transaction (ie atomic section), commit would throw a Xapian
exception, which notmuch_database_close would unnecessarily print to
stderr, even though notmuch_database_close would ultimately abort the
transaction anyway when it called close.
This patch simplifies notmuch_database_close to just call
Database::close. This works for both read-only and read/write
databases, takes care of committing changes, unifies the exception
handling path, and codifies aborting outstanding transactions. This
is currently the only way to abort an atomic section (and may remain
so, since it would be difficult to roll back things we may have cached
from rolled-back modifications).
---
lib/database.cc | 23 +++
lib/notmuch.h | 5 +
2 files changed, 12 insertions(+), 16 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index a3a7cd3..1f7ff2a 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -903,28 +903,19 @@ notmuch_database_close (notmuch_database_t *notmuch)
{
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
-try {
- if (notmuch-xapian_db != NULL
- notmuch-mode == NOTMUCH_DATABASE_MODE_READ_WRITE)
- (static_cast Xapian::WritableDatabase *
(notmuch-xapian_db))-flush ();
-} catch (const Xapian::Error error) {
- status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
- if (! notmuch-exception_reported) {
- fprintf (stderr, Error: A Xapian exception occurred flushing
database: %s\n,
-error.get_msg().c_str());
- }
-}
-
/* Many Xapian objects (and thus notmuch objects) hold references to
* the database, so merely deleting the database may not suffice to
- * close it. Thus, we explicitly close it here. */
+ * close it. Thus, we explicitly close it here. This will
+ * implicitly abort any outstanding transaction and commit changes. */
if (notmuch-xapian_db != NULL) {
try {
notmuch-xapian_db-close();
} catch (const Xapian::Error error) {
- /* don't clobber previous error status */
- if (status == NOTMUCH_STATUS_SUCCESS)
- status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ if (! notmuch-exception_reported) {
+ fprintf (stderr, Error: A Xapian exception occurred closing
database: %s\n,
+error.get_msg().c_str());
+ }
}
}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index fe2340b..5c40c67 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -292,6 +292,11 @@ notmuch_database_open (const char *path,
* notmuch_database_close can be called multiple times. Later calls
* have no effect.
*
+ * If the caller is currently in an atomic section (there was a
+ * notmuch_database_begin_atomic without a matching
+ * notmuch_database_end_atomic), this will abort the atomic section,
+ * discarding any modifications made in the atomic section.
+ *
* Return value:
*
* NOTMUCH_STATUS_SUCCESS: Successfully closed the database.
--
2.1.0
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] lib: Simplify close and codify aborting atomic section
On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote: This patch simplifies notmuch_database_close to just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. If we're dropping the flush call here, where will it be triggered instead? We'll need to flush/commit our changes to the database at some point before closing. Do clients now need an explicit flush/commit command (explicit, client-initiated flushes sound like a good idea to me). Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [announce] nmhive v0.1.0, a bookmarklet/server for nmbug tags
On Mon, Sep 22, 2014 at 10:19:35AM -0700, W. Trevor King wrote: I like nmbug's distributed tag maintenance, but not everyone has notmuch/nmbug installed locally (yet ;). However, everyone that I know does have a browser and a mail client. They can submit messages with their mail client already, but we've been missing a way for them to help tag messages. Ah, and the other piece to this workflow is the existing nmbug-status, which collects the results of canned searches so folks without a local notmuch can use the tags [1]. Folks using a local nmhive will probably want to run their own status-genertion via a post-commit hook in their nmhive repository. Then their users will have their search-results updated after each web-initiated change. If you also wanted them to see updates from changes to tethera's nmbug repository, you'd probably also want a cron job that tried to fetch and merge tethera's changes with the nmhive changes: -o---o---oo tethera/master \\ \o nmhive/status (auto-generated merge for nmbug-status) \ / o---oo nmhive/master (with web-initiated changes) You'd want to resolve conflicts somehow, but any resolution strategy is probably fine, since it's unlikely that we get conflicts very often. Cheers, Trevor [1]: http://nmbug.tethera.net/status/ -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] lib: Simplify close and codify aborting atomic section
Quoth W. Trevor King on Sep 22 at 9:59 am: On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote: This patch simplifies notmuch_database_close to just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. If we're dropping the flush call here, where will it be triggered instead? We'll need to flush/commit our changes to the database at some point before closing. Do clients now need an explicit flush/commit command (explicit, client-initiated flushes sound like a good idea to me). The call to Database::close implicitly flushes/commits, as mentioned in the comment in the patch, so there's no need for any new APIs or client changes. The call to Database::flush in notmuch_database_close was entirely redundant with the call to Database::close. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: jump: fix compile warning on emacs 23
LGTM. I'm a little surprised this is necessary, but whatever. I think the eval-and-compile has to be top-level; it's certainly not wrong for it to be top-level. (I like the comment in the eval-and-compile implementation: ;; Remember, it's magic.) On Thu, 04 Sep 2014, Mark Walters markwalters1...@gmail.com wrote: notmuch-jump uses window-body-width which is not defined in emacs 23. To get around this it does (unless (fboundp 'window-body-width) ;; Compatibility for Emacs pre-24 (defalias 'window-body-width 'window-width)) This makes sure window-body-width is defined and all should be well. But it seems that the byte compiler does not realise that this guarantees that window-body-width will be defined and so, when compiling with emacs 23, it gives an error In end of data: notmuch-jump.el:172:1:Warning: the function `window-body-width' is not known to be defined. Domo and I came to following on irc: wrap the (unless (fboundp ...)) inside eval-and-compile which ensures that both the test and the defalias (if needed) happen at both compile and load time. This fixes the warning. --- I think Domo and I were both not completely sure whether the eval-and-compile should be inside or outside the (unless fboundp ..) or not (ie should it wrap the unless fboundp or just the defalias) nor whether it should be eval-and-compile or eval-when-compile. We think this is the right version but it would be good to have confirmation. I tested notmuch jump inside emacs 23 and 24 with notmuch-emacs compiled with emacs 23 or 24 (ie all four combinations) and it seemed to work. Best wishes Mark emacs/notmuch-jump.el |7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/emacs/notmuch-jump.el b/emacs/notmuch-jump.el index 5eb0949..2706b6c 100644 --- a/emacs/notmuch-jump.el +++ b/emacs/notmuch-jump.el @@ -25,9 +25,10 @@ (require 'notmuch-lib) (require 'notmuch-hello) -(unless (fboundp 'window-body-width) - ;; Compatibility for Emacs pre-24 - (defalias 'window-body-width 'window-width)) +(eval-and-compile + (unless (fboundp 'window-body-width) +;; Compatibility for Emacs pre-24 +(defalias 'window-body-width 'window-width))) ;;;###autoload (defun notmuch-jump-search () -- 1.7.10.4 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] lib: Simplify close and codify aborting atomic section
On Mon, Sep 22, 2014 at 06:50:50PM +, Austin Clements wrote: Quoth W. Trevor King on Sep 22 at 9:59 am: On Mon, Sep 22, 2014 at 11:43:35AM -0400, Austin Clements wrote: This patch simplifies notmuch_database_close to just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. If we're dropping the flush call here, where will it be triggered instead? We'll need to flush/commit our changes to the database at some point before closing. Do clients now need an explicit flush/commit command (explicit, client-initiated flushes sound like a good idea to me). The call to Database::close implicitly flushes/commits, as mentioned in the comment in the patch, so there's no need for any new APIs or client changes. The call to Database::flush in notmuch_database_close was entirely redundant with the call to Database::close. Ah, I thought the implicit flush/commit was just in our code. Since it's also in the underlying Xapian close, then this patch looks pretty good to me. I'd mention Xapian's explicit close in the notmuch.h message. Xapain's docs say [1]: For a WritableDatabase, if a transaction is active it will be aborted, while if no transaction is active commit() will be implicitly called. Cheers, Trevor [1]: http://xapian.org/docs/apidoc/html/classXapian_1_1Database.html#a59f5f8b137723dcaaabdbdccbc0cf1eb -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Do path: searches handle spaces correctly?
On Mon, 2014-09-22 at 15:06 +, Austin Clements wrote: I assume you're doing this from the command line? Does the following work? notmuch search --output=files 'path:dir/INBOX/INBOX/Sent Items/**' Shell quoting and Xapian quoting interact in often confusing ways. In your original command line, the single quotes suppress shell argument splitting, but never make it to notmuch, so notmuch sees two search terms path:dir/INBOX/INBOX/Sent and Items/**. In the command line I suggested, the single quotes play the same role, but for the entire query. Because of the quoting, notmuch *does* see the inner double quotes, which makes the path a single term (note that Xapian only accepts double quotes, not single quotes). In general, it's good to enclose the entire command line query in single quotes so shell parsing doesn't get in the way of query parsing. Bingo. That was it. Thanks for clarifying how the quoting interacts. What do you think about adding a section to the notmuch-search-terms web page that talks in more detail about the quoting interactions? If there were on obvious section about this in the man page it might have helped me find the issue myself. If this seems like a good idea I'll take a crack at it later this week when I get time to research the Xapian quoting in a bit more detail. --- Keith ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
