a DoS vulnerability associated with conflated Message-IDs?

[Peter Wang]

On Thu, 08 Mar 2012 11:37:09 -0500, Daniel Kahn Gillmor  wrote:
> notmuch currently treats all messages with the same Message-ID as
> the same message.  I think this could be a vulnerability :(
> 
> If two messages have the same Message-ID, is there a guarantee of which
> of these messages will be produced during a notmuch show?
> 
> Either way, it seems to create a potential DoS attack on notmuch users.

Yesterday I was expecting a confirmation message which, seemingly, never
came.  It turns out my maildir already contained a message from the
same system.  From three years ago.  With the same Message-ID.

Malice has nothing on incompetence.

Could we distinguish messages with identical Message-IDs based on
some header fields, e.g. Date, From?

Peter

[PATCH] don't show x-foo tags in search view

[Austin Clements]

I like it.

Quoth James Vasile on Oct 29 at  5:19 pm:
> This patch hides any tags in search view that match the regex specified
> in `notmuch-search-hide-tag-regex`.  That variable can be set via setq
> or the customize interface.  To hide all tags that begin with "x-" or
> "X-", set `notmuch-search-hide-tag-regex` to "^X-".
> 
> ---
>  emacs/notmuch.el |   16 +++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index f9454d8..4bff538 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -775,6 +775,14 @@ non-authors is found, assume that all of the authors 
> match."
>   (overlay-put overlay 'isearch-open-invisible #'delete-overlay)))
>(insert padding
>  
> +(defcustom notmuch-search-hide-tag-regex ""
> +  "Regex specifying tags to hide in search view.

I have no idea why, but Emacs typically uses "regexp" instead of
"regex".

> +
> +Leave blank to disable hiding of tags in search view.

Saying "Leave blank" supposes that the user knows what the default
value is.  How about "An empty string disables hiding of tags in
search view."?

Even better, though, would be to use nil to indicate this, since "" is
a perfectly valid regexp and matches everything.  In that case, this
should say something like "If nil, no tags will be hidden in search
view."

> +Note: elisp regexes are case-insensitive"

Likewise, "regexps".  Also, Elisp regexps are not, in general,
case-insensitive.  If we want to control this, we should bind
case-fold-search to nil around the string-match below and say
something here like "Matching is case-insensitive."

> +  :type 'string

Better would be 'regexp.  Or, '(choice (const :tag "None" nil) regexp)
to allow nil or a regexp.

> +  :group 'notmuch-search)
> +
>  (defun notmuch-search-insert-field (field format-string result)
>(cond
> ((string-equal field "date")
> @@ -793,7 +801,13 @@ non-authors is found, assume that all of the authors 
> match."
>  (notmuch-search-insert-authors format-string (plist-get result 
> :authors)))
>  
> ((string-equal field "tags")
> -(let ((tags-str (mapconcat 'identity (plist-get result :tags) " ")))
> +(let ((tags-str (mapconcat 'identity
> +  (delq nil
> +(mapcar (lambda (x) (if (and (not (equal 
> notmuch-search-hide-tag-regex ""))
> + 
> (string-match notmuch-search-hide-tag-regex x))
> +nil
> +x)) (plist-get 
> result :tags)))
> +  " ")))

It would be simpler and more robust to use remove-if here.  What about
something like

  (let ((tags-str
 (mapconcat 'identity
(if notmuch-search-hide-tag-regex
(let ((case-fold-search t))
  (remove-if
   (apply-partially #'string-match
notmuch-search-hide-tag-regex)
   (plist-get result :tags)))
  (plist-get result :tags))
" ")))

?

>(insert (propertize (format format-string tags-str)
>   'face 'notmuch-tag-face))
>  

Notmuch search missing mail id

[Tomi Ollila]

On Mon, Oct 29 2012, James Vasile  wrote:

> I received the attached piece of spam telling me about an exciting
> investment opportunity.  Notmuch pulled it in to the database, noting
> the message id.  But then it seems to stop paying attention to the
> message id.
>
> notmuch can find the mail when I search by the from field:
>
> $ notmuch search "from:caroline horn"
> thread:bcb7  Today 09:29 [1/1] Caroline Horn; This Company is on 
> the rise (hv inbox unread)
>
> But not by the message id:
> $ notmuch search id:000801cdb5da$17673470$26448589 at microsof00t4ko3r
> $

Try 

$ notmuch search id:'000801cdb5da$17673470$26448589 at microsof00t4ko3r'

so that the $s are not expanded...

note that id:"..." does not suffice as shell expands $ & ` inside double
quotes.

Tomi


>
> Notmuch has correctly pulled the id into the database:
> $ notmuch show "from:caroline horn" | grep "id:" | sed "s/.*\(id:[^ 
> ]*\).*/\1/"
> id:000801cdb5da$17673470$26448589 at microsof00t4ko3r
>
> If notmuch knows the id but won't match it in a search, that looks like
> a bug to me.
>
> On a related matter, I have a script that pulls the message id header
> From a mail file and then uses notmuch tag id:foo to tag that file.  But
> if notmuch sometimes fails to find the id, maybe there is there a better
> way to do it?
>
> Thanks,
> James
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[BUG] Saving attachments containing UTF-8 chars

[Tomi Ollila]

On Mon, Oct 29 2012, Michael Stapelberg wrote:

> Hello,
>
> attached you can find a message which seems to trigger a bug somewhere
> in my notmuch setup. The message has been sent to me and includes a
> patchfile, which, when saved using the "w" key in notmuch-emacs, cannot
> be applied.
>
> When using munpack(1) on the raw mail file (I?ve attached rawmail.gz
> just to be sure that the message doesn?t get mangled in some way), the
> unpacked attachment can be attached just fine.
>
> When diff'ing both files, I see that munpack correctly stores the file,
> including the problematic UTF-8 apostrophe:
> 0200  67 3b 20 64 69 64 6e e2  80 99 74 20 72 65 61 64  |g; didn...t read|
>
> However, the version of the file saved with notmuch contains:
> 0200  67 3b 20 64 69 64 6e 19  74 20 72 65 61 64 22 20  |g; didn.t read" |
>
> If there is anything I can do to help debugging this (or if there is a
> known fix), please let me know.

I can verify this bug: I copied 'rawmail' to my mail store and attempted
to 'w' the attacment and got the same result (after notmuch new).

The saving code first does
notmuch show --format=raw id:"508953E6.70006 at gmail.com"
which decodes OK on command line, and to the buffer when 
kill-buffer is outcommented in (with-current-notmuch-show-message ...) 
macro.

I attempted a set of trial-&-error tricks to get the attachment
saved "correctly", and at least this seems to do the trick:

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index f273eb4..a6a85c0 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -203,9 +203,11 @@ For example, if you wanted to remove an \"unread\" tag and 
add a
  (let ((id (notmuch-show-get-message-id)))
(let ((buf (generate-new-buffer (concat "*notmuch-msg-" id "*"
  (with-current-buffer buf
-   (call-process notmuch-command nil t nil "show" "--format=raw" id)
-   , at body)
-(kill-buffer buf)
+  (let ((coding-system-for-read 'no-conversion)
+(coding-system-for-write 'no-conversion))
+(call-process notmuch-command nil t nil "show" "--format=raw" id)
+, at body))
+%%  (kill-buffer buf)

 (defun notmuch-show-turn-on-visual-line-mode ()
   "Enable Visual Line mode."

---

(kill-buffer is outcommented above for testing purposes)

To test this this needs to me evaluated and then the functions
using this macro (notmuch-show-save-attachments  in this case)

Smart suggestions for proper fix ?

Tomi

[PATCH] don't show x-foo tags in search view

[James Vasile]

This patch hides any tags in search view that match the regex specified
in `notmuch-search-hide-tag-regex`.  That variable can be set via setq
or the customize interface.  To hide all tags that begin with "x-" or
"X-", set `notmuch-search-hide-tag-regex` to "^X-".

---
 emacs/notmuch.el |   16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index f9454d8..4bff538 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -775,6 +775,14 @@ non-authors is found, assume that all of the authors 
match."
  (overlay-put overlay 'isearch-open-invisible #'delete-overlay)))
   (insert padding

+(defcustom notmuch-search-hide-tag-regex ""
+  "Regex specifying tags to hide in search view.
+
+Leave blank to disable hiding of tags in search view.
+Note: elisp regexes are case-insensitive"
+  :type 'string
+  :group 'notmuch-search)
+
 (defun notmuch-search-insert-field (field format-string result)
   (cond
((string-equal field "date")
@@ -793,7 +801,13 @@ non-authors is found, assume that all of the authors 
match."
 (notmuch-search-insert-authors format-string (plist-get result :authors)))

((string-equal field "tags")
-(let ((tags-str (mapconcat 'identity (plist-get result :tags) " ")))
+(let ((tags-str (mapconcat 'identity
+  (delq nil
+(mapcar (lambda (x) (if (and (not (equal 
notmuch-search-hide-tag-regex ""))
+ (string-match 
notmuch-search-hide-tag-regex x))
+nil
+x)) (plist-get 
result :tags)))
+  " ")))
   (insert (propertize (format format-string tags-str)
  'face 'notmuch-tag-face))

-- 
1.7.10.4

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121029/6509438e/attachment.pgp>

[PATCH] Add NEWS information about Bcc header in JSON output

[Michal Nazarewicz]

From: Michal Nazarewicz 

This commit adds a note to the NEWS file about Bcc header now being
available in the JSON output (and thus in Emacs) which has been
implemented by commit ffb629cc5d2c2d1505eb5aefcf04fb4d0af6c0c8.
---
 NEWS |7 +++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index ebd5c1e..24d743f 100644
--- a/NEWS
+++ b/NEWS
@@ -80,6 +80,13 @@ user-specified formatting
 The thread-id for the `target-thread` argument for `notmuch-search` should
 now be supplied without the "thread:" prefix.

+Bcc header is now returned to Emacs
+
+  This makes it possible to add Bcc header to the
+  `notmuch-message-headers` variable which will make it visible with
+  all the other headers.  This is possible thanks to `notmuch
+  show --format=jsonK` now including the Bcc header.
+
 Notmuch 0.13.2 (2012-06-02)
 ===

-- 
1.7.7.3

[PATCH] don't show x-foo tags in search view

[David Bremner]

James Vasile  writes:

> My filters create tags like x-bogotrained-spam that are for internal
> bookkeeping.  I don't mind seeing them in the 'show' view, but I didn't
> want them cluttering my 'search' view.  This patch omits x-foo and X-foo
> tags from the 'search' view.

I understand this scratches your itch, but what about something more
customizable?

d

[PATCH] don't show x-foo tags in search view

[James Vasile]

David Bremner  writes:
> James Vasile  writes:
>
>> My filters create tags like x-bogotrained-spam that are for internal
>> bookkeeping.  I don't mind seeing them in the 'show' view, but I didn't
>> want them cluttering my 'search' view.  This patch omits x-foo and X-foo
>> tags from the 'search' view.
>
> I understand this scratches your itch, but what about something more
> customizable?
>
> d

I thought it was fairly general as is, but you're right it could be more
customizable.  I'll rewrite it with a regex the user can customize.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121029/4933c10f/attachment.pgp>

[PATCH] don't show x-foo tags in search view

[James Vasile]

My filters create tags like x-bogotrained-spam that are for internal
bookkeeping.  I don't mind seeing them in the 'show' view, but I didn't
want them cluttering my 'search' view.  This patch omits x-foo and X-foo
tags from the 'search' view.

---
 emacs/notmuch.el |7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index f9454d8..90fafbf 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -793,7 +793,12 @@ non-authors is found, assume that all of the authors 
match."
 (notmuch-search-insert-authors format-string (plist-get result :authors)))

((string-equal field "tags")
-(let ((tags-str (mapconcat 'identity (plist-get result :tags) " ")))
+(let ((tags-str (mapconcat 'identity
+  (delq nil
+(mapcar (lambda (x) (if (equal (upcase 
(truncate-string-to-width  x 2)) "X-")
+nil
+(identity x))) 
(plist-get result :tags)))
+  " ")))
   (insert (propertize (format format-string tags-str)
  'face 'notmuch-tag-face))

-- 
1.7.10.4
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121029/15060333/attachment.pgp>

[BUG] Saving attachments containing UTF-8 chars

[Michael Stapelberg]

Hello,

attached you can find a message which seems to trigger a bug somewhere
in my notmuch setup. The message has been sent to me and includes a
patchfile, which, when saved using the "w" key in notmuch-emacs, cannot
be applied.

When using munpack(1) on the raw mail file (I?ve attached rawmail.gz
just to be sure that the message doesn?t get mangled in some way), the
unpacked attachment can be attached just fine.

When diff'ing both files, I see that munpack correctly stores the file,
including the problematic UTF-8 apostrophe:
0200  67 3b 20 64 69 64 6e e2  80 99 74 20 72 65 61 64  |g; didn...t read|

However, the version of the file saved with notmuch contains:
0200  67 3b 20 64 69 64 6e 19  74 20 72 65 61 64 22 20  |g; didn.t read" |

If there is anything I can do to help debugging this (or if there is a
known fix), please let me know.

-- next part --
An embedded message was scrubbed...
From: Conley Moorhous <conleymoorh...@gmail.com>
Subject: Re: User Guide Chapter 6 typo
Date: Thu, 25 Oct 2012 09:59:50 -0500
Size: 9355
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121029/ee441e65/attachment.mht>
-- next part --
A non-text attachment was scrubbed...
Name: rawmail.gz
Type: application/octet-stream
Size: 3762 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121029/ee441e65/attachment.obj>

Notmuch search missing mail id

[James Vasile]

Tomi Ollila  writes:
> On Mon, Oct 29 2012, James Vasile  wrote:
>
>> I received the attached piece of spam telling me about an exciting
>> investment opportunity.  Notmuch pulled it in to the database, noting
>> the message id.  But then it seems to stop paying attention to the
>> message id.
>>
>> notmuch can find the mail when I search by the from field:
>>
>> $ notmuch search "from:caroline horn"
>> thread:bcb7  Today 09:29 [1/1] Caroline Horn; This Company is on 
>> the rise (hv inbox unread)
>>
>> But not by the message id:
>> $ notmuch search id:000801cdb5da$17673470$26448589 at microsof00t4ko3r
>> $
>
> Try 
>
> $ notmuch search id:'000801cdb5da$17673470$26448589 at microsof00t4ko3r'
>
> so that the $s are not expanded...
>
> note that id:"..." does not suffice as shell expands $ & ` inside double
> quotes.

Tomi, that's a good catch and one I should have noticed.  Thanks much
for the help!
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121029/35aa6d42/attachment.pgp>

Notmuch search missing mail id

[James Vasile]

I received the attached piece of spam telling me about an exciting
investment opportunity.  Notmuch pulled it in to the database, noting
the message id.  But then it seems to stop paying attention to the
message id.

notmuch can find the mail when I search by the from field:

$ notmuch search "from:caroline horn"
thread:bcb7  Today 09:29 [1/1] Caroline Horn; This Company is on 
the rise (hv inbox unread)

But not by the message id:
$ notmuch search id:000801cdb5da$17673470$26448589 at microsof00t4ko3r
$

Notmuch has correctly pulled the id into the database:
$ notmuch show "from:caroline horn" | grep "id:" | sed "s/.*\(id:[^ ]*\).*/\1/"
id:000801cdb5da$17673470$26448589 at microsof00t4ko3r

If notmuch knows the id but won't match it in a search, that looks like
a bug to me.

On a related matter, I have a script that pulls the message id header

[PATCH] contrib: notmuch-pick: add tests

[Mark Walters]

The test should be run using the wrapper test-wrapper.sh.  This links
the tests into the normal notmuch TEST_DIRECTORY and runs them from
there. After the test is complete then the links are removed.
---

Here are some tests for notmuch-pick. I would have liked to include
the test structure and run them from inside the pick contrib directory
but I couldn't get all the paths to work.

Hence, this slightly hacky solution of linking the relevant stuff (one
test script and one expected output directory) into the normal notmuch
test directory and running from there. 

Best wishes

Mark

 contrib/notmuch-pick/README|5 +
 contrib/notmuch-pick/test-wrapper.sh   |   16 +++
 contrib/notmuch-pick/test/emacs-pick   |  104 
 .../notmuch-pick-single-thread |6 +
 .../pick.expected-output/notmuch-pick-tag-inbox|   53 ++
 5 files changed, 184 insertions(+), 0 deletions(-)
 create mode 100755 contrib/notmuch-pick/test-wrapper.sh
 create mode 100755 contrib/notmuch-pick/test/emacs-pick
 create mode 100644 
contrib/notmuch-pick/test/pick.expected-output/notmuch-pick-single-thread
 create mode 100644 
contrib/notmuch-pick/test/pick.expected-output/notmuch-pick-tag-inbox

diff --git a/contrib/notmuch-pick/README b/contrib/notmuch-pick/README
index 8eed974..98251f6 100644
--- a/contrib/notmuch-pick/README
+++ b/contrib/notmuch-pick/README
@@ -15,6 +15,11 @@ Then after the "(require 'notmuch)" line in your .emacs file 
add
 the line "(require 'notmuch-pick nil t)". This will load notmuch-pick on
 your next emacs start.

+TEST
+
+Just execute test-wrapper.sh from this directory and it should all
+work. It does require that notmuch has already been built.
+
 USING PICK

 The main key entries to notmuch pick are
diff --git a/contrib/notmuch-pick/test-wrapper.sh 
b/contrib/notmuch-pick/test-wrapper.sh
new file mode 100755
index 000..3ecf5b9
--- /dev/null
+++ b/contrib/notmuch-pick/test-wrapper.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+fail() {
+echo ERROR $1
+exit 1
+}
+
+export PICK_DIR=$PWD
+ln -s $PWD/test/emacs-pick ../../test/ || fail "emacs pick test exists"
+ln -s $PWD/test/pick.expected-output ../../test/ || fail "pick.expected-output 
exists"
+
+cd ../../test || fail "Cannot find test directory"
+./emacs-pick
+
+rm emacs-pick
+rm pick.expected-output
diff --git a/contrib/notmuch-pick/test/emacs-pick 
b/contrib/notmuch-pick/test/emacs-pick
new file mode 100755
index 000..9478732
--- /dev/null
+++ b/contrib/notmuch-pick/test/emacs-pick
@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+
+test_description="emacs interface"
+. test-lib.sh
+
+EXPECTED=$TEST_DIRECTORY/pick.expected-output
+
+add_email_corpus
+test_begin_subtest "Do we have emacs"
+test_emacs '(insert "hello\n")
+(test-output)'
+cat 

[PATCH 0/3] Add notmuch-pick to contrib

[Mark Walters]

Hello

On Sun, 28 Oct 2012, David Bremner  wrote:
> Mark Walters  writes:
>
>> This is a new version of notmuch-pick (previous version at
>> id:"1343164911-31589-1-git-send-email-markwalters1009 at gmail.com")following
>> suggestions of Tomi and David in that thread. The main change is that
>> it is now entirely self contained in contrib: all the user needs to do
>> is copy the notmuch-pick.el somewhere in the emacs load path and add
>> (require 'notmuch-pick nil t) to their .emacs file.
>
> I have pushed this to contrib.

That's great.

> We do need tests before this goes mainline. What about (untested) having
> the tests in contrib/notmuch-pick for now, including as needed from
> ../../test.

I haven't been able to get this to work but I have a slight variant
which seems to that I will post shortly.

> We should mention it in NEWS; at this point I think it is reasonable to
> expect to do at least one release with pick in contrib.

Yes that seems sensible.

> As a vague feature request, what about forwarding unknown keys to
> the notmuch-show buffer?

This would be good: but I can't see how to make it work. If anyone has
any suggestions or pointers do say!

Best wishes

Mark

[PATCH 0/2] automatic tagging based on imap maildir

[Andrei POPESCU]

[Remembered to Reply-All, my fingers are trained to Reply-List only ;) ]

On Du, 28 oct 12, 18:12:40, Austin Clements wrote:
> 
> Ah.  And I assume you want to see those in your inbox even if they're
> not a reply to some other message in your inbox?  I do recall Gmail
> suppressing list reflections.

Yes, except the Inbox part :p. As already mentioned, I sort all lists by 
List-Id and I want to have my own posts nicely in their place in the 
corresponding threads (cross-posts included). Most of the times I even 
read my posts again, to spot significant mistakes.

> Though I think that, by default, notmuch *will* put your own list
> postings in your inbox.  Assuming the list sends the message back to
> you (not all do), and assuming your notmuch new.tags contains inbox,
> notmuch will simply see it as a new message and tag it inbox.  If you
> have Fcc setup in your frontend of choice to save sent messages in a
> folder indexed by notmuch, then it should likewise tag the message
> inbox as soon as it indexes it.

Mmm, can't comment on that until I have a "full" notmuch client. Right 
now I'm using mutt and notmuch-mutt only.

Kind regards,
Andrei
-- 
If you can't explain it simply, you don't understand it well enough.
(Albert Einstein)
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20121029/4d69afaf/attachment.pgp>

[PATCH v5 8/9] man: document the date:since..until range queries

[Jani Nikula]

Many thanks, I'll incorporate most of your suggestions as-is.

BR,
Jani.


On Thu, 25 Oct 2012, Austin Clements  wrote:
> Quoth Jani Nikula on Oct 22 at 12:22 am:
>> ---
>>  man/man7/notmuch-search-terms.7 |  147 
>> +++
>>  1 file changed, 135 insertions(+), 12 deletions(-)
>> 
>> diff --git a/man/man7/notmuch-search-terms.7 
>> b/man/man7/notmuch-search-terms.7
>> index 17a109e..fbd3ee7 100644
>> --- a/man/man7/notmuch-search-terms.7
>> +++ b/man/man7/notmuch-search-terms.7
>> @@ -54,6 +54,8 @@ terms to match against specific portions of an email, 
>> (where
>>  
>>  folder:
>>  
>> +date:..
>> +
>>  The
>>  .B from:
>>  prefix is used to match the name or address of the sender of an email
>> @@ -104,6 +106,26 @@ contained within particular directories within the mail 
>> store. Only
>>  the directory components below the top-level mail database path are
>>  available to be searched.
>>  
>> +The
>> +.B date:
>> +prefix can be used to restrict the results to only messages within a
>> +particular time range (based on the Date: header) with a range syntax
>> +of:
>> +
>> +date:..
>> +
>> +See \fBDATE AND TIME SEARCH\fR below for details on the range
>> +expression, and supported syntax for  and  date and time
>> +expressions.
>> +
>> +The time range can also be specified using timestamps with a syntax
>> +of:
>> +
>> +..
>> +
>> +Each timestamp is a number representing the number of seconds since
>> +1970\-01\-01 00:00:00 UTC.
>> +
>>  In addition to individual terms, multiple terms can be
>>  combined with Boolean operators (
>>  .BR and ", " or ", " not
>> @@ -117,20 +139,121 @@ operators, but will have to be protected from 
>> interpretation by the
>>  shell, (such as by putting quotation marks around any parenthesized
>>  expression).
>>  
>> -Finally, results can be restricted to only messages within a
>> -particular time range, (based on the Date: header) with a syntax of:
>> +.SH DATE AND TIME SEARCH
>>  
>> -..
>> +This is a non-exhaustive description of the date and time search with
>> +some pseudo notation. Most of the constructs can be mixed freely, and
>> +in any order, but the same absolute date or time can't be expressed
>> +twice.
>
> I'm not sure what the end of this sentence means, though I assume it's
> related to the restrictions on repeated absolute components.  It would
> also be nice to give a broader view of the syntax here.  Maybe,
>
> notmuch understands a variety of standard and natural ways of
> expressing dates and times, both in absolute terms ("2012-10-24") and
> in relative terms ("yesterday").  Any number of relative terms can be
> combined ("1 hour 25 minutes") and an absolute date/time can be
> combined with relative terms to further adjust it.  A non-exhaustive
> description of the syntax supported for absolute and relative terms is
> given below.
>
>>  
>> -Each timestamp is a number representing the number of seconds since
>> -1970\-01\-01 00:00:00 UTC. This is not the most convenient means of
>> -expressing date ranges, but until notmuch is fixed to accept a more
>> -convenient form, one can use the date program to construct
>> -timestamps. For example, with the bash shell the following syntax would
>> -specify a date range to return messages from 2009\-10\-01 until the
>> -current time:
>> -
>> -$(date +%s \-d 2009\-10\-01)..$(date +%s)
>> +.RS 4
>> +.TP 4
>> +.B The range expression
>> +
>> +date:..
>> +
>> +The above expression restricts the results to only messages from
>> + to , based on the Date: header.
>> +
>> +If  or  describes time at an accuracy of days or less,
>> +the date/time is rounded, towards past for  and towards future
>> +for , to be inclusive. For example, date:january..february
>
> The accuracy doesn't seem to have have anything to do with days; if I
> say "date:1hour..1hour" I get a span of an hour.  Describing it as
> rounding also seems like it could be confusing to someone who hasn't
> thought a lot about this (though, as someone who has though a lot
> about this, I could be wrong).  What about something like,
>
>  and  can describe imprecise times, such as "yesterday".
> In this case,  is taken as the earliest time it could describe
> (the beginning of yesterday) and  is taken as the latest time
> it could describe (the end of yesterday).  Similarly,
> date:january..february matches from the beginning of January to the
> end of February.
>
>> +matches from the beginning of January until the end of
>> +February. Similarly, date:yesterday..yesterday matches from the
>> +beginning of yesterday until the end of yesterday.
>> +
>> +Open-ended ranges are supported (since Xapian 1.2.1), i.e. it's
>> +possible to specify date:.. or date:.. to not limit the
>> +start or end time, respectively. Unfortunately, pre-1.2.1 Xapian does
>
> No need for the "Unfortunately".
>
>> +not report an error on open ended ranges, but it does not work as
>> +expected either.
>> +
>> +Xapian does not support 

[PATCH v5 6/9] lib: add date range query support

[Jani Nikula]

On Tue, 23 Oct 2012, Austin Clements  wrote:
> Quoth Jani Nikula on Oct 22 at 12:22 am:
>> Add a custom value range processor to enable date and time searches of
>> the form date:since..until, where "since" and "until" are expressions
>> understood by the previously added date/time parser, to restrict the
>> results to messages within a particular time range (based on the Date:
>> header).
>> 
>> If "since" or "until" describes date/time at an accuracy of days or
>> less, the values are rounded according to the accuracy, towards past
>> for "since" and towards future for "until". For example,
>> date:november..yesterday would match from the beginning of November
>> until the end of yesterday. Expressions such as date:today..today
>> means since the beginning of today until the end of today.
>> 
>> Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
>> specify date:..until or date:since.. to not limit the start or end
>> date, respectively.
>> 
>> CAVEATS:
>> 
>> Xapian does not support spaces in range expressions. You can replace
>> the spaces with '_', or (in most cases) '-', or (in some cases) leave
>> the spaces out altogether.
>> 
>> Entering date:expr without ".." (for example date:yesterday) will not
>> work as you might expect. You can achieve the expected result by
>> duplicating the expr both sides of ".." (for example
>> date:yesterday..yesterday).
>> 
>> Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't
>> produce an error either.
>> 
>> Signed-off-by: Jani Nikula 
>> ---
>>  lib/Makefile.local |1 +
>>  lib/database-private.h |1 +
>>  lib/database.cc|5 +
>>  lib/parse-time-vrp.cc  |   40 
>>  lib/parse-time-vrp.h   |   19 +++
>>  5 files changed, 66 insertions(+)
>>  create mode 100644 lib/parse-time-vrp.cc
>>  create mode 100644 lib/parse-time-vrp.h
>> 
>> diff --git a/lib/Makefile.local b/lib/Makefile.local
>> index d1635cf..6c0f42f 100644
>> --- a/lib/Makefile.local
>> +++ b/lib/Makefile.local
>> @@ -58,6 +58,7 @@ libnotmuch_c_srcs =\
>>  
>>  libnotmuch_cxx_srcs =   \
>>  $(dir)/database.cc  \
>> +$(dir)/parse-time-vrp.cc\
>>  $(dir)/directory.cc \
>>  $(dir)/index.cc \
>>  $(dir)/message.cc   \
>> diff --git a/lib/database-private.h b/lib/database-private.h
>> index 88532d5..d3e65fd 100644
>> --- a/lib/database-private.h
>> +++ b/lib/database-private.h
>> @@ -52,6 +52,7 @@ struct _notmuch_database {
>>  Xapian::QueryParser *query_parser;
>>  Xapian::TermGenerator *term_gen;
>>  Xapian::ValueRangeProcessor *value_range_processor;
>> +Xapian::ValueRangeProcessor *date_range_processor;
>>  };
>>  
>>  /* Return the list of terms from the given iterator matching a prefix.
>> diff --git a/lib/database.cc b/lib/database.cc
>> index 761dc1a..4df3217 100644
>> --- a/lib/database.cc
>> +++ b/lib/database.cc
>> @@ -19,6 +19,7 @@
>>   */
>>  
>>  #include "database-private.h"
>> +#include "parse-time-vrp.h"
>>  
>>  #include 
>>  
>> @@ -710,12 +711,14 @@ notmuch_database_open (const char *path,
>>  notmuch->term_gen = new Xapian::TermGenerator;
>>  notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
>>  notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor 
>> (NOTMUCH_VALUE_TIMESTAMP);
>> +notmuch->date_range_processor = new ParseTimeValueRangeProcessor 
>> (NOTMUCH_VALUE_TIMESTAMP);
>>  
>>  notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
>>  notmuch->query_parser->set_database (*notmuch->xapian_db);
>>  notmuch->query_parser->set_stemmer (Xapian::Stem ("english"));
>>  notmuch->query_parser->set_stemming_strategy 
>> (Xapian::QueryParser::STEM_SOME);
>>  notmuch->query_parser->add_valuerangeprocessor 
>> (notmuch->value_range_processor);
>> +notmuch->query_parser->add_valuerangeprocessor 
>> (notmuch->date_range_processor);
>>  
>>  for (i = 0; i < ARRAY_SIZE (BOOLEAN_PREFIX_EXTERNAL); i++) {
>>  prefix_t *prefix = _PREFIX_EXTERNAL[i];
>> @@ -778,6 +781,8 @@ notmuch_database_close (notmuch_database_t *notmuch)
>>  notmuch->xapian_db = NULL;
>>  delete notmuch->value_range_processor;
>>  notmuch->value_range_processor = NULL;
>> +delete notmuch->date_range_processor;
>> +notmuch->date_range_processor = NULL;
>>  }
>>  
>>  void
>> diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
>> new file mode 100644
>> index 000..7e4eca4
>> --- /dev/null
>> +++ b/lib/parse-time-vrp.cc
>> @@ -0,0 +1,40 @@
>
> Should this file have the usual preamble?

Probably, yes.

>> +
>> +#include "database-private.h"
>> +#include "parse-time-vrp.h"
>> +#include "parse-time-string.h"
>> +
>> +#define PREFIX "date:"
>> +
>> +/* See *ValueRangeProcessor in xapian-core/api/valuerangeproc.cc */
>> +Xapian::valueno
>> +ParseTimeValueRangeProcessor::operator() (std::string , std::string 
>> )
>> +{
>> +  

[PATCH v5 4/9] test: add smoke tests for the date/time parser module

[Jani Nikula]

On Tue, 23 Oct 2012, Austin Clements  wrote:
> Quoth Jani Nikula on Oct 22 at 12:22 am:
>> Test the date/time parser module directly, independent of notmuch,
>> using the parse-time test tool.
>> 
>> Credits to Michal Sojka  for writing most of the
>> tests.
>> ---
>>  test/notmuch-test  |1 +
>>  test/parse-time-string |   71 
>> 
>>  2 files changed, 72 insertions(+)
>>  create mode 100755 test/parse-time-string
>> 
>> diff --git a/test/notmuch-test b/test/notmuch-test
>> index cc732c3..7eadfdf 100755
>> --- a/test/notmuch-test
>> +++ b/test/notmuch-test
>> @@ -60,6 +60,7 @@ TESTS="
>>emacs-hello
>>emacs-show
>>missing-headers
>> +  parse-time-string
>>  "
>>  TESTS=${NOTMUCH_TESTS:=$TESTS}
>>  
>> diff --git a/test/parse-time-string b/test/parse-time-string
>> new file mode 100755
>> index 000..862e701
>> --- /dev/null
>> +++ b/test/parse-time-string
>> @@ -0,0 +1,71 @@
>> +#!/usr/bin/env bash
>> +test_description="date/time parser module"
>> +. ./test-lib.sh
>> +
>> +# Sanity/smoke tests for the date/time parser independent of notmuch
>> +
>> +_date ()
>> +{
>> +date -d "$*" +%s
>> +}
>> +
>> +_parse_time ()
>> +{
>> +${TEST_DIRECTORY}/parse-time --format=%s "$*"
>> +}
>> +
>> +test_begin_subtest "date(1) default format without TZ code"
>> +test_expect_equal "$(_parse_time Fri Aug 3 23:06:06 2012)" "$(_date Fri Aug 
>> 3 23:06:06 2012)"
>> +
>> +test_begin_subtest "date(1) --rfc-2822 format"
>> +test_expect_equal "$(_parse_time Fri, 03 Aug 2012 23:07:46 +0100)" "$(_date 
>> Fri, 03 Aug 2012 23:07:46 +0100)"
>> +
>> +test_begin_subtest "date(1) --rfc=3339=seconds format"
>> +test_expect_equal "$(_parse_time 2012-08-03 23:09:37+03:00)" "$(_date 
>> 2012-08-03 23:09:37+03:00)"
>> +
>> +test_begin_subtest "Date parser tests"
>> +REFERENCE=$(_date Tue Jan 11 11:11:00 + 2011)
>> +cat < INPUT
>> +now  ==> Tue Jan 11 11:11:00 + 2011
>> +2010-1-1 ==> ERROR: 5
>
> It would be nice if these errors were strings.  I have no idea if "5"
> is the right error for this.

Good idea. Will fix.

>> +Jan 2==> Sun Jan 02 11:11:00 + 2011
>> +Mon  ==> Mon Jan 10 11:11:00 + 2011
>> +last Friday  ==> ERROR: 4
>> +2 hours ago  ==> ERROR: 1

I'll silently eat away "ago" too.

>> +last month   ==> Sat Dec 11 11:11:00 + 2010
>> +month ago==> ERROR: 1
>> +8am  ==> Tue Jan 11 08:00:00 + 2011
>> +9:15 ==> Tue Jan 11 09:15:00 + 2011
>> +12:34==> Tue Jan 11 12:34:00 + 2011
>> +monday   ==> Mon Jan 10 11:11:00 + 2011
>> +yesterday==> Mon Jan 10 11:11:00 + 2011
>> +tomorrow ==> ERROR: 1
>> + ==> Tue Jan 11 11:11:00 + 2011 # empty string is reference 
>> time
>> +
>> +Aug 3 23:06:06 2012 ==> Fri Aug 03 23:06:06 + 2012 # 
>> date(1) default format without TZ code
>> +Fri, 03 Aug 2012 23:07:46 +0100 ==> Fri Aug 03 22:07:46 + 2012 # 
>> rfc-2822
>> +2012-08-03 23:09:37+03:00   ==> Fri Aug 03 20:09:37 + 2012 # 
>> rfc-3339 seconds
>> +
>> +10s   ==> Tue Jan 11 11:10:50 + 2011
>> +19701223s ==> Fri May 28 10:37:17 + 2010
>> +19701223  ==> Wed Dec 23 11:11:00 + 1970
>> +
>> +19701223 +0100 ==> Wed Dec 23 11:11:00 + 1970 # Timezone is ignored 
>> without an error
>> +
>> +today ==^> Tue Jan 11 23:59:59 + 2011
>> +today ==_> Tue Jan 11 00:00:00 + 2011
>> +
>> +thisweek ==^> Sat Jan 15 23:59:59 + 2011
>> +thisweek ==_> Sun Jan 09 00:00:00 + 2011
>> +
>> +two months ago==> ERROR: 1 # "ago" is not supported
>> +two months ==> Thu Nov 11 11:11:00 + 2010
>> +
>> + at 1348569850 ==> Tue Sep 25 10:44:10 + 2012
>> + at 10 ==> Thu Jan 01 00:00:10 + 1970
>
> Very nice.  The only thing that jumps out at me is that there are no
> ==^^> tests, though it would be interesting to run a code coverage
> tool to see how thorough these tests are.

Again, most of the credit here goes to Michal Sojka.

Will add some ==^^> tests too.


BR,
Jani.


>
>> +EOF
>> +
>> +${TEST_DIRECTORY}/parse-time --ref=${REFERENCE} < INPUT > OUTPUT
>> +test_expect_equal_file INPUT OUTPUT
>> +
>> +test_done

[PATCH v5 2/9] parse-time-string: add a date/time parser to notmuch

[Jani Nikula]

On Mon, 22 Oct 2012, Austin Clements  wrote:
> Overall this looks pretty good to me, and I must say, this parser is
> amazingly flexible and copes well with a remarkably hostile grammar.
>
> A lot of little comments below (sorry if any of this ground has
> already been covered in the previous four versions).

Nope, apart from "postponed numbers are confusing".

> I do have one broad comment.  While I'm all for ad hoc parsers for ad
> hoc grammars like dates, there is one piece of the literature I think
> this parser suffers for by ignoring: tokenizing.  I think it would
> simplify a lot of this code if it did a tokenizing pass before the
> parsing pass.  It doesn't have to be a serious tokenizer with
> streaming and keywords and token types and junk; just something that
> first splits the input into substrings, possibly just non-overlapping
> matches of [[:digit:]]+|[[:alpha:]]+|[-+:/.].  This would simplify the
> handling of postponed numbers because, with trivial lookahead in the
> token stream, you wouldn't have to postpone them.  Likewise, it would
> eliminate last_field.  It would simplify keyword matching because you
> wouldn't have to worry about matching substrings (I spent a long time
> staring at that code before I figured out what it would and wouldn't
> accept).  Most important, I think it would make the parser more
> predictable for users; for example, the parser currently accepts
> things like "saturtoday" because it's aggressively single-pass.

I'll fix this to require a non-keyword character between keywords, but,
since you're not adamant about it, I'll pass adding the tokenizer, at
least for now.

> Quoth Jani Nikula on Oct 22 at 12:22 am:
>> Add a date/time parser to notmuch, to be used for adding date range
>> query support for notmuch lib later on. Add the parser to a directory
>> of its own to make it independent of the rest of the notmuch code
>> base.
>> 
>> Signed-off-by: Jani Nikula 
>> ---
>>  Makefile  |2 +-
>>  parse-time-string/Makefile|5 +
>>  parse-time-string/Makefile.local  |   12 +
>>  parse-time-string/README  |9 +
>>  parse-time-string/parse-time-string.c | 1477 
>> +
>>  parse-time-string/parse-time-string.h |  102 +++
>>  6 files changed, 1606 insertions(+), 1 deletion(-)
>>  create mode 100644 parse-time-string/Makefile
>>  create mode 100644 parse-time-string/Makefile.local
>>  create mode 100644 parse-time-string/README
>>  create mode 100644 parse-time-string/parse-time-string.c
>>  create mode 100644 parse-time-string/parse-time-string.h
>> 
>> diff --git a/Makefile b/Makefile
>> index e5e2e3a..bb9c316 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -3,7 +3,7 @@
>>  all:
>>  
>>  # List all subdirectories here. Each contains its own Makefile.local
>> -subdirs = compat completion emacs lib man util test
>> +subdirs = compat completion emacs lib man parse-time-string util test
>>  
>>  # We make all targets depend on the Makefiles themselves.
>>  global_deps = Makefile Makefile.config Makefile.local \
>> diff --git a/parse-time-string/Makefile b/parse-time-string/Makefile
>> new file mode 100644
>> index 000..fa25832
>> --- /dev/null
>> +++ b/parse-time-string/Makefile
>> @@ -0,0 +1,5 @@
>> +all:
>> +$(MAKE) -C .. all
>> +
>> +.DEFAULT:
>> +$(MAKE) -C .. $@
>> diff --git a/parse-time-string/Makefile.local 
>> b/parse-time-string/Makefile.local
>> new file mode 100644
>> index 000..53534f3
>> --- /dev/null
>> +++ b/parse-time-string/Makefile.local
>> @@ -0,0 +1,12 @@
>> +dir := parse-time-string
>> +extra_cflags += -I$(srcdir)/$(dir)
>> +
>> +libparse-time-string_c_srcs := $(dir)/parse-time-string.c
>> +
>> +libparse-time-string_modules := $(libparse-time-string_c_srcs:.c=.o)
>> +
>> +$(dir)/libparse-time-string.a: $(libparse-time-string_modules)
>> +$(call quiet,AR) rcs $@ $^
>> +
>> +SRCS := $(SRCS) $(libparse-time-string_c_srcs)
>> +CLEAN := $(CLEAN) $(libparse-time-string_modules) 
>> $(dir)/libparse-time-string.a
>> diff --git a/parse-time-string/README b/parse-time-string/README
>> new file mode 100644
>> index 000..300ff1f
>> --- /dev/null
>> +++ b/parse-time-string/README
>> @@ -0,0 +1,9 @@
>> +PARSE TIME STRING
>> +=
>> +
>> +parse_time_string() is a date/time parser originally written for
>> +notmuch by Jani Nikula . However, there is nothing
>> +notmuch specific in it, and it should be kept reusable for other
>> +projects, and ready to be packaged on its own as needed. Please do not
>> +add dependencies on or references to anything notmuch specific. The
>> +parser should only depend on the C library.
>> diff --git a/parse-time-string/parse-time-string.c 
>> b/parse-time-string/parse-time-string.c
>> new file mode 100644
>> index 000..942041a
>> --- /dev/null
>> +++ b/parse-time-string/parse-time-string.c
>> @@ -0,0 +1,1477 @@
>> +/*
>> + * parse time string - user friendly date and time parser
>> + * Copyright ? 

[PATCH 0/2] automatic tagging based on imap maildir

[Andrei POPESCU]

On Du, 28 oct 12, 11:31:38, Austin Clements wrote:
> 
> Why does that cause you to not receive mailing lists?  (Or did I
> misinterpret your statement?)

Yes. The main deterrent in Gmail silently dropping my own postings 
returning via the list, because it considers it identical to the sent 
message.

> If I understand what you're saying, this is exactly what notmuch does.
> When you remove the "unread" tag, notmuch will (should?) add the
> "seen" maildir flag to all copies of that message in the file system
> (barring some corner-cases that flag sync doesn't handle well; maybe
> you're encountering one of those?).  And since notmuch never deletes
> files, they will remain in the folders they were filed in.

Great. Then I only need a client that suits me. I've wanted to have a 
look at mutt-kz, but keep postponing it.

Kind regards,
Andrei
-- 
If you can't explain it simply, you don't understand it well enough.
(Albert Einstein)
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: 

Re: [PATCH 0/3] Add notmuch-pick to contrib

[Mark Walters]

Hello

On Sun, 28 Oct 2012, David Bremner da...@tethera.net wrote:
 Mark Walters markwalters1...@gmail.com writes:

 This is a new version of notmuch-pick (previous version at
 id:1343164911-31589-1-git-send-email-markwalters1...@gmail.com)following
 suggestions of Tomi and David in that thread. The main change is that
 it is now entirely self contained in contrib: all the user needs to do
 is copy the notmuch-pick.el somewhere in the emacs load path and add
 (require 'notmuch-pick nil t) to their .emacs file.

 I have pushed this to contrib.

That's great.

 We do need tests before this goes mainline. What about (untested) having
 the tests in contrib/notmuch-pick for now, including as needed from
 ../../test.

I haven't been able to get this to work but I have a slight variant
which seems to that I will post shortly.

 We should mention it in NEWS; at this point I think it is reasonable to
 expect to do at least one release with pick in contrib.

Yes that seems sensible.

 As a vague feature request, what about forwarding unknown keys to
 the notmuch-show buffer?

This would be good: but I can't see how to make it work. If anyone has
any suggestions or pointers do say!

Best wishes

Mark

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] contrib: notmuch-pick: add tests

[Mark Walters]

The test should be run using the wrapper test-wrapper.sh.  This links
the tests into the normal notmuch TEST_DIRECTORY and runs them from
there. After the test is complete then the links are removed.
---

Here are some tests for notmuch-pick. I would have liked to include
the test structure and run them from inside the pick contrib directory
but I couldn't get all the paths to work.

Hence, this slightly hacky solution of linking the relevant stuff (one
test script and one expected output directory) into the normal notmuch
test directory and running from there. 

Best wishes

Mark

 contrib/notmuch-pick/README|5 +
 contrib/notmuch-pick/test-wrapper.sh   |   16 +++
 contrib/notmuch-pick/test/emacs-pick   |  104 
 .../notmuch-pick-single-thread |6 +
 .../pick.expected-output/notmuch-pick-tag-inbox|   53 ++
 5 files changed, 184 insertions(+), 0 deletions(-)
 create mode 100755 contrib/notmuch-pick/test-wrapper.sh
 create mode 100755 contrib/notmuch-pick/test/emacs-pick
 create mode 100644 
contrib/notmuch-pick/test/pick.expected-output/notmuch-pick-single-thread
 create mode 100644 
contrib/notmuch-pick/test/pick.expected-output/notmuch-pick-tag-inbox

diff --git a/contrib/notmuch-pick/README b/contrib/notmuch-pick/README
index 8eed974..98251f6 100644
--- a/contrib/notmuch-pick/README
+++ b/contrib/notmuch-pick/README
@@ -15,6 +15,11 @@ Then after the (require 'notmuch) line in your .emacs file 
add
 the line (require 'notmuch-pick nil t). This will load notmuch-pick on
 your next emacs start.
 
+TEST
+
+Just execute test-wrapper.sh from this directory and it should all
+work. It does require that notmuch has already been built.
+
 USING PICK
 
 The main key entries to notmuch pick are
diff --git a/contrib/notmuch-pick/test-wrapper.sh 
b/contrib/notmuch-pick/test-wrapper.sh
new file mode 100755
index 000..3ecf5b9
--- /dev/null
+++ b/contrib/notmuch-pick/test-wrapper.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+fail() {
+echo ERROR $1
+exit 1
+}
+
+export PICK_DIR=$PWD
+ln -s $PWD/test/emacs-pick ../../test/ || fail emacs pick test exists
+ln -s $PWD/test/pick.expected-output ../../test/ || fail pick.expected-output 
exists
+
+cd ../../test || fail Cannot find test directory
+./emacs-pick
+
+rm emacs-pick
+rm pick.expected-output
diff --git a/contrib/notmuch-pick/test/emacs-pick 
b/contrib/notmuch-pick/test/emacs-pick
new file mode 100755
index 000..9478732
--- /dev/null
+++ b/contrib/notmuch-pick/test/emacs-pick
@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+
+test_description=emacs interface
+. test-lib.sh
+
+EXPECTED=$TEST_DIRECTORY/pick.expected-output
+
+add_email_corpus
+test_begin_subtest Do we have emacs
+test_emacs '(insert hello\n)
+(test-output)'
+cat EOF EXPECTED
+hello
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest Basic notmuch-pick view in emacs
+test_emacs '(add-to-list (quote load-path) '$PICK_DIR')
+(require (quote notmuch-pick))
+(notmuch-pick tag:inbox)
+(notmuch-test-wait)
+(test-output)'
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-pick-tag-inbox
+
+test_begin_subtest Basic notmuch-pick view in emacs (sync)
+test_emacs '(setq notmuch-pick-asynchronous-parser nil)
+(notmuch-pick tag:inbox)
+(notmuch-test-wait)
+(test-output)'
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-pick-tag-inbox
+
+test_begin_subtest Navigation of notmuch-hello to search results
+test_emacs '(notmuch-hello)
+(goto-char (point-min))
+(re-search-forward inbox)
+(widget-button-press (1- (point)))
+(notmuch-test-wait)
+   (notmuch-pick-from-search-current-query)
+(notmuch-test-wait)
+(test-output)'
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-pick-tag-inbox
+
+test_begin_subtest Navigation of notmuch-hello to search results (sync)
+test_emacs '(setq notmuch-pick-asynchronous-parser nil)
+(notmuch-hello)
+(goto-char (point-min))
+(re-search-forward inbox)
+(widget-button-press (1- (point)))
+(notmuch-test-wait)
+   (notmuch-pick-from-search-current-query)
+(notmuch-test-wait)
+(test-output)'
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-pick-tag-inbox
+
+test_begin_subtest Pick of a single thread (from search)
+test_emacs '(notmuch-hello)
+(goto-char (point-min))
+(re-search-forward inbox)
+(widget-button-press (1- (point)))
+(notmuch-test-wait)
+(notmuch-pick-from-search-thread)
+(notmuch-test-wait)
+(test-output)'
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-pick-single-thread
+
+test_begin_subtest Pick of a single thread (sync from search)
+test_emacs '(delete-other-windows)
+(setq 

Re: a DoS vulnerability associated with conflated Message-IDs?

[Peter Wang]

On Thu, 08 Mar 2012 11:37:09 -0500, Daniel Kahn Gillmor 
d...@fifthhorseman.net wrote:
 notmuch currently treats all messages with the same Message-ID as
 the same message.  I think this could be a vulnerability :(
 
 If two messages have the same Message-ID, is there a guarantee of which
 of these messages will be produced during a notmuch show?
 
 Either way, it seems to create a potential DoS attack on notmuch users.

Yesterday I was expecting a confirmation message which, seemingly, never
came.  It turns out my maildir already contained a message from the
same system.  From three years ago.  With the same Message-ID.

Malice has nothing on incompetence.

Could we distinguish messages with identical Message-IDs based on
some header fields, e.g. Date, From?

Peter
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[BUG] Saving attachments containing UTF-8 chars

[Michael Stapelberg]

Hello,

attached you can find a message which seems to trigger a bug somewhere
in my notmuch setup. The message has been sent to me and includes a
patchfile, which, when saved using the w key in notmuch-emacs, cannot
be applied.

When using munpack(1) on the raw mail file (I’ve attached rawmail.gz
just to be sure that the message doesn’t get mangled in some way), the
unpacked attachment can be attached just fine.

When diff'ing both files, I see that munpack correctly stores the file,
including the problematic UTF-8 apostrophe:
0200  67 3b 20 64 69 64 6e e2  80 99 74 20 72 65 61 64  |g; didn...t read|

However, the version of the file saved with notmuch contains:
0200  67 3b 20 64 69 64 6e 19  74 20 72 65 61 64 22 20  |g; didn.t read |

If there is anything I can do to help debugging this (or if there is a
known fix), please let me know.

---BeginMessage---
On 2012-10-25 04:44, Michael Stapelberg wrote:
 Hi Conley,

 Conley Moorhous conleymoorh...@gmail.com writes:
 Sorry, I accidentally used the master branch. I've attached the proper
 patch, matched against the next branch!
 Im sorry to bother you again about this, but I still cannot apply the
 patch you sent me on the next branch:

 midna /tmp $ git clone -b next git://code.i3wm.org/i3
 Cloning into 'i3'...
 remote: Counting objects: 20314, done.
 remote: Compressing objects: 100% (4651/4651), done.
 remote: Total 20314 (delta 15312), reused 20314 (delta 15312)
 Receiving objects: 100% (20314/20314), 4.98 MiB | 418 KiB/s, done.
 Resolving deltas: 100% (15312/15312), done.
 midna /tmp $ cd i3
 midna /tmp/i3 next $ git am /tmp/fix_alt.patch
 Applying: docs/userguide: s/alt/Alt/
 error: patch failed: docs/userguide:12
 error: docs/userguide: patch does not apply
 Patch failed at 0001 docs/userguide: s/alt/Alt/
 When you have resolved this problem run git am --resolved.
 If you would prefer to skip this patch, instead run git am --skip.
 To restore the original branch and stop patching run git am --abort.

 I then tried to make the changes you did on my own and generate a patch
 file using git format-patch HEAD^ and it looks very much like yours,
 except that it doesnt have broken UTF-8 in the first context line.

 Im not too sure where the problem lies here (that is, whether its your
 system, git version, locale setup, mail client), but you might want to
 investigate and re-send the patch. If you cant figure it out, I can
 also do the changes for you and commit it, though it seems desirable to
 me to fix such issues once and for all.

Wow, turns out I'm dumb :( It was Thunderbird, which, for some reason,
does not default to UTF-8. It /might/ work now, but knowing me, I'll
have done something else wrong (I tried applying it locally and it
worked)! I'm putting you through all this work for three measly words, ha!

I'll make up for it eventually, I hope. I'll submit a big ol' helpful patch.
From eda92a24e2460a36e2e59372120c92a8fac213b9 Mon Sep 17 00:00:00 2001
From: Conley Moorhous conleymoorh...@gmail.com
Date: Thu, 18 Oct 2012 22:12:11 -0500
Subject: [PATCH] docs/userguide: s/alt/Alt/

---
 docs/userguide | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/docs/userguide b/docs/userguide
index e76e4e0..2cdca05 100644
--- a/docs/userguide
+++ b/docs/userguide
@@ -12,9 +12,9 @@ contact us on IRC (preferred) or post your question(s) on the mailing list.
 For the too long; didnt read people, here is an overview of the default
 keybindings (click to see the full size image):
 
-*Keys to use with $mod (alt):*
+*Keys to use with $mod (Alt):*
 
-image:keyboard-layer1.png[Keys to use with $mod (alt),width=600,link=keyboard-layer1.png]
+image:keyboard-layer1.png[Keys to use with $mod (Alt),width=600,link=keyboard-layer1.png]
 
 *Keys to use with Shift+$mod:*
 
@@ -26,7 +26,7 @@ are your homerow.
 == Using i3
 
 Throughout this guide, the keyword +$mod+ will be used to refer to the
-configured modifier. This is the alt key (Mod1) by default, with windows (Mod4)
+configured modifier. This is the Alt key (Mod1) by default, with windows (Mod4)
 being a popular alternative.
 
 === Opening terminals and moving around
-- 
1.8.0

---End Message---


rawmail.gz
Description: Binary data
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] Add NEWS information about Bcc header in JSON output

[Michal Nazarewicz]

From: Michal Nazarewicz min...@mina86.com

This commit adds a note to the NEWS file about Bcc header now being
available in the JSON output (and thus in Emacs) which has been
implemented by commit ffb629cc5d2c2d1505eb5aefcf04fb4d0af6c0c8.
---
 NEWS |7 +++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index ebd5c1e..24d743f 100644
--- a/NEWS
+++ b/NEWS
@@ -80,6 +80,13 @@ user-specified formatting
 The thread-id for the `target-thread` argument for `notmuch-search` should
 now be supplied without the thread: prefix.
 
+Bcc header is now returned to Emacs
+
+  This makes it possible to add Bcc header to the
+  `notmuch-message-headers` variable which will make it visible with
+  all the other headers.  This is possible thanks to `notmuch
+  show --format=jsonK` now including the Bcc header.
+
 Notmuch 0.13.2 (2012-06-02)
 ===
 
-- 
1.7.7.3

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [BUG] Saving attachments containing UTF-8 chars

[Tomi Ollila]

On Mon, Oct 29 2012, Michael Stapelberg wrote:

 Hello,

 attached you can find a message which seems to trigger a bug somewhere
 in my notmuch setup. The message has been sent to me and includes a
 patchfile, which, when saved using the w key in notmuch-emacs, cannot
 be applied.

 When using munpack(1) on the raw mail file (I’ve attached rawmail.gz
 just to be sure that the message doesn’t get mangled in some way), the
 unpacked attachment can be attached just fine.

 When diff'ing both files, I see that munpack correctly stores the file,
 including the problematic UTF-8 apostrophe:
 0200  67 3b 20 64 69 64 6e e2  80 99 74 20 72 65 61 64  |g; didn...t read|

 However, the version of the file saved with notmuch contains:
 0200  67 3b 20 64 69 64 6e 19  74 20 72 65 61 64 22 20  |g; didn.t read |

 If there is anything I can do to help debugging this (or if there is a
 known fix), please let me know.

I can verify this bug: I copied 'rawmail' to my mail store and attempted
to 'w' the attacment and got the same result (after notmuch new).

The saving code first does
notmuch show --format=raw id:508953e6.70...@gmail.com
which decodes OK on command line, and to the buffer when 
kill-buffer is outcommented in (with-current-notmuch-show-message ...) 
macro.

I attempted a set of trial--error tricks to get the attachment
saved correctly, and at least this seems to do the trick:

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index f273eb4..a6a85c0 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -203,9 +203,11 @@ For example, if you wanted to remove an \unread\ tag and 
add a
  (let ((id (notmuch-show-get-message-id)))
(let ((buf (generate-new-buffer (concat *notmuch-msg- id *
  (with-current-buffer buf
-   (call-process notmuch-command nil t nil show --format=raw id)
-   ,@body)
-(kill-buffer buf)
+  (let ((coding-system-for-read 'no-conversion)
+(coding-system-for-write 'no-conversion))
+(call-process notmuch-command nil t nil show --format=raw id)
+,@body))
+%%  (kill-buffer buf)
 
 (defun notmuch-show-turn-on-visual-line-mode ()
   Enable Visual Line mode.

---

(kill-buffer is outcommented above for testing purposes)

To test this this needs to me evaluated and then the functions
using this macro (notmuch-show-save-attachments  in this case)

Smart suggestions for proper fix ?

Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Notmuch search missing mail id

[James Vasile]

I received the attached piece of spam telling me about an exciting
investment opportunity.  Notmuch pulled it in to the database, noting
the message id.  But then it seems to stop paying attention to the
message id.

notmuch can find the mail when I search by the from field:

$ notmuch search from:caroline horn
thread:bcb7  Today 09:29 [1/1] Caroline Horn; This Company is on 
the rise (hv inbox unread)

But not by the message id:
$ notmuch search id:000801cdb5da$17673470$26448589@microsof00t4ko3r
$

Notmuch has correctly pulled the id into the database:
$ notmuch show from:caroline horn | grep id: | sed s/.*\(id:[^ ]*\).*/\1/
id:000801cdb5da$17673470$26448589@microsof00t4ko3r

If notmuch knows the id but won't match it in a search, that looks like
a bug to me.

On a related matter, I have a script that pulls the message id header
From a mail file and then uses notmuch tag id:foo to tag that file.  But
if notmuch sometimes fails to find the id, maybe there is there a better
way to do it?

Thanks,
James



binDCNcXFt8Bc.bin
Description: spam mail


pgpM6XsgLXVyc.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Notmuch search missing mail id

[Tomi Ollila]

On Mon, Oct 29 2012, James Vasile ja...@hackervisions.org wrote:

 I received the attached piece of spam telling me about an exciting
 investment opportunity.  Notmuch pulled it in to the database, noting
 the message id.  But then it seems to stop paying attention to the
 message id.

 notmuch can find the mail when I search by the from field:

 $ notmuch search from:caroline horn
 thread:bcb7  Today 09:29 [1/1] Caroline Horn; This Company is on 
 the rise (hv inbox unread)

 But not by the message id:
 $ notmuch search id:000801cdb5da$17673470$26448589@microsof00t4ko3r
 $

Try 

$ notmuch search id:'000801cdb5da$17673470$26448589@microsof00t4ko3r'

so that the $s are not expanded...

note that id:... does not suffice as shell expands $  ` inside double
quotes.

Tomi



 Notmuch has correctly pulled the id into the database:
 $ notmuch show from:caroline horn | grep id: | sed s/.*\(id:[^ 
 ]*\).*/\1/
 id:000801cdb5da$17673470$26448589@microsof00t4ko3r

 If notmuch knows the id but won't match it in a search, that looks like
 a bug to me.

 On a related matter, I have a script that pulls the message id header
 From a mail file and then uses notmuch tag id:foo to tag that file.  But
 if notmuch sometimes fails to find the id, maybe there is there a better
 way to do it?

 Thanks,
 James

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Notmuch search missing mail id

[James Vasile]

Tomi Ollila tomi.oll...@iki.fi writes:
 On Mon, Oct 29 2012, James Vasile ja...@hackervisions.org wrote:

 I received the attached piece of spam telling me about an exciting
 investment opportunity.  Notmuch pulled it in to the database, noting
 the message id.  But then it seems to stop paying attention to the
 message id.

 notmuch can find the mail when I search by the from field:

 $ notmuch search from:caroline horn
 thread:bcb7  Today 09:29 [1/1] Caroline Horn; This Company is on 
 the rise (hv inbox unread)

 But not by the message id:
 $ notmuch search id:000801cdb5da$17673470$26448589@microsof00t4ko3r
 $

 Try 

 $ notmuch search id:'000801cdb5da$17673470$26448589@microsof00t4ko3r'

 so that the $s are not expanded...

 note that id:... does not suffice as shell expands $  ` inside double
 quotes.

Tomi, that's a good catch and one I should have noticed.  Thanks much
for the help!


pgpJYQeXk0U1n.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] don't show x-foo tags in search view

[James Vasile]

My filters create tags like x-bogotrained-spam that are for internal
bookkeeping.  I don't mind seeing them in the 'show' view, but I didn't
want them cluttering my 'search' view.  This patch omits x-foo and X-foo
tags from the 'search' view.

---
 emacs/notmuch.el |7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index f9454d8..90fafbf 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -793,7 +793,12 @@ non-authors is found, assume that all of the authors 
match.
 (notmuch-search-insert-authors format-string (plist-get result :authors)))
 
((string-equal field tags)
-(let ((tags-str (mapconcat 'identity (plist-get result :tags)  )))
+(let ((tags-str (mapconcat 'identity
+  (delq nil
+(mapcar (lambda (x) (if (equal (upcase 
(truncate-string-to-width  x 2)) X-)
+nil
+(identity x))) 
(plist-get result :tags)))
+   )))
   (insert (propertize (format format-string tags-str)
  'face 'notmuch-tag-face))
 
-- 
1.7.10.4


pgpw2rnBH8BS2.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] don't show x-foo tags in search view

[David Bremner]

James Vasile ja...@hackervisions.org writes:

 My filters create tags like x-bogotrained-spam that are for internal
 bookkeeping.  I don't mind seeing them in the 'show' view, but I didn't
 want them cluttering my 'search' view.  This patch omits x-foo and X-foo
 tags from the 'search' view.

I understand this scratches your itch, but what about something more
customizable?

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] don't show x-foo tags in search view

[James Vasile]

David Bremner da...@tethera.net writes:
 James Vasile ja...@hackervisions.org writes:

 My filters create tags like x-bogotrained-spam that are for internal
 bookkeeping.  I don't mind seeing them in the 'show' view, but I didn't
 want them cluttering my 'search' view.  This patch omits x-foo and X-foo
 tags from the 'search' view.

 I understand this scratches your itch, but what about something more
 customizable?

 d

I thought it was fairly general as is, but you're right it could be more
customizable.  I'll rewrite it with a regex the user can customize.


pgpQ16ElllsGO.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] don't show x-foo tags in search view

[James Vasile]

This patch hides any tags in search view that match the regex specified
in `notmuch-search-hide-tag-regex`.  That variable can be set via setq
or the customize interface.  To hide all tags that begin with x- or
X-, set `notmuch-search-hide-tag-regex` to ^X-.

---
 emacs/notmuch.el |   16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index f9454d8..4bff538 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -775,6 +775,14 @@ non-authors is found, assume that all of the authors 
match.
  (overlay-put overlay 'isearch-open-invisible #'delete-overlay)))
   (insert padding
 
+(defcustom notmuch-search-hide-tag-regex 
+  Regex specifying tags to hide in search view.
+
+Leave blank to disable hiding of tags in search view.
+Note: elisp regexes are case-insensitive
+  :type 'string
+  :group 'notmuch-search)
+
 (defun notmuch-search-insert-field (field format-string result)
   (cond
((string-equal field date)
@@ -793,7 +801,13 @@ non-authors is found, assume that all of the authors 
match.
 (notmuch-search-insert-authors format-string (plist-get result :authors)))
 
((string-equal field tags)
-(let ((tags-str (mapconcat 'identity (plist-get result :tags)  )))
+(let ((tags-str (mapconcat 'identity
+  (delq nil
+(mapcar (lambda (x) (if (and (not (equal 
notmuch-search-hide-tag-regex ))
+ (string-match 
notmuch-search-hide-tag-regex x))
+nil
+x)) (plist-get 
result :tags)))
+   )))
   (insert (propertize (format format-string tags-str)
  'face 'notmuch-tag-face))
 
-- 
1.7.10.4



pgp2GaMmTVEm7.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] don't show x-foo tags in search view

[Austin Clements]

I like it.

Quoth James Vasile on Oct 29 at  5:19 pm:
 This patch hides any tags in search view that match the regex specified
 in `notmuch-search-hide-tag-regex`.  That variable can be set via setq
 or the customize interface.  To hide all tags that begin with x- or
 X-, set `notmuch-search-hide-tag-regex` to ^X-.
 
 ---
  emacs/notmuch.el |   16 +++-
  1 file changed, 15 insertions(+), 1 deletion(-)
 
 diff --git a/emacs/notmuch.el b/emacs/notmuch.el
 index f9454d8..4bff538 100644
 --- a/emacs/notmuch.el
 +++ b/emacs/notmuch.el
 @@ -775,6 +775,14 @@ non-authors is found, assume that all of the authors 
 match.
   (overlay-put overlay 'isearch-open-invisible #'delete-overlay)))
(insert padding
  
 +(defcustom notmuch-search-hide-tag-regex 
 +  Regex specifying tags to hide in search view.

I have no idea why, but Emacs typically uses regexp instead of
regex.

 +
 +Leave blank to disable hiding of tags in search view.

Saying Leave blank supposes that the user knows what the default
value is.  How about An empty string disables hiding of tags in
search view.?

Even better, though, would be to use nil to indicate this, since  is
a perfectly valid regexp and matches everything.  In that case, this
should say something like If nil, no tags will be hidden in search
view.

 +Note: elisp regexes are case-insensitive

Likewise, regexps.  Also, Elisp regexps are not, in general,
case-insensitive.  If we want to control this, we should bind
case-fold-search to nil around the string-match below and say
something here like Matching is case-insensitive.

 +  :type 'string

Better would be 'regexp.  Or, '(choice (const :tag None nil) regexp)
to allow nil or a regexp.

 +  :group 'notmuch-search)
 +
  (defun notmuch-search-insert-field (field format-string result)
(cond
 ((string-equal field date)
 @@ -793,7 +801,13 @@ non-authors is found, assume that all of the authors 
 match.
  (notmuch-search-insert-authors format-string (plist-get result 
 :authors)))
  
 ((string-equal field tags)
 -(let ((tags-str (mapconcat 'identity (plist-get result :tags)  )))
 +(let ((tags-str (mapconcat 'identity
 +  (delq nil
 +(mapcar (lambda (x) (if (and (not (equal 
 notmuch-search-hide-tag-regex ))
 + 
 (string-match notmuch-search-hide-tag-regex x))
 +nil
 +x)) (plist-get 
 result :tags)))
 +   )))

It would be simpler and more robust to use remove-if here.  What about
something like

  (let ((tags-str
 (mapconcat 'identity
(if notmuch-search-hide-tag-regex
(let ((case-fold-search t))
  (remove-if
   (apply-partially #'string-match
notmuch-search-hide-tag-regex)
   (plist-get result :tags)))
  (plist-get result :tags))
 )))

?

(insert (propertize (format format-string tags-str)
   'face 'notmuch-tag-face))
  
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch