new "crypto" branch providing full PGP/MIME support

2011-02-03 Thread Darren McGuicken
On Wed, 02 Feb 2011 17:18:45 -0800, Jameson Rollins  wrote:
> Please test and provide feedback.  I would really like to see this
> series merged into the mainline for the next release, if at all
> possible.

Fan.  Tastic.  Thanks so much for this, incredible work!

Grabbed and testing now, the only slight niggle I see is that when I
reply to an encrypted mail I still get:

  On Thu, 03 Feb 2011 19:17:45 +, Someone  wrote:
  Non-text part: application/pgp-encrypted
  Non-text part: application/octet-stream

Can we pass the decrypted text to message mode on a reply?
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/ad85cf74/attachment.pgp>


notmuch-show-get-filename and maildir location

2011-02-03 Thread micah

I recently upgraded to the newer notmuch to test out the new folder
patch (its great!) and the new decryption/signature verification (its
fantastic!), and noticed something odd start happening.

I think that notmuch moves a new message to the maildir cur directory,


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
Hallo!

On Thu, 3 Feb 2011 12:06:20 -0500, Austin Clements  wrote:
> Is there a reason you keep the remaining data in a string instead of
> taking the more idiomatic elisp approach of leaving it in the process
> buffer?  In fact, the code would probably be simpler if you
> immediately appended the string to the process buffer like a normal
> process-filter and then peeled things away using buffer-oriented
> regexp functions like looking-at.  Elisp is a lot better at
> manipulating buffers than it is at manipulating strings.

Ha, I hear you -- this is what I meant to do originally.  But then, the
save-in-string approach (even though I always considered keeping state in
the string a bit ugly) seemed more simple to me.  As I said: writing
elisp code is not my primary profession...  :-) (Perhaps I should buy a
book about it, or something.)  Now that you confirmed my original idea,
I'll see about re-writing the code accordingly, so thanks for the input!


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/4b4df877/attachment.pgp>


always encrypting messages to self [was: Re: new "crypto" branch providing full PGP/MIME support]

2011-02-03 Thread Daniel Kahn Gillmor
On 02/03/2011 03:34 PM, Jameson Rollins wrote:
> On Thu, 03 Feb 2011 14:52:01 -0500, Daniel Kahn Gillmor  fifthhorseman.net> wrote:
>> You either want to fix this in your emacs config by putting your
>> fingerprint into mml2015-signers and setting mml2015-encrypt-to-self
>>
>> Or you want to set gpg's default-recipient-self option  (and
>> default-recipient option if you hold more than one secret key and want
>> to be sure it chooses the right one)
> 
> Actually, I think the gpg option we're looking for here is
> "encrypt-to".  "default-recipient-self" sets the recipient only if none
> other is specified.  I just set "encrypt-to " in my gpg.conf
> and it seems to do as expected (all encrypted messages are also
> encrypted to myself).

Yes, this is correct.  thanks for figuring that out, Jamie.

--dkg

-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1030 bytes
Desc: OpenPGP digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/65bc54a1/attachment.pgp>


new "crypto" branch providing full PGP/MIME support

2011-02-03 Thread Daniel Kahn Gillmor
Hi Micah--

just wanted to follow up on your points/questions:

On 02/03/2011 11:25 AM, micah anderson wrote:
> 1. I personally think notmuch-show-process-pgpmime should default to
> true

note that with it set to false, you can still M-RET (instead of RET) on
an item in the summary window to have it set for that particular view.

> 2. in-line pgp messages don't have any processing done on them. getting
> the mime-encoded processing work is a huge step and I'm happy that
> works, in-line can (and IMHO, should) come later

yeah, inline is a whole different thing, and much more difficult to
manage programmatically in the notmuch backend, i think.  I dealing with
inline signatures and encryption should be a job for the emacs (or vim
or whatever) frontend.

> 3. i'm not sure expired/revoked keys are handled properly - tested on a
> message that was encrypted by a key that was revoked and got "End of
> file during parsing"

when you say "encrypted by" do you mean "encrypted to"?  do you have
access to the corresponding secret key?

> 4. messages that I sent encrypted to someone are not also encrypted to
> myself, which means that a thread which contains my replies isn't able
> to decrypt my messages in that thread and results in a purple
> 'decryption error'. Perhaps this is an emacs UI tweak that needs to be
> made to get messages also encrypted to my own key?

this is an issue for the emacs message modes (or maybe for your gpg
configuration), not for notmuch.

You either want to fix this in your emacs config by putting your
fingerprint into mml2015-signers and setting mml2015-encrypt-to-self

Or you want to set gpg's default-recipient-self option  (and
default-recipient option if you hold more than one secret key and want
to be sure it chooses the right one)

> 5. unknown keys are represented in a long format,
> eg. '0x5585F58CC827A062' when most tools represent them just with their
> shortened keyid (in this case this one would be: 0xC827A062), is there a
> particular reason for this?

Short keyIDs are easily spoofable; believing anything based on a matched
short keyID is a mistake.  "unknown keys" themselves may or may not have
properly signed a message -- since we don't have the key handy, we don't
have a way of checking.

note that "unknown key" is different from "good signature from known key
but we do not know who controls the key"

> I recognize some people's keyids in the
> short form, but do not in the longform.

You can derive the short form from the long form by ignoring everything
but the last 8 hex characters.  But if you actually recognize the short
form, i'd expect you to have the relevant key in your keyring; is this
really a use case worth bothering with?

do you have a suggestion for how you think it should behave differently?

--dkg

-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1030 bytes
Desc: OpenPGP digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/8c4b917c/attachment.pgp>


new "crypto" branch providing full PGP/MIME support

2011-02-03 Thread Jameson Rollins
On Thu, 03 Feb 2011 20:42:38 +, Darren McGuicken  wrote:
> On Wed, 02 Feb 2011 17:18:45 -0800, Jameson Rollins  finestructure.net> wrote:
> > Please test and provide feedback.  I would really like to see this
> > series merged into the mainline for the next release, if at all
> > possible.
> 
> Fan.  Tastic.  Thanks so much for this, incredible work!

Thanks!  I hope people find it useful.

> Grabbed and testing now, the only slight niggle I see is that when I
> reply to an encrypted mail I still get:
> 
>   On Thu, 03 Feb 2011 19:17:45 +, Someone  wrote:
>   Non-text part: application/pgp-encrypted
>   Non-text part: application/octet-stream
> 
> Can we pass the decrypted text to message mode on a reply?

Yeah, some folks pointed this out on #notmuch this morning.  The
decryption is only happening in notmuch-show, but we need to do it in
notmuch-reply as well for the decrypted text to show up in the reply.
I'm working on it now.

jamie.

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/9b51ebd4/attachment.pgp>


new "crypto" branch providing full PGP/MIME support

2011-02-03 Thread Daniel Kahn Gillmor
On 02/02/2011 08:18 PM, Jameson Rollins wrote:
> Hi, all.  I have pushed a new branch called "crypto" to my notmuch
> repository [0].  This branch provides full support for PGP/MIME signed
> and encrypted messages, including emacs UI support.

I have tested this, and am now using it.  I'm very happy with it.  I
support its inclusion in the mainline.

Thanks for doing this, Jamie.  This is excellent!

 less important stuff follows

I want to raise one (non-blocking) question about the decryption to see
if anyone has any suggestions:

If you do "notmuch show --format=json" on a PGP/MIME-encrypted plaintext
message, it emits the base message, which is structured like this:

1 ???multipart/encrypted
2  ??application/pgp-encrypted attachment
3  ??application/octet-stream inline [msg.asc]

with these patches, if you do "notmuch show --format=json --decrypt", it
emits this instead:

1 ???multipart/encrypted
2  ??text/plain inline

and it attaches an encstatus (and possibly sigstatus, if the message was
signed) to part 1.  I'll call this "method A".

There are other methods that could be used as well, and it's worth
making sure we've chosen one that we think is what we'll want in the
future.  here are two other proposals:

Method B:

1 ???multipart/encrypted
2  ??application/pgp-encrypted attachment
3  ??text/plain inline

That is, just replace part 3 (the actual encrypted body) with the
decrypted material.

This has the advantage that for single-part encrypted messages, the
structure and part numbers of the message remains the same as without
--decrypt.


Method C:

1 ??text/plain inline

That is, replace the entire multipart/encrypted with the decrypted material.

This avoids having an explicitly-labeled "multipart/encrypted" wrapper
around cleartext (which might be considered odd).  It would mean
attaching the encstatus and sigstatus directly to the decrypted part,
though.



I don't actually see any of these methods as being significantly better
than the others -- i think they all have some inherent ugliness.  So i'm
fine with going with method A as Jamie chose it and has it working.  But
i wanted to see if anyone had strong arguments in favor of the other
methods (or if there are other --decrypt methods we could use, for that
matter)

--dkg

-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1030 bytes
Desc: OpenPGP digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/7c936d02/attachment.pgp>


new "crypto" branch providing full PGP/MIME support

2011-02-03 Thread Jameson Rollins
On Thu, 03 Feb 2011 14:52:01 -0500, Daniel Kahn Gillmor  wrote:
> On 02/03/2011 11:25 AM, micah anderson wrote:
> > 1. I personally think notmuch-show-process-pgpmime should default to
> > true
> 
> note that with it set to false, you can still M-RET (instead of RET) on
> an item in the summary window to have it set for that particular view.

This is also useful if you set notmuch-show-process-pgpmime and ever
come across a message that is causing crypto problems.  M-RET will
return you to the normal view.

> > 3. i'm not sure expired/revoked keys are handled properly - tested on a
> > message that was encrypted by a key that was revoked and got "End of
> > file during parsing"
> 
> when you say "encrypted by" do you mean "encrypted to"?  do you have
> access to the corresponding secret key?

I also seem to be noticing issues with revoked keys.  I'm looking in to
the issue.  If anyone else notices something similar, please do relay
your experience.

> > 4. messages that I sent encrypted to someone are not also encrypted to
> > myself, which means that a thread which contains my replies isn't able
> > to decrypt my messages in that thread and results in a purple
> > 'decryption error'. Perhaps this is an emacs UI tweak that needs to be
> > made to get messages also encrypted to my own key?
> 
> this is an issue for the emacs message modes (or maybe for your gpg
> configuration), not for notmuch.
> 
> You either want to fix this in your emacs config by putting your
> fingerprint into mml2015-signers and setting mml2015-encrypt-to-self
> 
> Or you want to set gpg's default-recipient-self option  (and
> default-recipient option if you hold more than one secret key and want
> to be sure it chooses the right one)

Actually, I think the gpg option we're looking for here is
"encrypt-to".  "default-recipient-self" sets the recipient only if none
other is specified.  I just set "encrypt-to " in my gpg.conf
and it seems to do as expected (all encrypted messages are also
encrypted to myself).

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/491434d0/attachment.pgp>


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Austin Clements
Nice catch.

Is there a reason you keep the remaining data in a string instead of
taking the more idiomatic elisp approach of leaving it in the process
buffer?  In fact, the code would probably be simpler if you
immediately appended the string to the process buffer like a normal
process-filter and then peeled things away using buffer-oriented
regexp functions like looking-at.  Elisp is a lot better at
manipulating buffers than it is at manipulating strings.

On Wed, Feb 2, 2011 at 6:56 PM, Thomas Schwinge  wrote:
> This issue has been lying in ambush as of 2009-11-24's commit
> 93af7b574598637c2766dd1f8ef343962c9a8efb.
>
> Signed-off-by: Thomas Schwinge 
> ---
> ?emacs/notmuch.el | ? 70 +++--
> ?1 files changed, 41 insertions(+), 29 deletions(-)
>
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index 3d82f0d..35ccee6 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors 
> match."
> ? ? (propertize authors 'face 'notmuch-search-matching-authors)))
>
> ?(defun notmuch-search-insert-authors (format-string authors)
> - ?;; Save the match data to avoid interfering with
> - ?;; `notmuch-search-process-filter'.
> - ?(save-match-data
> ? ? (let* ((formatted-authors (format format-string authors))
> ? ? ? ? ? (formatted-sample (format format-string ""))
> ? ? ? ? ? (visible-string formatted-authors)
> @@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors 
> match."
> ? ? ? ? ?(setq overlay (make-overlay start (point)))
> ? ? ? ? ?(overlay-put overlay 'invisible invis-spec)
> ? ? ? ? ?(overlay-put overlay 'isearch-open-invisible 
> #'notmuch-search-isearch-authors-show)))
> - ? ? ?(insert padding
> + ? ? ?(insert padding)))
>
> ?(defun notmuch-search-insert-field (field date count authors subject tags)
> ? (cond
> @@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors 
> match."
> ? ? ? ? ?do (notmuch-search-insert-field field date count authors subject 
> tags)))
> ? (insert "\n"))
>
> +(defvar notmuch-search-process-filter-data nil
> + ?"Data that has not yet been processed.")
> +(make-variable-buffer-local 'notmuch-search-process-filter-data)
> +
> ?(defun notmuch-search-process-filter (proc string)
> ? "Process and filter the output of \"notmuch search\""
> ? (let ((buffer (process-buffer proc))
> @@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors 
> match."
> ? ? (if (buffer-live-p buffer)
> ? ? ? ?(with-current-buffer buffer
> ? ? ? ? ?(save-excursion
> - ? ? ? ? ? (let ((line 0)
> - ? ? ? ? ? ? ? ? (more t)
> - ? ? ? ? ? ? ? ? (inhibit-read-only t))
> - ? ? ? ? ? ? (while more
> - ? ? ? ? ? ? ? (if (string-match "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
> \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$" string line)
> - ? ? ? ? ? ? ? ? ? (let* ((thread-id (match-string 1 string))
> - ? ? ? ? ? ? ? ? ? ? ? ? ?(date (match-string 2 string))
> - ? ? ? ? ? ? ? ? ? ? ? ? ?(count (match-string 3 string))
> - ? ? ? ? ? ? ? ? ? ? ? ? ?(authors (match-string 4 string))
> - ? ? ? ? ? ? ? ? ? ? ? ? ?(subject (match-string 5 string))
> - ? ? ? ? ? ? ? ? ? ? ? ? ?(tags (match-string 6 string))
> - ? ? ? ? ? ? ? ? ? ? ? ? ?(tag-list (if tags (save-match-data (split-string 
> tags)
> - ? ? ? ? ? ? ? ? ? ? (goto-char (point-max))
> - ? ? ? ? ? ? ? ? ? ? (let ((beg (point-marker)))
> - ? ? ? ? ? ? ? ? ? ? ? (notmuch-search-show-result date count authors 
> subject tags)
> - ? ? ? ? ? ? ? ? ? ? ? (notmuch-search-color-line beg (point-marker) 
> tag-list)
> - ? ? ? ? ? ? ? ? ? ? ? (put-text-property beg (point-marker) 
> 'notmuch-search-thread-id thread-id)
> - ? ? ? ? ? ? ? ? ? ? ? (put-text-property beg (point-marker) 
> 'notmuch-search-authors authors)
> - ? ? ? ? ? ? ? ? ? ? ? (put-text-property beg (point-marker) 
> 'notmuch-search-subject subject)
> - ? ? ? ? ? ? ? ? ? ? ? (if (string= thread-id notmuch-search-target-thread)
> - ? ? ? ? ? ? ? ? ? ? ? ? ? (progn
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? (set 'found-target beg)
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? (set 'notmuch-search-target-thread "found"
> - ? ? ? ? ? ? ? ? ? ? (set 'line (match-end 0)))
> - ? ? ? ? ? ? ? ? (set 'more nil)
> + ? ? ? ? ? (let ((inhibit-read-only t)
> + ? ? ? ? ? ? ? ? ;; We may have a partial line saved from the last iteration.
> + ? ? ? ? ? ? ? ? (string (concat notmuch-search-process-filter-data string))
> + ? ? ? ? ? ? ? ? (start 0))
> + ? ? ? ? ? ? (goto-char (point-max))
> + ? ? ? ? ? ? ;; Split `string' into lines.
> + ? ? ? ? ? ? (while (string-match "\n" string start)
> + ? ? ? ? ? ? ? (let ((line (substring string start (match-beginning 0
> + ? ? ? ? ? ? ? ? ;; Save the beginning of the next line already here, so that
> + ? ? ? ? ? ? ? ? ;; we can mangle the match data later on.
> + ? ? ? ? ? ? ? ? (setq start (match-end 0))
> + ? ? ? ? ? ? ? ? (if (string-match
> + ? ? ? ? ? ? ? ? ? ? ?"^\\(thread:[0-9A-Fa-f]*\\) 

notmuch's idea of concurrency / failing an invocation

2011-02-03 Thread micah anderson
On Thu, 27 Jan 2011 12:35:16 -0800, Jameson Rollins  wrote:
> On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson  
> wrote:
> > Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> > database growing, and perhaps some fragmentation somewhere, this has
> > become *incredibly* annoying for me. I am checking email every 30
> > minutes, and I'm nicing and ionicing the processes so I can use my
> > machine, but while those processes are running, I'm effectively locked
> > out of a good portion of my email. 
> 
> I also have a very slow disk, but this is very rarely a problem for me.
> I retrieve mail every 10 minutes, and the corresponding notmuch new
> usually takes a minute or so.  I really haven't found it to be much of a
> bother to just wait it out.

Sometimes I can have several thousand messages to add/remove from the
database. I know this is probably unusal, but for me its not due to
system emails. I suppose if I checked my email more frequently, I'd have
less to process on each run, but thats more side-stepping the
concurrency issue.

micah
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/e1123cff/attachment.pgp>


new "crypto" branch providing full PGP/MIME support

2011-02-03 Thread micah anderson
On Wed, 02 Feb 2011 17:18:45 -0800, Jameson Rollins  wrote:
> Hi, all.  I have pushed a new branch called "crypto" to my notmuch
> repository [0].  This branch provides full support for PGP/MIME signed
> and encrypted messages, including emacs UI support.  It has been applied
> on top of cworth's current master (21e97c50).  It includes the
> following:
> 
> * David Edmondson's improved multipart handling patch series (cherry-picked)
> * Daniel Gillmor's PGP/MIME signature verification patch series 
> (cherry-picked)
> * my PGP/MIME decryption+verification patch series
> * a test suite for signature verification and decryption
> * emacs support for the above

Don't forget that you also included man page changes!

> Please test and provide feedback.  I would really like to see this
> series merged into the mainline for the next release, if at all
> possible.

I've really really really wanted this functionality, so I pulled this
right away and have been testing it, its really slick! I like how the
emacs UI gives you good visual feedback for different signature states
(I had red for a signature from Sebastian Spaeth because I did not have
the key; orange for when I obtained that key; and green for Jameson and
dkg's mails because I have exchanged keys with them and have full
validity for them; and purple for a decryption error). The minor delay
in opening a thread with signatures is not bad, and is to be expected.

And messages that are PGP/MIME encrypted are decrypted automatically,
wow, this is amazing. I enthusiastically support merging this into
mainline for the next release.

I have a couple points of feedback that I do not think should hold up
merging this work:

1. I personally think notmuch-show-process-pgpmime should default to
true

2. in-line pgp messages don't have any processing done on them. getting
the mime-encoded processing work is a huge step and I'm happy that
works, in-line can (and IMHO, should) come later

3. i'm not sure expired/revoked keys are handled properly - tested on a
message that was encrypted by a key that was revoked and got "End of
file during parsing"

4. messages that I sent encrypted to someone are not also encrypted to
myself, which means that a thread which contains my replies isn't able
to decrypt my messages in that thread and results in a purple
'decryption error'. Perhaps this is an emacs UI tweak that needs to be
made to get messages also encrypted to my own key?

5. unknown keys are represented in a long format,
eg. '0x5585F58CC827A062' when most tools represent them just with their
shortened keyid (in this case this one would be: 0xC827A062), is there a
particular reason for this? I recognize some people's keyids in the
short form, but do not in the longform.

6. this is awesome, huge thanks to everyone who has worked on this!

micah
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/e0066155/attachment.pgp>


[PATCH 4/4] ruby: Add generated files to gitignore

2011-02-03 Thread Ali Polatel
On Mon, 31 Jan 2011 23:48:57 +0100, Thomas Schwinge  
wrote:
> Hallo!
> 
> On Mon, 10 Jan 2011 16:39:28 +0200, Ali Polatel  wrote:
> > --- a/.gitignore
> > +++ b/.gitignore
> > @@ -13,3 +13,7 @@ libnotmuch.so*
> >  .*.swp
> >  *.elc
> >  releases
> > +
> > +bindings/ruby/mkmf.log
> > +bindings/ruby/notmuch.so
> > +bindings/ruby/Makefile
> 
> These should rather be put into bindings/ruby/.gitignore, I'd say.

Good idea!
Just pushed ca69ce2397c0bb9b0e590a80b898ab4d58bb

> 
> Gr??e,
>  Thomas

-- 
Regards,
Ali Polatel
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110203/a4a9be91/attachment.pgp>


Folder search semantics (was Re: [RFC PATCH v2 0/8] Custom query parser, date search, folder search, and more)

2011-02-03 Thread Austin Clements
Quoth Carl Worth on Feb 02 at  2:48 pm:
> Restricting my reply to one tiny bit of your mail:
> 
> You wrote:
> > non-recursive is the only thing that makes sense for Maildir++ folders
> 
> Either I'm not understanding Maildir++ folders, or I don't agree with
> you.
> 
> I might have an email archive that looks like this:
> 
>   Maildir
> .work
>   .project1
>   .project2
>   .etc...
> .family
>   .dad
>   .mom
>   .brother
>   .etc...
> 
> With the above setup, what would be unreasonable about wanting to search
> for all work-related messages (across all projects, say) with a string
> like "folder:work" ?
> 
> Now, a person might definitely want to search for messages in the
> ".work" folder directly, (not including the sub-folders), so we should
> provide support for users to get at that behavior as well, (such as a
> proposed "folder:work$" or so).
> 
> To me, both cases are perfectly legitimate, and I don't understand an
> argument that claims that only one makes sense. (Or again, I may be
> misunderstanding something.)

(Somebody with more first-hand Maildir++ experience should jump in here.
I stopped using Maildir++ a long time ago, so I may have no idea what
I'm talking about.)

Both cases are perfectly legitimate.

However, the issue with Maildir++ is that the inbox is stored in the
top-level directory:

  Maildir
cur
new
tmp
.work
.work.project1

As a consequence, all folders are subfolders of the inbox.  With
recursive search, a search for your inbox folder returns *all* of your
messages.  I wasn't trying to say that we shouldn't support recursive
search (I'm all for flexibility), but it's a confusing default for
Maildir++ because of this.

Maildir++ has the added twist that the inbox folder has no name.  As a
result, currently notmuch can't search for a Maildir++ inbox folder,
which needs to be addressed somehow.  The least surprising approach
would compatibility with the Maildir++ convention of calling the
top-level folder INBOX, the subfolder INBOX.work, etc.


Maildir++ issues aside, I submit that rooted, non-recursive folder
searches are a more natural default with a more conventional syntactic
extension to non-rooted/recursive searches.  In
id:87aaiy3u65.fsf at yoom.home.cworth.org, you mentioned that you
implemented non-rooted folder search to mimic subject search.  But file
system paths are not natural language like subject lines.  File system
paths are hierarchical and rooted.

Of course, special query operators like ^ and $ can mitigate this, but
these queries *aren't* regexps and, furthermore, people don't usually
apply regexps to file names.  They apply globs.  Glob syntax has the
added benefit of congruity with Xapian wildcard syntax.  This naturally
leads to a rooted, non-recursive syntax by default (like globs), where a
* at the end means recursive and a * at the beginning means non-rooted.
In fact, we could easily generalize this to arbitrary shell globs.


Here's a proposal that, I think, addresses Maildir++ inboxes and
subfolders; rooted, non-rooted, recursive, and non-recursive queries;
and then some.  Plus, it wouldn't require many code changes; you've
already done the hard work.

Switch XFOLDER from a probabilistic prefix with word-splitting to a
boolean prefix without word-splitting.  When indexing, strip off the cur
or new and examine the resulting directory name.  If it's the mail root,
this is a Maildir++ inbox, so add the term XFOLDERINBOX.  If it starts
with a dot, it's a Maildir++ subfolder, so add the term
XFOLDERINBOX<.dirname>.  Otherwise, add the term XFOLDER.
Then, using a custom query transform for the "folder:" prefix, enumerate
XFOLDER terms and form a synonym query out of those that fnmatch the
user's folder query.


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
This issue has been lying in ambush as of 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.

Signed-off-by: Thomas Schwinge 
---
 emacs/notmuch.el |   70 +++--
 1 files changed, 41 insertions(+), 29 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 3d82f0d..35ccee6 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors match."
 (propertize authors 'face 'notmuch-search-matching-authors)))

 (defun notmuch-search-insert-authors (format-string authors)
-  ;; Save the match data to avoid interfering with
-  ;; `notmuch-search-process-filter'.
-  (save-match-data
 (let* ((formatted-authors (format format-string authors))
   (formatted-sample (format format-string ""))
   (visible-string formatted-authors)
@@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors match."
  (setq overlay (make-overlay start (point)))
  (overlay-put overlay 'invisible invis-spec)
  (overlay-put overlay 'isearch-open-invisible 
#'notmuch-search-isearch-authors-show)))
-  (insert padding
+  (insert padding)))

 (defun notmuch-search-insert-field (field date count authors subject tags)
   (cond
@@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors 
match."
  do (notmuch-search-insert-field field date count authors subject 
tags)))
   (insert "\n"))

+(defvar notmuch-search-process-filter-data nil
+  "Data that has not yet been processed.")
+(make-variable-buffer-local 'notmuch-search-process-filter-data)
+
 (defun notmuch-search-process-filter (proc string)
   "Process and filter the output of \"notmuch search\""
   (let ((buffer (process-buffer proc))
@@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors 
match."
 (if (buffer-live-p buffer)
(with-current-buffer buffer
  (save-excursion
-   (let ((line 0)
- (more t)
- (inhibit-read-only t))
- (while more
-   (if (string-match "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$" string line)
-   (let* ((thread-id (match-string 1 string))
-  (date (match-string 2 string))
-  (count (match-string 3 string))
-  (authors (match-string 4 string))
-  (subject (match-string 5 string))
-  (tags (match-string 6 string))
-  (tag-list (if tags (save-match-data (split-string 
tags)
- (goto-char (point-max))
- (let ((beg (point-marker)))
-   (notmuch-search-show-result date count authors subject 
tags)
-   (notmuch-search-color-line beg (point-marker) tag-list)
-   (put-text-property beg (point-marker) 
'notmuch-search-thread-id thread-id)
-   (put-text-property beg (point-marker) 
'notmuch-search-authors authors)
-   (put-text-property beg (point-marker) 
'notmuch-search-subject subject)
-   (if (string= thread-id notmuch-search-target-thread)
-   (progn
- (set 'found-target beg)
- (set 'notmuch-search-target-thread "found"
- (set 'line (match-end 0)))
- (set 'more nil)
+   (let ((inhibit-read-only t)
+ ;; We may have a partial line saved from the last iteration.
+ (string (concat notmuch-search-process-filter-data string))
+ (start 0))
+ (goto-char (point-max))
+ ;; Split `string' into lines.
+ (while (string-match "\n" string start)
+   (let ((line (substring string start (match-beginning 0
+ ;; Save the beginning of the next line already here, so that
+ ;; we can mangle the match data later on.
+ (setq start (match-end 0))
+ (if (string-match
+  "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$"
+  line)
+ (let* ((thread-id (match-string 1 line))
+(date (match-string 2 line))
+(count (match-string 3 line))
+(authors (match-string 4 line))
+(subject (match-string 5 line))
+(tags (match-string 6 line))
+(tag-list (if tags (split-string tags
+   (let ((beg (point-marker)))
+ (notmuch-search-show-result date count authors 
subject tags)
+ 

[PATCH 1/2] New test: Emacs' forgetfulness.

2011-02-03 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge 
---
 test/emacs-forgetfulness |   38 ++
 test/notmuch-test|1 +
 2 files changed, 39 insertions(+), 0 deletions(-)
 create mode 100755 test/emacs-forgetfulness

diff --git a/test/emacs-forgetfulness b/test/emacs-forgetfulness
new file mode 100755
index 000..e17b26f
--- /dev/null
+++ b/test/emacs-forgetfulness
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+test_description=Emacs\'\ forgetfulness
+
+. test-lib.sh
+
+# RFC822 imposes a 998 character limit per line.
+x=0123456789 # 10
+x=$x$x$x$x$x$x$x$x$x$x # 100
+x=$x$x$x$x$x$x$x$x$x # 900
+
+# If setting this ``too high'' (TODO: yet to be determined), Emacs will crash
+# with a segmentation fault.
+n=20
+for i in $(seq 1 $n); do
+  # Roughly 2 KiB per message.  That is, we need two messages in order to
+  # exceed the typical size of the pipe buffer (4 KiB on commodity systems).
+  generate_message [subject]=$i-$x [from]=$i-$x at x.x
+done
+# With 20 messages ?? 2 KiB, we have about 10 full pipe buffers, which should 
be
+# enough to trigger the erroneous behavior.
+
+notmuch new > /dev/null
+
+test_begin_subtest 'Search for all messages'
+output=$(exec 2>&1; \
+ diff -wu \
+   <(notmuch search \* \
+   | sed \
+   -e 's%^thread:[0-9a-f]*\ %%' \
+   -e 's%;%%'; \
+ echo 'End of search results.'; \
+ echo) \
+   <(test_emacs 2>&1 \
+   '(notmuch-search "*") (notmuch-test-wait) (message 
(buffer-string))'))
+test_expect_equal "$output" ''
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index 9d77c0f..2f11eac 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -36,6 +36,7 @@ TESTS="
   encoding
   emacs
   maildir-sync
+  emacs-forgetfulness
 "

 # Clean up any results from a previous run
-- 
1.7.1



[BUG] Emacs UI dropping every 25th line, roughly

2011-02-03 Thread Thomas Schwinge
Hallo!

Here is the problem, as suspected.

On Wed, 02 Feb 2011 17:12:16 +0100, I wrote:
> At this / in the middle of this point, the 4 KiB size is hit.  After
> accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
> first chunk (line breaks inserted for clarity):
> 
> 16:26:57.928798 read(8, "[first two dozen results]"
> "thread:06c7   2009-12-16 [2/2] 
> Thomas Schwinge; [subject] ([flags])\n"
> "t", 4096) = 4095
> 
> The last result line is obviously incomplete, only the `t' so far.

notmuch.el:notmuch-search-process-filter will be called for processing
these lines.  It'll do fine up to the single `t' -- which simply isn't a
conforming line and thus will be dropped (which is questionable behavior
anyway, I would think?).

> There is more to be read, so Emacs quickly goes on with the next chunk:
> 
> 16:26:57.966247 read(8, "hread:0ac4   2009-12-16 [1/1] jsm28 
> at sourceware.org; [subject] ([flags])\n"
> "[more results]", 4096) = 4095

Same thing; this time ``hread:[...]'' isn't a confirming line, and will
be dropped.  After that, regular processing continues.


This problem has been around as of the beginning of the incremental /
asynchronous search results changes, as documented in
id:87aayatnw4.fsf at yoom.home.cworth.org on 2009-11-25; 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.


Emacs LISP is not my speciality, neither is any other LISP dialect, so
someone please carefully review this.


Gr??e (und gute Nacht...),
 Thomas



Re: [BUG] Emacs UI dropping every 25th line, roughly

2011-02-03 Thread Thomas Schwinge
Hallo!

On Sun, 30 Jan 2011 22:02:03 +0100, I wrote:
 That is, roughly every 25th line is dropped in the Emacs notmuch-search
 buffer!  (Via setting notmuch-command to a shell script, I intercepted
 the notmuch / emacs pipeline with tee, and found that what comes out of
 ``notmuch | tee'' is still sane, so I'm fairly sure it's an Emacs /
 notmuch elisp code issue.)
 
 And, 25 times the medium length of a ``notmuch search [...]'' output line
 is... 4 KiB, the standard page size.  Is this ``just'' a problem in the
 notmuch elisp code, or is Emacs doing something awful with (wild
 speculation...) short reads on buffer page boundaries (or whatever else)?

I began to analyze this.  Here is a dump of my steps.

I used strace on the (already running) Emacs process.  strace would log
to several files: one for the main (Emacs) process, and a new one for
each forked process.  I can see that a ``notmuch search [...]'' process
is fork()ed / exec()uted.  It does a lot of read()ing from the DB, and
write()s out the results record by record (which in our case means line
by line).  Up to the point where I examined, there have been no short
writes.  The log:

[some write()s for the first two dozen search results]
16:26:54.143464 write(1, thread:06c7   2009-12-16 [2/2] Thomas 
Schwinge; [subject] ([flags])\n, 135) = 135
[...]
16:26:54.266866 write(1, thread:0ac4   2009-12-16 [1/1] 
js...@sourceware.org; [subject] ([flags])\n, 181) = 181

At this / in the middle of this point, the 4 KiB size is hit.  After
accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
first chunk (line breaks inserted for clarity):

16:26:57.928798 read(8, [first two dozen results]
thread:06c7   2009-12-16 [2/2] Thomas 
Schwinge; [subject] ([flags])\n
t, 4096) = 4095

The last result line is obviously incomplete, only the `t' so far.

There is more to be read, so Emacs quickly goes on with the next chunk:

16:26:57.966247 read(8, hread:0ac4   2009-12-16 [1/1] 
js...@sourceware.org; [subject] ([flags])\n
[more results], 4096) = 4095

That's the remainder of the partial line, plus further results.  This is
all fine.  It's now Emacs' job to re-assemble the two partial lines
(thread ac4).  I note that indeed this search result is missing in the
notmuch-search buffer.

(By the way, I wonder why read() returns 4095 bytes instead of 4096?)

I'll next move on to debugging what actually appears at the elisp layer.
I somehow don't think the Emacs core is faulty; probably rather our
asynchronous results presentation layer handles partial lines
incorrectly?  More to come later.


Grüße,
 Thomas


pgpXHIoQVpY05.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Shell programming: directories are ``executable'' (that is, searchable), too.

2011-02-03 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge tho...@schwinge.name
---
 test/test-lib.sh |5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index f536172..3471ead 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -815,13 +815,14 @@ EOF
 }
 
 
+# Locate the directory containing the `notmuch' executable we are to use.
 find_notmuch_path ()
 {
 dir=$1
 
 while [ -n $dir ]; do
bin=$dir/notmuch
-   if [ -x $bin ]; then
+   if [ -f $bin -a -x $bin ]; then
echo $dir
return
fi
@@ -858,7 +859,7 @@ then
 
make_valgrind_symlink () {
# handle only executables
-   test -x $1 || return
+   test -f $1 -a -x $1 || return
 
base=$(basename $1)
symlink_target=$TEST_DIRECTORY/../$base
-- 
1.7.1

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [RFC PATCH v2 0/8] Custom query parser, date search, folder search, and more

2011-02-03 Thread Carl Worth
Restricting my reply to one tiny bit of your mail:

You wrote:
 non-recursive is the only thing that makes sense for Maildir++ folders

Either I'm not understanding Maildir++ folders, or I don't agree with
you.

I might have an email archive that looks like this:

  Maildir
.work
  .project1
  .project2
  .etc...
.family
  .dad
  .mom
  .brother
  .etc...

With the above setup, what would be unreasonable about wanting to search
for all work-related messages (across all projects, say) with a string
like folder:work ?

Now, a person might definitely want to search for messages in the
.work folder directly, (not including the sub-folders), so we should
provide support for users to get at that behavior as well, (such as a
proposed folder:work$ or so).

To me, both cases are perfectly legitimate, and I don't understand an
argument that claims that only one makes sense. (Or again, I may be
misunderstanding something.)

-Carl

-- 
carl.d.wo...@intel.com


pgpdyeeZ7h1Pw.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
This issue has been lying in ambush as of 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.

Signed-off-by: Thomas Schwinge tho...@schwinge.name
---
 emacs/notmuch.el |   70 +++--
 1 files changed, 41 insertions(+), 29 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 3d82f0d..35ccee6 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors match.
 (propertize authors 'face 'notmuch-search-matching-authors)))
 
 (defun notmuch-search-insert-authors (format-string authors)
-  ;; Save the match data to avoid interfering with
-  ;; `notmuch-search-process-filter'.
-  (save-match-data
 (let* ((formatted-authors (format format-string authors))
   (formatted-sample (format format-string ))
   (visible-string formatted-authors)
@@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors match.
  (setq overlay (make-overlay start (point)))
  (overlay-put overlay 'invisible invis-spec)
  (overlay-put overlay 'isearch-open-invisible 
#'notmuch-search-isearch-authors-show)))
-  (insert padding
+  (insert padding)))
 
 (defun notmuch-search-insert-field (field date count authors subject tags)
   (cond
@@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors 
match.
  do (notmuch-search-insert-field field date count authors subject 
tags)))
   (insert \n))
 
+(defvar notmuch-search-process-filter-data nil
+  Data that has not yet been processed.)
+(make-variable-buffer-local 'notmuch-search-process-filter-data)
+
 (defun notmuch-search-process-filter (proc string)
   Process and filter the output of \notmuch search\
   (let ((buffer (process-buffer proc))
@@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors 
match.
 (if (buffer-live-p buffer)
(with-current-buffer buffer
  (save-excursion
-   (let ((line 0)
- (more t)
- (inhibit-read-only t))
- (while more
-   (if (string-match ^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$ string line)
-   (let* ((thread-id (match-string 1 string))
-  (date (match-string 2 string))
-  (count (match-string 3 string))
-  (authors (match-string 4 string))
-  (subject (match-string 5 string))
-  (tags (match-string 6 string))
-  (tag-list (if tags (save-match-data (split-string 
tags)
- (goto-char (point-max))
- (let ((beg (point-marker)))
-   (notmuch-search-show-result date count authors subject 
tags)
-   (notmuch-search-color-line beg (point-marker) tag-list)
-   (put-text-property beg (point-marker) 
'notmuch-search-thread-id thread-id)
-   (put-text-property beg (point-marker) 
'notmuch-search-authors authors)
-   (put-text-property beg (point-marker) 
'notmuch-search-subject subject)
-   (if (string= thread-id notmuch-search-target-thread)
-   (progn
- (set 'found-target beg)
- (set 'notmuch-search-target-thread found
- (set 'line (match-end 0)))
- (set 'more nil)
+   (let ((inhibit-read-only t)
+ ;; We may have a partial line saved from the last iteration.
+ (string (concat notmuch-search-process-filter-data string))
+ (start 0))
+ (goto-char (point-max))
+ ;; Split `string' into lines.
+ (while (string-match \n string start)
+   (let ((line (substring string start (match-beginning 0
+ ;; Save the beginning of the next line already here, so that
+ ;; we can mangle the match data later on.
+ (setq start (match-end 0))
+ (if (string-match
+  ^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
\\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$
+  line)
+ (let* ((thread-id (match-string 1 line))
+(date (match-string 2 line))
+(count (match-string 3 line))
+(authors (match-string 4 line))
+(subject (match-string 5 line))
+(tags (match-string 6 line))
+(tag-list (if tags (split-string tags
+   (let ((beg (point-marker)))
+ (notmuch-search-show-result date count authors 
subject tags)
+ 

Re: [BUG] Emacs UI dropping every 25th line, roughly

2011-02-03 Thread Thomas Schwinge
Hallo!

Here is the problem, as suspected.

On Wed, 02 Feb 2011 17:12:16 +0100, I wrote:
 At this / in the middle of this point, the 4 KiB size is hit.  After
 accumulating some more (notice the 3.6 seconds delay), Emacs read()s the
 first chunk (line breaks inserted for clarity):
 
 16:26:57.928798 read(8, [first two dozen results]
 thread:06c7   2009-12-16 [2/2] 
 Thomas Schwinge; [subject] ([flags])\n
 t, 4096) = 4095
 
 The last result line is obviously incomplete, only the `t' so far.

notmuch.el:notmuch-search-process-filter will be called for processing
these lines.  It'll do fine up to the single `t' -- which simply isn't a
conforming line and thus will be dropped (which is questionable behavior
anyway, I would think?).

 There is more to be read, so Emacs quickly goes on with the next chunk:
 
 16:26:57.966247 read(8, hread:0ac4   2009-12-16 [1/1] 
 js...@sourceware.org; [subject] ([flags])\n
 [more results], 4096) = 4095

Same thing; this time ``hread:[...]'' isn't a confirming line, and will
be dropped.  After that, regular processing continues.


This problem has been around as of the beginning of the incremental /
asynchronous search results changes, as documented in
id:87aayatnw4@yoom.home.cworth.org on 2009-11-25; 2009-11-24's commit
93af7b574598637c2766dd1f8ef343962c9a8efb.


Emacs LISP is not my speciality, neither is any other LISP dialect, so
someone please carefully review this.


Grüße (und gute Nacht...),
 Thomas

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


new crypto branch providing full PGP/MIME support

2011-02-03 Thread Jameson Rollins
Hi, all.  I have pushed a new branch called crypto to my notmuch
repository [0].  This branch provides full support for PGP/MIME signed
and encrypted messages, including emacs UI support.  It has been applied
on top of cworth's current master (21e97c50).  It includes the
following:

* David Edmondson's improved multipart handling patch series (cherry-picked)
* Daniel Gillmor's PGP/MIME signature verification patch series (cherry-picked)
* my PGP/MIME decryption+verification patch series
* a test suite for signature verification and decryption
* emacs support for the above

I have signed-off on all cherry-picked commits, as I have reviewed and
tested them extensively (I have been using them in my daily notmuch use
for many months now).  The multipart and sig-verification patches were
cherry-picked to get around a couple of accidental spurious commits in
their originating branches that would have needed to be reverted.

Please test and provide feedback.  I would really like to see this
series merged into the mainline for the next release, if at all
possible.

jamie.

[0] git://finestructure.net/notmuch


pgppeOSNllxAM.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Folder search semantics (was Re: [RFC PATCH v2 0/8] Custom query parser, date search, folder search, and more)

2011-02-03 Thread Austin Clements
Quoth Carl Worth on Feb 02 at  2:48 pm:
 Restricting my reply to one tiny bit of your mail:
 
 You wrote:
  non-recursive is the only thing that makes sense for Maildir++ folders
 
 Either I'm not understanding Maildir++ folders, or I don't agree with
 you.
 
 I might have an email archive that looks like this:
 
   Maildir
 .work
   .project1
   .project2
   .etc...
 .family
   .dad
   .mom
   .brother
   .etc...
 
 With the above setup, what would be unreasonable about wanting to search
 for all work-related messages (across all projects, say) with a string
 like folder:work ?
 
 Now, a person might definitely want to search for messages in the
 .work folder directly, (not including the sub-folders), so we should
 provide support for users to get at that behavior as well, (such as a
 proposed folder:work$ or so).
 
 To me, both cases are perfectly legitimate, and I don't understand an
 argument that claims that only one makes sense. (Or again, I may be
 misunderstanding something.)

(Somebody with more first-hand Maildir++ experience should jump in here.
I stopped using Maildir++ a long time ago, so I may have no idea what
I'm talking about.)

Both cases are perfectly legitimate.

However, the issue with Maildir++ is that the inbox is stored in the
top-level directory:

  Maildir
cur
new
tmp
.work
.work.project1

As a consequence, all folders are subfolders of the inbox.  With
recursive search, a search for your inbox folder returns *all* of your
messages.  I wasn't trying to say that we shouldn't support recursive
search (I'm all for flexibility), but it's a confusing default for
Maildir++ because of this.

Maildir++ has the added twist that the inbox folder has no name.  As a
result, currently notmuch can't search for a Maildir++ inbox folder,
which needs to be addressed somehow.  The least surprising approach
would compatibility with the Maildir++ convention of calling the
top-level folder INBOX, the subfolder INBOX.work, etc.


Maildir++ issues aside, I submit that rooted, non-recursive folder
searches are a more natural default with a more conventional syntactic
extension to non-rooted/recursive searches.  In
id:87aaiy3u65@yoom.home.cworth.org, you mentioned that you
implemented non-rooted folder search to mimic subject search.  But file
system paths are not natural language like subject lines.  File system
paths are hierarchical and rooted.

Of course, special query operators like ^ and $ can mitigate this, but
these queries *aren't* regexps and, furthermore, people don't usually
apply regexps to file names.  They apply globs.  Glob syntax has the
added benefit of congruity with Xapian wildcard syntax.  This naturally
leads to a rooted, non-recursive syntax by default (like globs), where a
* at the end means recursive and a * at the beginning means non-rooted.
In fact, we could easily generalize this to arbitrary shell globs.


Here's a proposal that, I think, addresses Maildir++ inboxes and
subfolders; rooted, non-rooted, recursive, and non-recursive queries;
and then some.  Plus, it wouldn't require many code changes; you've
already done the hard work.

Switch XFOLDER from a probabilistic prefix with word-splitting to a
boolean prefix without word-splitting.  When indexing, strip off the cur
or new and examine the resulting directory name.  If it's the mail root,
this is a Maildir++ inbox, so add the term XFOLDERINBOX.  If it starts
with a dot, it's a Maildir++ subfolder, so add the term
XFOLDERINBOX.dirname.  Otherwise, add the term XFOLDERdirname.
Then, using a custom query transform for the folder: prefix, enumerate
XFOLDER terms and form a synonym query out of those that fnmatch the
user's folder query.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 4/4] ruby: Add generated files to gitignore

2011-02-03 Thread Ali Polatel
On Mon, 31 Jan 2011 23:48:57 +0100, Thomas Schwinge tho...@schwinge.name 
wrote:
 Hallo!
 
 On Mon, 10 Jan 2011 16:39:28 +0200, Ali Polatel a...@exherbo.org wrote:
  --- a/.gitignore
  +++ b/.gitignore
  @@ -13,3 +13,7 @@ libnotmuch.so*
   .*.swp
   *.elc
   releases
  +
  +bindings/ruby/mkmf.log
  +bindings/ruby/notmuch.so
  +bindings/ruby/Makefile
 
 These should rather be put into bindings/ruby/.gitignore, I'd say.

Good idea!
Just pushed ca69ce2397c0bb9b0e590a80b898ab4d58bb

 
 Grüße,
  Thomas

-- 
Regards,
Ali Polatel


pgpwScQVgJMNS.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: notmuch's idea of concurrency / failing an invocation

2011-02-03 Thread micah anderson
On Thu, 27 Jan 2011 12:35:16 -0800, Jameson Rollins 
jroll...@finestructure.net wrote:
 On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson mi...@riseup.net wrote:
  Due to my harddisk in my laptop being slow (5400RPM), my notmuch
  database growing, and perhaps some fragmentation somewhere, this has
  become *incredibly* annoying for me. I am checking email every 30
  minutes, and I'm nicing and ionicing the processes so I can use my
  machine, but while those processes are running, I'm effectively locked
  out of a good portion of my email. 
 
 I also have a very slow disk, but this is very rarely a problem for me.
 I retrieve mail every 10 minutes, and the corresponding notmuch new
 usually takes a minute or so.  I really haven't found it to be much of a
 bother to just wait it out.

Sometimes I can have several thousand messages to add/remove from the
database. I know this is probably unusal, but for me its not due to
system emails. I suppose if I checked my email more frequently, I'd have
less to process on each run, but thats more side-stepping the
concurrency issue.

micah


pgp62R9y86RGi.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Austin Clements
Nice catch.

Is there a reason you keep the remaining data in a string instead of
taking the more idiomatic elisp approach of leaving it in the process
buffer?  In fact, the code would probably be simpler if you
immediately appended the string to the process buffer like a normal
process-filter and then peeled things away using buffer-oriented
regexp functions like looking-at.  Elisp is a lot better at
manipulating buffers than it is at manipulating strings.

On Wed, Feb 2, 2011 at 6:56 PM, Thomas Schwinge tho...@schwinge.name wrote:
 This issue has been lying in ambush as of 2009-11-24's commit
 93af7b574598637c2766dd1f8ef343962c9a8efb.

 Signed-off-by: Thomas Schwinge tho...@schwinge.name
 ---
  emacs/notmuch.el |   70 +++--
  1 files changed, 41 insertions(+), 29 deletions(-)

 diff --git a/emacs/notmuch.el b/emacs/notmuch.el
 index 3d82f0d..35ccee6 100644
 --- a/emacs/notmuch.el
 +++ b/emacs/notmuch.el
 @@ -641,9 +641,6 @@ non-authors is found, assume that all of the authors 
 match.
     (propertize authors 'face 'notmuch-search-matching-authors)))

  (defun notmuch-search-insert-authors (format-string authors)
 -  ;; Save the match data to avoid interfering with
 -  ;; `notmuch-search-process-filter'.
 -  (save-match-data
     (let* ((formatted-authors (format format-string authors))
           (formatted-sample (format format-string ))
           (visible-string formatted-authors)
 @@ -709,7 +706,7 @@ non-authors is found, assume that all of the authors 
 match.
          (setq overlay (make-overlay start (point)))
          (overlay-put overlay 'invisible invis-spec)
          (overlay-put overlay 'isearch-open-invisible 
 #'notmuch-search-isearch-authors-show)))
 -      (insert padding
 +      (insert padding)))

  (defun notmuch-search-insert-field (field date count authors subject tags)
   (cond
 @@ -736,6 +733,10 @@ non-authors is found, assume that all of the authors 
 match.
          do (notmuch-search-insert-field field date count authors subject 
 tags)))
   (insert \n))

 +(defvar notmuch-search-process-filter-data nil
 +  Data that has not yet been processed.)
 +(make-variable-buffer-local 'notmuch-search-process-filter-data)
 +
  (defun notmuch-search-process-filter (proc string)
   Process and filter the output of \notmuch search\
   (let ((buffer (process-buffer proc))
 @@ -743,31 +744,41 @@ non-authors is found, assume that all of the authors 
 match.
     (if (buffer-live-p buffer)
        (with-current-buffer buffer
          (save-excursion
 -           (let ((line 0)
 -                 (more t)
 -                 (inhibit-read-only t))
 -             (while more
 -               (if (string-match ^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
 \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$ string line)
 -                   (let* ((thread-id (match-string 1 string))
 -                          (date (match-string 2 string))
 -                          (count (match-string 3 string))
 -                          (authors (match-string 4 string))
 -                          (subject (match-string 5 string))
 -                          (tags (match-string 6 string))
 -                          (tag-list (if tags (save-match-data (split-string 
 tags)
 -                     (goto-char (point-max))
 -                     (let ((beg (point-marker)))
 -                       (notmuch-search-show-result date count authors 
 subject tags)
 -                       (notmuch-search-color-line beg (point-marker) 
 tag-list)
 -                       (put-text-property beg (point-marker) 
 'notmuch-search-thread-id thread-id)
 -                       (put-text-property beg (point-marker) 
 'notmuch-search-authors authors)
 -                       (put-text-property beg (point-marker) 
 'notmuch-search-subject subject)
 -                       (if (string= thread-id notmuch-search-target-thread)
 -                           (progn
 -                             (set 'found-target beg)
 -                             (set 'notmuch-search-target-thread found
 -                     (set 'line (match-end 0)))
 -                 (set 'more nil)
 +           (let ((inhibit-read-only t)
 +                 ;; We may have a partial line saved from the last iteration.
 +                 (string (concat notmuch-search-process-filter-data string))
 +                 (start 0))
 +             (goto-char (point-max))
 +             ;; Split `string' into lines.
 +             (while (string-match \n string start)
 +               (let ((line (substring string start (match-beginning 0
 +                 ;; Save the beginning of the next line already here, so that
 +                 ;; we can mangle the match data later on.
 +                 (setq start (match-end 0))
 +                 (if (string-match
 +                      ^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) 
 \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$
 +              

Re: [PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2011-02-03 Thread Thomas Schwinge
Hallo!

On Thu, 3 Feb 2011 12:06:20 -0500, Austin Clements amdra...@mit.edu wrote:
 Is there a reason you keep the remaining data in a string instead of
 taking the more idiomatic elisp approach of leaving it in the process
 buffer?  In fact, the code would probably be simpler if you
 immediately appended the string to the process buffer like a normal
 process-filter and then peeled things away using buffer-oriented
 regexp functions like looking-at.  Elisp is a lot better at
 manipulating buffers than it is at manipulating strings.

Ha, I hear you -- this is what I meant to do originally.  But then, the
save-in-string approach (even though I always considered keeping state in
the string a bit ugly) seemed more simple to me.  As I said: writing
elisp code is not my primary profession...  :-) (Perhaps I should buy a
book about it, or something.)  Now that you confirmed my original idea,
I'll see about re-writing the code accordingly, so thanks for the input!


Grüße,
 Thomas


pgpkcG4W4yhJj.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: new crypto branch providing full PGP/MIME support

2011-02-03 Thread Daniel Kahn Gillmor
On 02/02/2011 08:18 PM, Jameson Rollins wrote:
 Hi, all.  I have pushed a new branch called crypto to my notmuch
 repository [0].  This branch provides full support for PGP/MIME signed
 and encrypted messages, including emacs UI support.

I have tested this, and am now using it.  I'm very happy with it.  I
support its inclusion in the mainline.

Thanks for doing this, Jamie.  This is excellent!

 less important stuff follows

I want to raise one (non-blocking) question about the decryption to see
if anyone has any suggestions:

If you do notmuch show --format=json on a PGP/MIME-encrypted plaintext
message, it emits the base message, which is structured like this:

1 └┬╴multipart/encrypted
2  ├╴application/pgp-encrypted attachment
3  └╴application/octet-stream inline [msg.asc]

with these patches, if you do notmuch show --format=json --decrypt, it
emits this instead:

1 └┬╴multipart/encrypted
2  └╴text/plain inline

and it attaches an encstatus (and possibly sigstatus, if the message was
signed) to part 1.  I'll call this method A.

There are other methods that could be used as well, and it's worth
making sure we've chosen one that we think is what we'll want in the
future.  here are two other proposals:

Method B:

1 └┬╴multipart/encrypted
2  ├╴application/pgp-encrypted attachment
3  └╴text/plain inline

That is, just replace part 3 (the actual encrypted body) with the
decrypted material.

This has the advantage that for single-part encrypted messages, the
structure and part numbers of the message remains the same as without
--decrypt.


Method C:

1 └╴text/plain inline

That is, replace the entire multipart/encrypted with the decrypted material.

This avoids having an explicitly-labeled multipart/encrypted wrapper
around cleartext (which might be considered odd).  It would mean
attaching the encstatus and sigstatus directly to the decrypted part,
though.



I don't actually see any of these methods as being significantly better
than the others -- i think they all have some inherent ugliness.  So i'm
fine with going with method A as Jamie chose it and has it working.  But
i wanted to see if anyone had strong arguments in favor of the other
methods (or if there are other --decrypt methods we could use, for that
matter)

--dkg



signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: new crypto branch providing full PGP/MIME support

2011-02-03 Thread Daniel Kahn Gillmor
Hi Micah--

just wanted to follow up on your points/questions:

On 02/03/2011 11:25 AM, micah anderson wrote:
 1. I personally think notmuch-show-process-pgpmime should default to
 true

note that with it set to false, you can still M-RET (instead of RET) on
an item in the summary window to have it set for that particular view.

 2. in-line pgp messages don't have any processing done on them. getting
 the mime-encoded processing work is a huge step and I'm happy that
 works, in-line can (and IMHO, should) come later

yeah, inline is a whole different thing, and much more difficult to
manage programmatically in the notmuch backend, i think.  I dealing with
inline signatures and encryption should be a job for the emacs (or vim
or whatever) frontend.

 3. i'm not sure expired/revoked keys are handled properly - tested on a
 message that was encrypted by a key that was revoked and got End of
 file during parsing

when you say encrypted by do you mean encrypted to?  do you have
access to the corresponding secret key?

 4. messages that I sent encrypted to someone are not also encrypted to
 myself, which means that a thread which contains my replies isn't able
 to decrypt my messages in that thread and results in a purple
 'decryption error'. Perhaps this is an emacs UI tweak that needs to be
 made to get messages also encrypted to my own key?

this is an issue for the emacs message modes (or maybe for your gpg
configuration), not for notmuch.

You either want to fix this in your emacs config by putting your
fingerprint into mml2015-signers and setting mml2015-encrypt-to-self

Or you want to set gpg's default-recipient-self option  (and
default-recipient option if you hold more than one secret key and want
to be sure it chooses the right one)

 5. unknown keys are represented in a long format,
 eg. '0x5585F58CC827A062' when most tools represent them just with their
 shortened keyid (in this case this one would be: 0xC827A062), is there a
 particular reason for this?

Short keyIDs are easily spoofable; believing anything based on a matched
short keyID is a mistake.  unknown keys themselves may or may not have
properly signed a message -- since we don't have the key handy, we don't
have a way of checking.

note that unknown key is different from good signature from known key
but we do not know who controls the key

 I recognize some people's keyids in the
 short form, but do not in the longform.

You can derive the short form from the long form by ignoring everything
but the last 8 hex characters.  But if you actually recognize the short
form, i'd expect you to have the relevant key in your keyring; is this
really a use case worth bothering with?

do you have a suggestion for how you think it should behave differently?

--dkg



signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: new crypto branch providing full PGP/MIME support

2011-02-03 Thread Darren McGuicken
On Wed, 02 Feb 2011 17:18:45 -0800, Jameson Rollins 
jroll...@finestructure.net wrote:
 Please test and provide feedback.  I would really like to see this
 series merged into the mainline for the next release, if at all
 possible.

Fan.  Tastic.  Thanks so much for this, incredible work!

Grabbed and testing now, the only slight niggle I see is that when I
reply to an encrypted mail I still get:

  On Thu, 03 Feb 2011 19:17:45 +, Someone foo@bar wrote:
  Non-text part: application/pgp-encrypted
  Non-text part: application/octet-stream

Can we pass the decrypted text to message mode on a reply?


pgpvaIMNrRYvK.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: new crypto branch providing full PGP/MIME support

2011-02-03 Thread Jameson Rollins
On Thu, 03 Feb 2011 20:42:38 +, Darren McGuicken 
mailing-notm...@fernseed.info wrote:
 On Wed, 02 Feb 2011 17:18:45 -0800, Jameson Rollins 
 jroll...@finestructure.net wrote:
  Please test and provide feedback.  I would really like to see this
  series merged into the mainline for the next release, if at all
  possible.
 
 Fan.  Tastic.  Thanks so much for this, incredible work!

Thanks!  I hope people find it useful.

 Grabbed and testing now, the only slight niggle I see is that when I
 reply to an encrypted mail I still get:
 
   On Thu, 03 Feb 2011 19:17:45 +, Someone foo@bar wrote:
   Non-text part: application/pgp-encrypted
   Non-text part: application/octet-stream
 
 Can we pass the decrypted text to message mode on a reply?

Yeah, some folks pointed this out on #notmuch this morning.  The
decryption is only happening in notmuch-show, but we need to do it in
notmuch-reply as well for the decrypted text to show up in the reply.
I'm working on it now.

jamie.



pgpU7r4oBBd3U.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


always encrypting messages to self [was: Re: new crypto branch providing full PGP/MIME support]

2011-02-03 Thread Daniel Kahn Gillmor
On 02/03/2011 03:34 PM, Jameson Rollins wrote:
 On Thu, 03 Feb 2011 14:52:01 -0500, Daniel Kahn Gillmor 
 d...@fifthhorseman.net wrote:
 You either want to fix this in your emacs config by putting your
 fingerprint into mml2015-signers and setting mml2015-encrypt-to-self

 Or you want to set gpg's default-recipient-self option  (and
 default-recipient option if you hold more than one secret key and want
 to be sure it chooses the right one)
 
 Actually, I think the gpg option we're looking for here is
 encrypt-to.  default-recipient-self sets the recipient only if none
 other is specified.  I just set encrypt-to mykeyid in my gpg.conf
 and it seems to do as expected (all encrypted messages are also
 encrypted to myself).

Yes, this is correct.  thanks for figuring that out, Jamie.

--dkg



signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


notmuch-show-get-filename and maildir location

2011-02-03 Thread micah

I recently upgraded to the newer notmuch to test out the new folder
patch (its great!) and the new decryption/signature verification (its
fantastic!), and noticed something odd start happening.

I think that notmuch moves a new message to the maildir cur directory,
From the new directory when you first read it. Is that right? This is
actually what I would expect with maildirs, but notmuch wasn't doing
this before.

The reason why I care is because if I open a message,
notmuch-show-get-filename tells me that the file is located in the
'new/' path, but it is actually not there at all, looking around it
appears to be in the 'cur/' directory. 

This is because I made a function to let me do bayes training by putting
mistaken emails in my Mistakes folder, so that they can be retrained:

(defun notmuch-mark-as-mistake ()
Moves the current message into the Mistakes folder
  (interactive)
(let* ((fullpath (notmuch-show-get-filename))
  (filename (file-name-nondirectory fullpath)))
  (save-excursion
(rename-file fullpath (concat /home/micah/Maildir/INBOX.Mistakes/new/ 
filename)))
(notmuch-show-archive-thread-then-exit)
(notmuch-search-refresh-view)
(hl-line-mode 1)(hl-line-mode 1)))

(define-key notmuch-show-mode-map (kbd S) 'notmuch-mark-as-mistake)

if I hit the 'S' key on a message now, it complains that it cannot
rename the file because it doesn't exist.

Incidentally, does anyone know how I can do this without having to
actually do a notmuch-show on the message? Most spam I can recognize by
the subject in my inbox, and I dont need to open it to see that.

micah


pgpHTT9EUb8zD.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch