[PATCH] How to improve the mail handling workflow?
On Friday 12 November 2010 16:23:58 Matthieu Lemerre wrote:
> - Processing mails which do not have any automatically added tag is
>boring, because I need to press several keys to archive them: "+" to
>add a tag, and then "a". If I forget about +, then my mail is
>impossible to find.
I feel the same !
I do not know if notmuch/xapian can find mail with 0 tag or only "attachment"
tag for exemple ?
The dirty solution I found is to create a virtual folder called "orphan" that
contains none of all the other tags (ie not tag:foo and not tag:bar and not
...)
I do not considere "attachment" as a real tag. I should now do the same for
"replied" and all other new stuff
"""
(require 'cl)
(defun tagreduce(a b)
(if (or (string= a "attachment") (string= b "attachment"))
(if (string= a "attachment")
b
a
)
(if (not (string-match "and not" a))
(concat "not tag:" a " and not tag:" b)
(concat a " and not tag:" b)
)
)
)
(defun orphan()
(reduce 'tagreduce (split-string (with-output-to-string
(with-current-buffer standard-output
(apply 'call-process notmuch-command nil
t nil
"search-tags" nil))) "\n+" t))
)
(setq pnotmuch-orphans (append '("orphan") (orphan)))
(setq notmuch-folders (append notmuch-folders (list pnotmuch-orphans)))
"""
(excuse my lisp ...)
--
C?dric
[PATCH] How to improve the mail handling workflow?
On Sat, 13 Nov 2010 17:43:38 +0100, C?dric Cabessa wrote: > On Friday 12 November 2010 16:23:58 Matthieu Lemerre wrote: > > - Processing mails which do not have any automatically added tag is > >boring, because I need to press several keys to archive them: "+" to > >add a tag, and then "a". If I forget about +, then my mail is > >impossible to find. > > I feel the same ! > I do not know if notmuch/xapian can find mail with 0 tag or only "attachment" > tag for exemple ? I think you guys may have a misunderstanding about how notmuch indexes mail. Notmuch indexes multiple headers (To, From, Subject, Date) and the *entire* body of the message. That's kind of the whole point. In other words, messages don't have to have tags in order to be found. For instance, the following two search return just this message (for me): to:notmuch subject:"how improve workflow" from:ced at ryick.net "dirty solution" And none of the messages in this thread are tagged for me at all (except the first in the thread which has a "replied" tag). So don't worry if not all of your messages are tagged. They can all still be found. That's the beauty of fast search. jamie. -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20101113/667e2b6a/attachment.pgp>
[PATCH] How to improve the mail handling workflow?
On Sat, 13 Nov 2010 07:05:58 +0100, Michal Sojka wrote: > I agree with you in that in many cases tags can be replaced by saved > searches. Last time I did it, i.e. used saved searches to distinguish > messages from different mailing lists, the result was that it took > very long time (something like 5 seconds) to show notmuch-hello Interesting, what Xapian backend are you using? I moved to chert after id:"87ocl1lut1.fsf at yoom.home.cworth.org" and my notmuch-hello with 15 saved searches appears in a couple of seconds when freshly loaded, faster when switching back to it after use. That may well be slower than just tag searches but it's not yet at a threshold where I notice it. How many searches had you saved? > Additionally, I compared the speed of command line searches for tags > and for the whole email addresses and even without the bug mentioned > above, the search for to: is usually slower than the search for tag:. Very non-scientifically just using time and vm/drop_caches on my netbook, having tagged all mail sent to the list address with 'notmuch', I seem to get much the same performance: $ time notmuch search tag:notmuch > /dev/null real0m21.074s user0m4.740s sys 0m1.916s $ time notmuch search to:notmuch > /dev/null real0m20.280s user0m4.600s sys 0m2.048s $ time notmuch search to:notmuch at notmuchmail.org > /dev/null real0m21.790s user0m5.044s sys 0m2.008s -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20101113/352f2591/attachment.pgp>
a proposed change to JSON output to report verification of PGP/MIME signatures.
Thanks for the reply, David! On 11/13/2010 06:40 AM, David Bremner wrote: > Are both the forward and backward pointers needed? Technically, only the "signs" pointer is needed, i guess. I had included the "signedby" pointer so that frontends which process the list linearly know that a signed part will be referred to later by one or more signatures. If you think that's not actually useful, i'd be happy to drop the "signedby" pointer. What do other people think? >> Would it make more sense to do deeper structural modifications of the >> json output (e.g. return the full MIME tree instead of a list of parts) >> than to go with the current proposal? > > Yeah, this occured to me too, especially since I think David Edmondson > has some changes in mind to support better handling of > multipart/alternative parts. Another related concern that occurred to me is that parts E and F in my example mail are technically part of the aggregate that is signed by I, but they apparently the end-user won't even know they exist (unless the frontend uses "notmuch show --format=mbox"). This seems problematic if the message headers of the included message have relevance to the content. e.g. an e-mail that says "here's the phishing attempt i received" (Received: would be relevant) or "Look at the nasty things this guy said!" (From:, Subject:, and Date: would be relevant at least). David Edmonson, i'd be interested in hearing your proposal for restructuring the output to see how it might interact with my pieces here. I also found it interesting to consider the range of possible non-leaf MIME types: https://secure.wikimedia.org/wikipedia/en/wiki/MIME#Multipart_messages http://www.iana.org/assignments/media-types/multipart/ --dkg -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20101113/e3142cc4/attachment.pgp>
Second draft of logging patches
On Sat, 13 Nov 2010 06:25:57 +0100, Michal Sojka wrote: > On Wed, 10 Nov 2010, David Bremner wrote: > > I had a thought of a possibly interesting application of the (yet to be > > written) log playback code. It could be use to implement a simple > > queuing system where commands are only logged but not actually run on > > the database. > Hi, I think this could be very interesting and that it could make the > tagging operation asynchronous and thus faster as was suggested by > Sebastian in id:"87k4l5whwe.fsf at SSpaeth.de". > > In this case, however, the queuing should happen in the client, not in > the library. Agreed. The current implementation of automagically logging things in the library would be undesirable here. That specific thing shouldn't be too hard to change. Also worth thinking about is (optionally) using the Maildir to log most tag changes; Maildirs are designed (imperfectly, but still) to support simulataneous writes. The database could then catch up by reading the Maildir. d
a proposed change to JSON output to report verification of PGP/MIME signatures.
On Sat, 13 Nov 2010 02:55:50 -0500, Daniel Kahn Gillmor wrote: > A signed MIME part will contain a new element "signedby", which is a > list of part numbers identifying signatures that cover this part. > > Signature parts (Content-Type: application/pgp-signature) will contain a > new element "signs", which points back to the list of parts this > signature covers. Are both the forward and backward pointers needed? > > Would it make more sense to do deeper structural modifications of the > json output (e.g. return the full MIME tree instead of a list of parts) > than to go with the current proposal? > Yeah, this occured to me too, especially since I think David Edmondson has some changes in mind to support better handling of multipart/alternative parts. d
[PATCH] How to improve the mail handling workflow?
On Fri, 12 Nov 2010, Darren McGuicken wrote: > On Fri, 12 Nov 2010 16:23:58 +0100, Matthieu Lemerre wrote: > > Here is first a patch that copes with this last point. Whenever you > > want to archive a thread, it finds whether you forgot to add a custom > > "user" tag to a message, and if so asks you for a tag to add before > > archiving. That way, I no longer have messages without any tags. > > Hmm, this would be very irritating in my own workflow in which I really > only use a small number of tags on a fraction of my total mail archive > to differentiate mail type or content which can't otherwise be > determined from the indexed plain text of the message (I don't like to > add a 'notmuch' tag to mail from the list for instance since a saved > search for mail sent to the list address does exactly the same thing). I agree with you in that in many cases tags can be replaced by saved searches. Last time I did it, i.e. used saved searches to distinguish messages from different mailing lists, the result was that it took very long time (something like 5 seconds) to show notmuch-hello screen. There is a bug in notmuch hello, as I described in id:"87wrtz3pw2.fsf at steelpick.2x.cz". Additionally, I compared the speed of command line searches for tags and for the whole email addresses and even without the bug mentioned above, the search for to: is usually slower than the search for tag:. And finally, there are some mailing lists, which changed their address in the past or have multiple aliases and in this case you always has to use a more complex search to match all messages to that mailing list and these searches are even more slow than tag searches. -Michal
Second draft of logging patches
On Wed, 10 Nov 2010, David Bremner wrote: > On Sun, 24 Oct 2010 18:01:02 -0300, david at tethera.net wrote: > > Here is my second try at logging, taking into account the feedback I > > got from Rob and Michal. There is definitely some tidying to do; in > > particular I know the protoypes in public headers need > > documentation. Also, I should add a configuration option to > > enable configuration by command or something like that. > > I had a thought of a possibly interesting application of the (yet to be > written) log playback code. It could be use to implement a simple > queuing system where commands are only logged but not actually run on > the database. I'm not sure about the performance implications, but it > could be interesting because it eliminates the need to have a server > running in order to eliminate write contention for the tag database. > The "queue runner" could be as simple as a cron job, or it could be > something spawned by one of the queue operations; the point would be > that queueing could continue while the snapshot of the queue was run. Hi, I think this could be very interesting and that it could make the tagging operation asynchronous and thus faster as was suggested by Sebastian in id:"87k4l5whwe.fsf at SSpaeth.de". In this case, however, the queuing should happen in the client, not in the library. -Michal
a proposed change to JSON output to report verification of PGP/MIME signatures.
ld be responsible for resolution of key->userid bindings, but i'm OK punting on that question for the moment. Multipart messages can have some parts signed and other parts not signed: think of mailing lists which tack on a footer to each relayed mail; the footer isn't signed, though the rest of the message is. One MIME signature can cover more than one MIME part: Think of a signed e-mail with an attachment. In this case, the signature is actually over the aggregate, not the individual parts. For example, a signed two-part message that says: [ (A) "this is the budget for 2011", and (B) an attached spreadsheet ] is *not* the same as either (A) or (B) signed independently. A multipart MIME message can contain more than one distinct signature on different parts: Think of a digest of a mailing list discussion between several participants who each sign their own messages. Each signature needs to be bound to the relevant parts (and vice versa); and some signatures within a message can fail while others succeed. A single application/pgp-signature part could contain signing material from multiple signers. Think of a PGP/MIME-signed key transition document. MIME is actually a tree structure, and any subtree can be signed. But currently, "notmuch show" hides the tree structure and produces what appears to be a linear set of parts. Even more perversely, the tree structure means that a single MIME part could potentially be signed by multiple signatures, each of which potentially has independent origin and independent validity. I've attached a moderately nasty e-mail message to this one demonstrating a confluence of a bunch of these observations. The structure of the attached e-mail looks like this: A???multipart/signed 10936 bytes B ???multipart/mixed 7403 bytes C ???text/plain 77 bytes D ???image/jpeg attachment [dkg.jpg] 4753 bytes E message/rfc822 2072 bytes F ? ???multipart/signed 1914 bytes G ? ??text/plain 57 bytes H ? ??application/pgp-signature attachment [signature.asc] 900 bytes I ??application/pgp-signature attachment [signature.asc] 900 bytes "notmuch show" emits it as 5 parts (omitting A, B, E, and F): 1: C 2: D 3: G 4: H 5: I Note that while C and D are both signed by I, G is actually signed by both H and I. yuck. And since this example message is attached to the e-mail i'm writing right now (which itself will be signed) it can certainly get even yuckier. Questions: -- Am i missing any data or relationships you think we might want? Is anything broken, unexpected, or dangerous about the choice of JSON modifications? I realize i've gone down a bit of a rabbit hole in the corner cases here (driven mainly by my observations section). Are there any simplifying assumptions we can safely make about what kinds of messages are worth verifying? That is, are there ways to make this more intelligible that don't throw away our ability to accurately represent the verified state of some non-trivial subset of messages? If this method (or something similar to it) gets put into the notmuch backend, is this something we can actually represent to a human with a reasonable frontend? Would it make more sense to do deeper structural modifications of the json output (e.g. return the full MIME tree instead of a list of parts) than to go with the current proposal? It would be nice to also make this kind of reporting structure also work for S/MIME and maybe other crypto-signature structures like DKIM. Is that doable within this framework? are there other tweaks we might want to consider to cover that possibility? If you actually read this far, you are a champion! I look forward to any feedback you have. OK, off to bed! --dkg -- next part -- An embedded message was scrubbed... From: Daniel Kahn Gillmor <d...@fifthhorseman.net> Subject: Fwd: a test signature Date: Sat, 13 Nov 2010 00:00:29 -0500 Size: 10973 URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20101113/0bed0a25/attachment-0001.eml> -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20101113/0bed0a25/attachment-0001.pgp>
Re: Second draft of logging patches
On Sat, 13 Nov 2010 06:25:57 +0100, Michal Sojka sojk...@fel.cvut.cz wrote: On Wed, 10 Nov 2010, David Bremner wrote: I had a thought of a possibly interesting application of the (yet to be written) log playback code. It could be use to implement a simple queuing system where commands are only logged but not actually run on the database. Hi, I think this could be very interesting and that it could make the tagging operation asynchronous and thus faster as was suggested by Sebastian in id:87k4l5whwe@sspaeth.de. In this case, however, the queuing should happen in the client, not in the library. Agreed. The current implementation of automagically logging things in the library would be undesirable here. That specific thing shouldn't be too hard to change. Also worth thinking about is (optionally) using the Maildir to log most tag changes; Maildirs are designed (imperfectly, but still) to support simulataneous writes. The database could then catch up by reading the Maildir. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: a proposed change to JSON output to report verification of PGP/MIME signatures.
Thanks for the reply, David! On 11/13/2010 06:40 AM, David Bremner wrote: Are both the forward and backward pointers needed? Technically, only the signs pointer is needed, i guess. I had included the signedby pointer so that frontends which process the list linearly know that a signed part will be referred to later by one or more signatures. If you think that's not actually useful, i'd be happy to drop the signedby pointer. What do other people think? Would it make more sense to do deeper structural modifications of the json output (e.g. return the full MIME tree instead of a list of parts) than to go with the current proposal? Yeah, this occured to me too, especially since I think David Edmondson has some changes in mind to support better handling of multipart/alternative parts. Another related concern that occurred to me is that parts E and F in my example mail are technically part of the aggregate that is signed by I, but they apparently the end-user won't even know they exist (unless the frontend uses notmuch show --format=mbox). This seems problematic if the message headers of the included message have relevance to the content. e.g. an e-mail that says here's the phishing attempt i received (Received: would be relevant) or Look at the nasty things this guy said! (From:, Subject:, and Date: would be relevant at least). David Edmonson, i'd be interested in hearing your proposal for restructuring the output to see how it might interact with my pieces here. I also found it interesting to consider the range of possible non-leaf MIME types: https://secure.wikimedia.org/wikipedia/en/wiki/MIME#Multipart_messages http://www.iana.org/assignments/media-types/multipart/ --dkg signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] How to improve the mail handling workflow?
On Friday 12 November 2010 16:23:58 Matthieu Lemerre wrote: - Processing mails which do not have any automatically added tag is boring, because I need to press several keys to archive them: + to add a tag, and then a. If I forget about +, then my mail is impossible to find. I feel the same ! I do not know if notmuch/xapian can find mail with 0 tag or only attachment tag for exemple ? The dirty solution I found is to create a virtual folder called orphan that contains none of all the other tags (ie not tag:foo and not tag:bar and not ...) I do not considere attachment as a real tag. I should now do the same for replied and all other new stuff (require 'cl) (defun tagreduce(a b) (if (or (string= a attachment) (string= b attachment)) (if (string= a attachment) b a ) (if (not (string-match and not a)) (concat not tag: a and not tag: b) (concat a and not tag: b) ) ) ) (defun orphan() (reduce 'tagreduce (split-string (with-output-to-string (with-current-buffer standard-output (apply 'call-process notmuch-command nil t nil search-tags nil))) \n+ t)) ) (setq pnotmuch-orphans (append '(orphan) (orphan))) (setq notmuch-folders (append notmuch-folders (list pnotmuch-orphans))) (excuse my lisp ...) -- Cédric ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] How to improve the mail handling workflow?
On Sat, 13 Nov 2010 17:43:38 +0100, Cédric Cabessa c...@ryick.net wrote: On Friday 12 November 2010 16:23:58 Matthieu Lemerre wrote: - Processing mails which do not have any automatically added tag is boring, because I need to press several keys to archive them: + to add a tag, and then a. If I forget about +, then my mail is impossible to find. I feel the same ! I do not know if notmuch/xapian can find mail with 0 tag or only attachment tag for exemple ? I think you guys may have a misunderstanding about how notmuch indexes mail. Notmuch indexes multiple headers (To, From, Subject, Date) and the *entire* body of the message. That's kind of the whole point. In other words, messages don't have to have tags in order to be found. For instance, the following two search return just this message (for me): to:notmuch subject:how improve workflow from:c...@ryick.net dirty solution And none of the messages in this thread are tagged for me at all (except the first in the thread which has a replied tag). So don't worry if not all of your messages are tagged. They can all still be found. That's the beauty of fast search. jamie. pgpadIQDCK8Il.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
