[notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Mike Hommey <mh+notmuch at glandium.org> [2010.01.08.2220 +1300]:
> FYI, I have a good experience writing fuse filesystems, both with
> high-level and low-level APIs. I'd avise to use the low-level API,
> which allows for better performance.

I don't have any experience with FUSE yet, but the examples in
/usr/share/doc/libfuse-dev/examples/ look trivial. This is where
I would start, one function at a time. If you have a better
suggestion, I'd love to hear it; or to clone your repo! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

be careful of reading health books -- you might die of a misprint.
 -- mark twain

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/abdd/attachment.pgp>

[notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Ruben Pollan  [2010.01.08.2221 +1300]:
> I think that would be security hole. You should not store the
> encrypted messages on a decrypted database. A solution whould be
> to encrypt as well the xapian DB, but I think is too complex for
> the use.

As I said in <20100108091216.GC735 at lapse.rw.madduck.net>, I think it
should be optionally possible for those that are encrypting the
xapian DB in other ways.

> You should be still able, with the actual notmuch, to search over
> the headers of your encrypted messages, or any other non-encrypted
> part of the message. Is not like that?

Most of the time, I search headers, but I do search bodies
regularly. So no, that would not be enough, at least not with the
ideal solution. And notmuch comes close to ideal already! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

infinite loop: see 'loop, infinite'.
loop, infinite: see 'infinite loop'.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/6dd9e5de/attachment.pgp>

[notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Mike Hommey <mh+notmuch at glandium.org> [2010.01.08.2109 +1300]:
> That may leak decrypted form in the xapian index, though in
> a split manner. But that'd still be a problem IMHO.

Not for me, since the index is stored on encrypted media. Thus, this
should be off-by-default, but possible.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"academia is really just a way to help those with high volumes of
 nothing to say to social status."
 -- myself on #debian-devel, 01 Feb 2007

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/ce7c1bf1/attachment.pgp>

[notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Mike Hommey <mh+notmuch at glandium.org> [2010.01.08.2106 +1300]:
> I'm in \o_ (though I won't be in Wellington). I've been thinking
> about a fuse filesystem on top of notmuch for a while.

Grand news to see you interested! A FUSE filesystem is <25 functions
to implement, and each function is basically an entity of its own
and thus highly parallisable. Once we agreed on a general mapping
between filesystem i/o and notmuch interaction, 25 of us can write
a function each and be done. How's that for collaboration? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"courage is not the absence of fear, but the decision
 that something else is more important than fear."
  -- ambrose redmoon

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/94826290/attachment-0001.pgp>

[notmuch] Threading

[martin f krafft]

also sprach Carl Worth  [2009.12.11.0639 +1300]:
> On Wed, 9 Dec 2009 16:21:34 -0700, Mark Anderson  
> wrote:
> > I was wondering if there's a way in notmuch to group un-associated
> > threads into a single thread.
> 
> There's certainly nothing like that in notmuch currently.
> 
> Sup had user-level functionality in the interface for stitching
> messages into a single thread, and I definitely think that that
> doesn't make any sense.

Why doesn't it make sense? Mutt does it too, and stitching means
actually (re)writing In-Reply-To and References headers.

I think this is one of the most useful "productivity features" in
mutt.

I also think that threading is a preference thing. As Carl said in
a later message:

> Just this morning I sent a mail to the notmuch list, which was
> a reply, (and legitimately so), but also potentially of interest
> to everyone on the list, (since it was regarding a bug fix
> unrelated to the original topic of the thread I was replying to).
> 
> So I was stuck on whether I should break the thread or not, (at
> the sending end). I guess I could have just sent a quick "this is
> pushed" reply, and independently composed a separate message
> telling people about the fix.
> 
> I ended up keeping the threading intact in that case, (which
> I think is right).

I often thread forwarded messages (and their followups) with the
thread because all my information management currently is
thread-oriented.

I think being able to freely break and tie threads in a trivial way
is a definite plus!

> But I still have a hard time justifying user operations to
> manipulate threading. The whole point of threading is to make it
> faster to process and read messages. But manual operations like
> joining and splitting threads seem like the user just doing more
> work, and that *after* having read the messages. So that seems
> mostly backwards to me.

Reading is one thing. Information storage and organisation is
another. After a message is delivered (and read) to my mailbox, it's
really mine and I can (and should be able) to affix it and integrate
it into my organisational scheme any way I want, don't you think?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"if there's anything more important than my ego,
 i want it caught and shot now."
-- zaphod beeblebrox

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/e8d5d845/attachment.pgp>

[notmuch] notmuch and imap [musing, no code :)]

[martin f krafft]

also sprach David Bremner  [2009.12.17.0218 +1300]:
> I agree that the labels-in-headers approach has some nice
> advantages.  I haven't thought through merging of tag lists, but
> maybe that is no worse than other approaches.  One thing that
> worries me a bit is that notmuch updates tags often, and each of
> these updates would require rewriting the message, at least in the
> obvious implementation.

If we separated implicit and explicit tags, then that issue would
not exist, and from all I can tell, only the privacy issues, and the
risk of losing mail when files are rewritten persist.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

DISCLAIMER: this entire message is privileged communication, intended
for the sole use of its recipients only. If you read it even though
you know you aren't supposed to, you're a poopy-head.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/827e9650/attachment.pgp>

[notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Carl Worth  [2009.12.08.2001 +1300]:
> One concept in notmuch (compared to sup) is to (eventually) avoid people
> having to go through that pain by their current mail client becoming
> "notmuch enabled". For me, I had always liked composing email in emacs,
> so I just have to add notmuch support to the existing emacs
> message-mode.
> 
> Hopefully people working on other email interfaces will do similar
> things, (would be great to have Anjal and Thunderbird get some notmuch
> support for example).
> 
> I definitely didn't like that with sup, to get all the global-search and
> tagging features one had to accept the curses UI as well.

I am a bit late to the party, but re: this thread [0], I would
suggest to go the way of a fuse filesystem. That's effectively
a daemon, but one which also bridges a chasm between notmuch and all
kinds of existing mail tools, including IMAP servers, by way of the
standard filesystem interface.

0. http://notmuchmail.org/pipermail/notmuch/2009/000782.html

I don't want to harp on this too much right now for I have not yet
fully understood notmuch, but the basic idea would be that you'd
have ~/mail provided by notmuch-fuse-daemon, and there'd be a tool
like notmuch-fuse-cli with which you can add virtual folders, e.g.

  notmuch-fuse-cli new debianmail 'from:debian OR to:debian'

and that would create ~/mail/debianmail with mode 555 (since you
cannot write the results of a search) containing a Maildir with all
messages matching the query.

The benefit of this would be that I could use mutt, evolution, or an
IMAP server, or vi and shell tools to manipulate my mail without any
modifications to those tools.

There could be a separate hierarchy for tags, e.g. ~/mail/TAGS/foo
and ~/mail/TAGS/bar/baz matching on explicit tags (and maybe
~/mail/TAGS/notmuch with mode 555 for implicit tags). Writing mail
to those directories effectively adds tags, unlinking removes them.
~/mail/TAGS/UNTAGGED holds untagged mail for easier reference.

In addition to all of this, fuse could be used to index new messages
directly as they are delivered into ~/mail, rather than running
'notmuch new' regularly.

These ideas are not new, and I've written about them before:

http://madduck.net/blog/2007.07.24:a-user-space-filesystem-for-mail-labeling/

notmuch seems an excellent base for implementing such a filesystem.
I will try to make time before LCA to get up to speed on fuse, then
maybe Carl and Micah and I (and whoever else will be in Wellington)
can hack this up in a few hours and over a few beers.

If this resonates, or you want to work on this too, let's hear from
you!

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"no problem is so formidable
 that you can't just walk away from it."
  -- c. schulz

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/1bca16dc/attachment.pgp>

[notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Jameson Graef Rollins  
[2009.11.26.1901 +1300]:
> I would really like to start using notmuch with emacs beyond just
> testing, but I really need to be able to handle/read/send mail with
> PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> handle this?  Is there a separate emacs mode that people use for
> signing/verifying/{de,en}crypting mail buffers, or is this something
> that is going to have to be integrated into the notmuch mode?  I guess
> the notmuch-show mode at least will need to do some verifying and
> decrypting.

How about indexing GPG-encrypted messages?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"a scientist once wrote that all truth passes through three stages:
 first it is ridiculed, then violently opposed and eventually,
 accepted as self-evident."
   -- schopenhauer

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/a1faf36e/attachment.pgp>

[notmuch] [PATCH] Use libgcrypt for hashing.

[micah anderson]

On Fri, 27 Nov 2009 22:22:01 -0800, Carl Worth  wrote:
> On Fri, 27 Nov 2009 21:28:03 -0600, "Jeffrey C. Ollie"  
> wrote:
> > Instead of including a private implementation of the SHA1 hash, use
> > libgcrypt.  This means less code of our own to maintain and it will be
> > easier to switch to a different hash function like SHA256.
> 
> I don't believe we have a significant code-maintenance burden with
> libsha1.c. And as for different hash functions, the only use of sha-1 in
> notmuch is as a fallback in the case of a message not including a
> Message-ID header.
> 
> So I don't see it as important at all to try to remove this code.

Its good that this is not a burden to maintain for the notmuch project,
even better that Mikhail, the libsha1 maintainer, is currently active in
this project and has volunteered to maintain the in-tree copy. 

However, the problem that has been raised is about the code-maintenance
burden that distributions face. In fact, this is not an unique problem
to notmuch, if it was it wouldn't be such a big deal. The reality is
that the more projects which cargo-cult around 'convenience copies' of
code, the more of a burden is placed on the distributors.

In some ways, the notmuch project and the role of distributors are at
cross-purposes on this issue, each side has an argument that makes sense

[notmuch] indexing encrypted messages (was: OpenPGP support)

[micah anderson]

On Fri, 8 Jan 2010 10:21:21 +0100, Ruben Pollan  
wrote:
> On 15:56, Fri 08 Jan 10, martin f krafft wrote:
> > How about indexing GPG-encrypted messages?
> 
> I think that would be security hole. You should not store the
> encrypted messages on a decrypted database. A solution whould be to
> encrypt as well the xapian DB, but I think is too complex for the use.

Would you consider it a security hole if you stored your database on
encrypted media (such as on-disk block encryption)?

I know that sup does this, when it ran over my mail store, it would
trigger my gpg agent so that it could decrypt the encrypted
messages. This was annoying because this happened every time it ran,
which meant that unless I had used gpg recently, my agent would pop up
and ask me for my passphrase, which was often.

The way Mutt provides this functionality is by decrypting only when you
perform the search itself.

micah
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/5d884468/attachment.pgp>

[notmuch] Streaming data from process into buffer

[Ben Gamari]

Hey all,

I have recently been working on bringing Carl Worth's excellent
notmuch[1] mail indexing application into a state where I can rely on it
for everyday use. While most of the Intel folks developing notmuch use
emacs for both development and using notmuch, I prefer to avoid carpal
tunnel whenever possible and thus use vim. While there exists an
excellent notmuch frontend for vim, it suffers from the incredibly
annoying limitation of being unable to asynchronously stream data from
its notmuch subprocess. This can result in extremely long periods of
hanging during large searches.

Looking at the emacs frontend, it seems that emacs provides an excellent
subprocess interface, where one can supply a callback to be called when
data is available from the process. This interface, amazingly enough,
resembles an interface proposed on this list only a few hours ago.[2]

Has anyone examined what would be necessary to implement such an
interface in vim? Has anyone perhaps started work on such an interface?
Would this be a reasonable task for an experienced programmer unfamiliar
with the vi codebase to take on? I'll take a look at the code later
today.

I hope things are going well.

Cheers,

- Ben


[1] http://www.notmuch-mail.org/
[2] http://article.gmane.org/gmane.editors.vim.devel/25108

[notmuch] Streaming data from process into buffer

[Ben Gamari]

Hey all,

I have recently been working on bringing Carl Worth's excellent
notmuch[1] mail indexing application into a state where I can rely on it
for everyday use. While most of the Intel folks developing notmuch use
emacs for both development and using notmuch, I prefer to avoid carpal
tunnel whenever possible and thus use vim. While there exists an
excellent notmuch frontend for vim, it suffers from the incredibly
annoying limitation of being unable to asynchronously stream data from
its notmuch subprocess. This can result in extremely long periods of
hanging during large searches.

Looking at the emacs frontend, it seems that emacs provides an excellent
subprocess interface, where one can supply a callback to be called when
data is available from the process. This interface, amazingly enough,
resembles an interface proposed on this list only a few hours ago.[2]

Has anyone examined what would be necessary to implement such an
interface in vim? Has anyone perhaps started work on such an interface?
Would this be a reasonable task for an experienced programmer unfamiliar
with the vi codebase to take on? I'll take a look at the code later
today.

I hope things are going well.

Cheers,

- Ben


[1] http://www.notmuch-mail.org/
[2] http://article.gmane.org/gmane.editors.vim.devel/25108

[notmuch] indexing encrypted messages (was: OpenPGP support)

[James Westby]

On Fri, 8 Jan 2010 15:56:10 +1300, martin f krafft  
wrote:
> also sprach Jameson Graef Rollins  
> [2009.11.26.1901 +1300]:
> > I would really like to start using notmuch with emacs beyond just
> > testing, but I really need to be able to handle/read/send mail with
> > PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> > handle this?  Is there a separate emacs mode that people use for
> > signing/verifying/{de,en}crypting mail buffers, or is this something
> > that is going to have to be integrated into the notmuch mode?  I guess
> > the notmuch-show mode at least will need to do some verifying and
> > decrypting.
> 
> How about indexing GPG-encrypted messages?

I think the difficulty will be interactivity. If notmuch-new can
potentially block watiting for a passphrase then it's not going to be
much use for non-interactive use, and whether someone can respond to a
GPG prompt is harder to determine that isatty().

Configuration may be a possible way around that, but looking at other
things such as opportunistic indexing could be good. For instance,
it could be the job of the UIs to decrypt content, and there could be a
nomuch function which takes a message id and decrypted content and
indexes it in to the DB. That means it's under the UI's control, where
the decryption UI should be, gets you indexing of encrypted content.

That would leave an open question over whether future notmuch show
invocations would return the plaintext or ciphertext. If it is the
latter then it requires decrypting every time you want to view it, but
it does mean that there is less information leakage (you could find out
whether an encrypted message contained a particular term, but not read
the whole message directly).

Thanks,

James

[notmuch] indexing encrypted messages (was: OpenPGP support)

[Ruben Pollan]

On 15:56, Fri 08 Jan 10, martin f krafft wrote:
> How about indexing GPG-encrypted messages?

I think that would be security hole. You should not store the encrypted messages
on a decrypted database. A solution whould be to encrypt as well the xapian DB,
but I think is too complex for the use.

You should be still able, with the actual notmuch, to search over the headers 
of 
your encrypted messages, or any other non-encrypted part of the message. Is not
like that?

-- 
Rub?n Poll?n  | jabber:meskio at jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 Cuando los que mandan pierden la verg?enza,
los que obedecen pierden el respeto.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100108/0b5ad2c7/attachment.pgp>

[notmuch] Quick thoughts on a notmuch daemon

[Mike Hommey]

On Fri, Jan 08, 2010 at 10:03:21PM +1300, martin f krafft wrote:
> also sprach Mike Hommey  [2010.01.08.2106 +1300]:
> > I'm in \o_ (though I won't be in Wellington). I've been thinking
> > about a fuse filesystem on top of notmuch for a while.
> 
> Grand news to see you interested! A FUSE filesystem is <25 functions
> to implement, and each function is basically an entity of its own
> and thus highly parallisable. Once we agreed on a general mapping
> between filesystem i/o and notmuch interaction, 25 of us can write
> a function each and be done. How's that for collaboration? ;)

FYI, I have a good experience writing fuse filesystems, both with
high-level and low-level APIs. I'd avise to use the low-level API, which
allows for better performance.

Mike

[notmuch] indexing encrypted messages (was: OpenPGP support)

[Mike Hommey]

On Fri, Jan 08, 2010 at 03:56:10PM +1300, martin f krafft wrote:
> also sprach Jameson Graef Rollins  
> [2009.11.26.1901 +1300]:
> > I would really like to start using notmuch with emacs beyond just
> > testing, but I really need to be able to handle/read/send mail with
> > PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> > handle this?  Is there a separate emacs mode that people use for
> > signing/verifying/{de,en}crypting mail buffers, or is this something
> > that is going to have to be integrated into the notmuch mode?  I guess
> > the notmuch-show mode at least will need to do some verifying and
> > decrypting.
> 
> How about indexing GPG-encrypted messages?

That may leak decrypted form in the xapian index, though in a split
manner. But that'd still be a problem IMHO.

Mike

[notmuch] Quick thoughts on a notmuch daemon

[Mike Hommey]

On Fri, Jan 08, 2010 at 03:56:20PM +1300, martin f krafft wrote:
> These ideas are not new, and I've written about them before:
> 
> http://madduck.net/blog/2007.07.24:a-user-space-filesystem-for-mail-labeling/
> 
> notmuch seems an excellent base for implementing such a filesystem.
> I will try to make time before LCA to get up to speed on fuse, then
> maybe Carl and Micah and I (and whoever else will be in Wellington)
> can hack this up in a few hours and over a few beers.
> 
> If this resonates, or you want to work on this too, let's hear from
> you!

I'm in \o_ (though I won't be in Wellington). I've been thinking about a
fuse filesystem on top of notmuch for a while.

Mike