[notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Jameson Graef Rollins  
[2009.11.26.1901 +1300]:
> I would really like to start using notmuch with emacs beyond just
> testing, but I really need to be able to handle/read/send mail with
> PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> handle this?  Is there a separate emacs mode that people use for
> signing/verifying/{de,en}crypting mail buffers, or is this something
> that is going to have to be integrated into the notmuch mode?  I guess
> the notmuch-show mode at least will need to do some verifying and
> decrypting.

How about indexing GPG-encrypted messages?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"a scientist once wrote that all truth passes through three stages:
 first it is ridiculed, then violently opposed and eventually,
 accepted as self-evident."
   -- schopenhauer

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Carl Worth  [2009.12.08.2001 +1300]:
> One concept in notmuch (compared to sup) is to (eventually) avoid people
> having to go through that pain by their current mail client becoming
> "notmuch enabled". For me, I had always liked composing email in emacs,
> so I just have to add notmuch support to the existing emacs
> message-mode.
> 
> Hopefully people working on other email interfaces will do similar
> things, (would be great to have Anjal and Thunderbird get some notmuch
> support for example).
> 
> I definitely didn't like that with sup, to get all the global-search and
> tagging features one had to accept the curses UI as well.

I am a bit late to the party, but re: this thread [0], I would
suggest to go the way of a fuse filesystem. That's effectively
a daemon, but one which also bridges a chasm between notmuch and all
kinds of existing mail tools, including IMAP servers, by way of the
standard filesystem interface.

0. http://notmuchmail.org/pipermail/notmuch/2009/000782.html

I don't want to harp on this too much right now for I have not yet
fully understood notmuch, but the basic idea would be that you'd
have ~/mail provided by notmuch-fuse-daemon, and there'd be a tool
like notmuch-fuse-cli with which you can add virtual folders, e.g.

  notmuch-fuse-cli new debianmail 'from:debian OR to:debian'

and that would create ~/mail/debianmail with mode 555 (since you
cannot write the results of a search) containing a Maildir with all
messages matching the query.

The benefit of this would be that I could use mutt, evolution, or an
IMAP server, or vi and shell tools to manipulate my mail without any
modifications to those tools.

There could be a separate hierarchy for tags, e.g. ~/mail/TAGS/foo
and ~/mail/TAGS/bar/baz matching on explicit tags (and maybe
~/mail/TAGS/notmuch with mode 555 for implicit tags). Writing mail
to those directories effectively adds tags, unlinking removes them.
~/mail/TAGS/UNTAGGED holds untagged mail for easier reference.

In addition to all of this, fuse could be used to index new messages
directly as they are delivered into ~/mail, rather than running
'notmuch new' regularly.

These ideas are not new, and I've written about them before:

http://madduck.net/blog/2007.07.24:a-user-space-filesystem-for-mail-labeling/

notmuch seems an excellent base for implementing such a filesystem.
I will try to make time before LCA to get up to speed on fuse, then
maybe Carl and Micah and I (and whoever else will be in Wellington)
can hack this up in a few hours and over a few beers.

If this resonates, or you want to work on this too, let's hear from
you!

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"no problem is so formidable
 that you can't just walk away from it."
  -- c. schulz

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] notmuch and imap [musing, no code :)]

[martin f krafft]

also sprach David Bremner  [2009.12.17.0218 +1300]:
> I agree that the labels-in-headers approach has some nice
> advantages.  I haven't thought through merging of tag lists, but
> maybe that is no worse than other approaches.  One thing that
> worries me a bit is that notmuch updates tags often, and each of
> these updates would require rewriting the message, at least in the
> obvious implementation.

If we separated implicit and explicit tags, then that issue would
not exist, and from all I can tell, only the privacy issues, and the
risk of losing mail when files are rewritten persist.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

DISCLAIMER: this entire message is privileged communication, intended
for the sole use of its recipients only. If you read it even though
you know you aren't supposed to, you're a poopy-head.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Threading

[martin f krafft]

also sprach Carl Worth  [2009.12.11.0639 +1300]:
> On Wed, 9 Dec 2009 16:21:34 -0700, Mark Anderson  
> wrote:
> > I was wondering if there's a way in notmuch to group un-associated
> > threads into a single thread.
> 
> There's certainly nothing like that in notmuch currently.
> 
> Sup had user-level functionality in the interface for stitching
> messages into a single thread, and I definitely think that that
> doesn't make any sense.

Why doesn't it make sense? Mutt does it too, and stitching means
actually (re)writing In-Reply-To and References headers.

I think this is one of the most useful "productivity features" in
mutt.

I also think that threading is a preference thing. As Carl said in
a later message:

> Just this morning I sent a mail to the notmuch list, which was
> a reply, (and legitimately so), but also potentially of interest
> to everyone on the list, (since it was regarding a bug fix
> unrelated to the original topic of the thread I was replying to).
> 
> So I was stuck on whether I should break the thread or not, (at
> the sending end). I guess I could have just sent a quick "this is
> pushed" reply, and independently composed a separate message
> telling people about the fix.
> 
> I ended up keeping the threading intact in that case, (which
> I think is right).

I often thread forwarded messages (and their followups) with the
thread because all my information management currently is
thread-oriented.

I think being able to freely break and tie threads in a trivial way
is a definite plus!

> But I still have a hard time justifying user operations to
> manipulate threading. The whole point of threading is to make it
> faster to process and read messages. But manual operations like
> joining and splitting threads seem like the user just doing more
> work, and that *after* having read the messages. So that seems
> mostly backwards to me.

Reading is one thing. Information storage and organisation is
another. After a message is delivered (and read) to my mailbox, it's
really mine and I can (and should be able) to affix it and integrate
it into my organisational scheme any way I want, don't you think?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"if there's anything more important than my ego,
 i want it caught and shot now."
-- zaphod beeblebrox

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Mike Hommey  [2010.01.08.2106 +1300]:
> I'm in \o_ (though I won't be in Wellington). I've been thinking
> about a fuse filesystem on top of notmuch for a while.

Grand news to see you interested! A FUSE filesystem is <25 functions
to implement, and each function is basically an entity of its own
and thus highly parallisable. Once we agreed on a general mapping
between filesystem i/o and notmuch interaction, 25 of us can write
a function each and be done. How's that for collaboration? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"courage is not the absence of fear, but the decision
 that something else is more important than fear."
  -- ambrose redmoon

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Mike Hommey  [2010.01.08.2109 +1300]:
> That may leak decrypted form in the xapian index, though in
> a split manner. But that'd still be a problem IMHO.

Not for me, since the index is stored on encrypted media. Thus, this
should be off-by-default, but possible.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"academia is really just a way to help those with high volumes of
 nothing to say to social status."
 -- myself on #debian-devel, 01 Feb 2007

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Ruben Pollan  [2010.01.08.2221 +1300]:
> I think that would be security hole. You should not store the
> encrypted messages on a decrypted database. A solution whould be
> to encrypt as well the xapian DB, but I think is too complex for
> the use.

As I said in <20100108091216.GC735 at lapse.rw.madduck.net>, I think it
should be optionally possible for those that are encrypting the
xapian DB in other ways.

> You should be still able, with the actual notmuch, to search over
> the headers of your encrypted messages, or any other non-encrypted
> part of the message. Is not like that?

Most of the time, I search headers, but I do search bodies
regularly. So no, that would not be enough, at least not with the
ideal solution. And notmuch comes close to ideal already! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

infinite loop: see 'loop, infinite'.
loop, infinite: see 'infinite loop'.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Mike Hommey  [2010.01.08.2220 +1300]:
> FYI, I have a good experience writing fuse filesystems, both with
> high-level and low-level APIs. I'd avise to use the low-level API,
> which allows for better performance.

I don't have any experience with FUSE yet, but the examples in
/usr/share/doc/libfuse-dev/examples/ look trivial. This is where
I would start, one function at a time. If you have a better
suggestion, I'd love to hear it; or to clone your repo! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

be careful of reading health books -- you might die of a misprint.
 -- mark twain

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Idea for storing tags

[martin f krafft]

Folks, over in #notmuch, we just floated an idea that I'd like to
get out to you. We've been debating storing tags for messages.
Therefore I am cross-posting. Please forgive me.

So far, there are two approaches:

1. External database, which has the downside of not being
   synchronisable with standard IMAP, like the rest of your mail
   (assuming you use IMAP). Also, it's possible for mailstore and
   database to get out of sync.

2. In-headers, which has the downside of leaking (e.g. when
   bouncing), and incurs the risks associated with message rewrites
   (which I think is pretty much ignorable, but it's still there).
   Also, there's a performance issue, but in the context of an
   indexer like notmuch, this is negligible.

   The leakage is real, though and I think it makes in-headers
   unusable. After all, I don't ever want anyone else to know that
   I tag e-mails from my boss as "from-idiots", and I forward and
   bounce mail on a regular basis. I could tell my MTA to remove
   those headers, but I might forget to do that on a new system.

We also previously determined that IMAP keywords are pretty much
useless as they are stored per mailbox, not per message, not
standardised, and limited in their length anyway [0]. This also
means that we don't really need to investigate sensibly storing tags
in Maildir (e.g. with xattrs), because IMAP cannot transport them.

0. http://lists.madduck.net/pipermail/mailtags/2007-August/msg00016.html

Seriously, who implemented IMAPv4rev1 and what sort of crack were
they smoking??

I remember there was some KDE groupware contacts manager that used
IMAP to synchronise contacts. At first, this sounds horrible, but
when you detach IMAP from RFC822, it becomes a generic synchronising
protocol. The next step is then straight forward, and I want to
share this idea with you:

How about using pseudo-mails stored in Maildir and synchronised by
IMAP? E.g. every folder could have a subfolder .TAGS and if we find
a way to smartly pair messages between parent and subfolder, we'd
have a tag store alongside the mailstore it refers to, but without
the danger of leakage, and without having to rewrite messages.

The major problem with this is when clients don't understand this
"protocol", for then they will display all .TAGS folders as regular
IMAP folders, and try to treat the messages therein as regular
mails. Somewhere sometime this is bound to blow up and I don't
really know how to prevent that.

Anyway, the idea is out now. Thoughts?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

echo Prpv a\'rfg cnf har cvcr | tr Pacfghnrvp Cnpstuaeic

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Idea for storing tags

[martin f krafft]

also sprach Scott Robinson  [2010.01.12.1644 +1300]:
> I wrote a script to store and sync my tags.
> 
>   * One filename per message-ID.
>   * Line-feed seperated tags in each file.
> 
> Then the whole structure is controlled via git.
> Conflict-resolution and sync comes for free.

How do you ensure that the external tag store and your mail store do
not go out of sync? I assume that mails without a tagfile are simply
untagged, so that's hardly the issue. However, if you delete a mail,
how do you ensure that the tag database is cleaned up?

Also, do you attach tags automatically, e.g. with procmail on the
server? If so, how do you initiate git-pull locally?

Would you consider sharing your script?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"alle vorurteile kommen aus den eingeweiden."
 - friedrich nietzsche

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Scott Robinson  [2010.01.12.1644 +1300]:
> Then the whole structure is controlled via git.
> Conflict-resolution and sync comes for free.

I've just had a good think about this, also because the idea of
abandoning IMAP and using Git has been around for a while and
I have not really wrapped my head around it.

If the MDA delivers to Git, then potentially, you might get into
a situation where you cannot write your own changes back to the
repo. This is also a DoS scenario: I'll just keep sending you
e-mail, and if I manage to pass your mail filters, I'll basically
commit to your mail repository at regular intervals. Say those are
5 seconds. In order for you to write updates to the repo, e.g. to
update tags, then you would need to pull, rebase, and push all
within 5 seconds, for otherwise you'd try to push non-fast-forwards.

This a bit unrealistic, surely, but there's a real annoyance in it:
you'd have to pull/rebase/push until a push succeeds ? until you
found a time window between pull and push during which the MDA
didn't write to the repo. This might take a long time. If this
happens in the background by Cron, it's not a real concern, but if
this becomes a UI issue, I wouldn't know how to handle it.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

don't hate yourself in the morning -- sleep till noon.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Jameson Rollins  [2010.01.13.0838 
+1300]:
> What about if just the tag information is stored in the
> repository, and not the mail itself?  In that case only the user
> would be pushing into the repo and you wouldn't have to worry
> about the DoS scenario.

I certainly would like the ability to have messages
automatically-tagged on delivery, by procmail.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

may the bluebird of happiness twiddle your bits.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Idea for storing tags

[martin f krafft]

also sprach Scott Morrison  [2010.01.12.1711 +1300]:
> 1.  synchronization of tag data with emails -- if they are in
> a subfolder then it presents the issue of maintaining this
> subfolder when managing emails (moving, deleting, duplicating etc)
> and any .tag folder unaware clients are likely cause an breakage
> in tagdata/message association.  One way of doing this is to have
> a global .tag folder.

A global .tag folder indexed by e.g. message ID, as you state later,
would probably allow for this. Or a file-per-tag design. We'd have
to think carefully about pros and cons for each.

When thinking about this, I always have to remind myself that we are
targetting this at a design that has indexed search. If that weren't
the case, searches would be incredibly expensive.

Maybe a better approach would be content addressing (see below).

> 2. what happens if that message is archived or moved to an
> exclusively local cache -- eg. Mail.app on OS X can easily move
> IMAP messages to a folder resident on the computers computers?

Well, if the target can store tags, then ideally the MUA should know
how to transfer them along.

Maybe the right thing to do would be to use extended attributes
(which are stored in the inode!), even if they may not be
universally supported yet. If our solution scales, then this might
lead to a significant increase in xattr adoption.

> 3. what happens with duplicates of emails -- I would assume that
> the message id would be the key to match the tag data to the
> message.  In this system a duplicate of a message could not have
> a different set of tags from the original (not that this would
> necessarily be desirable.)

Duplicates need folders, and tags and folders are somewhat at odds
with each other. I mean, you can represent a folder hierarchy with
tags (and more), and if you have tags and folders, you are
potentially introducing a level of confusion/ambiguity that we don't
want in the first place. Maybe the ideal solution doesn't need
folders anymore (and IMAP-compatible (Maildir) subfolders have
always been a hack anyway).

There are also two types of duplicates: copies and links. The former
can diverge, the latter can't. I don't really see a reason for
either. It's not like you need to copy a mail before you edit it,
and I don't see a real reason for linking, assuming that the primary
means of browsing will be tag-searches anyway.

Duplicates always make me think of content addressing, like Git's
object cache. We could store the content hash of a message in its
filename, and also use the hash to index into the tag database.
I think that would be much cleaner than message IDs, and would make
handling true duplicates (links) much easier, while copies (diverged
ex-duplicates) would also be taken care of automatically.

> Your mention of potential leakage (aka inadvertent disclosure of
> tag data) is real -- but only if the client used to bounce/forward
> is not the one to tag the message (one would assume that if
> a client can tag, it can know to exclude the tags in a bounce.)

True, and it's probably the minority of people using multiple
clients. But those who do might also manipulate mail with sed and
use sendmail directly.

I don't think we can successfully enhance RFC 5351 to make MTAs
always ditch the Tags:-header.

> Mail.app -- which I am pluging into does not forward headers --

ew! ;) (I think one should be able to forward pristine mails)

> though it will include all headers in a bounce -- but chance are
> you aren't tagging messages you are bouncing.:)

That chance might well be very low. I bounce/forward-as-attachment
a lot of mail from the past to make it easier for others to
establish context.

> The performance issue is very real -- because it means that
> somehow messages have to rewritten to the IMAP server -- IMAP
> doesn't have a mechanism AFAIK for updates.

Not even UIDPLUS?
http://wiki.dovecot.org/FeatUIDPLUS

> Additionally, IMAP doesn't have a mechanism for simply replacing
> one message data with another -- a new message must be written and
> the old message must be deleted and the message IMAP UID will
> change, and the client will have to deal with this especially if
> it is cache the messages.

Yes, I am experiencing this pain regularly, since I currently use
a lot of message rewriting as part of my workflow ? one of the
reasons why I'd like to find an alternative.

> Also GMAIL IMAP is an issue-

Yeah, I bet. Is there anyone who doesn't think that that's Google's
problem, not ours, though?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"there's someone in my head but it's not me."
-- pink floyd, the dark side of the moon, 1972

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Idea for storing tags

[martin f krafft]

also sprach Scott Morrison  [2010.01.13.1752 +1300]:
> The problem with anything that is not universally supported is
> that for a package that is to appeal to a wide userbase, most
> don't know and don't care about the particulars of this IMAP
> server vs that IMAP server.  all they know it that for some reason
> it doesn't work with account X -- which leads to support head
> aches.
[...]
> Call it Googles problem as you like -- but when I have a product
> that doesn't work with GMAIL IMAP there are a lot of potential
> users that don't care about server peculiarities and rather just
> have it work.

Well, the way I see it: you cannot change all IMAP servers at once,
and you certainly cannot change Google. If it's possible to
implement tagging for email (dare say semantic e-mail) with standard
means (where standard means sub-standard, as exemplified by your
previous GMail IMAP example), then that's the best way, but if that
can't happen then we ought to try a better way. Should we find
a solution then, by the rate of standardisation on the 'Net, maybe
my grandchildren will finally be able to do proper e-mail. ;)

> I agree that conceptually duplicates should be buried but end
> users do have "peculiar" organization systems.

I think tags should help abstract e-mail away from underlying
storage and I'd love that to be a goal.

> From my reading, uidplus doesn't allow a delta modification of
> a message on a server -- just to write a portion of a message back
> -- you still have to write the whole thing back and that can mean
> real bandwidth issues for some messages.

Absolutely. It would indeed be better if you could just send
changes.

I just sent a blank mail to
imap-protocol-subscribe at mailman.u.washington.edu
and have started browsing the archives. So far, there's not really
anything relevant.

Anyway, looking back at the RFC on keywords, it's not exactly
encouraging:

  A keyword is defined by the server implementation. Keywords do not
  begin with "\". Servers MAY permit the client to define new
  keywords in the mailbox (see the description of the PERMANENTFLAGS
  response code for more information).

Anyway, I'll try to untangle the various issues re:IMAP we've been
seeing, write mails for each, and hopefully get to the point where
I can enquire about IMAPv5. ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

the unix philosophy basically involves
giving you enough rope to hang yourself.
and then some more, just to be sure.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Idea for storing tags

[martin f krafft]

also sprach Carl Worth  [2010.01.14.1432 +1300]:
> Yes. This approach requires some external means of synchronizing the
> tags from one system to another.
> 
> I don't understand what it would mean to have the mailstore and the
> database out of synch here. This approach doesn't have the tags in the
> mailstore by definition, right?

You might have marked a message 'read' on one machine and if the two
get out of sync on another machine, you might have the same message
unread there.

> > How about using pseudo-mails stored in Maildir and synchronised by
> > IMAP? E.g. every folder could have a subfolder .TAGS and if we find
> > a way to smartly pair messages between parent and subfolder, we'd
> > have a tag store alongside the mailstore it refers to, but without
> > the danger of leakage, and without having to rewrite messages.
> ...
> > Anyway, the idea is out now. Thoughts?
> 
> There are a couple of problems that I don't see addressed at all with
> this approach. The first is that there's not a one-to-one mapping
> between messages and files in the mail store. (I'm CCed on a lot of list
> mail meaning that I have multiple files in my mail store for a single
> message.)

Shouldn't this just be solved? I've had formail+procmail delete my
duplicates for 10+ years, and while I don't like the fact that
I usually get the CC before the list mail, and thus cannot filter on
Delivered-To, I have never looked back.

> Second, the only reason I would be interested in synchronizing mail
> between two systems is so that I could manipulate the tag data in
> multiple places, (that is, remove the "unread" tag whether on my
> network-disconnected laptop or via web-mail when away from my
> laptop). Using imap for synchronizing a file of tags within the mail
> store gives you no mechanism for doing any sort of conflict resolution,
> right? (Which I think in almost all cases is going to be quite trivial
> if there's a chance for a program to resolve it.)

I have not thought about this, but you are right. IMAP does not
really allow for conflict resolution, which may well be *the* reason
why you cannot update existing messages.

> [*] Though, I think a plain-text file with tags managed with
> something like git (and perhaps a custom merger) could save a lot
> of work. Or perhaps a plain-text journal of tag manipulations on
> either end that could be replayed on the other.

Git is good at conflict resolution if run interactively, but [0]
still makes me question whether it can ever take the place of IMAP.
However, Asheesh Laroia, who has floated the idea of Git-for-mail at
DebConf8 already, has some ideas and hopefully will soon reply to my
mail [0], which I just bounced.

0. http://notmuchmail.org/pipermail/notmuch/2010/001114.html

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

apt-get source --compile gentoo

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Asheesh Laroia  [2010.01.14.2112 +1300]:
> Sure. But the MDA doesn't need to do the commit immediately. Since
> (presumably) we're using Maildir, the MDA on the mail receiving
> server is going to generate filenames that won't cause conflicts.
> So it's okay to leave the files uncommitted.

So when does the commit happen?

> When I did the "git merge", git would create the Maildir files in
> ~/Maildir/cur/... non-atomically.

This might be something that the Git people could address if it was
brought up on the mailing list. Then again, it might not be possible
without going via a temporary file, which I doubt will fly.

I suppose that I never actually considered merges on the IMAP server
side, but obviously the IMAP server has to work off a clone, and
that means it needs to merge.

> Dovecot would notice the file in ~/Maildir/cur/ and think, "This
> file must be ready!" So it would parse it even though git hadn't
> finished writing it. This caused me to only see partial headers in
> Alpine since Dovecot parsed it before it was a complete message.

I wonder if a custom merge driver could address this to properly use
?/tmp/ to assemble the message and only then move it.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"this week dragged past me so slowly;
 the days fell on their knees..."
-- david bowie

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Idea for storing tags

[martin f krafft]

also sprach Carl Worth  [2010.01.15.1124 +1300]:
> > You might have marked a message 'read' on one machine and if the two
> > get out of sync on another machine, you might have the same message
> > unread there.
> 
> That's a different issue though. With two databases there's clearly the
> opportunity for the two databases to be out of synch.
> 
> But you talked about the database being out of synch with respect to the
> mailstore. And that's something I just don't understand, (given the
> assumption that all tags are stored in the database---which was the
> explicit description of the case of interest).

Yes, we are talking about the situation where the tagstore is
seperate from the mailstore, and that they are both synchronised
with a server, or between machines, separately. If for some reason
you only synchronise the mailstore ? say because the connection
drops before the sync of the tagstore completes ? then you end up
with an out-of-sync situation, because the mailstore-sync will have
pulled in a new message, but not the associated tags. So if you had
already read this message on another machine and tagged it 'done',
then it would show up on this machine as 'new' without the 'done'
tag, because the tags were not synchronised.

The only way to really solve this is by transferring a message and
its tags in a transactional way.

> > Shouldn't this just be solved? I've had formail+procmail delete my
> > duplicates for 10+ years, and while I don't like the fact that
> > I usually get the CC before the list mail, and thus cannot filter on
> > Delivered-To, I have never looked back.
> 
> Notmuch has access to all the information it needs to allow you to
> delete the CC version once the list mail arrives. So you could do
> notmuch-based deletion now and avoid losing the Delivered-To header if
> you want.

Of course. I hadn't thought that far.

However, there are still benefits to formail, namely avoiding having
to run duplicates through potentially expensive spamfilters.

> I think that synchronizing the mail store and synchronizing the
> tags information are tasks that have different requirements, and
> for which we may well want different tools.

Fair enough. Maybe I am just paranoid about the stores getting out
of sync (see above).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"we all know linux is great...
 it does infinite loops in 5 seconds."
 -- linus torvalds

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Threading

[martin f krafft]

also sprach Carl Worth  [2010.01.15.1108 +1300]:
> > Reading is one thing. Information storage and organisation is
> > another. After a message is delivered (and read) to my mailbox,
> > it's really mine and I can (and should be able) to affix it and
> > integrate it into my organisational scheme any way I want, don't
> > you think?
> 
> A fair point.
> 
> I don't see this being something I'm going to spend any time
> implementing. I just wouldn't use the functionality myself. But
> I would be happy to integrate patches if someone came up with
> some.

Maybe I should try to persuade you in person.

Just today I referenced a discussion I had with a client's ISP,
which was done via a web-based support system (custhelp.com). They
send you e-mail for every post you or they make to the thread, but
those e-mails do not reference each other. Fortunately, I stitched
them together and when I searched for the correspondence in my
mailstore, I had the entire thread available to me, which was handy
(thanks to mutt's useful thread handling abilities).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"this week dragged past me so slowly;
 the days fell on their knees..."
-- david bowie

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Thoughts on notmuch and Lua

[martin f krafft]

also sprach Carl Worth  [2010.01.15.1200 +1300]:
> > Lua has many advantages over other scripting languages when it
> > comes to integration with a C program. It has a very clean and
> > easy C API, the overhead of running Lua scripts is not noticable
> > among other things.
> 
> I've definitely heard lots of good things about "lua
> embedability". So if we do decide to provide hooks, then lua would
> seem like a logical option to look at first. I've never looked at
> it closely myself though.

Lua for hooks has the advantage that the hooks can be executed in
the context of manipulateable objects. On the other hand, hooks in
the style of run-parts directories are more flexible and accessible,
and could always be invoked as filters for the manipulateable data.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"imagine if every thursday your shoes exploded if you
 tied them the usual way. this happens to us all the time
 with computers, and nobody thinks of complaining."
-- jeff raskin

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git as notmuch object store (was: Potential problem using Git for mail)

[martin f krafft]

also sprach Asheesh Laroia  [2010.01.21.1928 +1300]:
> >I suppose that I never actually considered merges on the IMAP
> >server side, but obviously the IMAP server has to work off a clone,
> >and that means it needs to merge.
> 
> It's not "merge" that's unsafe; that just builds a tree in the git
> index (assuming no conflicts). It's the ensuing process of git
> writing a tree to the filesystem that is problematic.

There is no way to make that atomic, I am afraid. As you say.

> I could probably actually write a wrapper that locks the Maildir
> while git is operating. It would probably be specific to each IMAP
> server.

Ouch! I'd really rather not go there.

> Note that this mean git is fundamentally incompatible with
> Maildir, not just IMAP servers.

We had an idea about using Git to replace IMAP altogether, along
with making notmuch use a bare Git repository as object store. The
idea is that notmuch uses low-level Git commands to access the .git
repository (from which you can still checkout a tree tying the blobs
into a Maildir). The benefit would be compression, lower inode count
(due to packs), and backups using clones/merges.

You could either have the MDA write to a Git repo on the server side
and use git packs to download mail to a local clone, or one could
have e.g. offlineimap grow a Git storage backend. The interface to
notmuch would be the same.

If we used this, all the rename and delete code would be refactored
into Git and could be removed from notmuch. In addition, notmuch
could actually use Git tree objects to represent the results of
searches, and you could checkout these trees. However, deleting
messages from search results would not have any effect on the
message or its existence in other search results, much like what
happens with mairix nowadays.

I think we all kinda agreed that the Maildir flags should not be
used by notmuch and that things like Sebastian's notmuchsync should
be used if people wanted flags represented in Maildir filenames.

Instead of a Maildir checkout, notmuch could provide an interface to
browse the store contents in a way that could make it accessible to
mutt. The argument is that with 'notmuch {ls,cat,rm,?}', a mutt
backend could be trivially written. I am not sure about that, but
it's worth a try.

But there are still good reasons why you'd want to have IMAP
capability too, e.g. Webmail. Given the atomicity problems that come
from Git, maybe an IMAP server reading from the Git store would make
sense.

However, this all sounds like a lot of NIH and reinvention. It's
a bit like the marriage between the hypothetical Maildir2 and Git,
which is definitely worth pursuing. Before we embark on any of this,
however, we'd need to define the way in which Git stores mail.

Stewart, you've worked most on this so far. Would you like to share
your thoughts?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"reife des mannes, das ist es,
 den ernst wiedergefunden zu haben, den
 man hatte als kind beim spiel."
-- friedrich nietzsche

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git as notmuch object store (was: Potential problem using Git for mail)

[martin f krafft]

also sprach Asheesh Laroia  [2010.01.25.1819 +1300]:
> You say "Ouch" but you should know Dovecot *already* does this. I
> don't mind interoperating with that.
>
> See http://wiki.dovecot.org/MailboxFormat/Maildir, section "Issues
> with the specification", subsection "Locking". I term this theQ
> famous readdir() race.

Yikes. IMAP (including dovecot) just SUCKS.

> Without this lock, Maildir is fundamentally incompatible with IMAP
> -- one Maildir-using process modifying message flags could make
> a different Maildir-using process think said message is actually
> deleted. In the case of temporary disappearing mails in Mutt
> locally, that's not the end of the world. For IMAP, it will make
> the IMAP daemon (one of the Maildir-using processes) send a note
> to IMAP clients saying that the message has been deleted and
> expunged.
[?]
> Just don't fall into the trap of thinking Maildir is compatible
> with IMAP. It's not, because as I understand things, the
> filesystem doesn't guarantee that you can actually iterate across
> a directory's files if another process is modifying the list of
> files.

This is all perfect reason to concentrate even more on designing
a store that could potentially make IMAP obsolete once and for all!

The current idea is to sync Git downstream only, and find a way to
keep multiple copies of a tagstore in sync, by way of the "server
instance" (where mail is received/delivered). Deleting messages
would then be something like setting the notmuch::deleted tag, which
clients would honour; on the server, a cleanup process would run
regularly to actually delete the blobs associated with deleted
messages. This would then propogate the next time one pulls from
Git.

Whether to store history (commit objects) or just collections (tree
objects) needs to be investigated.

> >But there are still good reasons why you'd want to have IMAP
> >capability too, e.g. Webmail. Given the atomicity problems that
> >come from Git, maybe an IMAP server reading from the Git store
> >would make sense.
> 
> It wouldn't be too hard to write a FUSE filesystem that presented
> an interface to a Git repository that didn't allow the contents of
> files to be modified. Then Dovecot could think it's interacting
> with the filesystem.

Yes, a FUSE layer (which adds a daemon), or a lightweight access
API via libnotmuch. Probably the former using the latter. ;)

> Aww, I like Maildir flags, but if there's a sync tool, I'm fine
> with that.
[?]
> I'm not sure, but maybe it's safe if you refuse to ever modify
> a message's flags in the filename.

The main point is that there is nothing really in Maildir filenames
that you couldn't equally (and possibly better) represent in the
notmuch::* tag namespace, and then there is benefit in only having
one used primarily (which means notmuchsync can do whatever it
wants without affecting or messing with notmuch).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"if I can't dance, i don't want to be part of your revolution."
- emma goldman

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git as notmuch object store (was: Potential problem using Git for mail)

[martin f krafft]

also sprach Sebastian Spaeth  [2010.01.26.0249 +1300]:
> While notmuchsync fullfils my needs, it is a kludge. It needs to
> call "notmuch" for each mail where a MailDir flag has changed
> (which can be quite often on an initial run, where most mails are
> likely to be read), this can take a long, long time. It would
> makes sense IMHO to at least pick pioto's "don't set unread if 'S'
> flag is set" on notmuch new[1].

I am sure this could be implemented with libnotmuch if it proves to
be useful.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"it isn't pollution that's harming the environment.
 it's the impurities in our air and water that are doing it."
  - dan quayle

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] proposal for more streamlined mail flow in notmuch

[martin f krafft]

also sprach Jameson Rollins  [2010.01.17.0949 
+1300]:
> I would like to put forth here a proposal for a couple of changes
> to notmuch that I believe will considerably streamline message
> handling and new message flow through notmuch.  Notmuch is still
> new and I believe it hasn't quite figured out the best way to
> handle message flow in the new mail handling paradigm that it is
> working.  This is a proposal to fix that.
>
> I believe that most people are syncing their mail from remote IMAP or
> POP servers to local maildirs (via offlineimap for instance), and that
> most people's IMAP servers have limited storage capacity.  This
> practically means that people can not keep all of their mail in a
> single directory.  However, notmuch currently basically assumes that
> this is what is happening.  In order to keep synced maildirs from
> growing out of hand, messages need to be either deleted or moved out
> of the "inbox" where they initially show up.

We should keep this in mind when designing an object store, whether
or not that's based on Git. If we use Git, then a local branch is
all we really need to support this. \o/

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

above all, we should not wish to divest
our existence of its rich ambiguity.
 --friedrich nietzsche

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git feature branch

[martin f krafft]

also sprach Carl Worth  [2010.01.23.1010 +1300]:
> Anyway, I'll be on vacation for the next few days, so will likely
> not have much, (likely have not much?), time for patch merging.
> 
> But I *am* anxious to get back to the backlog. And in the
> meantime, I really appreciate others merging and sharing patches.

I discussed this with Carl at LCA a bit and ideally we should come
up with a way to relieve Carl of the bottleneck burden (obviously
without stealing away his dictator hat ;)

I think it would make sense to move the mainline to git.debian.org
for now, or another place where everyone can easily get an account.
As alternatives I propose repo.or.cz. I'd prefer to stay away from
commercial services like Github.

Once we're there, how about copying the branch structure from
Git development[0]:

  maint/? the stable release
  master/   ? the stablising head
  next/ ? testing branch
  pu/   ? patch integration branch (proposed updates)

Each of the four branches is usually a direct descendant of the one
above it. Conceptually, the feature enters at an unstable branch
(usually 'next' or 'pu'), and "graduates" to 'master' for the next
release once it is considered stable enough.

0. http://repo.or.cz/w/git.git/blob/HEAD:/Documentation/gitworkflows.txt

Sebastian, would it make sense to migrate your work into a 'pu'
branch?

What patch tracking workflow should we adopt? Keep sending patches
to the mailing list and have a few people who integrate them into
master or pu (or even maint/next), as appropriate? Use the Debian
bug tracker? Use Sebastian's Roundup instance? Set up a patch queue
manager on notmuchmail.org? Use patchwork [1]?

Cheers,

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"in all unimportant matters, style, not sincerity, is the essential.
 in all important matters, style, not sincerity, is the essential."
-- oscar wilde

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git feature branch

[martin f krafft]

also sprach micah anderson  [2010.01.27.1124 +1300]:
> Couldn't all of this be done without moving the existing git
> repository (don't forget that transition is a cost)? Those who
> wish to put together these proposed branches go ahead and do so,
> publishing those wherever they like (git.debian.org, gitorious,
> their own hosted git repositories, or even github) and then Carl
> can just add those as remotes as he sees fit.

I brought this up, so I should make the motivation explicit: I just
tried to relieve Carl from the burden of bottleneck. Since passing
out SSH accounts on his own machine does not really scale, I looked
elsewhere.

However, I agree that the easiest solution is just to add more
users. Potentially, Carl can make use of gitolite, I can investigate
that some time this week.

> Personally, I've found mailing lists that have patches sent to
> them tends to totally kill the list for anything else. It seems
> a bit weird to use Debian's bug tracker for a non-Debian native
> program (but using it for the Debian package of notmuch does make
> sense). I am not so familiar with Roundup, patch queue trackers or
> patchwork to have anything to say about those.

patchwork integrates with the mailing list and slurps patches and
related discussion and threads them into a webpage, where they can
be workflow-managed.

The Debian bug tracker has the benefit of being usable with e-mail
(and this is notmuch we're developing, don't forget). The others are
all exclusively web-based, with the exception of launchpad, AFAIK.

The idea of having a tracker is quite simply to make it easier for
Carl to stay on top, and for e.g. you and I to assist him by vetting
patches in our free minutes.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

there's an old proverb that says just about whatever you want it to.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Jameson Rollins  [2010.01.26.1046 
+1300]:
> > For example, I might have:
> > 
> > ~/.notmuch-config:
> > 
> > [database]
> > path=/home/pioto/mail
> > ...
> > [tags]
> > pioto at pioto.org/INBOX.ListMail.notmuch = notmuch
> > 
> > So, a 'tags' section, where each key is the folder name, relative to the
> > db path, and the value is one or more tag names
> 
> I think this idea is a really good one and I would like to pursue it as
> a tangent thread here.  I was going to propose something very similar to
> this.  I think it's a very flexible idea that would help in a lot of
> ways.

I think we need to carefully distinguish here. The above seems to
suggest a mapping from folder to tag, but we don't actually need
tags for folder locations, because those can (and should) be
implicitly determined from the database and storing the tag in
addition would just run the risk of getting out of sync: if I moved
a message, I would also have to remember to delete old and add new
tags, which is just asking for trouble.

> [tags]
> inbox = +inbox,+unread
> sent = +sent
> drafts = +draft
> archive = -inbox

This proposal, on the other hand, is an interesting one, but when is
it supposed to happen? It just feels wrong to make this happen as
part of 'notmuch new'.

What I would like to see is a notmuch-aware MDA, e.g. a programme
which reads an incoming mail on stdin and can do all this kind of
stuff, e.g. assign tags based on such rules (or take tags as
arguments, so that I could trivially set tags from procmail too),
write the message to the message store, and update the database.

This would allow us to get rid of 'notmuch new' altogether, at least
conceptually. We'd still need it if mail is being delivered
independently, e.g. with offlineimap.

On the performance side, it might make sense to write to a journal
instead of updating the database every time. SpamAssassin does this
with its Bayesian database, and it only merges the journal every
X updates (or when the user manually requests it). I am not sure
whether this is possible with Xapian. On the other hand, I think
notmuch needs to learn to journal anyway so that we can keep
different instances in sync.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"the only way to get rid of a temptation is to yield to it."
-- oscar wilde

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Jameson Rollins  [2010.01.27.0603 
+1300]:
> > This is getting involved. 
> > 
> > Maybe I'm missing something in this thread; but, why couldn't these complex 
> > and
> > context-sensitive decisions be delegated to sub-processes? ala git hooks?
> 
> I think this idea is completely independent of anything having to do
> with using git as a mail store.  That's why I was trying to separate it
> into a separate thread.

I think he meant "notmuch hooks like you have hooks for Git too",
e.g. thread:755741d13573c7642761d2a175cb146d

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"if i am occasionally a little overdressed, i make up for it by being
 always immensely over-educated."
-- oscar wilde

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach James Westby  [2010.01.28.1828 +1300]:
> Are you trying to use thread: such that it could be passed to
> notmuch show to see the conversation?
> 
> That's not going to work so well if so.

Why not? Works fine for me with the vim plugin...

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"perfection is achieved, not when there is nothing more to add, but
 when there is nothing left to take away."
 -- antoine de saint-exup?ry

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach martin f krafft  [2010.01.28.1834 +1300]:
> > That's not going to work so well if so.
> 
> Why not? Works fine for me with the vim plugin...

Now I get it. I was talking about
id:20100114084713.GA22273 at harikalardiyari

Sorry, I *am* new to notmuch ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"when zarathustra was alone... he said to his heart: 'could it be
 possible! this old saint in the forest hath not yet heard of it, that
 god is dead!'"
 - friedrich nietzsche

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100128/c2bedb2b/attachment.pgp>

[notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Servilio Afre Puentes  [2010.01.29.0132 
+1300]:
> >> [tags]
> >> inbox = +inbox,+unread
> >> sent = +sent
> >> drafts = +draft
> >> archive = -inbox
> >
> > This proposal, on the other hand, is an interesting one, but when is
> > it supposed to happen? It just feels wrong to make this happen as
> > part of 'notmuch new'.
> 
> Why so?

I guess I just dislike having to run notmuch new regularly, rather
than integrating the database more closely with the mail flow.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"to get back my youth i would do anything in the world, except take
 exercise, get up early, or be respectable."
-- oscar wilde

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Ben Gamari  [2010.01.29.0949 +1300]:
> > I guess I just dislike having to run notmuch new regularly,
> > rather than integrating the database more closely with the mail
> > flow.
> > 
> Sounds like you need to add a line to crontab.

It still feels like a hack. It's a bit like making many changes to
a source code repository (new mails get delivered) and committing
only once every hour (notmuch new), rather than making and
committing transactional changes (delivering and catalogueing mails
individually).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

a Hooloovoo is a superintelligent shade of the color blue.
-- douglas adams, "the hitchhiker's guide to the galaxy"

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Some thoughts about notmuch sync with other agents

[martin f krafft]

also sprach Paul R  [2010.01.28.0316 +1300]:
> As you see, I advocate a NotMuch <-> IMAP synchronisation ASAP :)

Given the limitations of IMAP (non-transactional, non-standard
keywords implementation, ?), I think chances of this are rapidly
diminishing, but I'd love to be proven wrong.

> First of all, processing mail with MUA involves some simple logic that
> is shared by most MUA. This is about receiving *new* mails, *reading*
> mail, *replying* to mail and so on... IMHO, this really belongs to the
> mail processing logic and not to the user logic. Hence my first
> request :
> 
>   1: Don't use user tags space to store MUA flags.
> 
>  That means no more "seen", "unread", "replied" as tags. These are
>  MUA processing *flags*, that must belong to an established set.
>  Tags, on the other hand, are user-land information. So no more
>  [seen, replied, grandma, important] tag sets :)

I disagree.

The MUA actually doesn't (shouldn't) care at all about any of these
flags, at least not for core functionality. Sure, a MUA should
probably hide messages tagged 'deleted', and it would be nice if it
could be configured to highlight messages tagged 'important', but
none of the others ? "seen", "unread", "replied", ? ? have any role
in the core functionality. They are purely user-specific.

They are leftovers from days when some MUA designer decided that
having these flags would be a useful way to organise e-mail
handling, but that person probably dealt with a dozen messages
a day, and didn't have half his/her life organised through
electronic mail.

Point being, times and needs have changed. And while we're in the
process of finding the technology that can suit those needs (in the
most generic way), we might just as well (and should) rid ourselves
from these leftovers.

Any solution must be generic enough so that if you rely on the
functionality provided by these tags, you can trivially make it
happen, e.g. with hooks to add/remove flags on certain actions (such
as sending a reply, or reading a message).

Neither IMAP nor Maildir is capable of storing an extensible,
freely-configurable set of tags (keywords). Therefore, we need a new
method anyway.

> Once this is done, notmuch will know, in a robust a predictable
> way, what happened to a mail. Simply put, NotMuch will store and
> expose MUA flags (Passed, Replied, Seen, Trashed, Draft, and
> Flagged [1]). For each , notmuch should associate
> a _synced flag. When changing  from notmuch, it should
> set the _synced bit to 0. These are MUA mail flags.

I think the semantics would be clearer the other way around: setting
*_changed when a flag is changed.

> Additionally, in a third ? space ?, notmuch should store its ? new ?
> bit, as well as a ? missing ? bit probably. Again, this is neither MUA
> logic or user logic, so this should not interfer with user
> classification facility provided by tags, nor with MUA flags. It,
> really, is something else, let's name it "status".

Or "lifetime".

> Once this is done, the 'notmuch new' command should find new mails
> and set the 'new' bit for them, and find the missing mails and set
> the 'missing' bit for them. This will allow for robust external
> processing.

Why would I want to keep around a record in the database when the
physical file is no longer present?

> # 1/ Sync from NotMuch to MailDir
> 
> notmuch list flags:(seen and not seen_synced) 
>   | notmuch-maildir --mark-mail seen
>   | notmuch move --stdin
>   | notmuch set flags:+seen_synced --stdin

Have you seen/look at notmuchsync?

> The output of the first command would be a list of filenames for mails
> 'seen' since last sync. The second one, an external notmuch--maildir
> helper, would propagate this logic to the MailDir store (easy, this is
> simply a rename), and output the list of (old-name ! new-name). Then
> notmuch would use this information, via a generic 'move' switch, to know
> that mail has been moved, and would output the list of the new places.
> Finaly, notmuch would set the seen_synced flag.

What happens if notmuch-move gets killed due to out-of-memory half
way through?

> As we have seen, this would allow most IMAP <-> MailDir <-> NotMuch
> synchronisation, without having to implement any kind of
> MailDir-specific logic inside notmuch.

It would not take care of any tags beyond the very strictly defined
and limited set of tags you listed in your mail. My point is that we
need to solve this problem more generally anyway ? why should
a "replied" tag be synchronised, but a "no-need-to-reply" tag not?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

#include 

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] patchwork test instance (was: Git feature branch)

[martin f krafft]

I investigated some patch/issue trackers over the weekend. Here's my
summary/reply.

The executive summary is that
http://patchwork.madduck.net/project/notmuch/list/ now exists.
I have not really used it for anything real, so if some of you feel
inclined to give it a shot, sign up and triage away! Feedback
welcome.

also sprach James Rowe  [2010.01.28.2005 +1300]:
>   Roundup has command line and email interfaces.  The email interface is
> quite similar to debian's.  I've never used a launchpad hosted project
> so I can't compare it.

Roundup is an issue tracker, while Patchwork is a patch tracker.
They are fundamentally distinct, but there are overlaps. What led me
to go the Patchwork-path is that projects like the kernel and Git
don't use issue trackers but work entirely patch-based.

I don't know if that is the right way to do things, but having an
issue tracker that fills up with bugs and wishlist items lacking
patches is no better in the long run than not having an issue
tracker.

Arguably, being patch-centric means that a project has a higher
barrier of entry, but it also means that if someone wants something,
they know that they'll have to somehow end up with a patch. The way
this happens on Git is that you either write it yourself and bring
it up to discussion (which is what patchwork facilitates), or
constructively theorise the functionality until someone else
submits a patch.

>   Google's codereview tool has a nice interface for collecting and
> commenting on patches, but I suspect that suggestion will also meet with
> a degree of friction.  To me codereview feels like patchwork with
> polish.

Maybe you could take some ideas from codereview and inform the
patchwork people about them?

>   Both gitorious and github have commenting functionality built in.
> Commenting on commits in a fork is as easy as opening the commit in
> a browser.  I use something along the lines of the following script to
> open commits on github:
> 
> #! /bin/sh
> BASE=$(git config remote.${2:-origin}.url | sed 
> 's,git\(@\|://\)\([^:/]*\)[:/]\(.*\).git,http://\2/\3/commit,')
> COMMIT=$(git rev-parse ${1:-HEAD})
> sensible-browser ${BASE}/${COMMIT}
> 
>   Using github or gitorious you can easily find and track forks from one
> place as well, which makes discovering new work much easier.  Github
> even provides a pretty single page interface to the work going on in
> other forks, gitorious requires a little more leg work to do the same
> but not much.

Git now has commit notes, but it doesn't seem like that's integrated
with Github/Gitorious.

Mind you, patchwork isn't integrated at all with Git. It should be
possible to set it up to automatically flag patches that are
accepted into mainline, next, or pu.

The benefit of patchwork is that discussion isn't moved to the web,
but patchwork hooks into the mailing list, so discussion can stay
where it should IMHO be.

>   For a couple of hosted projects we use at the office we email the
> individual entries from http://github.com/$user/$project/comments.atom
> to the mailing list so they're /forcibly/ seen by everybody :)

Right, but replying requires them to open a browser and be online at
the time, right?



Anyway, I suggest we give patchwork a try. It occurs to me that
notmuch can pretty much do all of what patchwork is doing ? after
all, it's just tagging patches/threads, but until we have
synchronisable tags and a mailing list archive based on notmuch
(which could then replace patchwork), I think we'll need to employ
a third tool.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"what's your conceptual continuity? --
 well, it should be easy to see:
 the crux of the bisquit is the apopstrophe!"
-- frank zappa

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Request for high-priority improvements to notmuch

[martin f krafft]

also sprach sebastian at sspaeth.de  [2010.02.02.0929 
+1300]:
> YES PLEASE :-). notmuch seems designed to work in an ecosystem of
> surrounding scripts, feeding data in and out. But we are all currently
> limited to regexes for that. And heck, I hard a hard time understanding
> why all hell broke out until I found that i had added a tag containing
> parentheses which made my regex fail. :-). XML, JSON, any structured
> output would be nice.

Shoot me, but I'd say mbox output would be nice too.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"time flies like an arrow. fruit flies like a banana."
   -- groucho marx

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Request for high-priority improvements to notmuch

[martin f krafft]

also sprach Scott Robinson  [2010.02.02.1043 +1300]:
> > YES PLEASE :-). notmuch seems designed to work in an ecosystem of
> > surrounding scripts, feeding data in and out. But we are all currently
> > limited to regexes for that. And heck, I hard a hard time understanding
> > why all hell broke out until I found that i had added a tag containing
> > parentheses which made my regex fail. :-). XML, JSON, any structured
> > output would be nice.
> > 
> > And as for filtering: YES, PLEASE :-). notmuchsync and many other 3rd
> > party apps would love that. As father of notmuchsync, I can tell you my
> > little script hickups very badly when slurping in 200k mails (including
> > text bodies) just to find out the maildir tags of those mails.
> > 
> 
> There's been a JSON patch waiting for a month now.

The last month has been busy for everyone, mainly due to LCA. We
should now all work together to help Carl go through the patch
queue. Maybe http://patchwork.madduck.net can help.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

beauty, brains, availability, personality; pick any two.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] [PATCH] Enforce foldmethod=manual when showing messages in vim

[martin f. krafft]

Vim's NotMuch mode relies on manual markers when rendering/showing
a message. If foldmethod is set to something else (marker in my case) by
default, then there are numerous errors, and folds don't work. Hence,
set foldmethod=manual for the local buffer upon showing a message.

Signed-off-by: martin f. krafft 
---
 vim/plugin/notmuch.vim |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/vim/plugin/notmuch.vim b/vim/plugin/notmuch.vim
index a226f20..2f9b05c 100644
--- a/vim/plugin/notmuch.vim
+++ b/vim/plugin/notmuch.vim
@@ -421,6 +421,7 @@ function! s:NM_cmd_show(words)
 let b:nm_raw_info = info
 let b:nm_prev_bufnr = prev_bufnr

+setlocal foldmethod=manual
 call NM_cmd_show_mkfolds()
 call NM_cmd_show_mksyntax()
 call NM_set_map('n', g:notmuch_show_maps)
-- 
1.6.6

[notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Marten Veldthuis  [2010.02.04.0353 +1300]:
> > Like this? http://notmuchmail.org/recentchanges/
> 
> No, more like this: http://git.notmuchmail.org/git/notmuch-wiki

To my knowledge, neither of these two give you diffs. This is what
a post-receive hook offers.

It's trivial to do with Git. Assuming a recent Git-on-Debian, it's
just (assuming you are in the bare repository)

  test -d refs || echo not in repo
  sed -e 's,^#\.,,' hooks/post-receive.sample > hooks/post-receive
  chmod 755 !$
  git config hooks.mailinglist notmuch at notmuchmail.org

One might also have to send hooks.envelopesender and
hooks.emailprefix (without trailing space) could be nice.

PS: speaking of prefixes, how about remving the subject prefix of
this list in general? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"arguments are extremely vulgar,
 for everyone in good society
 holds exactly the same opinion."
-- oscar wilde

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach David Bremner  [2010.02.04.0924 +1300]:
> > PS: speaking of prefixes, how about remving the subject prefix of
> > this list in general? ;)
> 
> I used to agree, but in notmuch, I actually find it convenient to have
> threads marked by mailing list. Perhaps this will change if/when my
> email setup or the notmuch emacs ui get better...

sed -e 's,^Subject:,& [notmuch],'

;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

apt-get source --compile gentoo

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git feature branch

[martin f krafft]

also sprach Carl Worth  [2010.02.04.1605 +1300]:
> >   maint/? the stable release
> >   master/   ? the stablising head
> >   next/ ? testing branch
> >   pu/   ? patch integration branch (proposed updates)
> 
> I'm not a fan of this scheme, (or maybe I've just never quite understood
> it).
> 
> When I first encountered this scheme, (when first using git), it was
> never clear to me what branch I should actually run or test as a
> user. Nor was it obvious which branches would be regularly rebased or
> not---nor which branches had features being discussed on the mailing
> list.

I'll happily explain if you want. I think that this workflow, in
combination with the distributed functionality of Git, makes for
really powerful collaboration.

To answer your two (implicit) questions directly:

- as a user, you'd probably be tracking master, if you aren't
  running a stable release anyway. if you are ready to deal with the
  occasional bug and want some juicy stuff, go with next. if you are
  a developer ready to fend of the sh*t as it hits the fan, use pu.

  Basically, it's a bit like Debian testing/unstable/experiemental,
  except that the default entry point for new patches (packages in
  that analogy) is pu (experimental), not next (unstable, as it is
  in Debian).

  maint is then the stable branch, see below.

- nothing is ever rebased. patches are cherry-picked downwards
  (pu?next?master) and branches are occasionally merged upwards
  (master into next, next into pu).

I don't think we will need 'next' anytime soon, not until we have
a situation where we are e.g. working on the next 1.x release by
already have 2.0 on the horizon.

The important distinction is between pu (proposed-updates) and
master. The goal for master should always be that it's usable.
Features that are too new to ensure that go to pu until they
matured.

> Instead of "maint" I'd much rather just have branches that are
> named directly after the stable releases being maintained. We've
> done this with the cairo repository with branch names like "1.2",
> "1.4", "1.6", etc. That way it's very clear what the branch
> represents and it's possible to have multiple concurrent "live"
> maintenance branches. But of course, until we actually have
> releases, this doesn't really matter. :-)

This is all possible to do. It has no impact on the pu/next/master
distinction. Basically, once a release is made, master is merged
into maint (I think) and tagged. If a maintenance release has to be
made, a maint/1.0 branch is created. I don't think Git/Linux do
that, but I do.

> I want to maintain a branch myself, (where I'm the only person
> pushing to the branch). [This is different than what I've done
> with the cairo repository where we have all core maintainer's
> pushing to a central repository. I'm intentionally trying
> something new here.]

Do you want to maintain that branch yourself for your own purposes,
or do you want to be the sole maintainer of the branch that is
advertised as *the* notmuch branch, and from which future releases
are made?

> Obviously, that branch that I maintain is currently called
> "master", but I wouldn't mind (and might actually prefer) to have
> it be called "~cworth" or so. Though we have the problem that we
> need "master" to point to *something*.

Actually, there is no reason for master to exist at all. ;)
It's just the default, but it's not intrinsic.

> Beyond that, I'm quite happy to have any number of branches similarly
> maintained by any other individuals. I want to get things setup so that
> those will be hosted and listed alongside my branch on
> notmuchmail.org.

So you are talking repos, not branches. I know they are almost the
same, but branches live in repos, and if you want to be the only
person pushing to a branch, you need to be the only one with write
access to the containing repo ? unless you use gitolite, which can
do in-repo per-branch access control.

> > What patch tracking workflow should we adopt? Keep sending
> > patches to the mailing list
> 
> I definitely like the patches on the list. I find that with
> notmuch, I can maintain a queue of outstanding patches very
> effectively, (meaning, that the queue is usable and doesn't forget
> even if I do get very far behind like I am currently).

The reason why I set up patchwork is because you might have spent
hours triaging some patches already, e.g. bringing the count from
400 to 100. Since I cannot really "pull" your tags, I am forced to
go through the entire list myself.

> > master or pu (or even maint/next), as appropriate? Use the
> > Debian bug tracker? Use Sebastian's Roundup instance? Set up
> > a patch queue manager on notmuchmail.org? Use patchwork [1]?
>
> I'm personally not interested in any system that requires me to
> push any additional buttons outside of notmuch and git itself.
> I am interested in tools that can generate reports and help
> provide visibility into things. So patchwork fixed to
> automatically notice wh

[notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Servilio Afre Puentes  [2010.02.04.1714 
+1300]:
> In the second link, the links with the text "commitdiff" provide
> it, and you have the Atom and RSS feeds at the bottom.

As I see it, the feeds have links to the commitdiffs, but that's not
quite the same as having the diffs inline. You might be offline
without having pulled before, then the items from RSS/Atom are
useless.

If there is indeed an RSS/Atom feed on gitweb that includes the
diffs inline, please give me the URL; I can't fine ond.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"es ist immer etwas wahnsinn in der liebe.
 es ist aber auch immer etwas vernunft im wahnsinn."
 - friedrich nietzsche

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Servilio Afre Puentes  [2010.02.04.1918 
+1300]:
> > If there is indeed an RSS/Atom feed on gitweb that includes the
> > diffs inline, please give me the URL; I can't fine ond.
> 
> Can't see that even in the code.

Carl, could you set up notmuch-commits at notmuchmail.org and enable
the commit and enable the mail hook as per

  http://notmuchmail.org/pipermail/notmuch/2010/001373.html

Sorry to burden you, but since you want to continue maintaining the
infrastructure, that's just what I have to do. ;)

Thanks,

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

beware of bugs in the above code;
i have only proved it correct, not tried it.
-- donald e. knuth

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] patchwork now auto-updates patches from Git (was: Git feature branch)

[martin f krafft]

also sprach martin f krafft  [2010.02.04.1650 +1300]:
> - ? which brings me to my second point: there are certain things
>   that patchwork can do, at least in theory:
> 
>   * mark patches accepted when they hit your (canonical) master
> branch
>   * mark patches rfc when they hit e.g. my (canonical) next branch
>   * mark patches "under review" when they hit the all-patches (or
> pu) branch.
> 
>   I have not yet tried any of these, and I am basing this theory
>   only on the idea that git-patch-id can come to the rescue, for
>   there is no other linkage between the patch on the mailing list
>   (and thus known to patchwork), and the commit in the repo.

Patchwork now marks patches Accepted once they hit Carl's master
branch (up to 10 minutes delay due to Cron). It uses an algorithm
similar (but not equal) to git-patch-id in that it hashes the diff.
This means that the commit message can be amended when patches are
applied/cherry-picked, but the patch itself must be verbatim.

I ran it on all history thus far and it found 99 patches.

It'll be trivial to set it up to mark other states when the
corresponding commits hit another branch.

Let me know if there are any problems, and feedback welcome.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"if they can get you asking the wrong questions,
 they don't have to worry about answers."
 -- thomas pynchon

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100205/a05b53e1/attachment.pgp>

[notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Carl Worth  [2010.02.07.1156 +1300]:
> > Sorry to burden you, but since you want to continue maintaining
> > the infrastructure, that's just what I have to do. ;)
> 
> No burden at all. I don't really care to do everything alone.
> I just want to ensure we keep things centralized enough that
> people can actually find things easily.

This is what DNS was designed for, right? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

to err is human - to moo, bovine

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] hack to retag a directory

[martin f krafft]

also sprach David Bremner  [2010.02.07.1238 +1300]:
> I have to say that merge conflicts are not very much fun. I tend
> to do a certain amount of oh, take all the changes from the
> server.  I wonder if the approach that someone else mentioned of
> keeping a file tags/message-id with the appropriate tags in it
> might make merging less painful.

Merge conflicts at the file level are even less fun. This reminds me
of my plan to introduce .gitignore.d/*:

http://kerneltrap.org/mailarchive/git/2007/10/1/326425

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"there was silence for a moment, and then out of the scrambled mess
 of arthur's brain crawled some words."
 -- hitchhiker's guide to the galaxy

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

Re: [notmuch] Notmuch shared library

[martin f krafft]

also sprach Michal Sojka  [2010.04.01.1310 +0200]:
> this can be solved by the following patch, but I don't know how portable
> it is. You can see the efect of this by

Please avoid rpath. The better solution is probably to create
a wrapper for notmuch, which prepends to LD_LIBRARY_PATH.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
when everything is coming your way, you're in the wrong lane.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Notmuch shared library

[martin f krafft]

also sprach martin f krafft  [2010.04.01.1324 +0200]:
> Please avoid rpath. The better solution is probably to create
> a wrapper for notmuch, which prepends to LD_LIBRARY_PATH.

E.g. http://wiki.debian.org/RpathIssue

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
a common mistake that people make
when trying to design something completely foolproof
was to underestimate the ingenuity of complete fools.
 -- douglas adams, "mostly harmless"
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.08.1713 +0200]:
>I'm working on the solution - if the mailstore cannot open the
>message with the name passed, it tries different names with
>different maildir flags.

Wouldn't it be better to postpone synchronisation of the tags until
after emacs is done with the message?

I understand this might be hard to make work with mailstore
abstraction. Wouldn't it make more sense to have emacs call 'notmuch
cat', which returns the entire message, removes the unread tag,
changes the filename, and updates the database?

The message returned by cat would be stored in a temporary file for
use by the client (emacs). And if the message was needed again, you
could just search for it again.

I dislike the idea of heuristically probing a Maildir for files.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"i don't think so," said rene descartes. just then, he vanished.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.12.1347 +0200]:
> > Wouldn't it be better to postpone synchronisation of the tags
> > until after emacs is done with the message?
> 
> Theoretically, it would be possible, but if, for some reason, the
> synchronization step would not happen, then the tags in the
> database and in the mailstore will be inconsistent and next run of
> notmuch new would reset the tags according to the state in
> mailstore.

Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
transactional anyway. There are many other ways in which the
database and store can get out of sync. And you are about to add
another redundancy.

> The current implementation takes tags in mailstore as
> authoritative and ensures that tags in mailstore are always
> updated before tags in the database.

So why store them in the database at all?

> > I understand this might be hard to make work with mailstore
> > abstraction. Wouldn't it make more sense to have emacs call
> > 'notmuch cat', which returns the entire message, removes the
> > unread tag, changes the filename, and updates the database?
> 
> I do not like the fact that cat would do two things - cat and tag.
> And then, 'unread' tag is not the only one which can be changed.

I wouldn't want an unread tag in the first place, especially not
with Maildir semantics. In this case, what should really happen is:

  1. cat feeds a message to client
  2. client instructs notmuch to update tags
 - some tags require changes in the database
 - others require filename changes, which must be completed in
   unison with a database update so the new filename is stored.
  3. user asks to see attachment, which the client can fulfill using
 either a cached copy from (1.) in a tempfile, or by simply
 asking for the message again, via notmuch search.

> > The message returned by cat would be stored in a temporary file
> > for use by the client (emacs). And if the message was needed
> > again, you could just search for it again.
> > 
> > I dislike the idea of heuristically probing a Maildir for files.
> 
> Well, I do not plan to use wired heuristics. At the end there will
> be readdir() to traverse the cur/ directory to find the file with
> the same part before flags. Since the S flag will probably be the
> most frequent change, I may add one single test for added S flag
> before trying more expensive readdir().

What is the point of storing these tags in the Maildir anyway? If
you want to make this information (e.g. new, seen, unread) available
to MUAs accessing Maildir directly, keep in mind that the database
and mailstore will very quickly grow inconsistent until the next
notmuch-new run, e.g. as messages are moved, or flags ('F') are
added in a way that the notmuch database is not updated.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"friendships last when each friend thinks he has
 a slight superiority over the other."
   -- honoré de balzac
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Mail in git

[martin f krafft]

also sprach Stewart Smith  [2010.02.17.1107 +0100]:
> On Wed, 17 Feb 2010 11:21:51 +1100, Stewart Smith  
> wrote:
> > Using fast-import is interesting. Does it update the working tree? The
> > big thing I wanted to avoid was creating a working tree (another million
> > inodes being created is not ever what I need)
> > 
> > Also interesting is the mention of creating packs on the fly... this
> > could save the time in first writing the object and then packing it (as
> > my script does).
> > 
> > I'm going to play with this
> 
> and I did.

Has anyone worked on this since?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"one should never allow one's mind
 and one's foot to wander at the same time."
-- edward perkins (yes, the librarian)
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

patchwork

[martin f krafft]

Are people using http://patchwork.notmuchmail.org at all? The reason
of my asking is that it receives quite a lot of spam and I do not
need to invest this time for a service noone uses.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"'oh, that was easy,' says Man, and for an encore goes on to prove
 that black is white and gets himself killed on the next zebra
 crossing."
-- douglas adams, "the hitchhiker's guide to the galaxy"
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Notmuch performance (literally, in my case)

[martin f krafft]

also sprach martin f krafft  [2010.03.16.1900 +0100]:
> I use ext4 with data=ordered, and while notmuch is writing the
> Xapian database, most I/O stalls on the machine:
> 
>   - Firefox does not get any mouse events
>   - Vim blocks writing the viminfo file
>   - All disk operations queue for multiple seconds.
> 
> So no, ext4 is not a solution. Is it just me, or should no
> filesystem of this world be able to hog a system this badly? I think
> the culprit is the IO-scheduler.

I just wanted to send a little update on this. Even though the Linux I/O
scheduler performs abysmally during the Xapian database updates,
I can report two improvements, at least to my situation:

  1. The 3.0 kernel seems to be better, but I did not quantify this
 in any way, and I might just as well be wrong.

  2. http://bugs.debian.org/635768 explains the (also I/O-related)
 lockups we've seen. Micah offered the tip that the actual fault
 lies with the awesome WM.

Cheers,

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"writing a book about debian
 is like hitting a moving target
 with a champagne bottle cork."
 -- arky
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Jameson Graef Rollins  [2009.11.26.1901 
+1300]:
> I would really like to start using notmuch with emacs beyond just
> testing, but I really need to be able to handle/read/send mail with
> PGP/MIME encoded attachments.  Do folks have any suggestions on how to
> handle this?  Is there a separate emacs mode that people use for
> signing/verifying/{de,en}crypting mail buffers, or is this something
> that is going to have to be integrated into the notmuch mode?  I guess
> the notmuch-show mode at least will need to do some verifying and
> decrypting.

How about indexing GPG-encrypted messages?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"a scientist once wrote that all truth passes through three stages:
 first it is ridiculed, then violently opposed and eventually,
 accepted as self-evident."
   -- schopenhauer
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Carl Worth  [2009.12.08.2001 +1300]:
> One concept in notmuch (compared to sup) is to (eventually) avoid people
> having to go through that pain by their current mail client becoming
> "notmuch enabled". For me, I had always liked composing email in emacs,
> so I just have to add notmuch support to the existing emacs
> message-mode.
> 
> Hopefully people working on other email interfaces will do similar
> things, (would be great to have Anjal and Thunderbird get some notmuch
> support for example).
> 
> I definitely didn't like that with sup, to get all the global-search and
> tagging features one had to accept the curses UI as well.

I am a bit late to the party, but re: this thread [0], I would
suggest to go the way of a fuse filesystem. That's effectively
a daemon, but one which also bridges a chasm between notmuch and all
kinds of existing mail tools, including IMAP servers, by way of the
standard filesystem interface.

0. http://notmuchmail.org/pipermail/notmuch/2009/000782.html

I don't want to harp on this too much right now for I have not yet
fully understood notmuch, but the basic idea would be that you'd
have ~/mail provided by notmuch-fuse-daemon, and there'd be a tool
like notmuch-fuse-cli with which you can add virtual folders, e.g.

  notmuch-fuse-cli new debianmail 'from:debian OR to:debian'

and that would create ~/mail/debianmail with mode 555 (since you
cannot write the results of a search) containing a Maildir with all
messages matching the query.

The benefit of this would be that I could use mutt, evolution, or an
IMAP server, or vi and shell tools to manipulate my mail without any
modifications to those tools.

There could be a separate hierarchy for tags, e.g. ~/mail/TAGS/foo
and ~/mail/TAGS/bar/baz matching on explicit tags (and maybe
~/mail/TAGS/notmuch with mode 555 for implicit tags). Writing mail
to those directories effectively adds tags, unlinking removes them.
~/mail/TAGS/UNTAGGED holds untagged mail for easier reference.

In addition to all of this, fuse could be used to index new messages
directly as they are delivered into ~/mail, rather than running
'notmuch new' regularly.

These ideas are not new, and I've written about them before:

http://madduck.net/blog/2007.07.24:a-user-space-filesystem-for-mail-labeling/

notmuch seems an excellent base for implementing such a filesystem.
I will try to make time before LCA to get up to speed on fuse, then
maybe Carl and Micah and I (and whoever else will be in Wellington)
can hack this up in a few hours and over a few beers.

If this resonates, or you want to work on this too, let's hear from
you!

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"no problem is so formidable
 that you can't just walk away from it."
  -- c. schulz
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] notmuch and imap [musing, no code :)]

[martin f krafft]

also sprach David Bremner  [2009.12.17.0218 +1300]:
> I agree that the labels-in-headers approach has some nice
> advantages.  I haven't thought through merging of tag lists, but
> maybe that is no worse than other approaches.  One thing that
> worries me a bit is that notmuch updates tags often, and each of
> these updates would require rewriting the message, at least in the
> obvious implementation.

If we separated implicit and explicit tags, then that issue would
not exist, and from all I can tell, only the privacy issues, and the
risk of losing mail when files are rewritten persist.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
DISCLAIMER: this entire message is privileged communication, intended
for the sole use of its recipients only. If you read it even though
you know you aren't supposed to, you're a poopy-head.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Threading

[martin f krafft]

also sprach Carl Worth  [2009.12.11.0639 +1300]:
> On Wed, 9 Dec 2009 16:21:34 -0700, Mark Anderson  
> wrote:
> > I was wondering if there's a way in notmuch to group un-associated
> > threads into a single thread.
> 
> There's certainly nothing like that in notmuch currently.
> 
> Sup had user-level functionality in the interface for stitching
> messages into a single thread, and I definitely think that that
> doesn't make any sense.

Why doesn't it make sense? Mutt does it too, and stitching means
actually (re)writing In-Reply-To and References headers.

I think this is one of the most useful "productivity features" in
mutt.

I also think that threading is a preference thing. As Carl said in
a later message:

> Just this morning I sent a mail to the notmuch list, which was
> a reply, (and legitimately so), but also potentially of interest
> to everyone on the list, (since it was regarding a bug fix
> unrelated to the original topic of the thread I was replying to).
> 
> So I was stuck on whether I should break the thread or not, (at
> the sending end). I guess I could have just sent a quick "this is
> pushed" reply, and independently composed a separate message
> telling people about the fix.
> 
> I ended up keeping the threading intact in that case, (which
> I think is right).

I often thread forwarded messages (and their followups) with the
thread because all my information management currently is
thread-oriented.

I think being able to freely break and tie threads in a trivial way
is a definite plus!

> But I still have a hard time justifying user operations to
> manipulate threading. The whole point of threading is to make it
> faster to process and read messages. But manual operations like
> joining and splitting threads seem like the user just doing more
> work, and that *after* having read the messages. So that seems
> mostly backwards to me.

Reading is one thing. Information storage and organisation is
another. After a message is delivered (and read) to my mailbox, it's
really mine and I can (and should be able) to affix it and integrate
it into my organisational scheme any way I want, don't you think?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"if there's anything more important than my ego,
 i want it caught and shot now."
-- zaphod beeblebrox
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Mike Hommey  [2010.01.08.2106 +1300]:
> I'm in \o_ (though I won't be in Wellington). I've been thinking
> about a fuse filesystem on top of notmuch for a while.

Grand news to see you interested! A FUSE filesystem is <25 functions
to implement, and each function is basically an entity of its own
and thus highly parallisable. Once we agreed on a general mapping
between filesystem i/o and notmuch interaction, 25 of us can write
a function each and be done. How's that for collaboration? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"courage is not the absence of fear, but the decision
 that something else is more important than fear."
  -- ambrose redmoon
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Mike Hommey  [2010.01.08.2109 +1300]:
> That may leak decrypted form in the xapian index, though in
> a split manner. But that'd still be a problem IMHO.

Not for me, since the index is stored on encrypted media. Thus, this
should be off-by-default, but possible.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"academia is really just a way to help those with high volumes of
 nothing to say to social status."
 -- myself on #debian-devel, 01 Feb 2007
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] indexing encrypted messages (was: OpenPGP support)

[martin f krafft]

also sprach Ruben Pollan  [2010.01.08.2221 +1300]:
> I think that would be security hole. You should not store the
> encrypted messages on a decrypted database. A solution whould be
> to encrypt as well the xapian DB, but I think is too complex for
> the use.

As I said in <20100108091216.gc...@lapse.rw.madduck.net>, I think it
should be optionally possible for those that are encrypting the
xapian DB in other ways.

> You should be still able, with the actual notmuch, to search over
> the headers of your encrypted messages, or any other non-encrypted
> part of the message. Is not like that?

Most of the time, I search headers, but I do search bodies
regularly. So no, that would not be enough, at least not with the
ideal solution. And notmuch comes close to ideal already! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
infinite loop: see 'loop, infinite'.
loop, infinite: see 'infinite loop'.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Quick thoughts on a notmuch daemon

[martin f krafft]

also sprach Mike Hommey  [2010.01.08.2220 +1300]:
> FYI, I have a good experience writing fuse filesystems, both with
> high-level and low-level APIs. I'd avise to use the low-level API,
> which allows for better performance.

I don't have any experience with FUSE yet, but the examples in
/usr/share/doc/libfuse-dev/examples/ look trivial. This is where
I would start, one function at a time. If you have a better
suggestion, I'd love to hear it; or to clone your repo! ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
be careful of reading health books -- you might die of a misprint.
 -- mark twain
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] Idea for storing tags

[martin f krafft]

Folks, over in #notmuch, we just floated an idea that I'd like to
get out to you. We've been debating storing tags for messages.
Therefore I am cross-posting. Please forgive me.

So far, there are two approaches:

1. External database, which has the downside of not being
   synchronisable with standard IMAP, like the rest of your mail
   (assuming you use IMAP). Also, it's possible for mailstore and
   database to get out of sync.

2. In-headers, which has the downside of leaking (e.g. when
   bouncing), and incurs the risks associated with message rewrites
   (which I think is pretty much ignorable, but it's still there).
   Also, there's a performance issue, but in the context of an
   indexer like notmuch, this is negligible.

   The leakage is real, though and I think it makes in-headers
   unusable. After all, I don't ever want anyone else to know that
   I tag e-mails from my boss as "from-idiots", and I forward and
   bounce mail on a regular basis. I could tell my MTA to remove
   those headers, but I might forget to do that on a new system.

We also previously determined that IMAP keywords are pretty much
useless as they are stored per mailbox, not per message, not
standardised, and limited in their length anyway [0]. This also
means that we don't really need to investigate sensibly storing tags
in Maildir (e.g. with xattrs), because IMAP cannot transport them.

0. http://lists.madduck.net/pipermail/mailtags/2007-August/msg00016.html

Seriously, who implemented IMAPv4rev1 and what sort of crack were
they smoking??

I remember there was some KDE groupware contacts manager that used
IMAP to synchronise contacts. At first, this sounds horrible, but
when you detach IMAP from RFC822, it becomes a generic synchronising
protocol. The next step is then straight forward, and I want to
share this idea with you:

How about using pseudo-mails stored in Maildir and synchronised by
IMAP? E.g. every folder could have a subfolder .TAGS and if we find
a way to smartly pair messages between parent and subfolder, we'd
have a tag store alongside the mailstore it refers to, but without
the danger of leakage, and without having to rewrite messages.

The major problem with this is when clients don't understand this
"protocol", for then they will display all .TAGS folders as regular
IMAP folders, and try to treat the messages therein as regular
mails. Somewhere sometime this is bound to blow up and I don't
really know how to prevent that.

Anyway, the idea is out now. Thoughts?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
echo Prpv a\'rfg cnf har cvcr | tr Pacfghnrvp Cnpstuaeic
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Idea for storing tags

[martin f krafft]

also sprach Scott Robinson  [2010.01.12.1644 +1300]:
> I wrote a script to store and sync my tags.
> 
>   * One filename per message-ID.
>   * Line-feed seperated tags in each file.
> 
> Then the whole structure is controlled via git.
> Conflict-resolution and sync comes for free.

How do you ensure that the external tag store and your mail store do
not go out of sync? I assume that mails without a tagfile are simply
untagged, so that's hardly the issue. However, if you delete a mail,
how do you ensure that the tag database is cleaned up?

Also, do you attach tags automatically, e.g. with procmail on the
server? If so, how do you initiate git-pull locally?

Would you consider sharing your script?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"alle vorurteile kommen aus den eingeweiden."
 - friedrich nietzsche
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Scott Robinson  [2010.01.12.1644 +1300]:
> Then the whole structure is controlled via git.
> Conflict-resolution and sync comes for free.

I've just had a good think about this, also because the idea of
abandoning IMAP and using Git has been around for a while and
I have not really wrapped my head around it.

If the MDA delivers to Git, then potentially, you might get into
a situation where you cannot write your own changes back to the
repo. This is also a DoS scenario: I'll just keep sending you
e-mail, and if I manage to pass your mail filters, I'll basically
commit to your mail repository at regular intervals. Say those are
5 seconds. In order for you to write updates to the repo, e.g. to
update tags, then you would need to pull, rebase, and push all
within 5 seconds, for otherwise you'd try to push non-fast-forwards.

This a bit unrealistic, surely, but there's a real annoyance in it:
you'd have to pull/rebase/push until a push succeeds — until you
found a time window between pull and push during which the MDA
didn't write to the repo. This might take a long time. If this
happens in the background by Cron, it's not a real concern, but if
this becomes a UI issue, I wouldn't know how to handle it.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
don't hate yourself in the morning -- sleep till noon.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Jameson Rollins  [2010.01.13.0838 
+1300]:
> What about if just the tag information is stored in the
> repository, and not the mail itself?  In that case only the user
> would be pushing into the repo and you wouldn't have to worry
> about the DoS scenario.

I certainly would like the ability to have messages
automatically-tagged on delivery, by procmail.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
may the bluebird of happiness twiddle your bits.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Idea for storing tags

[martin f krafft]

also sprach Scott Morrison  [2010.01.12.1711 +1300]:
> 1.  synchronization of tag data with emails -- if they are in
> a subfolder then it presents the issue of maintaining this
> subfolder when managing emails (moving, deleting, duplicating etc)
> and any .tag folder unaware clients are likely cause an breakage
> in tagdata/message association.  One way of doing this is to have
> a global .tag folder.

A global .tag folder indexed by e.g. message ID, as you state later,
would probably allow for this. Or a file-per-tag design. We'd have
to think carefully about pros and cons for each.

When thinking about this, I always have to remind myself that we are
targetting this at a design that has indexed search. If that weren't
the case, searches would be incredibly expensive.

Maybe a better approach would be content addressing (see below).

> 2. what happens if that message is archived or moved to an
> exclusively local cache -- eg. Mail.app on OS X can easily move
> IMAP messages to a folder resident on the computers computers?

Well, if the target can store tags, then ideally the MUA should know
how to transfer them along.

Maybe the right thing to do would be to use extended attributes
(which are stored in the inode!), even if they may not be
universally supported yet. If our solution scales, then this might
lead to a significant increase in xattr adoption.

> 3. what happens with duplicates of emails -- I would assume that
> the message id would be the key to match the tag data to the
> message.  In this system a duplicate of a message could not have
> a different set of tags from the original (not that this would
> necessarily be desirable.)

Duplicates need folders, and tags and folders are somewhat at odds
with each other. I mean, you can represent a folder hierarchy with
tags (and more), and if you have tags and folders, you are
potentially introducing a level of confusion/ambiguity that we don't
want in the first place. Maybe the ideal solution doesn't need
folders anymore (and IMAP-compatible (Maildir) subfolders have
always been a hack anyway).

There are also two types of duplicates: copies and links. The former
can diverge, the latter can't. I don't really see a reason for
either. It's not like you need to copy a mail before you edit it,
and I don't see a real reason for linking, assuming that the primary
means of browsing will be tag-searches anyway.

Duplicates always make me think of content addressing, like Git's
object cache. We could store the content hash of a message in its
filename, and also use the hash to index into the tag database.
I think that would be much cleaner than message IDs, and would make
handling true duplicates (links) much easier, while copies (diverged
ex-duplicates) would also be taken care of automatically.

> Your mention of potential leakage (aka inadvertent disclosure of
> tag data) is real -- but only if the client used to bounce/forward
> is not the one to tag the message (one would assume that if
> a client can tag, it can know to exclude the tags in a bounce.)

True, and it's probably the minority of people using multiple
clients. But those who do might also manipulate mail with sed and
use sendmail directly.

I don't think we can successfully enhance RFC 5351 to make MTAs
always ditch the Tags:-header.

> Mail.app -- which I am pluging into does not forward headers --

ew! ;) (I think one should be able to forward pristine mails)

> though it will include all headers in a bounce -- but chance are
> you aren't tagging messages you are bouncing.:)

That chance might well be very low. I bounce/forward-as-attachment
a lot of mail from the past to make it easier for others to
establish context.

> The performance issue is very real -- because it means that
> somehow messages have to rewritten to the IMAP server -- IMAP
> doesn't have a mechanism AFAIK for updates.

Not even UIDPLUS?
http://wiki.dovecot.org/FeatUIDPLUS

> Additionally, IMAP doesn't have a mechanism for simply replacing
> one message data with another -- a new message must be written and
> the old message must be deleted and the message IMAP UID will
> change, and the client will have to deal with this especially if
> it is cache the messages.

Yes, I am experiencing this pain regularly, since I currently use
a lot of message rewriting as part of my workflow — one of the
reasons why I'd like to find an alternative.

> Also GMAIL IMAP is an issue-

Yeah, I bet. Is there anyone who doesn't think that that's Google's
problem, not ours, though?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"there's someone in my head but it's not me."
-- pink floyd, the dark side of the moon, 1972
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Idea for storing tags

[martin f krafft]

also sprach Scott Morrison  [2010.01.13.1752 +1300]:
> The problem with anything that is not universally supported is
> that for a package that is to appeal to a wide userbase, most
> don't know and don't care about the particulars of this IMAP
> server vs that IMAP server.  all they know it that for some reason
> it doesn't work with account X -- which leads to support head
> aches.
[...]
> Call it Googles problem as you like -- but when I have a product
> that doesn't work with GMAIL IMAP there are a lot of potential
> users that don't care about server peculiarities and rather just
> have it work.

Well, the way I see it: you cannot change all IMAP servers at once,
and you certainly cannot change Google. If it's possible to
implement tagging for email (dare say semantic e-mail) with standard
means (where standard means sub-standard, as exemplified by your
previous GMail IMAP example), then that's the best way, but if that
can't happen then we ought to try a better way. Should we find
a solution then, by the rate of standardisation on the 'Net, maybe
my grandchildren will finally be able to do proper e-mail. ;)

> I agree that conceptually duplicates should be buried but end
> users do have "peculiar" organization systems.

I think tags should help abstract e-mail away from underlying
storage and I'd love that to be a goal.

> From my reading, uidplus doesn't allow a delta modification of
> a message on a server -- just to write a portion of a message back
> -- you still have to write the whole thing back and that can mean
> real bandwidth issues for some messages.

Absolutely. It would indeed be better if you could just send
changes.

I just sent a blank mail to
imap-protocol-subscr...@mailman.u.washington.edu
and have started browsing the archives. So far, there's not really
anything relevant.

Anyway, looking back at the RFC on keywords, it's not exactly
encouraging:

  A keyword is defined by the server implementation. Keywords do not
  begin with "\". Servers MAY permit the client to define new
  keywords in the mailbox (see the description of the PERMANENTFLAGS
  response code for more information).

Anyway, I'll try to untangle the various issues re:IMAP we've been
seeing, write mails for each, and hopefully get to the point where
I can enquire about IMAPv5. ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
the unix philosophy basically involves
giving you enough rope to hang yourself.
and then some more, just to be sure.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Idea for storing tags

[martin f krafft]

also sprach Carl Worth  [2010.01.14.1432 +1300]:
> Yes. This approach requires some external means of synchronizing the
> tags from one system to another.
> 
> I don't understand what it would mean to have the mailstore and the
> database out of synch here. This approach doesn't have the tags in the
> mailstore by definition, right?

You might have marked a message 'read' on one machine and if the two
get out of sync on another machine, you might have the same message
unread there.

> > How about using pseudo-mails stored in Maildir and synchronised by
> > IMAP? E.g. every folder could have a subfolder .TAGS and if we find
> > a way to smartly pair messages between parent and subfolder, we'd
> > have a tag store alongside the mailstore it refers to, but without
> > the danger of leakage, and without having to rewrite messages.
> ...
> > Anyway, the idea is out now. Thoughts?
> 
> There are a couple of problems that I don't see addressed at all with
> this approach. The first is that there's not a one-to-one mapping
> between messages and files in the mail store. (I'm CCed on a lot of list
> mail meaning that I have multiple files in my mail store for a single
> message.)

Shouldn't this just be solved? I've had formail+procmail delete my
duplicates for 10+ years, and while I don't like the fact that
I usually get the CC before the list mail, and thus cannot filter on
Delivered-To, I have never looked back.

> Second, the only reason I would be interested in synchronizing mail
> between two systems is so that I could manipulate the tag data in
> multiple places, (that is, remove the "unread" tag whether on my
> network-disconnected laptop or via web-mail when away from my
> laptop). Using imap for synchronizing a file of tags within the mail
> store gives you no mechanism for doing any sort of conflict resolution,
> right? (Which I think in almost all cases is going to be quite trivial
> if there's a chance for a program to resolve it.)

I have not thought about this, but you are right. IMAP does not
really allow for conflict resolution, which may well be *the* reason
why you cannot update existing messages.

> [*] Though, I think a plain-text file with tags managed with
> something like git (and perhaps a custom merger) could save a lot
> of work. Or perhaps a plain-text journal of tag manipulations on
> either end that could be replayed on the other.

Git is good at conflict resolution if run interactively, but [0]
still makes me question whether it can ever take the place of IMAP.
However, Asheesh Laroia, who has floated the idea of Git-for-mail at
DebConf8 already, has some ideas and hopefully will soon reply to my
mail [0], which I just bounced.

0. http://notmuchmail.org/pipermail/notmuch/2010/001114.html

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
apt-get source --compile gentoo
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Asheesh Laroia  [2010.01.14.2112 +1300]:
> Sure. But the MDA doesn't need to do the commit immediately. Since
> (presumably) we're using Maildir, the MDA on the mail receiving
> server is going to generate filenames that won't cause conflicts.
> So it's okay to leave the files uncommitted.

So when does the commit happen?

> When I did the "git merge", git would create the Maildir files in
> ~/Maildir/cur/... non-atomically.

This might be something that the Git people could address if it was
brought up on the mailing list. Then again, it might not be possible
without going via a temporary file, which I doubt will fly.

I suppose that I never actually considered merges on the IMAP server
side, but obviously the IMAP server has to work off a clone, and
that means it needs to merge.

> Dovecot would notice the file in ~/Maildir/cur/ and think, "This
> file must be ready!" So it would parse it even though git hadn't
> finished writing it. This caused me to only see partial headers in
> Alpine since Dovecot parsed it before it was a complete message.

I wonder if a custom merge driver could address this to properly use
…/tmp/ to assemble the message and only then move it.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"this week dragged past me so slowly;
 the days fell on their knees..."
-- david bowie
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Idea for storing tags

[martin f krafft]

also sprach Carl Worth  [2010.01.15.1124 +1300]:
> > You might have marked a message 'read' on one machine and if the two
> > get out of sync on another machine, you might have the same message
> > unread there.
> 
> That's a different issue though. With two databases there's clearly the
> opportunity for the two databases to be out of synch.
> 
> But you talked about the database being out of synch with respect to the
> mailstore. And that's something I just don't understand, (given the
> assumption that all tags are stored in the database---which was the
> explicit description of the case of interest).

Yes, we are talking about the situation where the tagstore is
seperate from the mailstore, and that they are both synchronised
with a server, or between machines, separately. If for some reason
you only synchronise the mailstore — say because the connection
drops before the sync of the tagstore completes — then you end up
with an out-of-sync situation, because the mailstore-sync will have
pulled in a new message, but not the associated tags. So if you had
already read this message on another machine and tagged it 'done',
then it would show up on this machine as 'new' without the 'done'
tag, because the tags were not synchronised.

The only way to really solve this is by transferring a message and
its tags in a transactional way.

> > Shouldn't this just be solved? I've had formail+procmail delete my
> > duplicates for 10+ years, and while I don't like the fact that
> > I usually get the CC before the list mail, and thus cannot filter on
> > Delivered-To, I have never looked back.
> 
> Notmuch has access to all the information it needs to allow you to
> delete the CC version once the list mail arrives. So you could do
> notmuch-based deletion now and avoid losing the Delivered-To header if
> you want.

Of course. I hadn't thought that far.

However, there are still benefits to formail, namely avoiding having
to run duplicates through potentially expensive spamfilters.

> I think that synchronizing the mail store and synchronizing the
> tags information are tasks that have different requirements, and
> for which we may well want different tools.

Fair enough. Maybe I am just paranoid about the stores getting out
of sync (see above).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"we all know linux is great...
 it does infinite loops in 5 seconds."
 -- linus torvalds
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Threading

[martin f krafft]

also sprach Carl Worth  [2010.01.15.1108 +1300]:
> > Reading is one thing. Information storage and organisation is
> > another. After a message is delivered (and read) to my mailbox,
> > it's really mine and I can (and should be able) to affix it and
> > integrate it into my organisational scheme any way I want, don't
> > you think?
> 
> A fair point.
> 
> I don't see this being something I'm going to spend any time
> implementing. I just wouldn't use the functionality myself. But
> I would be happy to integrate patches if someone came up with
> some.

Maybe I should try to persuade you in person.

Just today I referenced a discussion I had with a client's ISP,
which was done via a web-based support system (custhelp.com). They
send you e-mail for every post you or they make to the thread, but
those e-mails do not reference each other. Fortunately, I stitched
them together and when I searched for the correspondence in my
mailstore, I had the entire thread available to me, which was handy
(thanks to mutt's useful thread handling abilities).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"this week dragged past me so slowly;
 the days fell on their knees..."
-- david bowie
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Thoughts on notmuch and Lua

[martin f krafft]

also sprach Carl Worth  [2010.01.15.1200 +1300]:
> > Lua has many advantages over other scripting languages when it
> > comes to integration with a C program. It has a very clean and
> > easy C API, the overhead of running Lua scripts is not noticable
> > among other things.
> 
> I've definitely heard lots of good things about "lua
> embedability". So if we do decide to provide hooks, then lua would
> seem like a logical option to look at first. I've never looked at
> it closely myself though.

Lua for hooks has the advantage that the hooks can be executed in
the context of manipulateable objects. On the other hand, hooks in
the style of run-parts directories are more flexible and accessible,
and could always be invoked as filters for the manipulateable data.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"imagine if every thursday your shoes exploded if you
 tied them the usual way. this happens to us all the time
 with computers, and nobody thinks of complaining."
-- jeff raskin
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] Git as notmuch object store (was: Potential problem using Git for mail)

[martin f krafft]

also sprach Asheesh Laroia  [2010.01.21.1928 +1300]:
> >I suppose that I never actually considered merges on the IMAP
> >server side, but obviously the IMAP server has to work off a clone,
> >and that means it needs to merge.
> 
> It's not "merge" that's unsafe; that just builds a tree in the git
> index (assuming no conflicts). It's the ensuing process of git
> writing a tree to the filesystem that is problematic.

There is no way to make that atomic, I am afraid. As you say.

> I could probably actually write a wrapper that locks the Maildir
> while git is operating. It would probably be specific to each IMAP
> server.

Ouch! I'd really rather not go there.

> Note that this mean git is fundamentally incompatible with
> Maildir, not just IMAP servers.

We had an idea about using Git to replace IMAP altogether, along
with making notmuch use a bare Git repository as object store. The
idea is that notmuch uses low-level Git commands to access the .git
repository (from which you can still checkout a tree tying the blobs
into a Maildir). The benefit would be compression, lower inode count
(due to packs), and backups using clones/merges.

You could either have the MDA write to a Git repo on the server side
and use git packs to download mail to a local clone, or one could
have e.g. offlineimap grow a Git storage backend. The interface to
notmuch would be the same.

If we used this, all the rename and delete code would be refactored
into Git and could be removed from notmuch. In addition, notmuch
could actually use Git tree objects to represent the results of
searches, and you could checkout these trees. However, deleting
messages from search results would not have any effect on the
message or its existence in other search results, much like what
happens with mairix nowadays.

I think we all kinda agreed that the Maildir flags should not be
used by notmuch and that things like Sebastian's notmuchsync should
be used if people wanted flags represented in Maildir filenames.

Instead of a Maildir checkout, notmuch could provide an interface to
browse the store contents in a way that could make it accessible to
mutt. The argument is that with 'notmuch {ls,cat,rm,…}', a mutt
backend could be trivially written. I am not sure about that, but
it's worth a try.

But there are still good reasons why you'd want to have IMAP
capability too, e.g. Webmail. Given the atomicity problems that come
from Git, maybe an IMAP server reading from the Git store would make
sense.

However, this all sounds like a lot of NIH and reinvention. It's
a bit like the marriage between the hypothetical Maildir2 and Git,
which is definitely worth pursuing. Before we embark on any of this,
however, we'd need to define the way in which Git stores mail.

Stewart, you've worked most on this so far. Would you like to share
your thoughts?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"reife des mannes, das ist es,
 den ernst wiedergefunden zu haben, den
 man hatte als kind beim spiel."
-- friedrich nietzsche
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git as notmuch object store (was: Potential problem using Git for mail)

[martin f krafft]

also sprach Asheesh Laroia  [2010.01.25.1819 +1300]:
> You say "Ouch" but you should know Dovecot *already* does this. I
> don't mind interoperating with that.
>
> See http://wiki.dovecot.org/MailboxFormat/Maildir, section "Issues
> with the specification", subsection "Locking". I term this theQ
> famous readdir() race.

Yikes. IMAP (including dovecot) just SUCKS.

> Without this lock, Maildir is fundamentally incompatible with IMAP
> -- one Maildir-using process modifying message flags could make
> a different Maildir-using process think said message is actually
> deleted. In the case of temporary disappearing mails in Mutt
> locally, that's not the end of the world. For IMAP, it will make
> the IMAP daemon (one of the Maildir-using processes) send a note
> to IMAP clients saying that the message has been deleted and
> expunged.
[…]
> Just don't fall into the trap of thinking Maildir is compatible
> with IMAP. It's not, because as I understand things, the
> filesystem doesn't guarantee that you can actually iterate across
> a directory's files if another process is modifying the list of
> files.

This is all perfect reason to concentrate even more on designing
a store that could potentially make IMAP obsolete once and for all!

The current idea is to sync Git downstream only, and find a way to
keep multiple copies of a tagstore in sync, by way of the "server
instance" (where mail is received/delivered). Deleting messages
would then be something like setting the notmuch::deleted tag, which
clients would honour; on the server, a cleanup process would run
regularly to actually delete the blobs associated with deleted
messages. This would then propogate the next time one pulls from
Git.

Whether to store history (commit objects) or just collections (tree
objects) needs to be investigated.

> >But there are still good reasons why you'd want to have IMAP
> >capability too, e.g. Webmail. Given the atomicity problems that
> >come from Git, maybe an IMAP server reading from the Git store
> >would make sense.
> 
> It wouldn't be too hard to write a FUSE filesystem that presented
> an interface to a Git repository that didn't allow the contents of
> files to be modified. Then Dovecot could think it's interacting
> with the filesystem.

Yes, a FUSE layer (which adds a daemon), or a lightweight access
API via libnotmuch. Probably the former using the latter. ;)

> Aww, I like Maildir flags, but if there's a sync tool, I'm fine
> with that.
[…]
> I'm not sure, but maybe it's safe if you refuse to ever modify
> a message's flags in the filename.

The main point is that there is nothing really in Maildir filenames
that you couldn't equally (and possibly better) represent in the
notmuch::* tag namespace, and then there is benefit in only having
one used primarily (which means notmuchsync can do whatever it
wants without affecting or messing with notmuch).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"if I can't dance, i don't want to be part of your revolution."
- emma goldman
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git as notmuch object store (was: Potential problem using Git for mail)

[martin f krafft]

also sprach Sebastian Spaeth  [2010.01.26.0249 +1300]:
> While notmuchsync fullfils my needs, it is a kludge. It needs to
> call "notmuch" for each mail where a MailDir flag has changed
> (which can be quite often on an initial run, where most mails are
> likely to be read), this can take a long, long time. It would
> makes sense IMHO to at least pick pioto's "don't set unread if 'S'
> flag is set" on notmuch new[1].

I am sure this could be implemented with libnotmuch if it proves to
be useful.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"it isn't pollution that's harming the environment.
 it's the impurities in our air and water that are doing it."
  - dan quayle
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] proposal for more streamlined mail flow in notmuch

[martin f krafft]

also sprach Jameson Rollins  [2010.01.17.0949 
+1300]:
> I would like to put forth here a proposal for a couple of changes
> to notmuch that I believe will considerably streamline message
> handling and new message flow through notmuch.  Notmuch is still
> new and I believe it hasn't quite figured out the best way to
> handle message flow in the new mail handling paradigm that it is
> working.  This is a proposal to fix that.
>
> I believe that most people are syncing their mail from remote IMAP or
> POP servers to local maildirs (via offlineimap for instance), and that
> most people's IMAP servers have limited storage capacity.  This
> practically means that people can not keep all of their mail in a
> single directory.  However, notmuch currently basically assumes that
> this is what is happening.  In order to keep synced maildirs from
> growing out of hand, messages need to be either deleted or moved out
> of the "inbox" where they initially show up.

We should keep this in mind when designing an object store, whether
or not that's based on Git. If we use Git, then a local branch is
all we really need to support this. \o/

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
above all, we should not wish to divest
our existence of its rich ambiguity.
 --friedrich nietzsche
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git feature branch

[martin f krafft]

also sprach Carl Worth  [2010.01.23.1010 +1300]:
> Anyway, I'll be on vacation for the next few days, so will likely
> not have much, (likely have not much?), time for patch merging.
> 
> But I *am* anxious to get back to the backlog. And in the
> meantime, I really appreciate others merging and sharing patches.

I discussed this with Carl at LCA a bit and ideally we should come
up with a way to relieve Carl of the bottleneck burden (obviously
without stealing away his dictator hat ;)

I think it would make sense to move the mainline to git.debian.org
for now, or another place where everyone can easily get an account.
As alternatives I propose repo.or.cz. I'd prefer to stay away from
commercial services like Github.

Once we're there, how about copying the branch structure from
Git development[0]:

  maint/— the stable release
  master/   — the stablising head
  next/ — testing branch
  pu/   — patch integration branch (proposed updates)

Each of the four branches is usually a direct descendant of the one
above it. Conceptually, the feature enters at an unstable branch
(usually 'next' or 'pu'), and "graduates" to 'master' for the next
release once it is considered stable enough.

0. http://repo.or.cz/w/git.git/blob/HEAD:/Documentation/gitworkflows.txt

Sebastian, would it make sense to migrate your work into a 'pu'
branch?

What patch tracking workflow should we adopt? Keep sending patches
to the mailing list and have a few people who integrate them into
master or pu (or even maint/next), as appropriate? Use the Debian
bug tracker? Use Sebastian's Roundup instance? Set up a patch queue
manager on notmuchmail.org? Use patchwork [1]?

Cheers,

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"in all unimportant matters, style, not sincerity, is the essential.
 in all important matters, style, not sincerity, is the essential."
-- oscar wilde
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git feature branch

[martin f krafft]

also sprach micah anderson  [2010.01.27.1124 +1300]:
> Couldn't all of this be done without moving the existing git
> repository (don't forget that transition is a cost)? Those who
> wish to put together these proposed branches go ahead and do so,
> publishing those wherever they like (git.debian.org, gitorious,
> their own hosted git repositories, or even github) and then Carl
> can just add those as remotes as he sees fit.

I brought this up, so I should make the motivation explicit: I just
tried to relieve Carl from the burden of bottleneck. Since passing
out SSH accounts on his own machine does not really scale, I looked
elsewhere.

However, I agree that the easiest solution is just to add more
users. Potentially, Carl can make use of gitolite, I can investigate
that some time this week.

> Personally, I've found mailing lists that have patches sent to
> them tends to totally kill the list for anything else. It seems
> a bit weird to use Debian's bug tracker for a non-Debian native
> program (but using it for the Debian package of notmuch does make
> sense). I am not so familiar with Roundup, patch queue trackers or
> patchwork to have anything to say about those.

patchwork integrates with the mailing list and slurps patches and
related discussion and threads them into a webpage, where they can
be workflow-managed.

The Debian bug tracker has the benefit of being usable with e-mail
(and this is notmuch we're developing, don't forget). The others are
all exclusively web-based, with the exception of launchpad, AFAIK.

The idea of having a tracker is quite simply to make it easier for
Carl to stay on top, and for e.g. you and I to assist him by vetting
patches in our free minutes.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
there's an old proverb that says just about whatever you want it to.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Jameson Rollins  [2010.01.26.1046 
+1300]:
> > For example, I might have:
> > 
> > ~/.notmuch-config:
> > 
> > [database]
> > path=/home/pioto/mail
> > ...
> > [tags]
> > pi...@pioto.org/INBOX.ListMail.notmuch = notmuch
> > 
> > So, a 'tags' section, where each key is the folder name, relative to the
> > db path, and the value is one or more tag names
> 
> I think this idea is a really good one and I would like to pursue it as
> a tangent thread here.  I was going to propose something very similar to
> this.  I think it's a very flexible idea that would help in a lot of
> ways.

I think we need to carefully distinguish here. The above seems to
suggest a mapping from folder to tag, but we don't actually need
tags for folder locations, because those can (and should) be
implicitly determined from the database and storing the tag in
addition would just run the risk of getting out of sync: if I moved
a message, I would also have to remember to delete old and add new
tags, which is just asking for trouble.

> [tags]
> inbox = +inbox,+unread
> sent = +sent
> drafts = +draft
> archive = -inbox

This proposal, on the other hand, is an interesting one, but when is
it supposed to happen? It just feels wrong to make this happen as
part of 'notmuch new'.

What I would like to see is a notmuch-aware MDA, e.g. a programme
which reads an incoming mail on stdin and can do all this kind of
stuff, e.g. assign tags based on such rules (or take tags as
arguments, so that I could trivially set tags from procmail too),
write the message to the message store, and update the database.

This would allow us to get rid of 'notmuch new' altogether, at least
conceptually. We'd still need it if mail is being delivered
independently, e.g. with offlineimap.

On the performance side, it might make sense to write to a journal
instead of updating the database every time. SpamAssassin does this
with its Bayesian database, and it only merges the journal every
X updates (or when the user manually requests it). I am not sure
whether this is possible with Xapian. On the other hand, I think
notmuch needs to learn to journal anyway so that we can keep
different instances in sync.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"the only way to get rid of a temptation is to yield to it."
-- oscar wilde
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Jameson Rollins  [2010.01.27.0603 
+1300]:
> > This is getting involved. 
> > 
> > Maybe I'm missing something in this thread; but, why couldn't these complex 
> > and
> > context-sensitive decisions be delegated to sub-processes? ala git hooks?
> 
> I think this idea is completely independent of anything having to do
> with using git as a mail store.  That's why I was trying to separate it
> into a separate thread.

I think he meant "notmuch hooks like you have hooks for Git too",
e.g. thread:755741d13573c7642761d2a175cb146d

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"if i am occasionally a little overdressed, i make up for it by being
 always immensely over-educated."
-- oscar wilde
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach James Westby  [2010.01.28.1828 +1300]:
> Are you trying to use thread: such that it could be passed to
> notmuch show to see the conversation?
> 
> That's not going to work so well if so.

Why not? Works fine for me with the vim plugin...

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"perfection is achieved, not when there is nothing more to add, but
 when there is nothing left to take away."
 -- antoine de saint-exupéry
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach martin f krafft  [2010.01.28.1834 +1300]:
> > That's not going to work so well if so.
> 
> Why not? Works fine for me with the vim plugin...

Now I get it. I was talking about
id:20100114084713.ga22...@harikalardiyari

Sorry, I *am* new to notmuch ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"when zarathustra was alone... he said to his heart: 'could it be
 possible! this old saint in the forest hath not yet heard of it, that
 god is dead!'"
 - friedrich nietzsche
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Servilio Afre Puentes  [2010.01.29.0132 +1300]:
> >> [tags]
> >> inbox = +inbox,+unread
> >> sent = +sent
> >> drafts = +draft
> >> archive = -inbox
> >
> > This proposal, on the other hand, is an interesting one, but when is
> > it supposed to happen? It just feels wrong to make this happen as
> > part of 'notmuch new'.
> 
> Why so?

I guess I just dislike having to run notmuch new regularly, rather
than integrating the database more closely with the mail flow.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"to get back my youth i would do anything in the world, except take
 exercise, get up early, or be respectable."
-- oscar wilde
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] tag dir proposal [was: Re: Git as notmuch object store]

[martin f krafft]

also sprach Ben Gamari  [2010.01.29.0949 +1300]:
> > I guess I just dislike having to run notmuch new regularly,
> > rather than integrating the database more closely with the mail
> > flow.
> > 
> Sounds like you need to add a line to crontab.

It still feels like a hack. It's a bit like making many changes to
a source code repository (new mails get delivered) and committing
only once every hour (notmuch new), rather than making and
committing transactional changes (delivering and catalogueing mails
individually).

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
a Hooloovoo is a superintelligent shade of the color blue.
-- douglas adams, "the hitchhiker's guide to the galaxy"
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Some thoughts about notmuch sync with other agents

[martin f krafft]

also sprach Paul R  [2010.01.28.0316 +1300]:
> As you see, I advocate a NotMuch <-> IMAP synchronisation ASAP :)

Given the limitations of IMAP (non-transactional, non-standard
keywords implementation, …), I think chances of this are rapidly
diminishing, but I'd love to be proven wrong.

> First of all, processing mail with MUA involves some simple logic that
> is shared by most MUA. This is about receiving *new* mails, *reading*
> mail, *replying* to mail and so on... IMHO, this really belongs to the
> mail processing logic and not to the user logic. Hence my first
> request :
> 
>   1: Don't use user tags space to store MUA flags.
> 
>  That means no more "seen", "unread", "replied" as tags. These are
>  MUA processing *flags*, that must belong to an established set.
>  Tags, on the other hand, are user-land information. So no more
>  [seen, replied, grandma, important] tag sets :)

I disagree.

The MUA actually doesn't (shouldn't) care at all about any of these
flags, at least not for core functionality. Sure, a MUA should
probably hide messages tagged 'deleted', and it would be nice if it
could be configured to highlight messages tagged 'important', but
none of the others — "seen", "unread", "replied", … — have any role
in the core functionality. They are purely user-specific.

They are leftovers from days when some MUA designer decided that
having these flags would be a useful way to organise e-mail
handling, but that person probably dealt with a dozen messages
a day, and didn't have half his/her life organised through
electronic mail.

Point being, times and needs have changed. And while we're in the
process of finding the technology that can suit those needs (in the
most generic way), we might just as well (and should) rid ourselves
from these leftovers.

Any solution must be generic enough so that if you rely on the
functionality provided by these tags, you can trivially make it
happen, e.g. with hooks to add/remove flags on certain actions (such
as sending a reply, or reading a message).

Neither IMAP nor Maildir is capable of storing an extensible,
freely-configurable set of tags (keywords). Therefore, we need a new
method anyway.

> Once this is done, notmuch will know, in a robust a predictable
> way, what happened to a mail. Simply put, NotMuch will store and
> expose MUA flags (Passed, Replied, Seen, Trashed, Draft, and
> Flagged [1]). For each , notmuch should associate
> a _synced flag. When changing  from notmuch, it should
> set the _synced bit to 0. These are MUA mail flags.

I think the semantics would be clearer the other way around: setting
*_changed when a flag is changed.

> Additionally, in a third « space », notmuch should store its « new »
> bit, as well as a « missing » bit probably. Again, this is neither MUA
> logic or user logic, so this should not interfer with user
> classification facility provided by tags, nor with MUA flags. It,
> really, is something else, let's name it "status".

Or "lifetime".

> Once this is done, the 'notmuch new' command should find new mails
> and set the 'new' bit for them, and find the missing mails and set
> the 'missing' bit for them. This will allow for robust external
> processing.

Why would I want to keep around a record in the database when the
physical file is no longer present?

> # 1/ Sync from NotMuch to MailDir
> 
> notmuch list flags:(seen and not seen_synced) 
>   | notmuch-maildir --mark-mail seen
>   | notmuch move --stdin
>   | notmuch set flags:+seen_synced --stdin

Have you seen/look at notmuchsync?

> The output of the first command would be a list of filenames for mails
> 'seen' since last sync. The second one, an external notmuch--maildir
> helper, would propagate this logic to the MailDir store (easy, this is
> simply a rename), and output the list of (old-name ! new-name). Then
> notmuch would use this information, via a generic 'move' switch, to know
> that mail has been moved, and would output the list of the new places.
> Finaly, notmuch would set the seen_synced flag.

What happens if notmuch-move gets killed due to out-of-memory half
way through?

> As we have seen, this would allow most IMAP <-> MailDir <-> NotMuch
> synchronisation, without having to implement any kind of
> MailDir-specific logic inside notmuch.

It would not take care of any tags beyond the very strictly defined
and limited set of tags you listed in your mail. My point is that we
need to solve this problem more generally anyway — why should
a "replied" tag be synchronised, but a "no-need-to-reply" tag not?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
#include 
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] patchwork test instance (was: Git feature branch)

[martin f krafft]

I investigated some patch/issue trackers over the weekend. Here's my
summary/reply.

The executive summary is that
http://patchwork.madduck.net/project/notmuch/list/ now exists.
I have not really used it for anything real, so if some of you feel
inclined to give it a shot, sign up and triage away! Feedback
welcome.

also sprach James Rowe  [2010.01.28.2005 +1300]:
>   Roundup has command line and email interfaces.  The email interface is
> quite similar to debian's.  I've never used a launchpad hosted project
> so I can't compare it.

Roundup is an issue tracker, while Patchwork is a patch tracker.
They are fundamentally distinct, but there are overlaps. What led me
to go the Patchwork-path is that projects like the kernel and Git
don't use issue trackers but work entirely patch-based.

I don't know if that is the right way to do things, but having an
issue tracker that fills up with bugs and wishlist items lacking
patches is no better in the long run than not having an issue
tracker.

Arguably, being patch-centric means that a project has a higher
barrier of entry, but it also means that if someone wants something,
they know that they'll have to somehow end up with a patch. The way
this happens on Git is that you either write it yourself and bring
it up to discussion (which is what patchwork facilitates), or
constructively theorise the functionality until someone else
submits a patch.

>   Google's codereview tool has a nice interface for collecting and
> commenting on patches, but I suspect that suggestion will also meet with
> a degree of friction.  To me codereview feels like patchwork with
> polish.

Maybe you could take some ideas from codereview and inform the
patchwork people about them?

>   Both gitorious and github have commenting functionality built in.
> Commenting on commits in a fork is as easy as opening the commit in
> a browser.  I use something along the lines of the following script to
> open commits on github:
> 
> #! /bin/sh
> BASE=$(git config remote.${2:-origin}.url | sed 
> 's,git\(@\|://\)\([^:/]*\)[:/]\(.*\).git,http://\2/\3/commit,')
> COMMIT=$(git rev-parse ${1:-HEAD})
> sensible-browser ${BASE}/${COMMIT}
> 
>   Using github or gitorious you can easily find and track forks from one
> place as well, which makes discovering new work much easier.  Github
> even provides a pretty single page interface to the work going on in
> other forks, gitorious requires a little more leg work to do the same
> but not much.

Git now has commit notes, but it doesn't seem like that's integrated
with Github/Gitorious.

Mind you, patchwork isn't integrated at all with Git. It should be
possible to set it up to automatically flag patches that are
accepted into mainline, next, or pu.

The benefit of patchwork is that discussion isn't moved to the web,
but patchwork hooks into the mailing list, so discussion can stay
where it should IMHO be.

>   For a couple of hosted projects we use at the office we email the
> individual entries from http://github.com/$user/$project/comments.atom
> to the mailing list so they're /forcibly/ seen by everybody :)

Right, but replying requires them to open a browser and be online at
the time, right?



Anyway, I suggest we give patchwork a try. It occurs to me that
notmuch can pretty much do all of what patchwork is doing — after
all, it's just tagging patches/threads, but until we have
synchronisable tags and a mailing list archive based on notmuch
(which could then replace patchwork), I think we'll need to employ
a third tool.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"what's your conceptual continuity? --
 well, it should be easy to see:
 the crux of the bisquit is the apopstrophe!"
-- frank zappa
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Request for high-priority improvements to notmuch

[martin f krafft]

also sprach sebast...@sspaeth.de  [2010.02.02.0929 +1300]:
> YES PLEASE :-). notmuch seems designed to work in an ecosystem of
> surrounding scripts, feeding data in and out. But we are all currently
> limited to regexes for that. And heck, I hard a hard time understanding
> why all hell broke out until I found that i had added a tag containing
> parentheses which made my regex fail. :-). XML, JSON, any structured
> output would be nice.

Shoot me, but I'd say mbox output would be nice too.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"time flies like an arrow. fruit flies like a banana."
   -- groucho marx
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Request for high-priority improvements to notmuch

[martin f krafft]

also sprach Scott Robinson  [2010.02.02.1043 +1300]:
> > YES PLEASE :-). notmuch seems designed to work in an ecosystem of
> > surrounding scripts, feeding data in and out. But we are all currently
> > limited to regexes for that. And heck, I hard a hard time understanding
> > why all hell broke out until I found that i had added a tag containing
> > parentheses which made my regex fail. :-). XML, JSON, any structured
> > output would be nice.
> > 
> > And as for filtering: YES, PLEASE :-). notmuchsync and many other 3rd
> > party apps would love that. As father of notmuchsync, I can tell you my
> > little script hickups very badly when slurping in 200k mails (including
> > text bodies) just to find out the maildir tags of those mails.
> > 
> 
> There's been a JSON patch waiting for a month now.

The last month has been busy for everyone, mainly due to LCA. We
should now all work together to help Carl go through the patch
queue. Maybe http://patchwork.madduck.net can help.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
beauty, brains, availability, personality; pick any two.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] [PATCH] Enforce foldmethod=manual when showing messages in vim

[martin f. krafft]

Vim's NotMuch mode relies on manual markers when rendering/showing
a message. If foldmethod is set to something else (marker in my case) by
default, then there are numerous errors, and folds don't work. Hence,
set foldmethod=manual for the local buffer upon showing a message.

Signed-off-by: martin f. krafft 
---
 vim/plugin/notmuch.vim |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/vim/plugin/notmuch.vim b/vim/plugin/notmuch.vim
index a226f20..2f9b05c 100644
--- a/vim/plugin/notmuch.vim
+++ b/vim/plugin/notmuch.vim
@@ -421,6 +421,7 @@ function! s:NM_cmd_show(words)
 let b:nm_raw_info = info
 let b:nm_prev_bufnr = prev_bufnr
 
+setlocal foldmethod=manual
 call NM_cmd_show_mkfolds()
 call NM_cmd_show_mksyntax()
 call NM_set_map('n', g:notmuch_show_maps)
-- 
1.6.6


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Marten Veldthuis  [2010.02.04.0353 +1300]:
> > Like this? http://notmuchmail.org/recentchanges/
> 
> No, more like this: http://git.notmuchmail.org/git/notmuch-wiki

To my knowledge, neither of these two give you diffs. This is what
a post-receive hook offers.

It's trivial to do with Git. Assuming a recent Git-on-Debian, it's
just (assuming you are in the bare repository)

  test -d refs || echo not in repo
  sed -e 's,^#\.,,' hooks/post-receive.sample > hooks/post-receive
  chmod 755 !$
  git config hooks.mailinglist notmuch@notmuchmail.org

One might also have to send hooks.envelopesender and
hooks.emailprefix (without trailing space) could be nice.

PS: speaking of prefixes, how about remving the subject prefix of
this list in general? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"arguments are extremely vulgar,
 for everyone in good society
 holds exactly the same opinion."
-- oscar wilde
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach David Bremner  [2010.02.04.0924 +1300]:
> > PS: speaking of prefixes, how about remving the subject prefix of
> > this list in general? ;)
> 
> I used to agree, but in notmuch, I actually find it convenient to have
> threads marked by mailing list. Perhaps this will change if/when my
> email setup or the notmuch emacs ui get better...

sed -e 's,^Subject:,& [notmuch],'

;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
apt-get source --compile gentoo
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git feature branch

[martin f krafft]

also sprach Carl Worth  [2010.02.04.1605 +1300]:
> >   maint/— the stable release
> >   master/   — the stablising head
> >   next/ — testing branch
> >   pu/   — patch integration branch (proposed updates)
> 
> I'm not a fan of this scheme, (or maybe I've just never quite understood
> it).
> 
> When I first encountered this scheme, (when first using git), it was
> never clear to me what branch I should actually run or test as a
> user. Nor was it obvious which branches would be regularly rebased or
> not---nor which branches had features being discussed on the mailing
> list.

I'll happily explain if you want. I think that this workflow, in
combination with the distributed functionality of Git, makes for
really powerful collaboration.

To answer your two (implicit) questions directly:

- as a user, you'd probably be tracking master, if you aren't
  running a stable release anyway. if you are ready to deal with the
  occasional bug and want some juicy stuff, go with next. if you are
  a developer ready to fend of the sh*t as it hits the fan, use pu.

  Basically, it's a bit like Debian testing/unstable/experiemental,
  except that the default entry point for new patches (packages in
  that analogy) is pu (experimental), not next (unstable, as it is
  in Debian).

  maint is then the stable branch, see below.

- nothing is ever rebased. patches are cherry-picked downwards
  (pu→next→master) and branches are occasionally merged upwards
  (master into next, next into pu).

I don't think we will need 'next' anytime soon, not until we have
a situation where we are e.g. working on the next 1.x release by
already have 2.0 on the horizon.

The important distinction is between pu (proposed-updates) and
master. The goal for master should always be that it's usable.
Features that are too new to ensure that go to pu until they
matured.

> Instead of "maint" I'd much rather just have branches that are
> named directly after the stable releases being maintained. We've
> done this with the cairo repository with branch names like "1.2",
> "1.4", "1.6", etc. That way it's very clear what the branch
> represents and it's possible to have multiple concurrent "live"
> maintenance branches. But of course, until we actually have
> releases, this doesn't really matter. :-)

This is all possible to do. It has no impact on the pu/next/master
distinction. Basically, once a release is made, master is merged
into maint (I think) and tagged. If a maintenance release has to be
made, a maint/1.0 branch is created. I don't think Git/Linux do
that, but I do.

> I want to maintain a branch myself, (where I'm the only person
> pushing to the branch). [This is different than what I've done
> with the cairo repository where we have all core maintainer's
> pushing to a central repository. I'm intentionally trying
> something new here.]

Do you want to maintain that branch yourself for your own purposes,
or do you want to be the sole maintainer of the branch that is
advertised as *the* notmuch branch, and from which future releases
are made?

> Obviously, that branch that I maintain is currently called
> "master", but I wouldn't mind (and might actually prefer) to have
> it be called "~cworth" or so. Though we have the problem that we
> need "master" to point to *something*.

Actually, there is no reason for master to exist at all. ;)
It's just the default, but it's not intrinsic.

> Beyond that, I'm quite happy to have any number of branches similarly
> maintained by any other individuals. I want to get things setup so that
> those will be hosted and listed alongside my branch on
> notmuchmail.org.

So you are talking repos, not branches. I know they are almost the
same, but branches live in repos, and if you want to be the only
person pushing to a branch, you need to be the only one with write
access to the containing repo — unless you use gitolite, which can
do in-repo per-branch access control.

> > What patch tracking workflow should we adopt? Keep sending
> > patches to the mailing list
> 
> I definitely like the patches on the list. I find that with
> notmuch, I can maintain a queue of outstanding patches very
> effectively, (meaning, that the queue is usable and doesn't forget
> even if I do get very far behind like I am currently).

The reason why I set up patchwork is because you might have spent
hours triaging some patches already, e.g. bringing the count from
400 to 100. Since I cannot really "pull" your tags, I am forced to
go through the entire list myself.

> > master or pu (or even maint/next), as appropriate? Use the
> > Debian bug tracker? Use Sebastian's Roundup instance? Set up
> > a patch queue manager on notmuchmail.org? Use patchwork [1]?
>
> I'm personally not interested in any system that requires me to
> push any additional buttons outside of notmuch and git itself.
> I am interested in tools that can generate reports and help
> provide visibility into things. So patchwork fixed to
> automatically notice wh

Re: [notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Servilio Afre Puentes  [2010.02.04.1714 +1300]:
> In the second link, the links with the text "commitdiff" provide
> it, and you have the Atom and RSS feeds at the bottom.

As I see it, the feeds have links to the commitdiffs, but that's not
quite the same as having the diffs inline. You might be offline
without having pulled before, then the items from RSS/Atom are
useless.

If there is indeed an RSS/Atom feed on gitweb that includes the
diffs inline, please give me the URL; I can't fine ond.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"es ist immer etwas wahnsinn in der liebe.
 es ist aber auch immer etwas vernunft im wahnsinn."
 - friedrich nietzsche
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Servilio Afre Puentes  [2010.02.04.1918 +1300]:
> > If there is indeed an RSS/Atom feed on gitweb that includes the
> > diffs inline, please give me the URL; I can't fine ond.
> 
> Can't see that even in the code.

Carl, could you set up notmuch-comm...@notmuchmail.org and enable
the commit and enable the mail hook as per

  http://notmuchmail.org/pipermail/notmuch/2010/001373.html

Sorry to burden you, but since you want to continue maintaining the
infrastructure, that's just what I have to do. ;)

Thanks,

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
beware of bugs in the above code;
i have only proved it correct, not tried it.
-- donald e. knuth
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] patchwork now auto-updates patches from Git (was: Git feature branch)

[martin f krafft]

also sprach martin f krafft  [2010.02.04.1650 +1300]:
> - … which brings me to my second point: there are certain things
>   that patchwork can do, at least in theory:
> 
>   * mark patches accepted when they hit your (canonical) master
> branch
>   * mark patches rfc when they hit e.g. my (canonical) next branch
>   * mark patches "under review" when they hit the all-patches (or
> pu) branch.
> 
>   I have not yet tried any of these, and I am basing this theory
>   only on the idea that git-patch-id can come to the rescue, for
>   there is no other linkage between the patch on the mailing list
>   (and thus known to patchwork), and the commit in the repo.

Patchwork now marks patches Accepted once they hit Carl's master
branch (up to 10 minutes delay due to Cron). It uses an algorithm
similar (but not equal) to git-patch-id in that it hashes the diff.
This means that the commit message can be amended when patches are
applied/cherry-picked, but the patch itself must be verbatim.

I ran it on all history thus far and it found 99 patches.

It'll be trivial to set it up to mark other states when the
corresponding commits hit another branch.

Let me know if there are any problems, and feedback welcome.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"if they can get you asking the wrong questions,
 they don't have to worry about answers."
 -- thomas pynchon
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Git commit mails (was: New wiki instance on the notmuchmail.org website)

[martin f krafft]

also sprach Carl Worth  [2010.02.07.1156 +1300]:
> > Sorry to burden you, but since you want to continue maintaining
> > the infrastructure, that's just what I have to do. ;)
> 
> No burden at all. I don't really care to do everything alone.
> I just want to ensure we keep things centralized enough that
> people can actually find things easily.

This is what DNS was designed for, right? ;)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
to err is human - to moo, bovine
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] hack to retag a directory

[martin f krafft]

also sprach David Bremner  [2010.02.07.1238 +1300]:
> I have to say that merge conflicts are not very much fun. I tend
> to do a certain amount of oh, take all the changes from the
> server.  I wonder if the approach that someone else mentioned of
> keeping a file tags/message-id with the appropriate tags in it
> might make merging less painful.

Merge conflicts at the file level are even less fun. This reminds me
of my plan to introduce .gitignore.d/*:

http://kerneltrap.org/mailarchive/git/2007/10/1/326425

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"there was silence for a moment, and then out of the scrambled mess
 of arthur's brain crawled some words."
 -- hitchhiker's guide to the galaxy
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] patchwork test instance

[martin f krafft]

also sprach martin f krafft  [2010.02.02.1131 +1300]:
> I investigated some patch/issue trackers over the weekend. Here's my
> summary/reply.
> 
> The executive summary is that
> http://patchwork.madduck.net/project/notmuch/list/ now exists.
> I have not really used it for anything real, so if some of you feel
> inclined to give it a shot, sign up and triage away! Feedback
> welcome.

Are people actually using it? I know that merging patches is
impossible, and that sucks, but otherwise: is this something to keep
around, or should I take the site offline again?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"time flies like an arrow. fruit flies like a banana."
   -- groucho marx
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] patchwork test instance

[martin f krafft]

also sprach David Bremner  [2010.02.10.2149 +1300]:
> I'm not sure what merging patches means here; some kind of squash
> operation?  Anyway it seemed to me that every every patch series
> that I looked at was broken into individual patches. Maybe I am
> just unlucky, or does patchwork really not understand the concept
> of series of patches in a thread?

I don't think it does, this is what bundles are for, but these need
to be created manually at the moment. Patch series are pretty easy
to detect, so maybe the bundle could be automatically generated.

http://lists.ozlabs.org/pipermail/patchwork/2010-February/000226.html

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
because light travels faster than sound,
some people appear to be intelligent,
until you hear them speak.
 
spamtraps: madduck.bo...@madduck.net
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] patchwork test instance

[martin f krafft]

also sprach Sebastian Spaeth  [2010.02.10.2225 +1300]:
> "notmuch dump tag:notmuch and tag:patch" could be used to construct a
> list of "accepted", "willnottakeit","superseded" and "pending" lists or
> whatever. If that list were made accessible somewhere, this would be
> super useful. At least it would help me see whether Carl just hasn't
> gotten around to including my "press 'd' for delete" patch or whether he
> is not interested in merging it. :)

Carl, would you consider bouncing messages to addresses like
patchwork+rejec...@patchwork.notmuchmail.org? That would make it
trivial for me to write glue to update patchwork automatically.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"es ist gut, eine sache doppelt auszudrücken und ihr einen
 rechten und linken fuß zu geben. auf einem bein kann die wahrheit
 zwar stehen; mit zweien aber wird sie gehen und herumkommen."
-- friedrich nietzsche
 
spamtraps: madduck.bo...@madduck.net
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] notmuch reply template

[martin f krafft]

also sprach Sebastian Spaeth  [2010.02.12.0254 +1300]:
> I've had enough of the 
> 
>  On a sunny Sunday the 30th in timezone , male human  with mail
>  address  hit the reply button  microseconds after 1970 while being 
> at latitude :
> 
> type of reply template in notmuch. :-)
> 
> May I suggest to use a simpler:
> 
>  "On 11 Feb 2010, David Edmondson wrote:" 

This should be configurable so everyone can set their own.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
if god had meant for us to be naked,
we would have been born that way.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] [PATCH] notmuch: Respect maildir message flags

[martin f krafft]

also sprach Stewart Smith  [2010.02.16.1458 +1300]:
> + case 'R': /* replied */
> + notmuch_message_add_tag (message, "answered");
> + break;

'r' means replied, not 'answered'.

> + case 'T': /* trashed */
> + notmuch_message_add_tag (message, "deleted");
> + break;

Same. trashed and deleted are not the same thing.

I don't want to get into an argument over this, because I think this
already exposes a problem: you are putting into global namespace
something not everyone might want, or agree with.

Why not use 'maildirflags::replied' instead? People can always map
that to something in the global namespace.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"geld ist das brecheisen der macht."
 - friedrich nietzsche
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch