[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[Asheesh Laroia]

On Fri, 15 Jan 2010, martin f krafft wrote:

> also sprach Asheesh Laroia  [2010.01.14.2112 +1300]:
>> Sure. But the MDA doesn't need to do the commit immediately. Since
>> (presumably) we're using Maildir, the MDA on the mail receiving
>> server is going to generate filenames that won't cause conflicts.
>> So it's okay to leave the files uncommitted.
>
> So when does the commit happen?
>
>> When I did the "git merge", git would create the Maildir files in
>> ~/Maildir/cur/... non-atomically.
>
> This might be something that the Git people could address if it was
> brought up on the mailing list. Then again, it might not be possible
> without going via a temporary file, which I doubt will fly.

A temporary file + rename() is the only way, as far as I know.

> I suppose that I never actually considered merges on the IMAP server 
> side, but obviously the IMAP server has to work off a clone, and that 
> means it needs to merge.

It's not "merge" that's unsafe; that just builds a tree in the git index 
(assuming no conflicts). It's the ensuing process of git writing a tree to 
the filesystem that is problematic.

I could probably actually write a wrapper that locks the Maildir while git 
is operating. It would probably be specific to each IMAP server.

Note that this mean git is fundamentally incompatible with Maildir, not 
just IMAP servers.

>> Dovecot would notice the file in ~/Maildir/cur/ and think, "This file 
>> must be ready!" So it would parse it even though git hadn't finished 
>> writing it. This caused me to only see partial headers in Alpine since 
>> Dovecot parsed it before it was a complete message.
>
> I wonder if a custom merge driver could address this to properly use 
> ?/tmp/ to assemble the message and only then move it.

I don't think a merge driver can do it for the reason stated above.

-- Asheesh.

-- 
I always turn to the sports pages first, which record people's accomplishments.
The front page has nothing but man's failures.
-- Chief Justice Earl Warren

Re: [notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[Asheesh Laroia]

On Fri, 15 Jan 2010, martin f krafft wrote:


also sprach Asheesh Laroia ashe...@asheesh.org [2010.01.14.2112 +1300]:

Sure. But the MDA doesn't need to do the commit immediately. Since
(presumably) we're using Maildir, the MDA on the mail receiving
server is going to generate filenames that won't cause conflicts.
So it's okay to leave the files uncommitted.


So when does the commit happen?


When I did the git merge, git would create the Maildir files in
~/Maildir/cur/... non-atomically.


This might be something that the Git people could address if it was
brought up on the mailing list. Then again, it might not be possible
without going via a temporary file, which I doubt will fly.


A temporary file + rename() is the only way, as far as I know.

I suppose that I never actually considered merges on the IMAP server 
side, but obviously the IMAP server has to work off a clone, and that 
means it needs to merge.


It's not merge that's unsafe; that just builds a tree in the git index 
(assuming no conflicts). It's the ensuing process of git writing a tree to 
the filesystem that is problematic.


I could probably actually write a wrapper that locks the Maildir while git 
is operating. It would probably be specific to each IMAP server.


Note that this mean git is fundamentally incompatible with Maildir, not 
just IMAP servers.


Dovecot would notice the file in ~/Maildir/cur/ and think, This file 
must be ready! So it would parse it even though git hadn't finished 
writing it. This caused me to only see partial headers in Alpine since 
Dovecot parsed it before it was a complete message.


I wonder if a custom merge driver could address this to properly use 
…/tmp/ to assemble the message and only then move it.


I don't think a merge driver can do it for the reason stated above.

-- Asheesh.

--
I always turn to the sports pages first, which record people's accomplishments.
The front page has nothing but man's failures.
-- Chief Justice Earl Warren___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Asheesh Laroia  [2010.01.14.2112 +1300]:
> Sure. But the MDA doesn't need to do the commit immediately. Since
> (presumably) we're using Maildir, the MDA on the mail receiving
> server is going to generate filenames that won't cause conflicts.
> So it's okay to leave the files uncommitted.

So when does the commit happen?

> When I did the "git merge", git would create the Maildir files in
> ~/Maildir/cur/... non-atomically.

This might be something that the Git people could address if it was
brought up on the mailing list. Then again, it might not be possible
without going via a temporary file, which I doubt will fly.

I suppose that I never actually considered merges on the IMAP server
side, but obviously the IMAP server has to work off a clone, and
that means it needs to merge.

> Dovecot would notice the file in ~/Maildir/cur/ and think, "This
> file must be ready!" So it would parse it even though git hadn't
> finished writing it. This caused me to only see partial headers in
> Alpine since Dovecot parsed it before it was a complete message.

I wonder if a custom merge driver could address this to properly use
?/tmp/ to assemble the message and only then move it.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"this week dragged past me so slowly;
 the days fell on their knees..."
-- david bowie

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[Asheesh Laroia]

On Tue, 12 Jan 2010, martin f krafft wrote:

> If the MDA delivers to Git, then potentially, you might get into a 
> situation where you cannot write your own changes back to the repo. This 
> is also a DoS scenario: I'll just keep sending you e-mail, and if I 
> manage to pass your mail filters, I'll basically commit to your mail 
> repository at regular intervals. Say those are 5 seconds. In order for 
> you to write updates to the repo, e.g. to update tags, then you would 
> need to pull, rebase, and push all within 5 seconds, for otherwise you'd 
> try to push non-fast-forwards.

Sure. But the MDA doesn't need to do the commit immediately. Since 
(presumably) we're using Maildir, the MDA on the mail receiving server is 
going to generate filenames that won't cause conflicts. So it's okay to 
leave the files uncommitted.

If that's too scary, then have the MDA deliver to its own git branch with 
its own checkout. Then, if you can force linearity with a lock (!), your 
client can have a special "lock the repo and push" command. Your remote 
MUA could even ask the MDA to lock the Maildir while it does a merge and 
then pushes that, and then the MDA can go back to dequeuing messages from 
the MTA into the Maildir.

Not the beautiful lockless world the purists want, but I'm okay with that.

> This a bit unrealistic, surely, but there's a real annoyance in it: 
> you'd have to pull/rebase/push until a push succeeds ? until you found a 
> time window between pull and push during which the MDA didn't write to 
> the repo. This might take a long time. If this happens in the background 
> by Cron, it's not a real concern, but if this becomes a UI issue, I 
> wouldn't know how to handle it.

It's not entirely unreasonable. Cron caused issues like that for me when I 
tracked my Maildir in git.

I'm just learning about notmuchmail.org, but I'll keep listening here. 
Preferably CC: me on replies to this mail.

I will say, I'm interested in an email setup with with working IMAP on at 
least one side.

There's one other bad race I ran into when using git to manage my 
Maildirs. I was using Dovecot to serve my Maildir to an IMAP client, 
alpine. I separately did a "git merge" from origin/master, where the 
remote MTA had an MDA deliving messages and a layer on top of that 
committed them.

When I did the "git merge", git would create the Maildir files in 
~/Maildir/cur/... non-atomically. Dovecot would notice the file in 
~/Maildir/cur/ and think, "This file must be ready!" So it would parse it 
even though git hadn't finished writing it. This caused me to only see 
partial headers in Alpine since Dovecot parsed it before it was a complete 
message.

That kind of sucked.

-- Asheesh.

-- 
Almost anything derogatory you could say about today's software design
would be accurate.
-- K. E. Iverson

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Jameson Rollins  [2010.01.13.0838 
+1300]:
> What about if just the tag information is stored in the
> repository, and not the mail itself?  In that case only the user
> would be pushing into the repo and you wouldn't have to worry
> about the DoS scenario.

I certainly would like the ability to have messages
automatically-tagged on delivery, by procmail.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

may the bluebird of happiness twiddle your bits.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[martin f krafft]

also sprach Scott Robinson  [2010.01.12.1644 +1300]:
> Then the whole structure is controlled via git.
> Conflict-resolution and sync comes for free.

I've just had a good think about this, also because the idea of
abandoning IMAP and using Git has been around for a while and
I have not really wrapped my head around it.

If the MDA delivers to Git, then potentially, you might get into
a situation where you cannot write your own changes back to the
repo. This is also a DoS scenario: I'll just keep sending you
e-mail, and if I manage to pass your mail filters, I'll basically
commit to your mail repository at regular intervals. Say those are
5 seconds. In order for you to write updates to the repo, e.g. to
update tags, then you would need to pull, rebase, and push all
within 5 seconds, for otherwise you'd try to push non-fast-forwards.

This a bit unrealistic, surely, but there's a real annoyance in it:
you'd have to pull/rebase/push until a push succeeds ? until you
found a time window between pull and push during which the MDA
didn't write to the repo. This might take a long time. If this
happens in the background by Cron, it's not a real concern, but if
this becomes a UI issue, I wouldn't know how to handle it.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

don't hate yourself in the morning -- sleep till noon.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[notmuch] Potential problem using Git for mail (was: Idea for storing tags)

[Jameson Rollins]

On Tue, Jan 12, 2010 at 05:51:53PM +1300, martin f krafft wrote:
> If the MDA delivers to Git, then potentially, you might get into
> a situation where you cannot write your own changes back to the
> repo. This is also a DoS scenario: I'll just keep sending you
> e-mail, and if I manage to pass your mail filters, I'll basically
> commit to your mail repository at regular intervals. Say those are
> 5 seconds. In order for you to write updates to the repo, e.g. to
> update tags, then you would need to pull, rebase, and push all
> within 5 seconds, for otherwise you'd try to push non-fast-forwards.
> 
> This a bit unrealistic, surely, but there's a real annoyance in it:
> you'd have to pull/rebase/push until a push succeeds ? until you
> found a time window between pull and push during which the MDA
> didn't write to the repo. This might take a long time. If this
> happens in the background by Cron, it's not a real concern, but if
> this becomes a UI issue, I wouldn't know how to handle it.

What about if just the tag information is stored in the repository,
and not the mail itself?  In that case only the user would be pushing
into the repo and you wouldn't have to worry about the DoS scenario.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: