[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka  [2010.04.12.1347 +0200]:
> > > Wouldn't it be better to postpone synchronisation of the tags
> > > until after emacs is done with the message?
> > 
> > Theoretically, it would be possible, but if, for some reason, the
> > synchronization step would not happen, then the tags in the
> > database and in the mailstore will be inconsistent and next run of
> > notmuch new would reset the tags according to the state in
> > mailstore.
> 
> Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
> transactional anyway. There are many other ways in which the
> database and store can get out of sync. And you are about to add
> another redundancy.
> 
> > The current implementation takes tags in mailstore as
> > authoritative and ensures that tags in mailstore are always
> > updated before tags in the database.
> 
> So why store them in the database at all?

Because we want to use them in searches.

> > > I understand this might be hard to make work with mailstore
> > > abstraction. Wouldn't it make more sense to have emacs call
> > > 'notmuch cat', which returns the entire message, removes the
> > > unread tag, changes the filename, and updates the database?
> > 
> > I do not like the fact that cat would do two things - cat and tag.
> > And then, 'unread' tag is not the only one which can be changed.
> 
> I wouldn't want an unread tag in the first place, especially not
> with Maildir semantics. In this case, what should really happen is:

The bellow sounds reasonable. I only have a few notes.
> 
>   1. cat feeds a message to client

I'd not use cat here, I'd stay with show which transfers only text/*
parts. Recently I was surprised how fast notmuch is when client is
running locally and notmuch is invoked remotely over ssh (even over slow
connection).

>   2. client instructs notmuch to update tags
>  - some tags require changes in the database
>  - others require filename changes, which must be completed in
>unison with a database update so the new filename is stored.
>   3. user asks to see attachment, which the client can fulfill using
>  either a cached copy from (1.) in a tempfile, or by simply
>  asking for the message again, via notmuch search.

The latter option sounds well for me, but my elisp skills are not
sufficient for implementing this. Maybe, dme will do it in his new
client. On the other side, I'm not sure whether it has sense search for
the message again (in step 3) if you are not using maildir mailstore.
One possibility would be, when using maildir mailstore, to cheat the
client and use message-id instead of filename. Then, the client would
use message-id in cat command and implementation for open_file() in
maildir mailstore will search for the real file name.

What do you think?

> > > The message returned by cat would be stored in a temporary file
> > > for use by the client (emacs). And if the message was needed
> > > again, you could just search for it again.
> > > 
> > > I dislike the idea of heuristically probing a Maildir for files.
> > 
> > Well, I do not plan to use wired heuristics. At the end there will
> > be readdir() to traverse the cur/ directory to find the file with
> > the same part before flags. Since the S flag will probably be the
> > most frequent change, I may add one single test for added S flag
> > before trying more expensive readdir().
> 
> What is the point of storing these tags in the Maildir anyway? 

My point is to synchronize the status of reading of my emails with IMAP
server (through offlineimap). I use notmuch as my primary emails client,
but my mobile phone checks for new emails on the IMAP server and from
time to time I need to use other IMAP clients (on other computers).

I have offlineimap running in background so that whenever I read a
message in notmuch, after some short time, this information is
propagated back to IMAP server and when I go home from work, my mobile
phone shows zero unread mails.

> If you want to make this information (e.g. new, seen, unread)
> available to MUAs accessing Maildir directly, keep in mind that the
> database and mailstore will very quickly grow inconsistent until the
> next notmuch-new run, e.g. as messages are moved, or flags ('F') are
> added in a way that the notmuch database is not updated.

It depends on how you define quickly and if you run notmuch new also
very quickly, there is almost no inconsistency :))

-Michal

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.12.1347 +0200]:
> > Wouldn't it be better to postpone synchronisation of the tags
> > until after emacs is done with the message?
> 
> Theoretically, it would be possible, but if, for some reason, the
> synchronization step would not happen, then the tags in the
> database and in the mailstore will be inconsistent and next run of
> notmuch new would reset the tags according to the state in
> mailstore.

Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
transactional anyway. There are many other ways in which the
database and store can get out of sync. And you are about to add
another redundancy.

> The current implementation takes tags in mailstore as
> authoritative and ensures that tags in mailstore are always
> updated before tags in the database.

So why store them in the database at all?

> > I understand this might be hard to make work with mailstore
> > abstraction. Wouldn't it make more sense to have emacs call
> > 'notmuch cat', which returns the entire message, removes the
> > unread tag, changes the filename, and updates the database?
> 
> I do not like the fact that cat would do two things - cat and tag.
> And then, 'unread' tag is not the only one which can be changed.

I wouldn't want an unread tag in the first place, especially not
with Maildir semantics. In this case, what should really happen is:

  1. cat feeds a message to client
  2. client instructs notmuch to update tags
 - some tags require changes in the database
 - others require filename changes, which must be completed in
   unison with a database update so the new filename is stored.
  3. user asks to see attachment, which the client can fulfill using
 either a cached copy from (1.) in a tempfile, or by simply
 asking for the message again, via notmuch search.

> > The message returned by cat would be stored in a temporary file
> > for use by the client (emacs). And if the message was needed
> > again, you could just search for it again.
> > 
> > I dislike the idea of heuristically probing a Maildir for files.
> 
> Well, I do not plan to use wired heuristics. At the end there will
> be readdir() to traverse the cur/ directory to find the file with
> the same part before flags. Since the S flag will probably be the
> most frequent change, I may add one single test for added S flag
> before trying more expensive readdir().

What is the point of storing these tags in the Maildir anyway? If
you want to make this information (e.g. new, seen, unread) available
to MUAs accessing Maildir directly, keep in mind that the database
and mailstore will very quickly grow inconsistent until the next
notmuch-new run, e.g. as messages are moved, or flags ('F') are
added in a way that the notmuch database is not updated.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"friendships last when each friend thinks he has
 a slight superiority over the other."
   -- honor? de balzac

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka  [2010.04.08.1713 +0200]:
> >I'm working on the solution - if the mailstore cannot open the
> >message with the name passed, it tries different names with
> >different maildir flags.
> 
> Wouldn't it be better to postpone synchronisation of the tags until
> after emacs is done with the message?

Theoretically, it would be possible, but if, for some reason, the
synchronization step would not happen, then the tags in the database and
in the mailstore will be inconsistent and next run of notmuch new would
reset the tags according to the state in mailstore.

The current implementation takes tags in mailstore as authoritative and
ensures that tags in mailstore are always updated before tags in the
database.

> I understand this might be hard to make work with mailstore
> abstraction. Wouldn't it make more sense to have emacs call 'notmuch
> cat', which returns the entire message, removes the unread tag,
> changes the filename, and updates the database?

I do not like the fact that cat would do two things - cat and tag. And
then, 'unread' tag is not the only one which can be changed.

> The message returned by cat would be stored in a temporary file for
> use by the client (emacs). And if the message was needed again, you
> could just search for it again.
> 
> I dislike the idea of heuristically probing a Maildir for files.

Well, I do not plan to use wired heuristics. At the end there will be
readdir() to traverse the cur/ directory to find the file with the same
part before flags. Since the S flag will probably be the most frequent
change, I may add one single test for added S flag before trying more
expensive readdir().

-Michal

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.08.1713 +0200]:
>I'm working on the solution - if the mailstore cannot open the
>message with the name passed, it tries different names with
>different maildir flags.

Wouldn't it be better to postpone synchronisation of the tags until
after emacs is done with the message?

I understand this might be hard to make work with mailstore
abstraction. Wouldn't it make more sense to have emacs call 'notmuch
cat', which returns the entire message, removes the unread tag,
changes the filename, and updates the database?

The message returned by cat would be stored in a temporary file for
use by the client (emacs). And if the message was needed again, you
could just search for it again.

I dislike the idea of heuristically probing a Maildir for files.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"i don't think so," said rene descartes. just then, he vanished.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka  [2010.04.12.1347 +0200]:
> > > Wouldn't it be better to postpone synchronisation of the tags
> > > until after emacs is done with the message?
> > 
> > Theoretically, it would be possible, but if, for some reason, the
> > synchronization step would not happen, then the tags in the
> > database and in the mailstore will be inconsistent and next run of
> > notmuch new would reset the tags according to the state in
> > mailstore.
> 
> Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
> transactional anyway. There are many other ways in which the
> database and store can get out of sync. And you are about to add
> another redundancy.
> 
> > The current implementation takes tags in mailstore as
> > authoritative and ensures that tags in mailstore are always
> > updated before tags in the database.
> 
> So why store them in the database at all?

Because we want to use them in searches.

> > > I understand this might be hard to make work with mailstore
> > > abstraction. Wouldn't it make more sense to have emacs call
> > > 'notmuch cat', which returns the entire message, removes the
> > > unread tag, changes the filename, and updates the database?
> > 
> > I do not like the fact that cat would do two things - cat and tag.
> > And then, 'unread' tag is not the only one which can be changed.
> 
> I wouldn't want an unread tag in the first place, especially not
> with Maildir semantics. In this case, what should really happen is:

The bellow sounds reasonable. I only have a few notes.
> 
>   1. cat feeds a message to client

I'd not use cat here, I'd stay with show which transfers only text/*
parts. Recently I was surprised how fast notmuch is when client is
running locally and notmuch is invoked remotely over ssh (even over slow
connection).

>   2. client instructs notmuch to update tags
>  - some tags require changes in the database
>  - others require filename changes, which must be completed in
>unison with a database update so the new filename is stored.
>   3. user asks to see attachment, which the client can fulfill using
>  either a cached copy from (1.) in a tempfile, or by simply
>  asking for the message again, via notmuch search.

The latter option sounds well for me, but my elisp skills are not
sufficient for implementing this. Maybe, dme will do it in his new
client. On the other side, I'm not sure whether it has sense search for
the message again (in step 3) if you are not using maildir mailstore.
One possibility would be, when using maildir mailstore, to cheat the
client and use message-id instead of filename. Then, the client would
use message-id in cat command and implementation for open_file() in
maildir mailstore will search for the real file name.

What do you think?

> > > The message returned by cat would be stored in a temporary file
> > > for use by the client (emacs). And if the message was needed
> > > again, you could just search for it again.
> > > 
> > > I dislike the idea of heuristically probing a Maildir for files.
> > 
> > Well, I do not plan to use wired heuristics. At the end there will
> > be readdir() to traverse the cur/ directory to find the file with
> > the same part before flags. Since the S flag will probably be the
> > most frequent change, I may add one single test for added S flag
> > before trying more expensive readdir().
> 
> What is the point of storing these tags in the Maildir anyway? 

My point is to synchronize the status of reading of my emails with IMAP
server (through offlineimap). I use notmuch as my primary emails client,
but my mobile phone checks for new emails on the IMAP server and from
time to time I need to use other IMAP clients (on other computers).

I have offlineimap running in background so that whenever I read a
message in notmuch, after some short time, this information is
propagated back to IMAP server and when I go home from work, my mobile
phone shows zero unread mails.

> If you want to make this information (e.g. new, seen, unread)
> available to MUAs accessing Maildir directly, keep in mind that the
> database and mailstore will very quickly grow inconsistent until the
> next notmuch-new run, e.g. as messages are moved, or flags ('F') are
> added in a way that the notmuch database is not updated.

It depends on how you define quickly and if you run notmuch new also
very quickly, there is almost no inconsistency :))

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.12.1347 +0200]:
> > Wouldn't it be better to postpone synchronisation of the tags
> > until after emacs is done with the message?
> 
> Theoretically, it would be possible, but if, for some reason, the
> synchronization step would not happen, then the tags in the
> database and in the mailstore will be inconsistent and next run of
> notmuch new would reset the tags according to the state in
> mailstore.

Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
transactional anyway. There are many other ways in which the
database and store can get out of sync. And you are about to add
another redundancy.

> The current implementation takes tags in mailstore as
> authoritative and ensures that tags in mailstore are always
> updated before tags in the database.

So why store them in the database at all?

> > I understand this might be hard to make work with mailstore
> > abstraction. Wouldn't it make more sense to have emacs call
> > 'notmuch cat', which returns the entire message, removes the
> > unread tag, changes the filename, and updates the database?
> 
> I do not like the fact that cat would do two things - cat and tag.
> And then, 'unread' tag is not the only one which can be changed.

I wouldn't want an unread tag in the first place, especially not
with Maildir semantics. In this case, what should really happen is:

  1. cat feeds a message to client
  2. client instructs notmuch to update tags
 - some tags require changes in the database
 - others require filename changes, which must be completed in
   unison with a database update so the new filename is stored.
  3. user asks to see attachment, which the client can fulfill using
 either a cached copy from (1.) in a tempfile, or by simply
 asking for the message again, via notmuch search.

> > The message returned by cat would be stored in a temporary file
> > for use by the client (emacs). And if the message was needed
> > again, you could just search for it again.
> > 
> > I dislike the idea of heuristically probing a Maildir for files.
> 
> Well, I do not plan to use wired heuristics. At the end there will
> be readdir() to traverse the cur/ directory to find the file with
> the same part before flags. Since the S flag will probably be the
> most frequent change, I may add one single test for added S flag
> before trying more expensive readdir().

What is the point of storing these tags in the Maildir anyway? If
you want to make this information (e.g. new, seen, unread) available
to MUAs accessing Maildir directly, keep in mind that the database
and mailstore will very quickly grow inconsistent until the next
notmuch-new run, e.g. as messages are moved, or flags ('F') are
added in a way that the notmuch database is not updated.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"friendships last when each friend thinks he has
 a slight superiority over the other."
   -- honoré de balzac
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka  [2010.04.08.1713 +0200]:
> >I'm working on the solution - if the mailstore cannot open the
> >message with the name passed, it tries different names with
> >different maildir flags.
> 
> Wouldn't it be better to postpone synchronisation of the tags until
> after emacs is done with the message?

Theoretically, it would be possible, but if, for some reason, the
synchronization step would not happen, then the tags in the database and
in the mailstore will be inconsistent and next run of notmuch new would
reset the tags according to the state in mailstore.

The current implementation takes tags in mailstore as authoritative and
ensures that tags in mailstore are always updated before tags in the
database.

> I understand this might be hard to make work with mailstore
> abstraction. Wouldn't it make more sense to have emacs call 'notmuch
> cat', which returns the entire message, removes the unread tag,
> changes the filename, and updates the database?

I do not like the fact that cat would do two things - cat and tag. And
then, 'unread' tag is not the only one which can be changed.
 
> The message returned by cat would be stored in a temporary file for
> use by the client (emacs). And if the message was needed again, you
> could just search for it again.
> 
> I dislike the idea of heuristically probing a Maildir for files.

Well, I do not plan to use wired heuristics. At the end there will be
readdir() to traverse the cur/ directory to find the file with the same
part before flags. Since the S flag will probably be the most frequent
change, I may add one single test for added S flag before trying more
expensive readdir().

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.08.1713 +0200]:
>I'm working on the solution - if the mailstore cannot open the
>message with the name passed, it tries different names with
>different maildir flags.

Wouldn't it be better to postpone synchronisation of the tags until
after emacs is done with the message?

I understand this might be hard to make work with mailstore
abstraction. Wouldn't it make more sense to have emacs call 'notmuch
cat', which returns the entire message, removes the unread tag,
changes the filename, and updates the database?

The message returned by cat would be stored in a temporary file for
use by the client (emacs). And if the message was needed again, you
could just search for it again.

I dislike the idea of heuristically probing a Maildir for files.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"i don't think so," said rene descartes. just then, he vanished.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

This is the second part of mailstore abstraction patches. I do not
want this to be merged yet, but there might be some people interested
in testing this.

This patch adds a mailstore, which bi-directionally synchronizes
certain tags with maildir flags.

I use it already four weeks and it works quite well. There are the
following know bugs:

1) Viewing/storing of attachments of unread messages doesn't work. The
   reason is that when you view the message its unread tag is removed
   by elisp code. This leads to rename of the file, but Emacs still
   uses the original name to access message with the attachment.

   Workaround: close the message and open it again.

   I'm working on the solution - if the mailstore cannot open the
   message with the name passed, it tries different names with
   different maildir flags.

2) If there several messages with the same ID (e.g. one in sent folder
   and one sent back by mailing list), the flags are synchronized to
   only one of these files.

   I plan to do this:

   - When a tag is added/removed in notmuch, flags of all
 files corresponding to the message will be updated.
   - If we detect (during notmuch new) that flags of one file were
 changed by somebody else, we also change the flags for the other
 files corresponding to the message.
   - If we detect (during notmuch new) that flags of two or more files
 were changed by somebody else, we have to solve the conflict
 somehow, but I didn't invent how, yet.

The full series is available at
http://rtime.felk.cvut.cz/gitweb/notmuch.git/shortlog/refs/heads/mailstore-abstraction-v4
and can be pulled by

   git pull git://rtime.felk.cvut.cz/notmuch.git mailstore-abstraction-v4

Besides the patch sent here, there are also tests for the maildir
mailstore and a not finished implementation of the solution for 1)
above.

--Michal

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

This is the second part of mailstore abstraction patches. I do not
want this to be merged yet, but there might be some people interested
in testing this.

This patch adds a mailstore, which bi-directionally synchronizes
certain tags with maildir flags.

I use it already four weeks and it works quite well. There are the
following know bugs:

1) Viewing/storing of attachments of unread messages doesn't work. The
   reason is that when you view the message its unread tag is removed
   by elisp code. This leads to rename of the file, but Emacs still
   uses the original name to access message with the attachment.

   Workaround: close the message and open it again.

   I'm working on the solution - if the mailstore cannot open the
   message with the name passed, it tries different names with
   different maildir flags.

2) If there several messages with the same ID (e.g. one in sent folder
   and one sent back by mailing list), the flags are synchronized to
   only one of these files.

   I plan to do this:

   - When a tag is added/removed in notmuch, flags of all
 files corresponding to the message will be updated.
   - If we detect (during notmuch new) that flags of one file were
 changed by somebody else, we also change the flags for the other
 files corresponding to the message.
   - If we detect (during notmuch new) that flags of two or more files
 were changed by somebody else, we have to solve the conflict
 somehow, but I didn't invent how, yet.

The full series is available at
http://rtime.felk.cvut.cz/gitweb/notmuch.git/shortlog/refs/heads/mailstore-abstraction-v4
and can be pulled by

   git pull git://rtime.felk.cvut.cz/notmuch.git mailstore-abstraction-v4

Besides the patch sent here, there are also tests for the maildir
mailstore and a not finished implementation of the solution for 1)
above.

--Michal
   
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch