ANNOUNCE: muchsync 7 released

[David Mazieres]

I've just released version 7 of muchsync, a tool for pairwise
synchronization of maildirs and notmuch tags across machines.  There
aren't any new features.  However, I discovered that some of the sqlite
queries were generating full table scans rather than simple lookups.
The new version should be an order of magnitude faster when deleting
large numbers of mail messages or reorganizing your mail directory
structure.

You can download muchsync from the usual place:

https://www.muchsync.org/

Enjoy,
David
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

What do people use for calendar invites?

[David Mazieres]

Calendar invites and the text/calendar mime type seem to be getting
increasingly important.  I asked about this five years ago and didn't
get a good response, so I apologize for the repeat question, but I'm
wondering if anything has changed since then.

If you have found a good solution for integrating your calendar with
notmuchmail, would you mind sharing what you are doing?  Some specific
questions:

  * What's a good way to generate text/calendar attachments from within
notmuch (especially the emacs interface, which I use)?

  * What's a good way to apply text/calendar attachments to a calendar
from within notmuch?

  * For those of us who hate gmail and love notmuch, but are still stuck
using and hating google calendar, what alternative solutions should
we be considering?

Thanks,
David
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Problems with unicode characters under emacs and Xorg

[David Mazieres]

dm-list-email-notm...@scs.stanford.edu writes:

> I just installed the ttf-symbola package from AUR and ran fc-cache (not
> sure if necessary).  Now the problem is completely gone.  Not only that,
> but I even get the little memo symbol instead of a box with the hex code
> point number.
>
> Thank you so much!  This was driving me nuts for months.

Sadly, I spoke too soon.  This does fix the particular problem that I
posted, and makes the situation better, but I'm still getting occasional
lockups, presumably because symbola does not cover every possible
symbol.  For example, I had another email containing U+8BDD (CJK UNIFIED
IDEOGRAPH-8BDD), and this one still caused my emacs to spin for many
minutes, before displaying a box with 8BDD in it.

So unfortunately the problem seems to be that any character not
supported by an installed font takes about 2-5 minutes of CPU time to
resolve.  And of course most characters that I'd use are in installed
fonts, except that I can control what's in the emails I receive, so this
makes notmuch very painful to use.  But if no one else is running into
the problems, then I may be able to get around it by installing whatever
fonts other people have installed.

Thanks,
David
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Problems with unicode characters under emacs and Xorg

[David Mazieres]

I usually use notmuch in emacs under X windows on arch linux.  Recently,
I've had a problem where some screens in notmuch take several minutes of
100% CPU time to load.  For example, I'll just open a search, and emacs
will completely lock up (even Ctrl-G doesn't do anything) for 3 minutes
while my fan spins and my laptop battery drains significantly.

This appears to be related to the display of certain unicode characters
in email--particularly if they are in the email subject, because then
the whole search screen will freeze.  So far, the only workaround I've
found is to kill -15 emacs, start it again in an xterm or urxvt with
"emacs -nw", delete or archive the offending message, and then restart
the Xorg emacs.  This is quite painful particularly since it's not
always obvious which email message is causing the problem.

Has anyone else experienced this problem?  Is there any way to
workaround the problem by, for instance, defaulting to unibyte mode for
notmuch buffers?  I do use unicode for other languages, but I guess
wouldn't mind having to type "M-x toggle-enable-multibyte-characters" to
get them if as a result my emacs never locked up.

It's likely that this is an emacs-wide problem, but since whatever these
characters are only show up in email, I'm hoping there are people on
this list who know how to solve the problem or have better workarounds.

Thanks,
David
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: How do you synchronize your notmuch tags across multiple machines?

[David Mazieres]

Dan Čermák  writes:

>> The ideal solution would be to implement an imap server on top of
>> libnotmuch.  If we had that, then you could just use offlineimap and
>> isync through the imap (as opposed to file system) interface, and
>> everything will just work.
>
> Stupid question: how would the tags be synced with your local machines
> in this case? Via muchsync or would one keep the tags on the
> notmuch-imap server?

However you want.  If you run a local imap server and a remote imap
server, then you can just sync between the two imap servers with any
imap synching tool.  But if the two hosts happen to be running imap
servers implemented on top of libnotmuch, then you could use muchsync
which would be faster.

> How about a hacky and not ideal solution: "teach" notmuch to not only
> synchronize the read and deleted tags with the maildir, but all tags?

Unfortunately, there's no standard for how to encode these.  Also,
there's a pretty fundamental design decision in notmuch that it doesn't
edit the mail messages.  Now some imap servers, like dovecot, define
other flags on a per-maildir basis, but it only works up to a certain
number of flags, and you'd have to parse an extra file that tells you
what the mappings are.

David
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

ANNOUNCE: muchsync 3 released

[David Mazieres]

Muchsync 3 is now available from the usual place:

http://www.muchsync.org/

Muchsync synchronizes mail and tags in your notmuch database across
machines.  Not much has changed since the last release, because the tool
already worked well.  However, there are two changes people requested:

  * Muchsync only buffers 128MiB of data, rather than an unbounded
amount.  People synchronizing large databases on machines without
enough swap space were running out of memory.  A 128MiB buffer
should still be plenty to saturate the network.

  * There's a new option --newid to change the local replica id.  This
potentially allows a faster way of initializing muchsync:  Copying
your entire mail database and .notmuch-config file from the original
machine, then running "muchsync --newid" on the destination machine
to give the new replica a unique identity.

Enjoy.

David
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Breaking a really long thread

[David Mazieres]

Mark Walters  writes:

> By default the reference header is hidden. It is controlled by
> message-hidden-headers which you can customize. (Note notmuch adds
> user-agent to this list via notmuch-mua-hidden-header.)

Thanks for explaining this!  So I posted about one problem, and instead
got a solution to a problem I didn't even realize I had.  Adding:

  (setq message-hidden-headers (delete "^References:" message-hidden-headers))

to my eval-after-loaded notmuch-config.el file solved this problem
cold.  No more unintentional References: headers for me.

Arguably, I would say either both the In-Reply-To and the References
header should be hidden or neither.  Otherwise, what was happening is
that I was deleting the In-Reply-To header as it was the only one I saw,
and figuring that maybe References was adjusted after the fact based on
In-Reply-To.  After all, the message buffer doesn't keep track of the
parent message.

Unless there's a reason that someone would want to alter In-Reply-To
without altering References, it doesn't make sense to show one without
the other.

David
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: muchsync files renames

[David Mazieres]

Amadeusz Żołnowski  writes:

> Not necessarily. The recommended setup of notmuch for afew is that
> "notmuch new" tags messages with "new" tag only. Then afew processes all
> messages with "new" tag. So if it is a spam, then it gets "new" removed
> and "spam" added. A spam message at any time doesn't have "unread" tag
> assigned which should explain this behaviour.  So the problem is to be
> fixed on the afew side.

Let's just make sure I understand:  Your mail starts out like this:

Path:  spam/new/nnn.MnnnPnnnQnRn.machine
Tags:  new

Then you run afew, and afew runs

notmuch tag -new +spam 

You are saying that that even though maildir.synchronize_tags is true,
you end up with:

Path:  spam/new/nnn.MnnnPnnnQnRn.machine
Tags:  spam

That's a little surprising, because the next time you run "notmuch new,"
I would have expected it to add the unread flag based on the pathname.
But, I suppose it might make sense for notmuch to special-case that
flag.  In other words, if notmuch new finds a file called:

spam/new/nnn.MnnnPnnnQnRn.machine:2,

Then it will add the unread tag to the Xapian database.  But maybe if it
finds a file in the new folder it doesn't add the unread flag.

But why does notmuch_message_tags_to_maildir_flag() then feel the need
to rename the file when muchsync calls it.  Muchsync should ideally
behave exactly the same as the notmuch tag command.  Specifically, when
muchsync receives a new file from the server, it does the following:

 1. create file in same directory as the server (presumably spam/new)

 2. Call the following functions on this file:
  notmuch_database_add_message()
  notmuch_message_freeze()
  notmuch_message_remove_all_tags()
  notmuch_message_add_tag() for each tag in new.tags
  if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
  notmuch_message_thaw()

 3. get the current tags of the message from the server (presumably just
spam)

 4. Call the following functions on the Message-ID:
  notmuch_message_freeze()
  notmuch_message_remove_all_tags()
  notmuch_message_add_tag() for each tag sent *by the server*
  if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
  notmuch_message_thaw()

So what I'm wondering is how this is any different from what is already
happening on the server.  "notmuch new" should be doing what muchsync
does in step 2, and afew (via "notmuch tag") should be doing what
muchsync does in step 4.

Yet somehow you are saying that on the server the file stays in
spam/new/, while on the client muchsync's actions cause the file to move
to spam/cur/?  So that means there's still something I don't really
understand--possibly the series of notmuch library calls happening
server side (which I should then maybe emulate client side).

None of this is super serious, beyond a one-time extra cost, but I like
to understand things thoroughly, particularly when writing software that
manipulates critical data like mail...

It there any possibility that new.tags has a different setting on your
client and server machines?

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Resending the same email

[David Mazieres]

Sebastian Fischmeister sfisc...@uwaterloo.ca writes:

 Hi,

 My previous mail editor had a useful feature to resend already sent
 emails. It's essentially opening an already sent email and have the
 senders, subject, and body pre-filled as well as all attachments
 attached.

 Is this easy to achieve in notmuch? The attachments seem a bit tricky.

I often pipe the message to sendmail, using emacs's
notmuch-show-pipe-message command (bound to '|' by default).  However I
agree it would be much nicer to be able to edit the headers slightly, so
as to add headers like Resent-To.

On a related note, another feature I really miss is the ability to edit
and re-send a bounced message.  (I realize this could be problematic for
notmuch, in that you really want to re-send it with the same
message-ID.)

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: muchsync files renames

[David Mazieres]

Amadeusz Żołnowski aide...@aidecoe.name writes:

 Hi David,

 Fist of all thank you for such elaborate answer.

 I have missed the paragraph about maildir.synchronize_flags somehow.  I
 have it enabled.  So this must be source of a problem (?).

I've only ever tested with mailder.synchronize_flags = true, because I'm
worried that if I disable it I will have a hard time re-enabling it.  I
do think it is a source of inefficiency, but muchsync should still be
usable.

 Here follows steps I followed:

 1. I initialized server locally with muchsync -vv.  My mail is stored in
 ~/Mail on the server.
 2. I run muchsync --init ~/mail SERVER. (Directory names do not need to
 be the same, do they?)

Confirmed that directory names do not need to be the same.

 3. I run muchsync SERVER.
 4. When it lasted much longer then initialization I canceled it by
 single SIGINT (^c).

Interesting.  I wish I knew why this was taking much longer than running
it on the server, and whether the delay was caused by client activity or
server activity.

I don't suppose you'd be willing to make a copy of your mail database to
repeat the experiment without any risk of messing up your real maildir?
Basically what would be interesting to see is (assuming .notmuch-copy on
server is configured for this disposable copy):

# Log everything for later analysis
script
# Use new config file location for disposable copy
export NOTMUCH_CONFIG=$HOME/.notmuch-copy
# Set up a new initial database
muchsync - --init ~/test-copy SERVER - --config=.notmuch-copy

# Now initial copy is made, see if client is slow
# Is notmuch new itself slow?
notmuch new
# Is resynchronizing muchsync with notmuch slow?
muchsync -

# Now see if something weird happened on server to make notmuch new slow
ssh SERVER notmuch --config=.notmuch-copy new
# Now see if something weird happened on server to make muchsync slow
ssh SERVER muchsync - --config=.notmuch-copy

# Now finally try resynchronizing to see if this is slow
muchsync - SERVER - --config=.notmuch-copy
# Save script file
exit

Does something along these lines make sense?  If you can figure out
which of these is slow (other than --init, which always will be), and
what the verbose chatter is, that would help me narrow down and identify
any problem.

 5. I rerun muchsync SERVER and then it notified me that notmuch
 identified files names changes - more than 1000.

Were the link changes on the client (sent) or the server (received)
side?

 6. I waited a bit and then I canceled it by SIGINT.
 7. I run muchsync --noup SERVER. This took only seconds to finish.

So this suggests the issue is on the client side.  It sounds like a bug.
I wonder if I we can just reproduce this using a public email corpus, so
we don't have to worry about people's private email.

 I suspected that muchsync at step 3 and 5 tried to push files renames
 back to server.  But now I am not sure what was going on.  Have I
 desynchronized file mail flags?  It's hard to say if anything has broken
 for me, but I am a bit worried anyway.

 If I just disable maildir.synchronize_flags and rerun muchsync, will
 everything get synchronized properly?

I don't think that will change things.  maildir.synchronized_flags will
make things slower, but it shouldn't affect the SUMMARY numbers you see
at the end of muchsync, other than maybe files moving from .../new to
.../cur.  But presumably most of your mail files were already in cur
directories.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: muchsync files renames

[David Mazieres]

So just to follow up a bit.  I looked into things a bit further, and
here is what I found with maildir.synchronize_flags set to true.

Initially, when you run muchsync --init, it copies all the files to
your maildir, and for each file invokes
notmuch_message_tags_to_maildir_flag.  That changes the name of the
file, with the result that the sql database and the actual mail
directory end up out of sync.  That on it's own is not a big deal, but
it means that the next time muchsync, muchsync will have to rescan all
of the files, as their names are no longer correct.  That shouldn't
cause any extra traffic between the two machines, but it will require
time on the client.  That is likely the source of the delay you were
seeing.

However, if you C-c the client during this process, I still don't see
any problems arising that cause more links to be transferred between
machines.  So I'm kind of stumped about that part.

Maybe muchsync should create the original file name based on tags, so as
to avoid having to rescan in the common case.  I wish there were some
way of getting the renamed file after
notmuch_message_tags_to_maildir_flags.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: muchsync files renames

[David Mazieres]

Amadeusz Żołnowski aide...@aidecoe.name writes:

 Hi,

 I am testing muchsync-2 and it looks to me that files names across
 machines are different.  Moreover when syncing again after
 initialization it seems muchsync is working on something.  I have
 canceled this and rerun muchsync.  notmuch reported lots of files
 renames on server.  What and why it happens?

What muchsync specifically synchronizes for messages in the mapping:

(directory, SHA-1-hash, link-count)

So if a directory contains two copies of a file on one machine, it will
end up with two copies on the other machine.  However, the file names
themselves are not the same, but rather are created in accordance with
the maildir spec.  (Note SHA-1 wouldn't be my first choice of hash
function, but notmuch already uses this for messages with long message
IDs, so I figured I'd just be consistent with existing practice.)

In terms of what muchsync is working on, you can run it with - on
both sides to get an idea, as in muchsync - server -.  Better
yet, you can just run it on one side with muchsync -.  You'll get
a lot of output, so maybe run it inside the script command to save the
output.maybe run it inside the script command to save the output.  If
you have enabled maildir.synchronize_flags, it could be that notmuch is
initially renaming all of your files, in which case muchsync needs to
re-hash them to make sure they haven't changed.

How did you cancel muchsync?  If you send it a single SIGINT or SIGTERM,
it attempts to clean up after itself.  However, upon multiple signals or
other signals, it immediately exits.  Muchsync is conservative about
updating the database, to avoid missing tags or files that have been
changed.  It always updates the notmuch database first, then its own
sqlite database with a version number.  That means if you kill muchsync,
some number of files may get picked up as changed again even though
really they were just copied from a peer.

To mitigate this problem, the muchsync client syncs the database every
10 seconds, so that in theory you should only get 10 seconds of extra
work from killing the client.  However, the server does not sync
periodically, on the assumption that it is more likely to read an EOF
than get killed, although currently it doesn't appear to commit any
pending transactions to the sqlite database upon EOF, which may be an
oversight.

So to summarize:

  * File names are not the same across machine, only file contents and
directory structure.

  * Give muchsync lots of -v options to see what it is doing.

  * Try to avoid killing muchsync.  Doing so is safe, but likely to
generate extra work in the form of phantom renames or tag changes
that get synchronized even though they don't need to be.

  * Possibly the server should handle EOF more gracefully and commit any
pending transactions, or the client should periodically send a
commit command to the server.

If you think something is wrong, I can help you figure it out, but I
need to know what maildir.synchronize_flags is set to on each replica,
what you mean by canceled, and roughly what was happening when you
canceled (uploading or downloading).

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Enabling and disabling maildir.synchronize_flags

[David Mazieres]

David Bremner  writes:

> dm-list-email-notmuch at scs.stanford.edu writes:
>
>> It seems that disabling it should simply be safe.  But re-enabling, one
>> risks losing tags, as the next notmuch new will cause old maildir flags
>> to override the xapian database.  So that suggests something like:
>>
>>notmuch dump > backup
>>notmuch config set maildir.synchronize_flags false
>># Do I need to run notmuch new here?
>>notmuch restore < backup
>
> Hi David;
>
> Sorry about the long delay. I'm not sure I follow the connection between
> your paragraph above and the the example. The example seems safe, but as
> you say, disabling synching should bs safe anyway.

It's not an example, it's kind of a worst case scenario if there's no
easy and safe way way to enable synchronize_flags.  I want to try
disabling flags, but if I change my mind and the only way to get it back
is to do a full notmuch dump/restore, that's a pretty hefty penalty.
Also, even if I do a full notmuch dump / restore, I'm not sure if the
notmuch new is necessary in the middle.

So my question remains, what's the easiest safe way to re-enable
synchronize_flags after disabling it?  (Safe meaning it won't change any
tags.)  It could be that there's a very simple answer, in which case
sticking it in the man page might be nice.

Thanks,
David

Re: Enabling and disabling maildir.synchronize_flags

[David Mazieres]

David Bremner da...@tethera.net writes:

 dm-list-email-notm...@scs.stanford.edu writes:

 It seems that disabling it should simply be safe.  But re-enabling, one
 risks losing tags, as the next notmuch new will cause old maildir flags
 to override the xapian database.  So that suggests something like:

notmuch dump  backup
notmuch config set maildir.synchronize_flags false
# Do I need to run notmuch new here?
notmuch restore  backup

 Hi David;

 Sorry about the long delay. I'm not sure I follow the connection between
 your paragraph above and the the example. The example seems safe, but as
 you say, disabling synching should bs safe anyway.

It's not an example, it's kind of a worst case scenario if there's no
easy and safe way way to enable synchronize_flags.  I want to try
disabling flags, but if I change my mind and the only way to get it back
is to do a full notmuch dump/restore, that's a pretty hefty penalty.
Also, even if I do a full notmuch dump / restore, I'm not sure if the
notmuch new is necessary in the middle.

So my question remains, what's the easiest safe way to re-enable
synchronize_flags after disabling it?  (Safe meaning it won't change any
tags.)  It could be that there's a very simple answer, in which case
sticking it in the man page might be nice.

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

ANNOUNCE: muchsync 2 released

[David Mazieres]

I'm pleased to announce the release of muchsync version 2.  Muchsync is
a tool for replicating and synchronizing your notmuch databases across
machines.

The new version is reported to build out of the box on Mac OS X.

There's one new feature, which is a new notmuch config option,
muchsync.and_tags.  When a message experiences an update conflict,
muchsync deletes any tag in muchsync.and_tags unless that tag is present
in both replicas (a logical and).  For all other tags, muchsync adds the
tag if it exists on either replica (a logical or).  The default for
muchsync.and_tags is to use the value of new.tags, as in previous
versions of muchsync.

As usual, the distribution is available from the muchsync home page:

http://www.muchsync.org/

Enjoy.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Modify message after send...?

[David Mazieres]

mailinglists at nawaz.org writes:

> Hi,
>
> I use notmuch via Emacs.
>
> Here's what I want:
>
> When I hit C-c C-c to send a message, I'd like it to be passed to a
> script (likely a Python one, although I may consider an Elisp function
> if an external script is not possible) for modification of headers,
> before it is sent to the MTA (Postfix in my case). A bonus would be to
> have the modified message stored in the FCC location, instead of the
> original one.
>
> Is this possible? An alternative may be to modify the message /before/
> it goes to message-send-and-exit. I'm inexperienced in Elisp - would
> this be via what's called "advising"?
>
> BTW, all I really want to do is modify the From: field based on the
> recipients (for every email, with no default From). I'll welcome
> suggestions for existing ways to do that. I Googled a little, but didn't
> find a clear good solution. Furthermore, I expect over time the rules by
> which I pick the From: field will get more complex than my knowledge of
> Elisp.

For modifying the From field, I recommend doing it before you send the
message, as this gives you an opportunity to see what you are about to
send and possibly edit it by hand.

I do something like this using defadvice around notmuch-mua-mail, do
adjust things about my messages.  I also put a defadvice around
notmuch-call-notmuch-sexp to filter the To and Cc headers I get from
"notmuch reply" (because notmuch won't let me put wildcards in my
.notmuch-config file).

Finally, and possibly most relevant to you, I use a message-send-hook to
insert the Return-Path address and make a few more modifications before
sending a message.  Note that I only do this if there isn't already a
Return-Path header.  That way, if I type C-c C-c and don't like what I
see, I edit the result and send again, leaving the generated Return-Path
header, and this time it goes through unmodified.

David

notmuch-lazysync -- synchronizing tags using dropbox

[David Mazieres]

Daniel Schoepe  writes:

> The way tag changes are logged is a bit of a hack, but it could be
> improved in the future by adding a post-tag hook to notmuch.

One thing to look into, if you are thinking of a better logging
mechanism, is that Xapian itself has a change logging mechanism for
replicating databases (http://xapian.org/docs/replication.html).

I do think it would be cleaner to do this in a way that is integrated
with notmuch, but I think the best way to do this is to integrate a
"modtime" value into the Xapian database.  Having a modtime for each
record would not only allow incremental transfers (just record the
highest timestamp sent to each replica), it would also solve this
terrible problem that in emacs you can end up tagging messages you don't
see (because you apply a tag to the query result, when new mail has come
in--which would be solved by tagging only through the higest modtime
actually displayed).

When you have one mechanism (modtime) that solves multiple problems, it
is likely the right thing to use...

David

Re: Modify message after send...?

[David Mazieres]

mailingli...@nawaz.org writes:

 Hi,

 I use notmuch via Emacs.

 Here's what I want:

 When I hit C-c C-c to send a message, I'd like it to be passed to a
 script (likely a Python one, although I may consider an Elisp function
 if an external script is not possible) for modification of headers,
 before it is sent to the MTA (Postfix in my case). A bonus would be to
 have the modified message stored in the FCC location, instead of the
 original one.

 Is this possible? An alternative may be to modify the message /before/
 it goes to message-send-and-exit. I'm inexperienced in Elisp - would
 this be via what's called advising?

 BTW, all I really want to do is modify the From: field based on the
 recipients (for every email, with no default From). I'll welcome
 suggestions for existing ways to do that. I Googled a little, but didn't
 find a clear good solution. Furthermore, I expect over time the rules by
 which I pick the From: field will get more complex than my knowledge of
 Elisp.

For modifying the From field, I recommend doing it before you send the
message, as this gives you an opportunity to see what you are about to
send and possibly edit it by hand.

I do something like this using defadvice around notmuch-mua-mail, do
adjust things about my messages.  I also put a defadvice around
notmuch-call-notmuch-sexp to filter the To and Cc headers I get from
notmuch reply (because notmuch won't let me put wildcards in my
.notmuch-config file).

Finally, and possibly most relevant to you, I use a message-send-hook to
insert the Return-Path address and make a few more modifications before
sending a message.  Note that I only do this if there isn't already a
Return-Path header.  That way, if I type C-c C-c and don't like what I
see, I edit the result and send again, leaving the generated Return-Path
header, and this time it goes through unmodified.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch-lazysync -- synchronizing tags using dropbox

[David Mazieres]

Daniel Schoepe dan...@schoepe.org writes:

 The way tag changes are logged is a bit of a hack, but it could be
 improved in the future by adding a post-tag hook to notmuch.

One thing to look into, if you are thinking of a better logging
mechanism, is that Xapian itself has a change logging mechanism for
replicating databases (http://xapian.org/docs/replication.html).

I do think it would be cleaner to do this in a way that is integrated
with notmuch, but I think the best way to do this is to integrate a
modtime value into the Xapian database.  Having a modtime for each
record would not only allow incremental transfers (just record the
highest timestamp sent to each replica), it would also solve this
terrible problem that in emacs you can end up tagging messages you don't
see (because you apply a tag to the query result, when new mail has come
in--which would be solved by tagging only through the higest modtime
actually displayed).

When you have one mechanism (modtime) that solves multiple problems, it
is likely the right thing to use...

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

ANNOUNCE: muchsync 1 - share notmuch DB across machines

[David Mazieres]

Suvayu Ali  writes:

> I noticed that the configure script didn't notice notmuch-devel was not
> installed on my system.  I noticed it when the compilation failed.  Bug?

What OS and distribution are you using?  notmuch-devel sounds like
something a linux distribution might do, rather than anything that's
part of notmuch itself.  On arch linux, for example, there's just a
notmuch package that includes notmuch.

Obviously there's no way to compile muchsync without notmuch installed.
But if you elaborate on what it is you saw and what you would like to
see I might be able to address it in the next release.

> Besides that, I had a question.  I would like to synchronize just the
> tags, not the maildirs, I want to use OfflineIMAP for that.  Is that
> possible?

Not really, no.  You could run offlineimap followed by muchsync, but you
run the risk that mail will be delivered in the meantime.  You could
also run offlineimap to fetch your mail in the first place, and then
muchsync to sync it between your devices.

Can I ask why you would want to do this, anyway?  Muchsync should be
faster than notmuch, particularly if you have a lot of mail directories
and/or are going to pay the cost of tag synchronization anyway.

David

Re: ANNOUNCE: muchsync 1 - share notmuch DB across machines

[David Mazieres]

Suvayu Ali fatkasuvayu+li...@gmail.com writes:

 I noticed that the configure script didn't notice notmuch-devel was not
 installed on my system.  I noticed it when the compilation failed.  Bug?

What OS and distribution are you using?  notmuch-devel sounds like
something a linux distribution might do, rather than anything that's
part of notmuch itself.  On arch linux, for example, there's just a
notmuch package that includes notmuch.

Obviously there's no way to compile muchsync without notmuch installed.
But if you elaborate on what it is you saw and what you would like to
see I might be able to address it in the next release.

 Besides that, I had a question.  I would like to synchronize just the
 tags, not the maildirs, I want to use OfflineIMAP for that.  Is that
 possible?

Not really, no.  You could run offlineimap followed by muchsync, but you
run the risk that mail will be delivered in the meantime.  You could
also run offlineimap to fetch your mail in the first place, and then
muchsync to sync it between your devices.

Can I ask why you would want to do this, anyway?  Muchsync should be
faster than notmuch, particularly if you have a lot of mail directories
and/or are going to pay the cost of tag synchronization anyway.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

ANNOUNCE: muchsync 1 - share notmuch DB across machines

[David Mazieres]

I've just released a new version of muchsync, the tool for replicating
your notmuch database on multiple machines.  The source release is
available at:

http://www.muchsync.org/src/muchsync-1.tar.gz

The new version fixes a bug that incorrectly synchronized tags
containing '%' characters.  It also fixes several build problems people
reported.  Hence, the new version should work out of the box on more
operating systems.

More information about muchsync is available on the project home page:

http://www.muchsync.org/

Enjoy,
David

ANNOUNCE: muchsync 1 - share notmuch DB across machines

[David Mazieres]

I've just released a new version of muchsync, the tool for replicating
your notmuch database on multiple machines.  The source release is
available at:

http://www.muchsync.org/src/muchsync-1.tar.gz

The new version fixes a bug that incorrectly synchronized tags
containing '%' characters.  It also fixes several build problems people
reported.  Hence, the new version should work out of the box on more
operating systems.

More information about muchsync is available on the project home page:

http://www.muchsync.org/

Enjoy,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

How do I prevent notmuch-show in Emacs from automatically following links in emails?

[David Mazieres]

Philip  writes:

> Hi,
>
>   When I open emails in the notmuch-show mode on Emacs, it automatically
>   follows links which are present in the email. For instance, it
>   connects to various tracking objects included in the email. How can I
>   prevent it from doing this automatically all the time? Is there a
>   setting by which notmuch-show will not follow links in emails?

If you just want something quick and dirty, put the following in your
.emacs file:

(setq gnus-inhibit-images t)

I do that, and then in the very rare occasions where I want to see all
the images, I just open the message in my browser by typing ". v" on the
html part button.

I'm sure the patch Tomi linked to is much better, but this works with
unmodified notmuch.

David

Re: How do I prevent notmuch-show in Emacs from automatically following links in emails?

[David Mazieres]

Philip notm...@accounts.gphilip.in writes:

 Hi,

   When I open emails in the notmuch-show mode on Emacs, it automatically
   follows links which are present in the email. For instance, it
   connects to various tracking objects included in the email. How can I
   prevent it from doing this automatically all the time? Is there a
   setting by which notmuch-show will not follow links in emails?

If you just want something quick and dirty, put the following in your
.emacs file:

(setq gnus-inhibit-images t)

I do that, and then in the very rare occasions where I want to see all
the images, I just open the message in my browser by typing . v on the
html part button.

I'm sure the patch Tomi linked to is much better, but this works with
unmodified notmuch.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

ANNOUNCE: muchsync 0 - share notmuch DB across machines

[David Mazieres]

I'm pleased to announce the release of muchsync:

http://www.muchsync.org/

Muchsync is a notmuch mail synchronizer.  It lets you keep a full copy
of your notmuch database on every one of your computers.  Muchsync
properly synchronizes maildir contents and notmuch tags, obeying the no
lost updates rule and sensibly but conservatively resolving update
conflicts with no manual intervention.

After initialization (which is slow because it has to download and
re-index all of your mail), muchsync is bandwidth efficient and works
well over high-latency networks.  It is much faster than alternatives
such as offlineimap, particularly when you have lots of different
maildirs.

I originally had plans to rename the utility syncmuch and add tons of
features.  However, as soon as I ran the first tests of the system, I
found that I could not go back to any other way of reading email and my
test became production use.  Just this base functionality has been
enough to solve all of my email problems for the last 13 months.  I'm
therefore releasing it in the hopes that others will similarly find it
useful.  (And if muchsync ever takes off, there are small changes to the
notmuch API we can discuss that would make it even more efficient...)

David

ANNOUNCE: muchsync 0 - share notmuch DB across machines

[David Mazieres]

I'm pleased to announce the release of muchsync:

http://www.muchsync.org/

Muchsync is a notmuch mail synchronizer.  It lets you keep a full copy
of your notmuch database on every one of your computers.  Muchsync
properly synchronizes maildir contents and notmuch tags, obeying the no
lost updates rule and sensibly but conservatively resolving update
conflicts with no manual intervention.

After initialization (which is slow because it has to download and
re-index all of your mail), muchsync is bandwidth efficient and works
well over high-latency networks.  It is much faster than alternatives
such as offlineimap, particularly when you have lots of different
maildirs.

I originally had plans to rename the utility syncmuch and add tons of
features.  However, as soon as I ran the first tests of the system, I
found that I could not go back to any other way of reading email and my
test became production use.  Just this base functionality has been
enough to solve all of my email problems for the last 13 months.  I'm
therefore releasing it in the hopes that others will similarly find it
useful.  (And if muchsync ever takes off, there are small changes to the
notmuch API we can discuss that would make it even more efficient...)

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

folder and path completely broken in HEAD?

[David Mazieres expires 2014-07-31 PDT]

Mark Walters  writes:

> Hi
>
> Before checking other things: have you run notmuch new? That's needed to
> update the database. It is an irreversible update so notmuch-0.17 will
> not work with the updated database.

No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
The discussion of the NEWS was a bit confusing, so I wanted to check it
out.  I knew something had to be very wrong.

Weill the new primitives still allow me to group my mail hierarchically
in searches?  The new man page is not totally clear on what is being
matched.

Thanks,
David

Re: folder and path completely broken in HEAD?

[David Mazieres expires 2014-07-31 PDT]

Mark Walters markwalters1...@gmail.com writes:

 Hi

 Before checking other things: have you run notmuch new? That's needed to
 update the database. It is an irreversible update so notmuch-0.17 will
 not work with the updated database.

No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
The discussion of the NEWS was a bit confusing, so I wanted to check it
out.  I knew something had to be very wrong.

Weill the new primitives still allow me to group my mail hierarchically
in searches?  The new man page is not totally clear on what is being
matched.

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: github mirror

[David Mazieres expires 2014-07-26 PDT]

Sam Halliday sam.halli...@gmail.com writes:

 David Mazieres dm-list-email-notm...@scs.stanford.edu writes:
 The problem is that different imap servers store tags in different
 ways.  Since notmuch does not use imap, it would be hard for notmuch to
 synchronize the tags, other than the standard ones (for which notmuch
 already has support).

 One thing you could do is build an external tool that synchronizes
 notmuch tags and spawns an imap server in preauth mode to sync the tags.
 (That would be yet another use for the ctime values we have discussed on
 this list.)

 The improvements to offlineimap to use the mail header hack might work
 well for both of us. Currently the only way to add/remove labels (a
 gmail concept) is to copy/move mail between folders. And this is how
 notmuch tags are synced. But with outstanding pull request, this can
 all be managed via email headers and that means you *only* need to
 synchronise your All Mail folder.

 So, I'd be interested to see what your code could do in that world :-)

My code assumes shell access to your mail server, so it doesn't do squat
in the gmail world.  I suppose you could use gmail just as your SMTP
server, then download everything to your own server with offlineimap and
manage it with notmuch there, in which case my code gives you the
notmuch experience on all your devices.  However, there's a much better
SMTP server out than what google is running (www.mailavenger.org), so
such a setups is backwards... The right thing to do is to receive
everything on a mail server that you control, then push to gmail only
what you want to read on your phone and/or disclose to the NSA (which in
my case is only a tiny fraction of my email).

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

github mirror

[David Mazieres]

Gaute Hope  writes:

>> Really what you want is an imap server built on top of the notmuch
>> library.  That way you could use notmuch from your desktop and then use
>> imap from your phone, and everything would stay perfectly in sync.
>> Implementing such a server wouldn't be that hard, but it would help if
>> notmuch made the _notmuch_message_get_doc_id and
>> _notmuch_directory_get_document_id functions semi-public.  Then the imap
>> server could just use docids as uids.  (Plus then muchsync wouldn't have
>> to go through gross contortions to get docids information...)
>
> That would be nice, but a solution where the user does not need to run
> his own server is in my opinion pretty essential.

You don't have to "run" an imap server, you can use it in preauth mode.
Having a notmuch-based imap server would let you synchronize gmail with
your laptop and then read email offline, for instance.

David

github mirror

[David Mazieres]

Austin Clements  writes:

> As for storing this information directly in messages, in general, the
> notmuch community is opposed to modifying messages.  This causes many
> problems, and immutable messages are more robust and simplify so many
> things.  IMAP assumes messages are immutable.  Maildir assumes
> messages are immutable.  Notmuch new would get dramatically slower if
> it had to check for messages modifications.  What do you do if you
> change a tag and there are multiple copies of a message?  What do you
> do if there are multiple copies and they disagree about the tags?  How
> do you atomically update the tags stored in a message?  From an
> engineering standpoint, it's much better to avoid mutable messages.

The speed penalty would be very minor in the common case.  Muchsync
scans directories (since it has to scan file contents) and the cost to
compute SHA-1 hashes of modified files is under 50 msec or something in
the common case.  Extracting tags would be even cheaper.  The reason is
that A) you only need to scan modified directories, and B) you don't
need to open the file unless the inode, mtime, or size has changed.
Originally I was going to implement an optimization to detect renamed
files and avoid computing SHA-1 again (for the case where maildir flags
have changed), but in the end this wasn't even worth it because the cost
is so small.

That said, I agree that the complexity of altering files is not worth
it.  Especially since most imap servers will not know about this.  Also,
the question of what do you do with duplicate message IDs (which is
effectively what you have when the tags disagree) is a more general
problem still needing a solution, and would be exacerbated by embedding
important information like tags in the message.

Really what you want is an imap server built on top of the notmuch
library.  That way you could use notmuch from your desktop and then use
imap from your phone, and everything would stay perfectly in sync.
Implementing such a server wouldn't be that hard, but it would help if
notmuch made the _notmuch_message_get_doc_id and
_notmuch_directory_get_document_id functions semi-public.  Then the imap
server could just use docids as uids.  (Plus then muchsync wouldn't have
to go through gross contortions to get docids information...)

David

github mirror

[David Mazieres expires 2014-07-26 PDT]

Sam Halliday  writes:

> David Mazieres  writes:
>> The problem is that different imap servers store tags in different
>> ways.  Since notmuch does not use imap, it would be hard for notmuch to
>> synchronize the tags, other than the standard ones (for which notmuch
>> already has support).
>>
>> One thing you could do is build an external tool that synchronizes
>> notmuch tags and spawns an imap server in preauth mode to sync the tags.
>> (That would be yet another use for the ctime values we have discussed on
>> this list.)
>
> The improvements to offlineimap to use the mail header hack might work
> well for both of us. Currently the only way to add/remove "labels" (a
> gmail concept) is to copy/move mail between folders. And this is how
> notmuch "tags" are synced. But with outstanding pull request, this can
> all be managed via email headers and that means you *only* need to
> synchronise your "All Mail" folder.
>
> So, I'd be interested to see what your code could do in that world :-)

My code assumes shell access to your mail server, so it doesn't do squat
in the gmail world.  I suppose you could use gmail just as your SMTP
server, then download everything to your own server with offlineimap and
manage it with notmuch there, in which case my code gives you the
notmuch experience on all your devices.  However, there's a much better
SMTP server out than what google is running (www.mailavenger.org), so
such a setups is backwards... The right thing to do is to receive
everything on a mail server that you control, then push to gmail only
what you want to read on your phone and/or disclose to the NSA (which in
my case is only a tiny fraction of my email).

David

github mirror

[David Mazieres]

Sam Halliday  writes:

> Dear NotMuch,
>
> But in any case, my RFE/question was this: how hard would it be to have
> an optional mode of behaviour where tags are stored in the message
> itself, so that syncing with an IMAP server (e.g. via offlineimap)
> would make the tags available on all devices. This would negate the need
> for workarounds, such as shared notmuch databases, when users have
> multiple machines.

The problem is that different imap servers store tags in different
ways.  Since notmuch does not use imap, it would be hard for notmuch to
synchronize the tags, other than the standard ones (for which notmuch
already has support).

One thing you could do is build an external tool that synchronizes
notmuch tags and spawns an imap server in preauth mode to sync the tags.
(That would be yet another use for the ctime values we have discussed on
this list.)

> It would also allow applications like offlineimap to introduce a gmail
> plugin that would copy the message into a folder according to its tags,
> so gmail labels and notmuch tags would be in sync.

Unfortunately, offlineimap is dog slow, especially when you have a ton
of maildirs.  It just doesn't seem to pipeline very well at all.  Even
when I run 5 or 10 threads, it still seems to take a whole bunch of
round trip times.

I used to want what you want.  But then it got to the point that
offlineimap was taking 30 seconds to sync my 60 maildirs over 3G, even
when there was no new mail.  Worse, one thread seems to busy-wait for
the others to finish, so it often pegs the CPU and drains the battery on
my laptop.

As a result, I built a tool that synchronizes notmuch tags directly.
Now my problem is waiting for "notmuch new" to run, but at least the
synchronization part is really fast and pleasant.  I'll be releasing my
tool in the next couple of months once I've kicked the tires a bit more
and fixed up a few things.

David

Re: github mirror

[David Mazieres]

Sam Halliday sam.halli...@gmail.com writes:

 Dear NotMuch,

 But in any case, my RFE/question was this: how hard would it be to have
 an optional mode of behaviour where tags are stored in the message
 itself, so that syncing with an IMAP server (e.g. via offlineimap)
 would make the tags available on all devices. This would negate the need
 for workarounds, such as shared notmuch databases, when users have
 multiple machines.

The problem is that different imap servers store tags in different
ways.  Since notmuch does not use imap, it would be hard for notmuch to
synchronize the tags, other than the standard ones (for which notmuch
already has support).

One thing you could do is build an external tool that synchronizes
notmuch tags and spawns an imap server in preauth mode to sync the tags.
(That would be yet another use for the ctime values we have discussed on
this list.)

 It would also allow applications like offlineimap to introduce a gmail
 plugin that would copy the message into a folder according to its tags,
 so gmail labels and notmuch tags would be in sync.

Unfortunately, offlineimap is dog slow, especially when you have a ton
of maildirs.  It just doesn't seem to pipeline very well at all.  Even
when I run 5 or 10 threads, it still seems to take a whole bunch of
round trip times.

I used to want what you want.  But then it got to the point that
offlineimap was taking 30 seconds to sync my 60 maildirs over 3G, even
when there was no new mail.  Worse, one thread seems to busy-wait for
the others to finish, so it often pegs the CPU and drains the battery on
my laptop.

As a result, I built a tool that synchronizes notmuch tags directly.
Now my problem is waiting for notmuch new to run, but at least the
synchronization part is really fast and pleasant.  I'll be releasing my
tool in the next couple of months once I've kicked the tires a bit more
and fixed up a few things.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: github mirror

[David Mazieres]

Austin Clements amdra...@mit.edu writes:

 As for storing this information directly in messages, in general, the
 notmuch community is opposed to modifying messages.  This causes many
 problems, and immutable messages are more robust and simplify so many
 things.  IMAP assumes messages are immutable.  Maildir assumes
 messages are immutable.  Notmuch new would get dramatically slower if
 it had to check for messages modifications.  What do you do if you
 change a tag and there are multiple copies of a message?  What do you
 do if there are multiple copies and they disagree about the tags?  How
 do you atomically update the tags stored in a message?  From an
 engineering standpoint, it's much better to avoid mutable messages.

The speed penalty would be very minor in the common case.  Muchsync
scans directories (since it has to scan file contents) and the cost to
compute SHA-1 hashes of modified files is under 50 msec or something in
the common case.  Extracting tags would be even cheaper.  The reason is
that A) you only need to scan modified directories, and B) you don't
need to open the file unless the inode, mtime, or size has changed.
Originally I was going to implement an optimization to detect renamed
files and avoid computing SHA-1 again (for the case where maildir flags
have changed), but in the end this wasn't even worth it because the cost
is so small.

That said, I agree that the complexity of altering files is not worth
it.  Especially since most imap servers will not know about this.  Also,
the question of what do you do with duplicate message IDs (which is
effectively what you have when the tags disagree) is a more general
problem still needing a solution, and would be exacerbated by embedding
important information like tags in the message.

Really what you want is an imap server built on top of the notmuch
library.  That way you could use notmuch from your desktop and then use
imap from your phone, and everything would stay perfectly in sync.
Implementing such a server wouldn't be that hard, but it would help if
notmuch made the _notmuch_message_get_doc_id and
_notmuch_directory_get_document_id functions semi-public.  Then the imap
server could just use docids as uids.  (Plus then muchsync wouldn't have
to go through gross contortions to get docids information...)

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Add configurable changed tag to messages that have been changed on disk

[David Mazieres expires 2014-07-22 PDT]

Austin Clements amdra...@mit.edu writes:

 I'd like to have efficient change detection, too.  In my case, I'd
 like to use it to support efficient live search and show updates.  The
 design I'd sketched out for that used a log rather than ctimes, and
 I'm curious if you have thoughts on the relative merits and
 suitability for tag sync of these approaches.

Both logging ctime are very general mechanisms than can solve many
problems.  For example, is there still an issue that pressing * in
emacs notmuch-search mode can affect messages that are not visible if
someone ran notmuch new in a different window?  ctimes would be one way
to solve this.  (Conservatively exempt any messages that have changed
since the displayed search was run.)

 I'd been leaning toward logging because it can capture changes to
 things that aren't represented as documents in the database, such as
 thread membership.  This probably doesn't matter for synchronization,
 but it makes it much easier to figure out which threads in thread
 search results need to be refreshed.  A log can also capture message
 deletion easily, while ctimes would require tombstones (which may be a
 good idea for other reasons [1]).

 On the other hand, logging is obviously more mechanism than ctimes,
 and probably requires some garbage collection story.

The advantage of ctime over logging is that the interface is super
simple and intuitive.  How would the interface to the log work?

In terms of implementation, have you thought about leveraging the
XAPIAN_MAX_CHANGESETS mechanism to produce your logs?

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] Add configurable changed tag to messages that have been changed on disk

[David Mazieres expires 2014-07-22 PDT]

Austin Clements  writes:

> I'd like to have efficient change detection, too.  In my case, I'd
> like to use it to support efficient live search and show updates.  The
> design I'd sketched out for that used a log rather than ctimes, and
> I'm curious if you have thoughts on the relative merits and
> suitability for tag sync of these approaches.

Both logging ctime are very general mechanisms than can solve many
problems.  For example, is there still an issue that pressing "*" in
emacs notmuch-search mode can affect messages that are not visible if
someone ran notmuch new in a different window?  ctimes would be one way
to solve this.  (Conservatively exempt any messages that have changed
since the displayed search was run.)

> I'd been leaning toward logging because it can capture changes to
> things that aren't represented as documents in the database, such as
> thread membership.  This probably doesn't matter for synchronization,
> but it makes it much easier to figure out which threads in thread
> search results need to be refreshed.  A log can also capture message
> deletion easily, while ctimes would require tombstones (which may be a
> good idea for other reasons [1]).
>
> On the other hand, logging is obviously more mechanism than ctimes,
> and probably requires some garbage collection story.

The advantage of ctime over logging is that the interface is super
simple and intuitive.  How would the interface to the log work?

In terms of implementation, have you thought about leveraging the
XAPIAN_MAX_CHANGESETS mechanism to produce your logs?

David

[PATCH] Add configurable changed tag to messages that have been changed on disk

[David Mazieres]

Gaute Hope  writes:

> A db-tick or a _good_ ctime solution can as far as I can see solve both
> David M's (correct me if I am wrong) and my purposes, as well as
> probably have more use cases in the future. It would even be an
> interesting direct search: show me everything that changed lately,
> sorted.

I could live with a db-tick scheme.  I would prefer a ctime scheme,
since then I can answer questions such as "what has changed in the last
five minutes"?  I mean all kinds of other stuff starts to break if your
clock goes backwards on a mail server machine, not the least of which is
that incremental backups will fail silently, so you risk losing your
mail.

A middle ground might be to use the maximum of two values: 1) the
time-of-day at which notmuch started executing, and 2) the highest ctime
in the database plus 100 microseconds (leaving plenty of slop to store
timestamps as IEEE doubles with 52 significant bits).  Since the values
will be Btree-indexed, computing the max plus one will be cheap.

Incidentally, if you are really this paranoid about time stamps, it
should bother you that notmuch's directory timestamps only have one
second granularity.  It's not that hard to get a new message delivered
in the same second that notmuch new finished running.  In my
synchronizer, I convert st_mtim (a struct timespec) into a double and
keep that plus size in the database to decide if I need to re-hash
files.  But for directories, I'm stuck with NOTMUCH_VALUE_TIMESTAMP,
which are quantized to the second.  (Ironically, I think
Xapian::sortable_serialize converts time_ts to doubles anyway, so
avoiding st_mtim is not really helping performance.)

David

notmuch, OpenBSD issues

[David Mazieres]

David Bremner  writes:

> Allan Streib  writes:
>
>> I've been using notmuch on OpenBSD for a while, and I really appreciate
>> the project.

Since there is a thread on this...

I'm using notmuch 0.17 on openbsd (from the ports tree).  My problem is
that notmuch new is just unbearably slow.  I don't know if it's because
I'm running the 32-bit (i386) mode or what, but it takes over one second
per mail message.  E.g., this is typical of what I see when checking for
new mail:

Processed 18 total files in 20s (0 files/sec.).
Added 5 new messages to the database.

Linux is 10 times faster.  Have you seen any similar performance issues?

David

Re: notmuch, OpenBSD issues

[David Mazieres]

David Bremner da...@tethera.net writes:

 Allan Streib astr...@indiana.edu writes:

 I've been using notmuch on OpenBSD for a while, and I really appreciate
 the project.

Since there is a thread on this...

I'm using notmuch 0.17 on openbsd (from the ports tree).  My problem is
that notmuch new is just unbearably slow.  I don't know if it's because
I'm running the 32-bit (i386) mode or what, but it takes over one second
per mail message.  E.g., this is typical of what I see when checking for
new mail:

Processed 18 total files in 20s (0 files/sec.).
Added 5 new messages to the database.

Linux is 10 times faster.  Have you seen any similar performance issues?

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Synchronization success stories?

[David Mazieres]

Tilmann Singer  writes:

> The steps performed on a sync run are roughly like this:
>
> - local: notmuch new
> - local: notmuch search --output=messages ..
> - remote: notmuch new
> - remote: notmuch search --output=messages ..
> - compare search results
> - run rsync for mails that only exist locally
>   (using notmuch search --output=files to get the filenames)
> - run rsync for mails that only exist remotely
>   (using notmuch search --output=files to get the filenames)

What happens if you get a message that's been stuck in a queue for a few
days and has an old Date: header?  Or if you get new messages that have
the same Message-ID as old ones?

> Synchronization of the notmuch tags database is only necessary when I
> switch between different client computers, which happens less
> frequently.

Do you use a laptop everywhere?  I've found that for switching between
my desktop machine at home, my laptop on the train, and my desktop at
work (which amounts to five switches a day), the notmuch dump time is
painfully slow--like well over 10 seconds for 100,000 messages.  Hook
that into notmuch-poll and you have a recipe for hanging emacs every
time you type "G".

Of course, I'm also experiencing the problem of "notmuch new" itself
being painfully slow, but at least that's now my bottleneck in switching
machines.  I suspect the source of the notmuch new problem is that I
have some huge, huge mailboxes.  Some of my maildir/cur directories are
multiple megabytes on a BSD FFS file system (no hashing, so linear
filename lookups in something that doesn't fit in the dcache).  On linux
ext4 things are much faster.  I intend to reorganize my maildir so that
there is a top-level directory with the year and hence no single
directory ever contains mail from more than one year.

David

Re: Synchronization success stories?

[David Mazieres]

Tilmann Singer t...@tils.net writes:

 The steps performed on a sync run are roughly like this:

 - local: notmuch new
 - local: notmuch search --output=messages some time ago..now
 - remote: notmuch new
 - remote: notmuch search --output=messages some time ago..now
 - compare search results
 - run rsync for mails that only exist locally
   (using notmuch search --output=files to get the filenames)
 - run rsync for mails that only exist remotely
   (using notmuch search --output=files to get the filenames)

What happens if you get a message that's been stuck in a queue for a few
days and has an old Date: header?  Or if you get new messages that have
the same Message-ID as old ones?

 Synchronization of the notmuch tags database is only necessary when I
 switch between different client computers, which happens less
 frequently.

Do you use a laptop everywhere?  I've found that for switching between
my desktop machine at home, my laptop on the train, and my desktop at
work (which amounts to five switches a day), the notmuch dump time is
painfully slow--like well over 10 seconds for 100,000 messages.  Hook
that into notmuch-poll and you have a recipe for hanging emacs every
time you type G.

Of course, I'm also experiencing the problem of notmuch new itself
being painfully slow, but at least that's now my bottleneck in switching
machines.  I suspect the source of the notmuch new problem is that I
have some huge, huge mailboxes.  Some of my maildir/cur directories are
multiple megabytes on a BSD FFS file system (no hashing, so linear
filename lookups in something that doesn't fit in the dcache).  On linux
ext4 things are much faster.  I intend to reorganize my maildir so that
there is a top-level directory with the year and hence no single
directory ever contains mail from more than one year.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Synchronization success stories?

[David Mazieres]

David Bremner  writes:

> Brian Sniffen  writes:
>
>> I'm thrilled by using notmuch to manage my mail.  Low-latency search is
>> very important to me.  But I use computers in a couple of
>> places---several of which are laptops.  Has anyone stories to share of
>> successful multi-computer notmuch sync, for a corpus of a
>> quarter-million messages or so?  
>
> I use syncmaildir to sync the actual messages, and a copy of the output
> of "notmuch dump" in git to sync the metadata.
>
> It works OK. A bit slow; depends how often you need to fetch new mail.

If you want to see my solution, it is here:

http://www.scs.stanford.edu/~dm/muchsync-0.tar.gz

I'm a little embarrassed by this code, as I just started to test it a
week ago then instantly became completely dependent on it.  I will
probably change the name (from muchsync to syncmuch) and the database
format before releasing.  But if you feel like beta-testing and giving
me feedback, have a look.

Beware that if you have been using notmuch dump, you may become
instantly hooked on my solution...

David

gnus-alias integration bug

[David Mazieres]

Ian Kelling  writes:

> gnus-alias allows you to set your identity based on the headers in the
> message you are replying. This is an essential feature in a mail client
> for me, I want to automatically reply as the person an email was sent to
> for certain addresses. This does not work in notmuch.

I need this feature, too.  There is some support for the same
functionality built into notmuch.  The big problem for me is that
.notmuch-config does not support wildcards (e.g., *@mydomain.com or
user-*@mydomain.com).  Compounding this issue is that, other than
notmuch reply's internals, notmuch pretty much suppresses the
Delivered-To header, which is crucial for knowing in a non-heuristic way
where an email was actually delivered to.  I want to compute my From:
header based on the Delivered-To, so I created a
mail-delivered-to-reply-from alist that maps one to the other.

It's pretty horrifying what I had to do to get this to work, but it
looks something like the following.  (Apologies for any mistakes, I had
to strip out additional complexity I have for other things in there, so
it's not verbatim the code that I'm using, but it's pretty close.)

If someone knows a better way of doing this with stock notmuch, please
let me know.  I know it's simpler to fix notmuch-reply, but I really
don't want to start depending on a custom-forked version on the notmuch
binary, as I read my mail from multiple operating systems and I'm sure
I'd forget to update one of them.

David


;;; Regular expression matching all of my own email addresses.  Note
;;; this is also used by mail-dont-reply-to (which ships with emacs).
(setq mail-dont-reply-to-names
  (concat "REXEXP MATCHING ALL YOUR EMAIL ALIASES"))

;;; Alist mapping Delivered-To addresses to the address that should be
;;; in the From header of replies.
(setq mail-delivered-to-reply-from
  '(("delivered-to-\\(.*\\)@address.com"
 . "Name ")
;; ...
))

(defun compute-from (email)
  (let ((case-fold-search t)
(subst (assoc-default email mail-delivered-to-reply-from
  #'string-match)))
(and subst
 (concat (notmuch-user-name) " <"
 (replace-match subst t nil email) ">"

(defun get-delivered-to-from-path (path)
  (with-temp-buffer
; It's weird to use sed, but there's also no reason to read a huge
; file into a buffer right here when we reply to messags with
; attachments.  (notmuch show doesn't support --body=false in
; --format=raw mode, and the other modes all get rid of Delivered-To.)
(call-process "sed" nil t t "-ne"
  "/^Delivered-To: */{s///p;q;}; /^$/q" path)
(goto-char (point-max))
(or (bobp) (delete-char -1))
(and (not (bobp)) (buffer-string

(defun get-msg-pathnames (query)
  (notmuch-call-notmuch-sexp
   "search" "--format=sexp" "--format-version=1"
   "--output=files" "--" query))

(defun compute-reply-from (pl)
  (let* ((orig (plist-get pl :original))
 (path (plist-get orig :filename))
 (orig-from (plist-get (plist-get orig :headers) :From))
 (addr
  (cond
   ((get-delivered-to-from-path path))
   ;; in case of maildir.synchronize_flags=true, the pathname
   ;; might already be incorrect, so try getting it again
   ((and (setq path (car (get-msg-pathnames query-string)))
 (get-delivered-to-from-path path)))
   ((string-match mail-dont-reply-to-names
  (setq orig-from (car (rfc822-addresses orig-from
orig-from)
   )))
(and addr (compute-from addr

;;; Somehow no plist-del function in emacs
(defun my-plist-del (plist0 prop)
  (if (eq prop (car plist0))
  (cddr plist0)
(let ((p (cdr plist0)))
  (while (and p (not (eq prop (cadr p
(setq p (cddr p)))
  (if p (setcdr p (cdddr p)))
  plist0)))

(defun fix-reply-sexp (pl)
  (let* ((rh (plist-get pl :reply-headers))
 (from (compute-reply-from pl))
 h)
(if from (setq rh (plist-put rh :From from)))
(and (setq h (plist-get rh :To))
 (setq h (mail-dont-reply-to h))
 (not (equal h ""))
 (setq rh (plist-put rh :To h)))
(and (setq h (plist-get rh :Cc))
 (setq h (mail-dont-reply-to h))
 (if (equal h "")
 (setq rh (my-plist-del rh :Cc))
   (setq rh (plist-put rh :Cc h
(plist-put pl :reply-headers rh)))
(defadvice notmuch-call-notmuch-sexp (after fix-reply activate)
  "Fix From/To/CC headers in reply"
  (if (equal (ad-get-arg 0) "reply")
  (setq ad-return-value (fix-reply-sexp ad-return-value

Re: gnus-alias integration bug

[David Mazieres]

Ian Kelling i...@iankelling.org writes:

 gnus-alias allows you to set your identity based on the headers in the
 message you are replying. This is an essential feature in a mail client
 for me, I want to automatically reply as the person an email was sent to
 for certain addresses. This does not work in notmuch.

I need this feature, too.  There is some support for the same
functionality built into notmuch.  The big problem for me is that
.notmuch-config does not support wildcards (e.g., *@mydomain.com or
user-*@mydomain.com).  Compounding this issue is that, other than
notmuch reply's internals, notmuch pretty much suppresses the
Delivered-To header, which is crucial for knowing in a non-heuristic way
where an email was actually delivered to.  I want to compute my From:
header based on the Delivered-To, so I created a
mail-delivered-to-reply-from alist that maps one to the other.

It's pretty horrifying what I had to do to get this to work, but it
looks something like the following.  (Apologies for any mistakes, I had
to strip out additional complexity I have for other things in there, so
it's not verbatim the code that I'm using, but it's pretty close.)

If someone knows a better way of doing this with stock notmuch, please
let me know.  I know it's simpler to fix notmuch-reply, but I really
don't want to start depending on a custom-forked version on the notmuch
binary, as I read my mail from multiple operating systems and I'm sure
I'd forget to update one of them.

David


;;; Regular expression matching all of my own email addresses.  Note
;;; this is also used by mail-dont-reply-to (which ships with emacs).
(setq mail-dont-reply-to-names
  (concat REXEXP MATCHING ALL YOUR EMAIL ALIASES))

;;; Alist mapping Delivered-To addresses to the address that should be
;;; in the From header of replies.
(setq mail-delivered-to-reply-from
  '((delivered-to-\\(.*\\)@address.com
 . Name send-from-\\1...@address.com)
;; ...
))

(defun compute-from (email)
  (let ((case-fold-search t)
(subst (assoc-default email mail-delivered-to-reply-from
  #'string-match)))
(and subst
 (concat (notmuch-user-name)  
 (replace-match subst t nil email) 

(defun get-delivered-to-from-path (path)
  (with-temp-buffer
; It's weird to use sed, but there's also no reason to read a huge
; file into a buffer right here when we reply to messags with
; attachments.  (notmuch show doesn't support --body=false in
; --format=raw mode, and the other modes all get rid of Delivered-To.)
(call-process sed nil t t -ne
  /^Delivered-To: */{s///p;q;}; /^$/q path)
(goto-char (point-max))
(or (bobp) (delete-char -1))
(and (not (bobp)) (buffer-string

(defun get-msg-pathnames (query)
  (notmuch-call-notmuch-sexp
   search --format=sexp --format-version=1
   --output=files -- query))

(defun compute-reply-from (pl)
  (let* ((orig (plist-get pl :original))
 (path (plist-get orig :filename))
 (orig-from (plist-get (plist-get orig :headers) :From))
 (addr
  (cond
   ((get-delivered-to-from-path path))
   ;; in case of maildir.synchronize_flags=true, the pathname
   ;; might already be incorrect, so try getting it again
   ((and (setq path (car (get-msg-pathnames query-string)))
 (get-delivered-to-from-path path)))
   ((string-match mail-dont-reply-to-names
  (setq orig-from (car (rfc822-addresses orig-from
orig-from)
   )))
(and addr (compute-from addr

;;; Somehow no plist-del function in emacs
(defun my-plist-del (plist0 prop)
  (if (eq prop (car plist0))
  (cddr plist0)
(let ((p (cdr plist0)))
  (while (and p (not (eq prop (cadr p
(setq p (cddr p)))
  (if p (setcdr p (cdddr p)))
  plist0)))

(defun fix-reply-sexp (pl)
  (let* ((rh (plist-get pl :reply-headers))
 (from (compute-reply-from pl))
 h)
(if from (setq rh (plist-put rh :From from)))
(and (setq h (plist-get rh :To))
 (setq h (mail-dont-reply-to h))
 (not (equal h ))
 (setq rh (plist-put rh :To h)))
(and (setq h (plist-get rh :Cc))
 (setq h (mail-dont-reply-to h))
 (if (equal h )
 (setq rh (my-plist-del rh :Cc))
   (setq rh (plist-put rh :Cc h
(plist-put pl :reply-headers rh)))
(defadvice notmuch-call-notmuch-sexp (after fix-reply activate)
  Fix From/To/CC headers in reply
  (if (equal (ad-get-arg 0) reply)
  (setq ad-return-value (fix-reply-sexp ad-return-value
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] Add configurable changed tag to messages that have been changed on disk

[David Mazieres]

Gaute Hope  writes:

> When one of the source files for a message is changed on disk, renamed,
> deleted or a new source file is added. A configurable changed tag is
> is added. The tag can be configured under the option 'changed_tags' in
> the [new] section, the default is none. Tests have been updated to
> accept the new config option.
>
> notmuch-setup now asks for a changed tag after the new tags question.
>
> This could be useful for for example 'afew' to detect remote changes in
> IMAP folders and update the FolderNameFilter to also add tags or remove
> tags when a _existing_ message has been added to or removed from a
> maildir.

I think this is the wrong way to achieve such functionality, because
then the change tag A) is expensive to remove, B) is easy to misuse
(remember to call fsync everywhere before deleting the change tag), and
C) can be used by only one application.

A better approach would be to add a new "modtime" xapian value that is
updated whenever the tags or any other terms (such as XFDIRENTRY) are
added to or deleted from a docid.  If it's a Xapian value, rather than a
term, then modtime will be queriable just like date, allowing multiple
applications to query all docids modified since the last time they ran.

I currently have multiple applications that could significantly benefit
from such a modtime.  An obvious one is proper incremental backups with
notmuch-dump.

Another example is a tool I have that synchromizes maildirs and notmuch
tags across machines.  With the current interface, there is no way to do
this without scanning the entire database, because any message, even a
very old one, may have changed tags or links.  Moreover, something like
notmuch-dump is way, way too slow to run every time you want to check
for new mail.  notmuch-dump costs 5-10 seconds on my 110,000-message
maildir!  In fact, any approach the gathers tags associated with each
individual docid is a complete non-starter, forcing me to violate
abstraction and examine the postlists associated with each tag and
XFDIRENTRY term.  Even my highly optimized implementation takes about
250 msec (1400 msec on a 32-bit machine), which adds perceptible latency
to synchronizing my clients' notmuch maildirs with my server's when I
poll for new mail.

Yet another application is something like nottoomuch-addresses, which
currently uses an occasionally incorrect heuristic to detect new
messages based on the Date header.

Let me make a stronger statement, which is that not only are
modification times an incredibly useful and general primitive, but lack
of modification times is the single thing that kept me away from notmuch
despite years of wanting to switch.  In the end, I invested months
developing a highly-optimized change detector that efficiently diffs
Xapian's Btrees against a mysql database with a snapshot of the same
information.  My solution works, and I now enjoy a replicated notmuch
setup synchronized across three machines, including offline access on my
laptop.  But my 4,000-line C++ program might have been a 400-line shell
script if only notmuch supported docid mod times.

Also, to put this in perspective, how long does it take to remove the
changed tags from a bunch of messages?  If it's longer than 300 msec on
a 64-bit machine, then even with a single application you'd be better
off using my crazy on-the-side mysql version vector scheme.

David

Re: [PATCH] Add configurable changed tag to messages that have been changed on disk

[David Mazieres]

Gaute Hope e...@gaute.vetsj.com writes:

 When one of the source files for a message is changed on disk, renamed,
 deleted or a new source file is added. A configurable changed tag is
 is added. The tag can be configured under the option 'changed_tags' in
 the [new] section, the default is none. Tests have been updated to
 accept the new config option.

 notmuch-setup now asks for a changed tag after the new tags question.

 This could be useful for for example 'afew' to detect remote changes in
 IMAP folders and update the FolderNameFilter to also add tags or remove
 tags when a _existing_ message has been added to or removed from a
 maildir.

I think this is the wrong way to achieve such functionality, because
then the change tag A) is expensive to remove, B) is easy to misuse
(remember to call fsync everywhere before deleting the change tag), and
C) can be used by only one application.

A better approach would be to add a new modtime xapian value that is
updated whenever the tags or any other terms (such as XFDIRENTRY) are
added to or deleted from a docid.  If it's a Xapian value, rather than a
term, then modtime will be queriable just like date, allowing multiple
applications to query all docids modified since the last time they ran.

I currently have multiple applications that could significantly benefit
from such a modtime.  An obvious one is proper incremental backups with
notmuch-dump.

Another example is a tool I have that synchromizes maildirs and notmuch
tags across machines.  With the current interface, there is no way to do
this without scanning the entire database, because any message, even a
very old one, may have changed tags or links.  Moreover, something like
notmuch-dump is way, way too slow to run every time you want to check
for new mail.  notmuch-dump costs 5-10 seconds on my 110,000-message
maildir!  In fact, any approach the gathers tags associated with each
individual docid is a complete non-starter, forcing me to violate
abstraction and examine the postlists associated with each tag and
XFDIRENTRY term.  Even my highly optimized implementation takes about
250 msec (1400 msec on a 32-bit machine), which adds perceptible latency
to synchronizing my clients' notmuch maildirs with my server's when I
poll for new mail.

Yet another application is something like nottoomuch-addresses, which
currently uses an occasionally incorrect heuristic to detect new
messages based on the Date header.

Let me make a stronger statement, which is that not only are
modification times an incredibly useful and general primitive, but lack
of modification times is the single thing that kept me away from notmuch
despite years of wanting to switch.  In the end, I invested months
developing a highly-optimized change detector that efficiently diffs
Xapian's Btrees against a mysql database with a snapshot of the same
information.  My solution works, and I now enjoy a replicated notmuch
setup synchronized across three machines, including offline access on my
laptop.  But my 4,000-line C++ program might have been a 400-line shell
script if only notmuch supported docid mod times.

Also, to put this in perspective, how long does it take to remove the
changed tags from a bunch of messages?  If it's longer than 300 msec on
a 64-bit machine, then even with a single application you'd be better
off using my crazy on-the-side mysql version vector scheme.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch