Re: [PATCH] emacs: process crypto for reply only when specified
On Sun, Apr 13 2014, David Bremner wrote: > Jameson Graef Rollins writes: > >> This is a tweak to patch "emacs: sign/encrypt replies to >> signed/encrypted messages" to only add mml crypto flags for replys >> when crypto processing has been activated. >> > > The (merged) patch seems straightforward and seems to work. I'll > probably push it tomorrow if nobody complains. Code looks OK. +1. I'm interested to see whether replies to signed emails work on this system where sign/(de|en)cryption just doesn't work (out of the box) :D > d Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Synchronization success stories?
I have experimented with offlineimap, syncmaildir and rsync. The append-only approach of notmuch makes synchronization of the mail corpus simpler, so there are lots of options. With ssh access to the server, I found rsync to be conceptually the simplest, but it turned out to be too slow for me (with ~110k mails) when frequently checking for new mails. What I have settled with is a hacked together ruby script that uses the notmuch command line both on the server and on the client to determine unsynced mails, and then runs rsync explicitely for the necessary files. The notmuch index on the server is only used to find new files for this synchronization process, and is different from the notmuch indexes I have on my client machines. A prerequisite for this is of course ssh access and the ability to set up notmuch on the server. The steps performed on a sync run are roughly like this: - local: notmuch new - local: notmuch search --output=messages .. - remote: notmuch new - remote: notmuch search --output=messages .. - compare search results - run rsync for mails that only exist locally (using notmuch search --output=files to get the filenames) - run rsync for mails that only exist remotely (using notmuch search --output=files to get the filenames) With a reused ssh connection this is sufficiently fast for me (<2s). If there is interest I can clean up the script of hardcoded paths etc. and put it on github. Synchronization of the notmuch tags database is only necessary when I switch between different client computers, which happens less frequently. Like David I have a dump file committed to git for that. I found it useful to sort the output before adding it to git, to avoid huge unreadable diffs: notmuch dump --format=batch-tag | sort > /path/to/notmuch.dump Til pgpkJPsZdB3wk.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] test: use test_expect_equal for PATH test, update message
- The old test was quite impossible to debug; the new one shows the difference
between the two directories, if any.
- "repository" doesn't make sense for out of tree builds. Or tarball
builds, for that matter.
---
I was tracking down a weird issue with out of tree builds. I didn't
find the issue (it may have just been cruft from a previously broken
build). In any case, I think the patch is a small improvement.
test/T000-basic.sh | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index 9c94b62..ebbb6d2 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -86,8 +86,9 @@ test_expect_success \
'NOTMUCH_CONFIG is set and points to an existing file' \
'test -f "${NOTMUCH_CONFIG}"'
-test_expect_success \
-'PATH is set to this repository' \
-'test "`echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,'`" =
"`dirname ${TEST_DIRECTORY}`"'
+test_begin_subtest 'PATH is set to build directory'
+test_expect_equal \
+"$(dirname ${TEST_DIRECTORY})" \
+"$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
test_done
--
1.9.1
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] test: use test_expect_equal for PATH test, update message
On Sun, Apr 13 2014, David Bremner wrote:
> - The old test was quite impossible to debug; the new one shows the difference
> between the two directories, if any.
>
> - "repository" doesn't make sense for out of tree builds. Or tarball
> builds, for that matter.
> ---
Looks better than the old one. +1
Tomi
>
> I was tracking down a weird issue with out of tree builds. I didn't
> find the issue (it may have just been cruft from a previously broken
> build). In any case, I think the patch is a small improvement.
>
> test/T000-basic.sh | 7 ---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/test/T000-basic.sh b/test/T000-basic.sh
> index 9c94b62..ebbb6d2 100755
> --- a/test/T000-basic.sh
> +++ b/test/T000-basic.sh
> @@ -86,8 +86,9 @@ test_expect_success \
> 'NOTMUCH_CONFIG is set and points to an existing file' \
> 'test -f "${NOTMUCH_CONFIG}"'
>
> -test_expect_success \
> -'PATH is set to this repository' \
> -'test "`echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,'`" =
> "`dirname ${TEST_DIRECTORY}`"'
> +test_begin_subtest 'PATH is set to build directory'
> +test_expect_equal \
> +"$(dirname ${TEST_DIRECTORY})" \
> +"$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
>
> test_done
> --
> 1.9.1
>
> ___
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: Synchronization success stories?
Tilmann Singer writes: > > With a reused ssh connection this is sufficiently fast for me (<2s). If > there is interest I can clean up the script of hardcoded paths etc. and > put it on github. Sure, sounds at least as good as what I am using. Also, syncmaildir recently did something pretty annoying for upward compatibility, so in the long term I'm interested in alternatives. Maybe we should start a notmuchmail.org wiki page with the various solutions on it. > Synchronization of the notmuch tags database is only necessary when I > switch between different client computers, which happens less > frequently. Like David I have a dump file committed to git for that. I > found it useful to sort the output before adding it to git, to avoid > huge unreadable diffs: > > notmuch dump --format=batch-tag | sort > /path/to/notmuch.dump same here. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: process crypto for reply only when specified
Tomi Ollila writes: > > Code looks OK. +1. I'm interested to see whether replies to signed emails > work on this system where sign/(de|en)cryption just doesn't work (out of > the box) :D > It's a good point. I need to insert my smartcard to sign things, which I'm sometimes too lazy to do. In my case, maybe I should stop being so lazy; I suspect my particular case is a but unusual. Perhaps people with no ability to sign are less likely to have "notmuch-crypto-process-mime" set? Or we can add another configuration variable initialized from notmuch-crypto-process-mime, but allowing people to shut this off. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Synchronization success stories?
Tilmann Singer writes: > The steps performed on a sync run are roughly like this: > > - local: notmuch new > - local: notmuch search --output=messages .. > - remote: notmuch new > - remote: notmuch search --output=messages .. > - compare search results > - run rsync for mails that only exist locally > (using notmuch search --output=files to get the filenames) > - run rsync for mails that only exist remotely > (using notmuch search --output=files to get the filenames) What happens if you get a message that's been stuck in a queue for a few days and has an old Date: header? Or if you get new messages that have the same Message-ID as old ones? > Synchronization of the notmuch tags database is only necessary when I > switch between different client computers, which happens less > frequently. Do you use a laptop everywhere? I've found that for switching between my desktop machine at home, my laptop on the train, and my desktop at work (which amounts to five switches a day), the notmuch dump time is painfully slow--like well over 10 seconds for 100,000 messages. Hook that into notmuch-poll and you have a recipe for hanging emacs every time you type "G". Of course, I'm also experiencing the problem of "notmuch new" itself being painfully slow, but at least that's now my bottleneck in switching machines. I suspect the source of the notmuch new problem is that I have some huge, huge mailboxes. Some of my maildir/cur directories are multiple megabytes on a BSD FFS file system (no hashing, so linear filename lookups in something that doesn't fit in the dcache). On linux ext4 things are much faster. I intend to reorganize my maildir so that there is a top-level directory with the year and hence no single directory ever contains mail from more than one year. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Synchronization success stories?
David Mazieres writes: > What happens if you get a message that's been stuck in a queue for a few > days and has an old Date: header? It would be missed. I have set the timespan to look backwards for new mail to one month to be a bit safer against the stuck-in-queue cases, but mails with older Date: headers would definitely get missed. The current output of notmuch count "*" is the same on both the client and the server, so it seems I didn't run into this problem yet (maybe I was just lucky). > Or if you get new messages that have > the same Message-ID as old ones? Is that even possible? I thought that notmuch guarantees the uniqueness of indexed message ids. The only reference I could find without trying to read the code was this thread id:87mwyz3s9d@star.eba from 2012, which supports the assumption. >> Synchronization of the notmuch tags database is only necessary when I >> switch between different client computers, which happens less >> frequently. > > Do you use a laptop everywhere? I've found that for switching between > my desktop machine at home, my laptop on the train, and my desktop at > work (which amounts to five switches a day), the notmuch dump time is > painfully slow--like well over 10 seconds for 100,000 messages. Hook > that into notmuch-poll and you have a recipe for hanging emacs every > time you type "G". I have one laptop and one desktop and switch between them almost daily, and run a hibernate script that does notmuch dump + git push, and a resume script that does git pull + notmuch restore. For hibernate / resume the speed of those operations is acceptable, but I wouldn't want to incur that wait for every time checking for new mail. Here is how long they take (on a machine with an SSD, which certainly helps): $ time notmuch dump --format=batch-tag | sort > /tmp/notmuch.dump real0m3.643s user0m3.593s sys 0m0.140s $ time notmuch restore < /tmp/notmuch.dump real0m3.719s user0m3.357s sys 0m0.357s $ notmuch count 117118 Til pgp9eSTUw7Arc.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Synchronization success stories?
Tilmann Singer writes: > David Mazieres writes: >> What happens if you get a message that's been stuck in a queue for a few >> days and has an old Date: header? > > It would be missed. I have set the timespan to look backwards for new > mail to one month to be a bit safer against the stuck-in-queue cases, > but mails with older Date: headers would definitely get missed. > > The current output of notmuch count "*" is the same on both the client > and the server, so it seems I didn't run into this problem yet (maybe I > was just lucky). I've been playing around with reorganizing my maildir, and found a couple of messages (on mailing lists) with clearly invalid dates years in the past. But checking with notmuch count is a good idea. Then you can always fall back to the slow path in the unlikely event that your counts don't match up. Well, except that A) count is just unique message-IDs, not messages, and B) when synchronizing in both directions you could still miss something. You have to assume that the invalid dates are only ever going to occur at one end of a synchronization event. >> Or if you get new messages that have >> the same Message-ID as old ones? > > Is that even possible? I thought that notmuch guarantees the uniqueness > of indexed message ids. The only reference I could find without trying > to read the code was this thread id:87mwyz3s9d@star.eba from 2012, > which supports the assumption. Sadly, yes it is quite possible, and even opens up a slight security issue. Suppose I know you are on a mailing list, and some message appears on that mailing list that I don't want you to see. I can send you an innocuous-looking message that just happens to have the same message-id, and you may never see the original mailing list message. Even better, depending on how your spam filtering is setup, if I include the GTUBE string in my message you may never see mine or the original. That's why with muchsync, I replicate actual mail messages, rather than message-IDs. Then you can always periodically check for message-IDs that appear in more than one file. (In fact, thought I haven't published an interface for this, the SQL database kept my muchsync makes it trivial to check for this and detect certain attacks.) I understand why notmuch went with message IDs. For instance you have sent this reply both directly to me and to a mailing list I am subscribed to. So I will get two slightly different copies of the message (one will have the standard notmuch mailing list signature, the other won't). And this way once I've marked it read, the message will be read even once the second copy comes in. But personally I'd rather see the occasional duplicate message than risk not seeing messages. In particular, if the goal is to see fewer unread messages, some sort of feature that pro-actively skips all future messages in a thread or subthread would be more useful... > Here is how long they take (on a machine with an SSD, which certainly > helps): > > $ time notmuch dump --format=batch-tag | sort > /tmp/notmuch.dump > real0m3.643s > user0m3.593s > sys 0m0.140s > $ time notmuch restore < /tmp/notmuch.dump > real0m3.719s > user0m3.357s > sys 0m0.357s > $ notmuch count > 117118 That's crazy. I'm jealous. Then again, this is how fast muchsync runs (including a full database scan to detect changed messages and tags) when there is no new mail: $ time ./muchsync -v [notmuch] No new mail. synchronizing muchsync database with Xapian... 0.038506 (+0.038506) starting scan of Xapian database... 0.039069 (+0.000563) opened Xapian... 0.040851 (+0.001782) scanned message IDs... 0.137647 (+0.096796) scanned tags... 0.170404 (+0.032757) scanned directories in xapian... 0.172100 (+0.001696) scanned filenames in xapian... 0.172376 (+0.000276) adjusted link counts... 0.199461 (+0.027085) finished synchronizing muchsync database with Xapian... 0.212965 (+0.013505) real0m0.220s user0m0.173s sys 0m0.023s David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: process crypto for reply only when specified
On Sun, Apr 13 2014, David Bremner wrote: > Tomi Ollila writes: > >> >> Code looks OK. +1. I'm interested to see whether replies to signed emails >> work on this system where sign/(de|en)cryption just doesn't work (out of >> the box) :D >> > > It's a good point. I need to insert my smartcard to sign things, which > I'm sometimes too lazy to do. In my case, maybe I should stop being so > lazy; I suspect my particular case is a but unusual. Below is what happened to me when I failed to remove the "<#secure method=pgpmime mode=sign>" part from the beginning of the message It is very easy to detect and there is nothing much one can use if they have configured notmuch-crypto-process-mime to be t (like I seem to have ;) --8<8<8<8<8<8<8<8<8<8<8<8<-- notmuch-crypto-process-mime is a variable defined in `one-notmuch.el'. Its value is t Original value was nil --8<8<8<8<8<8<8<8<8<8<8<8<-- Emacs stopped responding my keypresses after C-c C-c; C-g brought control back to me -- and then I tried again... ps output 19028 ? SLs 0:00 /usr/bin/gpg --no-tty --status-fd 1 --yes --command-f 19029 ? SL 0:00 gpg-agent --server 19030 ? RL 0:50 /usr/bin/pinentry-curses 19034 ? SLs 0:00 /usr/bin/gpg --no-tty --status-fd 1 --yes --command-f 19035 ? SL 0:00 gpg-agent --server 19036 ? RL 0:04 /usr/bin/pinentry-curses 19037 pts/6 R+ 0:00 ps x The 'pinentry-curses' and this emacs doesn't play along well (I've seen this happening before when I tried to encrypt some messages). I had to pkill gpg and pinentry-curses to get rid of the above processes. > Perhaps people with no ability to sign are less likely to have > "notmuch-crypto-process-mime" set? Or we can add another configuration > variable initialized from notmuch-crypto-process-mime, but allowing > people to shut this off. Well, I set notmuch-crypto-process-mime to nil -- it still wants to sign the message and runs gpg... ... my case may be unique enough no-one else has the same problem; anyway ideas how to automatically kill the gpg process(es) when one aborts send attempt? Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: process crypto for reply only when specified
Tomi Ollila writes: > > Well, I set notmuch-crypto-process-mime to nil -- it still wants to > sign the message and runs gpg... > Was it nil when you replied, i.e. when the mml tags were created? d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: process crypto for reply only when specified
On Sun, Apr 13 2014, Tomi Ollila wrote: >> Perhaps people with no ability to sign are less likely to have >> "notmuch-crypto-process-mime" set? Or we can add another configuration >> variable initialized from notmuch-crypto-process-mime, but allowing >> people to shut this off. > > Well, I set notmuch-crypto-process-mime to nil -- it still wants to > sign the message and runs gpg... Was my followup patch applied? My patch controls the insertion of the mml tag depending on whether or not notmuch-crypto-process-mime is t or not. If notmuch-crypto-process-mime is nil the tag won't be added. Presumably you either did not have that patch applied, or had manually set it to t? In any event, if the mml tag is present, it's no longer in notmuch's hands; emacs's mail processing is handling things and calling gpg-agent to sign/encrypt the message. Can you clarify what exactly your situation was? Presumably people who have not set up any crypto processing should not have notmuch-crypto-process-mime set t. jamie. pgp3U5wDLQOHU.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] emacs: process crypto for reply only when specified
On Sun, Apr 13 2014, David Bremner wrote: > Jameson Graef Rollins writes: > >> This is a tweak to patch "emacs: sign/encrypt replies to >> signed/encrypted messages" to only add mml crypto flags for replys >> when crypto processing has been activated. >> > > The (merged) patch seems straightforward and seems to work. I'll > probably push it tomorrow if nobody complains. Code looks OK. +1. I'm interested to see whether replies to signed emails work on this system where sign/(de|en)cryption just doesn't work (out of the box) :D > d Tomi
Synchronization success stories?
I have experimented with offlineimap, syncmaildir and rsync. The append-only approach of notmuch makes synchronization of the mail corpus simpler, so there are lots of options. With ssh access to the server, I found rsync to be conceptually the simplest, but it turned out to be too slow for me (with ~110k mails) when frequently checking for new mails. What I have settled with is a hacked together ruby script that uses the notmuch command line both on the server and on the client to determine unsynced mails, and then runs rsync explicitely for the necessary files. The notmuch index on the server is only used to find new files for this synchronization process, and is different from the notmuch indexes I have on my client machines. A prerequisite for this is of course ssh access and the ability to set up notmuch on the server. The steps performed on a sync run are roughly like this: - local: notmuch new - local: notmuch search --output=messages .. - remote: notmuch new - remote: notmuch search --output=messages .. - compare search results - run rsync for mails that only exist locally (using notmuch search --output=files to get the filenames) - run rsync for mails that only exist remotely (using notmuch search --output=files to get the filenames) With a reused ssh connection this is sufficiently fast for me (<2s). If there is interest I can clean up the script of hardcoded paths etc. and put it on github. Synchronization of the notmuch tags database is only necessary when I switch between different client computers, which happens less frequently. Like David I have a dump file committed to git for that. I found it useful to sort the output before adding it to git, to avoid huge unreadable diffs: notmuch dump --format=batch-tag | sort > /path/to/notmuch.dump Til -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140413/69158c4d/attachment.pgp>
[PATCH] test: use test_expect_equal for PATH test, update message
- The old test was quite impossible to debug; the new one shows the difference
between the two directories, if any.
- "repository" doesn't make sense for out of tree builds. Or tarball
builds, for that matter.
---
I was tracking down a weird issue with out of tree builds. I didn't
find the issue (it may have just been cruft from a previously broken
build). In any case, I think the patch is a small improvement.
test/T000-basic.sh | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index 9c94b62..ebbb6d2 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -86,8 +86,9 @@ test_expect_success \
'NOTMUCH_CONFIG is set and points to an existing file' \
'test -f "${NOTMUCH_CONFIG}"'
-test_expect_success \
-'PATH is set to this repository' \
-'test "`echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,'`" =
"`dirname ${TEST_DIRECTORY}`"'
+test_begin_subtest 'PATH is set to build directory'
+test_expect_equal \
+"$(dirname ${TEST_DIRECTORY})" \
+"$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
test_done
--
1.9.1
[PATCH] test: use test_expect_equal for PATH test, update message
On Sun, Apr 13 2014, David Bremner wrote:
> - The old test was quite impossible to debug; the new one shows the difference
> between the two directories, if any.
>
> - "repository" doesn't make sense for out of tree builds. Or tarball
> builds, for that matter.
> ---
Looks better than the old one. +1
Tomi
>
> I was tracking down a weird issue with out of tree builds. I didn't
> find the issue (it may have just been cruft from a previously broken
> build). In any case, I think the patch is a small improvement.
>
> test/T000-basic.sh | 7 ---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/test/T000-basic.sh b/test/T000-basic.sh
> index 9c94b62..ebbb6d2 100755
> --- a/test/T000-basic.sh
> +++ b/test/T000-basic.sh
> @@ -86,8 +86,9 @@ test_expect_success \
> 'NOTMUCH_CONFIG is set and points to an existing file' \
> 'test -f "${NOTMUCH_CONFIG}"'
>
> -test_expect_success \
> -'PATH is set to this repository' \
> -'test "`echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,'`" =
> "`dirname ${TEST_DIRECTORY}`"'
> +test_begin_subtest 'PATH is set to build directory'
> +test_expect_equal \
> +"$(dirname ${TEST_DIRECTORY})" \
> +"$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
>
> test_done
> --
> 1.9.1
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
Synchronization success stories?
Tilmann Singer writes: > > With a reused ssh connection this is sufficiently fast for me (<2s). If > there is interest I can clean up the script of hardcoded paths etc. and > put it on github. Sure, sounds at least as good as what I am using. Also, syncmaildir recently did something pretty annoying for upward compatibility, so in the long term I'm interested in alternatives. Maybe we should start a notmuchmail.org wiki page with the various solutions on it. > Synchronization of the notmuch tags database is only necessary when I > switch between different client computers, which happens less > frequently. Like David I have a dump file committed to git for that. I > found it useful to sort the output before adding it to git, to avoid > huge unreadable diffs: > > notmuch dump --format=batch-tag | sort > /path/to/notmuch.dump same here.
[PATCH] emacs: process crypto for reply only when specified
Tomi Ollila writes: > > Code looks OK. +1. I'm interested to see whether replies to signed emails > work on this system where sign/(de|en)cryption just doesn't work (out of > the box) :D > It's a good point. I need to insert my smartcard to sign things, which I'm sometimes too lazy to do. In my case, maybe I should stop being so lazy; I suspect my particular case is a but unusual. Perhaps people with no ability to sign are less likely to have "notmuch-crypto-process-mime" set? Or we can add another configuration variable initialized from notmuch-crypto-process-mime, but allowing people to shut this off.
Synchronization success stories?
Tilmann Singer writes: > The steps performed on a sync run are roughly like this: > > - local: notmuch new > - local: notmuch search --output=messages .. > - remote: notmuch new > - remote: notmuch search --output=messages .. > - compare search results > - run rsync for mails that only exist locally > (using notmuch search --output=files to get the filenames) > - run rsync for mails that only exist remotely > (using notmuch search --output=files to get the filenames) What happens if you get a message that's been stuck in a queue for a few days and has an old Date: header? Or if you get new messages that have the same Message-ID as old ones? > Synchronization of the notmuch tags database is only necessary when I > switch between different client computers, which happens less > frequently. Do you use a laptop everywhere? I've found that for switching between my desktop machine at home, my laptop on the train, and my desktop at work (which amounts to five switches a day), the notmuch dump time is painfully slow--like well over 10 seconds for 100,000 messages. Hook that into notmuch-poll and you have a recipe for hanging emacs every time you type "G". Of course, I'm also experiencing the problem of "notmuch new" itself being painfully slow, but at least that's now my bottleneck in switching machines. I suspect the source of the notmuch new problem is that I have some huge, huge mailboxes. Some of my maildir/cur directories are multiple megabytes on a BSD FFS file system (no hashing, so linear filename lookups in something that doesn't fit in the dcache). On linux ext4 things are much faster. I intend to reorganize my maildir so that there is a top-level directory with the year and hence no single directory ever contains mail from more than one year. David
Synchronization success stories?
David Mazieres writes: > What happens if you get a message that's been stuck in a queue for a few > days and has an old Date: header? It would be missed. I have set the timespan to look backwards for new mail to one month to be a bit safer against the stuck-in-queue cases, but mails with older Date: headers would definitely get missed. The current output of notmuch count "*" is the same on both the client and the server, so it seems I didn't run into this problem yet (maybe I was just lucky). > Or if you get new messages that have > the same Message-ID as old ones? Is that even possible? I thought that notmuch guarantees the uniqueness of indexed message ids. The only reference I could find without trying to read the code was this thread id:87mwyz3s9d.fsf at star.eba from 2012, which supports the assumption. >> Synchronization of the notmuch tags database is only necessary when I >> switch between different client computers, which happens less >> frequently. > > Do you use a laptop everywhere? I've found that for switching between > my desktop machine at home, my laptop on the train, and my desktop at > work (which amounts to five switches a day), the notmuch dump time is > painfully slow--like well over 10 seconds for 100,000 messages. Hook > that into notmuch-poll and you have a recipe for hanging emacs every > time you type "G". I have one laptop and one desktop and switch between them almost daily, and run a hibernate script that does notmuch dump + git push, and a resume script that does git pull + notmuch restore. For hibernate / resume the speed of those operations is acceptable, but I wouldn't want to incur that wait for every time checking for new mail. Here is how long they take (on a machine with an SSD, which certainly helps): $ time notmuch dump --format=batch-tag | sort > /tmp/notmuch.dump real0m3.643s user0m3.593s sys 0m0.140s $ time notmuch restore < /tmp/notmuch.dump real0m3.719s user0m3.357s sys 0m0.357s $ notmuch count 117118 Til -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140413/3ae1dfe0/attachment.pgp>
Synchronization success stories?
Tilmann Singer writes: > David Mazieres writes: >> What happens if you get a message that's been stuck in a queue for a few >> days and has an old Date: header? > > It would be missed. I have set the timespan to look backwards for new > mail to one month to be a bit safer against the stuck-in-queue cases, > but mails with older Date: headers would definitely get missed. > > The current output of notmuch count "*" is the same on both the client > and the server, so it seems I didn't run into this problem yet (maybe I > was just lucky). I've been playing around with reorganizing my maildir, and found a couple of messages (on mailing lists) with clearly invalid dates years in the past. But checking with notmuch count is a good idea. Then you can always fall back to the slow path in the unlikely event that your counts don't match up. Well, except that A) count is just unique message-IDs, not messages, and B) when synchronizing in both directions you could still miss something. You have to assume that the invalid dates are only ever going to occur at one end of a synchronization event. >> Or if you get new messages that have >> the same Message-ID as old ones? > > Is that even possible? I thought that notmuch guarantees the uniqueness > of indexed message ids. The only reference I could find without trying > to read the code was this thread id:87mwyz3s9d.fsf at star.eba from 2012, > which supports the assumption. Sadly, yes it is quite possible, and even opens up a slight security issue. Suppose I know you are on a mailing list, and some message appears on that mailing list that I don't want you to see. I can send you an innocuous-looking message that just happens to have the same message-id, and you may never see the original mailing list message. Even better, depending on how your spam filtering is setup, if I include the GTUBE string in my message you may never see mine or the original. That's why with muchsync, I replicate actual mail messages, rather than message-IDs. Then you can always periodically check for message-IDs that appear in more than one file. (In fact, thought I haven't published an interface for this, the SQL database kept my muchsync makes it trivial to check for this and detect certain attacks.) I understand why notmuch went with message IDs. For instance you have sent this reply both directly to me and to a mailing list I am subscribed to. So I will get two slightly different copies of the message (one will have the standard notmuch mailing list signature, the other won't). And this way once I've marked it read, the message will be read even once the second copy comes in. But personally I'd rather see the occasional duplicate message than risk not seeing messages. In particular, if the goal is to see fewer unread messages, some sort of feature that pro-actively skips all future messages in a thread or subthread would be more useful... > Here is how long they take (on a machine with an SSD, which certainly > helps): > > $ time notmuch dump --format=batch-tag | sort > /tmp/notmuch.dump > real0m3.643s > user0m3.593s > sys 0m0.140s > $ time notmuch restore < /tmp/notmuch.dump > real0m3.719s > user0m3.357s > sys 0m0.357s > $ notmuch count > 117118 That's crazy. I'm jealous. Then again, this is how fast muchsync runs (including a full database scan to detect changed messages and tags) when there is no new mail: $ time ./muchsync -v [notmuch] No new mail. synchronizing muchsync database with Xapian... 0.038506 (+0.038506) starting scan of Xapian database... 0.039069 (+0.000563) opened Xapian... 0.040851 (+0.001782) scanned message IDs... 0.137647 (+0.096796) scanned tags... 0.170404 (+0.032757) scanned directories in xapian... 0.172100 (+0.001696) scanned filenames in xapian... 0.172376 (+0.000276) adjusted link counts... 0.199461 (+0.027085) finished synchronizing muchsync database with Xapian... 0.212965 (+0.013505) real0m0.220s user0m0.173s sys 0m0.023s David
[PATCH] emacs: process crypto for reply only when specified
On Sun, Apr 13 2014, David Bremner wrote: > Tomi Ollila writes: > >> >> Code looks OK. +1. I'm interested to see whether replies to signed emails >> work on this system where sign/(de|en)cryption just doesn't work (out of >> the box) :D >> > > It's a good point. I need to insert my smartcard to sign things, which > I'm sometimes too lazy to do. In my case, maybe I should stop being so > lazy; I suspect my particular case is a but unusual. Below is what happened to me when I failed to remove the "<#secure method=pgpmime mode=sign>" part from the beginning of the message It is very easy to detect and there is nothing much one can use if they have configured notmuch-crypto-process-mime to be t (like I seem to have ;) --8<8<8<8<8<8<8<8<8<8<8<8<-- notmuch-crypto-process-mime is a variable defined in `one-notmuch.el'. Its value is t Original value was nil --8<8<8<8<8<8<8<8<8<8<8<8<-- Emacs stopped responding my keypresses after C-c C-c; C-g brought control back to me -- and then I tried again... ps output 19028 ? SLs 0:00 /usr/bin/gpg --no-tty --status-fd 1 --yes --command-f 19029 ? SL 0:00 gpg-agent --server 19030 ? RL 0:50 /usr/bin/pinentry-curses 19034 ? SLs 0:00 /usr/bin/gpg --no-tty --status-fd 1 --yes --command-f 19035 ? SL 0:00 gpg-agent --server 19036 ? RL 0:04 /usr/bin/pinentry-curses 19037 pts/6 R+ 0:00 ps x The 'pinentry-curses' and this emacs doesn't play along well (I've seen this happening before when I tried to encrypt some messages). I had to pkill gpg and pinentry-curses to get rid of the above processes. > Perhaps people with no ability to sign are less likely to have > "notmuch-crypto-process-mime" set? Or we can add another configuration > variable initialized from notmuch-crypto-process-mime, but allowing > people to shut this off. Well, I set notmuch-crypto-process-mime to nil -- it still wants to sign the message and runs gpg... ... my case may be unique enough no-one else has the same problem; anyway ideas how to automatically kill the gpg process(es) when one aborts send attempt? Tomi
[PATCH] emacs: process crypto for reply only when specified
Tomi Ollila writes: > > Well, I set notmuch-crypto-process-mime to nil -- it still wants to > sign the message and runs gpg... > Was it nil when you replied, i.e. when the mml tags were created? d
[PATCH] emacs: process crypto for reply only when specified
On Sun, Apr 13 2014, Tomi Ollila wrote: >> Perhaps people with no ability to sign are less likely to have >> "notmuch-crypto-process-mime" set? Or we can add another configuration >> variable initialized from notmuch-crypto-process-mime, but allowing >> people to shut this off. > > Well, I set notmuch-crypto-process-mime to nil -- it still wants to > sign the message and runs gpg... Was my followup patch applied? My patch controls the insertion of the mml tag depending on whether or not notmuch-crypto-process-mime is t or not. If notmuch-crypto-process-mime is nil the tag won't be added. Presumably you either did not have that patch applied, or had manually set it to t? In any event, if the mml tag is present, it's no longer in notmuch's hands; emacs's mail processing is handling things and calling gpg-agent to sign/encrypt the message. Can you clarify what exactly your situation was? Presumably people who have not set up any crypto processing should not have notmuch-crypto-process-mime set t. jamie. -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140413/d5f75bda/attachment.pgp>
Synchronization success stories?
David Mazieres writes: > David Bremner writes: > >> Brian Sniffen writes: >> >>> I'm thrilled by using notmuch to manage my mail. Low-latency search is >>> very important to me. But I use computers in a couple of >>> places---several of which are laptops. Has anyone stories to share of >>> successful multi-computer notmuch sync, for a corpus of a >>> quarter-million messages or so? >> >> I use syncmaildir to sync the actual messages, and a copy of the output >> of "notmuch dump" in git to sync the metadata. >> >> It works OK. A bit slow; depends how often you need to fetch new mail. > > If you want to see my solution, it is here: > > http://www.scs.stanford.edu/~dm/muchsync-0.tar.gz Thanks! Much sync. "Wow." It sounds like you're paying very careful attention to correctness and performance, so I'm very glad to be able to start from that basis. -Brian > I'm a little embarrassed by this code, as I just started to test it a > week ago then instantly became completely dependent on it. I will > probably change the name (from muchsync to syncmuch) and the database > format before releasing. But if you feel like beta-testing and giving > me feedback, have a look. > > Beware that if you have been using notmuch dump, you may become > instantly hooked on my solution... > > David -- Brian Sniffen Information Security Akamai Technologies
