notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Thomas Schwinge
Hallo!

Stepping away from the current code base -- what is notmuch's original
idea of concurrency?  That is, all of us probably know that one:

A Xapian exception occurred opening database: Unable to get write
  lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
  already locked

I recently saw that one while using the Emacs UI (that one tried to
remove a unread tag or similar), and in parallel a delivery to the
notmuch DB was going on.

Apparently the DB we're using doesn't allow for simultaneous writing
(even though it can't even possibly have been dangerous in this case).

Which is the original idea here?  Is it that...

  * each and every client should catch these kinds of errors, and retry,
or eventually give up at some point, and report the status to the
user; or is it that...

  * notmuch internally should catch these concurrency cases, and retry,
or eventually give up at some point (``notmuch --maximum-wait=30s tag
[...]''), and fail as seen above?

This one is an obvious temporary error due to a concurrency situation.
Wouldn't the latter suggestion be preferable here?  I guess that in most
cases the DB isn't locked for long periods of time, and thus the
concurrency situation would decline quickly.


One difficulty I see is judging which errors are temporary and which are
permanent -- which is obvious in a lot of cases (concurrent DB access,
memory starved or any other OS resource), but may not be, for example in
case of I/O errors (is ``disk full'' a permanent error?).  And then, for
some of these cases, waiting does make sense (concurrent DB access, as
suggested above), and for other (temporary?) errors it doesn't make (a
lot of) sense (out of memory: only sensible thing is to abort, and have
the caller re-try, or disk full: waiting for some free space may be worth
it, or it may be not).


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/17349725/attachment.pgp>


[PATCH 1/3] new: Do not defer maildir flag synchronization during the first run

2011-01-27 Thread Austin Clements
Looks like eagerly synchronizing tags is easy and works fine, though I need
to beef up the tests and put some transactions around things before I'm
satisfied.

I added a "notmuch_database_remove_message_get" to the public API that's
just like "notmuch_database_remove_message" except that it also returns a
notmuch_message_t if other instances of the message still exist.  It feels
clunky to have two almost identical variants of this function.  Is this the
preferred way to change the public API?  Or should I just add the argument
to the existing function and fix the other three calls to it?
On Thu, Jan 27, 2011 at 12:43 AM, Austin Clements  wrote:

> Sure. I've been wanting to take a crack at notmuch new's atomicity for a
> while. Though you'll have to get through some of my outstanding patches. I
> can only keep so many branches in my head. ]:--8)
>
-- next part --
An HTML attachment was scrubbed...
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/9e7c3dfc/attachment.html>


notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Austin Clements
I'm looking into breaking notmuch new up into small transactions.  It
wouldn't be much a leap from there to simply close and reopen the database
between transactions if another task wants to use it, which would release
the lock and let the queued notmuch task have the database for a bit.  It
seems silly to have a daemon when all of notmuch's state is already on disk
and queue on a lock is as good as a queue in a daemon, but without the
accompanying architectural shenanigans.

On Thu, Jan 27, 2011 at 3:35 PM, Jameson Rollins  wrote:

> On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson 
> wrote:
> > Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> > database growing, and perhaps some fragmentation somewhere, this has
> > become *incredibly* annoying for me. I am checking email every 30
> > minutes, and I'm nicing and ionicing the processes so I can use my
> > machine, but while those processes are running, I'm effectively locked
> > out of a good portion of my email.
>
> I also have a very slow disk, but this is very rarely a problem for me.
> I retrieve mail every 10 minutes, and the corresponding notmuch new
> usually takes a minute or so.  I really haven't found it to be much of a
> bother to just wait it out.
>
> One of the suggested ways to develop around this problem would be a
> notmuch daemon that would queue database modification requests.  I don't
> think anyone has been working on this yet, but if this is a big problem
> for you guys, you might start looking into putting one together.
>
> jamie.
-- next part --
An HTML attachment was scrubbed...
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/a987c67d/attachment.html>


[PATCH 2/3] new: Add all initial tags at once

2011-01-27 Thread Carl Worth
On Thu, 27 Jan 2011 15:03:49 +1000, Carl Worth  wrote:
> Just a few hours ago I attended an interesting talk by Rusty Russell in
> which he talks about a CCAN module he has written called failtest which
> provides an implementation of this kind of testing.

I meant to include some links with the above. CCAN is here:

http://ccan.ozlabs.org/

And the failtest module is here:

http://ccan.ozlabs.org/info/failtest.html

I talked to Rusty about adding copyright attribution and a one-line
pointer to the LICENSE file to this. Once that's in place, we could
incorporate failtest.c into notmuch if it would be useful.

-Carl

-- 
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/21677515/attachment.pgp>


[PATCH] emacs: send notmuch-query stderr to /dev/null

2011-01-27 Thread Jameson Rollins
On Thu, 27 Jan 2011 14:49:43 +1000, Carl Worth  wrote:
> On Wed, 26 Jan 2011 12:57:50 -0800, Jameson Rollins  finestructure.net> wrote:
> > The call-process to notmuch in notmuch-query.el was previously sending
> > stderr into the output buffer.  This means that if there is any stderr
> > the JSON parsing breaks.  Unfortunately call-process does not support
> > sending stderr to a separate buffer or to the minibuffer [0], but it
> > does support sending it to /dev/null.  So we do that here instead.
> > 
> > [0] a bug was filed against emacs (#7842)
> 
> Thanks! I had wondered what those json errors were about. I've committed
> this.
> 
> I am a bit concerned about throwing the error output away, of course,
> (so we'll wait for that fix to emacs---thanks for submitting a bug
> report). Do you have a sense of what kinds of output we are getting on
> stderr?

So the only stderr output I've ever seen seen is in the
signature-verification branch (which I now run).  Emails with S/MIME
signatures are currently not handled (because of GMIME limitations, I
think) and notmuch throws the following error when trying to process
them:

servo:~ 0$ notmuch show --format=json --verify id:"4D25D062.1000103 at 
sixdemonbag.org" >/dev/null
Failed to verify signed part: no error explanation given
servo:~ 0$ 

Obviously the emacs call-process function should be capturing and
displaying those error messages.  Hopefully we can eventually get that
fixed.  In the mean time it's probably better to throw out the messages
rather than have them break the JSON output.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/eb9e93ae/attachment.pgp>


[PATCH 1/3] new: Do not defer maildir flag synchronization during the first run

2011-01-27 Thread Carl Worth
On Wed, 26 Jan 2011 10:07:28 -0500, Austin Clements  wrote:
> Quoth Carl Worth on Jan 26 at  9:59 pm:
> 
> I believe you're right that synchronization could always be done
> eagerly.  In fact, I believe this is true even with the addition of
> conjunctive and disjunctive flags.

Excellent! Want to take a whack at implementing that? And seeing if you
can find any test cases that stress this more than existing test cases
might?

-Carl

-- 
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/59847bc2/attachment.pgp>


[PATCH 2/3] new: Add all initial tags at once

2011-01-27 Thread Carl Worth
On Wed, 26 Jan 2011 17:52:53 +0100, Thomas Schwinge  
wrote:
> I do support the patch's idea (which was recently committed; and what
> follows in this message is not at all directed towards Michal, who wrote
> this patch) -- but what about return values checking?  This is one aspect
> of the notmuch C code (which I generally consider to be nice to read and
> of high quality, as I said before already), that I consider totally
> lacking -- there are literally hundreds of C functions calls where the
> return values are just discarded.  This is bad.  For example (simulating
> a full disk):
> 
> $ notmuch dump > /dev/full
> $ echo $?
> 0

All very well pointed out. This is clearly something we need to fix.

> Other languages have the concept of exceptions; C doesn't, so we're
> supposed to put some ``ABORT_IF_NOT_NOTMUCH_STATUS_SUCCESS(ret)''
> statements after each and every non-void (etc.) C function call.  Or make
> the functions abort themselves (which is not a too good idea, as we
> surely agree).  Or use a different programming language -- now, at the
> present state, it wouldn't be too painful to switch, in my opinion.  (I
> won't suggest any specific language, though.)

I wouldn't have any problem with anyone re-implementing notmuch in some
other language than C. But that's not something I would be likely to
work on myself, I don't think. 

>If staying with C (which I
> don't object, either), then this needs a whole code audit, and a lot of
> discipline in the future.

Even a code audit and developer discipline won't be enough here. We'll
still miss things.

What we need is exhaustive testing. A great approach is to take calls
like malloc, open, read, write, etc. and at each site, fork() and fail
the call along one path, (which should then exit with a failure), and
then let the other path continue.

Just a few hours ago I attended an interesting talk by Rusty Russell in
which he talks about a CCAN module he has written called failtest which
provides an implementation of this kind of testing.

I'd love to see something like that integrated with notmuch.

> Comments?  (And I hope this doesn't sound too harsh :-) -- but it is a
> serious programming issue.)

Please don't apologize! It would be a shame if people didn't share
problems they notice in the code. Being able to hear those kinds of
things is one of the great benefits I get from publishing this code as
free software.

So, please, keep the suggestions coming!

-Carl

-- 
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/bb600e96/attachment.pgp>


[PATCH] emacs: send notmuch-query stderr to /dev/null

2011-01-27 Thread Carl Worth
On Wed, 26 Jan 2011 12:57:50 -0800, Jameson Rollins  wrote:
> The call-process to notmuch in notmuch-query.el was previously sending
> stderr into the output buffer.  This means that if there is any stderr
> the JSON parsing breaks.  Unfortunately call-process does not support
> sending stderr to a separate buffer or to the minibuffer [0], but it
> does support sending it to /dev/null.  So we do that here instead.
> 
> [0] a bug was filed against emacs (#7842)

Thanks! I had wondered what those json errors were about. I've committed
this.

I am a bit concerned about throwing the error output away, of course,
(so we'll wait for that fix to emacs---thanks for submitting a bug
report). Do you have a sense of what kinds of output we are getting on
stderr?

-Carl

-- 
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/7cc33d89/attachment.pgp>


notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread micah anderson
On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge  
wrote:
> Stepping away from the current code base -- what is notmuch's original
> idea of concurrency?  That is, all of us probably know that one:
> 
> A Xapian exception occurred opening database: Unable to get write
>   lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
>   already locked
> 
> I recently saw that one while using the Emacs UI (that one tried to
> remove a unread tag or similar), and in parallel a delivery to the
> notmuch DB was going on.

Due to my harddisk in my laptop being slow (5400RPM), my notmuch
database growing, and perhaps some fragmentation somewhere, this has
become *incredibly* annoying for me. I am checking email every 30
minutes, and I'm nicing and ionicing the processes so I can use my
machine, but while those processes are running, I'm effectively locked
out of a good portion of my email. 

Usually, I switch to another task until my disk light has ceased being
solid, because the update time is too slow for me to wait. 

Now that folders are making it in, the two remaining features that are
driving me nuts with notmuch is this one and the
verification/decryption/encryption process (replying to an encrypted
message is 12 distinct steps for me, which is discouraging me from doing
that at all). 

I really don't want to complain, because I have no time to help in these
areas,  rather I'm interested  to know  if anyone  has any  pointers for
making this less annoying, and I'm  hoping that at some point I can free
up time to help. Perhaps I need to dump/restore my notmuch DB? Or index
less mail?

micah
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/f8728db1/attachment-0001.pgp>


notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Jameson Rollins
On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson  wrote:
> Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> database growing, and perhaps some fragmentation somewhere, this has
> become *incredibly* annoying for me. I am checking email every 30
> minutes, and I'm nicing and ionicing the processes so I can use my
> machine, but while those processes are running, I'm effectively locked
> out of a good portion of my email. 

I also have a very slow disk, but this is very rarely a problem for me.
I retrieve mail every 10 minutes, and the corresponding notmuch new
usually takes a minute or so.  I really haven't found it to be much of a
bother to just wait it out.

One of the suggested ways to develop around this problem would be a
notmuch daemon that would queue database modification requests.  I don't
think anyone has been working on this yet, but if this is a big problem
for you guys, you might start looking into putting one together.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20110127/10747b55/attachment.pgp>


[PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.

2011-01-27 Thread Michal Sojka
On Thu, 27 Jan 2011, Thomas Schwinge wrote:
> Signed-off-by: Thomas Schwinge 
> ---
>  test/test-lib.sh |5 -
>  1 files changed, 4 insertions(+), 1 deletions(-)
> 
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index d179426..f536172 100755
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -351,8 +351,11 @@ ${additional_headers}"
>  ${additional_headers}"
>  fi
>  
> +# Note that in the way we're setting it above and using it below,
> +# `additional_headers' will also serve as the header / body separator
> +# (empty line in between).

Yes, this will definitely help the future readers of this code. Thanks.

-Michal


[PATCH] emacs: send notmuch-query stderr to /dev/null

2011-01-27 Thread Michal Sojka
On Thu, 27 Jan 2011, Carl Worth wrote:
> On Wed, 26 Jan 2011 12:57:50 -0800, Jameson Rollins  finestructure.net> wrote:
> > The call-process to notmuch in notmuch-query.el was previously sending
> > stderr into the output buffer.  This means that if there is any stderr
> > the JSON parsing breaks.  Unfortunately call-process does not support
> > sending stderr to a separate buffer or to the minibuffer [0], but it
> > does support sending it to /dev/null.  So we do that here instead.
> > 
> > [0] a bug was filed against emacs (#7842)
> 
> Thanks! I had wondered what those json errors were about. I've committed
> this.
> 
> I am a bit concerned about throwing the error output away, of course,
> (so we'll wait for that fix to emacs---thanks for submitting a bug
> report). Do you have a sense of what kinds of output we are getting on
> stderr?

I do not know which errors Jameson experienced, but sometimes I have to
use LD_PRELOADed 32bit libraries with some software and when I
accidentally run my emacs (which is 64bit) with LD_PRELOAD set this way,
every execution of notmuch prints

ERROR: ld.so: object '/opt/xilinx/usb-driver/libusb-driver.so' from LD_PRELOAD 
cannot be preloaded: ignored.

and this breaks the parsing. So thanks Jameson for solving this issue
for me.

-Michal


[PATCH 2/3] new: Add all initial tags at once

2011-01-27 Thread Michal Sojka
On Thu, 27 Jan 2011, Carl Worth wrote:
> On Thu, 27 Jan 2011 15:03:49 +1000, Carl Worth  wrote:
> > Just a few hours ago I attended an interesting talk by Rusty Russell in
> > which he talks about a CCAN module he has written called failtest which
> > provides an implementation of this kind of testing.
> 
> I meant to include some links with the above. CCAN is here:
> 
>   http://ccan.ozlabs.org/
> 
> And the failtest module is here:
> 
>   http://ccan.ozlabs.org/info/failtest.html

Yes, this looks interesting. We may also want to use GCC function
attribute "__attribute__ ((warn_unused_result))" to get warnings when
the returned value is not checked.

-Michal


Remote usage script updated

2011-01-27 Thread Jesse Rosenthal
Dear all,

Just a note to say that I finally got around to updating the remote
usage script on the wiki to what I'm using now. With "--format=raw" in,
it's all pretty straightforward. The only things the script does now
are:

1. Produces a slight pause in the "notmuch show" output to avoid that
   weird bug where emacs leaves off every tenth message or so.

2. Locally caches raw messages (i.e. when --format=raw) is called. This
   usually happens when getting attachments, so this is a nice way to
   avoid having to download large attachments repeatedly.

   Note this just caches based on msg-id (or a hash thereof, to avoid
   strange characters in file names). That means that if an attachment
   is deleted on the server, the cache will be out of date. An easy way
   to fix this would be to make the cache file name a concatenation of
   the msg-id hash (check that first) and the hash of the actual message
   (check that if the msg-id hash is there). I might put this in in the
   future, especially if anyone else is using the script.

3. Escapes dollar signs in the msg-id to make shell-quoting over ssh
   work.

I've actually switched over to keeping my messages on my IMAP server and
using this remote script on all of my computers. It avoids any need for
syncing. It's been working very well for me so far.

A future feature might be to integrate the ControlMaster feature of
openssh into the script, instead of having to open a connection
manually, but there are some complications there (dead sockets still
around if you go offline, etc.).

Best,
Jesse



[PATCH] Add a few tests for searching LWN emails.

2011-01-27 Thread Thomas Schwinge
These tests should pass -- but they currently don't.

Signed-off-by: Thomas Schwinge 

---

Hallo!

I reported this on IRC some weeks ago; here is a more elaborate report.

What we get from these emails, is an author named ``LWN.net'', and the
``Weekly Notification'' / ``Mailing Lists'' bits are stripped away.  I
suspect this may be a misinterpretation in the notmuch address parser,
related to the dot in the name.  I have not yet looked at the relevant
code.

The same problem might exist for the To: parser, which I have not yet
checked.


Gre,
 Thomas

 test/search |   36 
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/test/search b/test/search
index b180c7f..2ae5640 100755
--- a/test/search
+++ b/test/search
@@ -3,6 +3,12 @@ test_description='"notmuch search" in several variations'
 . ./test-lib.sh

 add_email_corpus
+add_message \
+  '[from]="LWN.net Weekly Notification "' \
+  '[subject]="LWN.net Weekly Edition for January 27, 2011 available"'
+add_message \
+  '[from]="LWN.net Mailing Lists "' \
+  '[subject]="LWN.net newly freed content for January 27, 2011"'

 test_begin_subtest "Search body"
 add_message '[subject]="body search"' '[date]="Sat, 01 Jan 2000 12:00:00 
-"' [body]=bodysearchtest
@@ -57,6 +63,34 @@ add_message '[subject]="search by from (name)"' 
'[date]="Sat, 01 Jan 2000 12:00:
 output=$(notmuch search from:"Search By From Name" | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Search By From 
Name; search by from (name) (inbox unread)"

+test_begin_subtest "LWN, I:"
+output=$(notmuch search from:'lwn.net weekly notification' | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Weekly 
Notification; LWN.net Weekly Edition for January 27, 2011 available (inbox 
unread)"
+
+test_begin_subtest "LWN, II:"
+output=$(notmuch search from:'lwn.net mailing lists' | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Mailing 
Lists; LWN.net newly freed content for January 27, 2011 (inbox unread)"
+
+test_begin_subtest "LWN, III:"
+output=$(notmuch search from:lwn and from:weekly | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Weekly 
Notification; LWN.net Weekly Edition for January 27, 2011 available (inbox 
unread)"
+
+test_begin_subtest "LWN, IV:"
+output=$(notmuch search from:lwn and from:mailing | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Mailing 
Lists; LWN.net newly freed content for January 27, 2011 (inbox unread)"
+
+test_begin_subtest "LWN, V:"
+output=$(notmuch search from:lwn at lwn.net and subject:weekly | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Weekly 
Notification; LWN.net Weekly Edition for January 27, 2011 available (inbox 
unread)"
+
+test_begin_subtest "LWN, VI:"
+output=$(notmuch search from:lwn at lwn.net and subject:mailing | 
notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] LWN.net Mailing 
Lists; LWN.net newly freed content for January 27, 2011 (inbox unread)"
+
+test_begin_subtest "LWN, VII:"
+output=$(notmuch count from:lwn at lwn.net)
+test_expect_equal "$output" 2
+
 test_begin_subtest "Search by to: (address)"
 add_message '[subject]="search by to (address)"' '[date]="Sat, 01 Jan 2000 
12:00:00 -"' [to]=searchbyto at example.com
 output=$(notmuch search to:searchbyto at example.com | notmuch_search_sanitize)
@@ -97,6 +131,8 @@ thread:XXX   2009-11-18 [1/1] Stewart Smith; [notmuch] 
[PATCH] Fix linking with
 thread:XXX   2009-11-18 [2/2] Lars Kellogg-Stedman; [notmuch] \"notmuch help\" 
outputs to stderr? (attachment inbox unread)
 thread:XXX   2009-11-17 [1/1] Mikhail Gusarov; [notmuch] [PATCH] Handle rename 
of message file (inbox unread)
 thread:XXX   2009-11-17 [2/2] Alex Botero-Lowry, Carl Worth; [notmuch] 
preliminary FreeBSD support (attachment inbox unread)
+thread:XXX   2001-01-05 [1/1] LWN.net Weekly Notification; LWN.net Weekly 
Edition for January 27, 2011 available (inbox unread)
+thread:XXX   2001-01-05 [1/1] LWN.net Mailing Lists; LWN.net newly freed 
content for January 27, 2011 (inbox unread)
 thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; body search (inbox unread)
 thread:XXX   2000-01-01 [1/1] searchbyfrom; search by from (inbox unread)
 thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; search by to (inbox unread)
-- 
tg: (74cb76a..) t/from-lwn (depends on: master)


[PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.

2011-01-27 Thread Thomas Schwinge
Signed-off-by: Thomas Schwinge 
---
 test/test-lib.sh |5 -
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index d179426..f536172 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -351,8 +351,11 @@ ${additional_headers}"
 ${additional_headers}"
 fi

+# Note that in the way we're setting it above and using it below,
+# `additional_headers' will also serve as the header / body separator
+# (empty line in between).

-cat <"$gen_msg_filename"
+cat <"$gen_msg_filename"
 From: ${template[from]}
 To: ${template[to]}
 Message-Id: <${gen_msg_id}>
-- 
1.7.1



[PATCH 1/3] new: Do not defer maildir flag synchronization during the first run

2011-01-27 Thread Austin Clements
Sure. I've been wanting to take a crack at notmuch new's atomicity for a while. 
Though you'll have to get through some of my outstanding patches. I can only 
keep so many branches in my head. ]:--8)

rlb, you expressed an interest in solving this problem, too. Did you make any 
headway?

"Carl Worth"  wrote:

>On Wed, 26 Jan 2011 10:07:28 -0500, Austin Clements 
>wrote:
>> Quoth Carl Worth on Jan 26 at  9:59 pm:
>> 
>> I believe you're right that synchronization could always be done
>> eagerly.  In fact, I believe this is true even with the addition of
>> conjunctive and disjunctive flags.
>
>Excellent! Want to take a whack at implementing that? And seeing if you
>can find any test cases that stress this more than existing test cases
>might?
>
>-Carl
>
>-- 
>carl.d.worth at intel.com

-- 
Sent from my Android. Please excuse my brevity.


Re: [PATCH 2/3] new: Add all initial tags at once

2011-01-27 Thread Michal Sojka
On Thu, 27 Jan 2011, Carl Worth wrote:
 On Thu, 27 Jan 2011 15:03:49 +1000, Carl Worth cwo...@cworth.org wrote:
  Just a few hours ago I attended an interesting talk by Rusty Russell in
  which he talks about a CCAN module he has written called failtest which
  provides an implementation of this kind of testing.
 
 I meant to include some links with the above. CCAN is here:
 
   http://ccan.ozlabs.org/
 
 And the failtest module is here:
 
   http://ccan.ozlabs.org/info/failtest.html

Yes, this looks interesting. We may also want to use GCC function
attribute __attribute__ ((warn_unused_result)) to get warnings when
the returned value is not checked.

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] emacs: send notmuch-query stderr to /dev/null

2011-01-27 Thread Michal Sojka
On Thu, 27 Jan 2011, Carl Worth wrote:
 On Wed, 26 Jan 2011 12:57:50 -0800, Jameson Rollins 
 jroll...@finestructure.net wrote:
  The call-process to notmuch in notmuch-query.el was previously sending
  stderr into the output buffer.  This means that if there is any stderr
  the JSON parsing breaks.  Unfortunately call-process does not support
  sending stderr to a separate buffer or to the minibuffer [0], but it
  does support sending it to /dev/null.  So we do that here instead.
  
  [0] a bug was filed against emacs (#7842)
 
 Thanks! I had wondered what those json errors were about. I've committed
 this.
 
 I am a bit concerned about throwing the error output away, of course,
 (so we'll wait for that fix to emacs---thanks for submitting a bug
 report). Do you have a sense of what kinds of output we are getting on
 stderr?

I do not know which errors Jameson experienced, but sometimes I have to
use LD_PRELOADed 32bit libraries with some software and when I
accidentally run my emacs (which is 64bit) with LD_PRELOAD set this way,
every execution of notmuch prints

ERROR: ld.so: object '/opt/xilinx/usb-driver/libusb-driver.so' from LD_PRELOAD 
cannot be preloaded: ignored.

and this breaks the parsing. So thanks Jameson for solving this issue
for me.

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Thomas Schwinge
Hallo!

Stepping away from the current code base -- what is notmuch's original
idea of concurrency?  That is, all of us probably know that one:

A Xapian exception occurred opening database: Unable to get write
  lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
  already locked

I recently saw that one while using the Emacs UI (that one tried to
remove a unread tag or similar), and in parallel a delivery to the
notmuch DB was going on.

Apparently the DB we're using doesn't allow for simultaneous writing
(even though it can't even possibly have been dangerous in this case).

Which is the original idea here?  Is it that...

  * each and every client should catch these kinds of errors, and retry,
or eventually give up at some point, and report the status to the
user; or is it that...

  * notmuch internally should catch these concurrency cases, and retry,
or eventually give up at some point (``notmuch --maximum-wait=30s tag
[...]''), and fail as seen above?

This one is an obvious temporary error due to a concurrency situation.
Wouldn't the latter suggestion be preferable here?  I guess that in most
cases the DB isn't locked for long periods of time, and thus the
concurrency situation would decline quickly.


One difficulty I see is judging which errors are temporary and which are
permanent -- which is obvious in a lot of cases (concurrent DB access,
memory starved or any other OS resource), but may not be, for example in
case of I/O errors (is ``disk full'' a permanent error?).  And then, for
some of these cases, waiting does make sense (concurrent DB access, as
suggested above), and for other (temporary?) errors it doesn't make (a
lot of) sense (out of memory: only sensible thing is to abort, and have
the caller re-try, or disk full: waiting for some free space may be worth
it, or it may be not).


Grüße,
 Thomas


pgpQ4jpU3xiMy.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread micah anderson
On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge tho...@schwinge.name 
wrote:
 Stepping away from the current code base -- what is notmuch's original
 idea of concurrency?  That is, all of us probably know that one:
 
 A Xapian exception occurred opening database: Unable to get write
   lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
   already locked
 
 I recently saw that one while using the Emacs UI (that one tried to
 remove a unread tag or similar), and in parallel a delivery to the
 notmuch DB was going on.

Due to my harddisk in my laptop being slow (5400RPM), my notmuch
database growing, and perhaps some fragmentation somewhere, this has
become *incredibly* annoying for me. I am checking email every 30
minutes, and I'm nicing and ionicing the processes so I can use my
machine, but while those processes are running, I'm effectively locked
out of a good portion of my email. 

Usually, I switch to another task until my disk light has ceased being
solid, because the update time is too slow for me to wait. 

Now that folders are making it in, the two remaining features that are
driving me nuts with notmuch is this one and the
verification/decryption/encryption process (replying to an encrypted
message is 12 distinct steps for me, which is discouraging me from doing
that at all). 

I really don't want to complain, because I have no time to help in these
areas,  rather I'm interested  to know  if anyone  has any  pointers for
making this less annoying, and I'm  hoping that at some point I can free
up time to help. Perhaps I need to dump/restore my notmuch DB? Or index
less mail?

micah


pgpd38QjpX1DO.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Jameson Rollins
On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson mi...@riseup.net wrote:
 Due to my harddisk in my laptop being slow (5400RPM), my notmuch
 database growing, and perhaps some fragmentation somewhere, this has
 become *incredibly* annoying for me. I am checking email every 30
 minutes, and I'm nicing and ionicing the processes so I can use my
 machine, but while those processes are running, I'm effectively locked
 out of a good portion of my email. 

I also have a very slow disk, but this is very rarely a problem for me.
I retrieve mail every 10 minutes, and the corresponding notmuch new
usually takes a minute or so.  I really haven't found it to be much of a
bother to just wait it out.

One of the suggested ways to develop around this problem would be a
notmuch daemon that would queue database modification requests.  I don't
think anyone has been working on this yet, but if this is a big problem
for you guys, you might start looking into putting one together.

jamie.


pgpz6tFW8ZWjT.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 1/3] new: Do not defer maildir flag synchronization during the first run

2011-01-27 Thread Austin Clements
Looks like eagerly synchronizing tags is easy and works fine, though I need
to beef up the tests and put some transactions around things before I'm
satisfied.

I added a notmuch_database_remove_message_get to the public API that's
just like notmuch_database_remove_message except that it also returns a
notmuch_message_t if other instances of the message still exist.  It feels
clunky to have two almost identical variants of this function.  Is this the
preferred way to change the public API?  Or should I just add the argument
to the existing function and fix the other three calls to it?
On Thu, Jan 27, 2011 at 12:43 AM, Austin Clements amdra...@mit.edu wrote:

 Sure. I've been wanting to take a crack at notmuch new's atomicity for a
 while. Though you'll have to get through some of my outstanding patches. I
 can only keep so many branches in my head. ]:--8)

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.

2011-01-27 Thread Carl Worth
On Thu, 27 Jan 2011 02:17:21 -0700, Thomas Schwinge tho...@schwinge.name 
wrote:
 +# Note that in the way we're setting it above and using it below,
 +# `additional_headers' will also serve as the header / body separator
 +# (empty line in between).

Thanks, Thomas.

I've merged this now.

I'd even prefer to have the newline explicitly in the HERE document, but
it's awkward to avoid having the extra newline at the end of
${additional_headers} the way I'm constructing it incrementally. So just
documenting the current approach is probably best for now.

-Carl

-- 
carl.d.wo...@intel.com


pgp7Bw5UGDQgf.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Clarify usage of `additional_headers' in test/test-lib.sh:generate_message.

2011-01-27 Thread Thomas Schwinge
Hallo!

On Fri, 28 Jan 2011 15:36:25 +1000, Carl Worth cwo...@cworth.org wrote:
 On Thu, 27 Jan 2011 02:17:21 -0700, Thomas Schwinge tho...@schwinge.name 
 wrote:
  +# Note that in the way we're setting it above and using it below,
  +# `additional_headers' will also serve as the header / body separator
  +# (empty line in between).

 I'd even prefer to have the newline explicitly in the HERE document, but
 it's awkward to avoid having the extra newline at the end of
 ${additional_headers} the way I'm constructing it incrementally. So just
 documenting the current approach is probably best for now.

Matches my thoughts :-) -- and as it occurs to me right now, doing it in
one here document should be possible like this, if additional_headers is
changed to have the newline *at the beginning* of the string:

cat EOF $gen_msg_filename
From: ${template[from]}
To: ${template[to]}
Message-Id: ${gen_msg_id}
Subject: ${template[subject]}
Date: ${template[date]}${additional_headers}

${template[body]}
EOF

Or, of course, we could split the here document: base header,
conditionally (if set at all) additional_headers, new line, body.

If you'd like me to prepare (and test) any of these, please tell.


Grüße,
 Thomas


PS: Didn't know you'd be doing a presentation of notmuch at LCA2011 -- I
saw your announcement on the IRC channel (re live stream) what it was too
late already.  But then, it would have been a rather inconvenient time /
timezone anyways, being based in Germany.  So, how has it been?


pgpKO5IrlXe81.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch