notmuch's idea of concurrency / failing an invocation

[Austin Clements]

Now that I've split notmuch new up in to lots of small transactions, I
think the database locking issue is quite approachable.  Here's a
proposed locking protocol where a notmuch operation that wants to
modify the database blocks if there's another operation in progress
(instead of immediately failing like now), but indicates to the
in-progress operation that, when convenient, it should temporarily
abdicate the database.

Add a file to the .notmuch directory, say "lock", which we'll use for
fcntl locks (fcntl locks have nice properties, like automatic cleanup
on process exit and NFS compatibility).

To open the database for write,
1. Acquire an exclusive lock on byte 0 of the lock file (in blocking mode)
2. Acquire an exclusive lock on byte 1 of the lock file
3. Release the lock on byte 0
4. Open the Xapian database

When it's convenient to abdicate the lock, test if there are pending
operations by testing for a lock on byte 0 of the lock file using
F_GETLK.  If there's no lock on byte 0, just continue without
releasing the database.  Otherwise,
1. Close the Xapian database
2. Release the lock on byte 1
3. Re-lock and re-open the database.

In effect, this acts like one lock, since byte 1 is only ever acquired
while byte 0 is held, but splitting it across two locks like this lets
us "peek" at the waiter queue and see if someone is waiting.

Some possible extensions: We may want a timeout for how long to wait
for the lock (in case the lock holder gets wedged).  We could work
around DatabaseModified exceptions by having readers do essentially
the same thing as writers, but take the locks in shared mode.  Readers
wouldn't proceed in parallel with writers, but long-running writers
would relinquish the lock, so this isn't so bad.  Finally, concurrent
notmuch new's should probably be serialized (instead of repeatedly
abdicating to each other), so it may make sense to have an additional
"notmuch new lock".

On Thu, Jan 27, 2011 at 5:20 PM, Austin Clements  wrote:
> I'm looking into breaking notmuch new up into small transactions. ?It
> wouldn't be much a leap from there to simply close and reopen the database
> between transactions if another task wants to use it, which would release
> the lock and let the queued notmuch task have the database for a bit. ?It
> seems silly to have a daemon when all of notmuch's state is already on disk
> and queue on a lock is as good as a queue in a daemon, but without the
> accompanying architectural?shenanigans.

notmuch's idea of concurrency / failing an invocation

[micah anderson]

On Thu, 27 Jan 2011 12:35:16 -0800, Jameson Rollins  wrote:
> On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson  
> wrote:
> > Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> > database growing, and perhaps some fragmentation somewhere, this has
> > become *incredibly* annoying for me. I am checking email every 30
> > minutes, and I'm nicing and ionicing the processes so I can use my
> > machine, but while those processes are running, I'm effectively locked
> > out of a good portion of my email. 
> 
> I also have a very slow disk, but this is very rarely a problem for me.
> I retrieve mail every 10 minutes, and the corresponding notmuch new
> usually takes a minute or so.  I really haven't found it to be much of a
> bother to just wait it out.

Sometimes I can have several thousand messages to add/remove from the
database. I know this is probably unusal, but for me its not due to
system emails. I suppose if I checked my email more frequently, I'd have
less to process on each run, but thats more side-stepping the
concurrency issue.

micah
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 

Re: notmuch's idea of concurrency / failing an invocation

[micah anderson]

On Thu, 27 Jan 2011 12:35:16 -0800, Jameson Rollins 
jroll...@finestructure.net wrote:
 On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson mi...@riseup.net wrote:
  Due to my harddisk in my laptop being slow (5400RPM), my notmuch
  database growing, and perhaps some fragmentation somewhere, this has
  become *incredibly* annoying for me. I am checking email every 30
  minutes, and I'm nicing and ionicing the processes so I can use my
  machine, but while those processes are running, I'm effectively locked
  out of a good portion of my email. 
 
 I also have a very slow disk, but this is very rarely a problem for me.
 I retrieve mail every 10 minutes, and the corresponding notmuch new
 usually takes a minute or so.  I really haven't found it to be much of a
 bother to just wait it out.

Sometimes I can have several thousand messages to add/remove from the
database. I know this is probably unusal, but for me its not due to
system emails. I suppose if I checked my email more frequently, I'd have
less to process on each run, but thats more side-stepping the
concurrency issue.

micah


pgp62R9y86RGi.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch's idea of concurrency / failing an invocation

[Stewart Smith]

On Sat, 29 Jan 2011 19:14:27 -0500, Daniel Kahn Gillmor  wrote:
> On 01/28/2011 08:05 PM, Stewart Smith wrote:
> > I'm about at the point where I'm going to take my git mail store
> > experiments and get them really to work (and everyone will have to use
> > 'notmuch cat' or the like to access the messages)
> 
> Would this hypothetical git-based mail store retain the atomicity and
> lockless concurrent-access of a maildir?  That is, could it be used in a
> server environment?

My idea is that it would be... at least with the experiments conducted
so far.

> > which should provide
> > both great storage efficiency, much faster backups of your Maildir as
> > well as having way fewer paths to traverse checking for new mail.
> 
> when you say "backups of your Maildir" do you mean "backups of your
> git-based mail store" ?  or is this somehow a literal Maildir stored in git?

I'll write more "soon" when there is more code behind it... and I figure
out a good upgrade path to something that is also self-consistently sane.

-- 
Stewart Smith

Re: notmuch's idea of concurrency / failing an invocation

[Stewart Smith]

On Sat, 29 Jan 2011 19:14:27 -0500, Daniel Kahn Gillmor 
d...@fifthhorseman.net wrote:
 On 01/28/2011 08:05 PM, Stewart Smith wrote:
  I'm about at the point where I'm going to take my git mail store
  experiments and get them really to work (and everyone will have to use
  'notmuch cat' or the like to access the messages)
 
 Would this hypothetical git-based mail store retain the atomicity and
 lockless concurrent-access of a maildir?  That is, could it be used in a
 server environment?

My idea is that it would be... at least with the experiments conducted
so far.

  which should provide
  both great storage efficiency, much faster backups of your Maildir as
  well as having way fewer paths to traverse checking for new mail.
 
 when you say backups of your Maildir do you mean backups of your
 git-based mail store ?  or is this somehow a literal Maildir stored in git?

I'll write more soon when there is more code behind it... and I figure
out a good upgrade path to something that is also self-consistently sane.

-- 
Stewart Smith
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch's idea of concurrency / failing an invocation

[Daniel Kahn Gillmor]

On 01/28/2011 08:05 PM, Stewart Smith wrote:
> I'm about at the point where I'm going to take my git mail store
> experiments and get them really to work (and everyone will have to use
> 'notmuch cat' or the like to access the messages)

Would this hypothetical git-based mail store retain the atomicity and
lockless concurrent-access of a maildir?  That is, could it be used in a
server environment?

> which should provide
> both great storage efficiency, much faster backups of your Maildir as
> well as having way fewer paths to traverse checking for new mail.

when you say "backups of your Maildir" do you mean "backups of your
git-based mail store" ?  or is this somehow a literal Maildir stored in git?

Intrigued,

--dkg

-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1030 bytes
Desc: OpenPGP digital signature
URL: 

notmuch's idea of concurrency / failing an invocation

[Mike Kelly]

On Fri, 28 Jan 2011 11:57:34 -0500
Austin Clements  wrote:

> Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the
> ability to block on the lock (see the fcntl call in
> backends/flint_lock.cc, which is hard-coded to the non-blocking
> F_SETLK instead of F_SETLKW), so we'd either need a new Xapian
> option, or we would just have to wrap our own flock/fcntl lock around
> things as you suggest.

Hrm. Do you know if Xapian upstream would be open to a patch to support
optional blocking locks? We can't be the only ones hitting these sorts
of issues.

-- 
Mike Kelly
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: 

notmuch's idea of concurrency / failing an invocation

[Stewart Smith]

On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson  wrote:
> Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> database growing, and perhaps some fragmentation somewhere, this has
> become *incredibly* annoying for me. I am checking email every 30
> minutes, and I'm nicing and ionicing the processes so I can use my
> machine, but while those processes are running, I'm effectively locked
> out of a good portion of my email. 

I used to use spinning rust and also noticed things were slow. This
is in fact mostly not xapian - but rather crawling the Maildir. I
improved this early on in notmuch history by reducing the number of
seeks needed when traversing the Maildir hierarchy (e.g. stat in
i-node order, which is roughly on-disk order).

I'm about at the point where I'm going to take my git mail store
experiments and get them really to work (and everyone will have to use
'notmuch cat' or the like to access the messages) which should provide
both great storage efficiency, much faster backups of your Maildir as
well as having way fewer paths to traverse checking for new mail.

-- 
Stewart Smith

Re: notmuch's idea of concurrency / failing an invocation

[Mike Kelly]

On Fri, 28 Jan 2011 11:57:34 -0500
Austin Clements amdra...@mit.edu wrote:

 Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the
 ability to block on the lock (see the fcntl call in
 backends/flint_lock.cc, which is hard-coded to the non-blocking
 F_SETLK instead of F_SETLKW), so we'd either need a new Xapian
 option, or we would just have to wrap our own flock/fcntl lock around
 things as you suggest.

Hrm. Do you know if Xapian upstream would be open to a patch to support
optional blocking locks? We can't be the only ones hitting these sorts
of issues.

-- 
Mike Kelly


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch's idea of concurrency / failing an invocation

[Stewart Smith]

On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson mi...@riseup.net wrote:
 Due to my harddisk in my laptop being slow (5400RPM), my notmuch
 database growing, and perhaps some fragmentation somewhere, this has
 become *incredibly* annoying for me. I am checking email every 30
 minutes, and I'm nicing and ionicing the processes so I can use my
 machine, but while those processes are running, I'm effectively locked
 out of a good portion of my email. 

I used to use spinning rust and also noticed things were slow. This
is in fact mostly not xapian - but rather crawling the Maildir. I
improved this early on in notmuch history by reducing the number of
seeks needed when traversing the Maildir hierarchy (e.g. stat in
i-node order, which is roughly on-disk order).

I'm about at the point where I'm going to take my git mail store
experiments and get them really to work (and everyone will have to use
'notmuch cat' or the like to access the messages) which should provide
both great storage efficiency, much faster backups of your Maildir as
well as having way fewer paths to traverse checking for new mail.

-- 
Stewart Smith
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch's idea of concurrency / failing an invocation

[Daniel Kahn Gillmor]

On 01/28/2011 08:05 PM, Stewart Smith wrote:
 I'm about at the point where I'm going to take my git mail store
 experiments and get them really to work (and everyone will have to use
 'notmuch cat' or the like to access the messages)

Would this hypothetical git-based mail store retain the atomicity and
lockless concurrent-access of a maildir?  That is, could it be used in a
server environment?

 which should provide
 both great storage efficiency, much faster backups of your Maildir as
 well as having way fewer paths to traverse checking for new mail.

when you say backups of your Maildir do you mean backups of your
git-based mail store ?  or is this somehow a literal Maildir stored in git?

Intrigued,

--dkg



signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch's idea of concurrency / failing an invocation

[Mike Kelly]

On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge  
wrote:
> > It would definitely be nice to avoid the complexity inherent in having a
> > daemon, but how do you imagine "queue on a lock" to work? We don't have
> > anything like that in place now.
>
> I suppose what he means is trying to get the lock, and if that fails wait
> a bit / wait until it is available again.
>
> Actually, as a next step, wouldn't it also be possible to add some
> heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
> some interactive user (UI; high-priority task)?  But we can pursue such
> schemes as soon as the basic infrastructure is in place.

Couldn't we pretty much get the desired behavior by using flock(2)?
Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
we only need to read. Using the blocking form, things should pretty much
just queue up and take their turn, right?

I'm not familiar with Xapian, but if it doesn't give us something we
could use this sort of locking on, couldn't we just add some
/path/to/mail/.notmuch.lock file that we open to hold a lock on?

We already have to specify if we want a read-only or read-write database
handle in notmuch_database_open, so it seems like it'd be easy enough to
hook in there.

-- 
Mike Kelly

notmuch's idea of concurrency / failing an invocation

[Carl Worth]

On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements  wrote:
> I'm looking into breaking notmuch new up into small transactions.  It
> wouldn't be much a leap from there to simply close and reopen the database
> between transactions if another task wants to use it, which would release
> the lock and let the queued notmuch task have the database for a bit.

That sounds like something very useful to pursue. Please continue!

> It seems silly to have a daemon when all of notmuch's state is already on disk
> and queue on a lock is as good as a queue in a daemon, but without the
> accompanying architectural shenanigans.

It would definitely be nice to avoid the complexity inherent in having a
daemon, but how do you imagine "queue on a lock" to work? We don't have
anything like that in place now.

Another advantage that can happen with queueing (wherever it occurs) is
to allow a client to be very responsive without waiting for an operation
to complete. Though that can of course be band if the operation isn't
reliably committed.

-Carl

-- 
carl.d.worth at intel.com
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

notmuch's idea of concurrency / failing an invocation

[Carl Worth]

On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge  
wrote:
> Which is the original idea here?  Is it that...

There's no original idea yet. It's essentially an unsolved problem right now.

>   * each and every client should catch these kinds of errors, and retry,
> or eventually give up at some point, and report the status to the
> user; or is it that...
> 
>   * notmuch internally should catch these concurrency cases, and retry,
> or eventually give up at some point (``notmuch --maximum-wait=30s tag
> [...]''), and fail as seen above?

Some people have actually already done work solutions in one way or
another. Here are a few of the messages I found in my "outstanding
notmuch mail to read"[*] queue:

James Vasile patched the emacs interface to call notmuch
asynchronously and to repeatedly call it if it fails (he also
wonders if it should have some sort of timeout):

id:"87vddnlxos.wl%james at hackervisions.org"

James also wrote a shell script that repeatedly calls the notmuch
binary as necessary (and he wonders if this retrying should happen
inside notmuch itself):

id:"87pr3sw43a.fsf at hackervisions.org"


"servilio" wrote a new "notmuch repl" command which can accept
notmuch operations expressed in text form on stdin, and then
interpret and execute them. That's a good start on a notmuch daemon:

id:"AANLkTi=7eCt0=NqUiJFrGDcaZ17LOd3qNNqN1-ASwYzr at mail.gmail.com"

I'm not sure yet which approach (or approaches) we want. But I would
love to see some of the limitations described in the messages above
addressed. That would definitely make some of the patches more
acceptable.

-Carl

[*] And yes, my queue really does span a year(!) or so. That's
embarrassing. I'm committed to making progress on this queue and staying
up-to-date with new patches, so I've made a couple of recent changes:

1. I'm now processing the queue largely in reverse-chronological
   order. The idea here is that I can stay on top of new posts, while
   also making progress on previously-sent items.

   This does mean that you can hack my workflow by replying to an old
   thread, (and thereby bringing it back to my attention). Please feel
   free to do that---ideally by mentioning any new information such as
   "these patches are now rebased " or "I've tested these patches
   in daily use for X months and they still apply fine to master" or
   similar.

2. I've date-limited my saved search for my notmuch queue to show a
   small number of messages. This is a cheap psychological hack. If the
   number on the queue is too large it makes me hesitant to even look at
   it. But with a small number, it's easier to make progress since the
   end is apparently in sight.

   Of course, once I reduce my date-limited queue to 0, I'll extend the
   date back into the past and try to keep working through things.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 

notmuch's idea of concurrency / failing an invocation

[Austin Clements]

Actually, this is trivial to play with.  Here's a stop-gap wrapper
script for people having trouble with Xapian locking,

#!/bin/bash

NOTMUCH_BIN="/path/to/notmuch"
MAIL_DIR="/path/to/mailroot"

(
case "$1" in
setup|help)
;;
search|show|count|reply|dump|search-tags|part)
flock -s 200;;
*)
flock -x 200;;
esac
"$NOTMUCH_BIN" "$@" 200>&-
) 200>"$MAIL_DIR"/.notmuch/lock

On Fri, Jan 28, 2011 at 11:57 AM, Austin Clements  wrote:
>
> On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly  wrote:
>>
>> On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge > schwinge.name> wrote:
>> > > It would definitely be nice to avoid the complexity inherent in having a
>> > > daemon, but how do you imagine "queue on a lock" to work? We don't have
>> > > anything like that in place now.
>> >
>> > I suppose what he means is trying to get the lock, and if that fails wait
>> > a bit / wait until it is available again.
>> >
>> > Actually, as a next step, wouldn't it also be possible to add some
>> > heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
>> > some interactive user (UI; high-priority task)? ?But we can pursue such
>> > schemes as soon as the basic infrastructure is in place.
>>
>> Couldn't we pretty much get the desired behavior by using flock(2)?
>> Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
>> we only need to read. Using the blocking form, things should pretty much
>> just queue up and take their turn, right?
>>
>> I'm not familiar with Xapian, but if it doesn't give us something we
>> could use this sort of locking on, couldn't we just add some
>> /path/to/mail/.notmuch.lock file that we open to hold a lock on?
>
> Yes, exactly. ?All of this. ?Unfortunately, Xapian doesn't expose the ability 
> to block on the lock (see the fcntl call in backends/flint_lock.cc, which is 
> hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so we'd either 
> need a new Xapian option, or we would just have to wrap our own flock/fcntl 
> lock around things as you suggest.

notmuch's idea of concurrency / failing an invocation

[Austin Clements]

On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly  wrote:

> On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge 
> wrote:
> > > It would definitely be nice to avoid the complexity inherent in having
> a
> > > daemon, but how do you imagine "queue on a lock" to work? We don't have
> > > anything like that in place now.
> >
> > I suppose what he means is trying to get the lock, and if that fails wait
> > a bit / wait until it is available again.
> >
> > Actually, as a next step, wouldn't it also be possible to add some
> > heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
> > some interactive user (UI; high-priority task)?  But we can pursue such
> > schemes as soon as the basic infrastructure is in place.
>
> Couldn't we pretty much get the desired behavior by using flock(2)?
> Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
> we only need to read. Using the blocking form, things should pretty much
> just queue up and take their turn, right?
>
> I'm not familiar with Xapian, but if it doesn't give us something we
> could use this sort of locking on, couldn't we just add some
> /path/to/mail/.notmuch.lock file that we open to hold a lock on?
>

Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the
ability to block on the lock (see the fcntl call in backends/flint_lock.cc,
which is hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so
we'd either need a new Xapian option, or we would just have to wrap our own
flock/fcntl lock around things as you suggest.
-- next part --
An HTML attachment was scrubbed...
URL: 

notmuch's idea of concurrency / failing an invocation

[Austin Clements]

On Fri, Jan 28, 2011 at 4:45 AM, Thomas Schwinge wrote:

> On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth  wrote:
> > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements 
> wrote:
> > > I'm looking into breaking notmuch new up into small transactions.  It
> > > wouldn't be much a leap from there to simply close and reopen the
> database
> > > between transactions if another task wants to use it, which would
> release
> > > the lock and let the queued notmuch task have the database for a bit.
> >
> > That sounds like something very useful to pursue. Please continue!
>
> Ack!  And actually -- I just wondered about that: what happens if
> ``notmuch new'' has executed notmuch_database_add_message for a new
> message M, but then is killed before adding any tags and finishing up
> (and supposing that the DB isn't in an invalid state now).  This process
> of adding M to the DB and applying any tags isn't one single transaction
> currently, right?  (And this is exactly what you're working on
> chainging?)  Am I right that what currently happens is that upon the next
> ``notmuch new'' run, notmuch will not reconsider M (given that it already
> is present in the DB), but continue with the next messages -- thus
> leaving M without any tags?  This isn't a very likely scenario, but still
> a possible one.


There are quite a few bugs like this.  In fact, last night I added a test
that interrupts notmuch new (for real, not SIGINT) after every database
write, and on each interrupted database snapshot, re-runs notmuch new to
completion, then checks that the database winds up in the correct state.
 There are dozens of interruption points where it doesn't, many of which are
permanent, even if you force notmuch new to rescan the maildir.

> Another advantage that can happen with queueing (wherever it occurs) is
> > to allow a client to be very responsive without waiting for an operation
> > to complete. Though that can of course be band if the operation isn't
> > reliably committed.
>
> (Obviously this can only work as long as we don't immediatelly need the
> operation's result; think ``notmuch show''.)
>
> So, if the DB has the functionality to internally queue and immediatelly
> acknowledge transactions, and only later (reliably) commit them, wouldn't
> that be fine indeed?  For example, ``notmuch tag'' then wouldn't have to
> wait for the DB to be writable.  (And note that I have no idea whether
> Xapian supports such things.)  But on the other hand we would like to
> immediatelly display the requested change in the UI, right?
>

This would be fantastic, if the client could indicate the difference between
a "pending" change and a "committed" change as you suggest below.  I don't
think having the database lie about its commit state is the right way to do
this, though (nor should the client lie about this, thus the "pending"
display).  A better way would be for the client to update the display to
"pending", start the notmuch operation asynchronously, have the notmuch
operation block and queue up on the database lock, then have the client
update the display to "committed" when the asynchronous operation returns.
 No weird database operations or transactional semantics and the client side
is fairly straightforward.
-- next part --
An HTML attachment was scrubbed...
URL: 

notmuch's idea of concurrency / failing an invocation

[Thomas Schwinge]

Hallo!

On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth  wrote:
> On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements  
> wrote:
> > I'm looking into breaking notmuch new up into small transactions.  It
> > wouldn't be much a leap from there to simply close and reopen the database
> > between transactions if another task wants to use it, which would release
> > the lock and let the queued notmuch task have the database for a bit.
> 
> That sounds like something very useful to pursue. Please continue!

Ack!  And actually -- I just wondered about that: what happens if
``notmuch new'' has executed notmuch_database_add_message for a new
message M, but then is killed before adding any tags and finishing up
(and supposing that the DB isn't in an invalid state now).  This process
of adding M to the DB and applying any tags isn't one single transaction
currently, right?  (And this is exactly what you're working on
chainging?)  Am I right that what currently happens is that upon the next
``notmuch new'' run, notmuch will not reconsider M (given that it already
is present in the DB), but continue with the next messages -- thus
leaving M without any tags?  This isn't a very likely scenario, but still
a possible one.

> > It seems silly to have a daemon when all of notmuch's state is already on 
> > disk
> > and queue on a lock is as good as a queue in a daemon, but without the
> > accompanying architectural shenanigans.

Ack, too.  A daemon seems one abstraction layer too much.  (But I'm not
actively opposed either, if someone has a valid use for such a scheme.)

> It would definitely be nice to avoid the complexity inherent in having a
> daemon, but how do you imagine "queue on a lock" to work? We don't have
> anything like that in place now.

I suppose what he means is trying to get the lock, and if that fails wait
a bit / wait until it is available again.

Actually, as a next step, wouldn't it also be possible to add some
heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
some interactive user (UI; high-priority task)?  But we can pursue such
schemes as soon as the basic infrastructure is in place.

> Another advantage that can happen with queueing (wherever it occurs) is
> to allow a client to be very responsive without waiting for an operation
> to complete. Though that can of course be band if the operation isn't
> reliably committed.

(Obviously this can only work as long as we don't immediatelly need the
operation's result; think ``notmuch show''.)

So, if the DB has the functionality to internally queue and immediatelly
acknowledge transactions, and only later (reliably) commit them, wouldn't
that be fine indeed?  For example, ``notmuch tag'' then wouldn't have to
wait for the DB to be writable.  (And note that I have no idea whether
Xapian supports such things.)  But on the other hand we would like to
immediatelly display the requested change in the UI, right?

What notmuch-show.el:notmuch-show-remove-tag currently does is *not*
re-asking the DB for a message's current tags after having removed a
specific one, but instead it interprets the tag removal command itself --
which is easy enough in this case, and rather unlikely to ever yield
different results, at least unless there's another process operating on
the DB concurrently.

Otherwise, the other way round, the client could maintain a list of to-do
items, to which actions are added if the DB is currently busy, and this
list is periodically worked on in order to get it empty.  For example,
tag changes that are in this list, but not yet committed in the DB could
be displayed in another color in the UI.  But doing so would shift the
responsibility to the UI, which should be in the DB, in my humble
opinion.  (Actually, this issue feels similar to the one who should be
doing the re-trying in case the DB is busy: the UI, or the notmuch
process itself, which we're discussing in another thread.)


As you can guess I'm not very much into DBs, and neither too much into
concurrent systems, so if my ideas don't make sense, please feel free to
refer me to literature.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 

Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements]

On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly pi...@pioto.org wrote:

 On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge tho...@schwinge.name
 wrote:
   It would definitely be nice to avoid the complexity inherent in having
 a
   daemon, but how do you imagine queue on a lock to work? We don't have
   anything like that in place now.
 
  I suppose what he means is trying to get the lock, and if that fails wait
  a bit / wait until it is available again.
 
  Actually, as a next step, wouldn't it also be possible to add some
  heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
  some interactive user (UI; high-priority task)?  But we can pursue such
  schemes as soon as the basic infrastructure is in place.

 Couldn't we pretty much get the desired behavior by using flock(2)?
 Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
 we only need to read. Using the blocking form, things should pretty much
 just queue up and take their turn, right?

 I'm not familiar with Xapian, but if it doesn't give us something we
 could use this sort of locking on, couldn't we just add some
 /path/to/mail/.notmuch.lock file that we open to hold a lock on?


Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the
ability to block on the lock (see the fcntl call in backends/flint_lock.cc,
which is hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so
we'd either need a new Xapian option, or we would just have to wrap our own
flock/fcntl lock around things as you suggest.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch's idea of concurrency / failing an invocation

[Austin Clements]

Actually, this is trivial to play with.  Here's a stop-gap wrapper
script for people having trouble with Xapian locking,

#!/bin/bash

NOTMUCH_BIN=/path/to/notmuch
MAIL_DIR=/path/to/mailroot

(
case $1 in
setup|help)
;;
search|show|count|reply|dump|search-tags|part)
flock -s 200;;
*)
flock -x 200;;
esac
$NOTMUCH_BIN $@ 200-
) 200$MAIL_DIR/.notmuch/lock

On Fri, Jan 28, 2011 at 11:57 AM, Austin Clements amdra...@mit.edu wrote:

 On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly pi...@pioto.org wrote:

 On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge tho...@schwinge.name 
 wrote:
   It would definitely be nice to avoid the complexity inherent in having a
   daemon, but how do you imagine queue on a lock to work? We don't have
   anything like that in place now.
 
  I suppose what he means is trying to get the lock, and if that fails wait
  a bit / wait until it is available again.
 
  Actually, as a next step, wouldn't it also be possible to add some
  heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
  some interactive user (UI; high-priority task)?  But we can pursue such
  schemes as soon as the basic infrastructure is in place.

 Couldn't we pretty much get the desired behavior by using flock(2)?
 Basically, take out a LOCK_EX when we need to write, and a LOCK_SH when
 we only need to read. Using the blocking form, things should pretty much
 just queue up and take their turn, right?

 I'm not familiar with Xapian, but if it doesn't give us something we
 could use this sort of locking on, couldn't we just add some
 /path/to/mail/.notmuch.lock file that we open to hold a lock on?

 Yes, exactly.  All of this.  Unfortunately, Xapian doesn't expose the ability 
 to block on the lock (see the fcntl call in backends/flint_lock.cc, which is 
 hard-coded to the non-blocking F_SETLK instead of F_SETLKW), so we'd either 
 need a new Xapian option, or we would just have to wrap our own flock/fcntl 
 lock around things as you suggest.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch's idea of concurrency / failing an invocation

[Thomas Schwinge]

Hallo!

Stepping away from the current code base -- what is notmuch's original
idea of concurrency?  That is, all of us probably know that one:

A Xapian exception occurred opening database: Unable to get write
  lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
  already locked

I recently saw that one while using the Emacs UI (that one tried to
remove a unread tag or similar), and in parallel a delivery to the
notmuch DB was going on.

Apparently the DB we're using doesn't allow for simultaneous writing
(even though it can't even possibly have been dangerous in this case).

Which is the original idea here?  Is it that...

  * each and every client should catch these kinds of errors, and retry,
or eventually give up at some point, and report the status to the
user; or is it that...

  * notmuch internally should catch these concurrency cases, and retry,
or eventually give up at some point (``notmuch --maximum-wait=30s tag
[...]''), and fail as seen above?

This one is an obvious temporary error due to a concurrency situation.
Wouldn't the latter suggestion be preferable here?  I guess that in most
cases the DB isn't locked for long periods of time, and thus the
concurrency situation would decline quickly.


One difficulty I see is judging which errors are temporary and which are
permanent -- which is obvious in a lot of cases (concurrent DB access,
memory starved or any other OS resource), but may not be, for example in
case of I/O errors (is ``disk full'' a permanent error?).  And then, for
some of these cases, waiting does make sense (concurrent DB access, as
suggested above), and for other (temporary?) errors it doesn't make (a
lot of) sense (out of memory: only sensible thing is to abort, and have
the caller re-try, or disk full: waiting for some free space may be worth
it, or it may be not).


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 

notmuch's idea of concurrency / failing an invocation

[Austin Clements]

I'm looking into breaking notmuch new up into small transactions.  It
wouldn't be much a leap from there to simply close and reopen the database
between transactions if another task wants to use it, which would release
the lock and let the queued notmuch task have the database for a bit.  It
seems silly to have a daemon when all of notmuch's state is already on disk
and queue on a lock is as good as a queue in a daemon, but without the
accompanying architectural shenanigans.

On Thu, Jan 27, 2011 at 3:35 PM, Jameson Rollins  wrote:

> On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson 
> wrote:
> > Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> > database growing, and perhaps some fragmentation somewhere, this has
> > become *incredibly* annoying for me. I am checking email every 30
> > minutes, and I'm nicing and ionicing the processes so I can use my
> > machine, but while those processes are running, I'm effectively locked
> > out of a good portion of my email.
>
> I also have a very slow disk, but this is very rarely a problem for me.
> I retrieve mail every 10 minutes, and the corresponding notmuch new
> usually takes a minute or so.  I really haven't found it to be much of a
> bother to just wait it out.
>
> One of the suggested ways to develop around this problem would be a
> notmuch daemon that would queue database modification requests.  I don't
> think anyone has been working on this yet, but if this is a big problem
> for you guys, you might start looking into putting one together.
>
> jamie.
-- next part --
An HTML attachment was scrubbed...
URL: 

notmuch's idea of concurrency / failing an invocation

[micah anderson]

On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge  
wrote:
> Stepping away from the current code base -- what is notmuch's original
> idea of concurrency?  That is, all of us probably know that one:
> 
> A Xapian exception occurred opening database: Unable to get write
>   lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
>   already locked
> 
> I recently saw that one while using the Emacs UI (that one tried to
> remove a unread tag or similar), and in parallel a delivery to the
> notmuch DB was going on.

Due to my harddisk in my laptop being slow (5400RPM), my notmuch
database growing, and perhaps some fragmentation somewhere, this has
become *incredibly* annoying for me. I am checking email every 30
minutes, and I'm nicing and ionicing the processes so I can use my
machine, but while those processes are running, I'm effectively locked
out of a good portion of my email. 

Usually, I switch to another task until my disk light has ceased being
solid, because the update time is too slow for me to wait. 

Now that folders are making it in, the two remaining features that are
driving me nuts with notmuch is this one and the
verification/decryption/encryption process (replying to an encrypted
message is 12 distinct steps for me, which is discouraging me from doing
that at all). 

I really don't want to complain, because I have no time to help in these
areas,  rather I'm interested  to know  if anyone  has any  pointers for
making this less annoying, and I'm  hoping that at some point I can free
up time to help. Perhaps I need to dump/restore my notmuch DB? Or index
less mail?

micah
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 

notmuch's idea of concurrency / failing an invocation

[Jameson Rollins]

On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson  wrote:
> Due to my harddisk in my laptop being slow (5400RPM), my notmuch
> database growing, and perhaps some fragmentation somewhere, this has
> become *incredibly* annoying for me. I am checking email every 30
> minutes, and I'm nicing and ionicing the processes so I can use my
> machine, but while those processes are running, I'm effectively locked
> out of a good portion of my email. 

I also have a very slow disk, but this is very rarely a problem for me.
I retrieve mail every 10 minutes, and the corresponding notmuch new
usually takes a minute or so.  I really haven't found it to be much of a
bother to just wait it out.

One of the suggested ways to develop around this problem would be a
notmuch daemon that would queue database modification requests.  I don't
think anyone has been working on this yet, but if this is a big problem
for you guys, you might start looking into putting one together.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 

notmuch's idea of concurrency / failing an invocation

[Thomas Schwinge]

Hallo!

Stepping away from the current code base -- what is notmuch's original
idea of concurrency?  That is, all of us probably know that one:

A Xapian exception occurred opening database: Unable to get write
  lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
  already locked

I recently saw that one while using the Emacs UI (that one tried to
remove a unread tag or similar), and in parallel a delivery to the
notmuch DB was going on.

Apparently the DB we're using doesn't allow for simultaneous writing
(even though it can't even possibly have been dangerous in this case).

Which is the original idea here?  Is it that...

  * each and every client should catch these kinds of errors, and retry,
or eventually give up at some point, and report the status to the
user; or is it that...

  * notmuch internally should catch these concurrency cases, and retry,
or eventually give up at some point (``notmuch --maximum-wait=30s tag
[...]''), and fail as seen above?

This one is an obvious temporary error due to a concurrency situation.
Wouldn't the latter suggestion be preferable here?  I guess that in most
cases the DB isn't locked for long periods of time, and thus the
concurrency situation would decline quickly.


One difficulty I see is judging which errors are temporary and which are
permanent -- which is obvious in a lot of cases (concurrent DB access,
memory starved or any other OS resource), but may not be, for example in
case of I/O errors (is ``disk full'' a permanent error?).  And then, for
some of these cases, waiting does make sense (concurrent DB access, as
suggested above), and for other (temporary?) errors it doesn't make (a
lot of) sense (out of memory: only sensible thing is to abort, and have
the caller re-try, or disk full: waiting for some free space may be worth
it, or it may be not).


Grüße,
 Thomas


pgpQ4jpU3xiMy.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch's idea of concurrency / failing an invocation

[micah anderson]

On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge tho...@schwinge.name 
wrote:
 Stepping away from the current code base -- what is notmuch's original
 idea of concurrency?  That is, all of us probably know that one:
 
 A Xapian exception occurred opening database: Unable to get write
   lock on /home/thomas/Mail-schwinge.name-thomas/.notmuch/xapian:
   already locked
 
 I recently saw that one while using the Emacs UI (that one tried to
 remove a unread tag or similar), and in parallel a delivery to the
 notmuch DB was going on.

Due to my harddisk in my laptop being slow (5400RPM), my notmuch
database growing, and perhaps some fragmentation somewhere, this has
become *incredibly* annoying for me. I am checking email every 30
minutes, and I'm nicing and ionicing the processes so I can use my
machine, but while those processes are running, I'm effectively locked
out of a good portion of my email. 

Usually, I switch to another task until my disk light has ceased being
solid, because the update time is too slow for me to wait. 

Now that folders are making it in, the two remaining features that are
driving me nuts with notmuch is this one and the
verification/decryption/encryption process (replying to an encrypted
message is 12 distinct steps for me, which is discouraging me from doing
that at all). 

I really don't want to complain, because I have no time to help in these
areas,  rather I'm interested  to know  if anyone  has any  pointers for
making this less annoying, and I'm  hoping that at some point I can free
up time to help. Perhaps I need to dump/restore my notmuch DB? Or index
less mail?

micah


pgpd38QjpX1DO.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch's idea of concurrency / failing an invocation

[Jameson Rollins]

On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson mi...@riseup.net wrote:
 Due to my harddisk in my laptop being slow (5400RPM), my notmuch
 database growing, and perhaps some fragmentation somewhere, this has
 become *incredibly* annoying for me. I am checking email every 30
 minutes, and I'm nicing and ionicing the processes so I can use my
 machine, but while those processes are running, I'm effectively locked
 out of a good portion of my email. 

I also have a very slow disk, but this is very rarely a problem for me.
I retrieve mail every 10 minutes, and the corresponding notmuch new
usually takes a minute or so.  I really haven't found it to be much of a
bother to just wait it out.

One of the suggested ways to develop around this problem would be a
notmuch daemon that would queue database modification requests.  I don't
think anyone has been working on this yet, but if this is a big problem
for you guys, you might start looking into putting one together.

jamie.


pgpz6tFW8ZWjT.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch