Re: notmuch's idea of concurrency / failing an invocation

2011-02-23 Thread Austin Clements
Now that I've split notmuch new up in to lots of small transactions, I think the database locking issue is quite approachable. Here's a proposed locking protocol where a notmuch operation that wants to modify the database blocks if there's another operation in progress (instead of immediately fail

Re: notmuch's idea of concurrency / failing an invocation

2011-02-03 Thread micah anderson
On Thu, 27 Jan 2011 12:35:16 -0800, Jameson Rollins wrote: > On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson wrote: > > Due to my harddisk in my laptop being slow (5400RPM), my notmuch > > database growing, and perhaps some fragmentation somewhere, this has > > become *incredibly* annoying fo

Re: notmuch's idea of concurrency / failing an invocation

2011-02-01 Thread Stewart Smith
On Sat, 29 Jan 2011 19:14:27 -0500, Daniel Kahn Gillmor wrote: > On 01/28/2011 08:05 PM, Stewart Smith wrote: > > I'm about at the point where I'm going to take my git mail store > > experiments and get them really to work (and everyone will have to use > > 'notmuch cat' or the like to access the

Re: notmuch's idea of concurrency / failing an invocation

2011-01-29 Thread Daniel Kahn Gillmor
On 01/28/2011 08:05 PM, Stewart Smith wrote: > I'm about at the point where I'm going to take my git mail store > experiments and get them really to work (and everyone will have to use > 'notmuch cat' or the like to access the messages) Would this hypothetical git-based mail store retain the atomi

Re: notmuch's idea of concurrency / failing an invocation

2011-01-29 Thread Stewart Smith
On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson wrote: > Due to my harddisk in my laptop being slow (5400RPM), my notmuch > database growing, and perhaps some fragmentation somewhere, this has > become *incredibly* annoying for me. I am checking email every 30 > minutes, and I'm nicing and ioni

Re: notmuch's idea of concurrency / failing an invocation

2011-01-29 Thread Mike Kelly
On Fri, 28 Jan 2011 11:57:34 -0500 Austin Clements wrote: > Yes, exactly. All of this. Unfortunately, Xapian doesn't expose the > ability to block on the lock (see the fcntl call in > backends/flint_lock.cc, which is hard-coded to the non-blocking > F_SETLK instead of F_SETLKW), so we'd either

Re: notmuch's idea of concurrency / failing an invocation

2011-01-28 Thread Austin Clements
Actually, this is trivial to play with. Here's a stop-gap wrapper script for people having trouble with Xapian locking, #!/bin/bash NOTMUCH_BIN="/path/to/notmuch" MAIL_DIR="/path/to/mailroot" ( case "$1" in setup|help) ;; search|show|count|reply|dump|search-tags|

Re: notmuch's idea of concurrency / failing an invocation

2011-01-28 Thread Austin Clements
On Fri, Jan 28, 2011 at 10:36 AM, Mike Kelly wrote: > On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge > wrote: > > > It would definitely be nice to avoid the complexity inherent in having > a > > > daemon, but how do you imagine "queue on a lock" to work? We don't have > > > anything like th

Re: notmuch's idea of concurrency / failing an invocation

2011-01-28 Thread Austin Clements
On Fri, Jan 28, 2011 at 4:45 AM, Thomas Schwinge wrote: > On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth wrote: > > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements > wrote: > > > I'm looking into breaking notmuch new up into small transactions. It > > > wouldn't be much a leap from there to

Re: notmuch's idea of concurrency / failing an invocation

2011-01-28 Thread Mike Kelly
On Fri, 28 Jan 2011 10:45:19 +0100, Thomas Schwinge wrote: > > It would definitely be nice to avoid the complexity inherent in having a > > daemon, but how do you imagine "queue on a lock" to work? We don't have > > anything like that in place now. > > I suppose what he means is trying to get the

Re: notmuch's idea of concurrency / failing an invocation

2011-01-28 Thread Thomas Schwinge
Hallo! On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth wrote: > On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements wrote: > > I'm looking into breaking notmuch new up into small transactions. It > > wouldn't be much a leap from there to simply close and reopen the database > > between transacti

Re: notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Carl Worth
On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge wrote: > Which is the original idea here? Is it that... There's no original idea yet. It's essentially an unsolved problem right now. > * each and every client should catch these kinds of errors, and retry, > or eventually give up at so

Re: notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Carl Worth
On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements wrote: > I'm looking into breaking notmuch new up into small transactions. It > wouldn't be much a leap from there to simply close and reopen the database > between transactions if another task wants to use it, which would release > the lock and

Re: notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Austin Clements
I'm looking into breaking notmuch new up into small transactions. It wouldn't be much a leap from there to simply close and reopen the database between transactions if another task wants to use it, which would release the lock and let the queued notmuch task have the database for a bit. It seems

Re: notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread Jameson Rollins
On Thu, 27 Jan 2011 13:40:25 -0500, micah anderson wrote: > Due to my harddisk in my laptop being slow (5400RPM), my notmuch > database growing, and perhaps some fragmentation somewhere, this has > become *incredibly* annoying for me. I am checking email every 30 > minutes, and I'm nicing and ioni

Re: notmuch's idea of concurrency / failing an invocation

2011-01-27 Thread micah anderson
On Thu, 27 Jan 2011 19:20:00 +0100, Thomas Schwinge wrote: > Stepping away from the current code base -- what is notmuch's original > idea of concurrency? That is, all of us probably know that one: > > A Xapian exception occurred opening database: Unable to get write > lock on /home/t