[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-30 Thread Sebastian Spaeth
On Thu, 29 Sep 2011 11:01:47 -0400, Austin Clements  wrote:
> Quoth Sebastian Spaeth on Sep 28 at  6:36 pm:
> > db.find_message_by_filename("moo")
> > Internal error: Failure to ensure database is writable
> > (lib/directory.cc:100).

> It appears that looking up a directory requires a writable database
> because notmuch will try to *create* a database document for the
> directory if one doesn't already exist.  This is clearly wrong
> behavior for a "find" function.

First of all, I consider libnotmuch exiting, taking python down, an
outright bug. So we should modify this case to return:
NOTMUCH_STATUS_READ_ONLY_DATABASE

so I can at least give some sensible error than having an angry mob run
with pitchforks towards me.

Ideally, we don't need READ-WRITE dbs for the find :-)

As it is, the same issue happens with the
notmuch_database_get_directory call, which should be protected in the
same manner (and probably has the same root cause). It is debatable
whether get_directory should be creating a directory on demand, I guess,
but it should never crash.

I have documented this in the python bindings, so it's not super urgent.
But it's dangerous and wrong nonetheless.

Sebastian
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 



Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-30 Thread Sebastian Spaeth
On Thu, 29 Sep 2011 11:01:47 -0400, Austin Clements amdra...@mit.edu wrote:
 Quoth Sebastian Spaeth on Sep 28 at  6:36 pm:
  db.find_message_by_filename(moo)
  Internal error: Failure to ensure database is writable
  (lib/directory.cc:100).

 It appears that looking up a directory requires a writable database
 because notmuch will try to *create* a database document for the
 directory if one doesn't already exist.  This is clearly wrong
 behavior for a find function.

First of all, I consider libnotmuch exiting, taking python down, an
outright bug. So we should modify this case to return:
NOTMUCH_STATUS_READ_ONLY_DATABASE

so I can at least give some sensible error than having an angry mob run
with pitchforks towards me.

Ideally, we don't need READ-WRITE dbs for the find :-)

As it is, the same issue happens with the
notmuch_database_get_directory call, which should be protected in the
same manner (and probably has the same root cause). It is debatable
whether get_directory should be creating a directory on demand, I guess,
but it should never crash.

I have documented this in the python bindings, so it's not super urgent.
But it's dangerous and wrong nonetheless.

Sebastian


pgpB73zrYUUlp.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-29 Thread Austin Clements
Quoth Sebastian Spaeth on Sep 28 at  6:36 pm:
> On Sat, 11 Jun 2011 16:04:26 -0400, Austin Clements  
> wrote:
> > Here's the reworked patch series that uses atomic sections more
> > heavily rather than changing the removal API.  This is atomic-new-v6
> > on http://awakening.csail.mit.edu/git/notmuch.git .
> 
> I just caught up implementing find_message_by_filename and
> begin|end_atomic
> 
> One oddity, since databases are opened in read-only by default, I was
> surprise to see find_message_by_filename on such a database have my
> python instance crash...
> 
> > lib: Add an API to find a message by filename.
> >   Culled from "lib: Add API's to find by filename and ..." in the old
> >   series.  What I kept is identical.
> 
> db.find_message_by_filename("moo")
> Internal error: Failure to ensure database is writable
> (lib/directory.cc:100).
> 
> Outch?

Oof.

It appears that looking up a directory requires a writable database
because notmuch will try to *create* a database document for the
directory if one doesn't already exist.  This is clearly wrong
behavior for a "find" function.

The exact code path is
  notmuch_database_find_message_by_filename
  _notmuch_database_filename_to_direntry
  _notmuch_database_find_directory_id
  _notmuch_directory_create

_notmuch_message_add_filename currently depends on
_notmuch_database_filename_to_direntry to create the directory
document if it doesn't exist.  Possibly
_notmuch_database_filename_to_direntry,
_notmuch_database_find_directory_id, and _notmuch_directory_create
should acquire a flags argument with a "create" flag that controls
this behavior.


Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-29 Thread Austin Clements
Quoth Sebastian Spaeth on Sep 28 at  6:36 pm:
 On Sat, 11 Jun 2011 16:04:26 -0400, Austin Clements amdra...@mit.edu wrote:
  Here's the reworked patch series that uses atomic sections more
  heavily rather than changing the removal API.  This is atomic-new-v6
  on http://awakening.csail.mit.edu/git/notmuch.git .
 
 I just caught up implementing find_message_by_filename and
 begin|end_atomic
 
 One oddity, since databases are opened in read-only by default, I was
 surprise to see find_message_by_filename on such a database have my
 python instance crash...
 
  lib: Add an API to find a message by filename.
Culled from lib: Add API's to find by filename and ... in the old
series.  What I kept is identical.
 
 db.find_message_by_filename(moo)
 Internal error: Failure to ensure database is writable
 (lib/directory.cc:100).
 
 Outch?

Oof.

It appears that looking up a directory requires a writable database
because notmuch will try to *create* a database document for the
directory if one doesn't already exist.  This is clearly wrong
behavior for a find function.

The exact code path is
  notmuch_database_find_message_by_filename
  _notmuch_database_filename_to_direntry
  _notmuch_database_find_directory_id
  _notmuch_directory_create

_notmuch_message_add_filename currently depends on
_notmuch_database_filename_to_direntry to create the directory
document if it doesn't exist.  Possibly
_notmuch_database_filename_to_direntry,
_notmuch_database_find_directory_id, and _notmuch_directory_create
should acquire a flags argument with a create flag that controls
this behavior.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-28 Thread Sebastian Spaeth
On Sat, 11 Jun 2011 16:04:26 -0400, Austin Clements  wrote:
> Here's the reworked patch series that uses atomic sections more
> heavily rather than changing the removal API.  This is atomic-new-v6
> on http://awakening.csail.mit.edu/git/notmuch.git .

I just caught up implementing find_message_by_filename and
begin|end_atomic

One oddity, since databases are opened in read-only by default, I was
surprise to see find_message_by_filename on such a database have my
python instance crash...

> lib: Add an API to find a message by filename.
>   Culled from "lib: Add API's to find by filename and ..." in the old
>   series.  What I kept is identical.

db.find_message_by_filename("moo")
Internal error: Failure to ensure database is writable
(lib/directory.cc:100).

Outch?
Sebastian
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: 



[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-26 Thread Austin Clements
Quoth David Bremner on Sep 24 at 11:36 pm:
> On Sat, 24 Sep 2011 00:03:02 -0400, Austin Clements  
> wrote:
> > 
> > Awesome.  Only seven more to go!
> 
> The remaining seven are pushed, along with some related debian packaging
> things.

Huzzah!

> Austin, could I bug you for some atomicity related items for NEWS? I
> guess at least the 3 new library calls should be mentioned.

In fact, I've been looking forward to writing some NEWS items!


Correct handling of interruptions during "notmuch new"

  "notmuch new" now operates as a series of small, self-consistent
  transactions, so it can correctly resume after an interruption or
  crash.  Previously, interruption could lose existing tags, fail to
  detect messages on resume, or leave the database in a state
  temporarily or permanently inconsistent with the mail store.

Library changes
---

New functions

  notmuch_database_begin_atomic and notmuch_database_end_atomic allow
  multiple database operations to be performed atomically.

  notmuch_database_find_message_by_filename does exactly what it says.


Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-26 Thread Austin Clements
Quoth David Bremner on Sep 24 at 11:36 pm:
 On Sat, 24 Sep 2011 00:03:02 -0400, Austin Clements amdra...@mit.edu wrote:
  
  Awesome.  Only seven more to go!
 
 The remaining seven are pushed, along with some related debian packaging
 things.

Huzzah!

 Austin, could I bug you for some atomicity related items for NEWS? I
 guess at least the 3 new library calls should be mentioned.

In fact, I've been looking forward to writing some NEWS items!


Correct handling of interruptions during notmuch new

  notmuch new now operates as a series of small, self-consistent
  transactions, so it can correctly resume after an interruption or
  crash.  Previously, interruption could lose existing tags, fail to
  detect messages on resume, or leave the database in a state
  temporarily or permanently inconsistent with the mail store.

Library changes
---

New functions

  notmuch_database_begin_atomic and notmuch_database_end_atomic allow
  multiple database operations to be performed atomically.

  notmuch_database_find_message_by_filename does exactly what it says.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-25 Thread David Bremner
On Sat, 24 Sep 2011 00:03:02 -0400, Austin Clements  wrote:
> 
> Awesome.  Only seven more to go!

The remaining seven are pushed, along with some related debian packaging
things.

Austin, could I bug you for some atomicity related items for NEWS? I
guess at least the 3 new library calls should be mentioned.

d


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-24 Thread Austin Clements
Quoth David Bremner on Sep 23 at 11:14 pm:
> On Tue, 13 Sep 2011 09:39:15 -0300, David Bremner  
> wrote:
> > On Fri, 8 Jul 2011 14:09:16 -0400, Austin Clements  
> > wrote:
> > 
> > I have rebased these yet again, and pushed the first 4 commits to
> > master. So far this is mainly testing stuff, along with one oneline
> > bugfix. I also added a couple testing related commits:
> > 
> > 73ed66a test: use test_expect_equal_file in atomicity
> > 05a522c test: Convert atomicity test to use 
> > test_subtest_known_broken
> 
> I have pushed 5 more commits in the sequence, without changes.  

Awesome.  Only seven more to go!


Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-24 Thread David Bremner
On Sat, 24 Sep 2011 00:03:02 -0400, Austin Clements amdra...@mit.edu wrote:
 
 Awesome.  Only seven more to go!

The remaining seven are pushed, along with some related debian packaging
things.

Austin, could I bug you for some atomicity related items for NEWS? I
guess at least the 3 new library calls should be mentioned.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-23 Thread David Bremner
On Tue, 13 Sep 2011 09:39:15 -0300, David Bremner da...@tethera.net wrote:
 On Fri, 8 Jul 2011 14:09:16 -0400, Austin Clements amdra...@mit.edu wrote:
 
 I have rebased these yet again, and pushed the first 4 commits to
 master. So far this is mainly testing stuff, along with one oneline
 bugfix. I also added a couple testing related commits:
 
 73ed66a test: use test_expect_equal_file in atomicity
 05a522c test: Convert atomicity test to use test_subtest_known_broken

I have pushed 5 more commits in the sequence, without changes.  

d

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-23 Thread Austin Clements
Quoth David Bremner on Sep 23 at 11:14 pm:
 On Tue, 13 Sep 2011 09:39:15 -0300, David Bremner da...@tethera.net wrote:
  On Fri, 8 Jul 2011 14:09:16 -0400, Austin Clements amdra...@mit.edu wrote:
  
  I have rebased these yet again, and pushed the first 4 commits to
  master. So far this is mainly testing stuff, along with one oneline
  bugfix. I also added a couple testing related commits:
  
  73ed66a test: use test_expect_equal_file in atomicity
  05a522c test: Convert atomicity test to use 
  test_subtest_known_broken
 
 I have pushed 5 more commits in the sequence, without changes.  

Awesome.  Only seven more to go!
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-09-13 Thread David Bremner
On Fri, 8 Jul 2011 14:09:16 -0400, Austin Clements  wrote:
> Rebased against current master and now atomic-new-v7 on
>   http://awakening.csail.mit.edu/git/notmuch.git
> There were a handful of conflicts, but nothing substantive.
> 

I have rebased these yet again, and pushed the first 4 commits to
master. So far this is mainly testing stuff, along with one oneline
bugfix. I also added a couple testing related commits:

73ed66a test: use test_expect_equal_file in atomicity
05a522c test: Convert atomicity test to use test_subtest_known_broken

d


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-07-08 Thread Austin Clements
Rebased against current master and now atomic-new-v7 on
  http://awakening.csail.mit.edu/git/notmuch.git
There were a handful of conflicts, but nothing substantive.

On Wed, Jun 22, 2011 at 3:39 PM, Austin Clements  wrote:
> Ping.
>
> On Sat, Jun 11, 2011 at 4:04 PM, Austin Clements  wrote:
>> Here's the reworked patch series that uses atomic sections more
>> heavily rather than changing the removal API. ?This is atomic-new-v6
>> on http://awakening.csail.mit.edu/git/notmuch.git .
>>
>> (I was planning to make this series on Monday while stuck on a plane,
>> but an opportunity presented itself when I needed something better to
>> do than fix the sink. ?As a result, if you can look over this before
>> Monday, I can use that time to address any further comments.)
>>
>> The beginning of the sequence---"test: Fix message when skipping
>> test_expect_equal* tests" through "new: Defer updating directory
>> mtimes until the end."---has not changed besides than the minor things
>> you pointed out earlier. ?Here's a quick summary of the differences in
>> the rest of the sequence:
>>
>> lib: Add notmuch_database_{begin,end}_atomic.
>> ?Rebased but otherwise identical.
>>
>> lib: Add support for nested atomic sections.
>> ?New.
>>
>> lib: Indicate if there are more filenames after removal.
>> ?Rebased but otherwise identical.
>>
>> lib: Remove message document directly after removing the last file
>> name.
>> ?New. ?Supersedes the patches that rewrote
>> ?notmuch_database_remove_message and made sync delete messages.
>>
>> lib: Add an API to find a message by filename.
>> ?Culled from "lib: Add API's to find by filename and ..." in the old
>> ?series. ?What I kept is identical.
>>
>> lib: Wrap notmuch_database_add_message in an atomic section.
>> ?New.
>>
>> new: Cleanup. Put removed/renamed message count in add_files_state_t.
>> new: Cleanup. De-duplicate file name removal code.
>> ?Both new, but closely related to previous remove_filename patch.
>>
>> new: Synchronize maildir flags eagerly.
>> ?New, but very closely related to previous patch of the same name.
>> ?This patch is simpler now because of the previous two cleanups.
>>
>> new: Wrap adding and removing messages in atomic sections.
>> ?Rebased and added an atomic section around removal.
>>
>> lib: Improve notmuch_database_{add,remove}_message documentation.
>> ?New.
>>
>> Ultimately, I still think the existing removal API is weird because it
>> forces you to do things like lookup a message by file name just in
>> case you need that message object after removal tells you that it
>> didn't actually remove the message. ?But I don't know what to do about
>> it that isn't equally weird.
>>
>> ___
>> notmuch mailing list
>> notmuch at notmuchmail.org
>> http://notmuchmail.org/mailman/listinfo/notmuch
>>
>


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-06-29 Thread Austin Clements
Just found one more atomicity bug in notmuch-new.  I would prefer to
not update this patch series yet again, but rather to follow up with a
separate fix for this.  The full bug is described below, but the gist
is that how add_files_recursive computes new_directory from the mtime
isn't sound.  The simplest fix is to remove the new_directory
optimization, but then we wouldn't scan files in inode order.  The
best fix is probably to add an out-argument to
notmuch_database_get_directory that indicates if the directory really
was just created in the database (and hence has no files or subdirs).
Unfortunately, that requires an API change.  If that's undesirable, an
alternate would be to record that bit in the notmuch_directory_t and
add some notmuch_directory_is_new API that returns it.  Thoughts?


Bug description:

add_files_recursive determines whether a directory is "new" by
inspecting the database mtime returned by notmuch_directory_get_mtime.
 However, if there's an interruption after adding
messages/subdirectories from a new directory but before updating the
directory's mtime, it will be considered a new directory again on the
next run.  As a result, on the next run, db_files and db_subdirs will
not be loaded from the database (even though they *wouldn't* be NULL).
 As a result, we'll miss removing messages or entire subdirectories
that have been deleted from the "new" directory.  (We'll also re-add
the messages in this directory to the database rather than skipping
them, which will throw off the notmuch new statistics, but that's
fine.)

This didn't show up in the atomicity test because throwing off
anything besides the statistics requires *two* interruptions.  In
fact, I don't even know how I would write an automated test for this.


Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-06-29 Thread Austin Clements
Just found one more atomicity bug in notmuch-new.  I would prefer to
not update this patch series yet again, but rather to follow up with a
separate fix for this.  The full bug is described below, but the gist
is that how add_files_recursive computes new_directory from the mtime
isn't sound.  The simplest fix is to remove the new_directory
optimization, but then we wouldn't scan files in inode order.  The
best fix is probably to add an out-argument to
notmuch_database_get_directory that indicates if the directory really
was just created in the database (and hence has no files or subdirs).
Unfortunately, that requires an API change.  If that's undesirable, an
alternate would be to record that bit in the notmuch_directory_t and
add some notmuch_directory_is_new API that returns it.  Thoughts?


Bug description:

add_files_recursive determines whether a directory is new by
inspecting the database mtime returned by notmuch_directory_get_mtime.
 However, if there's an interruption after adding
messages/subdirectories from a new directory but before updating the
directory's mtime, it will be considered a new directory again on the
next run.  As a result, on the next run, db_files and db_subdirs will
not be loaded from the database (even though they *wouldn't* be NULL).
 As a result, we'll miss removing messages or entire subdirectories
that have been deleted from the new directory.  (We'll also re-add
the messages in this directory to the database rather than skipping
them, which will throw off the notmuch new statistics, but that's
fine.)

This didn't show up in the atomicity test because throwing off
anything besides the statistics requires *two* interruptions.  In
fact, I don't even know how I would write an automated test for this.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-06-22 Thread Austin Clements
Ping.

On Sat, Jun 11, 2011 at 4:04 PM, Austin Clements  wrote:
> Here's the reworked patch series that uses atomic sections more
> heavily rather than changing the removal API. ?This is atomic-new-v6
> on http://awakening.csail.mit.edu/git/notmuch.git .
>
> (I was planning to make this series on Monday while stuck on a plane,
> but an opportunity presented itself when I needed something better to
> do than fix the sink. ?As a result, if you can look over this before
> Monday, I can use that time to address any further comments.)
>
> The beginning of the sequence---"test: Fix message when skipping
> test_expect_equal* tests" through "new: Defer updating directory
> mtimes until the end."---has not changed besides than the minor things
> you pointed out earlier. ?Here's a quick summary of the differences in
> the rest of the sequence:
>
> lib: Add notmuch_database_{begin,end}_atomic.
> ?Rebased but otherwise identical.
>
> lib: Add support for nested atomic sections.
> ?New.
>
> lib: Indicate if there are more filenames after removal.
> ?Rebased but otherwise identical.
>
> lib: Remove message document directly after removing the last file
> name.
> ?New. ?Supersedes the patches that rewrote
> ?notmuch_database_remove_message and made sync delete messages.
>
> lib: Add an API to find a message by filename.
> ?Culled from "lib: Add API's to find by filename and ..." in the old
> ?series. ?What I kept is identical.
>
> lib: Wrap notmuch_database_add_message in an atomic section.
> ?New.
>
> new: Cleanup. Put removed/renamed message count in add_files_state_t.
> new: Cleanup. De-duplicate file name removal code.
> ?Both new, but closely related to previous remove_filename patch.
>
> new: Synchronize maildir flags eagerly.
> ?New, but very closely related to previous patch of the same name.
> ?This patch is simpler now because of the previous two cleanups.
>
> new: Wrap adding and removing messages in atomic sections.
> ?Rebased and added an atomic section around removal.
>
> lib: Improve notmuch_database_{add,remove}_message documentation.
> ?New.
>
> Ultimately, I still think the existing removal API is weird because it
> forces you to do things like lookup a message by file name just in
> case you need that message object after removal tells you that it
> didn't actually remove the message. ?But I don't know what to do about
> it that isn't equally weird.
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>


Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-06-22 Thread Austin Clements
Ping.

On Sat, Jun 11, 2011 at 4:04 PM, Austin Clements amdra...@mit.edu wrote:
 Here's the reworked patch series that uses atomic sections more
 heavily rather than changing the removal API.  This is atomic-new-v6
 on http://awakening.csail.mit.edu/git/notmuch.git .

 (I was planning to make this series on Monday while stuck on a plane,
 but an opportunity presented itself when I needed something better to
 do than fix the sink.  As a result, if you can look over this before
 Monday, I can use that time to address any further comments.)

 The beginning of the sequence---test: Fix message when skipping
 test_expect_equal* tests through new: Defer updating directory
 mtimes until the end.---has not changed besides than the minor things
 you pointed out earlier.  Here's a quick summary of the differences in
 the rest of the sequence:

 lib: Add notmuch_database_{begin,end}_atomic.
  Rebased but otherwise identical.

 lib: Add support for nested atomic sections.
  New.

 lib: Indicate if there are more filenames after removal.
  Rebased but otherwise identical.

 lib: Remove message document directly after removing the last file
 name.
  New.  Supersedes the patches that rewrote
  notmuch_database_remove_message and made sync delete messages.

 lib: Add an API to find a message by filename.
  Culled from lib: Add API's to find by filename and ... in the old
  series.  What I kept is identical.

 lib: Wrap notmuch_database_add_message in an atomic section.
  New.

 new: Cleanup. Put removed/renamed message count in add_files_state_t.
 new: Cleanup. De-duplicate file name removal code.
  Both new, but closely related to previous remove_filename patch.

 new: Synchronize maildir flags eagerly.
  New, but very closely related to previous patch of the same name.
  This patch is simpler now because of the previous two cleanups.

 new: Wrap adding and removing messages in atomic sections.
  Rebased and added an atomic section around removal.

 lib: Improve notmuch_database_{add,remove}_message documentation.
  New.

 Ultimately, I still think the existing removal API is weird because it
 forces you to do things like lookup a message by file name just in
 case you need that message object after removal tells you that it
 didn't actually remove the message.  But I don't know what to do about
 it that isn't equally weird.

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-06-11 Thread Austin Clements
On Sat, Jun 11, 2011 at 4:04 PM, Austin Clements  wrote:
> Here's the reworked patch series that uses atomic sections more
> heavily rather than changing the removal API. ?This is atomic-new-v6
> on http://awakening.csail.mit.edu/git/notmuch.git .

(Patch 01/17 is out of date order.  My mail queue hiccuped, so I had
to resend it.)


[PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-06-11 Thread Austin Clements
Here's the reworked patch series that uses atomic sections more
heavily rather than changing the removal API.  This is atomic-new-v6
on http://awakening.csail.mit.edu/git/notmuch.git .

(I was planning to make this series on Monday while stuck on a plane,
but an opportunity presented itself when I needed something better to
do than fix the sink.  As a result, if you can look over this before
Monday, I can use that time to address any further comments.)

The beginning of the sequence---"test: Fix message when skipping
test_expect_equal* tests" through "new: Defer updating directory
mtimes until the end."---has not changed besides than the minor things
you pointed out earlier.  Here's a quick summary of the differences in
the rest of the sequence:

lib: Add notmuch_database_{begin,end}_atomic.
  Rebased but otherwise identical.

lib: Add support for nested atomic sections.
  New.

lib: Indicate if there are more filenames after removal.
  Rebased but otherwise identical.

lib: Remove message document directly after removing the last file
name.
  New.  Supersedes the patches that rewrote
  notmuch_database_remove_message and made sync delete messages.

lib: Add an API to find a message by filename.
  Culled from "lib: Add API's to find by filename and ..." in the old
  series.  What I kept is identical.

lib: Wrap notmuch_database_add_message in an atomic section.
  New.

new: Cleanup. Put removed/renamed message count in add_files_state_t.
new: Cleanup. De-duplicate file name removal code.
  Both new, but closely related to previous remove_filename patch.

new: Synchronize maildir flags eagerly.
  New, but very closely related to previous patch of the same name.
  This patch is simpler now because of the previous two cleanups.

new: Wrap adding and removing messages in atomic sections.
  Rebased and added an atomic section around removal.

lib: Improve notmuch_database_{add,remove}_message documentation.
  New.

Ultimately, I still think the existing removal API is weird because it
forces you to do things like lookup a message by file name just in
case you need that message object after removal tells you that it
didn't actually remove the message.  But I don't know what to do about
it that isn't equally weird.



Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues

2011-06-11 Thread Austin Clements
On Sat, Jun 11, 2011 at 4:04 PM, Austin Clements amdra...@mit.edu wrote:
 Here's the reworked patch series that uses atomic sections more
 heavily rather than changing the removal API.  This is atomic-new-v6
 on http://awakening.csail.mit.edu/git/notmuch.git .

(Patch 01/17 is out of date order.  My mail queue hiccuped, so I had
to resend it.)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch