notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 13 at  3:32 pm:
> On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements  wrote:
> > There should be no way to corrupt the database at this level through
> > the Xapian API, which means nothing libnotmuch can do (much less users
> > of libnotmuch) should be able to corrupt the database.  If you can
> > reproduce the problem, it's probably a serious bug in Xapian, but it
> > could also have been a file system bug or even random file system
> > corruption.
> 
> Well that's... troubling.
> 
> Patrick: could you please backup and try to reproduce? Otherwise I'll
> assume this was a one-off situation.
> 
> 
> Austin-- think you could do a quick review of the script to double
> check and confirm I'm not doing anything nefarious?
> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

In theory the only way you could cause corruption besides tickling a
bug would be to access the same database object concurrently from
different threads (since it's not thread-safe), but you don't appear
to be doing that.

I did spot something that could corrupt delivered email, though.  The
way you deliver to the Maildir is resilient to process termination,
but not to system failures such as power outages.  In particular, you
need to at least os.fsync before the os.link.  I'd recommend looking
at Python's mailbox module, which has a robust Maildir delivery
implementation (though it appears it doesn't let you control the file
name, so you probably can't use it directly).

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Sat, Dec 15, 2012 at 7:18 AM, Austin Clements amdra...@mit.edu wrote:

 need to at least os.fsync before the os.link.


Fixed, thanks for the suggestion.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3f9646058bfd91d7d0e2eda035521f97de92eabc
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 13 at  3:32 pm:
 On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements amdra...@mit.edu wrote:
  There should be no way to corrupt the database at this level through
  the Xapian API, which means nothing libnotmuch can do (much less users
  of libnotmuch) should be able to corrupt the database.  If you can
  reproduce the problem, it's probably a serious bug in Xapian, but it
  could also have been a file system bug or even random file system
  corruption.
 
 Well that's... troubling.
 
 Patrick: could you please backup and try to reproduce? Otherwise I'll
 assume this was a one-off situation.
 
 
 Austin-- think you could do a quick review of the script to double
 check and confirm I'm not doing anything nefarious?
 http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

In theory the only way you could cause corruption besides tickling a
bug would be to access the same database object concurrently from
different threads (since it's not thread-safe), but you don't appear
to be doing that.

I did spot something that could corrupt delivered email, though.  The
way you deliver to the Maildir is resilient to process termination,
but not to system failures such as power outages.  In particular, you
need to at least os.fsync before the os.link.  I'd recommend looking
at Python's mailbox module, which has a robust Maildir delivery
implementation (though it appears it doesn't let you control the file
name, so you probably can't use it directly).
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-13 14:32:53)
> On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements  wrote:
> > There should be no way to corrupt the database at this level through
> > the Xapian API, which means nothing libnotmuch can do (much less users
> > of libnotmuch) should be able to corrupt the database.  If you can
> > reproduce the problem, it's probably a serious bug in Xapian, but it
> > could also have been a file system bug or even random file system
> > corruption.
> 
> Well that's... troubling.
> 
> Patrick: could you please backup and try to reproduce? Otherwise I'll
> assume this was a one-off situation.


I tried it again, concurrently triggered some modifications
and killed the process afer a while.
(my modifications were not flushed during the run of your script as
the index was locked)
The index seems to be OK. So I'm afraid I cannot reproduce this reliably.
/p

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 10:26 PM, David Bremner  wrote:
>
> One thing that Olly Betts mentioned is that there is a recently fixed
> bug in xapian with respect to multiple threads in the same process

The script's actually single threaded.

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements  wrote:
> There should be no way to corrupt the database at this level through
> the Xapian API, which means nothing libnotmuch can do (much less users
> of libnotmuch) should be able to corrupt the database.  If you can
> reproduce the problem, it's probably a serious bug in Xapian, but it
> could also have been a file system bug or even random file system
> corruption.

Well that's... troubling.

Patrick: could you please backup and try to reproduce? Otherwise I'll
assume this was a one-off situation.


Austin-- think you could do a quick review of the script to double
check and confirm I'm not doing anything nefarious?
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 10:26 PM, David Bremner da...@tethera.net wrote:

 One thing that Olly Betts mentioned is that there is a recently fixed
 bug in xapian with respect to multiple threads in the same process

The script's actually single threaded.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-13 14:32:53)
 On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements amdra...@mit.edu wrote:
  There should be no way to corrupt the database at this level through
  the Xapian API, which means nothing libnotmuch can do (much less users
  of libnotmuch) should be able to corrupt the database.  If you can
  reproduce the problem, it's probably a serious bug in Xapian, but it
  could also have been a file system bug or even random file system
  corruption.
 
 Well that's... troubling.
 
 Patrick: could you please backup and try to reproduce? Otherwise I'll
 assume this was a one-off situation.


I tried it again, concurrently triggered some modifications
and killed the process afer a while.
(my modifications were not flushed during the run of your script as
the index was locked)
The index seems to be OK. So I'm afraid I cannot reproduce this reliably.
/p


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke  
wrote:
> OK, i assumed i'd not have to download all messages because i already have 
> them
> on disk -- synced via offlineimap. but then i remembered that OI stores uses
> its own naming scheme for the files in a maildir. so thats probably why your 
> script
> did not realize theyre already there. (maybe it should? it could look up the 
> MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being "${gmail-unique-number}.gmail",
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

> i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

> All i did was start your script, writing to the index in another terminal in 
> between (using alot)
> possibly my cronjob fired a "notmuch new; afew.." in between also.
> when I killed your script the index war broken.
>
> Can i turn on some global logging for notmuch to debug this?
> Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.

notmuch python bindings corrupt db index (was: gmail importer script)

[David Bremner]

Austin Clements  writes:

>
> There should be no way to corrupt the database at this level through
> the Xapian API, which means nothing libnotmuch can do (much less users
> of libnotmuch) should be able to corrupt the database.  If you can
> reproduce the problem, it's probably a serious bug in Xapian, but it
> could also have been a file system bug or even random file system
> corruption.

One thing that Olly Betts mentioned is that there is a recently fixed
bug in xapian with respect to multiple threads in the same process

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695542

I don't _think_ this is relevant here, but I mention it for
completeness.

d

notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 12 at  7:35 pm:
> On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke  
> wrote:
> > i am kind of reluctant to reproduce this again (on my work-desktop).
> 
> It'd really be a huge help if you could backup and try...
> 
> > All i did was start your script, writing to the index in another terminal 
> > in between (using alot)
> > possibly my cronjob fired a "notmuch new; afew.." in between also.
> > when I killed your script the index war broken.
> >
> > Can i turn on some global logging for notmuch to debug this?
> > Maybe libnotmuch had trouble with so many db-writes at a time?
> 
> That is very wrong and bizarre. I open the database in read/write
> mode, which is supposed to lock other programs from accessing it in
> this way... Very strange.

There should be no way to corrupt the database at this level through
the Xapian API, which means nothing libnotmuch can do (much less users
of libnotmuch) should be able to corrupt the database.  If you can
reproduce the problem, it's probably a serious bug in Xapian, but it
could also have been a file system bug or even random file system
corruption.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke patricktot...@gmail.com wrote:
 OK, i assumed i'd not have to download all messages because i already have 
 them
 on disk -- synced via offlineimap. but then i remembered that OI stores uses
 its own naming scheme for the files in a maildir. so thats probably why your 
 script
 did not realize theyre already there. (maybe it should? it could look up the 
 MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being ${gmail-unique-number}.gmail,
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

 i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

 All i did was start your script, writing to the index in another terminal in 
 between (using alot)
 possibly my cronjob fired a notmuch new; afew.. in between also.
 when I killed your script the index war broken.

 Can i turn on some global logging for notmuch to debug this?
 Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 12 at  7:35 pm:
 On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke patricktot...@gmail.com 
 wrote:
  i am kind of reluctant to reproduce this again (on my work-desktop).
 
 It'd really be a huge help if you could backup and try...
 
  All i did was start your script, writing to the index in another terminal 
  in between (using alot)
  possibly my cronjob fired a notmuch new; afew.. in between also.
  when I killed your script the index war broken.
 
  Can i turn on some global logging for notmuch to debug this?
  Maybe libnotmuch had trouble with so many db-writes at a time?
 
 That is very wrong and bizarre. I open the database in read/write
 mode, which is supposed to lock other programs from accessing it in
 this way... Very strange.

There should be no way to corrupt the database at this level through
the Xapian API, which means nothing libnotmuch can do (much less users
of libnotmuch) should be able to corrupt the database.  If you can
reproduce the problem, it's probably a serious bug in Xapian, but it
could also have been a file system bug or even random file system
corruption.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[David Bremner]

Austin Clements amdra...@mit.edu writes:


 There should be no way to corrupt the database at this level through
 the Xapian API, which means nothing libnotmuch can do (much less users
 of libnotmuch) should be able to corrupt the database.  If you can
 reproduce the problem, it's probably a serious bug in Xapian, but it
 could also have been a file system bug or even random file system
 corruption.

One thing that Olly Betts mentioned is that there is a recently fixed
bug in xapian with respect to multiple threads in the same process

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695542

I don't _think_ this is relevant here, but I mention it for
completeness.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 13:47:32)
> On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
>  wrote:
> > I ended up killing the process after i saw that ETA was >3h.
> 
> Gmail's bottleneck. Very unfortunate. Leave it going over night.

OK, i assumed i'd not have to download all messages because i already have them
on disk -- synced via offlineimap. but then i remembered that OI stores uses
its own naming scheme for the files in a maildir. so thats probably why your 
script
did not realize theyre already there. (maybe it should? it could look up the 
MIDs from notmuch)

> > This broke my index!
> >
> > afterwards, a `notmuch new` printed
> >
> > A Xapian exception occurred creating a directory: Expected block
> > 485 to be level 1, not 0.
> >
> > I wasnt able to properly read the index anymore and had to
> > restore an old index dump. not cool!
> 
> 
> Can you provide any more information about this? The script properly
> uses atomic transactions. Any corruption here is the result of a
> notmuch library bug, which should be investigated.

i am kind of reluctant to reproduce this again (on my work-desktop).
All i did was start your script, writing to the index in another terminal in 
between (using alot)
possibly my cronjob fired a "notmuch new; afew.." in between also.
when I killed your script the index war broken.

Can i turn on some global logging for notmuch to debug this?
Maybe libnotmuch had trouble with so many db-writes at a time?

/p


-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: signature
URL: 

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
 wrote:
> I ended up killing the process after i saw that ETA was >3h.

Gmail's bottleneck. Very unfortunate. Leave it going over night.

> This broke my index!
>
> afterwards, a `notmuch new` printed
>
> A Xapian exception occurred creating a directory: Expected block
> 485 to be level 1, not 0.
>
> I wasnt able to properly read the index anymore and had to
> restore an old index dump. not cool!


Can you provide any more information about this? The script properly
uses atomic transactions. Any corruption here is the result of a
notmuch library bug, which should be investigated.

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
patricktot...@gmail.com wrote:
 I ended up killing the process after i saw that ETA was 3h.

Gmail's bottleneck. Very unfortunate. Leave it going over night.

 This broke my index!

 afterwards, a `notmuch new` printed

 A Xapian exception occurred creating a directory: Expected block
 485 to be level 1, not 0.

 I wasnt able to properly read the index anymore and had to
 restore an old index dump. not cool!


Can you provide any more information about this? The script properly
uses atomic transactions. Any corruption here is the result of a
notmuch library bug, which should be investigated.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 13:47:32)
 On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
 patricktot...@gmail.com wrote:
  I ended up killing the process after i saw that ETA was 3h.
 
 Gmail's bottleneck. Very unfortunate. Leave it going over night.

OK, i assumed i'd not have to download all messages because i already have them
on disk -- synced via offlineimap. but then i remembered that OI stores uses
its own naming scheme for the files in a maildir. so thats probably why your 
script
did not realize theyre already there. (maybe it should? it could look up the 
MIDs from notmuch)

  This broke my index!
 
  afterwards, a `notmuch new` printed
 
  A Xapian exception occurred creating a directory: Expected block
  485 to be level 1, not 0.
 
  I wasnt able to properly read the index anymore and had to
  restore an old index dump. not cool!
 
 
 Can you provide any more information about this? The script properly
 uses atomic transactions. Any corruption here is the result of a
 notmuch library bug, which should be investigated.

i am kind of reluctant to reproduce this again (on my work-desktop).
All i did was start your script, writing to the index in another terminal in 
between (using alot)
possibly my cronjob fired a notmuch new; afew.. in between also.
when I killed your script the index war broken.

Can i turn on some global logging for notmuch to debug this?
Maybe libnotmuch had trouble with so many db-writes at a time?

/p




signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch