Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

David Bremner  writes:

> Gregor Zattler  writes:
>
>>
>> On my way reducing the corpus I had an intermediate corpus of
>> ~43000 files on which notmuch new produced a xapian exeption.  I
>> kept this corpus but did not include the test results in my email
>> because the later, smaller corpus seemed more interesting.
>>
>> I redid the test on this mid-sized corpus and I got an
>> exeption (again):
> [snip]
>> Error: A Xapian exception occurred. Halting processing.
>> Processed 27611 total files in 21m 2s (21 files/sec.).
>> Added 27481 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> [Inferior 1 (process 6752) exited with code 01]
>> (gdb) 
>
> I can finally replicate the crash. I do crash in a different place
> (after processing 25184 files), so it suggests a certain non-determinism
> or dependence on the host configuration.
>
> Anyway, now that I have a test case, I'll try and figure out what the
> underlying Xapian exeception is.

As far as I know this was dixed in xapian 1.4.7, so marking it fixed for
notmuch.

d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Leonard Lausen]

I run notmuch 0.27 and xapian 1.4.3 (from the Debian stretch repo).

Now that Xapian 1.4.6 is released (which should contain fixes to prevent
the database corruption that may occur due to this bug) I will upgrade
to it.

David Bremner  writes:
> While we're sorting out test data, what versions of notmuch and xapian
> are you running?
>
> d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Leonard Lausen  writes:

> I run notmuch 0.27 and xapian 1.4.3 (from the Debian stretch repo).
>
> Now that Xapian 1.4.6 is released (which should contain fixes to prevent
> the database corruption that may occur due to this bug) I will upgrade
> to it.
>

I believe the problem is a Xapian bug still present in 1.4.6. The
following patch (to Xapian) seems to fix it for me:

  https://oligarchy.co.uk/xapian/patches/glass-cursor-rebuild-fix.patch

If I understand correctly Olly will release Xapian 1.4.7 pretty soon
with this fix (and another one that affects the notmuch test suite).
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, Leonard,
* David Bremner  [2018-07-07; 23:07]:
> Gregor Zattler  writes:
>> * Leonard Lausen  [2018-07-01; 15:36]:
>>> Also it seems that the midsize corpus
>>> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
>>> available anymore.
>>
>> I still have this archive on my local hard drive.  If you (or
>> anyone else) are interested in this, I will upload it again.
>>
> Please do. I seem to have deleted my copy.

Done.  It's at
https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz
again.

Thanks for looking into this, Gregor


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Leonard Lausen  writes:

> I also experience this error trying to import the emacs-orgmode mailing
> list archive. Has there been any progress on fixing the issue? What
> steps should be taken to help?
>
> Also it seems that the midsize corpus
> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
> available anymore.
>
> David introduced a patch to improve the error message, below I paste the
> message I get:
>
> add_file: A Xapian exception occurred
> A Xapian exception occurred adding message: Unexpected end of posting list 
> for 'G000255c6'.
> Processed 4949 total files in 10m 13s (8 files/sec.).
> Added 4921 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred

While we're sorting out test data, what versions of notmuch and xapian
are you running?

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

> Hi Leonard, nomuch developers,
> * Leonard Lausen  [2018-07-01; 15:36]:
>> I also experience this error trying to import the emacs-orgmode mailing
>> list archive. Has there been any progress on fixing the issue? What
>> steps should be taken to help?
>>
>> Also it seems that the midsize corpus
>> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
>> available anymore.
>
> I still have this archive on my local hard drive.  If you (or
> anyone else) are interested in this, I will upload it again.
>

Please do. I seem to have deleted my copy.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi Leonard, nomuch developers,
* Leonard Lausen  [2018-07-01; 15:36]:
> I also experience this error trying to import the emacs-orgmode mailing
> list archive. Has there been any progress on fixing the issue? What
> steps should be taken to help?
>
> Also it seems that the midsize corpus
> https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
> available anymore.

I still have this archive on my local hard drive.  If you (or
anyone else) are interested in this, I will upload it again.

Ciao; Gregor 

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Leonard Lausen]

I also experience this error trying to import the emacs-orgmode mailing
list archive. Has there been any progress on fixing the issue? What
steps should be taken to help?

Also it seems that the midsize corpus
https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz is not
available anymore.

David introduced a patch to improve the error message, below I paste the
message I get:

add_file: A Xapian exception occurred
A Xapian exception occurred adding message: Unexpected end of posting list for 
'G000255c6'.
Processed 4949 total files in 10m 13s (8 files/sec.).
Added 4921 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred


Gregor Zattler  writes:

> Hi David,
> * David Bremner  [14. Jul. 2017]:
>> Gregor Zattler  writes:
>>> I redid the test on this mid-sized corpus and I got an
>>> exeption (again):
>> [snip]
>>> Error: A Xapian exception occurred. Halting processing.
>>> Processed 27611 total files in 21m 2s (21 files/sec.).
>>> Added 27481 new messages to the database.
>>> Note: A fatal error was encountered: A Xapian exception occurred
>>> [Inferior 1 (process 6752) exited with code 01]
>>> (gdb) 
>> 
>> I can finally replicate the crash. I do crash in a different place
>> (after processing 25184 files), so it suggests a certain non-determinism
>> or dependence on the host configuration.
>
> I would assume the first difference is in the file system:
> Perhaps the files are accessed in a different order!?
>
>> Anyway, now that I have a test case, I'll try and figure out what the
>> underlying Xapian exeception is.
>
>
> Thanks.
>
> Ciao, Gregor
> -- 
>  -... --- .-. . -.. ..--.. ...-.-
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner  [14. Jul. 2017]:
> Gregor Zattler  writes:
>> I redid the test on this mid-sized corpus and I got an
>> exeption (again):
> [snip]
>> Error: A Xapian exception occurred. Halting processing.
>> Processed 27611 total files in 21m 2s (21 files/sec.).
>> Added 27481 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> [Inferior 1 (process 6752) exited with code 01]
>> (gdb) 
> 
> I can finally replicate the crash. I do crash in a different place
> (after processing 25184 files), so it suggests a certain non-determinism
> or dependence on the host configuration.

I would assume the first difference is in the file system:
Perhaps the files are accessed in a different order!?

> Anyway, now that I have a test case, I'll try and figure out what the
> underlying Xapian exeception is.


Thanks.

Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

>
> On my way reducing the corpus I had an intermediate corpus of
> ~43000 files on which notmuch new produced a xapian exeption.  I
> kept this corpus but did not include the test results in my email
> because the later, smaller corpus seemed more interesting.
>
> I redid the test on this mid-sized corpus and I got an
> exeption (again):
[snip]
> Error: A Xapian exception occurred. Halting processing.
> Processed 27611 total files in 21m 2s (21 files/sec.).
> Added 27481 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 6752) exited with code 01]
> (gdb) 

I can finally replicate the crash. I do crash in a different place
(after processing 25184 files), so it suggests a certain non-determinism
or dependence on the host configuration.

Anyway, now that I have a test case, I'll try and figure out what the
underlying Xapian exeception is.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner  [13. Jul. 2017]:
>> * David Bremner  [13. Jul. 2017]:
>>> Gregor Zattler  writes:
 I then downloaded parts of the archive, merged and sliced it and
 now I have a sample of 25001 total files (that's not much mail),
 on which notmuch new produces a xapian exeption:
[...]
>>> For me this corpus does not produce an exception (although it does have
>>> one thread with 5067 messages).

>>> I'm running Debian testing, so there could be slightly different
>>> versions of xapian and gmime. What versions of libxapian-dev and
>>> libgmime-2.6-dev have you built notmuch against?
[...]
> I tried on a debian stretch with the same library versions, and I still
> can't duplicate the crash.

This is strange.  I rerun the self compiled notmuch new on this
corpus and it did not throw an exeption, nore did the debian
stretch binary.  I have no idea why this is different with the
first run.

On my way reducing the corpus I had an intermediate corpus of
~43000 files on which notmuch new produced a xapian exeption.  I
kept this corpus but did not include the test results in my email
because the later, smaller corpus seemed more interesting.

I redid the test on this mid-sized corpus and I got an
exeption (again):

0 (master *) grfz@len:~$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 43255 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x557d2820, 
format=0x5558d258 "A Xapian exception occurred adding message: %s.\n") at 
lib/database.cc:426
426 {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x557d2820, format=0x5558d258 "A 
Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x55577f58 in notmuch_database_add_message 
(notmuch=notmuch@entry=0x557d2820, filename=filename@entry=0x56fa4870 
"/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len",

  message_ret=message_ret@entry=0x7fffd2e8) at 
lib/database.cc:2597
#2  0x5556802f in add_file (state=0x7fffd560, 
filename=0x56fa4870 
"/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len", 
notmuch=0x557d2820) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x557d2820, 
path=path@entry=0x557d26e0 "/tmp/reduced-sample.xapian-exeption/new", 
state=state@entry=0x7fffd560) at notmuch-new.c:599
#4  0x55567b44 in add_files (notmuch=0x557d2820, 
path=path@entry=0x557cf730 "/tmp/reduced-sample.xapian-exeption", 
state=state@entry=0x7fffd560) at notmuch-new.c:483
#5  0x555689ed in notmuch_new_command (config=0x557ce1d0, 
argc=, argv=) at notmuch-new.c:1099
#6  0x55561a27 in main (argc=, argv=0x7fffda88) at 
notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 27611 total files in 21m 2s (21 files/sec.).
Added 27481 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 6752) exited with code 01]
(gdb) 



I redid the test with this medium sized corpus with the notmuch
binary from debian stretch, the xapian exeption hits after the
same number of scanned/added files:

0 (master *) grfz@len:~$ gdb --args /usr/bin/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other 

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

> Hi David,
> * David Bremner  [13. Jul. 2017]:
>> Gregor Zattler  writes:
>>> I then downloaded parts of the archive, merged and sliced it and
>>> now I have a sample of 25001 total files (that's not much mail),
>>> on which notmuch new produces a xapian exeption:
>>>
>> 
>>> As a tar.xz it weighs 28 MB.  I could provide this for download
>>> if someone is interested.
>> 
>> For me this corpus does not produce an exception (although it does have
>> one thread with 5067 messages).
>
> I cannot view this ATM because I cannot index these emails.  When
> viewd with mutt there are no such gargantuan threads.
>
>> I'm running Debian testing, so there could be slightly different
>> versions of xapian and gmime. What versions of libxapian-dev and
>> libgmime-2.6-dev have you built notmuch against?
>
> Mhm.  I'm running debian stretch.  The versions are:
>

I tried on a debian stretch with the same library versions, and I still
can't duplicate the crash.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner  [13. Jul. 2017]:
> Gregor Zattler  writes:
>> I then downloaded parts of the archive, merged and sliced it and
>> now I have a sample of 25001 total files (that's not much mail),
>> on which notmuch new produces a xapian exeption:
>>
> 
>> As a tar.xz it weighs 28 MB.  I could provide this for download
>> if someone is interested.
> 
> For me this corpus does not produce an exception (although it does have
> one thread with 5067 messages).

I cannot view this ATM because I cannot index these emails.  When
viewd with mutt there are no such gargantuan threads.

> I'm running Debian testing, so there could be slightly different
> versions of xapian and gmime. What versions of libxapian-dev and
> libgmime-2.6-dev have you built notmuch against?

Mhm.  I'm running debian stretch.  The versions are:

0 grfz@len:/tmp$ apt-cache policy libxapian-dev libgmime-2.6-dev
libxapian-dev:
Installed: 1.4.3-2
Candidate: 1.4.3-2
Version table:
*** 1.4.3-2 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status
libgmime-2.6-dev:
Installed: 2.6.22+dfsg2-1
Candidate: 2.6.22+dfsg2-1
Version table:
*** 2.6.22+dfsg2-1 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status


BTW:


0 grfz@len:/tmp$ apt-cache policy gcc
gcc:
Installed: 4:6.3.0-4
Candidate: 4:6.3.0-4
Version table:
*** 4:6.3.0-4 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status
0 grfz@len:/tmp$ gcc --version
gcc (Debian 6.3.0-18) 6.3.0 20170516





HTH, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

>
> I then downloaded parts of the archive, merged and sliced it and
> now I have a sample of 25001 total files (that's not much mail),
> on which notmuch new produces a xapian exeption:
>

> As a tar.xz it weighs 28 MB.  I could provide this for download
> if someone is interested.

For me this corpus does not produce an exception (although it does have
one thread with 5067 messages).

I'm running Debian testing, so there could be slightly different
versions of xapian and gmime. What versions of libxapian-dev and
libgmime-2.6-dev have you built notmuch against?

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner  [13. Jul. 2017]:
>> * David Bremner  [09. Jul. 2017]:
>>> Gregor Zattler  writes:
>>> I was wondering if the exception could result from overflowing some
>>> internal limit due to threads many thousands of messages long. I will be
>>> hard to know until I can replicate the exception. Perhaps a more
>>> complete org-mode list archive would do it.
>>
>> Quite possible, when I do 
>> notmuch show  --entire-thread=true --format=mbox 
>> path:Mail/~ml/emacs-orgm...@gnu.org/**   date 64 bit 32 > /tmp/emo.mbox
>> and open the resulting mbox with mutt, it shows 10199 messages.
> 
> Are those literal search terms "date 64 bit 32"?

yes.

>> As a tar.xz it weighs 28 MB.  I could provide this for download
>> if someone is interested.

I'll do so in a private email.  


Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

> Hi David, notmuch developers,
> * David Bremner  [09. Jul. 2017]:
>> Gregor Zattler  writes:
>> 
>> I was wondering if the exception could result from overflowing some
>> internal limit due to threads many thousands of messages long. I will be
>> hard to know until I can replicate the exception. Perhaps a more
>> complete org-mode list archive would do it.
>
> Quite possible, when I do 
> notmuch show  --entire-thread=true --format=mbox 
> path:Mail/~ml/emacs-orgm...@gnu.org/**   date 64 bit 32 > /tmp/emo.mbox
> and open the resulting mbox with mutt, it shows 10199 messages.

Are those literal search terms "date 64 bit 32"?

> As a tar.xz it weighs 28 MB.  I could provide this for download
> if someone is interested.

Sure, send me a download URL.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner  [09. Jul. 2017]:
> Gregor Zattler  writes:
>> Short version: Some of the messages on this mailinglist had very
>> weired References: headers mot probably causing notmuch to
>> misbehave while threading the messages.  But then there was no
>> xapian exeption involved.
>>
> 
> Right. I looks like there are indeed many message-ids in the resulting
> database that look like valid email addresses. So that problem persists,
> despite an attempted fix discussed in that thread.
> 
> I was wondering if the exception could result from overflowing some
> internal limit due to threads many thousands of messages long. I will be
> hard to know until I can replicate the exception. Perhaps a more
> complete org-mode list archive would do it.

Quite possible, when I do 
notmuch show  --entire-thread=true --format=mbox 
path:Mail/~ml/emacs-orgm...@gnu.org/**   date 64 bit 32 > /tmp/emo.mbox
and open the resulting mbox with mutt, it shows 10199 messages.

There is fun to have regarding message ids:

0 grfz@len:/tmp$ grep -c "^Message-ID:[[:space:]]" eom.mbox 
8758
0 grfz@len:/tmp$ grep -ci "^Message-ID:[[:space:]]" eom.mbox 
10174

I then split the mbox with mutt in a maildir folder.  This were
10199 individual files.  I then extracted the message ids via
formail from this files and piped this through sort -u.  These
were 10105 message ids.

The maildir with the emacs-orgm...@gnu.org emails contained 114563
individual files.  I removed every file somewhere containing one
of those message ids.  Now there were only 102164 individual files
in the maildir.

And after that I indexed this cleaned up maildir.  If somehow
this message threading problem affects the indexing it should now
index the files without xapian exeption:


This is with newest notmuch:
0 grfz@len:/tmp$ /home/grfz/src/notmuch/notmuch --version
notmuch 0.24.2+112~g37d1fa5


0 grfz@len:/tmp$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 102164 total files (that's not much mail).
Processed 102164 total files in 5m 58s (284 files/sec.).
Added 102151 new messages to the database.
[Inferior 1 (process 26339) exited normally]
(gdb) 


While indexing the original emacs-org mode mailing list maildir
with it's 114563 files with the very same binary results in a
xapian exeption:


0 grfz@len:/tmp$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 114563 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x557d2780, 
format=0x5558d258 "A Xapian exception occurred adding message: %s.\n") at 
lib/database.cc:426
426 {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x557d2780, format=0x5558d258 "A 
Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x55577f58 in notmuch_database_add_message 
(notmuch=notmuch@entry=0x557d2780, filename=filename@entry=

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

> Hi David,
> * David Bremner  [09. Jul. 2017]:
>> Gregor Zattler  writes:
>>> Since its an email from an open mailing list I attach it to
>>> this very email.
>>>
>> 
>> It might or might not be related, but I noticed that those two messages
>> get merged for me into a thread 1734 messages long of org-mode messages
>> (I have some old org messages, but I think I not the two you mention).
>> 
>> I'm not sure yet if it's a bug in notmuch, or something strange about
>> mail from the org-mode list.
>
> This *might* be related with some other problem I reported
> regarding false threading of messages, as discussed in the thread
> containing amongst others id:874nvcekjk@qmul.ac.uk
>
> Short version: Some of the messages on this mailinglist had very
> weired References: headers mot probably causing notmuch to
> misbehave while threading the messages.  But then there was no
> xapian exeption involved.
>

Right. I looks like there are indeed many message-ids in the resulting
database that look like valid email addresses. So that problem persists,
despite an attempted fix discussed in that thread.

I was wondering if the exception could result from overflowing some
internal limit due to threads many thousands of messages long. I will be
hard to know until I can replicate the exception. Perhaps a more
complete org-mode list archive would do it.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David,
* David Bremner  [09. Jul. 2017]:
> Gregor Zattler  writes:
>> Since its an email from an open mailing list I attach it to
>> this very email.
>>
> 
> It might or might not be related, but I noticed that those two messages
> get merged for me into a thread 1734 messages long of org-mode messages
> (I have some old org messages, but I think I not the two you mention).
> 
> I'm not sure yet if it's a bug in notmuch, or something strange about
> mail from the org-mode list.

This *might* be related with some other problem I reported
regarding false threading of messages, as discussed in the thread
containing amongst others id:874nvcekjk@qmul.ac.uk

Short version: Some of the messages on this mailinglist had very
weired References: headers mot probably causing notmuch to
misbehave while threading the messages.  But then there was no
xapian exeption involved.


Otherwise I did an extensive memory test: no prblem and an
smartctl -t long and -t offline test: no problems.  Remains to do
a bisecting but I have no free time atm.  Will report back later.

Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

>
> Since its an email from an open mailing list I attach it to
> this very email.
>

It might or might not be related, but I noticed that those two messages
get merged for me into a thread 1734 messages long of org-mode messages
(I have some old org messages, but I think I not the two you mention).

I'm not sure yet if it's a bug in notmuch, or something strange about
mail from the org-mode list.

d

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner  [03. Jul. 2017]:
> Try setting a break in _notmuch_database_log (b _notmuch_database_log)
> and run "bt" at that break point. It might (or might not) be worth
> continuing after the first breakpoint to inspect other errors.

I did so with newest notmuch:

0 (master *) grfz@len:~/tmp$ notmuch --version
notmuch 0.24.2+84~g1ec6344
0 (master *) grfz@len:~/tmp$

0 (master=) grfz@len:~/src/notmuch$ gdb --args /home/grfz/src/notmuch/notmuch 
new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 941498 total files (that's not much mail).
q
Breakpoint 1, _notmuch_database_log (notmuch=0x55825dd0, 
format=0x5558d258 "A Xapian exception occurred adding message: %s.\n") at 
lib/database.cc:426
426 {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x55825dd0, format=0x5558d258 "A 
Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x55577f58 in notmuch_database_add_message 
(notmuch=notmuch@entry=0x55825dd0, filename=filename@entry=0x7b204400 
"/home/grfz/Mail/~ml/emacs-orgm...@gnu.org/cur/1488666048.R6065357166258349555.len:2,",
message_ret=message_ret@entry=0x7fffd048) at lib/database.cc:2597
#2  0x5556802f in add_file (state=0x7fffd540, 
filename=0x7b204400 
"/home/grfz/Mail/~ml/emacs-orgm...@gnu.org/cur/1488666048.R6065357166258349555.len:2,",
 notmuch=0x55825dd0) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x55825dd0, 
path=path@entry=0x74a15b80 "/home/grfz/Mail/~ml/emacs-orgm...@gnu.org/cur", 
state=state@entry=0x7fffd540) at notmuch-new.c:599
#4  0x55567b44 in add_files (notmuch=notmuch@entry=0x55825dd0, 
path=path@entry=0x8520f1b0 "/home/grfz/Mail/~ml/emacs-orgm...@gnu.org", 
state=state@entry=0x7fffd540) at notmuch-new.c:483
#5  0x55567b44 in add_files (notmuch=notmuch@entry=0x55825dd0, 
path=path@entry=0x638993b0 "/home/grfz/Mail/~ml", 
state=state@entry=0x7fffd540) at notmuch-new.c:483
#6  0x55567b44 in add_files (notmuch=0x55825dd0, 
path=path@entry=0x557cf710 "/home/grfz/Mail", 
state=state@entry=0x7fffd540) at notmuch-new.c:483
#7  0x555689ed in notmuch_new_command (config=0x557ce1d0, 
argc=, argv=) at notmuch-new.c:1099
#8  0x55561a27 in main (argc=, argv=0x7fffda68) at 
notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 660118 total files in 8h 59m 43s (20 files/sec.).
Added 584749 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 14593) exited with code 01]
(gdb) continue
The program is not being run.
(gdb)



Since the gdb output shows a filename I looked at its contents:
There is a second "Content-Length:" -Header which does not make
sense but otherwise should not be a problem.  

Since its an email from an open mailing list I attach it to
this very email.

I then configured notmuch to scan the whole mail folder
/home/grfz/Mail/~ml/emacs-orgm...@gnu.org: 

0 (master=) grfz@len:~/src/notmuch$ gdb --args /home/grfz/src/notmuch/notmuch 
new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Tomi Ollila]

You could also try notmuch new --verbose 

--- and then look into the last file it was indexing when crash occurred...


Tomi

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

>
> I have no clue about gdb, there was no backtrace:
>
...
> Error: A Xapian exception occurred. Halting processing.
> Processed 662357 total files in 1h 52m 51s (97 files/sec.).
> Added 587023 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 4509) exited with code 01]
> (gdb) bt
> No stack.
> (gdb) 

Try setting a break in _notmuch_database_log (b _notmuch_database_log)
and run "bt" at that break point. It might (or might not) be worth
continuing after the first breakpoint to inspect other errors.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[Gregor Zattler]

Hi David, notmuch developers,
* David Bremner  [03. Jul. 2017]:
> Gregor Zattler  writes:
>> notmuch new fails while indexing my emails and when executed a
>> second time it obviously starts over again.  This is on a
>> up-to-date debian stetch with notmuch from git:
>> Any ideas how to proceed in order to get a working notmuch setup?

> 1. Check disk space

Sorry, forgot to mention: there is enough free space on RAM (8
GB) and SSD (40 G free).

> 2. run xapian-fsck

xapian-check does not find any errors, only spelling and synonym
are "Lazily created, and not yet used."

> 3. run "notmuch new" under gdb to try to get a backtrace

I have no clue about gdb, there was no backtrace:

0 (master) grfz@len:~$ gdb --args notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from notmuch...done.
(gdb) run
Starting program: /usr/local/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Error: A Xapian exception occurred. Halting processing.
Processed 662357 total files in 1h 52m 51s (97 files/sec.).
Added 587023 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 4509) exited with code 01]
(gdb) bt
No stack.
(gdb) 


> 4. use "git bisect" to try to find what commit introduced this problem

I'll do so in the next few days.

Thanks, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

[David Bremner]

Gregor Zattler  writes:

> Dear notmuch developers,
>
> notmuch new fails while indexing my emails and when executed a
> second time it obviously starts over again.  This is on a
> up-to-date debian stetch with notmuch from git:
> Any ideas how to proceed in order to get a working notmuch setup?
>

> Ciao, Gregor

Some things to try

1. Check disk space
2. run xapian-fsck
3. run "notmuch new" under gdb to try to get a backtrace
4. use "git bisect" to try to find what commit introduced this problem

d




signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch