[PATCH 1/6] test: two new messages for the 'broken' corpus
These have an 'In-Reply-To' loop, which currently confuses "notmuch new". --- test/corpora/broken/loop/loop-12 | 8 test/corpora/broken/loop/loop-21 | 8 2 files changed, 16 insertions(+) create mode 100644 test/corpora/broken/loop/loop-12 create mode 100644 test/corpora/broken/loop/loop-21 diff --git a/test/corpora/broken/loop/loop-12 b/test/corpora/broken/loop/loop-12 new file mode 100644 index ..b5c3af7e --- /dev/null +++ b/test/corpora/broken/loop/loop-12 @@ -0,0 +1,8 @@ +From: Alice +To: Daniel +Subject: referencing in-reply-to-loop-21 +Message-ID: +In-Reply-To: +Date: Thu, 16 Jun 2016 22:14:41 -0400 + +Note Message-ID and In-Reply-To: in file in-reply-to-loop-21 diff --git a/test/corpora/broken/loop/loop-21 b/test/corpora/broken/loop/loop-21 new file mode 100644 index ..234f0323 --- /dev/null +++ b/test/corpora/broken/loop/loop-21 @@ -0,0 +1,8 @@ +From: Alice +To: Daniel +Subject: referencing in-reply-to-loop-12 +Message-ID: +In-Reply-To: +Date: Fri, 17 Jun 2016 22:14:41 -0400 + +Note Message-ID and In-Reply-To: in file in-reply-to-loop-12 -- 2.17.0 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 6/6] lib: choose oldest message when breaking reference loops
This preserves a sensible thread order
---
lib/thread.cc| 33 -
test/T050-new.sh | 1 -
2 files changed, 24 insertions(+), 10 deletions(-)
diff --git a/lib/thread.cc b/lib/thread.cc
index dbac002f..a6dc4e5a 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -390,20 +390,18 @@ _thread_add_matched_message (notmuch_thread_t *thread,
static void
_resolve_thread_relationships (notmuch_thread_t *thread)
{
-notmuch_message_node_t *node;
+notmuch_message_node_t *node, *first_node;
notmuch_message_t *message, *parent;
const char *in_reply_to;
-for (node = thread->message_list->head; node; node = node->next) {
+first_node = thread->message_list->head;
+if (!first_node)
+ return;
+
+for (node = first_node->next; node; node = node->next) {
message = node->message;
in_reply_to = _notmuch_message_get_in_reply_to (message);
- /*
-* if we reach the end of the list without finding a top-level
-* message, that means the thread is a cycle (or set of
-* cycles) and any message can be considered top-level
-*/
- if ((thread->toplevel_list->head || node->next) &&
-in_reply_to && strlen (in_reply_to) &&
+ if (in_reply_to && strlen (in_reply_to) &&
g_hash_table_lookup_extended (thread->message_hash,
in_reply_to, NULL,
(void **) &parent))
@@ -412,6 +410,23 @@ _resolve_thread_relationships (notmuch_thread_t *thread)
_notmuch_message_list_add_message (thread->toplevel_list, message);
}
+/*
+ * if we reach the end of the list without finding a top-level
+ * message, that means the thread is a cycle (or set of cycles)
+ * and any message can be considered top-level. Choose the oldest
+ * message, which happens to be first in our list.
+ */
+message=first_node->message;
+in_reply_to = _notmuch_message_get_in_reply_to (message);
+if (thread->toplevel_list->head &&
+ in_reply_to && strlen (in_reply_to) &&
+ g_hash_table_lookup_extended (thread->message_hash,
+ in_reply_to, NULL,
+ (void **) &parent))
+ _notmuch_message_add_reply (parent, message);
+else
+ _notmuch_message_list_add_message (thread->toplevel_list, message);
+
/* XXX: After scanning through the entire list looking for parents
* via "In-Reply-To", we should do a second pass that looks at the
* list of messages IDs in the "References" header instead. (And
diff --git a/test/T050-new.sh b/test/T050-new.sh
index 320a7646..9025fa7a 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -359,7 +359,6 @@ test_begin_subtest "reference loop does not crash"
test_expect_code 0 "notmuch show --format=json id:mid-loop...@example.org
id:mid-loop...@example.org > OUTPUT"
test_begin_subtest "reference loop ordered by date"
-test_subtest_known_broken
threadid=$(notmuch search --output=threads id:mid-loop...@example.org)
notmuch show --format=mbox $threadid | grep '^Date' > OUTPUT
cat < EXPECTED
--
2.17.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 4/6] NEWS: add item for reference loop fix.
--- NEWS | 9 + 1 file changed, 9 insertions(+) diff --git a/NEWS b/NEWS index 39ce7707..e9b1dcd9 100644 --- a/NEWS +++ b/NEWS @@ -1,3 +1,12 @@ +Notmuch 0.26.2 (UNRELEASED) +=== + +Library Changes +--- + +Make thread indexing more robust against reference loops. Fix a +related abort in "notmuch show". + Notmuch 0.26.1 (2018-04-02) === -- 2.17.0 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 5/6] test: add known broken test for thread ordering from a loop
The previous loop handling code chooses the last message in the message list, which turns out to be the last in date order. See the comment in _notmuch_thread_create. --- test/T050-new.sh | 10 ++ 1 file changed, 10 insertions(+) diff --git a/test/T050-new.sh b/test/T050-new.sh index f3bfe7b1..320a7646 100755 --- a/test/T050-new.sh +++ b/test/T050-new.sh @@ -358,4 +358,14 @@ add_email_corpus broken test_begin_subtest "reference loop does not crash" test_expect_code 0 "notmuch show --format=json id:mid-loop...@example.org id:mid-loop...@example.org > OUTPUT" +test_begin_subtest "reference loop ordered by date" +test_subtest_known_broken +threadid=$(notmuch search --output=threads id:mid-loop...@example.org) +notmuch show --format=mbox $threadid | grep '^Date' > OUTPUT +cat < EXPECTED +Date: Thu, 16 Jun 2016 22:14:41 -0400 +Date: Fri, 17 Jun 2016 22:14:41 -0400 +EOF +test_expect_equal_file EXPECTED OUTPUT + test_done -- 2.17.0 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 3/6] lib: break reference loop by choosing arbitrary top level msg
Other parts of notmuch (e.g. notmuch show) expect each thread to
contain at least one top level message, and crash if this expectation
is not met.
---
lib/thread.cc| 8 +++-
test/T050-new.sh | 1 -
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/lib/thread.cc b/lib/thread.cc
index 3561b27f..dbac002f 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -397,7 +397,13 @@ _resolve_thread_relationships (notmuch_thread_t *thread)
for (node = thread->message_list->head; node; node = node->next) {
message = node->message;
in_reply_to = _notmuch_message_get_in_reply_to (message);
- if (in_reply_to && strlen (in_reply_to) &&
+ /*
+* if we reach the end of the list without finding a top-level
+* message, that means the thread is a cycle (or set of
+* cycles) and any message can be considered top-level
+*/
+ if ((thread->toplevel_list->head || node->next) &&
+in_reply_to && strlen (in_reply_to) &&
g_hash_table_lookup_extended (thread->message_hash,
in_reply_to, NULL,
(void **) &parent))
diff --git a/test/T050-new.sh b/test/T050-new.sh
index b9854978..f3bfe7b1 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -356,7 +356,6 @@ test_expect_equal_file EXPECTED OUTPUT
add_email_corpus broken
test_begin_subtest "reference loop does not crash"
-test_subtest_known_broken
test_expect_code 0 "notmuch show --format=json id:mid-loop...@example.org
id:mid-loop...@example.org > OUTPUT"
test_done
--
2.17.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/6] test: add known broken test for indexing an In-Reply-To loop.
This documents the bug discussed in id:87d10042pu@curie.anarc.at --- test/T050-new.sh | 5 + 1 file changed, 5 insertions(+) diff --git a/test/T050-new.sh b/test/T050-new.sh index cd522364..b9854978 100755 --- a/test/T050-new.sh +++ b/test/T050-new.sh @@ -354,4 +354,9 @@ exit status: 75 EOF test_expect_equal_file EXPECTED OUTPUT +add_email_corpus broken +test_begin_subtest "reference loop does not crash" +test_subtest_known_broken +test_expect_code 0 "notmuch show --format=json id:mid-loop...@example.org id:mid-loop...@example.org > OUTPUT" + test_done -- 2.17.0 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
v3 reference loop fix
this version obsoletes id:20180414014610.15438-1-da...@tethera.net changes: - simplified test messages from Tomi - deterministic thread order suggested by dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Bug? notmuch-emacs: "All tags" in notmuch-hello does not show all tags
Dear notmuch-emacs developers, I use a certain tag which until now is only used with regards to one single message which is also tagged "spam". This certain tag is among the output of notmuch search --output=tags '*' The part of notmuch-hello which is supposed to show "All tags" does not show this certain tag. But it shows the certain tag with count = 1 if I remove the tag "spam" from this message. The same happens if this message is tagged "deleted" instead of "spam". My ~/.notmuch-config contains: [search] exclude_tags=deleted;spam; The relevant section of my customzation.el contains: '(notmuch-hello-sections '(notmuch-hello-insert-search notmuch-hello-insert-saved-searches notmuch-hello-insert-recent-searches (notmuch-hello-insert-tags-section "All tags" :initially-hidden t nil nil))) I can show the message in question with a search for (is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted) AND is:certaintag notmuch-emacs then shows the single message and its tags. Among those is the certain tag. I expected the certain tag to be shown in the notmuch-hello section "All tags" and consider it a bug that this is not the case. My guess is that somewhere in the code notmuch count is called without --exclude=false. Ciao; Gregor -- -... --- .-. . -.. ..--.. ...-.- ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Database error
Quoting David Bremner (2018-04-20 14:44:25)
> > » xapian-check .notmuch/xapian docdata:
> > blocksize=8K items=564 firstunused=7 revision=7 levels=1 root=2
> > B-tree checked okay
> > docdata table structure checked OK
> >
> > termlist:
> > blocksize=8K items=305882 firstunused=64506 revision=7 levels=2 root=63491
> > xapian-check: DatabaseError: 1 unused block(s) missing from the free
> > list, first is 0
>
>
> This is almost certainly a known and recently fixed xapian bug. notmuch
> compact is the recommended workaround (or just ignore the message).
Ok, I'll ignore it for now.
>
> >
> > After this I try to tun "afew --tag --new -v" but it fails with (I don't
> > know if
> > this is related though):
> >
> > Traceback (most recent call last):
> > File "/usr/bin/afew", line 11, in
> > load_entry_point('afew==1.3.0', 'console_scripts', 'afew')()
> > File "/usr/lib/python3/dist-packages/afew/commands.py", line 159, in
> > main
> > inner_main(args, database, query_string)
> > File "/usr/lib/python3/dist-packages/afew/main.py", line 23, in main
> > filter_.run(query_string)
> > File "/usr/lib/python3/dist-packages/afew/filters/BaseFilter.py", line
> > 59, in run
> > for message in self.database.get_messages(query):
> > File "/usr/lib/python3/dist-packages/afew/Database.py", line 93, in
> > get_messages
> > for message in self.do_query(query).search_messages():
> > File "/usr/lib/python3/dist-packages/notmuch/query.py", line 182, in
> > search_messages
> > raise NotmuchError(status)
> > notmuch.errors.XapianError
> >
>
> It's hard to say for sure, but it's possible this would be fixed by the
> patch
>
> 20180414014610.15438-4-david@tethera.net">https://mid.mail-archive.com/20180414014610.15438-4-david@tethera.net
>
> That should be part of a bug-fix release of notmuch fairly soon. You can
> also try to find a subset of mail reproducing the problem. I don't
> know what that afew invokation is trying to do. Can you reproduce the
> problem using the notmuch CLI?
It turned out that this was a bad syntax in afew, where I quoted a query in
single quotes.
Greetings, Varac
signature.asc
Description: signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] lib: work around xapian bug with get_mset(0,0, x)
Daniel Kahn Gillmor writes: > But if we want a compile-time check, i don't think we'd need anything > fancier than something like (potentially even directly in query.cc): > > #if XAPIAN_AT_LEAST(1,4,6) > #define MSET_GET_MIN_COUNT 0 > #else > #define MSET_GET_MIN_COUNT 1 > #endif > > what else were you thinking we'd need? > > --dkg I forgot about the XAPIAN_AT_LEAST macro. Nonetheless, I'm not keen to ship untested/uncompiled code (no matter how "obviously correct" it is), so I'd prefer to wait until Xapian 1.4.6 actually exists before adding an ifdef like that. FWIW I doubt that the performance impact of passing 1 instead of 0 here is measurable. For me, getting a bug fix release out is a matter of some urgency; fine tuning and testing against xapian 1.4.6 can wait. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Database error
Varac writes:
> Hi,
>
> I get a DatabaseError right after initial indexing.
> I can reproduce this with my mails every time.
> First, I remove the whole .notmuch/xapian directory and
> run "notmuch new", which results in a DatabaseError when
> using "xapian-check":
>
>
> » notmuch new
> Welcome to a new version of notmuch! Your database will now be upgraded.
> This process is safe to interrupt.
> Backing up tags to
> /home/varac/.offlineimap/Mail/.notmuch/dump-20180419T160138.gz...
> Your notmuch database has now been upgraded.
> Processed 148440 total files in 13m 38s (181 files/sec.).
> Added 140351 new messages to the database.
>
> » xapian-check .notmuch/xapian
> docdata:
> blocksize=8K items=564 firstunused=7 revision=7 levels=1 root=2
> B-tree checked okay
> docdata table structure checked OK
>
> termlist:
> blocksize=8K items=305882 firstunused=64506 revision=7 levels=2 root=63491
> xapian-check: DatabaseError: 1 unused block(s) missing from the free list,
> first is 0
This is almost certainly a known and recently fixed xapian bug. notmuch
compact is the recommended workaround (or just ignore the message).
>
> After this I try to tun "afew --tag --new -v" but it fails with (I don't know
> if
> this is related though):
>
> Traceback (most recent call last):
> File "/usr/bin/afew", line 11, in
> load_entry_point('afew==1.3.0', 'console_scripts', 'afew')()
> File "/usr/lib/python3/dist-packages/afew/commands.py", line 159, in main
> inner_main(args, database, query_string)
> File "/usr/lib/python3/dist-packages/afew/main.py", line 23, in main
> filter_.run(query_string)
> File "/usr/lib/python3/dist-packages/afew/filters/BaseFilter.py", line
> 59, in run
> for message in self.database.get_messages(query):
> File "/usr/lib/python3/dist-packages/afew/Database.py", line 93, in
> get_messages
> for message in self.do_query(query).search_messages():
> File "/usr/lib/python3/dist-packages/notmuch/query.py", line 182, in
> search_messages
> raise NotmuchError(status)
> notmuch.errors.XapianError
>
It's hard to say for sure, but it's possible this would be fixed by the
patch
20180414014610.15438-4-david@tethera.net">https://mid.mail-archive.com/20180414014610.15438-4-david@tethera.net
That should be part of a bug-fix release of notmuch fairly soon. You can
also try to find a subset of mail reproducing the problem. I don't
know what that afew invokation is trying to do. Can you reproduce the
problem using the notmuch CLI?
d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
