Package: release.debian.org Severity: normal User: release.debian....@packages.debian.org Usertags: unblock
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Please unblock package notmuch [ Reason ] The new version contains a backported (well, cherry-picked) fix for a database corruption bug fixed in 0.32.2. [ Impact ] The corruption is relatively mild, if that is possible. Roughly speaking, when multiple mail files have the same message-id, the message document with that message id gets added to multiple threads. I introduced this bug in 2017, but there has only been occasional mystified users complaining about misthreading of messages. Still, it's not very nice to have a subtle failure in a core feature. [ Tests ] I have backported the new integration test, and it (and the rest of the test suite) passes). I ran for about 24 hours with the proposed version as my MUA, didn't notice anything untoward. [ Risks ] The changed code is confined to about 6 lines of code in add-message.c. Most of the diff is adjusting the test suite (which had unknowingly encoded the bug in several tests). On the other hand, this is a change in the core of notmuch, so potentially affects several rdeps, most notably (from a popcon point of view) neomutt. [ Checklist ] [x] all changes are documented in the d/changelog [x] I reviewed all changes and I approve them [x] attach debdiff against the package in testing [ Other info ] I also attach the output of git diff --diff-filter=MRC, to hide the contents of debian/patches unblock notmuch/0.31.4-2 -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEkiyHYXwaY0SiY6fqA0U5G1WqFSEFAmDa/4gACgkQA0U5G1Wq FSELyA//ZHb3pqhnMRXvxlM6yHUYrinl2yvIl0ZVizKTOTVMvU3TpijbJ5ozUPe5 7HD6vgOz0YovosCr/FvBleCSp6n8z5tspsiIoTnaDFZfaQ8QslYRCTpAaFZ2u939 L4nE13J8aWP58Az8Pq6zKf+P88xFZkd2kzKBtgiBwl2mB6guIc9XDtlz4LvFAUZP XuqGD6wIO+hX72HIDTzeN/NsC43fEkEtmIOpbDnlX0qLOm1pEf8AqwzCvkkFwwgs Ml2Bq6jZJwSfzCAKXQohl7fu+SAgvd5ifBngCQq/RDcj1T+u4EiOfAVlQC2cylwL vIjAb/Kg0/QpjA1glj3zgv7eX18iVWbQ6MKZDnlGzeqzAUfRSd8BvN8SY9uBhTqX S12pXsD0ZYW2Qr2bRVIChFHkoAxppU38shpQ81gAXyP1W0kicfmMMLIidr8R22Xg PB4SBhtbfsorc7KiJ2Y3lCs/pcTaq3zupm2UA3xAdFSDj9ugXJg2iye7wqNnmmV9 uchNeLoY9LD6dnJN1FP3rH5HPL4DMejFoKzhjBKwlZwsLjCyNdu/pQPczSopWDcN pgI969jNUzfEV95VpexTpRbFJtNwNUuwF1g5Yy1p36ZKnDDOph99TOvmJks0V9+I /RXMLJdA4oohmWu069q6VmdR1nC+mnMLwsbRJq1ld4FfLowW0ZU= =Pb2W -----END PGP SIGNATURE-----
diff -Nru notmuch-0.31.4/debian/changelog notmuch-0.31.4/debian/changelog --- notmuch-0.31.4/debian/changelog 2021-02-18 07:23:00.000000000 -0400 +++ notmuch-0.31.4/debian/changelog 2021-06-28 22:09:09.000000000 -0300 @@ -1,3 +1,11 @@ +notmuch (0.31.4-2) unstable; urgency=medium + + * Cherry pick upstream commit 3f4de98e7c8, which fixes a bug where + duplicate message-ids can cause multiple thread-ids for some message + documents. + + -- David Bremner <brem...@debian.org> Mon, 28 Jun 2021 22:09:09 -0300 + notmuch (0.31.4-1) unstable; urgency=medium * New upstream bugfix release diff -Nru notmuch-0.31.4/debian/control notmuch-0.31.4/debian/control --- notmuch-0.31.4/debian/control 2021-02-18 07:23:00.000000000 -0400 +++ notmuch-0.31.4/debian/control 2021-06-28 22:09:09.000000000 -0300 @@ -37,6 +37,7 @@ ruby, ruby-dev (>>1:1.9.3~), texinfo, + xapian-tools <!nocheck>, Standards-Version: 4.4.1 Homepage: https://notmuchmail.org/ Vcs-Git: https://git.notmuchmail.org/git/notmuch -b release diff -Nru notmuch-0.31.4/debian/patches/debian-changes notmuch-0.31.4/debian/patches/debian-changes --- notmuch-0.31.4/debian/patches/debian-changes 1969-12-31 20:00:00.000000000 -0400 +++ notmuch-0.31.4/debian/patches/debian-changes 2021-06-28 22:09:09.000000000 -0300 @@ -0,0 +1,136 @@ +This is an autogenerated patch header for a single-debian-patch file. The +delta against upstream is either kept as a single patch, or maintained +in some VCS, and exported as a single patch instead of more manageable +atomic patches. + +--- notmuch-0.31.4.orig/lib/add-message.cc ++++ notmuch-0.31.4/lib/add-message.cc +@@ -407,14 +407,17 @@ static notmuch_status_t + _notmuch_database_link_message (notmuch_database_t *notmuch, + notmuch_message_t *message, + notmuch_message_file_t *message_file, +- bool is_ghost) ++ bool is_ghost, ++ bool is_new) + { + void *local = talloc_new (NULL); + notmuch_status_t status; + const char *thread_id = NULL; + + /* Check if the message already had a thread ID */ +- if (notmuch->features & NOTMUCH_FEATURE_GHOSTS) { ++ if (! is_new) { ++ thread_id = notmuch_message_get_thread_id (message); ++ } else if (notmuch->features & NOTMUCH_FEATURE_GHOSTS) { + if (is_ghost) + thread_id = notmuch_message_get_thread_id (message); + } else { +@@ -539,7 +542,7 @@ notmuch_database_index_file (notmuch_dat + } + + ret = _notmuch_database_link_message (notmuch, message, +- message_file, is_ghost); ++ message_file, is_ghost, is_new); + if (ret) + goto DONE; + +--- notmuch-0.31.4.orig/test/T357-index-decryption.sh ++++ notmuch-0.31.4/test/T357-index-decryption.sh +@@ -112,12 +112,10 @@ test_expect_equal \ + "$expected" + + # try inserting it with decryption, should appear as a single copy +-# (note: i think thread id skips 4 because of duplicate message-id +-# insertion, above) + test_begin_subtest "message cleartext is present with insert --decrypt=true" + notmuch insert --folder=sent --decrypt=true <<<"$contents" +-output=$(notmuch search wumpus) +-expected='thread:0000000000000005 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (encrypted inbox unread)' ++output=$(notmuch search wumpus | notmuch_search_sanitize) ++expected='thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (encrypted inbox unread)' + test_expect_equal \ + "$output" \ + "$expected" +@@ -127,9 +125,9 @@ test_expect_equal \ + test_begin_subtest 'tagging all messages' + test_expect_success 'notmuch tag +blarney "encrypted message"' + test_begin_subtest "verify that tags have not changed" +-output=$(notmuch search tag:blarney) +-expected='thread:0000000000000001 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) +-thread:0000000000000005 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' ++output=$(notmuch search tag:blarney | notmuch_search_sanitize) ++expected='thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) ++thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' + test_expect_equal \ + "$output" \ + "$expected" +@@ -138,14 +136,14 @@ test_expect_equal \ + test_begin_subtest 'reindex old messages' + test_expect_success 'notmuch reindex --decrypt=true tag:encrypted and not property:index.decryption=success' + test_begin_subtest "reindexed encrypted message, including cleartext" +-output=$(notmuch search wumpus) ++output=$(notmuch search wumpus | notmuch_search_sanitize) + test_expect_equal \ + "$output" \ + "$expected" + + # and the same search, but by property ($expected is untouched): + test_begin_subtest "emacs search by property for both messages" +-output=$(notmuch search property:index.decryption=success) ++output=$(notmuch search property:index.decryption=success | notmuch_search_sanitize) + test_expect_equal \ + "$output" \ + "$expected" +@@ -154,7 +152,7 @@ test_expect_equal \ + test_begin_subtest 'reindex in auto mode' + test_expect_success 'notmuch reindex tag:encrypted and property:index.decryption=success' + test_begin_subtest "reindexed encrypted messages, should not have changed" +-output=$(notmuch search wumpus) ++output=$(notmuch search wumpus | notmuch_search_sanitize) + test_expect_equal \ + "$output" \ + "$expected" +@@ -188,9 +186,9 @@ test_expect_equal \ + + # ensure that the tags remain even when we are dropping the cleartext. + test_begin_subtest "verify that tags remain without cleartext" +-output=$(notmuch search tag:blarney) +-expected='thread:0000000000000001 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) +-thread:0000000000000005 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' ++output=$(notmuch search tag:blarney | notmuch_search_sanitize) ++expected='thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) ++thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' + test_expect_equal \ + "$output" \ + "$expected" +@@ -199,7 +197,7 @@ test_begin_subtest "index cleartext with + test_expect_success "notmuch reindex --decrypt=nostash tag:blarney" + + test_begin_subtest "Ensure that the indexed terms are present" +-output=$(notmuch search wumpus) ++output=$(notmuch search wumpus | notmuch_search_sanitize) + test_expect_equal \ + "$output" \ + "$expected" +--- notmuch-0.31.4.orig/test/T670-duplicate-mid.sh ++++ notmuch-0.31.4/test/T670-duplicate-mid.sh +@@ -6,6 +6,19 @@ add_message '[id]="duplicate"' '[subject + add_message '[id]="duplicate"' '[subject]="message 2" [filename]=copy2' + + add_message '[id]="duplicate"' '[subject]="message 0" [filename]=copy0' ++ ++test_begin_subtest 'at most 1 thread-id per xapian document' ++db=${MAIL_DIR}/.notmuch/xapian ++for doc in $(xapian-delve -1 -t '' "$db" | grep '^[1-9]'); do ++ xapian-delve -1 -r "$doc" "$db" | grep -c '^G' ++done > OUTPUT.raw ++sort -u < OUTPUT.raw > OUTPUT ++cat <<EOF > EXPECTED ++0 ++1 ++EOF ++test_expect_equal_file EXPECTED OUTPUT ++ + test_begin_subtest 'search: first indexed subject preserved' + cat <<EOF > EXPECTED + thread:XXX 2001-01-05 [1/1(3)] Notmuch Test Suite; message 1 (inbox unread) diff -Nru notmuch-0.31.4/debian/patches/series notmuch-0.31.4/debian/patches/series --- notmuch-0.31.4/debian/patches/series 1969-12-31 20:00:00.000000000 -0400 +++ notmuch-0.31.4/debian/patches/series 2021-06-28 22:09:09.000000000 -0300 @@ -0,0 +1 @@ +debian-changes
diff --git a/debian/changelog b/debian/changelog index ec3de63d..ebfd9209 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,12 @@ +notmuch (0.31.4-2) unstable; urgency=medium + + * Cherry pick upstream commit 3f4de98e7c8, which fixes a bug where + duplicate message-ids can cause multiple thread-ids for some message + documents. + * Add build-dependency on xapian-tools, for new test + + -- David Bremner <brem...@debian.org> Mon, 28 Jun 2021 22:48:02 -0300 + notmuch (0.31.4-1) unstable; urgency=medium * New upstream bugfix release diff --git a/debian/control b/debian/control index 8b34d9ed..633d0be8 100644 --- a/debian/control +++ b/debian/control @@ -37,6 +37,7 @@ Build-Depends: ruby, ruby-dev (>>1:1.9.3~), texinfo, + xapian-tools <!nocheck>, Standards-Version: 4.4.1 Homepage: https://notmuchmail.org/ Vcs-Git: https://git.notmuchmail.org/git/notmuch -b release diff --git a/lib/add-message.cc b/lib/add-message.cc index 485debad..b47aa501 100644 --- a/lib/add-message.cc +++ b/lib/add-message.cc @@ -407,14 +407,17 @@ static notmuch_status_t _notmuch_database_link_message (notmuch_database_t *notmuch, notmuch_message_t *message, notmuch_message_file_t *message_file, - bool is_ghost) + bool is_ghost, + bool is_new) { void *local = talloc_new (NULL); notmuch_status_t status; const char *thread_id = NULL; /* Check if the message already had a thread ID */ - if (notmuch->features & NOTMUCH_FEATURE_GHOSTS) { + if (! is_new) { + thread_id = notmuch_message_get_thread_id (message); + } else if (notmuch->features & NOTMUCH_FEATURE_GHOSTS) { if (is_ghost) thread_id = notmuch_message_get_thread_id (message); } else { @@ -539,7 +542,7 @@ notmuch_database_index_file (notmuch_database_t *notmuch, } ret = _notmuch_database_link_message (notmuch, message, - message_file, is_ghost); + message_file, is_ghost, is_new); if (ret) goto DONE; diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh index 1ed5f28c..1951ef78 100755 --- a/test/T357-index-decryption.sh +++ b/test/T357-index-decryption.sh @@ -112,12 +112,10 @@ test_expect_equal \ "$expected" # try inserting it with decryption, should appear as a single copy -# (note: i think thread id skips 4 because of duplicate message-id -# insertion, above) test_begin_subtest "message cleartext is present with insert --decrypt=true" notmuch insert --folder=sent --decrypt=true <<<"$contents" -output=$(notmuch search wumpus) -expected='thread:0000000000000005 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (encrypted inbox unread)' +output=$(notmuch search wumpus | notmuch_search_sanitize) +expected='thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (encrypted inbox unread)' test_expect_equal \ "$output" \ "$expected" @@ -127,9 +125,9 @@ test_expect_equal \ test_begin_subtest 'tagging all messages' test_expect_success 'notmuch tag +blarney "encrypted message"' test_begin_subtest "verify that tags have not changed" -output=$(notmuch search tag:blarney) -expected='thread:0000000000000001 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) -thread:0000000000000005 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' +output=$(notmuch search tag:blarney | notmuch_search_sanitize) +expected='thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) +thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' test_expect_equal \ "$output" \ "$expected" @@ -138,14 +136,14 @@ test_expect_equal \ test_begin_subtest 'reindex old messages' test_expect_success 'notmuch reindex --decrypt=true tag:encrypted and not property:index.decryption=success' test_begin_subtest "reindexed encrypted message, including cleartext" -output=$(notmuch search wumpus) +output=$(notmuch search wumpus | notmuch_search_sanitize) test_expect_equal \ "$output" \ "$expected" # and the same search, but by property ($expected is untouched): test_begin_subtest "emacs search by property for both messages" -output=$(notmuch search property:index.decryption=success) +output=$(notmuch search property:index.decryption=success | notmuch_search_sanitize) test_expect_equal \ "$output" \ "$expected" @@ -154,7 +152,7 @@ test_expect_equal \ test_begin_subtest 'reindex in auto mode' test_expect_success 'notmuch reindex tag:encrypted and property:index.decryption=success' test_begin_subtest "reindexed encrypted messages, should not have changed" -output=$(notmuch search wumpus) +output=$(notmuch search wumpus | notmuch_search_sanitize) test_expect_equal \ "$output" \ "$expected" @@ -188,9 +186,9 @@ test_expect_equal \ # ensure that the tags remain even when we are dropping the cleartext. test_begin_subtest "verify that tags remain without cleartext" -output=$(notmuch search tag:blarney) -expected='thread:0000000000000001 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) -thread:0000000000000005 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' +output=$(notmuch search tag:blarney | notmuch_search_sanitize) +expected='thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 001 (blarney encrypted inbox) +thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (blarney encrypted inbox unread)' test_expect_equal \ "$output" \ "$expected" @@ -199,7 +197,7 @@ test_begin_subtest "index cleartext without keeping session keys" test_expect_success "notmuch reindex --decrypt=nostash tag:blarney" test_begin_subtest "Ensure that the indexed terms are present" -output=$(notmuch search wumpus) +output=$(notmuch search wumpus | notmuch_search_sanitize) test_expect_equal \ "$output" \ "$expected" diff --git a/test/T670-duplicate-mid.sh b/test/T670-duplicate-mid.sh index 4e5672ab..7ae60595 100755 --- a/test/T670-duplicate-mid.sh +++ b/test/T670-duplicate-mid.sh @@ -6,6 +6,19 @@ add_message '[id]="duplicate"' '[subject]="message 1" [filename]=copy1' add_message '[id]="duplicate"' '[subject]="message 2" [filename]=copy2' add_message '[id]="duplicate"' '[subject]="message 0" [filename]=copy0' + +test_begin_subtest 'at most 1 thread-id per xapian document' +db=${MAIL_DIR}/.notmuch/xapian +for doc in $(xapian-delve -1 -t '' "$db" | grep '^[1-9]'); do + xapian-delve -1 -r "$doc" "$db" | grep -c '^G' +done > OUTPUT.raw +sort -u < OUTPUT.raw > OUTPUT +cat <<EOF > EXPECTED +0 +1 +EOF +test_expect_equal_file EXPECTED OUTPUT + test_begin_subtest 'search: first indexed subject preserved' cat <<EOF > EXPECTED thread:XXX 2001-01-05 [1/1(3)] Notmuch Test Suite; message 1 (inbox unread)