cleanups / fixes from olly betts

[David Bremner]

Olly passed me a patch on IRC today. I'm not sure when I'll have a
chance to turn it into proper patches, so here it is so it doesn't get
lost.

diff --git a/gmime-filter-reply.c b/gmime-filter-reply.c
index a1ba4b45..f673c0a2 100644
--- a/gmime-filter-reply.c
+++ b/gmime-filter-reply.c
@@ -113,7 +113,7 @@ filter_filter (GMimeFilter *filter, char *inbuf, size_t 
inlen, size_t prespace,
   char **outbuf, size_t *outlen, size_t *outprespace)
 {
GMimeFilterReply *reply = (GMimeFilterReply *) filter;
-   register const char *inptr = inbuf;
+   const char *inptr = inbuf;
const char *inend = inbuf + inlen;
char *outptr;
 
diff --git a/lib/database.cc b/lib/database.cc
index 02444e09..b3d5bf56 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1185,12 +1185,9 @@ notmuch_database_compact (const char *path,
 (void) rmtree (compact_xapian_path);
 
 try {
+   Xapian::Database db (xapian_path);
NotmuchCompactor compactor (status_cb, closure);
-
-   compactor.set_renumber (false);
-   compactor.add_source (xapian_path);
-   compactor.set_destdir (compact_xapian_path);
-   compactor.compact ();
+   db.compact (compact_xapian_path, Xapian::DBCOMPACT_NO_RENUMBER, 0, 
compactor);
 } catch (const Xapian::Error ) {
_notmuch_database_log (notmuch, "Error while compacting: %s\n", 
error.get_msg().c_str());
ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
diff --git a/lib/index.cc b/lib/index.cc
index 0ad683fa..146a8928 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -176,7 +176,7 @@ filter_filter (GMimeFilter *gmime_filter, char *inbuf, 
size_t inlen, size_t pres
 {
 NotmuchFilterDiscardNonTerm *filter = (NotmuchFilterDiscardNonTerm *) 
gmime_filter;
 const scanner_state_t *states = filter->states;
-register const char *inptr = inbuf;
+const char *inptr = inbuf;
 const char *inend = inbuf + inlen;
 char *outptr;
 
diff --git a/notmuch-show.c b/notmuch-show.c
index 9871159d..e5280e66 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -326,13 +326,10 @@ signature_status_to_string (GMimeSignatureStatus status)
 if (g_mime_signature_status_bad (status))
return "bad";
 
-if (g_mime_signature_status_error (status))
-   return "error";
-
 if (g_mime_signature_status_good (status))
return "good";
 
-return "unknown";
+return "error";
 }
 
 /* Print signature flags */
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: tip: how to not forget attachments

[Antoine Beaupré]

On 2018-03-19 17:40:40, Brian Sniffen wrote:
> Throw your function name, catch it outside the save-excursion, and raise an 
> error there?

You mean to catch/throw to have save-excursion save the point correctly?
But my tests show the point is moved by something else in message-send
anyways, so I'm not sure I should even bother at that point...

Or should I?

a.

-- 
Il faut tout un village pour élever un enfant.
- Proverbe africain
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: tip: how to not forget attachments

[Brian Sniffen]

Throw your function name, catch it outside the save-excursion, and raise an 
error there?

-- 
Brian Sniffen

> On Mar 19, 2018, at 4:16 PM, Antoine Beaupré  wrote:
> 
>> On 2018-03-19 15:57:05, Brian Sniffen wrote:
>> `error` doesn’t do any unwinding; it leaves the program state wherever it 
>> was for analysis.  You probably want throw/catch, as described at 
>> https://www.gnu.org/software/emacs/manual/html_node/elisp/Catch-and-Throw.html#Catch-and-Throw
> 
> Wait, but what tag would I throw? message-send doesn't do any catching
> around the hook calls...
> 
> a.
> 
> -- 
> The United States is a nation of laws:
> badly written and randomly enforced.
>- Frank Zappa

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: tip: how to not forget attachments

[Antoine Beaupré]

On 2018-03-19 15:57:05, Brian Sniffen wrote:
> `error` doesn’t do any unwinding; it leaves the program state wherever it was 
> for analysis.  You probably want throw/catch, as described at 
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Catch-and-Throw.html#Catch-and-Throw

Wait, but what tag would I throw? message-send doesn't do any catching
around the hook calls...

a.

-- 
The United States is a nation of laws:
badly written and randomly enforced.
- Frank Zappa
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: bug: "no top level messages" crash on Zen email loops

[David Bremner]

Antoine Beaupré  writes:

> On 2018-03-19 13:36:49, David Bremner wrote:
>>
>> I can't duplicate that part.  
>
> That's very strange. I can reproduce this on my workstation here, but
> taking the tarball I sent in the original message, I can't reproduce
> anymore. So something changed! I suspect it's the "flags" on the
> message. I have "F" everywhere because I'm experimenting with syncing
> (badly) my inbox tag everywhere, through the flagged tag. All post-new
> hooks stuff that shouldn't affect this because it's in a new
> environment, but it does change the flag on the files sometimes.
>
> So attached is a *new* reproducer, with which I *can* reproduce in a
> clean VM with notmuch from stretch (0.23?).

I can confirm this reproduces both the xapian-check and the notmuch-show
error. Olly agrees that whatever notmuch is doing wrong, it shouldn't
lead to a corrupted database (unless we reach around the API and access
files directly, which I don't think we do).

d






___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: tip: how to not forget attachments

[Brian Sniffen]

`error` doesn’t do any unwinding; it leaves the program state wherever it was 
for analysis.  You probably want throw/catch, as described at 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Catch-and-Throw.html#Catch-and-Throw

-- 
Brian Sniffen

> On Mar 19, 2018, at 3:25 PM, Antoine Beaupré  wrote:
> 
>> On 2018-03-19 13:56:54, Antoine Beaupré wrote:
>> PS: don't we have a "you forgot to actually attach the damn file" plugin
>> when we detect the word "attachment" and there's no attach? :p
> 
> So I figured that one out, I think. Before adding it to the wiki, I'd
> like a review of the code (attached) from more adept elisp programmers.
> 
> I'm particularly surprised that save-excursion doesn't work the way I
> expect: when I answer "no" to the question, I go back to the email
> buffer, but the point is invariably at the end of the buffer, whereas I
> would expect it to be where it was when I send the message. It looks
> like something else moves the mark before my hook, but I'm not sure
> what...
> 
> How else than (error) or (keyboard-quit) am I supposed to abort email
> sending? (message-send) uses the latter but it would seem better to use
> an actual error message than to just "beep" our way out here..
> 
> Other advice? (save-excursion) + (goto-char (point-min)) +
> (re-search-forward), is that idiomatic? or is there something more
> clever that should be done?
> 
> thanks!
> 
> -- 
> Tu connaîtras la vérité de ton chemin à ce qui te rend heureux.
>- Aristote
> 
> ___
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

tip: how to not forget attachments

[Antoine Beaupré]

On 2018-03-19 13:56:54, Antoine Beaupré wrote:
> PS: don't we have a "you forgot to actually attach the damn file" plugin
> when we detect the word "attachment" and there's no attach? :p

So I figured that one out, I think. Before adding it to the wiki, I'd
like a review of the code (attached) from more adept elisp programmers.

I'm particularly surprised that save-excursion doesn't work the way I
expect: when I answer "no" to the question, I go back to the email
buffer, but the point is invariably at the end of the buffer, whereas I
would expect it to be where it was when I send the message. It looks
like something else moves the mark before my hook, but I'm not sure
what...

How else than (error) or (keyboard-quit) am I supposed to abort email
sending? (message-send) uses the latter but it would seem better to use
an actual error message than to just "beep" our way out here..

Other advice? (save-excursion) + (goto-char (point-min)) +
(re-search-forward), is that idiomatic? or is there something more
clever that should be done?

thanks!

-- 
Tu connaîtras la vérité de ton chemin à ce qui te rend heureux.
- Aristote


notmuch-buddha.el
Description: application/emacs-lisp
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: bug: "no top level messages" crash on Zen email loops

[Antoine Beaupré]

And obviously I forget the frigging attachment.



zendesk-email-loop2.tgz
Description: application/gtar-compressed

PS: don't we have a "you forgot to actually attach the damn file" plugin
when we detect the word "attachment" and there's no attach? :p
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: bug: "no top level messages" crash on Zen email loops

[Antoine Beaupré]

On 2018-03-19 13:36:49, David Bremner wrote:
> Antoine Beaupré  writes:
>
>> Hi!
>>
>> Here's a fun bug for you Xapian tricksters.
>>
>> Two emails attached make notmuch crash when trying to display the
>> folder.
>>
>> $ notmuch show thread:0001
>> Internal error: Thread 0001 has no toplevel messages.
>>  (notmuch-show.c:1012)
>>
>> Those are the two messages:
>>
>> $ notmuch search --output messages  thread:0001
>> id:9379qm5z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sp...@zendesk.com
>> id:9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com
>>
>> `notmuch show` on either messages crashes the same way:
>>
>> $ notmuch show 
>> id:9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com
>> Internal error: Thread 0001 has no toplevel messages.
>>  (notmuch-show.c:1012)
>
> I can't duplicate that part.  

That's very strange. I can reproduce this on my workstation here, but
taking the tarball I sent in the original message, I can't reproduce
anymore. So something changed! I suspect it's the "flags" on the
message. I have "F" everywhere because I'm experimenting with syncing
(badly) my inbox tag everywhere, through the flagged tag. All post-new
hooks stuff that shouldn't affect this because it's in a new
environment, but it does change the flag on the files sometimes.

So attached is a *new* reproducer, with which I *can* reproduce in a
clean VM with notmuch from stretch (0.23?).

To reproduce, with a `debian/stretch64` vagrant VM:

host$ vagrant init debian/stretch64 && vagrant up && vagrant ssh
guest$ sudo apt install notmuch
guest$ notmuch setup # pick all defaults
guest$ wget $url_of_the_reproducer
guest$ tar zxfv zendesk-mail-loop2.tgz
guest$ mv gitlab mail # to put it where notmuch expects
guest$ notmuch new
guest$ notmuch show thread:0001
Internal error: Thread 0001 has no toplevel messages.
 (notmuch-show.c:957)

I can reproduce this reproducibly here now.

Phew, that is definitely weird! For what it's worth, here's the diff
between the two tarballs:

[429]anarcat@curie:~1$ diffoscope zendesk-email-loop.tgz zendesk-email-loop2.tgz
 
||
  100% Time: 0:00:00 
--- zendesk-email-loop.tgz
+++ zendesk-email-loop2.tgz
├── metadata
│ @@ -1 +1 @@
│ -gzip compressed data, last modified: Mon Mar 19 13:21:40 2018, from Unix
│ +gzip compressed data, last modified: Mon Mar 19 17:38:29 2018, from Unix
│   --- zendesk-email-loop.tgz-content
├── +++ zendesk-email-loop2.tgz-content
├── file list
│ │ @@ -1,5 +1,5 @@
│ │ -drwx--   0 anarcat   (1000) anarcat   (1000)0 2018-03-19 
13:11:45.00 gitlab/cur/
│ │ --rw---   0 anarcat   (1000) anarcat   (1000) 8858 2018-03-14 
00:27:37.00 gitlab/cur/1521465105.R3423354954039434325.curie:2,FS
│ │ --rw---   0 anarcat   (1000) anarcat   (1000)11861 2018-03-19 
08:50:10.00 gitlab/cur/1521464914.R16228666356894086807.curie:2,F
│ │ +drwx--   0 anarcat   (1000) anarcat   (1000)0 2018-03-19 
17:35:32.00 gitlab/cur/
│ │ +-rw---   0 anarcat   (1000) anarcat   (1000) 8858 2018-03-14 
00:27:37.00 gitlab/cur/1521463753.R9368947314807690338.curie:2,FS
│ │ +-rw---   0 anarcat   (1000) anarcat   (1000) 8720 2018-03-14 
00:30:59.00 gitlab/cur/1521463752.R13151765805797588408.curie:2,FS
│ │  drwx--   0 anarcat   (1000) anarcat   (1000)0 2018-03-19 
12:49:12.00 gitlab/new/
│ │ -drwx--   0 anarcat   (1000) anarcat   (1000)0 2018-03-19 
13:11:45.00 gitlab/tmp/
│ │ +drwx--   0 anarcat   (1000) anarcat   (1000)0 2018-03-19 
12:49:13.00 gitlab/tmp/
│   --- gitlab/cur/1521465105.R3423354954039434325.curie:2,FS
├── +++ gitlab/cur/1521463753.R9368947314807690338.curie:2,FS

ie. the files are identical, but the serial numbers, timestamps and
flags differ. Maybe this makes the directory ordering (so the load order
in notmuch new) differ? No idea.

But hopefully this will allow you to reproduce more reliably.

A.

-- 
La seule excuse de Dieu, c'est qu'il n'existe pas.
- Stendhal
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: bug: "no top level messages" crash on Zen email loops

[David Bremner]

Antoine Beaupré  writes:

> Hi!
>
> Here's a fun bug for you Xapian tricksters.
>
> Two emails attached make notmuch crash when trying to display the
> folder.
>
> $ notmuch show thread:0001
> Internal error: Thread 0001 has no toplevel messages.
>  (notmuch-show.c:1012)
>
> Those are the two messages:
>
> $ notmuch search --output messages  thread:0001
> id:9379qm5z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sp...@zendesk.com
> id:9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com
>
> `notmuch show` on either messages crashes the same way:
>
> $ notmuch show 
> id:9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com
> Internal error: Thread 0001 has no toplevel messages.
>  (notmuch-show.c:1012)

I can't duplicate that part.  

>
> Note that displaying the messages weith `--format raw` doesn't crash, so
> it's really the thread structure that's broken. Obviously, emacs can't
> display the messages either and doesn't touch the unread tags when
> trying to load the message, which is to be expected I guess.
>
> Xapian is also unhappy with the database created by notmuch new:
>
> $ xapian-check gitlab/.notmuch/xapian/
> docdata:
> blocksize=8K items=1 firstunused=1 revision=7 levels=0 root=0
> B-tree checked okay
> docdata table structure checked OK
>
> termlist:
> blocksize=8K items=12 firstunused=4 revision=7 levels=0 root=3
> xapian-check: DatabaseError: 1 unused block(s) missing from the free list, 
> first is 0

Surprisingly (to me), I can duplicate that, so that's something to
pursue.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

bug: "no top level messages" crash on Zen email loops

[Antoine Beaupré]

Hi!

Here's a fun bug for you Xapian tricksters.

Two emails attached make notmuch crash when trying to display the
folder.

$ notmuch show thread:0001
Internal error: Thread 0001 has no toplevel messages.
 (notmuch-show.c:1012)

Those are the two messages:

$ notmuch search --output messages  thread:0001
id:9379qm5z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sp...@zendesk.com
id:9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com

`notmuch show` on either messages crashes the same way:

$ notmuch show 
id:9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com
Internal error: Thread 0001 has no toplevel messages.
 (notmuch-show.c:1012)

Note that displaying the messages weith `--format raw` doesn't crash, so
it's really the thread structure that's broken. Obviously, emacs can't
display the messages either and doesn't touch the unread tags when
trying to load the message, which is to be expected I guess.

Xapian is also unhappy with the database created by notmuch new:

$ xapian-check gitlab/.notmuch/xapian/
docdata:
blocksize=8K items=1 firstunused=1 revision=7 levels=0 root=0
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=12 firstunused=4 revision=7 levels=0 root=3
xapian-check: DatabaseError: 1 unused block(s) missing from the free list, 
first is 0

Valgrind is not particularly unhappy with notmuch, so it doesn't seem
like a memory error:

==26723== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

I tried to track this down in gdb, i got as far as finding that, in
`notmuch_thread_get_toplevel_messages`, the `list` object is corrupt (?)
already (`list->head == NULL`) which obviously makes it hard to, er,
list messages in a thread. :p I lost the exact backtrace and so on, but
I'm not sure there's much we can get from gdb: it seems the problem
might be in notmuch-new, but I'm a little out of my depth to debug
*that* without any further pointers.

This is with 0.26-1~bpo9+1 on Debian stretch, but I can also reproduce
with 0.23 on another Debian stretch machine, using a similar mail
spool.

My guess is that those messages are somewhat special: notice how the
reply-to identifiers *loop* between the two messages?

Message one:

Message-ID: <9379qm5z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sp...@zendesk.com>
In-Reply-To: <9379qm5...@zendesk.com>
 <9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com>

Message two:

Message-ID: <9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com>
In-Reply-To: <9379qm5...@zendesk.com>
 <9379qm5z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sp...@zendesk.com>

And indeed, a mailbox with only *one* of those messages doesn't cause
the crash. But also: the original thread is now made of *three*
messages, and taking any one of the two messages above with that *third*
message doesn't cause the crash:

Message three:

Message-ID: <9379qm5z39_5aaf79c126a_94233ffb30ecb9982187c0_sp...@zendesk.com>
In-Reply-To: <9379qm5...@zendesk.com>
 <9379qm5z39_5aa86b1350504_174eb3fc97a2cb98c71674_sp...@zendesk.com>

This message also shows correctly threaded with message two if present,
otherwise the thread is (obviously) broken with only message one and
three.

Mutt displays those messages as a "normal" three-level thread:

   1   ! mar 14 GitLab Support  (4,7K) Your GitLab support request has been 
received
   2   ! mar 14 GitLab Support  (4,5K) └>comments not showing up?
   3 O ! mar 19 XXX (7,9K)  └>[GitLab, Inc.] Re: comments not 
showing up?

The numbers on the left (1, 2, 3) correspond to the labeling I used
above as well (one, two, three).

The third message is not included here because it's an actual reply from
a human from GitLab (yay gitlab! :) which I'd need approval before
sharing here. The first message is an automated response so I thought it
was fair game to share publicly. The second is a copy of my own message
which triggered the autoreply, which is probably the source of the
loop. The software generating this mess is Zendesk.com. I haven't had
that problem with other interactions with Zendesk, maybe because I
never talked with a Zendesk that sent autoreplies.

To reproduce this, untar the attachment anywhere (say $HOME) and then
hack a notmuch config file pointing there, e.g.:

$ diff .notmuch-config*
15c15
< path=/home/anarcat/Maildir/
---
> path=/home/anarcat/gitlab/

Then point notmuch to that config (export
NOTMUCH_CONFIG=~/.notmuch-config-test) and run notmuch new (which should
find only two messages). Then run the commands from the above of this
email, of course. :)

Thanks for any input,

A.

PS: I must say I am grateful and impressed by the reliability of
notmuch. I've been using notmuch for *years* now and it's the *first*
time, for as long as I remember, that I had to go back to mutt to read
email. So kudos to the team, good job. :)

-- 
Si les élections n'étaient pas indispensables à la prospérité du
capital, on ne nous 

Re: multilingual notmuch (and Content-Language)

[Daniel Kahn Gillmor]

On Sun 2018-03-18 21:32:35 +0200, Jani Nikula wrote:
> On Sun, 18 Mar 2018, Daniel Kahn Gillmor  wrote:
>>  * if we know our index expects english, and we have a message part that
>>*is not* english (e.g. Content-Language: es), we could avoid indexing
>>that part.
>
> Why would we do that? Search mostly works just fine for non-English
> languages, it's just that the *stemming* is not right.
>
>> what do you think?  what ideas are missing from the branstorm above?  I'd
>> love to hear from people with multilingual mailboxes about how we might
>> be able to make notmuch work better for them.
>
> With my limited understanding of this, stemming happens both at indexing
> and searching. Basically at indexing, the term generator indexes both
> the full and the stemmed version of words. I'm wondering if we could
> look at Content-Language (and missing that, heuristics), and (if the
> user so desires) use multiple term generators with different stemmers on
> a per document basis. Or, use non-stemming indexing for unidentified or
> unsupported languages. How far would that take us? Then, perhaps, we
> could also perform language specific queries?
>
> I don't know how feasible that is, or if it would require Xapian
> changes.

thanks, this is exactly the kind of promising idea i was hoping my dumb
questions and half-baked suggestions would provoke :)

Maybe Olly or someone else with deeper knowledge of xapian can weigh in
about the feasibility of this proposal?

  --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch