[PATCH 1/3] emacs: Introduce `notmuch-call-notmuch-sexp'

2013-06-24 Thread Mark Walters

That is fine.

(I think I only moved your code from one file to another but in any case
what you suggest is fine.)

Best wishes

Mark


On Mon, 24 Jun 2013, Austin Clements  wrote:
> Yes, and thanks for reminding me about the require's.  I'll do that in
> a follow up.  I was thinking of just moving the streaming JSON parser
> to a repo on my GitHub account, since maybe somebody some day will
> find a use for it, or at least take inspiration from the API (I looked
> into streaming JSON parser APIs before embarking on that one and
> they're all terrible!)  Mark, you're the only other person who has
> touched that code.  Is this plan good with you?
>
> Quoth Mark Walters on Jun 24 at  8:00 pm:
>> 
>> This series looks good to me +1. 
>> 
>> Is it worth removing all the json (3 files with (require 'json) and the
>> async json parser) too?
>> 
>> Best wishes
>> 
>> Mark
>> 
>> On Mon, 24 Jun 2013, Austin Clements  wrote:
>> > This is just like `notmuch-call-notmuch-json', but parses S-expression
>> > output.  Note that, also like `notmuch-call-notmuch-json', this
>> > doesn't consider trailing data to be an error, which may or may not be
>> > what we want in the long run.
>> > ---
>> >  emacs/notmuch-lib.el |   17 +
>> >  1 file changed, 17 insertions(+)
>> >
>> > diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
>> > index 534f217..36eacc1 100644
>> > --- a/emacs/notmuch-lib.el
>> > +++ b/emacs/notmuch-lib.el
>> > @@ -484,6 +484,23 @@ an error."
>> >  (json-read)))
>> >(delete-file err-file)
>> >  
>> > +(defun notmuch-call-notmuch-sexp ( args)
>> > +  "Invoke `notmuch-command' with ARGS and return the parsed S-exp output.
>> > +
>> > +If notmuch exits with a non-zero status, this will pop up a
>> > +buffer containing notmuch's output and signal an error."
>> > +
>> > +  (with-temp-buffer
>> > +(let ((err-file (make-temp-file "nmerr")))
>> > +  (unwind-protect
>> > +(let ((status (apply #'call-process
>> > + notmuch-command nil (list t err-file) nil args)))
>> > +  (notmuch-check-exit-status status (cons notmuch-command args)
>> > + (buffer-string) err-file)
>> > +  (goto-char (point-min))
>> > +  (read (current-buffer)))
>> > +  (delete-file err-file)
>> > +
>> >  (defun notmuch-start-notmuch (name buffer sentinel  args)
>> >"Start and return an asynchronous notmuch command.


[PATCH v7b] cli: add insert command

2013-06-24 Thread Mark Walters

Just to confirm my +1 for this series with patch 7b.

Mark

On Sun, 23 Jun 2013, Peter Wang  wrote:
> The notmuch insert command reads a message from standard input,
> writes it to a Maildir folder, and then incorporates the message into
> the notmuch database.  Essentially it moves the functionality of
> notmuch-deliver into notmuch.
>
> Though it could be used as an alternative to notmuch new, the reason
> I want this is to allow my notmuch frontend to add postponed or sent
> messages to the mail store and notmuch database, without resorting to
> another tool (e.g. notmuch-deliver) nor directly modifying the maildir.
> ---
>  Makefile.local   |   1 +
>  notmuch-client.h |   3 +
>  notmuch-insert.c | 337 
> +++
>  notmuch.c|   2 +
>  4 files changed, 343 insertions(+)
>  create mode 100644 notmuch-insert.c
>
> diff --git a/Makefile.local b/Makefile.local
> index 644623f..84043fe 100644
> --- a/Makefile.local
> +++ b/Makefile.local
> @@ -261,6 +261,7 @@ notmuch_client_srcs = \
>   notmuch-config.c\
>   notmuch-count.c \
>   notmuch-dump.c  \
> + notmuch-insert.c\
>   notmuch-new.c   \
>   notmuch-reply.c \
>   notmuch-restore.c   \
> diff --git a/notmuch-client.h b/notmuch-client.h
> index 4a3c7ac..dfe81e6 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -181,6 +181,9 @@ int
>  notmuch_new_command (notmuch_config_t *config, int argc, char *argv[]);
>  
>  int
> +notmuch_insert_command (notmuch_config_t *config, int argc, char *argv[]);
> +
> +int
>  notmuch_reply_command (notmuch_config_t *config, int argc, char *argv[]);
>  
>  int
> diff --git a/notmuch-insert.c b/notmuch-insert.c
> new file mode 100644
> index 000..1228afa
> --- /dev/null
> +++ b/notmuch-insert.c
> @@ -0,0 +1,337 @@
> +/* notmuch - Not much of an email program, (just index and search)
> + *
> + * Copyright ? 2013 Peter Wang
> + *
> + * Based in part on notmuch-deliver
> + * Copyright ? 2010 Ali Polatel
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see http://www.gnu.org/licenses/ .
> + *
> + * Author: Peter Wang 
> + */
> +
> +#include "notmuch-client.h"
> +#include "tag-util.h"
> +
> +#include 
> +#include 
> +#include 
> +
> +static volatile sig_atomic_t interrupted;
> +
> +static void
> +handle_sigint (unused (int sig))
> +{
> +static char msg[] = "Stopping... \n";
> +
> +/* This write is "opportunistic", so it's okay to ignore the
> + * result.  It is not required for correctness, and if it does
> + * fail or produce a short write, we want to get out of the signal
> + * handler as quickly as possible, not retry it. */
> +IGNORE_RESULT (write (2, msg, sizeof (msg) - 1));
> +interrupted = 1;
> +}
> +
> +/* Like gethostname but guarantees that a null-terminated hostname is
> + * returned, even if it has to make one up. Invalid characters are
> + * substituted such that the hostname can be used within a filename.
> + */
> +static void
> +safe_gethostname (char *hostname, size_t len)
> +{
> +char *p;
> +
> +if (gethostname (hostname, len) == -1) {
> + strncpy (hostname, "unknown", len);
> +}
> +hostname[len - 1] = '\0';
> +
> +for (p = hostname; *p != '\0'; p++) {
> + if (*p == '/' || *p == ':')
> + *p = '_';
> +}
> +}
> +
> +/* Call fsync() on a directory path. */
> +static notmuch_bool_t
> +sync_dir (const char *dir)
> +{
> +notmuch_bool_t ret;
> +int fd;
> +
> +fd = open (dir, O_RDONLY);
> +if (fd == -1) {
> + fprintf (stderr, "Error: open() dir failed: %s\n", strerror (errno));
> + return FALSE;
> +}
> +ret = (fsync (fd) == 0);
> +if (! ret) {
> + fprintf (stderr, "Error: fsync() dir failed: %s\n", strerror (errno));
> +}
> +close (fd);
> +return ret;
> +}
> +
> +/* Open a unique file in the 'tmp' sub-directory of dir.
> + * Returns the file descriptor on success, or -1 on failure.
> + * On success, file paths for the message in the 'tmp' and 'new'
> + * directories are returned via tmppath and newpath,
> + * and the path of the 'new' directory itself in newdir. */
> +static int
> +maildir_open_tmp_file (void *ctx, const char *dir,
> +char **tmppath, char **newpath, char **newdir)
> +{
> +pid_t pid;
> +char 

[PATCH 1/3] emacs: Introduce `notmuch-call-notmuch-sexp'

2013-06-24 Thread Mark Walters

This series looks good to me +1. 

Is it worth removing all the json (3 files with (require 'json) and the
async json parser) too?

Best wishes

Mark

On Mon, 24 Jun 2013, Austin Clements  wrote:
> This is just like `notmuch-call-notmuch-json', but parses S-expression
> output.  Note that, also like `notmuch-call-notmuch-json', this
> doesn't consider trailing data to be an error, which may or may not be
> what we want in the long run.
> ---
>  emacs/notmuch-lib.el |   17 +
>  1 file changed, 17 insertions(+)
>
> diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
> index 534f217..36eacc1 100644
> --- a/emacs/notmuch-lib.el
> +++ b/emacs/notmuch-lib.el
> @@ -484,6 +484,23 @@ an error."
> (json-read)))
>   (delete-file err-file)
>  
> +(defun notmuch-call-notmuch-sexp ( args)
> +  "Invoke `notmuch-command' with ARGS and return the parsed S-exp output.
> +
> +If notmuch exits with a non-zero status, this will pop up a
> +buffer containing notmuch's output and signal an error."
> +
> +  (with-temp-buffer
> +(let ((err-file (make-temp-file "nmerr")))
> +  (unwind-protect
> +   (let ((status (apply #'call-process
> +notmuch-command nil (list t err-file) nil args)))
> + (notmuch-check-exit-status status (cons notmuch-command args)
> +(buffer-string) err-file)
> + (goto-char (point-min))
> + (read (current-buffer)))
> + (delete-file err-file)
> +
>  (defun notmuch-start-notmuch (name buffer sentinel  args)
>"Start and return an asynchronous notmuch command.
>  
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/2] notmuch.el:notmuch-search-process-filter: Rewritten. Cope with incomplete lines.

2013-06-24 Thread Austin Clements
On Tue, 15 Nov 2011, Thomas Schwinge  wrote:
> Hi!
>
> On Thu, 10 Mar 2011 18:02:09 -0800, Carl Worth  wrote:
>> On Thu,  3 Feb 2011 00:56:39 +0100, Thomas Schwinge > schwinge.name> wrote:
>> > This issue has been lying in ambush as of 2009-11-24's commit
>> > 93af7b574598637c2766dd1f8ef343962c9a8efb.
>> 
>> Thanks very much for tracking down this bug, Thomas. What a nasty bug to
>> have in notmuch!
>> 
>> Your fix seems to drop the last thread from the search results
>> view. I've now committed a slightly modified fix that avoids that
>> problem. I also made the test case provide slightly cleaner results.
>> 
>> Let me know if you see any problems.
>
> That is much better, thanks!
>
> But we're not there yet...  %-| That is, today I hit another issue that
> appears to hide in the same elisp code.  See ``Error: Unexpected output
> From notmuch search''.  (And, thanks to eBay for long subject lines...)
>
> Unfortunately I'm totally out of time at the moment (final month of
> writing and wrapping up my diploma thesis), so I'm just dumping my state
> here, for now.

*snip*

I happened to notice that this is still marked as a bug.  I'm marking it
as fixed.  This code was completely rewritten to use the streaming JSON
parser (and now the streaming S-expression parser) and there's now a
test that feeds the process filter the search data one character at a
time to catch these sorts of buffer boundary bugs.


[PATCH 1/3] emacs: Introduce `notmuch-call-notmuch-sexp'

2013-06-24 Thread Tomi Ollila
On Mon, Jun 24 2013, Austin Clements  wrote:

> This is just like `notmuch-call-notmuch-json', but parses S-expression
> output.  Note that, also like `notmuch-call-notmuch-json', this
> doesn't consider trailing data to be an error, which may or may not be
> what we want in the long run.
> ---

This series looks good (if not trivial), works in use and tests pass. +1

Tomi


>  emacs/notmuch-lib.el |   17 +
>  1 file changed, 17 insertions(+)
>
> diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
> index 534f217..36eacc1 100644
> --- a/emacs/notmuch-lib.el
> +++ b/emacs/notmuch-lib.el
> @@ -484,6 +484,23 @@ an error."
> (json-read)))
>   (delete-file err-file)
>  
> +(defun notmuch-call-notmuch-sexp ( args)
> +  "Invoke `notmuch-command' with ARGS and return the parsed S-exp output.
> +
> +If notmuch exits with a non-zero status, this will pop up a
> +buffer containing notmuch's output and signal an error."
> +
> +  (with-temp-buffer
> +(let ((err-file (make-temp-file "nmerr")))
> +  (unwind-protect
> +   (let ((status (apply #'call-process
> +notmuch-command nil (list t err-file) nil args)))
> + (notmuch-check-exit-status status (cons notmuch-command args)
> +(buffer-string) err-file)
> + (goto-char (point-min))
> + (read (current-buffer)))
> + (delete-file err-file)
> +
>  (defun notmuch-start-notmuch (name buffer sentinel  args)
>"Start and return an asynchronous notmuch command.
>  
> -- 
> 1.7.10.4
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] emacs: Echo the output of notmuch new as it runs

2013-06-24 Thread Austin Clements
Previously, when the user pressed "G" to invoke notmuch new, Emacs
would go out to lunch until it finished, giving the user no sense that
the (potentially long-running) notmuch new process was making
progress.  This patch fixes this by continuously updating the echo
area to display the last output line of notmuch new as it runs.
---

This turned out to be a little more complex than I was expecting, but
the effect is really nice, especially if you have a slow computer.

 emacs/notmuch-lib.el |   87 ++
 emacs/notmuch.el |5 +--
 2 files changed, 90 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 534f217..5329146 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -372,6 +372,9 @@ returned by FUNC."
   (put-text-property start next prop (funcall func value) object)
   (setq start next

+;;
+;; Process helpers
+
 (defun notmuch-logged-error (msg  extra)
   "Log MSG and EXTRA to *Notmuch errors* and signal MSG.

@@ -554,6 +557,90 @@ status."
(message "%s" (error-message-string err
 (ignore-errors (delete-file err-file

+(defun notmuch-call-process-with-progress (msg-prefix program  args)
+  "Call PROGRAM with ARGS, tailing its last line in the echo area.
+
+This is useful for potentially long-running commands that print
+their progress, since it will continuously display the last line
+of the command's output in the echo area as it runs.  In other
+respects, this is very similar to `call-process': it's
+synchronous, handles quits the same way, and its return value is
+the same.
+
+MSG-PREFIX is the string to prefix echo area messages with.  If
+nil, the message will be constructed from PROGRAM."
+
+  (unless msg-prefix
+(setq msg-prefix (format "Running %s" program)))
+
+  (with-temp-buffer
+(let* (;; Inhibit quit until we're ready to handle it properly
+  (inhibit-quit t)
+  (proc
+   (let ((process-environment
+  ;; We emulate a (very lame) VT100
+  (cons "TERM=vt100" process-environment)))
+ (apply #'start-process program (current-buffer) program args)))
+  (filter
+   (lambda (proc string)
+ (when (buffer-live-p (process-buffer proc))
+   (with-current-buffer (process-buffer proc)
+ (goto-char (point-max))
+ ;; Treat both \r and \n as newline
+ (insert (replace-regexp-in-string "\r" "\n" string))
+ ;; Find the beginning of the last line with content
+ ;; (which might be the line we're on)
+ (while (and (bolp) (not (bobp)))
+   (backward-char))
+ (beginning-of-line)
+ (delete-region (point-min) (point))
+ ;; Strip VT100 control sequences.  This isn't
+ ;; perfect, but it's simple and it'll handle anything
+ ;; we're likely to see.
+ (save-excursion
+   (while (re-search-forward "\e\\[[0-9;?$]*[@a-zA-Z]" nil t)
+ (replace-match "")))
+ ;; Update the minibuffer.  The text is after the
+ ;; "..." so that Emacs will update the line in
+ ;; *Messages* rather than flooding the log.
+ (message "%s... %s" (process-get proc 'msg-prefix)
+  (buffer-substring (point) (line-end-position)))
+  (sentinel
+   (lambda (proc event)
+ ;; This is the only way to get signal names
+ (process-put proc 'sentinel-event (substring event 0 -1)
+  (process-put proc 'msg-prefix msg-prefix)
+  (set-process-filter proc filter)
+  (set-process-sentinel proc sentinel)
+  (process-send-eof proc)
+  (message "%s..." msg-prefix)
+
+  ;; Wait for termination, emulating `call-process'
+  (unwind-protect
+ (while (eq (process-status proc) 'run)
+   (let ((inhibit-quit nil))
+ (accept-process-output proc nil nil t)))
+   (when (eq (process-status proc) 'run)
+ (interrupt-process proc t)
+ (message "Waiting for process to die...(type C-g again to kill it 
instantly)")
+ (unwind-protect
+ (while (eq (process-status proc) 'run)
+   (let ((inhibit-quit nil))
+ (accept-process-output proc nil nil t)))
+   (delete-process proc))
+ (message "Waiting for process to die...done")))
+
+  ;; Print the final status message and return like `call-process'
+  (let ((event (process-get proc 'sentinel-event))
+   (status (process-status proc))
+   (exit-status (process-exit-status proc)))
+   (if (eq status 'exit)
+   (progn
+ (message "%s...%s" msg-prefix (if (= exit-status 0) 

[PATCH 1/3] emacs: Introduce `notmuch-call-notmuch-sexp'

2013-06-24 Thread Austin Clements
Yes, and thanks for reminding me about the require's.  I'll do that in
a follow up.  I was thinking of just moving the streaming JSON parser
to a repo on my GitHub account, since maybe somebody some day will
find a use for it, or at least take inspiration from the API (I looked
into streaming JSON parser APIs before embarking on that one and
they're all terrible!)  Mark, you're the only other person who has
touched that code.  Is this plan good with you?

Quoth Mark Walters on Jun 24 at  8:00 pm:
> 
> This series looks good to me +1. 
> 
> Is it worth removing all the json (3 files with (require 'json) and the
> async json parser) too?
> 
> Best wishes
> 
> Mark
> 
> On Mon, 24 Jun 2013, Austin Clements  wrote:
> > This is just like `notmuch-call-notmuch-json', but parses S-expression
> > output.  Note that, also like `notmuch-call-notmuch-json', this
> > doesn't consider trailing data to be an error, which may or may not be
> > what we want in the long run.
> > ---
> >  emacs/notmuch-lib.el |   17 +
> >  1 file changed, 17 insertions(+)
> >
> > diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
> > index 534f217..36eacc1 100644
> > --- a/emacs/notmuch-lib.el
> > +++ b/emacs/notmuch-lib.el
> > @@ -484,6 +484,23 @@ an error."
> >   (json-read)))
> > (delete-file err-file)
> >  
> > +(defun notmuch-call-notmuch-sexp ( args)
> > +  "Invoke `notmuch-command' with ARGS and return the parsed S-exp output.
> > +
> > +If notmuch exits with a non-zero status, this will pop up a
> > +buffer containing notmuch's output and signal an error."
> > +
> > +  (with-temp-buffer
> > +(let ((err-file (make-temp-file "nmerr")))
> > +  (unwind-protect
> > + (let ((status (apply #'call-process
> > +  notmuch-command nil (list t err-file) nil args)))
> > +   (notmuch-check-exit-status status (cons notmuch-command args)
> > +  (buffer-string) err-file)
> > +   (goto-char (point-min))
> > +   (read (current-buffer)))
> > +   (delete-file err-file)
> > +
> >  (defun notmuch-start-notmuch (name buffer sentinel  args)
> >"Start and return an asynchronous notmuch command.


header continuation issue in notmuch frontend/alot/pythons email module

2013-06-24 Thread Thomas Schwinge
Hi!

On Mon, 24 Jun 2013 10:57:10 +0200, Justus Winter <4winter at 
informatik.uni-hamburg.de> wrote:
> Quoting Austin Clements (2013-06-23 18:59:39)
> > Quoth Justus Winter on Jun 23 at  3:11 pm:
> > > I recently had a problem replying to a mail written by Thomas Schwinge
> > > using an oldish notmuch. Not sure if it has been fixed in more recent

"Oldish", yeah, yeah, I know...  (Mumbles someting about long TODO list.)

> > > versions, but I think notmuch could improve uppon its header
> > > generation (see below). Problematic part of the mail:
> > > 
> > > ~~~ snip ~~~
> > > [...]
> > > To: someone at example.org, "line
> > >  break" , someoneelse at example.org
> > > User-Agent: Notmuch/0.9-101-g81dad07 (http://notmuchmail.org) 
> > > Emacs/23.4.1 (i486-pc-linux-gnu)
> > > [...]
> > > ~~~ snap ~~~

> > Do you happen to know how the strangely folded "to" header was
> > produced for this message?

I just entered/copied all the addresses into one long To: line, and then
let message-mode do its thing.

> No, but Thomas might. Thomas, the problematic message is
> id:877ghpqckb.fsf at kepler.schwinge.homeip.net

Here is the header from the message as I sent it:

To: Samuel Thibault , Justus Winter
 <4winter at informatik.uni-hamburg.de>, fotis.koutoulakis at gmail.com, Ian
 Lance Taylor , toscano.pino at tiscali.it, Luis Machado
 , =?utf-8?B?6ZmG5bKz?=
 

And this is what I received from the bug-hurd mailing list:

To: Samuel Thibault , Justus Winter
<4winter at informatik.uni-hamburg.de>, , "Ian
Lance Taylor" , , Luis 
Machado
,
=?utf-8?B?6ZmG5bKz?= 

So the "corruption" (if it is declared as one; I don't have time right
now to follow your RFC interpretation) must have happened after sending
it off -- perhaps my company's Microsoft Exchange server (as Justus
received a direct copy from that one), or even msmtp used as the local
MTA.


Gr??e,
 Thomas
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20130624/0ecdd369/attachment-0001.pgp>


Emacs not finding keys to verify signatures

2013-06-24 Thread Daniel Patterson

Apologies if this is addressed somewhere in the docs - I haven't found
it!

I am testing out signing of messages. I set
(add-hook 'message-setup-hook mml-secure-message-sign-pgpmime)
As per the docs, and I can send a message and it gets signed. When I
view the message (which was Bcc'd to me) and hit $, I see this message:

[ Unknown key ID 0x2747EC48A98D4AF0 or unsupported algorithm ]

I click on it, and it requests _my_ public key from the key server, and
of course nothing changes - because I already had the key.

Any ideas? Is this because of some missing package or misconfigured
setting? I'm using the latest released version of notmuch (0.15.2),
emacs 24.2.1, and gpg 1.4.11.

Thanks!
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20130624/1adc93ca/attachment.pgp>


header continuation issue in notmuch frontend/alot/pythons email module

2013-06-24 Thread Justus Winter
Quoting Austin Clements (2013-06-23 18:59:39)
> Quoth Justus Winter on Jun 23 at  3:11 pm:
> > Hi,
> > 
> > I recently had a problem replying to a mail written by Thomas Schwinge
> > using an oldish notmuch. Not sure if it has been fixed in more recent
> > versions, but I think notmuch could improve uppon its header
> > generation (see below). Problematic part of the mail:
> > 
> > ~~~ snip ~~~
> > [...]
> > To: someone at example.org, "line
> >  break" , someoneelse at example.org
> > User-Agent: Notmuch/0.9-101-g81dad07 (http://notmuchmail.org) Emacs/23.4.1 
> > (i486-pc-linux-gnu)
> > [...]
> > ~~~ snap ~~~
> > 
> > http://tools.ietf.org/html/rfc2822#section-2.2.3 says:
> > 
> >Note: Though structured field bodies are defined in such a way that
> >folding can take place between many of the lexical tokens (and even
> >within some of the lexical tokens), folding SHOULD be limited to
> >placing the CRLF at higher-level syntactic breaks.  For instance, if
> >a field body is defined as comma-separated values, it is recommended
> >that folding occur after the comma separating the structured items in
> >preference to other places where the field could be folded, even if
> >it is allowed elsewhere.
> > 
> > So notmuch "rfc-SHOULD" place the newlines after the comma.
> > 
> > The rfc goes on:
> > 
> >The process of moving from this folded multiple-line representation
> >of a header field to its single line representation is called
> >"unfolding". Unfolding is accomplished by simply removing any CRLF
> >that is immediately followed by WSP.  Each header field should be
> >treated in its unfolded form for further syntactic and semantic
> >evaluation.
> > 
> > My interpretation is that unfolding simply removes any linebreaks
> > first, so the value does not contain any newlines. But pythons email
> > module discriminates quoted and unquoted parts of the value:
> > 
> > ~~~ snip ~~~
> > from __future__ import print_function
> > import email
> > from email.utils import getaddresses
> > 
> > m = email.message_from_string('''To: "line
> >  break" , line
> >  break ''')
> > print("m['To'] = ", m['To'])
> > print("getaddresses(m.get_all('To')) = ", getaddresses(m.get_all('To')))
> > ~~~ snap ~~~
> > 
> > % python3 test.py
> > m['To'] =  "line
> >  break" , line
> >  break 
> > getaddresses(m.get_all('To')) =  [('line\n break', 'linebreak at 
> > example.org'), ('line break', 'linebreak at example.org')]
> > 
> > I believe that is what's preventing me from replying to the message
> > using alot without sanitizing the To header first. Not really sure who
> > is wrong or right here... any thoughts?
> 
> There are at least two bugs here.  Regardless of what we RFC-should
> do, that folding *is* permitted by RFC2822, since quoted
> strings can contain folding whitespace:
> 
>   http://tools.ietf.org/html/rfc2822#section-3.2.5
> 
> For completeness, the full derivation for this "To" header is:
> 
> to  =   "To:" address-list CRLF
> address-list=   (address *("," address)) / obs-addr-list
> address =   mailbox / group
> mailbox =   name-addr / addr-spec
> name-addr   =   [display-name] angle-addr
> display-name=   phrase
> phrase  =   1*word / obs-phrase
> word=   atom / quoted-string
> quoted-string   =   [CFWS]
> DQUOTE *([FWS] qcontent) [FWS] DQUOTE
> [CFWS]
> 
> Do you happen to know how the strangely folded "to" header was
> produced for this message?

No, but Thomas might. Thomas, the problematic message is
id:877ghpqckb.fsf at kepler.schwinge.homeip.net

>  In notmuch-emacs, a user can put whatever
> they want in a message-mode buffer's headers and mm will dutifully
> pass it on to their MTA.  We could validate it, but that's a slippery
> slope and I would hope that the MTA itself is validating it (and
> probably more thoroughly than we could).
> 
> That said, the first bug here is in Python.  As I mentioned above,
> foldable whitespace is allowed in quoted strings.  In fact, though the
> standard is rather long-winded about whitespace, if you dig into the
> grammar, you'll find that *all whitespace can be folded* (except in
> the obsolete grammar, which allowed whitespace between the header name
> and the colon, which obviously can't be folded).  I'm not sure what
> Python is doing, but I bet it's going to a lot of effort to
> mis-implement something very simple.

Yes, I'm glad you came to the same conclusion.

> There also appears to be a bug in the notmuch CLI's reply command
> where it omits addresses that were folded in the original message.  I
> don't know if alot uses the CLI's reply command, so this may or may
> not be related to your specific issue.  I haven't dug into this yet,
> other than to confirm that it's the CLI's fault and not
> notmuch-emacs's.

No, alot does not use notmuchs reply command.

Thanks,
Justus