Re: trouble with my_hdr and $format_flowed [SOLVED]

[Cameron Simpson]

On 02Sep2022 15:27, Cameron Simpson  wrote:

On 01Sep2022 20:18, Kevin J. McCarthy  wrote:

I'm not sure if it helps at this point, but I can report that I've
taken your template above and passed it to 'mutt -H' and the resulting
sent file has format=flowed in its content-type header.  :-/

Can you try directly doing this too, outside of your tmux-editing
framework?


Yes. I should have done that already. [...]
I'll be back after I've done some cleaner experiments.


So, a bare "mutt -H" here acts like yours, honouring the Content-Type 
header. This points the finger directly at my script, where I discovered 
a problem.


First up, it is necessary to hand apply a Content-Type header to the 
template file (as implied by the $text_flowed documentation).


Second, my script was:
- grabbing the template filename from the command line as $filename
- making the copy to be used by the tmux session's "mutt -H" as $tmpf
- patching the copy with a Content-Type header
- assigning filename=$tmpf after the copy was complete, for use in the 
  following code


During debugging I'd stuck that in a subshell, a habit when I turn on 
"set -x" in a script, thus:


blah blah ...
( set -x
  code beging traced
  ...
  ...
)

So the filename= assignment was lost, and the subsequent run-in-tmux 
code was using the unpatched no-Content-Type file.


Fixed and deployed.

This message should be format=flowed.

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 01Sep2022 20:18, Kevin J. McCarthy  wrote:
>On Fri, Sep 02, 2022 at 08:28:01AM +1000, Cameron Simpson wrote:
>>and the file ~/var/mutt/mutt-fleet2-501-83365-16591433053586932493
>>passwd to "mutt -H" is already gone. Does mutt remove it?
>>
>>Ah, mutt might have move it to ~/var/mutt/compose/muttedit.94328, which
>>exists with the header above.
>>
>>Let's look at the Content-Type on this message...
>
>I'm not sure if it helps at this point, but I can report that I've 
>taken your template above and passed it to 'mutt -H' and the resulting 
>sent file has format=flowed in its content-type header.  :-/
>
>Can you try directly doing this too, outside of your tmux-editing 
>framework?

Yes. I should have done that already. I'm running Mutt 2.2.7 
(2022-08-07) BTW, not that I expect any of this to have recent changes.

I'll be back after I've done some cleaner experiments.

Thanks,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Kevin J. McCarthy]

On Fri, Sep 02, 2022 at 08:28:01AM +1000, Cameron Simpson wrote:

and the file ~/var/mutt/mutt-fleet2-501-83365-16591433053586932493
passwd to "mutt -H" is already gone. Does mutt remove it?

Ah, mutt might have move it to ~/var/mutt/compose/muttedit.94328, which
exists with the header above.

Let's look at the Content-Type on this message...


I'm not sure if it helps at this point, but I can report that I've taken 
your template above and passed it to 'mutt -H' and the resulting sent 
file has format=flowed in its content-type header.  :-/


Can you try directly doing this too, outside of your tmux-editing 
framework?


--
Kevin J. McCarthy
GPG Fingerprint: 8975 A9B3 3AA3 7910 385C  5308 ADEF 7684 8031 6BDA


signature.asc
Description: PGP signature

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 02Sep2022 08:15, Cameron Simpson  wrote:
>Trying that with this reply.

Well, that tossed my Content-Type header and sent with:

Content-Type: text/plain; charset=us-ascii

I'm trying again here, for thoroughness. The source file starts like 
this:

Content-Type: text/plain; charset=utf-8; format=flowed
From: Cameron Simpson 
To: mutt-users@mutt.org
Cc:
Bcc: c...@cskk.id.au
Subject: Re: trouble with my_hdr and $format_flowed
Reply-To:
In-Reply-To: 

and the Content-Type is not presented in this current editor.

Interestingly, the process tree looks like this:

cameron 94359  1116 4299624  1220   \_ /bin/sh ~/bin/rmafter 
~/var/mutt/mutt-fleet2-501-83365-16591433053586932493 mutt -e set 
editor=vim-flowed -e unset signature -e set sidebar_visible=no -H 
~/var/mutt/mutt-fleet2-501-83365-16591433053586932493
cameron 94361 94359 4345688 11628 \_ mutt -e set editor=vim-flowed -e 
unset signature -e set sidebar_visible=no -H 
~/var/mutt/mutt-fleet2-501-83365-16591433053586932493
cameron 94383 94361 4403212 12656   \_ vim -c silent 1,/^$/s/ *$// -c 
set filetype=mail -c set formatoptions=walqj 
~/var/mutt/mutt-fleet2-501-94361-4463746866273406678

and the file ~/var/mutt/mutt-fleet2-501-83365-16591433053586932493 
passwd to "mutt -H" is already gone. Does mutt remove it?

Ah, mutt might have move it to ~/var/mutt/compose/muttedit.94328, which 
exists with the header above.

Let's look at the Content-Type on this message...

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 01Sep2022 09:09, Kevin J. McCarthy  wrote:
>On Wed, Aug 31, 2022 at 06:45:08PM -0700, Kevin J. McCarthy wrote:
>>In my testing, a header 'Content-Type: text/plain; format=flowed' in 
>>the draft file does result in the sent email being format=flowed.
>
>My example wasn't great: you *should* add a charset parameter set to 
>your system charset (I'm assuming that's utf-8).

Trying that with this reply.

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Kevin J. McCarthy]

On Wed, Aug 31, 2022 at 06:45:08PM -0700, Kevin J. McCarthy wrote:
In my testing, a header 'Content-Type: text/plain; format=flowed' in 
the draft file does result in the sent email being format=flowed.


My example wasn't great: you *should* add a charset parameter set to 
your system charset (I'm assuming that's utf-8).  For replies, Mutt 
generated the quoted contents of the reply in your $charset, and it uses 
the charset parameter when reading in the draft file.


If it's unset, Mutt will use $assumed_charset or ascii - which would 
likely result in '?' characters for non-ascii characters in the reply.


--
Kevin J. McCarthy
GPG Fingerprint: 8975 A9B3 3AA3 7910 385C  5308 ADEF 7684 8031 6BDA


signature.asc
Description: PGP signature

Re: trouble with my_hdr and $format_flowed

[Kevin J. McCarthy]

On Thu, Sep 01, 2022 at 08:50:33AM +1000, Cameron Simpson wrote:

On 31Aug2022 15:37, Kevin J. McCarthy  wrote:

I think this may work best for you, if the my_hdr approach doesn't work.


It seems to have no effect; the Content-Type eg from my reply just now
to you is still not format=flowed. I did discover that an illegal
Content-Type causes the "mutt -H" to abort.


In my testing, a header 'Content-Type: text/plain; format=flowed' in the 
draft file does result in the sent email being format=flowed.


Perhaps something else is going on with the config or details of how you 
are doing this?


--
Kevin J. McCarthy
GPG Fingerprint: 8975 A9B3 3AA3 7910 385C  5308 ADEF 7684 8031 6BDA


signature.asc
Description: PGP signature

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 01Sep2022 08:50, Cameron Simpson  wrote:
>>If you haven't played with it before, you might also look into
>>background editing and the contrib/bgedit-screen-tmux.sh script.  See
>>.
>
>I will; the background editing feature postdated my tmux games.

I don't think these things can do what I want.

The background editing feature seems to let you use an asynchronous 
editor (a GUI or a tmux/screen session with suitable 
wait-for-completion) and return immediately to mutt. Then have the 
primary (the only) mutt do the post-edit sending when the editor 
finishes. While that would let me, say, detach the tmux session and 
resume it elsewhere, it wouldn't let me quit the primary mutt.

My approach runs an entirely separate mutt to edit-then-send the 
message, and runs that as the "editor" in the primary mutt. I can finish 
in the tmux session and send, and it feels exactly like invoking vim 
directly from mutt. Or I can detach from the session, resume the primary 
mutt, do whatever, and _quit_ the primary mutt. The tmux session lurks 
around, ready for resumption later.

For example, as I type this my tmux listing contains:

 14 mutt-01sep2022-09_09-Re__trouble_with_my_hdr_and_format_flowed: 1 
windows (created Thu Sep  1 09:09:57 2022) (attached)

which is running "mutt -H -copy-of-the-original".

So I'm still left with the problem: how to invoke a "mutt -H" to send a 
format=flowed message. Still striking out with that so far.

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 31Aug2022 15:37, Kevin J. McCarthy  wrote:
>On Thu, Sep 01, 2022 at 08:11:50AM +1000, Cameron Simpson wrote:
>>The essential problem is that I run a separate mutt instance using "mutt
>>-H" on a copy of my message template. The manual does say that
>>$text_flowed has no effect then, but I seem to have no other way to
>>apply the needed header - adding it by hand during composition is
>>ineffective.
>>
>>I _might_ do better adding it to the copy before starting the separate
>>mutt; I'll see if that works.
>
>I think this may work best for you, if the my_hdr approach doesn't work.

It seems to have no effect; the Content-Type eg from my reply just now 
to you is still not format=flowed. I did discover that an illegal 
Content-Type causes the "mutt -H" to abort.

>Content-Type and other MIME headers aren't included in the "edit 
>headers" generated headers, and they aren't parsed afterwards either.

Ok. I can see why that might be done, since in effect I'm editing a text 
attachment in some ways. I'm a little surprised one can't modify 
Content-Type though. That seems like something which would make sense in 
the "edit headers"; it feels like an "in the attachment" header. What if 
I wanted to send markdown? (I know there's a whole thread about this 
somewhere, to which I didn't pay enough attention at the time; I'll find 
it).

>If you haven't played with it before, you might also look into 
>background editing and the contrib/bgedit-screen-tmux.sh script.  See 
>.

I will; the background editing feature postdated my tmux games.

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 31Aug2022 15:28, Kevin J. McCarthy  wrote:
>The parser for my_hdr is a bit different - it allows quote characters 
>in the header, so it doesn't treat them specially like elsewhere.  
>However, semicolons are still parsed normally.

Ah, right.

>I think this was a mistake, but would be a breaking change to fix now.

Aye.

>Anyway, a workaround would be to put a '\' in front of the semicolons:
>  my_hdr X-Foo: bar\; baz

Thank you!

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Kevin J. McCarthy]

On Thu, Sep 01, 2022 at 08:11:50AM +1000, Cameron Simpson wrote:

The essential problem is that I run a separate mutt instance using "mutt
-H" on a copy of my message template. The manual does say that
$text_flowed has no effect then, but I seem to have no other way to
apply the needed header - adding it by hand during composition is
ineffective.

I _might_ do better adding it to the copy before starting the separate
mutt; I'll see if that works.


I think this may work best for you, if the my_hdr approach doesn't work.

Content-Type and other MIME headers aren't included in the "edit 
headers" generated headers, and they aren't parsed afterwards either.


If you haven't played with it before, you might also look into 
background editing and the contrib/bgedit-screen-tmux.sh script.  See 
.


--
Kevin J. McCarthy
GPG Fingerprint: 8975 A9B3 3AA3 7910 385C  5308 ADEF 7684 8031 6BDA


signature.asc
Description: PGP signature

Re: trouble with my_hdr and $format_flowed

[Kevin J. McCarthy]

On Thu, Sep 01, 2022 at 08:08:21AM +1000, Cameron Simpson wrote:

I forgot the my_hdr issue. This line:

   my_hdr "Content-Type: teAxt/plain; charset=usascii; format-flowed"

results in this from from mutt at startup:

   Error in /Users/cameron/rc/mutt/settings, line 129: charset=usascii: unknown 
command

So it appears that I can't use a semicolon in my_hdr?


The parser for my_hdr is a bit different - it allows quote characters in 
the header, so it doesn't treat them specially like elsewhere.  However, 
semicolons are still parsed normally.


I think this was a mistake, but would be a breaking change to fix now.

Anyway, a workaround would be to put a '\' in front of the semicolons:
  my_hdr X-Foo: bar\; baz

--
Kevin J. McCarthy
GPG Fingerprint: 8975 A9B3 3AA3 7910 385C  5308 ADEF 7684 8031 6BDA


signature.asc
Description: PGP signature

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 31Aug2022 18:06, David Lowry-Duda  wrote:
>>I have a pretty weird way of composing replies and new messages: I like
>>to be able to detach from the new message window. So I compose in a
>>separate tmux session.
>
>If you compose directly in mutt+vim without composing in a separate 
>tmux window, does it work? (I would assume so, as I use mutt+vim and 
>format flowed works fine).

Yes.

>I don't quite understand how your composing system works well enough to 
>have a more meaningful comment.

The essential problem is that I run a separate mutt instance using "mutt 
-H" on a copy of my message template. The manual does say that 
$text_flowed has no effect then, but I seem to have no other way to 
apply the needed header - adding it by hand during composition is 
ineffective.

I _might_ do better adding it to the copy before starting the separate 
mutt; I'll see if that works.

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

I forgot the my_hdr issue. This line:

my_hdr "Content-Type: text/plain; charset=usascii; format-flowed"

results in this from from mutt at startup:

Error in /Users/cameron/rc/mutt/settings, line 129: charset=usascii: 
unknown command

So it appears that I can't use a semicolon in my_hdr?

Not that it would help me, since mutt seems to eat that header anyway :-(

Cheers,
Cameron Simpson 

Re: trouble with my_hdr and $format_flowed

[David Lowry-Duda]

I have a pretty weird way of composing replies and new messages: I like
to be able to detach from the new message window. So I compose in a
separate tmux session.


If you compose directly in mutt+vim without composing in a separate tmux 
window, does it work? (I would assume so, as I use mutt+vim and format 
flowed works fine).


I don't quite understand how your composing system works well enough to 
have a more meaningful comment.


- DLD

Re: trouble with my_hdr and $format_flowed

[Cameron Simpson]

On 01Sep2022 07:58, Cameron Simpson  wrote:
>I've put a format=flowed into this message by hand, and I'll see if it
>survives.

It did not. The message sent by mutt discarded my Content-Type header 
and used:

Content-Type: text/plain; charset=us-ascii

So I've actually got a significant problem here, as modifying my wrapper 
script to provide the header isn't enough.

Cheers,
Cameron Simpson 

trouble with my_hdr and $format_flowed

[Cameron Simpson]

I've long had my mutt+vim setup to compose format=flowed email. There's 
just one problem: it wasn't working. So I'm digging into why.

I've got:

set text_flowed=yes

in my setup. And if I pare down enough mutt settings, my reply goes out 
with this header:

Content-Type: text/plain; charset=us-ascii; format=flowed

which is what I want (I've got send_charset="us-ascii:utf-8" so I 
presume if I put some nonascii in then mutt would bump that to "utf-8").

But without paring things down, I do not get that header.

I have a pretty weird way of composing replies and new messages: I like 
to be able to detach from the new message window. So I compose in a 
separate tmux session. That is done via a special script I use for 
$editor, which essentially goes:
- copy the message to a temp file
- start a tmux session running "mutt -H the-temp-file"
- quit

Because mutt silently drops back (and doesn't send) if the new message 
didn't get modified, this gives me a tmux session where I compose the 
message and send it, and then mutt silently discards my original, and 
I'm back in the index or pager as before. This is seamless.

So what's wrong?

It appears that mutt does not put "format=flowed" in the "Content-Type" 
header until message send time. The message I see in the editor before 
compose does not have the header.

I think the fix would be to presupply the "Content-Type" header. I also 
think such a fix might break the send_charset magic, unless mutt picks 
up an existing Content-Type header, parses it, and fills in the 
parameters if missing.

I've put a format=flowed into this message by hand, and I'll see if it 
survives.

Thoughts?
Cameron Simpson