Re: Question about message id
On Thu, Apr 11, 2024 at 02:35:30PM -0700, googly.negotiator...@aceecat.org wrote: > On Thu, Apr 11, 2024 at 01:13:52PM +1000, raf via Mutt-users wrote: > - if the mailing list does mess with msgid, it absolutely must do it > consistently for all copies of the message. For existing software that has problems with '/' (because it is a path separator), the solution is actually trivially simple. On the way in, replace it with one of the "specials" that are not allowed in a message ID, but IS allowed in your system's path names. On the way out, do the reverse transformation. You need only write a simple function that does each thing, and call them once, when the data is "on the way in" (from whichever source, either the RFC2822 message form or the on-disk form). Memory is cheap these days so just save both forms in your data structure and use the right one for the given context. Or don't, the perf hit will be imperceptible to the user (but it's more typing to call the function every time you need one or the other). There are probably only a handful of places you'd need to change, if you wrote suitably modular software--even in mail software, there are few cases where the message ID is at all interesting. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. signature.asc Description: PGP signature
Re: Question about message id
On Thu, Apr 11, 2024 at 01:13:52PM +1000, raf via Mutt-users wrote: > On Tue, Apr 09, 2024 at 10:53:41AM -0400, Derek Martin > wrote: > > > On Sun, Apr 07, 2024 at 01:19:09PM +, Ебрашка wrote: > > > Question, what should I write in .muttrc to make my outgoing > > > mails have the same beautiful message-ID as Yandex mail? > > > > The unfathomable thing about this question is why you (or anyone) > > should care in the slightest what your message ID looks like. > > It's an esoteric detail about e-mail transfer, the specific > > contents of which have no value to users [...] > The link to a kernel mailing list message that was provided earlier > in this discussion said that the choice to use base64 results in the > possibility of / characters being included in the message id which > causes problems for their archived messages accessible via a web > browser. Yeah, but... > So it seems that there is a reason to care about this. I disagree, both for the actual stated case of wanting "the same beautiful message-ID as Yandex mail" and for the more general case of controlling message ID format. It's not something the USER should ever have to think about, for the reasons I said, quoted above--it's something only developers should ever have to care about, and even then only to the point where it complies with standards, which it does. If Mutt wants to accommodate broken list archive software (it IS broken, because it fails to handle the full range of decades-old standard, documented valid input values), then it should use a format that does that. No user should ever have reason to care about this. Seriously. > Although one could argue that the mailing list archiving system > should accept the responsibility of munging message ids to suit its > own needs. Indeed. But even that presupposes the design flaw that leads to the necessity of doing that. They could have simply organized the data differently (e.g. using a database, etc.) that does not have Unix path name issues. Also worth pointing out that if the software in question runs on Windows, there are additional allowed characters that cause breakage there. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. signature.asc Description: PGP signature
Re: Question about message id
On Thu, Apr 11, 2024 at 01:13:52PM +1000, raf via Mutt-users wrote: > > > Question, what should I write in .muttrc to make my outgoing > > > mails have the same beautiful message-ID as Yandex mail? > > The unfathomable thing about this question is why you (or anyone) > > should care in the slightest what your message ID looks like. > > It's an esoteric detail about e-mail transfer, the specific > > contents of which have no value to users, who in most cases won't > > even ever see the message ID, since most mail clients hide that > > detail from you by default. You have no practical reason to care > > what it is as long as everything is working correctly. It's > > literally not for you--it's for your MUA software. > The link to a kernel mailing list message that was provided earlier > in this discussion said that the choice to use base64 results in the > possibility of / characters being included in the message id which > causes problems for their archived messages accessible via a web > browser. So it seems that there is a reason to care about this. I believe it has to be a *syntactically* valid email address, according to the RFC. So, for example, an all-alphabetical random string would fail, because it'd lack a "@". > Although one could argue that the mailing list archiving system > should accept the responsibility of munging message ids to suit its > own needs. I've certainly seen mailing list archives on the web that > did munge the message id (but to replace @ characters, I think). I myself see not much wrong with a mailing list doing that, but: - others seem to feel differently, that modifying the msgid once a valid one has been generated by a MUA is gauche. And I understand that it makes some things slightly harder, especially if you use fcc or similar feature in other MUAs. - if the mailing list does mess with msgid, it absolutely must do it consistently for all copies of the message. Not like the mailman managed GNU lists where the msgid was (or still is?) munged if and only if the message was crossing a mail / news boundary. I stopped following the GNU lists because of that and I never went back. -- Ian
Re: Question about message id
On Tue, Apr 09, 2024 at 10:53:41AM -0400, Derek Martin wrote: > On Sun, Apr 07, 2024 at 01:19:09PM +, Ебрашка wrote: > > Question, what should I write in .muttrc to make my outgoing mails have > > the same beautiful message-ID as Yandex mail? > > The unfathomable thing about this question is why you (or anyone) > should care in the slightest what your message ID looks like. It's an > esoteric detail about e-mail transfer, the specific contents of which > have no value to users, who in most cases won't even ever see the > message ID, since most mail clients hide that detail from you by > default. You have no practical reason to care what it is as long as > everything is working correctly. It's literally not for you--it's for > your MUA software. The link to a kernel mailing list message that was provided earlier in this discussion said that the choice to use base64 results in the possibility of / characters being included in the message id which causes problems for their archived messages accessible via a web browser. So it seems that there is a reason to care about this. Although one could argue that the mailing list archiving system should accept the responsibility of munging message ids to suit its own needs. I've certainly seen mailing list archives on the web that did munge the message id (but to replace @ characters, I think). cheers, raf > -- > Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 > -=-=-=-=- > This message is posted from an invalid address. Replying to it will result in > undeliverable mail due to spam prevention. Sorry for the inconvenience. >
Re: Question about message id
On Wed, Apr 10, 2024 at 01:05:09PM +0800, Kevin J. McCarthy wrote: > On Tue, Apr 09, 2024 at 09:33:06AM -0700, Will Yardley wrote: > > side note: using quotes around the template screws things up, that is, > > set message_id_format="" > > vs > > set message_id_format= > > > > results in a broken message-id with the quotes inside the angle brackets > > (Message-ID: <"xyz">) > > That shouldn't be the case. The outer double quotes are stripped off during > muttrc evaluation. > > You may want to double check whether you accidentally copy/pasted unicode > curly quotes. You're right. That said, maybe we could setup the online version of the docs to be more copy / pasta compliant? http://www.mutt.org/doc/manual/#message-id-format I'd normally expect to be able to copy the whole string (including the quotes) into a config Seems like mutt is using "\fC" and "\fP" consistently (intentionally?) in various sections of the docs in init.h, though. The quotes in 2202 are also a little odd with the left / right double quotes, then outer regular double quotes, then inner regular ones. https://gitlab.com/muttmua/mutt/-/blob/master/init.h?ref_type=heads#L2202 https://gitlab.com/muttmua/mutt/-/blob/master/init.h?ref_type=heads#L2208 /w
Re: Question about message id
Thanks everyone for the replies and discussion, something to ponder. Всем спасибо за ответы и дискуссию, есть над чем поразмышлять.
Re: Question about message id
On Tue, Apr 09, 2024 at 09:33:06AM -0700, Will Yardley wrote: side note: using quotes around the template screws things up, that is, set message_id_format="" vs set message_id_format= results in a broken message-id with the quotes inside the angle brackets (Message-ID: <"xyz">) That shouldn't be the case. The outer double quotes are stripped off during muttrc evaluation. You may want to double check whether you accidentally copy/pasted unicode curly quotes. You can also check the value mutt has read in via :set ?message_id_format -- Kevin J. McCarthy GPG Fingerprint: 8975 A9B3 3AA3 7910 385C 5308 ADEF 7684 8031 6BDA signature.asc Description: PGP signature
Re: Question about message id
On 09Apr2024 07:11, Michael Kjörling wrote: On 9 Apr 2024 07:32 +1000, from c...@cskk.id.au (Cameron Simpson): _Or_ you could "set sendmail=" to a script of your own to add a message-id header - that is what mutt uses to deliver the message to a mail system - you could add a header there then pass the message on to whatever you had mutt using before. Wouldn't that at a minimum cause a discrepancy between the message-ID of what is actually sent, and of what is stored in the Fcc? If mutt made one, which I gather it actually does (I had thought not). Agree such a script should never replace an existing message-id. Cheers, Cameron Simpson
Re: Question about message id
Cameron Simpson wrote: > On 07Apr2024 13:19, Ебрашка wrote: > > Question, what should I write in .muttrc to make my outgoing mails have > > the same beautiful message-ID as Yandex mail? > > For example Message-Id: <43265...@example.com> consisting of random > > digits and domain name > > I think it would be enough to supply your own message-id header; I don't > think it is necessarily mutt making it at all. Mutt hands messages to a mail > system, and a mail system adds a message-id if one is missing. > > You could (a) turn on "set edit_headers=yes" so the headers are visible in > your message and (b) have your editor startup make one? > > _Or_ you could "set sendmail=" to a script of your own to add a message-id > header - that is what mutt uses to deliver the message to a mail system - > you could add a header there then pass the message on to whatever you had > mutt using before. And don't be overly clever and set a custom header to a script that produces a message-id; that will get evaluated when mutt starts and not again for every message, thus defeating the point. Please don't ask me how I know. -dsr-
Re: Question about message id
On Tue, Apr 09, 2024 at 09:25:56AM -0700, Will Yardley wrote: > On Tue, Apr 09, 2024 at 11:33:55AM -0400, Derek Martin wrote: > > If so that feature may not be working correctly... > > That is the old format for the mutt version I'm using. The new format > looks like > > > > Thought you were on that long thread about this when it changed a while > back, but maybe remembering wrong. I was--but I guess the source of my confusion is I don't actually care what the message ID format is (as should no user), so I never bothered to internalize it, but I vaguely (mis)remembered it had (I thought one, but at least one) more dot-separated components. My issue was never with the specifics of the format, but rather the practical benefit of the changes and the spurious logic in favor of them vs. the cost of implementing and maintaining them. Cheers -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. signature.asc Description: PGP signature
Re: Question about message id
On Tue, Apr 09, 2024 at 11:33:55AM -0400, Derek Martin wrote: > > I think some people mentioned some minor threading issues with the new > > format? > > Is that so? I don't recall noticing anything about that... I'd love a > pointer to some details, if anyone has that. It's possible that it was since fixed. The issue isn't with threading _in_ a main client, I think, but with certain web archiving software in the case where the message-id has a "/" character. https://people.kernel.org/monsieuricon/fix-your-mutt > Anyway, I said then, and > maintain now, that changing the format and making it configurable was > pointless and added complexity without good technical justification. > You may have just proven that. =8^) Right, I was also fine with the old format, which is why I'm glad there's an option to put it back. I'm not personally interested in reviving the old argument about whether the new format is better or worse, as long as there's a simple way to restore the old behavior with a config option. side note: using quotes around the template screws things up, that is, set message_id_format="" vs set message_id_format= results in a broken message-id with the quotes inside the angle brackets (Message-ID: <"xyz">) if folks think that's a bug, I'm happy to file an issue about it, though just not using the quotes worked fine. /w
Re: Question about message id
On Sun, Apr 07, 2024 at 01:19:09PM +, Ебрашка wrote: > Question, what should I write in .muttrc to make my outgoing mails > have the same beautiful message-ID as Yandex mail? Talk about bikeshedding :-) -- Ian
Re: Question about message id
On Tue, Apr 09, 2024 at 11:33:55AM -0400, Derek Martin wrote: > On Tue, Apr 09, 2024 at 08:05:06AM -0700, Will Yardley wrote: > > On Tue, Apr 09, 2024 at 10:53:41AM -0400, Derek Martin wrote: > > > The unfathomable thing about this question is why you (or anyone) > > > should care in the slightest what your message ID looks like. > > > > That's totally true, but I still like the classic Mutt message-id > > format. > > Did you set your message ID format accordingly, then? Because the > message I'm replying to carries this: > > Message-ID: <20240409150506.ga47...@aura.veggiechinese.net> > > If so that feature may not be working correctly... That is the old format for the mutt version I'm using. The new format looks like Thought you were on that long thread about this when it changed a while back, but maybe remembering wrong. w
Re: Question about message id
On Tue, Apr 09, 2024 at 08:05:06AM -0700, Will Yardley wrote: > On Tue, Apr 09, 2024 at 10:53:41AM -0400, Derek Martin wrote: > > The unfathomable thing about this question is why you (or anyone) > > should care in the slightest what your message ID looks like. > > That's totally true, but I still like the classic Mutt message-id > format. Did you set your message ID format accordingly, then? Because the message I'm replying to carries this: Message-ID: <20240409150506.ga47...@aura.veggiechinese.net> If so that feature may not be working correctly... > Maybe it's just nostalgia, but either way, I'm not super > worried about leaking my PID etc... Nor should you be--as I pointed out then, the PID is only of any use to an attacker who is already on your system, which means either: 1. Your system is already compromised, or 2. They are a legitimate user on your system, in which case they can already easily get the pid of your mail client In no way can a process ID be construed as sensitive. If it were you'd have to disable the ps command, modify the logging of virtually every network daemon ever, and a whole bunch of other things. > I think some people mentioned some minor threading issues with the new > format? Is that so? I don't recall noticing anything about that... I'd love a pointer to some details, if anyone has that. Anyway, I said then, and maintain now, that changing the format and making it configurable was pointless and added complexity without good technical justification. You may have just proven that. =8^) -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. signature.asc Description: PGP signature
Re: Question about message id
On Tue, Apr 09, 2024 at 10:53:41AM -0400, Derek Martin wrote: > The unfathomable thing about this question is why you (or anyone) > should care in the slightest what your message ID looks like. That's totally true, but I still like the classic Mutt message-id format. Maybe it's just nostalgia, but either way, I'm not super worried about leaking my PID etc... so I'm happy someone added the feature to make it configurable. I think some people mentioned some minor threading issues with the new format? w
Re: Question about message id
On Sun, Apr 07, 2024 at 01:19:09PM +, Ебрашка wrote: > Question, what should I write in .muttrc to make my outgoing mails have > the same beautiful message-ID as Yandex mail? The unfathomable thing about this question is why you (or anyone) should care in the slightest what your message ID looks like. It's an esoteric detail about e-mail transfer, the specific contents of which have no value to users, who in most cases won't even ever see the message ID, since most mail clients hide that detail from you by default. You have no practical reason to care what it is as long as everything is working correctly. It's literally not for you--it's for your MUA software. From a practical standpoint, the only thing that matters is that the generator intelligently makes a genuine effort to ensure the generated message ID is unique (and perhaps that it does not needlessly leak sensitive details about the user, though no message ID format that I'm aware of--including Mutt's original default format--ever did that, emphasis on "sensitive"). You should be able to trust your mail client to handle that detail on your behalf. If for some reason you can't, you should be using different e-mail software. Truthfully, Mutt typically shouldn't even be in the business of generating message IDs, where the expectation is that it will hand the mail off to the MTA running on the user's system, which should handle message ID generation, so that there is no possibility of overlap. [In practice this is unlikely to be a problem, because the format Mutt uses is different from the major mail transfer agents. The probability of an overlap is therefore exceedingly low, and thus uniqueness is maintained.] Other than that, as long as replies and message threading work properly in your mail client, don't even look at it--it is (or should be) completely opaque to you. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. signature.asc Description: PGP signature
Re: Question about message id
On 9 Apr 2024 07:32 +1000, from c...@cskk.id.au (Cameron Simpson): > You could (a) turn on "set edit_headers=yes" so the headers are visible in > your message and (b) have your editor startup make one? This is what I do in order to have a UUID message-ID, which Mutt doesn't support natively. > _Or_ you could "set sendmail=" to a script of your own to add a message-id > header - that is what mutt uses to deliver the message to a mail system - > you could add a header there then pass the message on to whatever you had > mutt using before. Wouldn't that at a minimum cause a discrepancy between the message-ID of what is actually sent, and of what is stored in the Fcc? -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: Question about message id
On Sun, Apr 07, 2024 at 01:19:09PM +, Ебрашка wrote: > Question, what should I write in .muttrc to make my outgoing mails have > the same beautiful message-ID as Yandex mail? > For example Message-Id: <43265...@example.com> consisting of random > digits and domain name There was a long debate about the message-id format that you could probably dig up. For a long time, the Mutt ones were a date string followed by an epoch timestamp. The change in 2.0 was somewhat controversial. Note that this is now configurable (which I hadn't known and / or had forgotten), so, assuming you're using mutt >= 2.1, you can customize it if you'd like using this config setting: http://www.mutt.org/doc/manual/#message-id-format [I'm testing it out myself on this response!] w
Re: Question about message id
On 07Apr2024 13:19, Ебрашка wrote: Question, what should I write in .muttrc to make my outgoing mails have the same beautiful message-ID as Yandex mail? For example Message-Id: <43265...@example.com> consisting of random digits and domain name I think it would be enough to supply your own message-id header; I don't think it is necessarily mutt making it at all. Mutt hands messages to a mail system, and a mail system adds a message-id if one is missing. You could (a) turn on "set edit_headers=yes" so the headers are visible in your message and (b) have your editor startup make one? _Or_ you could "set sendmail=" to a script of your own to add a message-id header - that is what mutt uses to deliver the message to a mail system - you could add a header there then pass the message on to whatever you had mutt using before. -- Cameron Simpson (formerly c...@zip.com.au) Madness takes its toll. Please have exact change. - Janice Hanes
Re: Question about message id
On 07Apr2024 18:23, Anton Sharonov wrote: On Sun, Apr 07, 2024 at 09:23:07AM -0600, Charles Cazabon via Mutt-users wrote: There's a good reason for that; it help to ensure uniqueness, which prevents problems with threading. By limiting itself to only digits, Yandex's IDs are much more likely to collide. Hm, not sure about that... Given that string is long enough, random string which consists only out of digits can perhaps compete with (much shorter) random string of alpha-numeric characters - in terms of uniqueness probability? It doesn't matter. The yandex people only need to ensure the uniqueness of the stuff left of the "@" (because they're the owners of the stuff on the right), and can do whatever is reliably unique for them. Regardless, just using digits only means the string needs to be a bit longer. Mutt uses more characters because it lets you make the message-id a shorter piece of text.
Re: Question about message id
On 7 Apr 2024 18:23 +0200, from anton.sharo...@gmail.com (Anton Sharonov): >>> For example Message-Id: <43265...@example.com> consisting of random >>> digits and domain name >> >> There's a good reason for that; it help to ensure uniqueness, which prevents >> problems with threading. By limiting itself to only digits, Yandex's IDs are >> much more likely to collide. > > Hm, not sure about that... Given that string is long enough, > random string which consists only out of digits can perhaps > compete with (much shorter) random string of alpha-numeric > characters - in terms of uniqueness probability? Sure; assuming a Base64 charset, each character encodes log_2(64) = 6 bits; for decimal digits, each character encodes log_2(10) ~ 3.3 bits. Eight decimal digits encode only about 26 bits; eight Base64 characters encode 48 bits. (To encode 48 bits using only decimal digits you need at least 15 digits.) So the decimal digits case has a very much non-trivial chance of a random collision within a fixed domain name in 8 characters, whereas the Base64 case is significantly less likely to have collisions within the same number of characters; in both cases assuming that the part before the @ is assigned at random. Look up the "birthday paradox". The value of the Message-ID header is required by RFC 5322 to be unique. https://www.rfc-editor.org/rfc/rfc5322.html#section-3.6.4 That has been the case since RFC 822 https://www.rfc-editor.org/rfc/rfc822 (section 4.6.1, page 23). RFC 5322 puts the uniqueness of the message ID as a "MUST" level requirement, even going so far as to state that _twice_ within two consecutive sentences. _How you guarantee that uniqueness_ is up to you, but not following that requirement _is_ going to cause a variety of issues, quite possibly up to and including messages not being delivered properly. Violate that requirement at your own peril. -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: Question about message id
On Sun, Apr 07, 2024 at 09:23:07AM -0600, Charles Cazabon via Mutt-users wrote: > Ебрашка wrote: > > my mails have Message-ID: . Question, what > > should I write in .muttrc to make my outgoing mails have the same beautiful > > message-ID as Yandex mail? > > My first question would be, why do you care what the Message-ID: field > contents look like? Virtually no-one will ever look at it. Yes, this here is fully true ))) > > > For example Message-Id: <43265...@example.com> consisting of random > > digits and domain name > > There's a good reason for that; it help to ensure uniqueness, which prevents > problems with threading. By limiting itself to only digits, Yandex's IDs are > much more likely to collide. Hm, not sure about that... Given that string is long enough, random string which consists only out of digits can perhaps compete with (much shorter) random string of alpha-numeric characters - in terms of uniqueness probability? > > If you absolutely must do this, have a look at the `message_id_format` > variable. Good hint ))) Attached is the example (perl) script, which can be used as: set message_id_format = "$HOME/bin/mutt_message_id_format_filter '<%z@%f>'|" This converts "left_part" provided by Mutt %z part of Message-ID code (ZhJUjJRaE9InvXxm in your example above), into (much longer) string of only digits (592493855977705398853375435077558962105453 for same example). String of digits is 1:1 mapping of each byte code of every next symbol in the source string, used in the triples with some padding by spaces. Every tripple generates decimal number. All numbers are put after each other into long result. Code is certainly ugly but seems to work. Cheers, Anton #!/usr/bin/perl -w use 5.010; use strict; my $val = $ARGV[0]; my $debug = 0; if (! ($val =~ m/^<([^@]+)@([^>]+)>/)) { print $val; # message format is not recognized exit 0; } my $logfile = undef; if ($debug) { my $log_fname = "/tmp/mutt_message_id_format_filter.log"; open ($logfile, ">>", $log_fname) or die ("Cannot open [$log_fname] for write: $!"); } my $left_part = $1; my $host_n_domain = $2; #abc print $logfile "left_part = $left_part; host_n_domain = $host_n_domain\n" if ($debug); my $new_left_part = ""; while (length($left_part)) { # extract 3 symbols, pad with blanks my $bytes = sprintf("%3.3s", substr($left_part, 0, 3)); my $len = length($left_part); $left_part = substr($left_part, $len >= 3 ? 3 : $len); print $logfile " bytes = {" . $bytes . "}" if ($debug); my $num = 0; for my $i (0..2) { $num *= 256; $num += ord(substr($bytes, $i, 1)); } print $logfile " num = {" . $num . "}\n" if ($debug); $new_left_part .= $num; } print $logfile "\n" if ($debug); print $logfile "<$new_left_part@" . $host_n_domain . ">" if ($debug); print "<$new_left_part@" . $host_n_domain . ">"; print $logfile "\n" if ($debug);
Re: Question about message id
Ебрашка wrote: > my mails have Message-ID: . Question, what > should I write in .muttrc to make my outgoing mails have the same beautiful > message-ID as Yandex mail? My first question would be, why do you care what the Message-ID: field contents look like? Virtually no-one will ever look at it. > For example Message-Id: <43265...@example.com> consisting of random > digits and domain name There's a good reason for that; it help to ensure uniqueness, which prevents problems with threading. By limiting itself to only digits, Yandex's IDs are much more likely to collide. If you absolutely must do this, have a look at the `message_id_format` variable. Charles -- --- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ---