Re: WIP: filter out envelope headers in notmuch-insert.
On Mon, 14 Feb 2022, David Bremner wrote: > "NeilBrown" writes: > > > On Sat, 12 Feb 2022, David Bremner wrote: > >> This is a bit rough and ready, but before I fine tune it, I want to > >> make sure the overall idea of stripping envelope headers in > >> notmuch-insert makes sense. > > > > I think that it would be highly inappropriate to modify the message > > passed to notmuch-insert AT ALL. > > It *might* be appropriate to check that it looks generally like an email > > message (e.g. headers followed by a blank line), but changing it would > > be wrong. > > > > Here we're just changing "From foo" into "X-Envelope-From: foo", > so no information is lost, unless you consider being an mbox file > information. That one small change is probably defensible - though I would probably prefer that such messages were either completely rejected or completely supported. Your broader discussion in the sequence of emails seemed to suggest you were considering a lot more than that - sorry if I misunderstood. > > > Why do you think it might be helpful? > > Currently we throw an error from notmuch-insert when trying to index > those mbox files (as passed to notmuch-insert by postfix). This means > the message is either bounced or (with --keep), invisible in the notmuch > index. If notmuch-insert is given an 'mbox', shouldn't it either reject it, or split it up into individual messages and insert each one individually (discarding the "From" line because it shouldn't add any new information)? I don't suppose there is some config option to postfix to tell it to provide an RFC-2822 email message, not an mbox ??? Thanks, NeilBrown ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: WIP: filter out envelope headers in notmuch-insert.
NeilBrown writes: > And on digging into postfix a bit, it seem that when forwarding to a > program, it *doesn't* quote other "From " lines in the email, so > splitting up the input would break things. I wonder if that is a bug... > the documentation is clear on the details of delivering to a program It's a least partly "mbox is trainwreck, and they probably don't really want to do mbox". Imagine a "From " line inside a signed part. But I really don't know the intent of including a "From " line. > So it seems... In the case of postfix the sender information is provided > both in "From " and "Return-Path:", so just stripping "From " would be > safe. It is hard to argue against the extra caution of retaining the > from line in an "X-Envolope-from" though. > > Thanks, > NeilBrown Thanks for thinking about these design issues with me. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: WIP: filter out envelope headers in notmuch-insert.
On Tue, 15 Feb 2022, David Bremner wrote: > "NeilBrown" writes: > > > On Mon, 14 Feb 2022, David Bremner wrote: > > > > If notmuch-insert is given an 'mbox', shouldn't it either reject it, or > > split it up into individual messages and insert each one individually > > (discarding the "From" line because it shouldn't add any new > > information)? > > I guess you could think of what the patch does as a conservative version > of that. I'm not aware of any case where a "real" mbox with multiple > messages is passed to notmuch-insert. So the modification effectively > discards the From line, although keeping the contents, because who knows > what people rely on. And on digging into postfix a bit, it seem that when forwarding to a program, it *doesn't* quote other "From " lines in the email, so splitting up the input would break things. I wonder if that is a bug... the documentation is clear on the details of delivering to a program > > > I don't suppose there is some config option to postfix to tell it to > > provide an RFC-2822 email message, not an mbox ??? > > It seems not, at least for the case of invoking notmuch-insert via a > .forward file, which I would guess is the most common case (nothing > specific to postfix, it's just the obvious way to invoke > notmuch-insert). So it seems... In the case of postfix the sender information is provided both in "From " and "Return-Path:", so just stripping "From " would be safe. It is hard to argue against the extra caution of retaining the from line in an "X-Envolope-from" though. Thanks, NeilBrown ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: WIP: filter out envelope headers in notmuch-insert.
"NeilBrown" writes: > On Mon, 14 Feb 2022, David Bremner wrote: > > If notmuch-insert is given an 'mbox', shouldn't it either reject it, or > split it up into individual messages and insert each one individually > (discarding the "From" line because it shouldn't add any new > information)? I guess you could think of what the patch does as a conservative version of that. I'm not aware of any case where a "real" mbox with multiple messages is passed to notmuch-insert. So the modification effectively discards the From line, although keeping the contents, because who knows what people rely on. > I don't suppose there is some config option to postfix to tell it to > provide an RFC-2822 email message, not an mbox ??? It seems not, at least for the case of invoking notmuch-insert via a .forward file, which I would guess is the most common case (nothing specific to postfix, it's just the obvious way to invoke notmuch-insert). d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: WIP: filter out envelope headers in notmuch-insert.
"NeilBrown" writes: > On Sat, 12 Feb 2022, David Bremner wrote: >> This is a bit rough and ready, but before I fine tune it, I want to >> make sure the overall idea of stripping envelope headers in >> notmuch-insert makes sense. > > I think that it would be highly inappropriate to modify the message > passed to notmuch-insert AT ALL. > It *might* be appropriate to check that it looks generally like an email > message (e.g. headers followed by a blank line), but changing it would > be wrong. > Here we're just changing "From foo" into "X-Envelope-From: foo", so no information is lost, unless you consider being an mbox file information. > Why do you think it might be helpful? Currently we throw an error from notmuch-insert when trying to index those mbox files (as passed to notmuch-insert by postfix). This means the message is either bounced or (with --keep), invisible in the notmuch index. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: WIP: filter out envelope headers in notmuch-insert.
David Bremner writes: > David Bremner writes: > >> I thought about a more ambitious version that would replace any >> existing "Return-Path" headers, but it seems like significantly more >> work (the current code is not line based), and not obviously >> better. Or maybe I missed the wording in the RFCs that talks about how >> MDAs should behave here. > > The RFC wording I _did_ find is as follows. > > RFC5598 says that (§4.3.3) > > The MDA records the RFC5321.MailFrom address into the > RFC5321.Return-Path field. > > where the first is the (address in) envelope from and the second is the > Return-Path: header. I'm not sure how authoritative RFC5598 really is; > it is "Category: Informational", whatever that means. Starting at RFCs some more (as one does), I think that the "envelope header" added by e.g. postfix's local(8) does not count as a RFC5321.MailFrom (although it should contain the same information). These mbox style envelope headers do not seem to super well defined. RFC 4155, which is _about_ mbox, seems to go out of its way not to define the syntax. The following (from RFC5598) seems to indicate that MDAs speak SMTP. Transfer into the MDA is accomplished by a normal MTA transfer mechanism. If correct, that means that notmuch-insert is not an MDA in the sense meant by RFC5598 As an experiment I converted copy_fd to line oriented form (which I guess I'll post seperately), but I realized that further progress would require parsing the "envelope header" line. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: WIP: filter out envelope headers in notmuch-insert.
David Bremner writes: > I thought about a more ambitious version that would replace any > existing "Return-Path" headers, but it seems like significantly more > work (the current code is not line based), and not obviously > better. Or maybe I missed the wording in the RFCs that talks about how > MDAs should behave here. The RFC wording I _did_ find is as follows. RFC5598 says that (§4.3.3) The MDA records the RFC5321.MailFrom address into the RFC5321.Return-Path field. where the first is the (address in) envelope from and the second is the Return-Path: header. I'm not sure how authoritative RFC5598 really is; it is "Category: Informational", whatever that means. On the other hand RFC5321 §4.4 says When the delivery SMTP server makes the "final delivery" of a message, it inserts a return-path line at the beginning of the mail data. This use of return-path is required; mail systems MUST support it. The return-path line preserves the information in the from the MAIL command. Here, final delivery means the message has left the SMTP environment. Normally, this would mean it had been delivered to the destination user or an associated mail drop, but in some cases it may be further processed and transmitted by another mail system. At least postfix seems to follow the latter, but also includes the envelope header when passing a message to a program via .forward. This seems to imply that if we can trust the MTA to DTRT, the Return-Path should already be present. OTOH, that is obviously not the most robust assumption possible. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
WIP: filter out envelope headers in notmuch-insert.
This is a bit rough and ready, but before I fine tune it, I want to make sure the overall idea of stripping envelope headers in notmuch-insert makes sense. The first patch is duplicated from id:20220205195215.166213-2-da...@tethera.net I thought about a more ambitious version that would replace any existing "Return-Path" headers, but it seems like significantly more work (the current code is not line based), and not obviously better. Or maybe I missed the wording in the RFCs that talks about how MDAs should behave here. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
