Yeah this sounds like what Volkan discovered in his research. The mailing list software is fucking up the metadata in such a way as to make it invalid in various spam checks (some of which silently drop malformed emails). It doesn’t help that said software is ancient and no longer developed.
> On Oct 15, 2024, at 10:58, Dominik Psenner <dpsen...@gmail.com> wrote: > > I would guess, if mail doesn't work it probably relates to spf, dkim or > dmark. > > Your description matches exactly this: Some service providers gradually > increase the rate of rejected mails that come from servers/domains refusing > to implement spf and dkim or even fail spf/dkim checks. This can be seen as > an incentive so that other service providers harden their infrastructure. > > At my dayjob we reject all messages that fail spf and dkim checks. Further > we actively monitor dmark activity to be informed if third parties are > attempting impersonation of company owned domains (aka identity spoofing). > > Unfortunately you do not get any messages as a "client", be it sender or > recipient. Mailserver administrators would see failures, but only if they > monitor the right metrics. And, yes, the world is a messy place. ;-) > > I had to learn some of this craft. Even if I did not master all aspects, I > could try my best and do an analysis of some sample message headers? Do not > hesitate to send me sample messages that made it into your inbox, please as > attachments so that message headers are preserved. > > On Tue, 15 Oct 2024, 15:25 Ralph Goers, <ralph.go...@dslextreme.com> wrote: > >> Volkan has taken to pinging me in Slack every time he sends a message to >> this list because at least 25% of the time I wasn’t getting his emails. No >> one knows why. I checked with my mail provider and have looked everywhere I >> have access to. >> >> Lately I have been getting all of his emails. Again, I don’t know why. >> Also, it seemed to only be emails from him. But his emails always made it >> to the list. >> >> I would like email more if it could automatically filter. As it stands I >> have something like 60 folders under the email address where I receive >> these. That means I have at least 60 filters set up to route my mail into >> them. But even that isn’t perfect. GitHub generates so much noise that it >> is impossible to keep up with all the notifications it sends me. That is >> one of the major reasons I am not in love with the idea of using >> discussions. >> >> Ralph >> >>> On Oct 15, 2024, at 4:54 AM, Dominik Psenner <dpsen...@gmail.com> wrote: >>> >>> What concerns me is that email appears to be unreliable where it >> shouldn't. >>> Either it works always or it doesn't at all. I experience it to be 100% >>> reliable. >>> >>> Short downtimes are OK and to be expected while systems are patched. But >>> then email are delivered later. The technology is much like a postman who >>> comes back later if he finds the front gate closed. The postman should >> not >>> burn the letter in front of the gate. >>> >>> On Mon, 14 Oct 2024, 22:56 Matt Sicker, <m...@musigma.org> wrote: >>> >>>> That’s because a lot of other things are also using Slack. On the other >>>> hand, I had to disable notifications from Slack due to people misusing >> it >>>> to DM me instead of sending emails to the Secretary properly (unrelated >> to >>>> Log4j). >>>> >>>>> On Oct 14, 2024, at 15:53, Ralph Goers <ralph.go...@dslextreme.com> >>>> wrote: >>>>> >>>>> I can’t say I agree with that. It didn’t take me very long to get used >>>> to using Slack. >>>>> >>>>> Ralph >>>>> >>>>>> On Oct 14, 2024, at 1:47 PM, Matt Sicker <m...@musigma.org> wrote: >>>>>> >>>>>> There’s a very, very small chance I’ll ever remember to visit a >> website >>>> to find out about what are essentially emails that could have been sent >> to >>>> me. I have a regular habit of reading email nearly every day, but >>>> developing new habits is unlikely to stick. >>>>>> >>>>>>> On Oct 14, 2024, at 14:50, Ralph Goers <ralph.go...@dslextreme.com> >>>> wrote: >>>>>>> >>>>>>> Maybe we just need to start contributing to PonyMail to improve the >> UI >>>> to eliminate actually needing the email delivered to our accounts. >>>>>>> >>>>>>> I am only 1/4 serious about this. There has to be a better solution. >>>>>>> >>>>>>> Ralph >>>>>>> >>>>>>>> On Oct 14, 2024, at 10:25 AM, Matt Sicker <m...@musigma.org> wrote: >>>>>>>> >>>>>>>> I didn’t get the original email in this thread once again, so I >> think >>>> I’d support trying somewhere else to host discussions. Besides archiving >>>> those messages into a mailing list, it would be great if the solution >>>> provided allowed for email interactivity (e.g., you can reply to the >>>> notification of a message and it’s added to the thread appropriately; >> this >>>> is how GitHub notification emails typically work). >>>>>>>> >>>>>>>>> On Oct 10, 2024, at 05:40, Christian Grobmeier < >> grobme...@apache.org> >>>> wrote: >>>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> I am generally open to such experiments. I would start with the >>>> easiest parts, such as users@, and see where it goes. >>>>>>>>> >>>>>>>>> I would advise against mirroring it to users@ behind the scenes, >> as >>>> it may cause privacy problems (we need user consensus to mirror it). >> When a >>>> user uses GitHub, they know what to expect. >>>>>>>>> >>>>>>>>> As for Discourse, many use that now, but I find it very >> overwhelming >>>> and stressful. I prefer the clean Github discussions approach. >>>>>>>>> >>>>>>>>> I haven't checked against ASF policies but feel positive about this >>>> move for the arguments you have given >>>>>>>>> >>>>>>>>> Kind regards >>>>>>>>> Christian >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Oct 10, 2024, at 10:58, Volkan Yazıcı wrote: >>>>>>>>>> *Abstract:* Modern email system security measures make it >>>> practically >>>>>>>>>> impossible for mailing lists to work – many subscribers don't get >>>> all >>>>>>>>>> emails. This not only hinders communication, but blocks an >> inclusive >>>>>>>>>> one. *Shall >>>>>>>>>> we, as Logging Services, experiment with alternatives?* >>>>>>>>>> >>>>>>>>>> *Motivation #1: mailing lists technically don't work* >>>>>>>>>> >>>>>>>>>> Several widely-used email providers (GMail, Yahoo, iCloud, etc.) >>>> have in >>>>>>>>>> the last couple of years enabled new measures (DMARC, SPF, DKIM, >>>> etc.) to >>>>>>>>>> verify the authenticity of emails. In short, these measures enrich >>>> email >>>>>>>>>> content with checksums and signatures capturing its authenticity. >>>> When a >>>>>>>>>> mailing list system (e.g., ezmlm, mailman) receives such an email, >>>> it >>>>>>>>>> performs several changes on its content (adds information about >> the >>>> mailing >>>>>>>>>> list, etc.), and delivers it to all subscribers. When the mail >>>> server of a >>>>>>>>>> subscriber receives such tampered mail, and if that mail server >>>> happens to >>>>>>>>>> have earlier shared authenticity checks enabled, it discards the >>>> email, or >>>>>>>>>> at best, marks it as spam. >>>>>>>>>> >>>>>>>>>> Ralph, Matt, Piotr stated many times that they don't receive all >>>> emails. >>>>>>>>>> Ralph actually stated many ASF mailing list emails end up in his >>>> spam >>>>>>>>>> box >>>>>>>>>> < >>>> >> https://the-asf.slack.com/archives/CBX4TSBQ8/p1728032221080189?thread_ts=1727958807.348019&cid=CBX4TSBQ8 >>>>> . >>>>>>>>>> Recently we witnessed even Brian Proffitt (VP, Marketing & >>>> Publicity) >>>>>>>>>> suffer >>>>>>>>>> from the same problem >>>>>>>>>> <https://lists.apache.org/thread/yfmrpjslcbo5jmsqqpvtok1o6lht11rb >>> . >>>> >>>>>>>>>> INFRA >>>>>>>>>> is crawling with related tickets: INFRA-24574 >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24574>, INFRA-24790 >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24790>, INFRA-24845 >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24845>, INFRA-24850 >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24850>, INFRA-24872 >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24872>, INFRA-25947 >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-25947>, INFRA-26171 >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-26171> – there are >>>> dozens >>>>>>>>>> more. >>>>>>>>>> >>>>>>>>>> This technical difficulty is not only known to us. AFAIK, this is >>>> one of >>>>>>>>>> the main reasons PSF (Python Software Foundation) decided to >> switch >>>> from >>>>>>>>>> mailing lists to Discourse. Mailman documents several DMARC >>>> mitigations >>>>>>>>>> < >>>> >> https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/dmarc-mitigations.html >>>>> , >>>>>>>>>> but I think these are workarounds/hacks rather than >> well-established >>>>>>>>>> solutions. >>>>>>>>>> >>>>>>>>>> *Motivation #2: ezmlm is dead* >>>>>>>>>> >>>>>>>>>> ezmlm, the mailing list software ASF uses, is dead – it is neither >>>>>>>>>> developed, nor maintained anymore. (Last official release was in >>>> 1997, >>>>>>>>>> there >>>>>>>>>> is the `ezmlm-idx` add-on, which later on became a successor >>>>>>>>>> < >>>> >> https://untroubled.org/ezmlm/faq/What-is-the-difference-between-ezmlm-and-ezmlm_002didx_003f.html >>>>> , >>>>>>>>>> which last produced a release in 2014, and so on. Long, dead >>>> story.) >>>>>>>>>> INFRA >>>>>>>>>> maintains a very big, sophisticated set of Perl rules for running >>>> ASF >>>>>>>>>> ezmlm >>>>>>>>>> instances. If you look closely at the INFRA tickets I cited above, >>>> some >>>>>>>>>> suggest INFRA to fork ezmlm and fix some long standing bugs, etc. >>>> We can >>>>>>>>>> discuss the possibility of migrating from ezmlm to mailman (yet >>>> another >>>>>>>>>> mailing list software, but one that is still maintained), whether >>>> such a >>>>>>>>>> migration should be practiced ASF-wide or only for Logging >>>> Services, >>>>>>>>>> etc. >>>>>>>>>> But eventually, we will still be using a mailing list, and as I >>>> tried to >>>>>>>>>> explain above, IMO, they just don't work good. >>>>>>>>>> >>>>>>>>>> *Proposal #1: Experimenting with GitHub Discussions* >>>>>>>>>> >>>>>>>>>> GitHub is our development bread and butter. We use its tickets, >> PRs, >>>>>>>>>> reviews, discussions, CI, security & code quality checks, etc. It >>>> works >>>>>>>>>> perfectly and components are integrated well, i.e., you can link >>>> issues, >>>>>>>>>> comments, PRs, CI runs, etc. Users like it too – we all witnessed >>>> the >>>>>>>>>> sudden increase in user interactions after migrating to GitHub >>>> Issues >>>>>>>>>> and >>>>>>>>>> Discussions. We can configure sections & categories in Discussions >>>>>>>>>> < >>>> >> https://docs.github.com/en/discussions/managing-discussions-for-your-community/managing-categories-for-discussions >>>>> >>>>>>>>>> to make it serve as our main communication medium. It also >> provides >>>> mail >>>>>>>>>> notifications and the possibility to respond to them for those who >>>> still >>>>>>>>>> prefer their email client over a browser. >>>>>>>>>> >>>>>>>>>> In short, we can quickly configure Discussions, update our support >>>> policy >>>>>>>>>> page, and start experimenting with it. >>>>>>>>>> >>>>>>>>>> One can raise the argument that what if Discussions disappear? We >>>> can >>>>>>>>>> mirror communication there to a mailing list to be on the safe >>>> side. Yet, >>>>>>>>>> we need to evaluate the necessity of this. >>>>>>>>>> >>>>>>>>>> *Proposal #2: Experimenting with Discourse* >>>>>>>>>> >>>>>>>>>> We can get a VM from INFRA and manage our Discourse instance. >>>> Though, >>>>>>>>>> AFAIC, this will result in a "GitHub Discussions"-like setup with >>>> all the >>>>>>>>>> integration goodies missing and added server maintenance burden. >>>>>>>>>> >>>>>>>>>> *F.A.Q.* >>>>>>>>>> >>>>>>>>>> *What if GitHub Discussions disappear?* >>>>>>>>>> >>>>>>>>>> In such a case, I presume they will allow us to download the >>>> existing >>>>>>>>>> archives. In the worst case, we can decide to mirror the >>>> communication >>>>>>>>>> there to a mailing list. Yet, we need to evaluate the necessity of >>>> this. In >>>>>>>>>> particular, how big of a problem is this at the experimentation >>>> stage? >>>>>>>>>> >>>>>>>>>> *How will private communication work with GitHub Discussions?* >>>>>>>>>> >>>>>>>>>> We can create private repositories for internal/private >>>> communication. >>>>>>>>>> For >>>>>>>>>> users/researchers wanting to submit & discuss security issues, >> they >>>> can >>>>>>>>>> get >>>>>>>>>> in touch with us (either via email to `security@logging` or some >>>> other >>>>>>>>>> ASF/INFRA mailing list), we can grant them permissions to >>>> collaborate >>>>>>>>>> privately on a repository security advisory >>>>>>>>>> < >>>> >> https://docs.github.com/en/code-security/security-advisories/working-with-repository-security-advisories/about-repository-security-advisories >>>>> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>>> *Don't the ASF legals require mailing lists?* >>>>>>>>>> >>>>>>>>>> I am aware that several ASF policies require mailing list >>>> communication, >>>>>>>>>> e.g., for voting and such. I first want to establish a consensus >>>> among us, >>>>>>>>>> and then pitch to the board for exemption as a pilot. >>>>>>>>>> >>>>>>>>>> *Shouldn't this proposal be practiced ASF-wide?* >>>>>>>>>> >>>>>>>>>> This will be a very (very very very, actually!) daunting route to >>>> pursue. >>>>>>>>>> I'd rather start small, solve our problem first (if we can), and >>>> then think >>>>>>>>>> about widening the scope. >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >> >>
