This is one item I would like us, Apache, spend our money on. Gary
On Tue, Oct 15, 2024, 12:33 PM Matt Sicker <m...@musigma.org> wrote: > Yeah this sounds like what Volkan discovered in his research. The mailing > list software is fucking up the metadata in such a way as to make it > invalid in various spam checks (some of which silently drop malformed > emails). It doesn’t help that said software is ancient and no longer > developed. > > > On Oct 15, 2024, at 10:58, Dominik Psenner <dpsen...@gmail.com> wrote: > > > > I would guess, if mail doesn't work it probably relates to spf, dkim or > > dmark. > > > > Your description matches exactly this: Some service providers gradually > > increase the rate of rejected mails that come from servers/domains > refusing > > to implement spf and dkim or even fail spf/dkim checks. This can be seen > as > > an incentive so that other service providers harden their infrastructure. > > > > At my dayjob we reject all messages that fail spf and dkim checks. > Further > > we actively monitor dmark activity to be informed if third parties are > > attempting impersonation of company owned domains (aka identity > spoofing). > > > > Unfortunately you do not get any messages as a "client", be it sender or > > recipient. Mailserver administrators would see failures, but only if they > > monitor the right metrics. And, yes, the world is a messy place. ;-) > > > > I had to learn some of this craft. Even if I did not master all aspects, > I > > could try my best and do an analysis of some sample message headers? Do > not > > hesitate to send me sample messages that made it into your inbox, please > as > > attachments so that message headers are preserved. > > > > On Tue, 15 Oct 2024, 15:25 Ralph Goers, <ralph.go...@dslextreme.com> > wrote: > > > >> Volkan has taken to pinging me in Slack every time he sends a message to > >> this list because at least 25% of the time I wasn’t getting his emails. > No > >> one knows why. I checked with my mail provider and have looked > everywhere I > >> have access to. > >> > >> Lately I have been getting all of his emails. Again, I don’t know why. > >> Also, it seemed to only be emails from him. But his emails always made > it > >> to the list. > >> > >> I would like email more if it could automatically filter. As it stands I > >> have something like 60 folders under the email address where I receive > >> these. That means I have at least 60 filters set up to route my mail > into > >> them. But even that isn’t perfect. GitHub generates so much noise that > it > >> is impossible to keep up with all the notifications it sends me. That is > >> one of the major reasons I am not in love with the idea of using > >> discussions. > >> > >> Ralph > >> > >>> On Oct 15, 2024, at 4:54 AM, Dominik Psenner <dpsen...@gmail.com> > wrote: > >>> > >>> What concerns me is that email appears to be unreliable where it > >> shouldn't. > >>> Either it works always or it doesn't at all. I experience it to be 100% > >>> reliable. > >>> > >>> Short downtimes are OK and to be expected while systems are patched. > But > >>> then email are delivered later. The technology is much like a postman > who > >>> comes back later if he finds the front gate closed. The postman should > >> not > >>> burn the letter in front of the gate. > >>> > >>> On Mon, 14 Oct 2024, 22:56 Matt Sicker, <m...@musigma.org> wrote: > >>> > >>>> That’s because a lot of other things are also using Slack. On the > other > >>>> hand, I had to disable notifications from Slack due to people misusing > >> it > >>>> to DM me instead of sending emails to the Secretary properly > (unrelated > >> to > >>>> Log4j). > >>>> > >>>>> On Oct 14, 2024, at 15:53, Ralph Goers <ralph.go...@dslextreme.com> > >>>> wrote: > >>>>> > >>>>> I can’t say I agree with that. It didn’t take me very long to get > used > >>>> to using Slack. > >>>>> > >>>>> Ralph > >>>>> > >>>>>> On Oct 14, 2024, at 1:47 PM, Matt Sicker <m...@musigma.org> wrote: > >>>>>> > >>>>>> There’s a very, very small chance I’ll ever remember to visit a > >> website > >>>> to find out about what are essentially emails that could have been > sent > >> to > >>>> me. I have a regular habit of reading email nearly every day, but > >>>> developing new habits is unlikely to stick. > >>>>>> > >>>>>>> On Oct 14, 2024, at 14:50, Ralph Goers <ralph.go...@dslextreme.com > > > >>>> wrote: > >>>>>>> > >>>>>>> Maybe we just need to start contributing to PonyMail to improve the > >> UI > >>>> to eliminate actually needing the email delivered to our accounts. > >>>>>>> > >>>>>>> I am only 1/4 serious about this. There has to be a better > solution. > >>>>>>> > >>>>>>> Ralph > >>>>>>> > >>>>>>>> On Oct 14, 2024, at 10:25 AM, Matt Sicker <m...@musigma.org> > wrote: > >>>>>>>> > >>>>>>>> I didn’t get the original email in this thread once again, so I > >> think > >>>> I’d support trying somewhere else to host discussions. Besides > archiving > >>>> those messages into a mailing list, it would be great if the solution > >>>> provided allowed for email interactivity (e.g., you can reply to the > >>>> notification of a message and it’s added to the thread appropriately; > >> this > >>>> is how GitHub notification emails typically work). > >>>>>>>> > >>>>>>>>> On Oct 10, 2024, at 05:40, Christian Grobmeier < > >> grobme...@apache.org> > >>>> wrote: > >>>>>>>>> > >>>>>>>>> Hi > >>>>>>>>> > >>>>>>>>> I am generally open to such experiments. I would start with the > >>>> easiest parts, such as users@, and see where it goes. > >>>>>>>>> > >>>>>>>>> I would advise against mirroring it to users@ behind the scenes, > >> as > >>>> it may cause privacy problems (we need user consensus to mirror it). > >> When a > >>>> user uses GitHub, they know what to expect. > >>>>>>>>> > >>>>>>>>> As for Discourse, many use that now, but I find it very > >> overwhelming > >>>> and stressful. I prefer the clean Github discussions approach. > >>>>>>>>> > >>>>>>>>> I haven't checked against ASF policies but feel positive about > this > >>>> move for the arguments you have given > >>>>>>>>> > >>>>>>>>> Kind regards > >>>>>>>>> Christian > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Oct 10, 2024, at 10:58, Volkan Yazıcı wrote: > >>>>>>>>>> *Abstract:* Modern email system security measures make it > >>>> practically > >>>>>>>>>> impossible for mailing lists to work – many subscribers don't > get > >>>> all > >>>>>>>>>> emails. This not only hinders communication, but blocks an > >> inclusive > >>>>>>>>>> one. *Shall > >>>>>>>>>> we, as Logging Services, experiment with alternatives?* > >>>>>>>>>> > >>>>>>>>>> *Motivation #1: mailing lists technically don't work* > >>>>>>>>>> > >>>>>>>>>> Several widely-used email providers (GMail, Yahoo, iCloud, etc.) > >>>> have in > >>>>>>>>>> the last couple of years enabled new measures (DMARC, SPF, DKIM, > >>>> etc.) to > >>>>>>>>>> verify the authenticity of emails. In short, these measures > enrich > >>>> email > >>>>>>>>>> content with checksums and signatures capturing its > authenticity. > >>>> When a > >>>>>>>>>> mailing list system (e.g., ezmlm, mailman) receives such an > email, > >>>> it > >>>>>>>>>> performs several changes on its content (adds information about > >> the > >>>> mailing > >>>>>>>>>> list, etc.), and delivers it to all subscribers. When the mail > >>>> server of a > >>>>>>>>>> subscriber receives such tampered mail, and if that mail server > >>>> happens to > >>>>>>>>>> have earlier shared authenticity checks enabled, it discards the > >>>> email, or > >>>>>>>>>> at best, marks it as spam. > >>>>>>>>>> > >>>>>>>>>> Ralph, Matt, Piotr stated many times that they don't receive all > >>>> emails. > >>>>>>>>>> Ralph actually stated many ASF mailing list emails end up in his > >>>> spam > >>>>>>>>>> box > >>>>>>>>>> < > >>>> > >> > https://the-asf.slack.com/archives/CBX4TSBQ8/p1728032221080189?thread_ts=1727958807.348019&cid=CBX4TSBQ8 > >>>>> . > >>>>>>>>>> Recently we witnessed even Brian Proffitt (VP, Marketing & > >>>> Publicity) > >>>>>>>>>> suffer > >>>>>>>>>> from the same problem > >>>>>>>>>> < > https://lists.apache.org/thread/yfmrpjslcbo5jmsqqpvtok1o6lht11rb > >>> . > >>>> > >>>>>>>>>> INFRA > >>>>>>>>>> is crawling with related tickets: INFRA-24574 > >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24574>, > INFRA-24790 > >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24790>, > INFRA-24845 > >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24845>, > INFRA-24850 > >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24850>, > INFRA-24872 > >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-24872>, > INFRA-25947 > >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-25947>, > INFRA-26171 > >>>>>>>>>> <https://issues.apache.org/jira/browse/INFRA-26171> – there are > >>>> dozens > >>>>>>>>>> more. > >>>>>>>>>> > >>>>>>>>>> This technical difficulty is not only known to us. AFAIK, this > is > >>>> one of > >>>>>>>>>> the main reasons PSF (Python Software Foundation) decided to > >> switch > >>>> from > >>>>>>>>>> mailing lists to Discourse. Mailman documents several DMARC > >>>> mitigations > >>>>>>>>>> < > >>>> > >> > https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/dmarc-mitigations.html > >>>>> , > >>>>>>>>>> but I think these are workarounds/hacks rather than > >> well-established > >>>>>>>>>> solutions. > >>>>>>>>>> > >>>>>>>>>> *Motivation #2: ezmlm is dead* > >>>>>>>>>> > >>>>>>>>>> ezmlm, the mailing list software ASF uses, is dead – it is > neither > >>>>>>>>>> developed, nor maintained anymore. (Last official release was in > >>>> 1997, > >>>>>>>>>> there > >>>>>>>>>> is the `ezmlm-idx` add-on, which later on became a successor > >>>>>>>>>> < > >>>> > >> > https://untroubled.org/ezmlm/faq/What-is-the-difference-between-ezmlm-and-ezmlm_002didx_003f.html > >>>>> , > >>>>>>>>>> which last produced a release in 2014, and so on. Long, dead > >>>> story.) > >>>>>>>>>> INFRA > >>>>>>>>>> maintains a very big, sophisticated set of Perl rules for > running > >>>> ASF > >>>>>>>>>> ezmlm > >>>>>>>>>> instances. If you look closely at the INFRA tickets I cited > above, > >>>> some > >>>>>>>>>> suggest INFRA to fork ezmlm and fix some long standing bugs, > etc. > >>>> We can > >>>>>>>>>> discuss the possibility of migrating from ezmlm to mailman (yet > >>>> another > >>>>>>>>>> mailing list software, but one that is still maintained), > whether > >>>> such a > >>>>>>>>>> migration should be practiced ASF-wide or only for Logging > >>>> Services, > >>>>>>>>>> etc. > >>>>>>>>>> But eventually, we will still be using a mailing list, and as I > >>>> tried to > >>>>>>>>>> explain above, IMO, they just don't work good. > >>>>>>>>>> > >>>>>>>>>> *Proposal #1: Experimenting with GitHub Discussions* > >>>>>>>>>> > >>>>>>>>>> GitHub is our development bread and butter. We use its tickets, > >> PRs, > >>>>>>>>>> reviews, discussions, CI, security & code quality checks, etc. > It > >>>> works > >>>>>>>>>> perfectly and components are integrated well, i.e., you can link > >>>> issues, > >>>>>>>>>> comments, PRs, CI runs, etc. Users like it too – we all > witnessed > >>>> the > >>>>>>>>>> sudden increase in user interactions after migrating to GitHub > >>>> Issues > >>>>>>>>>> and > >>>>>>>>>> Discussions. We can configure sections & categories in > Discussions > >>>>>>>>>> < > >>>> > >> > https://docs.github.com/en/discussions/managing-discussions-for-your-community/managing-categories-for-discussions > >>>>> > >>>>>>>>>> to make it serve as our main communication medium. It also > >> provides > >>>> mail > >>>>>>>>>> notifications and the possibility to respond to them for those > who > >>>> still > >>>>>>>>>> prefer their email client over a browser. > >>>>>>>>>> > >>>>>>>>>> In short, we can quickly configure Discussions, update our > support > >>>> policy > >>>>>>>>>> page, and start experimenting with it. > >>>>>>>>>> > >>>>>>>>>> One can raise the argument that what if Discussions disappear? > We > >>>> can > >>>>>>>>>> mirror communication there to a mailing list to be on the safe > >>>> side. Yet, > >>>>>>>>>> we need to evaluate the necessity of this. > >>>>>>>>>> > >>>>>>>>>> *Proposal #2: Experimenting with Discourse* > >>>>>>>>>> > >>>>>>>>>> We can get a VM from INFRA and manage our Discourse instance. > >>>> Though, > >>>>>>>>>> AFAIC, this will result in a "GitHub Discussions"-like setup > with > >>>> all the > >>>>>>>>>> integration goodies missing and added server maintenance burden. > >>>>>>>>>> > >>>>>>>>>> *F.A.Q.* > >>>>>>>>>> > >>>>>>>>>> *What if GitHub Discussions disappear?* > >>>>>>>>>> > >>>>>>>>>> In such a case, I presume they will allow us to download the > >>>> existing > >>>>>>>>>> archives. In the worst case, we can decide to mirror the > >>>> communication > >>>>>>>>>> there to a mailing list. Yet, we need to evaluate the necessity > of > >>>> this. In > >>>>>>>>>> particular, how big of a problem is this at the experimentation > >>>> stage? > >>>>>>>>>> > >>>>>>>>>> *How will private communication work with GitHub Discussions?* > >>>>>>>>>> > >>>>>>>>>> We can create private repositories for internal/private > >>>> communication. > >>>>>>>>>> For > >>>>>>>>>> users/researchers wanting to submit & discuss security issues, > >> they > >>>> can > >>>>>>>>>> get > >>>>>>>>>> in touch with us (either via email to `security@logging` or > some > >>>> other > >>>>>>>>>> ASF/INFRA mailing list), we can grant them permissions to > >>>> collaborate > >>>>>>>>>> privately on a repository security advisory > >>>>>>>>>> < > >>>> > >> > https://docs.github.com/en/code-security/security-advisories/working-with-repository-security-advisories/about-repository-security-advisories > >>>>> > >>>>>>>>>> . > >>>>>>>>>> > >>>>>>>>>> *Don't the ASF legals require mailing lists?* > >>>>>>>>>> > >>>>>>>>>> I am aware that several ASF policies require mailing list > >>>> communication, > >>>>>>>>>> e.g., for voting and such. I first want to establish a consensus > >>>> among us, > >>>>>>>>>> and then pitch to the board for exemption as a pilot. > >>>>>>>>>> > >>>>>>>>>> *Shouldn't this proposal be practiced ASF-wide?* > >>>>>>>>>> > >>>>>>>>>> This will be a very (very very very, actually!) daunting route > to > >>>> pursue. > >>>>>>>>>> I'd rather start small, solve our problem first (if we can), and > >>>> then think > >>>>>>>>>> about widening the scope. > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >> > >> > >
