Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
On 2021-07-12 at 14:38:43 UTC-0400 (Mon, 12 Jul 2021 20:38:43 +0200) Robert Harnischmacher is rumored to have said: > Hi Bill, > > thanks for the detailed explanations. I understand the purpose of the > def_whitelist_auth list better now, but wonder if its benefit is not > overcompensated by significant negative effects, certainly not desired by the > community. > > First of all, I would like to contribute some statistical findings: > > A look at the exemplary group of the largest 1,000 U.S. online stores > according to Alexa Rank shows that about 15 percent of the domains are > whitelisted in 60_whitelist_auth.cf. There are no significant and especially > no consistent differences in the email reputation of these 15 percent > compared to the rest of the top 1,000. Sure there is: they have a default WL entry in SpamAssasin. That's a reputation metric in itself, albeit a very noisy and incomplete metric. I feel like I need to point out that SpamAssassin is designed on the premise that there is no such thing as a perfect rule for discriminating between spam and ham. Imperfection in a SA rule or feature is not enough of a reason for it to be removed altogether. > This would not be a problem if the Spamassassin whitelist did not > unintentionally give the 15 percent a competitive advantage. Based on the > high spam score bonus of 7.5 points, which USER_IN_DEF_DKIM_WL and > USER_IN_DEF_SPF_WL bring, one can for example risk a higher frequency of mass > mailings, run riskier reactivation campaigns or write to "broader" > distribution lists: Possible spam scores, for example due to blacklisting, > would be ironed out by the above-mentioned "bonus". And indeed: With some > stores from the 15 percent group I see again and again - partly even > consistently - serious blacklistings. If you (or anyone) has definitive evidence of a sender with a default WL entry sending spam which is classified by SA as ham incorrectly, you can submit it in a bug report and there's a very strong chance that the WL entry will be removed. Hand-waving about what the general feature of a default WL might enable for hypothetically listed senders is not an actionable bug report. If we removed every feature in SA that had hypothetical negative effects, it would be a useless and tiny bit of software. > There are about 16 DKIM rules and 12 SPF rules in Spamassassin that are meant > to evaluate in a technically automated way whether and how good the SPF and > DKIM implementation of a sender is. Interestingly, the comment in > 60_whitelist_auth.cf. says: "These listings are intended to (...) reward > senders that send with good SPF, DKIM, and DMARC." With this in mind, it > seems like a logical overlap to me that USER_IN_DEF_DKIM_WL and > USER_IN_DEF_SPF_WL introduce additional high "bonus" scores based solely on > human judgment at the one-time point of a check. Almost all of the senders > listed in 60_whitelist_auth.cf have changed their email service providers one > or more times over the years, with sometimes significant changes in the > quality of their deliverability settings and with significant differences in > list hygiene, sending frequency, etc. But the spam score bonus of 7.5 remains > nailed down all the time! Anyone using SA can set that value to anything they like, including zero, or knock out individual senders from the list as they like. For example, I knock the bonus for those rules down to 2 points on systems where I control scoring. Anyone running SA can do the same. > In short, I would recommend considering removing the DKIM and SPF whitelists > in Spamassassin altogether. It would make the spam-fighting world a better > and fairer place! I am unconvinced that "better" is true and I am quite sure that "fairer" doesn't have a useful definition in this context. Nothing in SA is designed to provide "fairness" between different senders or between senders and recipients. We are heavily biased towards our users, who are the recipients and their immediate service providers. If a feature benefits THEM uniformly while unevenly punishing/benefitting various senders, that's just fine. The purpose of the default WL is to eliminate "false positive" spam identification for mail whose senders have had such problems for affirmatively wanted mail (as noticed by SA users) while NOT sending any discernible spam. As far as I can tell, there's never been a case of a sender successfully advocating for inclusion or even being consulted about inclusion or removal. We've not had any cases of a listing being a matter of meaningful controversy. I have no idea how we'd handle anything resembling an edge case. >> Am 29.06.2021 um 06:52 schrieb Bill Cole >> : >> >> On 2021-06-28 at 17:04:05 UTC-0400 (Mon, 28 Jun 2021 23:04:05 +0200) >> Robert Harnischmacher >> is rumored to have said: >> >>> In which form can one submit the subdomain of a mail sender for the >>> integration in
Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
Hi Bill, thanks for the detailed explanations. I understand the purpose of the def_whitelist_auth list better now, but wonder if its benefit is not overcompensated by significant negative effects, certainly not desired by the community. First of all, I would like to contribute some statistical findings: A look at the exemplary group of the largest 1,000 U.S. online stores according to Alexa Rank shows that about 15 percent of the domains are whitelisted in 60_whitelist_auth.cf. There are no significant and especially no consistent differences in the email reputation of these 15 percent compared to the rest of the top 1,000. This would not be a problem if the Spamassassin whitelist did not unintentionally give the 15 percent a competitive advantage. Based on the high spam score bonus of 7.5 points, which USER_IN_DEF_DKIM_WL and USER_IN_DEF_SPF_WL bring, one can for example risk a higher frequency of mass mailings, run riskier reactivation campaigns or write to "broader" distribution lists: Possible spam scores, for example due to blacklisting, would be ironed out by the above-mentioned "bonus". And indeed: With some stores from the 15 percent group I see again and again - partly even consistently - serious blacklistings. There are about 16 DKIM rules and 12 SPF rules in Spamassassin that are meant to evaluate in a technically automated way whether and how good the SPF and DKIM implementation of a sender is. Interestingly, the comment in 60_whitelist_auth.cf. says: "These listings are intended to (...) reward senders that send with good SPF, DKIM, and DMARC." With this in mind, it seems like a logical overlap to me that USER_IN_DEF_DKIM_WL and USER_IN_DEF_SPF_WL introduce additional high "bonus" scores based solely on human judgment at the one-time point of a check. Almost all of the senders listed in 60_whitelist_auth.cf have changed their email service providers one or more times over the years, with sometimes significant changes in the quality of their deliverability settings and with significant differences in list hygiene, sending frequency, etc. But the spam score bonus of 7.5 remains nailed down all the time! In short, I would recommend considering removing the DKIM and SPF whitelists in Spamassassin altogether. It would make the spam-fighting world a better and fairer place! Best, Robert > Am 29.06.2021 um 06:52 schrieb Bill Cole > : > > On 2021-06-28 at 17:04:05 UTC-0400 (Mon, 28 Jun 2021 23:04:05 +0200) > Robert Harnischmacher > is rumored to have said: > >> In which form can one submit the subdomain of a mail sender for the >> integration in 60_whitelist_auth.cf. Which information is required for >> consideration? > > There is no process by which a sender can pro-actively apply for the addition > of a def_whitelist_auth entry in that file. Entries are added rarely, when a > committer to the project sees a need for an entry due to false positives or > borderline scoring of messages from a sender who is not known to send ANY > spam and is known to send "ham" that users value highly. Removal of entries > is equally ad hoc and unilateral, and more rare. If a committer is convinced > that an entry is causing spam to be misclassified as ham, they can remove > that entry. > > Note that the above describes concrete process and vague criteria, not any > sort of objective formal policy. There is no objective official policy. The > normal state for any sender is to not have an entry. I believe that most > committers to the project would agree with me that ideally there would be no > such list because high-value ham would be more readily distinguishable from > spam. Additions and removals happen when they are believed to address a > concrete problem being experienced by actual SpamAssassin users. I don't > recall any significant disagreements about entries in that list, but if there > were any they could be discussed here or on the 'dev' list. Ultimately, the > PMC would be the final authority on including an entry or not, however our > processes for deciding anything that becomes an issue for the PMC is biased > towards stability, not agility. > > > > > -- > Bill Cole > b...@scconsult.com or billc...@apache.org > (AKA @grumpybozo and many *@billmail.scconsult.com addresses) > Not Currently Available For Hire smime.p7s Description: S/MIME cryptographic signature
Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
On Thu, 2021-07-01 at 16:32 -0600, @lbutlr wrote: > Sending spam, viruses, ransom demands, and/or spearfishing from > "known" addresses is extremely common, so how effective that is > depends a lot on the sort of mail and the amount of mail you receive. > Agreed, but I'm not silly enough to have the whitelisting check trigger shortcircuiting or use a whitelist score high enough to override a bunch of spam hits: I wouldn't be using it in that case. > It is very common for me to get spam mail that appears to be from > known addresses, mostly clients and the less sophisticated family > members (computer sophisticated, at least) who have the bad habit of > sharing their contacts with whatever random app they download. > Different sender population, then. I get very little spam that spoofs regular correspondents. > > If a company insists on sending me advertising mail I do not want, I > don't want to do any business with that company. > Agreed, but most of those, at least in my experience, use a different sending address for adspam than they use for invoices, dispatch notes, etc. so my adspam blacklist works rather well. Before you ask, my daily logwatch reports monitor the performance of local SA rules: I wrote report modules to do that. Seems to me there's little point in writing, testing and tuning local rule sets if you can't easily see how well they're working. Martin
Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
On 01 Jul 2021, at 16:43, Reindl Harald wrote: > Am 02.07.21 um 00:32 schrieb @lbutlr: >>> I also manually maintain a private blacklist, which contains the 'From' >>> addresses of advertising e-mails from companies that I've dealt with in >>> the past. This works because many (most?) companies use different >>> subdomains for advertising messages than they use for order >>> confirmations etc. This makes blacklisting the advertising 'From' >>> addresses very simple to do and is a manual process. >> If a company insists on sending me advertising mail I do not want, I don't >> want to do any business with that company > > not everyones mailserver is just for him and his family > in your case you may reject the world except whitelists > fine for a child setup On my mail server any domain admin can blacklist any email address, either for the domain itself or for specific addresses. -- "We take off our Republican hats and put on our American hats" -- Many Republicans in Sep 2008
Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
On 29 Jun 2021, at 04:50, Martin Gregorie wrote: > On Tue, 2021-06-29 at 00:52 -0400, Bill Cole wrote: >> On 2021-06-28 at 17:04:05 UTC-0400 (Mon, 28 Jun 2021 23:04:05 +0200) >> Robert Harnischmacher >> is rumored to have said: >>> In which form can one submit the subdomain of a mail sender for the >>> integration in 60_whitelist_auth.cf. Which information is required >>> for >>> consideration? > There's nothing preventing yo from maintaining your own whitelist (and > blacklist). > > I wrote my own automatic whitelister, which whitelists mail from anybody > I've sent mail to. It works by scanning my outgoing mail stream: almost > no maintenance needed and it would be quite difficult to spoof. Sending spam, viruses, ransom demands, and/or spearfishing from "known" addresses is extremely common, so how effective that is depends a lot on the sort of mail and the amount of mail you receive. It is very common for me to get spam mail that appears to be from known addresses, mostly clients and the less sophisticated family members (computer sophisticated, at least) who have the bad habit of sharing their contacts with whatever random app they download. > I also manually maintain a private blacklist, which contains the 'From' > addresses of advertising e-mails from companies that I've dealt with in > the past. This works because many (most?) companies use different > subdomains for advertising messages than they use for order > confirmations etc. This makes blacklisting the advertising 'From' > addresses very simple to do and is a manual process. If a company insists on sending me advertising mail I do not want, I don't want to do any business with that company. -- 'You're your own worst enemy, Rincewind,' said the sword. Rincewind looked up at the grinning men. 'Bet?' --Colour of Magic
Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
On Tue, 29 Jun 2021 11:50:46 +0100 Martin Gregorie wrote: > > On 2021-06-28 at 17:04:05 UTC-0400 (Mon, 28 Jun 2021 23:04:05 +0200) > > Robert Harnischmacher > > is rumored to have said: > > > > > In which form can one submit the subdomain of a mail sender for > > > the integration in 60_whitelist_auth.cf. Which information is > > > required for > > > consideration? > > > > > There's nothing preventing yo from maintaining your own whitelist (and > blacklist). I'd be surprised if this has anything to do with the incoming mail of Publicare Marketing Communications.
Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
On Tue, 2021-06-29 at 00:52 -0400, Bill Cole wrote: > On 2021-06-28 at 17:04:05 UTC-0400 (Mon, 28 Jun 2021 23:04:05 +0200) > Robert Harnischmacher > is rumored to have said: > > > In which form can one submit the subdomain of a mail sender for the > > integration in 60_whitelist_auth.cf. Which information is required > > for > > consideration? > > There's nothing preventing yo from maintaining your own whitelist (and blacklist). I wrote my own automatic whitelister, which whitelists mail from anybody I've sent mail to. It works by scanning my outgoing mail stream: almost no maintenance needed and it would be quite difficult to spoof. I also manually maintain a private blacklist, which contains the 'From' addresses of advertising e-mails from companies that I've dealt with in the past. This works because many (most?) companies use different subdomains for advertising messages than they use for order confirmations etc. This makes blacklisting the advertising 'From' addresses very simple to do and is a manual process. Martin
Re: Process of domain submission for inclusion in 60_whitelist_auth.cf
On 2021-06-28 at 17:04:05 UTC-0400 (Mon, 28 Jun 2021 23:04:05 +0200) Robert Harnischmacher is rumored to have said: In which form can one submit the subdomain of a mail sender for the integration in 60_whitelist_auth.cf. Which information is required for consideration? There is no process by which a sender can pro-actively apply for the addition of a def_whitelist_auth entry in that file. Entries are added rarely, when a committer to the project sees a need for an entry due to false positives or borderline scoring of messages from a sender who is not known to send ANY spam and is known to send "ham" that users value highly. Removal of entries is equally ad hoc and unilateral, and more rare. If a committer is convinced that an entry is causing spam to be misclassified as ham, they can remove that entry. Note that the above describes concrete process and vague criteria, not any sort of objective formal policy. There is no objective official policy. The normal state for any sender is to not have an entry. I believe that most committers to the project would agree with me that ideally there would be no such list because high-value ham would be more readily distinguishable from spam. Additions and removals happen when they are believed to address a concrete problem being experienced by actual SpamAssassin users. I don't recall any significant disagreements about entries in that list, but if there were any they could be discussed here or on the 'dev' list. Ultimately, the PMC would be the final authority on including an entry or not, however our processes for deciding anything that becomes an issue for the PMC is biased towards stability, not agility. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire
Process of domain submission for inclusion in 60_whitelist_auth.cf
In which form can one submit the subdomain of a mail sender for the integration in 60_whitelist_auth.cf. Which information is required for consideration? Thank you! Best, Robert
