This hit us again yesterday. Lot's of yahoo spam, from Yahoo mail servers, slipping through because of pbWhite.
Quick summary: I want to be able to block any yahoo mail based on HMM /bayes alone, and I don't want a PB white listing for the sending IP to undo that score. In genreal, the PBWhite is great, but as I've explained, there's too much spam mixed in with the generally good mail from Yahoo IP's. My workaround is to not whitelist IP's for the big carriers using their SPF records to see what IP's they should be sending from. This isn't possible with yahoo, because they use PTR records. Here's a real world example. Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org DKIM-Signature found Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org info: found DKIM signature identity '@yahoo.com' Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org [scoring] DKIM signature verified-OK - header-passed - identity is: @yahoo.com - sender policy is: neutral - author policy is: neutral Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org Message-Score: added -15 (pbwValencePB) for In Penalty White Box, total score for this message is now -15 Sep-12-19 01:17:38 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org HMM Check [scoring] - Prob: 1.00000 - Confidence: 0.21703 => confident.spam - answer/query relation: 100% of 384 Sep-12-19 01:17:38 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org Message-Score: added 50 for HMM Probability: 1.00000, total score for this message is now 35 Sep-12-19 01:17:38 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org [Plugin] calling plugin ASSP_AFC Sep-12-19 01:17:40 84116-18877 [MessageOK] 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org message ok [Hello] -> messages/okmail/Hello--3560453.txt Sep-12-19 01:17:40 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org finished message - received DATA size: 7.96 kByte - sent DATA size: 8.94 kByte Sep-12-19 01:17:40 84116-18877 74.6.129.83 <spam...@yahoo.com> to: u...@ourcharity.org disconnected: session:5151B32A 74.6.129.83 - processing time 4 seconds Everything is good about this mail except its content: it's DKIM signed, passes SPF. ASSP is certain this is HMM spam, but the IP whitelist, takes away 15 and lets it through. Because I have the -15 scoring, it doesn't even prefix the subject. So I'll ask again, can you suggest a way, for Yahoo specifically, but I guess for selected senders in general too, to stop the IP score from reducing the score? I don't want to give a bonus to any Yahoo sender. Having a magic way of not considering the pbwhite/black lists for any IP that reverses to a wildcard (*.yahoo.com for example) would be great. I'm all ears for (and desperate for) suggestions. thanks On Mon, Sep 9, 2019 at 3:51 PM K Post <nntp.p...@gmail.com> wrote: > The problem that I'm talking about with these large mail providers is two > fold: > > First, the overwhelming majority of mails from GMail, Yahoo, and Office365 > are *not* spam. They're DKIM signed, SPF passes, and the content is > good. That quickly gets the sending IP's into the PB whitelist. That in > change reduces the score for any mail from these providers. That in itself > isn't a problem. > > The issue is that there's plenty of spam coming out of the actual > gmail/yahoo/office365 servers too. Are you not encountering this? I can't > imagine that it's just our charity that's seeing this. SO many people > create trial Office365 accounts, sending DKIM validated mail on their own > domain, that's SPAM! Microsoft isn't stopping enough of it. > > These spam messages are validly DKIM signed. pass SPF coming from the > provider's servers, and pass other ip checks. The content usually gets > caught, but because the IP is in the PB white, we take off points and the > message passes often without the spam subject prefix (bayes score minus > pbwhite).. Stopping these major providers from ever being pbwhite or > pbblack lets us still use bayesian filtering to block or at least tag spam > message sent through them that are only caught with bayes. That's worked > well for us for 4+ years using the concept I explained before or parsing > SPF to get allowable sending IP's . I feel like we should ignore the IP if > it's known to be a publicly accessible big email provider, don't penalize > or negatively score. Yahoo using the depreciated PTR syntax in their SPF, > makes this impossible unless ASSP could do PTR lookups and let us do > hostname exceptions. Fortunately gmail, Office365, and others ultimately > list IP ranges. > > Can you help me understand why it's a bad idea / illogical? > > I suppose I could use a spambomb score for anything from > gmail/outlook/hotmail/yahoo to negate the pbWhite, but if it comes from a > new ip that's no in pbWhite yet, now I'm more likely to reject legit mail. > And both gmail and outlook/office365 have services allowing senders to use > their own domain name. > > Second, we get a lot of legitimate, but not signed yahoo mail. This isn't > so much the case with gmail, but for whatever reason, Yahoo users have a > much higher rate of unsigned mail. We've seen order confirmations from > legitimate but poorly configured small ecommerce sites which send with > yahoo addresses, but not through yahoo servers. I'm talking about a > "thanks for your trinket order" automated emails that a etsy type of seller > sends automatically through their ecommerce site using their yahoo > address. I'd love to be able to strictly reject any non-signed yahoo > messages, but it's not a choice. > > I hear you on reporting, but that's not going to fix the broader problem. > Accounts will be shut down, but others will open up. > > And last, this really is a moot point, and I won't belabor the point, but > I'm surprised that you don't think it's possible with some providers to > have them (or do it yourself if you manage a netblock as I once did at a > small ISP) create a ptr record that resolves to a hostname of a domain that > you don't control. I've personally seen a big US business cable company do > this. We wanted website operators to be able to see that web traffic > originating from our office was us. Instead of > local123.region.BigCableISP.net, we wanted our firewall's public IP to > reverse to firewall.OurCharity.org. The ISP mistakenly entered the .com > instead of the .org. For a short while, we'd show up as > firewall.OurCharity.COM, even though we didn't own the .com. Another > time a colleague requested that one of our public IPs reverse to > www.google.com because he thought that would help with a proxy server we > used to us (totally illogical, and he's gone now, but my point is that the > ISP blindly followed the request in the ticket). And I know that plenty of > colocation providers will create PTR records on demand without validation > of domain ownership. Is it an IANNA violoation? I'm sure, but when did > spammers start caring about rules? This is such a minor thing, but if > you're going to essentially lecture me about being stupid, I'm going to > explain. > > > On Mon, Sep 9, 2019 at 4:46 AM Thomas Eckardt <thomas.ecka...@thockar.com> > wrote: > >> >Are you sure??? If I run a DNS server that handles the REVERSE lookups >> for a specific block of IP's, >> >> Yes, I'm 100% sure. YOU (and everyone else) will NOT be able to create a >> PTR record for a not owned domain and a NOT associated IP. PTR's are >> controlled by ISP's, which got IP-ranges for public distribution from and >> under terms of ARIN,RIPE,LACNIC,AFRINIC,KRNIC,APNIC ... - observed by the >> rules of IANA. >> What a (100%) senseless discussion! Repeating anything stupid, unwise or >> wrong over and over. doesn't make it anyhow better nor will it become true. >> Your assumtion, that anyone can easily get or hold its own public reverse >> lookup zone is currently the same way far away from reality, like the >> flight to the center of the milky way. >> >> >What if a yahoo user doesn't send through yahoo.?? >> >> block the mail - valid yahoo users will never able to send valid mails >> without using their yahoo-ISP-authentication >> >> DKIM will fail. SPF will fail if yahoo.com is in >> strictSPFRe >> and/or >> blockstrictSPFRe >> >> "v=spf1 ptr:*yahoo.com* <http://yahoo.com/> ptr:*yahoo.net* >> <http://yahoo.net/> ?all" is NOT strict enougth - "v=spf1 ptr:*yahoo.com* >> <http://yahoo.com/> ptr:*yahoo.net* <http://yahoo.net/> -all" should be >> used (strictSPFRe, blockstrictSPFRe ) >> >> SenderBase and WHOIS-IP may also help in such cases (not only yahoo). >> >> > You don't think that reporting to the carrier is a waste of time? >> The global players care about their reputation. But if no one reports >> abuse - nothing will change. >> BTW: I've got not a single malicious email from any global players >> SMTP-host for a very long time. Yes - from abused/faked addresses - but not >> from their registered hosts. >> Before you report - think about what an abuse is! It is not a mail you >> don't want (others may want them) - abuse is related to malicious code or >> links. Advertising mails are not an abuse outsite the EU! >> >> >> >So can you think of another way to insure that any ip that reverses to >> a yahoo domain isn't ever added to a pbwhite or pbblack? >> No. >> Collecting IP's of global players like yahoo, google...... for pbwhite >> or pbblack does not make any sense to me. They are DKIM signing every >> mail - and they never send mails from outsite their SPF ranges. >> >> >Most US ISP's let you authenticate with the ISP credentials, but send >> as anything once that's done. >> This is commonly done everywhere and nothing bad. SPFoverride may help. >> >> >> the 'elsif (/ptr:/' part of the script will not work in 99% - e.g. for >> all PTR definitions with domain definitions (not hosts) >> >> Thomas >> >> >> >> >> >> Von: "K Post" <nntp.p...@gmail.com> >> An: "ASSP development mailing list" < >> assp-test@lists.sourceforge.net> >> Datum: 09.09.2019 03:46 >> Betreff: Re: [Assp-test] noPB and NoPBWhite based on reverse dns >> ------------------------------ >> >> >> >> Thanks as always for entertaining my questions Thomas. >> My comments below inline. >> >> On Thu, Aug 29, 2019 at 3:17 AM Thomas Eckardt < >> *thomas.ecka...@thockar.com* <thomas.ecka...@thockar.com>> wrote: >> >Is there a way to have an entry like *.*yahoo.com* <http://yahoo.com/> >> in noPB or noPBWhite? >> >> No - hostnames in IP lists are forward lookedup, not reverse >> >> >I've got a little script that takes IP's from SPF records for major >> providers. >> >> How should such a script work for yahoo? - "v=spf1 ptr:*yahoo.com* >> <http://yahoo.com/> ptr:*yahoo.net* <http://yahoo.net/> ?all" >> You would need a list of all defined yahoo related PTR's. >> Yep, that's what my script does, but they're using PTR records unlike the >> others I use (like google, *outlook.com* <http://outlook.com/>, etc) >> >> >However, 66.163.184.147 doesn't match their SPF >> >> it matches yahoo's SFP record - the IP resolves to >> *sonic309-21.consmr.mail.ne1.yahoo.com* >> <http://sonic309-21.consmr.mail.ne1.yahoo.com/> >> the SPF record is "v=spf1 ptr:*yahoo.com* <http://yahoo.com/> ptr: >> *yahoo.net* <http://yahoo.net/> ?all" >> >> True, as long as the IP reverses to a yahoo name, they say it's valid. >> That's why PTR isn't recommended / is depreciated in SPF, more below. >> >> *yahoo.com* <http://yahoo.com/> should be in >> strictSPFRe >> and/or >> blockstrictSPFRe >> I don't know about that. Lots of silly users send yahoo mail through >> home ISP connections and their home ISP's smtp servers. I hate it, but it's >> the reality. Most US ISP's let you authenticate with the ISP credentials, >> but send as anything once that's done. Shameful. >> >> >> Think about the logic - if a mail is valid DKIM signed by *yahoo.com* >> <http://yahoo.com/>, it is impossible that it was sent from an invalid >> SPF IP. >> >> If SPAM are sent from valid *yahoo.com* <http://yahoo.com/> accounts and >> you expect to receive also HAM from there - only the personal black list >> and/or content base checks will help. >> >> If you get attacked with malicious mails from valid yahoo accounts, >> report the abuse to yahoo (or any other major provider). >> We get a TON of spam mails from actual yahoo accounts, sent through yahoo >> servers. You don't think that reporting to the carrier is a waste of time? >> >> >> >I'm aware that a spammer could easily have their ip reverse to a yahoo >> hostname >> >> No this should never be possible (even not in the US). To create a custom >> PTR-record - you need to create the related A or AAAA record first (you >> have to be the domain owner). >> Are you sure??? If I run a DNS server that handles the REVERSE lookups >> for a specific block of IP's, I could have 1.2.3.4 (provided my dns is the >> reverse for the *1.2.3.0/24* <http://1.2.3.0/24> block), reverse to >> *mail.yahoo.com* <http://mail.yahoo.com/>. I'm not saying that >> *mail.yahoo.com* <http://mail.yahoo.com/> will then become 1.2.3.4 but >> if I did a reverse lookup of 1.2.3.4, it would show *mail.yahoo.com* >> <http://mail.yahoo.com/>. ASSP would be angry, because the IP of the >> *mail.yahoo.com* <http://mail.yahoo.com/> A record obviously wouldn't >> match. I feel like I could pass SPF for yahoo if I sent from an IP that I >> control the reverse DNS for. It would definitely fail DKIM. >> >> So can you think of another way to insure that any ip that reverses to a >> yahoo domain isn't ever added to a pbwhite or pbblack? I want to do all >> the other processing, including scoring (but not blocking for the >> previously explained reasons) SPF, DKIM, etc. If we could magically also >> use the PTR record for the sending IP to either match a single hostname, or >> wildcard like *.*yahoo.com* <http://yahoo.com/> in noPBWhite and/or >> noPBBlack, the original issue goes away. It's still imperfect, but at >> least it would allow me to stop heavily used Yahoo ip's which are generally >> sending good mail from getting on the pbwhite and then decreasing the score >> which allows hmm only span through. Do you think it's a good idea to >> negate the pbwhite status by then assigning a negating score to anything >> from an @*yahoo.com* <http://yahoo.com/> sender (net zero sum)? What if >> a yahoo user doesn't send through yahoo.?? >> >> Like I said, my nopb method has been working great with gmail and >> *outlook.com* <http://outlook.com/>, by periodically parsing all of the >> big provider's SPF records to find out which IP's to never PBwhite or >> PBblack. I works really well. There's just too much mail from these >> providers to whilelist or blacklist by IP. Making them good will have too >> much spam slip through. making them black will block way too much >> legitimate mail. Spammers will always abuse these free providers, but the >> services are too prevalent to penalize! It's a bit of a catch 22 that the >> method I use fixes - but not for yahoo because of their stupid SPF record >> using PTR's. >> >> I'm hoping with all my might that I've not been doing something >> incredibly stupid all along, but I know you'll tell me if that's the case! >> Might it be that I've got a system in place that could be rolled into ASSP >> as something that's universally useful? >> >> The script writes IP blocks to the files. Then in the group config, I'll >> do something like this: >> [GROUP-GOOGLE-IPS] >> # include IP-Lists/IPS-google.com.cfg >> >> From there, I can add the group definition that I want to to noPBWhite >> and noPBBlack >> >> Here's the script: >> >> #!/usr/bin/perl -- >> >> # GetDomainIPSfromSPF v0.2 >> >> # Output all IP4 addresses to a file, one per line, from a hostname's SPF >> record(s) >> # does NOT consider PTR records >> >> # Copyright (C) 2015 Ken Post under the terms of GPL v3 >> # >> # This program is free software; you can redistribute it and/or modify >> # it under the terms of the GNU General Public License as published by >> # the Free Software Foundation; either version 3 of the License, or >> # (at your option) any later version. >> # >> # This program is distributed in the hope that it will be useful, >> # but WITHOUT ANY WARRANTY; without even the implied warranty of >> # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> # GNU General Public License (*http://www.gnu.org/licenses/* >> <http://www.gnu.org/licenses/>) for more details. >> >> use strict; >> use warnings; >> >> use Mail::SPF::Query; >> use Net::Nslookup; >> >> >> my @names = ( >> "*ccsend.com* <http://ccsend.com/>", >> "*salesforce.com* <http://salesforce.com/>", >> "*force.com* <http://force.com/>", >> "*mandrillapp.com* <http://mandrillapp.com/>", >> "*outlook.com* <http://outlook.com/>", >> "*hotmail.com* <http://hotmail.com/>", >> "*google.com* <http://google.com/>", >> "*amazon.com* <http://amazon.com/>", >> "*facebook.com* <http://facebook.com/>", >> "*facebookmail.com* <http://facebookmail.com/>", >> "*verticalresponse.com* <http://verticalresponse.com/>", >> "*mailchimp.com* <http://mailchimp.com/>", >> "*bluestatedigital.com* <http://bluestatedigital.com/>", >> "*yahoo.com* <http://yahoo.com/>", >> "*pphosted.com* <http://pphosted.com/>" >> ); >> >> my $hostname = ""; >> my $ipcomment = ""; >> >> foreach my $i (0 .. $#names) { >> >> $hostname = $names[$i]; >> $ipcomment = ""; >> open(my $fh, '>', 'IP-Lists\\IPs-' . $hostname . '.cfg'); >> >> >> print $fh "# \n"; >> print $fh "# Generated by WriteFile-GetDomainIPSFromSPF.pl \n"; >> print $fh "# \n"; >> >> >> RecurseSPF($hostname,'','FROMSPF:' . $ipcomment, $fh); >> >> close $fh; >> } >> >> sub RecurseSPF { >> my ($hostname,$ipcomment,$originalipcomment, $fh) = @_ ; >> >> my @SplitSPFLines; >> >> print "Working on " . $hostname . "\n"; >> >> # get SPF record for the hostname. Using Mail::SPF::Query out of >> convenience, >> # bogus IP and helo sent >> my $query = eval { new Mail::SPF::Query ( >> ip => '1.1.1.1', >> sender => 'someone@' . $hostname, >> helo => 'helo' >> )}; >> >> # spf_record gets populated with the SPF record >> my ($result, $smtp_comment, $header_comment, $spf_record, $detail) = >> $query->result(); >> if (defined $spf_record) { >> # split into an array of words based on spaces >> @SplitSPFLines = split /\s+/, $spf_record; >> } >> >> foreach (@SplitSPFLines) { >> >> # if the word starts include: or redirect: run RecurseSPF >> recursively again, >> # pulling up the SPF record for the referenced hostname >> if (/(include|redirect):/) { >> # strip off include:/redirect: >> s/(include|redirect)://; >> # run it recursively >> #print "# Include SPF for $_\n"; >> RecurseSPF($_,$_,$originalipcomment,$fh); >> >> #if we've found and IP4 record, print that IP address or range >> to stdout >> } elsif (/ip4:/) { >> s/ip4://; >> >> print $fh $_." $originalipcomment $ipcomment\n"; >> } elsif (/ptr:/) { >> s/ptr://; >> >> my @addrs = nslookup(type => "A", domain => $_); >> my $ThePTR = $_; >> >> foreach (@addrs) { >> print $fh $_." $originalipcomment from ptr $ThePTR - >> $ipcomment\n"; >> >> } >> >> } >> } >> } >> >> >> An: "ASSP development mailing list" < >> *assp-test@lists.sourceforge.net* <assp-test@lists.sourceforge.net>> >> Datum: 29.08.2019 02:34 >> Betreff: [Assp-test] noPB and NoPBWhite based on reverse dns >> ------------------------------ >> >> >> >> Is there a way to have an entry like *.*yahoo.com* <http://yahoo.com/> >> in noPB or noPBWhite? I know we can put something like >> *sonic309-21.consmr.mail.ne1.yahoo.com* >> <http://sonic309-21.consmr.mail.ne1.yahoo.com/> but what if I never want >> any IP that reversed to any *yahoo.com* <http://yahoo.com/> name to be >> penalized? I'm aware that a spammer could easily have their ip reverse to >> a yahoo hostname, but I'd hope to catch using other methods. >> >> I've got a little script that takes IP's from SPF records for major >> providers. (I've posted it here before). Those IP's get added to group >> definitions and can be used from there. >> >> One thing I've done for a long time is having the IP's from gmail's and >> yahoo's SPF records in noPB and noPBWhite. This way, these email providers >> are never penalized nor pbWhite. Too many spammers send mail through real >> yahoo and gmail accounts, but we can't negatively score because about 20% >> of our legit inbound mail comes from these 2 providers. We also don't want >> to pbWhite the IP's or bayesian/hmm spam will get 15 points removed and >> pass. This has worked great for a long long time. >> >> However, with yahoo, I'm noticing now that there's inbound mail coming >> from non-SPF matching IP addresses. For example: >> Aug-24-19 12:27:31 61051-11848 66.163.184.147 <*sen...@yahoo.com* >> <sen...@yahoo.com>> to: *ouru...@domain.org* <ouru...@domain.org> >> [scoring] DKIM signature verified-OK - header-passed - identity is: @ >> *yahoo.com* <http://yahoo.com/> - sender policy is: neutral - author >> policy is: neutral >> Aug-24-19 12:27:32 61051-11848 66.163.184.147 <*sen...@yahoo.com* >> <sen...@yahoo.com>> to: *ouru...@domain.org* <ouru...@domain.org> >> Message-Score: added -15 (pbwValencePB) for In Penalty White Box, total >> score for this message is now -15 >> >> That message DKIM verified. It really came through yahoo. However, >> 66.163.184.147 doesn't match their SPF, so it wasn't excluded from my IP >> whitelist. It's in the pbWhite. Even though the message gets 50 for >> bayesian, it starts at -15, so passes. >> >> Any other suggestions are very welcome!! >> thanks >> _______________________________________________ >> Assp-test mailing list >> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net> >> *https://lists.sourceforge.net/lists/listinfo/assp-test* >> <https://lists.sourceforge.net/lists/listinfo/assp-test> >> >> >> >> >> DISCLAIMER: >> ******************************************************* >> This email and any files transmitted with it may be confidential, legally >> privileged and protected in law and are intended solely for the use of the >> individual to whom it is addressed. >> This email was multiple times scanned for viruses. There should be no >> known virus in this email! >> ******************************************************* >> >> _______________________________________________ >> Assp-test mailing list >> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net> >> *https://lists.sourceforge.net/lists/listinfo/assp-test* >> <https://lists.sourceforge.net/lists/listinfo/assp-test> >> _______________________________________________ >> Assp-test mailing list >> Assp-test@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/assp-test >> >> >> >> >> DISCLAIMER: >> ******************************************************* >> This email and any files transmitted with it may be confidential, legally >> privileged and protected in law and are intended solely for the use of the >> individual to whom it is addressed. >> This email was multiple times scanned for viruses. There should be no >> known virus in this email! >> ******************************************************* >> >> _______________________________________________ >> Assp-test mailing list >> Assp-test@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/assp-test >> >
_______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
