Thanks as always for entertaining my questions Thomas. My comments below inline.
On Thu, Aug 29, 2019 at 3:17 AM Thomas Eckardt <thomas.ecka...@thockar.com> wrote: > >Is there a way to have an entry like *.*yahoo.com* <http://yahoo.com/> > in noPB or noPBWhite? > > No - hostnames in IP lists are forward lookedup, not reverse > > >I've got a little script that takes IP's from SPF records for major > providers. > > How should such a script work for yahoo? - "v=spf1 ptr:yahoo.com ptr: > yahoo.net ?all" > You would need a list of all defined yahoo related PTR's. > Yep, that's what my script does, but they're using PTR records unlike the others I use (like google, outlook.com, etc) > > >However, 66.163.184.147 doesn't match their SPF > > it matches yahoo's SFP record - the IP resolves to > sonic309-21.consmr.mail.ne1.yahoo.com > the SPF record is "v=spf1 ptr:yahoo.com ptr:yahoo.net ?all" > > True, as long as the IP reverses to a yahoo name, they say it's valid. That's why PTR isn't recommended / is depreciated in SPF, more below. > yahoo.com should be in > strictSPFRe > and/or > blockstrictSPFRe > I don't know about that. Lots of silly users send yahoo mail through home ISP connections and their home ISP's smtp servers. I hate it, but it's the reality. Most US ISP's let you authenticate with the ISP credentials, but send as anything once that's done. Shameful. > > Think about the logic - if a mail is valid DKIM signed by yahoo.com, it > is impossible that it was sent from an invalid SPF IP. > > If SPAM are sent from valid yahoo.com accounts and you expect to receive > also HAM from there - only the personal black list and/or content base > checks will help. > > If you get attacked with malicious mails from valid yahoo accounts, report > the abuse to yahoo (or any other major provider). > We get a TON of spam mails from actual yahoo accounts, sent through yahoo servers. You don't think that reporting to the carrier is a waste of time? > >I'm aware that a spammer could easily have their ip reverse to a yahoo > hostname > > No this should never be possible (even not in the US). To create a custom > PTR-record - you need to create the related A or AAAA record first (you > have to be the domain owner). > Are you sure??? If I run a DNS server that handles the REVERSE lookups for a specific block of IP's, I could have 1.2.3.4 (provided my dns is the reverse for the 1.2.3.0/24 block), reverse to mail.yahoo.com. I'm not saying that mail.yahoo.com will then become 1.2.3.4 but if I did a reverse lookup of 1.2.3.4, it would show mail.yahoo.com. ASSP would be angry, because the IP of the mail.yahoo.com A record obviously wouldn't match. I feel like I could pass SPF for yahoo if I sent from an IP that I control the reverse DNS for. It would definitely fail DKIM. So can you think of another way to insure that any ip that reverses to a yahoo domain isn't ever added to a pbwhite or pbblack? I want to do all the other processing, including scoring (but not blocking for the previously explained reasons) SPF, DKIM, etc. If we could magically also use the PTR record for the sending IP to either match a single hostname, or wildcard like *.yahoo.com in noPBWhite and/or noPBBlack, the original issue goes away. It's still imperfect, but at least it would allow me to stop heavily used Yahoo ip's which are generally sending good mail from getting on the pbwhite and then decreasing the score which allows hmm only span through. Do you think it's a good idea to negate the pbwhite status by then assigning a negating score to anything from an @yahoo.com sender (net zero sum)? What if a yahoo user doesn't send through yahoo.?? Like I said, my nopb method has been working great with gmail and outlook.com, by periodically parsing all of the big provider's SPF records to find out which IP's to never PBwhite or PBblack. I works really well. There's just too much mail from these providers to whilelist or blacklist by IP. Making them good will have too much spam slip through. making them black will block way too much legitimate mail. Spammers will always abuse these free providers, but the services are too prevalent to penalize! It's a bit of a catch 22 that the method I use fixes - but not for yahoo because of their stupid SPF record using PTR's. I'm hoping with all my might that I've not been doing something incredibly stupid all along, but I know you'll tell me if that's the case! Might it be that I've got a system in place that could be rolled into ASSP as something that's universally useful? The script writes IP blocks to the files. Then in the group config, I'll do something like this: [GROUP-GOOGLE-IPS] # include IP-Lists/IPS-google.com.cfg >From there, I can add the group definition that I want to to noPBWhite and noPBBlack Here's the script: #!/usr/bin/perl -- # GetDomainIPSfromSPF v0.2 # Output all IP4 addresses to a file, one per line, from a hostname's SPF record(s) # does NOT consider PTR records # Copyright (C) 2015 Ken Post under the terms of GPL v3 # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License (http://www.gnu.org/licenses/) for more details. use strict; use warnings; use Mail::SPF::Query; use Net::Nslookup; my @names = ( "ccsend.com", "salesforce.com", "force.com", "mandrillapp.com", "outlook.com", "hotmail.com", "google.com", "amazon.com", "facebook.com", "facebookmail.com", "verticalresponse.com", "mailchimp.com", "bluestatedigital.com", "yahoo.com", "pphosted.com" ); my $hostname = ""; my $ipcomment = ""; foreach my $i (0 .. $#names) { $hostname = $names[$i]; $ipcomment = ""; open(my $fh, '>', 'IP-Lists\\IPs-' . $hostname . '.cfg'); print $fh "# \n"; print $fh "# Generated by WriteFile-GetDomainIPSFromSPF.pl \n"; print $fh "# \n"; RecurseSPF($hostname,'','FROMSPF:' . $ipcomment, $fh); close $fh; } sub RecurseSPF { my ($hostname,$ipcomment,$originalipcomment, $fh) = @_ ; my @SplitSPFLines; print "Working on " . $hostname . "\n"; # get SPF record for the hostname. Using Mail::SPF::Query out of convenience, # bogus IP and helo sent my $query = eval { new Mail::SPF::Query ( ip => '1.1.1.1', sender => 'someone@' . $hostname, helo => 'helo' )}; # spf_record gets populated with the SPF record my ($result, $smtp_comment, $header_comment, $spf_record, $detail) = $query->result(); if (defined $spf_record) { # split into an array of words based on spaces @SplitSPFLines = split /\s+/, $spf_record; } foreach (@SplitSPFLines) { # if the word starts include: or redirect: run RecurseSPF recursively again, # pulling up the SPF record for the referenced hostname if (/(include|redirect):/) { # strip off include:/redirect: s/(include|redirect)://; # run it recursively #print "# Include SPF for $_\n"; RecurseSPF($_,$_,$originalipcomment,$fh); #if we've found and IP4 record, print that IP address or range to stdout } elsif (/ip4:/) { s/ip4://; print $fh $_." $originalipcomment $ipcomment\n"; } elsif (/ptr:/) { s/ptr://; my @addrs = nslookup(type => "A", domain => $_); my $ThePTR = $_; foreach (@addrs) { print $fh $_." $originalipcomment from ptr $ThePTR - $ipcomment\n"; } } } } An: "ASSP development mailing list" <assp-test@lists.sourceforge.net> > Datum: 29.08.2019 02:34 > Betreff: [Assp-test] noPB and NoPBWhite based on reverse dns > ------------------------------ > > > > Is there a way to have an entry like *.*yahoo.com* <http://yahoo.com/> in > noPB or noPBWhite? I know we can put something like > *sonic309-21.consmr.mail.ne1.yahoo.com* > <http://sonic309-21.consmr.mail.ne1.yahoo.com/> but what if I never want > any IP that reversed to any *yahoo.com* <http://yahoo.com/> name to be > penalized? I'm aware that a spammer could easily have their ip reverse to > a yahoo hostname, but I'd hope to catch using other methods. > > I've got a little script that takes IP's from SPF records for major > providers. (I've posted it here before). Those IP's get added to group > definitions and can be used from there. > > One thing I've done for a long time is having the IP's from gmail's and > yahoo's SPF records in noPB and noPBWhite. This way, these email providers > are never penalized nor pbWhite. Too many spammers send mail through real > yahoo and gmail accounts, but we can't negatively score because about 20% > of our legit inbound mail comes from these 2 providers. We also don't want > to pbWhite the IP's or bayesian/hmm spam will get 15 points removed and > pass. This has worked great for a long long time. > > However, with yahoo, I'm noticing now that there's inbound mail coming > from non-SPF matching IP addresses. For example: > Aug-24-19 12:27:31 61051-11848 66.163.184.147 <*sen...@yahoo.com* > <sen...@yahoo.com>> to: *ouru...@domain.org* <ouru...@domain.org> > [scoring] DKIM signature verified-OK - header-passed - identity is: @ > *yahoo.com* <http://yahoo.com/> - sender policy is: neutral - author > policy is: neutral > Aug-24-19 12:27:32 61051-11848 66.163.184.147 <*sen...@yahoo.com* > <sen...@yahoo.com>> to: *ouru...@domain.org* <ouru...@domain.org> > Message-Score: added -15 (pbwValencePB) for In Penalty White Box, total > score for this message is now -15 > > That message DKIM verified. It really came through yahoo. However, > 66.163.184.147 doesn't match their SPF, so it wasn't excluded from my IP > whitelist. It's in the pbWhite. Even though the message gets 50 for > bayesian, it starts at -15, so passes. > > Any other suggestions are very welcome!! > thanks > _______________________________________________ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > DISCLAIMER: > ******************************************************* > This email and any files transmitted with it may be confidential, legally > privileged and protected in law and are intended solely for the use of the > individual to whom it is addressed. > This email was multiple times scanned for viruses. There should be no > known virus in this email! > ******************************************************* > > _______________________________________________ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test >
_______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
