Here's a real world example.
Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org DKIM-Signature found
Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org info: found DKIM
signature identity '@yahoo.com <http://yahoo.com>'
Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org [scoring] DKIM
signature verified-OK - header-passed - identity is: @yahoo.com
<http://yahoo.com> - sender policy is: neutral - author policy is: neutral
Sep-12-19 01:17:37 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org Message-Score: added
-15 (pbwValencePB) for In Penalty White Box, total score for this
message is now -15
Sep-12-19 01:17:38 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org HMM Check [scoring]
- Prob: 1.00000 - Confidence: 0.21703 => confident.spam - answer/query
relation: 100% of 384
Sep-12-19 01:17:38 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org Message-Score: added
50 for HMM Probability: 1.00000, total score for this message is now 35
Sep-12-19 01:17:38 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org [Plugin] calling
plugin ASSP_AFC
Sep-12-19 01:17:40 84116-18877 [MessageOK] 74.6.129.83
<spam...@yahoo.com <mailto:spam...@yahoo.com>> to: u...@ourcharity.org
message ok [Hello] -> messages/okmail/Hello--3560453.txt
Sep-12-19 01:17:40 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org finished message -
received DATA size: 7.96 kByte - sent DATA size: 8.94 kByte
Sep-12-19 01:17:40 84116-18877 74.6.129.83 <spam...@yahoo.com
<mailto:spam...@yahoo.com>> to: u...@ourcharity.org disconnected:
session:5151B32A 74.6.129.83 - processing time 4 seconds
Everything is good about this mail except its content: it's DKIM signed,
passes SPF. ASSP is certain this is HMM spam, but the IP whitelist,
takes away 15 and lets it through. Because I have the -15 scoring, it
doesn't even prefix the subject.
So I'll ask again, can you suggest a way, for Yahoo specifically, but I
guess for selected senders in general too, to stop the IP score from
reducing the score? I don't want to give a bonus to any Yahoo sender.
Having a magic way of not considering the pbwhite/black lists for any IP
that reverses to a wildcard (*.yahoo.com <http://yahoo.com> for example)
would be great. I'm all ears for (and desperate for) suggestions.
thanks
On Mon, Sep 9, 2019 at 3:51 PM K Post <nntp.p...@gmail.com
<mailto:nntp.p...@gmail.com>> wrote:
The problem that I'm talking about with these large mail providers
is two fold:
First, the overwhelming majority of mails from GMail, Yahoo, and
Office365 are _not_ spam. They're DKIM signed, SPF passes, and the
content is good. That quickly gets the sending IP's into the PB
whitelist. That in change reduces the score for any mail from these
providers. That in itself isn't a problem.
The issue is that there's plenty of spam coming out of the actual
gmail/yahoo/office365 servers too. Are you not encountering this? I
can't imagine that it's just our charity that's seeing this. SO
many people create trial Office365 accounts, sending DKIM validated
mail on their own domain, that's SPAM! Microsoft isn't stopping
enough of it.
These spam messages are validly DKIM signed. pass SPF coming from
the provider's servers, and pass other ip checks. The content
usually gets caught, but because the IP is in the PB white, we take
off points and the message passes often without the spam subject
prefix (bayes score minus pbwhite).. Stopping these major providers
from ever being pbwhite or pbblack lets us still use bayesian
filtering to block or at least tag spam message sent through them
that are only caught with bayes. That's worked well for us for 4+
years using the concept I explained before or parsing SPF to get
allowable sending IP's . I feel like we should ignore the IP if
it's known to be a publicly accessible big email provider, don't
penalize or negatively score. Yahoo using the depreciated PTR
syntax in their SPF, makes this impossible unless ASSP could do PTR
lookups and let us do hostname exceptions. Fortunately gmail,
Office365, and others ultimately list IP ranges.
Can you help me understand why it's a bad idea / illogical?
I suppose I could use a spambomb score for anything from
gmail/outlook/hotmail/yahoo to negate the pbWhite, but if it comes
from a new ip that's no in pbWhite yet, now I'm more likely to
reject legit mail. And both gmail and outlook/office365 have
services allowing senders to use their own domain name.
Second, we get a lot of legitimate, but not signed yahoo mail. This
isn't so much the case with gmail, but for whatever reason, Yahoo
users have a much higher rate of unsigned mail. We've seen order
confirmations from legitimate but poorly configured small ecommerce
sites which send with yahoo addresses, but not through yahoo
servers. I'm talking about a "thanks for your trinket order"
automated emails that a etsy type of seller sends automatically
through their ecommerce site using their yahoo address. I'd love to
be able to strictly reject any non-signed yahoo messages, but it's
not a choice.
I hear you on reporting, but that's not going to fix the broader
problem. Accounts will be shut down, but others will open up.
And last, this really is a moot point, and I won't belabor the
point, but I'm surprised that you don't think it's possible with
some providers to have them (or do it yourself if you manage a
netblock as I once did at a small ISP) create a ptr record that
resolves to a hostname of a domain that you don't control. I've
personally seen a big US business cable company do this. We wanted
website operators to be able to see that web traffic originating
from our office was us. Instead of local123.region.BigCableISP.net
<http://local123.region.BigCableISP.net>, we wanted our firewall's
public IP to reverse to firewall.OurCharity.org
<http://firewall.OurCharity.org>. The ISP mistakenly entered the
.com instead of the .org. For a short while, we'd show up as
firewall.OurCharity.COM <http://firewall.OurCharity.COM>, even
though we didn't own the .com. Another time a colleague requested
that one of our public IPs reverse to www.google.com
<http://www.google.com> because he thought that would help with a
proxy server we used to us (totally illogical, and he's gone now,
but my point is that the ISP blindly followed the request in the
ticket). And I know that plenty of colocation providers will create
PTR records on demand without validation of domain ownership. Is it
an IANNA violoation? I'm sure, but when did spammers start caring
about rules? This is such a minor thing, but if you're going to
essentially lecture me about being stupid, I'm going to explain.
On Mon, Sep 9, 2019 at 4:46 AM Thomas Eckardt
<thomas.ecka...@thockar.com <mailto:thomas.ecka...@thockar.com>> wrote:
>Are you sure??? If I run a DNS server that handles the REVERSE
lookups for a specific block
of IP's,
Yes, I'm 100% sure. YOU (and everyone else) will NOT be able to
create a PTR record for a not owned domain and a NOT associated
IP. PTR's are controlled by ISP's, which got IP-ranges for
public distribution from and under terms of
ARIN,RIPE,LACNIC,AFRINIC,KRNIC,APNIC ... - observed by the rules
of IANA.
What a (100%) senseless discussion! Repeating anything stupid,
unwise or wrong over and over. doesn't make it anyhow better nor
will it become true.
Your assumtion, that anyone can easily get or hold its own
public reverse lookup zone is currently the same way far away
from reality, like the flight to the center of the milky way.
>What if a yahoo user doesn't send through yahoo.??
block the mail - valid yahoo users will never able to send valid
mails without using their yahoo-ISP-authentication
DKIM will fail. SPF will fail if yahoo.com <http://yahoo.com> is in
strictSPFRe
and/or
blockstrictSPFRe
"v=spf1 ptr:_yahoo.com_ <http://yahoo.com/>ptr:_yahoo.net_
<http://yahoo.net/>?all" is NOT strict enougth - "v=spf1
ptr:_yahoo.com_ <http://yahoo.com/>ptr:_yahoo.net_
<http://yahoo.net/>-all" should be used (strictSPFRe,
blockstrictSPFRe )
SenderBase and WHOIS-IP may also help in such cases (not only
yahoo).
> You don't think that reporting to the carrier is a waste of time?
The global players care about their reputation. But if no one
reports abuse - nothing will change.
BTW: I've got not a single malicious email from any global
players SMTP-host for a very long time. Yes - from abused/faked
addresses - but not from their registered hosts.
Before you report - think about what an abuse is! It is not a
mail you don't want (others may want them) - abuse is related to
malicious code or links. Advertising mails are not an abuse
outsite the EU!
>So can you think of another way to insure that any ip that
reverses to a yahoo domain isn't ever added to a pbwhite or
pbblack?
No.
Collecting IP's of global players like yahoo, google...... for
pbwhite or pbblack does not make any sense to me. They are DKIM
signing every mail - and they never send mails from outsite
their SPF ranges.
>Most US ISP's let you authenticate with the ISP credentials, but
send as anything once that's done.
This is commonly done everywhere and nothing bad. SPFoverride
may help.
the 'elsif(/ptr:/' part of the script will not work in 99% -
e.g. for all PTR definitions with domain definitions (not hosts)
Thomas
Von: "K Post" <nntp.p...@gmail.com <mailto:nntp.p...@gmail.com>>
An: "ASSP development mailing list"
<assp-test@lists.sourceforge.net
<mailto:assp-test@lists.sourceforge.net>>
Datum: 09.09.2019 03:46
Betreff: Re: [Assp-test] noPB and NoPBWhite based on reverse dns
------------------------------------------------------------------------
Thanks as always for entertaining my questions Thomas.
My comments below inline.
On Thu, Aug 29, 2019 at 3:17 AM Thomas Eckardt
<_Thomas.Eckardt@thockar.com_
<mailto:thomas.ecka...@thockar.com>> wrote:
>Is there a way to have an entry like *._yahoo.com_
<http://yahoo.com/>in noPB or noPBWhite?
No - hostnames in IP lists are forward lookedup, not reverse
>I've got a little script that takes IP's from SPF records for
major providers.
How should such a script work for yahoo? - "v=spf1
ptr:_yahoo.com_ <http://yahoo.com/>ptr:_yahoo.net_
<http://yahoo.net/>?all"
You would need a list of all defined yahoo related PTR's.
Yep, that's what my script does, but they're using PTR records
unlike the others I use (like google, _outlook.com_
<http://outlook.com/>, etc)
>However, 66.163.184.147 doesn't match their SPF
it matches yahoo's SFP record - the IP resolves to
_sonic309-21.consmr.mail.ne1.yahoo.com_
<http://sonic309-21.consmr.mail.ne1.yahoo.com/>
the SPF record is "v=spf1 ptr:_yahoo.com_
<http://yahoo.com/>ptr:_yahoo.net_ <http://yahoo.net/>?all"
True, as long as the IP reverses to a yahoo name, they say it's
valid. That's why PTR isn't recommended / is depreciated in
SPF, more below.
_yahoo.com_ <http://yahoo.com/>should be in
strictSPFRe
and/or
blockstrictSPFRe
I don't know about that. Lots of silly users send yahoo mail
through home ISP connections and their home ISP's smtp servers.
I hate it, but it's the reality. Most US ISP's let you
authenticate with the ISP credentials, but send as anything once
that's done. Shameful.
Think about the logic - if a mail is valid DKIM signed by
_yahoo.com_ <http://yahoo.com/>, it is impossible that it was
sent from an invalid SPF IP.
If SPAM are sent from valid _yahoo.com_
<http://yahoo.com/>accounts and you expect to receive also HAM
from there - only the personal black list and/or content base
checks will help.
If you get attacked with malicious mails from valid yahoo
accounts, report the abuse to yahoo (or any other major provider).
We get a TON of spam mails from actual yahoo accounts, sent
through yahoo servers. You don't think that reporting to the
carrier is a waste of time?
>I'm aware that a spammer could easily have their ip reverse to a
yahoo hostname
No this should never be possible (even not in the US). To create
a custom PTR-record - you need to create the related A or AAAA
record first (you have to be the domain owner).
Are you sure??? If I run a DNS server that handles the REVERSE
lookups for a specific block of IP's, I could have 1.2.3.4
(provided my dns is the reverse for the _1.2.3.0/24_
<http://1.2.3.0/24>block), reverse to _mail.yahoo.com_
<http://mail.yahoo.com/>. I'm not saying that _mail.yahoo.com_
<http://mail.yahoo.com/>will then become 1.2.3.4 but if I did a
reverse lookup of 1.2.3.4, it would show _mail.yahoo.com_
<http://mail.yahoo.com/>. ASSP would be angry, because the IP of
the _mail.yahoo.com_ <http://mail.yahoo.com/>A record obviously
wouldn't match. I feel like I could pass SPF for yahoo if I
sent from an IP that I control the reverse DNS for. It would
definitely fail DKIM.
So can you think of another way to insure that any ip that
reverses to a yahoo domain isn't ever added to a pbwhite or
pbblack? I want to do all the other processing, including
scoring (but not blocking for the previously explained reasons)
SPF, DKIM, etc. If we could magically also use the PTR record
for the sending IP to either match a single hostname, or
wildcard like *._yahoo.com_ <http://yahoo.com/>in noPBWhite
and/or noPBBlack, the original issue goes away. It's still
imperfect, but at least it would allow me to stop heavily used
Yahoo ip's which are generally sending good mail from getting on
the pbwhite and then decreasing the score which allows hmm only
span through. Do you think it's a good idea to negate the
pbwhite status by then assigning a negating score to anything
from an @_yahoo.com_ <http://yahoo.com/>sender (net zero sum)?
What if a yahoo user doesn't send through yahoo.??
Like I said, my nopb method has been working great with gmail
and _outlook.com_ <http://outlook.com/>, by periodically parsing
all of the big provider's SPF records to find out which IP's to
never PBwhite or PBblack. I works really well. There's just too
much mail from these providers to whilelist or blacklist by IP.
Making them good will have too much spam slip through. making
them black will block way too much legitimate mail. Spammers
will always abuse these free providers, but the services are too
prevalent to penalize! It's a bit of a catch 22 that the method
I use fixes - but not for yahoo because of their stupid SPF
record using PTR's.
I'm hoping with all my might that I've not been doing something
incredibly stupid all along, but I know you'll tell me if that's
the case! Might it be that I've got a system in place that
could be rolled into ASSP as something that's universally useful?
The script writes IP blocks to the files. Then in the group
config, I'll do something like this:
[GROUP-GOOGLE-IPS]
# include IP-Lists/IPS-google.com.cfg
From there, I can add the group definition that I want to to
noPBWhite and noPBBlack
Here's the script:
#!/usr/bin/perl --
# GetDomainIPSfromSPF v0.2
# Output all IP4 addresses to a file, one per line, from a
hostname's SPF record(s)
# does NOT consider PTR records
# Copyright (C) 2015 Ken Post under the terms of GPL v3
#
# This program is free software; you can redistribute it and/or
modify
# it under the terms of the GNU General Public License as
published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License (_http://www.gnu.org/licenses/_)
for more details.
usestrict;
usewarnings;
useMail::SPF::Query;
useNet::Nslookup;
my@names= (
"_ccsend.com_ <http://ccsend.com/>",
"_salesforce.com_ <http://salesforce.com/>",
"_force.com_ <http://force.com/>",
"_mandrillapp.com_ <http://mandrillapp.com/>",
"_outlook.com_ <http://outlook.com/>",
"_hotmail.com_ <http://hotmail.com/>",
"_google.com_ <http://google.com/>",
"_amazon.com_ <http://amazon.com/>",
"_facebook.com_ <http://facebook.com/>",
"_facebookmail.com_ <http://facebookmail.com/>",
"_verticalresponse.com_ <http://verticalresponse.com/>",
"_mailchimp.com_ <http://mailchimp.com/>",
"_bluestatedigital.com_ <http://bluestatedigital.com/>",
"_yahoo.com_ <http://yahoo.com/>",
"_pphosted.com_ <http://pphosted.com/>"
);
my$hostname= "";
my$ipcomment= "";
foreachmy$i(0 .. $#names) {
$hostname= $names[$i];
$ipcomment= "";
open(my$fh, '>', 'IP-Lists\\IPs-'. $hostname. '.cfg');
print$fh"# \n";
print$fh"# Generated by WriteFile-GetDomainIPSFromSPF.pl \n";
print$fh"# \n";
RecurseSPF($hostname,'','FROMSPF:'. $ipcomment, $fh);
close$fh;
}
subRecurseSPF{
my($hostname,$ipcomment,$originalipcomment, $fh) = @_;
my@SplitSPFLines;
print"Working on ". $hostname. "\n";
# get SPF record for the hostname. Using Mail::SPF::Query out of
convenience,
# bogus IP and helo sent
my$query= eval{ new Mail::SPF::Query (
ip => '1.1.1.1',
sender => 'someone@'. $hostname,
helo => 'helo'
)};
# spf_record gets populated with the SPF record
my($result, $smtp_comment, $header_comment, $spf_record,
$detail) = $query->result();
if(defined$spf_record) {
# split into an array of words based on spaces
@SplitSPFLines= split/\s+/, $spf_record;
}
foreach(@SplitSPFLines) {
# if the word starts include: or redirect: run RecurseSPF
recursively again,
# pulling up the SPF record for the referenced hostname
if(/(include|redirect):/) {
# strip off include:/redirect:
s/(include|redirect)://;
# run it recursively
#print "# Include SPF for $_\n";
RecurseSPF($_,$_,$originalipcomment,$fh);
#if we've found and IP4 record, print that IP address or range
to stdout
} elsif(/ip4:/) {
s/ip4://;
print$fh$_." $originalipcomment$ipcomment\n";
} elsif(/ptr:/) {
s/ptr://;
my@addrs= nslookup(type => "A", domain => $_);
my$ThePTR= $_;
foreach(@addrs) {
print$fh$_." $originalipcommentfrom ptr $ThePTR- $ipcomment\n";
}
}
}
}
An: "ASSP development mailing list"
<_assp-test@lists.sourceforge.net_
<mailto:assp-test@lists.sourceforge.net>>
Datum: 29.08.2019 02:34
Betreff: [Assp-test] noPB and NoPBWhite based on reverse dns
------------------------------------------------------------------------
Is there a way to have an entry like *._yahoo.com_
<http://yahoo.com/>in noPB or noPBWhite? I know we can put
something like _sonic309-21.consmr.mail.ne1.yahoo.com_
<http://sonic309-21.consmr.mail.ne1.yahoo.com/>but what if I
never want any IP that reversed to any _yahoo.com_
<http://yahoo.com/>name to be penalized? I'm aware that a
spammer could easily have their ip reverse to a yahoo hostname,
but I'd hope to catch using other methods.
I've got a little script that takes IP's from SPF records for
major providers. (I've posted it here before). Those IP's get
added to group definitions and can be used from there.
One thing I've done for a long time is having the IP's from
gmail's and yahoo's SPF records in noPB and noPBWhite. This
way, these email providers are never penalized nor pbWhite. Too
many spammers send mail through real yahoo and gmail accounts,
but we can't negatively score because about 20% of our legit
inbound mail comes from these 2 providers. We also don't want to
pbWhite the IP's or bayesian/hmm spam will get 15 points removed
and pass. This has worked great for a long long time.
However, with yahoo, I'm noticing now that there's inbound mail
coming from non-SPF matching IP addresses. For example:
Aug-24-19 12:27:31 61051-11848 66.163.184.147
<_sender@yahoo.com_ <mailto:sen...@yahoo.com>> to:
_ouruser@domain.org_ <mailto:ouru...@domain.org>[scoring] DKIM
signature verified-OK - header-passed - identity is:
@_yahoo.com_ <http://yahoo.com/>- sender policy is: neutral -
author policy is: neutral
Aug-24-19 12:27:32 61051-11848 66.163.184.147
<_sender@yahoo.com_ <mailto:sen...@yahoo.com>> to:
_ouruser@domain.org_ <mailto:ouru...@domain.org>Message-Score:
added -15 (pbwValencePB) for In Penalty White Box, total score
for this message is now -15
That message DKIM verified. It really came through yahoo.
However, 66.163.184.147 doesn't match their SPF, so it wasn't
excluded from my IP whitelist. It's in the pbWhite. Even
though the message gets 50 for bayesian, it starts at -15, so
passes.
Any other suggestions are very welcome!!
thanks
_______________________________________________
Assp-test mailing list_
__Assp-test@lists.sourceforge.net_
<mailto:Assp-test@lists.sourceforge.net>_
__https://lists.sourceforge.net/lists/listinfo/assp-test_
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be
confidential, legally privileged and protected in law and are
intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should
be no known virus in this email!
*******************************************************
_______________________________________________
Assp-test mailing list_
__Assp-test@lists.sourceforge.net_
<mailto:Assp-test@lists.sourceforge.net>_
__https://lists.sourceforge.net/lists/listinfo/assp-test________________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
<mailto:Assp-test@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be
confidential, legally privileged and protected in law and are
intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should
be no known virus in this email!
*******************************************************
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
<mailto:Assp-test@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/assp-test
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
