Zones2 Dying - HW Failures likely a key cause in masscheck Failures

[Kevin A. McGrail]

Just a heads up that I've been working on getting a new VM stood up to 
replace two of our current VMs for the project involved with masscheck.


At the same time, I've been trying to identify some of the issues 
especially since some of the issue involved processing we handled that 
was not getting the correct svn trunk, etc.


It was getting to be my guess it was hardware and infra has just 
confirmed it.


ERROR: spamassassin2/home/automc/tmp/ham.log.8 failed verification -- 
update discarded.
rsync: read errors mapping 
"/zonestorage/spamassassin2/home/automc/tmp/ham.log.8": I/O error (5)

Anyway, good that we uncovered it, good that we have a copy on another VM now 
but expect that things might fail worse before I can get the new box up and 
running.

Best,
KAM
 

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Fri, 27 Jan 2017 22:23:55 +0100
Benny Pedersen  wrote:

> with use of PTR its always up2date, problem is just that none spf 
> testers are doing FcRDNS checked before saying spf pass

Unlikely.  The SPF spec says that you must do that, and most SPF libraries
probably do the checks.  I use Perl's Net::SPF module and it checks FcRDNS.

> PTR on its own is by defination same as +all

Not true; see above and the spec: http://www.openspf.org/SPF_Record_Syntax#ptr

-- Dianne.

Re: Legit Yahoo mail servers list

[RW]

On Fri, 27 Jan 2017 22:23:55 +0100
Benny Pedersen wrote:

> Dianne Skoll skrev den 2017-01-27 19:02:
> > On Fri, 27 Jan 2017 12:40:16 -0500
> > Rob McEwen  wrote:
> >   
> >> While I have Yahoo sending IPs extensively covered in my whitelist,
> >> I've been trying to get their complete official list of sending IPs
> >> for years.  
> > 
> > Yahoo might want the flexibility to change this list on a regular
> > basis.  
> 
> with use of PTR its always up2date, problem is just that none spf 
> testers are doing FcRDNS checked before saying spf pass

The RFC requires that they do.

Re: Legit Yahoo mail servers list

[Benny Pedersen]

Dianne Skoll skrev den 2017-01-27 19:02:

On Fri, 27 Jan 2017 12:40:16 -0500
Rob McEwen  wrote:


While I have Yahoo sending IPs extensively covered in my whitelist,
I've been trying to get their complete official list of sending IPs
for years.


Yahoo might want the flexibility to change this list on a regular
basis.


with use of PTR its always up2date, problem is just that none spf 
testers are doing FcRDNS checked before saying spf pass


PTR on its own is by defination same as +all

Re: Legit Yahoo mail servers list

[Matus UHLAR - fantomas]

the SPF record can change too, so that makes no difference.


On 27.01.17 16:57, David Jones wrote:

We have to assume that a competent mail sysadmin would
make that SPF record change.  It has to be trusted since that's
the whole point of SPF.


The easy workaround is to put ptr: into the SPF record, which is clearly
what yahoo did.  Then it's enough to maintain servers' fcrdns - no
incompetence here.

however, in both cases, some IPs can be added to, as well as removed from
pool. That means, one should do the comparison at time mail is received, not
far later (because the information might be obsolete at later time).

Still no practical difference between using IP ranges or rdns in SPF.


I get it as you need parse mail logs to find out what to put into
postscreen list, since postscreen doesn't use rdns...


Hmm, are you sure about that?


I have checked (just for sure) before sending my email.

what exactly did you mean when talking about log parsing, if not this?

Well - if postscreen was able to use rdns, this discussion would be useless,
since you'd whitelist .yahoo.com in postscreen, wouldn't you?


and postwhite (https://github.com/stevejenkins/postwhite) script can only
parse SPF records, not logs. Luckily ita page shows something that can help
you with yahoo:
https://help.yahoo.com/kb/SLN23997.html


Cool.  Thank you.  This is what I was looking for.

I think I have this solved in Postfix based on FCrDNS but
it good to know that Steve Jenkins is working on the same
thing.


postfix' smtpd can do rdns (and whitelist based on it). postscreen can't.
you mentioned postscren (and postwhite, which is whitelisting for
postscreen), I don't get why you mix postfix here...
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watson.  -- Daffy Duck & Porky Pig

Re: Legit Yahoo mail servers list

[Dianne Skoll]

On Fri, 27 Jan 2017 12:40:16 -0500
Rob McEwen  wrote:

> While I have Yahoo sending IPs extensively covered in my whitelist,
> I've been trying to get their complete official list of sending IPs
> for years.

Yahoo might want the flexibility to change this list on a regular
basis.

Regards,

Dianne.

Re: Legit Yahoo mail servers list

[Rob McEwen]

While I have Yahoo sending IPs extensively covered in my whitelist, I've 
been trying to get their complete official list of sending IPs for years.


I'm amazed that Yahoo doesn't participate in these conversations - or 
that nobody ever says, "I'll ask my colleague over at Yahoo"


seems very odd...

--
Rob McEwen

Re: Legit Yahoo mail servers list

[David Jones]

>the SPF record can change too, so that makes no difference.

We have to assume that a competent mail sysadmin would 
make that SPF record change.  It has to be trusted since that's
the whole point of SPF.

>MailScanner can still (and its SA plugin will) use the results described
>above.

I know that but I have to get the message past Postfix/postscreen
so it can make it to SA.

>I get it as you need parse mail logs to find out what to put into
>postscreen list, since postscreen doesn't use rdns...

Hmm, are you sure about that?

>and postwhite (https://github.com/stevejenkins/postwhite) script can only
>parse SPF records, not logs. Luckily ita page shows something that can help
>you with yahoo:
>https://help.yahoo.com/kb/SLN23997.html

Cool.  Thank you.  This is what I was looking for.

I think I have this solved in Postfix based on FCrDNS but
it good to know that Steve Jenkins is working on the same
thing.

Re: Legit Yahoo mail servers list

[Matus UHLAR - fantomas]

On 26.01.17 19:53, David Jones wrote:

Their SPF record can really only be evaluated by the MTA during
the SMTP conversation.



From: Matus UHLAR - fantomas 
SPF records can be perfectly parser by SA or other software at
different time.


  On 27.01.17 12:43, David Jones wrote:

I think you misunderstood.  PTR records don't change often but
they could.  Their matching A records for FCrDNS could change
too so you can't rely on later processing to know what happened
when that message arrived.


the SPF record can change too, so that makes no difference.

The best we can do here is to put sending host's fcrdns into headers,
probably together with Received-SPF: header, so spam filter will process
there.

Luckily most MTAs do the first, unless you turn off DNS check at SMTP time.


The main problem with parsing mail logs is the chicken-and-the-egg
issue where you may block a Yahoo mail server with an RBL for a
short period until you process the logs.



what informations do you search in logs that are not in mail headers?


I use MailScanner which is not a milter or otherwise directly part of the
MTA (Postfix in my setup).  This basically creates 2 levels of filtering:
the MTA and MailScanner (SpamAssassin plus many other checks).
My RBLs are done by postscreen (really awesome, everyone should
use it) so I have to allow Yahoo mail servers in the first level of filtering
independent of SA.


MailScanner can still (and its SA plugin will) use the results described
above.

I get it as you need parse mail logs to find out what to put into
postscreen list, since postscreen doesn't use rdns...

and postwhite (https://github.com/stevejenkins/postwhite) script can only
parse SPF records, not logs. Luckily ita page shows something that can help
you with yahoo:
https://help.yahoo.com/kb/SLN23997.html


I think I have solved this issue.  Postfix smtpd_client_restrictions
check_client_access does use FCrDNS for domains listed. I will
watch my logs for a few days and make sure this is working properly.


unluckily this is not something for postscreen...
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I don't have lysdexia. The Dog wouldn't allow that.

Re: Legit Yahoo mail servers list

[David Jones]

>From: Matus UHLAR - fantomas 
>Sent: Thursday, January 26, 2017 2:15 PM
    
>On 26.01.17 19:53, David Jones wrote:
>>I  understand what their SPF record means and how it works
>>but what they are publishing in their SPF record is not common.
>>Normally this would expand out to a list of IPs and CIDRs or DNS
>>records that can be turned into IPs that postwhite can use to build
>>a list for bypassing RBL checks.

>SPF was never designed to create such lists. They can get easily obsolete,
>miss some IPs and/or have some IPS that don't belong there.

I agree. But it turns out it works pretty well since SPF has been taken
more seriously the past couple of years.  When Gmail and others
started putting SPF failed messages into the Junk folder, it's starting
to be worth something.

I am only doing postwhite exclusions for 2 types of senders:
1. Large mail hosting providers that are too large to block and don't
keep their mail server IPs off of RBLs
2. Highly trusted senders that know what they are doing and keep
their SPF record properly maintained that would already score very
low in SA.

>>Their SPF record can really only be evaluated by the MTA during
>>the SMTP conversation. 

>SPF records can be perfectly parser by SA or other software at
>different time.

I think you misunderstood.  PTR records don't change often but
they could.  Their matching A records for FCrDNS could change
too so you can't rely on later processing to know what happened
when that message arrived.

>>The main problem with parsing mail logs is the chicken-and-the-egg
>>issue where you may block a Yahoo mail server with an RBL for a
>>short period until you process the logs.

>what informations do you search in logs that are not in mail headers?

I use MailScanner which is not a milter or otherwise directly part of the
MTA (Postfix in my setup).  This basically creates 2 levels of filtering:
the MTA and MailScanner (SpamAssassin plus many other checks).
My RBLs are done by postscreen (really awesome, everyone should
use it) so I have to allow Yahoo mail servers in the first level of filtering
independent of SA.

>>I think they publish their SPF like this because they have no good
>>list of outbound mail servers themselves so they take the lazy
>>approach.

>I believe that ptr method is one of best methods to implement in spf,
>contrary what the authors say. (I believe) Most of MTAs verify fcrdns of 
>connecting
>server so all required information are available in DNS cache at the time of
>SPF processing.

I think I have solved this issue.  Postfix smtpd_client_restrictions
check_client_access does use FCrDNS for domains listed. I will
watch my logs for a few days and make sure this is working properly.