Re: [OT] Re: Check for valid MX of sender and rspamd testing

[John Hardin]

On Tue, 10 Apr 2018, Rupert Gallagher wrote:


Microsoft shits on your head, and you pay for it. Go on, enjoy it.

On Tue, Apr 10, 2018 at 17:19, Reindl Harald  wrote:


Am 10.04.2018 um 17:17 schrieb Rupert Gallagher: > Microsoft should be blacklisted globally, until they fix 
their > own software.  go on - you won't realize that your damned job is to make sure you receive mails for your 
users while reject as much spam as possible every single false-positive does a lot of more harm than some slipped 
spam mails > On Tue, Apr 10, 2018 at 16:00, Sebastian Arcus > wrote: >> Hence why I have to have a 
local whitelist and skip verification for >> all MX's of the form *.outlook.com (which include Microsoft 
cloud >> hosted domains) @open-t.co.uk> @open-t.co.uk>


Sigmonster agree...

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Your mouse has moved. Your Windows Operating System must be
  relicensed due to this hardware change. Please contact Microsoft
  to obtain a new activation key. If this hardware change results in
  added functionality you may be subject to additional license fees.
  Your system will now shut down. Thank you for choosing Microsoft.
---
 3 days until Thomas Jefferson's 275th Birthday

Re: [OT] Re: Check for valid MX of sender and rspamd testing

[Rupert Gallagher]

Microsoft shits on your head, and you pay for it. Go on, enjoy it.

Sent from ProtonMail Mobile

On Tue, Apr 10, 2018 at 17:19, Reindl Harald  wrote:

> Am 10.04.2018 um 17:17 schrieb Rupert Gallagher: > Microsoft should be 
> blacklisted globally, until they fix their > own software.  go on - you won't 
> realize that your damned job is to make sure you receive mails for your users 
> while reject as much spam as possible every single false-positive does a lot 
> of more harm than some slipped spam mails > On Tue, Apr 10, 2018 at 16:00, 
> Sebastian Arcus > wrote: >> Hence why I have to have a local whitelist and 
> skip verification for >> all MX's of the form *.outlook.com (which include 
> Microsoft cloud >> hosted domains) @open-t.co.uk> @open-t.co.uk>

Re: [OT] Re: Check for valid MX of sender and rspamd testing

[Pedro David Marco]

 
 >Microsoft should be blacklisted globally, until they fix their own software.  
   
 They even change the order of many headers (Receiveds included) remove the 
ones they do not like, etc... i am sure they like playing dices...
PedroD  

Re: [OT] Re: Check for valid MX of sender and rspamd testing

[Rupert Gallagher]

Microsoft should be blacklisted globally, until they fix their own software.

On Tue, Apr 10, 2018 at 16:00, Sebastian Arcus  wrote:

> Hence why I have to have a local whitelist and skip verification for all MX's 
> of the form *.outlook.com (which include Microsoft cloud hosted domains).

[OT] Re: Check for valid MX of sender and rspamd testing

[Sebastian Arcus]

On 10/04/18 08:41, Daniele Duca wrote:

On 09/04/2018 20:40, Sebastian Arcus wrote:



This might not really answer your question, but I've had really good 
results leaving all this to the MTA (Exim in my case). I actually go 
for the whole hog full callout verification - checking with the MX 
that the sender really exists. I know that some people are against 
this and say that you get blacklisted - but I've been doing this for 
about 8 months on 4 sites and it has worked very well. I have a local 
full callout verification whitelist - to skip callout verification 
mainly for Microsoft operated domains - which will blacklist you at 
the drop of the hat.

Hello Sebastian,

I'm curious about this approach. I never tried it, but, assuming that 
you check the MX of the envelope from domain, how do you deal with 
poorly-configured-but-legit VPS that use, in example, 
www-d...@hostname.of.the.server ? I have live examples of wordpress and 
vbulletin installations that have not existent envelope from mailboxes 
or VPS hostnames without MX records. There are also other services that 
actively send email in the form of "nore...@domain.com". If I understood 
correctly, your approach would heavily penalize these senders.


I know that in the ideal world everyone should configure their systems 
neatly, but unfortunately we are far from ideal conditions in real life :/


I'm happy to discuss this technique but I can't really afforhttps://www.exim.org/exim-html-current/doc/html/spec_html/ch-access_control_lists.htmld the 
administrative overhead I would have with users complaining about 
rejected emails..


Hi Daniele. I agree that configuring a real life system is often a 
balancing act between having a standards compliant and efficient system 
on one side - but at the same time compromising so that the users are 
not too inconvenienced. I started with a configuration which was as 
strict as I preferred, and then gradually loosened things up.


I also think that there is some scope to penalizing badly configured 
systems - if time and circumstances allow. Accepting crap often means 
condoning it - and encouraging systems administrators in sloppy 
practices. Of course, if you can find the time to do this - and not end 
up inconveniencing your own users too much :-)


Generally if emails come from poorly configured servers and they are 
relatively small providers or organisations, I try and liaise with them 
and get them to implement better settings. Fortunately I can do this as 
most of the setups at my end are relatively small - but in larger ones 
that is probably not possible.


For larger providers and domains at the sending end, sometimes I have to 
implement local workarounds and whitelists - as there isn't usually much 
chance to get any cooperation from them.


I believe (but I could be wrong) that the envelope from address should 
be able to receive bounce messages - so I don't think an address of the 
type www-data@server_hostname is acceptable.


Also, I found that most noreply@ type of addresses from clued-up 
providers seem to react correctly to callout verifications and confirm 
the address is real and valid (although they might return a bounceback 
message if you actually try to email them). I think this should be the 
correct way to configure noreply@ addresses. The exception to this is 
pretty much all Microsoft controlled domains and systems - which seem to 
be rubbish at both following standards and also configuring a decent 
email setup. Hence why I have to have a local whitelist and skip 
verification for all MX's of the form *.outlook.com (which include 
Microsoft cloud hosted domains).

Re: Check for valid MX of sender and rspamd testing

[Daniele Duca]

On 09/04/2018 20:40, Sebastian Arcus wrote:



This might not really answer your question, but I've had really good 
results leaving all this to the MTA (Exim in my case). I actually go 
for the whole hog full callout verification - checking with the MX 
that the sender really exists. I know that some people are against 
this and say that you get blacklisted - but I've been doing this for 
about 8 months on 4 sites and it has worked very well. I have a local 
full callout verification whitelist - to skip callout verification 
mainly for Microsoft operated domains - which will blacklist you at 
the drop of the hat.

Hello Sebastian,

I'm curious about this approach. I never tried it, but, assuming that 
you check the MX of the envelope from domain, how do you deal with 
poorly-configured-but-legit VPS that use, in example, 
www-d...@hostname.of.the.server ? I have live examples of wordpress and 
vbulletin installations that have not existent envelope from mailboxes 
or VPS hostnames without MX records. There are also other services that 
actively send email in the form of "nore...@domain.com". If I understood 
correctly, your approach would heavily penalize these senders.


I know that in the ideal world everyone should configure their systems 
neatly, but unfortunately we are far from ideal conditions in real life :/


I'm happy to discuss this technique but I can't really afford the 
administrative overhead I would have with users complaining about 
rejected emails..


Daniele Duca

Re: Check for valid MX of sender and rspamd testing

[Sebastian Arcus]

On 09/04/18 15:24, David Jones wrote:
I was wondering if anyone knows of an SA plugin or another method to 
determine if the envelope-from domain has a valid MX record that is 
listening on TCP port 25.  I don't think it would be a major scorer but 
it could be useful in meta rules.


This might not really answer your question, but I've had really good 
results leaving all this to the MTA (Exim in my case). I actually go for 
the whole hog full callout verification - checking with the MX that the 
sender really exists. I know that some people are against this and say 
that you get blacklisted - but I've been doing this for about 8 months 
on 4 sites and it has worked very well. I have a local full callout 
verification whitelist - to skip callout verification mainly for 
Microsoft operated domains - which will blacklist you at the drop of the 
hat. Pretty much everybody else on the internet seems to understand the 
full callout verification has more advantages than disadvantages in 
fighting spam. I also use Exim to keep count of how many callout 
verifications have failed for an origin IP address and then start 
rejecting connections after 10/24 hours - to stop spammers from using my 
boxes as dictionary attacks proxies against other domains (and getting 
me blacklisted in the process).


All of this seems to have worked out very well so far - but I realise 
that it will depend on the size of the email system and number of 
mailboxes and all sorts of other things - so it might not work so well 
elsewhere.

Re: Check for valid MX of sender and rspamd testing

[RW]

On Mon, 9 Apr 2018 09:24:23 -0500
David Jones wrote:

> I was wondering if anyone knows of an SA plugin or another method to 
> determine if the envelope-from domain has a valid MX record that is 
> listening on TCP port 25.  I don't think it would be a major scorer
> but it could be useful in meta rules.

There's NO_DNS_FOR_FROM which tests for MX or A.  I don't know if it's
still true, but historically this has, in my experience, been largely a
surrogate test for made-up domains.

Having a single test for DNS and port 25 doesn't sound like a good
idea since you can't determine how much benefit come from each of the
two parts or score them separately.   

Re: Check for valid MX of sender and rspamd testing

[Benny Pedersen]

Kevin A. McGrail skrev den 2018-04-09 16:46:


If you are interested, let me know.


i am interested to learn how to setup mimedefang, not how to test mx :=)

that will always be a job for mta to make sure this is valid

Re: Check for valid MX of sender and rspamd testing

[Benny Pedersen]

David Jones skrev den 2018-04-09 16:24:

I was wondering if anyone knows of an SA plugin or another method to
determine if the envelope-from domain has a valid MX record that is
listening on TCP port 25.  I don't think it would be a major scorer
but it could be useful in meta rules.


thats a job of mta, not content filters

so if postfix reject non existing senders, then it remain received in sa 
valid mx/a/



Been playing around with rspamd over the weekend to see how it
compares and so far not that impressed.  It has a few features that
are interesting like the MX check but other than that it's not as
impressive as the author makes it out to be on he website comparing
it to SA


if it's still just check that a mx exists, without A/ then its buggy 
and should not be used, i do not need a mx, but there is fools around 
that say homepage as well must start with www


i give up with this fools not understanding it


It claims to have better Bayes but so far I am seeing identical
results after identical training.


marketing is better :=)


The Universal Configuration Language is terrible and hard to wrap your
head around it when the structure is so loose.  Since it's not well
defined nor well documented it takes a lot of trial and error to
figure it out.


xml files is very hard to manage so the ucl was created to make it even 
more hard to make it right :=)



It doesn't seem to be as flexible as SA in many regards.


yep, thats why i only tested rspamd live as a second spam filter, not 
take off spamassing while tested it, so i could see errors fast in both 
content filters, and later use the best of both, this stopped me as a 
rspamd ebuild maintainer aswell on gentoo, i was the first one adding 
rspamd / rmilter to gentoo, i still love what i did, but the kids have 
to learn why i use spamassassin now :=)



Right now I have rspamd only adding headers so I can compare with SA.
Tuning it out to match SA's accuracy is proving to be very challenging
and time consuming.


yep one more faktor why i stopped using it

Re: Check for valid MX of sender and rspamd testing

[Daniele Duca]

On 09/04/2018 16:24, David Jones wrote:



Been playing around with rspamd over the weekend to see how it 
compares and so far not that impressed.  It has a few features that 
are interesting like the MX check but other than that it's not as 
impressive as the author makes it out to be on the website comparing 
it to SA.


It claims to have better Bayes but so far I am seeing identical 
results after identical training.
It's a few months that I'm using rspamd. I wrote a dedicated plugin for 
amavisd-new and I use it's scoring together with SA's.


IMHO to reach satisfying results you have to train it a lot more that 
SA, but in the long run it's a nice addition. My empirical observations 
suggests that it gets better after at least 3000 ham and spam email 
learned. It's also cool that you can train both global and per-domain 
bayes, very useful if you have a multitenant installation with a lot of 
different domains.


Daniele

Re: Check for valid MX of sender and rspamd testing

[Kevin A. McGrail]

Well, here's the code I use in filter_sender in MD to check for a validMX.
The module needs a public release with some updates and doesn't work great
with IPv6 but the code is solid and been in use for a long time at my firm.

 #IF NOT A BOUNCE, THEN CHECK VALID MX RECORDS
  if ($sender ne '<>') {
#CHECK IF SENDER HAS VALID MX RECORDS
($rv, $reason) = _valid_mx($sender);

#IF WE GOT A RETURN VALUE OF 1 CHECK WHAT IT IS
if ($rv) {
  #RESOLUTION ISSUE? LOG ERROR AND CONTINUE AS A SAFETY VALVE
  if ($reason =~ /Resolution Problem/i) {
md_syslog('error', "ERROR: check valid MX Resolution Problem:
$sender - $reason.");
  } else {
#OTHERWISE PASSED CHECK VALID MX
md_syslog('info', "DEBUG: Passed check valid MX: $sender");
  }
} else {
  #FAILED CHECK VALID MX
  md_syslog('warning', "DEBUG: Rejecting $sender - Invalid MX:
$reason.");
  return ('REJECT', "$QueueID: Sorry, mail not accepted. $sender has an
invalid MX record: $reason.");
}
  }

For the check against port 25, Diane's caveat aside, look
at md_check_against_smtp_server which you can run in filter_recipient.  I
can share how we use a Redis backend to store the data and our routines.

The validmx check hits about 90% of the issues and the cached check really
helps us shutdown DDoS and dictionary attacks.

Regards,
KAM

--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

On Mon, Apr 9, 2018 at 10:58 AM, Dianne Skoll 
wrote:

> On Mon, 9 Apr 2018 09:56:20 -0500
> David Jones  wrote:
>
> > On 04/09/2018 09:44 AM, Reindl Harald wrote:
> > > you simply don't want connect to every innocent MX which inbound
> > > mail is forged because for the sake of god you are attacking the
> > > victim of spoofed mails and you are easily part of a distributed
> > > DOS when your few connections back are only a small part
>
> Also, if an innocent domain's MX server just happens to be down
> when you check, you could get a FP.
>
> Checking for the existence of a sane MX record is good practice.
> I'm not so sure about actually trying to connect to said MX, even if
> you take basic precautions to minimize connections.
>
> Regards,
>
> Dianne.
>

Re: Check for valid MX of sender and rspamd testing

[David Jones]

On 04/09/2018 09:58 AM, Dianne Skoll wrote:

On Mon, 9 Apr 2018 09:56:20 -0500
David Jones  wrote:


On 04/09/2018 09:44 AM, Reindl Harald wrote:

you simply don't want connect to every innocent MX which inbound
mail is forged because for the sake of god you are attacking the
victim of spoofed mails and you are easily part of a distributed
DOS when your few connections back are only a small part


Also, if an innocent domain's MX server just happens to be down
when you check, you could get a FP.

Checking for the existence of a sane MX record is good practice.
I'm not so sure about actually trying to connect to said MX, even if
you take basic precautions to minimize connections.

Regards,

Dianne.



https://rspamd.com/doc/modules/mx_check.html

I guess I could check the X-Spamd-Result header in SA from rspamd for 
/MX_GOOD/ and let rspamd do the heavy lifting.


X-Spamd-Result: default: False [1.18 / 999.00]
  TO_DN_NONE(0.00)[]
  NEURAL_HAM(-0.00)[-0.792,0]
  DKIM_TRACE(0.00)[email.symantec.com:+]
  ASN(0.00)[asn:7160, ipnet:142.0.160.0/21, country:US]
  RCVD_NO_TLS_LAST(0.00)[]
  R_SPF_ALLOW(-0.20)[+ip4:142.0.160.0/20]
  DMARC_POLICY_ALLOW(-0.25)[email.symantec.com,none]
  MID_RHS_NOT_FQDN(0.50)[]

FROM_NEQ_ENVFROM(0.00)[co...@email.symantec.com,boun...@email.symantec.com]
  ARC_NA(0.00)[]
  RCVD_IN_DNSWL_NONE(0.00)[28.163.0.142.list.dnswl.org : 127.0.15.0]
  RCVD_COUNT_TWO(0.00)[2]
  MX_GOOD(-0.01)[cached: S912704989.m.en25.com]
  HTML_SHORT_LINK_IMG_2(1.00)[]
  MIME_GOOD(-0.10)[multipart/alternative,text/plain]
  FROM_HAS_DN(0.00)[]
  FORGED_SENDER(0.30)[]
  REPLYTO_DN_EQ_FROM_DN(0.00)[]
  HAS_REPLYTO(0.00)[symantec_communications-...@symantec.com]
  TO_MATCH_ENVRCPT_ALL(0.00)[]
  REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]
  RCPT_COUNT_ONE(0.00)[1]
  HAS_LIST_UNSUB(-0.01)[]
  IP_SCORE(0.05)[ipnet: 142.0.160.0/21(0.08), asn: 7160(0.13), 
country: US(0.02)]

  MIME_BASE64_TEXT(0.10)[]
  R_DKIM_ALLOW(-0.20)[email.symantec.com]

--
David Jones

Re: Check for valid MX of sender and rspamd testing

[Dianne Skoll]

On Mon, 9 Apr 2018 09:56:20 -0500
David Jones  wrote:

> On 04/09/2018 09:44 AM, Reindl Harald wrote:
> > you simply don't want connect to every innocent MX which inbound
> > mail is forged because for the sake of god you are attacking the
> > victim of spoofed mails and you are easily part of a distributed
> > DOS when your few connections back are only a small part

Also, if an innocent domain's MX server just happens to be down
when you check, you could get a FP.

Checking for the existence of a sane MX record is good practice.
I'm not so sure about actually trying to connect to said MX, even if
you take basic precautions to minimize connections.

Regards,

Dianne.

Re: Check for valid MX of sender and rspamd testing

[David Jones]

On 04/09/2018 09:46 AM, Kevin A. McGrail wrote:

Hi Dave,

I do similar work in MIMEDefang using the a redis backend for caching 
valid recipients combined with Net::validMX that can check to see if a 
sender has valid MX before sending.  I have a release of Net::validMX 
I'm about to post this week in fact.


If you are interested, let me know.

Regards,
KAM



I am interested in both learning MIMEDefang and your valid MX check.


--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

On Mon, Apr 9, 2018 at 10:24 AM, David Jones > wrote:


I was wondering if anyone knows of an SA plugin or another method to
determine if the envelope-from domain has a valid MX record that is
listening on TCP port 25.  I don't think it would be a major scorer
but it could be useful in meta rules.

Been playing around with rspamd over the weekend to see how it
compares and so far not that impressed.  It has a few features that
are interesting like the MX check but other than that it's not as
impressive as the author makes it out to be on the website comparing
it to SA.

It claims to have better Bayes but so far I am seeing identical
results after identical training.

The Universal Configuration Language is terrible and hard to wrap
your head around it when the structure is so loose.  Since it's not
well defined nor well documented it takes a lot of trial and error
to figure it out.

It doesn't seem to be as flexible as SA in many regards.

Right now I have rspamd only adding headers so I can compare with
SA. Tuning it out to match SA's accuracy is proving to be very
challenging and time consuming.

-- 
David Jones






--
David Jones

Re: Check for valid MX of sender and rspamd testing

[David Jones]

On 04/09/2018 09:44 AM, Reindl Harald wrote:



Am 09.04.2018 um 16:24 schrieb David Jones:

I was wondering if anyone knows of an SA plugin or another method to
determine if the envelope-from domain has a valid MX record that is
listening on TCP port 25.  I don't think it would be a major scorer but
it could be useful in meta rules.


you simply don't want connect to every innocent MX which inbound mail is
forged because for the sake of god you are attacking the victim of
spoofed mails and you are easily part of a distributed DOS when your few
connections back are only a small part

at least combine it with SPF_PASS and let alone domains without SPF



Rspamd is doing this and caching the information in Redis so it doesn't 
check every single email.  I am sure that it's only checking the valid 
MX once it has passed some basic checks to prevent "attacking the victim 
of spoofed emails."


--
David Jones

Re: Check for valid MX of sender and rspamd testing

[Kevin A. McGrail]

Hi Dave,

I do similar work in MIMEDefang using the a redis backend for caching valid
recipients combined with Net::validMX that can check to see if a sender has
valid MX before sending.  I have a release of Net::validMX I'm about to
post this week in fact.

If you are interested, let me know.

Regards,
KAM

--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

On Mon, Apr 9, 2018 at 10:24 AM, David Jones  wrote:

> I was wondering if anyone knows of an SA plugin or another method to
> determine if the envelope-from domain has a valid MX record that is
> listening on TCP port 25.  I don't think it would be a major scorer but it
> could be useful in meta rules.
>
> Been playing around with rspamd over the weekend to see how it compares
> and so far not that impressed.  It has a few features that are interesting
> like the MX check but other than that it's not as impressive as the author
> makes it out to be on the website comparing it to SA.
>
> It claims to have better Bayes but so far I am seeing identical results
> after identical training.
>
> The Universal Configuration Language is terrible and hard to wrap your
> head around it when the structure is so loose.  Since it's not well defined
> nor well documented it takes a lot of trial and error to figure it out.
>
> It doesn't seem to be as flexible as SA in many regards.
>
> Right now I have rspamd only adding headers so I can compare with SA.
> Tuning it out to match SA's accuracy is proving to be very challenging and
> time consuming.
>
> --
> David Jones
>