Re: the whole greylisting, spam filtering thing

[Marc Peters]

On Mon, Oct 02, 2017 at 07:16:43AM +, rosjat wrote:
> Hi there again,
> 
> so I will try to ask the question about implementing rspam on a dedicated
> machine oder at the mailsystem again because I don't know if it was lost in
> the converstion :).

How is you setup now? Do you do any analysis or antivirus checking at
all?

I would start to put it on the same machine, as it is designed to use
less resources. If the machine bogs down, take another one and implement
it there. Nobody knows your mail volume or your machines or your
actual setup you're running now except from you ;).

hth,
Marc

> 
> Is there some effort in NOT run rspamd on the same machine as the
> mailsystem? I was just wondering because it could make some transitioning a
> little easier but if the amount of "workarounds" to relays mails through
> another instance is not worth it then I will go with spamfilterting on the
> mailsystem.
> 
> regards
> 
> -- 
> Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de
> 
> G+H Webservice GbR Gorzolla, Herrmann
> Königsbrücker Str. 70, 01099 Dresden
> 
> http://www.ghweb.de
> fon: +49 351 8107220   fax: +49 351 8107227
> 
> Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you
> print it, think about your responsibility and commitment to the ENVIRONMENT
> 

Re: the whole greylisting, spam filtering thing

[Boudewijn Dijkstra]

Op Sun, 01 Oct 2017 22:11:27 +0200 schreef Rupert Gallagher  
:
Spammers keep trying, from the same IPs, for days here, so graylisting  
is useless for us.


All of them?  On my end about 90% only try once.



--
Gemaakt met Opera's e-mailprogramma: http://www.opera.com/mail/

Re: the whole greylisting, spam filtering thing

[rosjat]

Hi there again,

so I will try to ask the question about implementing rspam on a 
dedicated machine oder at the mailsystem again because I don't know if 
it was lost in the converstion :).


Is there some effort in NOT run rspamd on the same machine as the 
mailsystem? I was just wondering because it could make some 
transitioning a little easier but if the amount of "workarounds" to 
relays mails through another instance is not worth it then I will go 
with spamfilterting on the mailsystem.


regards

--
Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de

G+H Webservice GbR Gorzolla, Herrmann
Königsbrücker Str. 70, 01099 Dresden

http://www.ghweb.de
fon: +49 351 8107220   fax: +49 351 8107227

Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before 
you print it, think about your responsibility and commitment to the 
ENVIRONMENT

Re: the whole greylisting, spam filtering thing

[Rupert Gallagher]

Spammers keep trying, from the same IPs, for days here, so graylisting is 
useless for us.

On SA and other things that require training, this is a nice story for you. A 
client received an average of 60 spam items per day on his own inbox alone. He 
trusted Kasperski, was confident on the accountant's ability to filter 
manually, and was happy to pay for everybody's waste of time in filtering their 
own daily dose of junk. This is a company with a cash-flow of 12M per year. One 
day, the accountant printed a fake bill: both archives and backups were 
encrypted by a ransomware. The boss paid for it, against the advice of the 
police, because funding international organised crime is a penal offence here. 
He obtained the antivirus, but it did not work.

Sent from ProtonMail Mobile

On Fri, Sep 29, 2017 at 3:06 PM, Markus Rosjat  wrote:

> Hi there, my boss is getting on my nerves that greylisting is basically out 
> of date because of things like outlook.com and mails ending up delayed for 
> ever. So the next logical step would be to deploy a tool like rspamd or 
> spamassasin to examin mail content. These tools need to be trained and if you 
> have a small mailserver with less accounts this could take a while I imagine. 
> So my question is, is there some source that you could use to train these 
> kind of tools (like a database that you could connect to for training 
> conntent ) or is every one here, that uses these tools, lucky enough to have 
> a shit load of users that do the training for your systems? some informations 
> about this would be helpful regards -- Markus Rosjat fon: +49 351 8107223 
> mail: ros...@ghweb.de G+H Webservice GbR Gorzolla, Herrmann Königsbrücker 
> Str. 70, 01099 Dresden http://www.ghweb.de fon: +49 351 8107220 fax: +49 351 
> 8107227 Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! 
> Before you print it, think about your responsibility and commitment to the 
> ENVIRONMENT

[rspamd and smtpd] (was: the whole greylisting, spam filtering thing)

[Thuban]

By the way, does anyone has some instructions to use rspamd with the
default smtpd ?

Regards.

-- 
thuban

Re: the whole greylisting, spam filtering thing

[Craig Skinner]

Hi Markus/all,

On Fri, 29 Sep 2017 15:06:29 +0200 Markus Rosjat wrote:
> ... greylisting ... like outlook.com and mails ending up delayed
> for ever 

The 'ungrey-robins' tool automatically solves this problem for
round-robin sending servers (Google, Outlook, Amazon, Yahoo, BT, etc.)

Start with the README & see the logs directory for evidence:
http://web.Britvault.Co.UK/products/ungrey-robins/





Otherwise;- simply set spamd's greylisting expire time to 4 days, not 4
hours. I ran servers this way for years - the mail does come through...



Cheers,
-- 
Craig Skinner | http://twitter.com/Craig_Skinner | http://linkd.in/yGqkv7

Re: the whole greylisting, spam filtering thing

[Peter N. M. Hansteen]

On 09/30/17 04:39, Stuart Henderson wrote:
>> It won't surprise anyone here that I disagree with the assertion that
>> greylisting is in any way outdated. Come back with that assertion when
>> the SMTP RFC is amended to drop the retry requirement.
> 
> These senders do retry, but not always from the same source address.
> Are you aware of any requirement in RFC5321 about source addresses
> of retries? I didn't find any when I looked (or even a requirement that
> retries are done over the same IP protocol version).

We had hoped for a clarification of the relevant parts in that RFC
update, but unfortunately the RFC still does not require retrying from
the same IP address. Back when the original was written it may have been
the default assumption that retries would come from the same host, but
even then at least some site would have had more than one outgoing mail
exchanger in place.

> Greylisting still has its place, but with the way email operates today,
> exemptions are unavoidable if you have a requirement to communicate
> reliably with users of many email services. Especially with a strict
> per-host greylisting implementation, where you don't get any benefit
> from the common thing where senders often arrange to retry from within
> the same v4 /24.

Unfortunately some senders (IIRC outlook.com being one) don't
necessarily stay within the same /24, even. That's why we need the
nospamd trick. And this will become incrementally more fun (fsvo) as
more of the traffic moves to IPv6.

-- 
Peter N. M. Hansteen, member of the first RFC 1149 implementation team
http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
"Remember to set the evil bit on all malicious network traffic"
delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.

Re: the whole greylisting, spam filtering thing

[Pierre-Philipp Braun]

I start greylisting on the firewall and thats ok but should I implement 
a dedicated system for rspamd and relay the "ok-Mails" from there to the 
mailsystem or simply run rspamd on the mailsystem und plug it front of 
the mailserver like postfix?


aha so if you are using Postfix then there are plenty anti-spam features 
that truly reduces the amount of spam and almost wipes it all out 
**during the SMTP session**: `man 5 postconf` and search for those 
patterns (this is postfix 3.1).


# NETWORK restrictions (smtpd_client_restrictions)
check_policy_service unix:private/policy
reject_unknown_client_hostname
check_client_access hash:/etc/postfix/client_access
reject_rbl_client ...
reject_unauth_pipelining

unknown_client_reject_code   = 554
smtpd_data_restrictions = reject_unauth_pipelining

# HELO/EHLO restrictions
reject_invalid_helo_hostname
reject_non_fqdn_helo_hostname
reject_unknown_helo_hostname
regexp:/etc/postfix/helo.regexp

# MAIL FROM restrictions
check_sender_access hash:/etc/postfix/sender_access,
reject_non_fqdn_sender,
reject_unknown_sender_domain

# RCPT TO restrictions
reject_non_fqdn_recipient,
reject_unknown_recipient_domain

unknown_address_reject_code  = 554

if some spam comes through that, it is a pretty one (and even passed tru 
the SPF check).  This already gets rid of 98% of the spam for me. 
Adding rspamd or whatever milter on top of that would clearly get you to 
99%.  No greylisting is needed.


Eventually make sure STARTTLS is enabled so the MX talk through TLS, 
setup your SPF records for your domain and eventually setup DKIM.

Re: the whole greylisting, spam filtering thing

[Markus Rosjat]

Hi,

thank you all for the helpful input on that subject. I have one last 
thing to ask about it.


What would be a good approach to implementing rspamd?

I start greylisting on the firewall and thats ok but should I implement 
a dedicated system for rspamd and relay the "ok-Mails" from there to the 
mailsystem or simply run rspamd on the mailsystem und plug it front of 
the mailserver like postfix?


Regards

--
Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de

G+H Webservice GbR Gorzolla, Herrmann
Königsbrücker Str. 70, 01099 Dresden

http://www.ghweb.de
fon: +49 351 8107220   fax: +49 351 8107227

Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before 
you print it, think about your responsibility and commitment to the 
ENVIRONMENT

Re: the whole greylisting, spam filtering thing

[Stuart Henderson]

On 2017-09-29, Peter N. M. Hansteen  wrote:
> On 09/29/17 15:06, Markus Rosjat wrote:
>
>> my boss is getting on my nerves that greylisting is basically out of
>> date because of things like outlook.com and mails ending up delayed for
>> ever. So the next logical step would be to deploy a tool like rspamd or
>> spamassasin to examin mail content. These tools need to be trained and
>> if you have a small mailserver with less accounts this could take a
>> while I imagine.
>
> It won't surprise anyone here that I disagree with the assertion that
> greylisting is in any way outdated. Come back with that assertion when
> the SMTP RFC is amended to drop the retry requirement.

These senders do retry, but not always from the same source address.
Are you aware of any requirement in RFC5321 about source addresses
of retries? I didn't find any when I looked (or even a requirement that
retries are done over the same IP protocol version).

Greylisting still has its place, but with the way email operates today,
exemptions are unavoidable if you have a requirement to communicate
reliably with users of many email services. Especially with a strict
per-host greylisting implementation, where you don't get any benefit
from the common thing where senders often arrange to retry from within
the same v4 /24.

What you can do with rspamd is only greylist mail that looks spammy
but isn't scored highly enough to block outright. (Or you could think
of that as making an exemption for mail that doesn't look too spammy).
This works quite well in my experience. Unfortunately it's a lot more
complex to configure than spamd, though once you start adding
scripts and trying to work out who to whitelist, the spamd setup
doesn't seem quite so straightforward either.

Most of the spam that reaches my mailbox is forwarded by a (high
IP reputation) host that sits behind spamd. (I'm looking at you,
Chinese state-owned enterprise trying to order a batch of fox
fur from my @openbsd address! And others.) That's a lot trickier
to block on my side without false positives..

Re: the whole greylisting, spam filtering thing

[Stuart Henderson]

On 2017-09-29, Larry Hynes  wrote:
> Markus Rosjat  wrote:
>> my boss is getting on my nerves
>
> It may be mutual.
>
>> that greylisting is basically out of date because of things like
>> outlook.com and mails ending up delayed for ever. So the next logical
>> step would be to deploy a tool like rspamd or spamassasin to examin
>> mail content. These tools need to be trained and if you have a small
>> mailserver with less accounts this could take a while I imagine.
>
> Specifically in relation to rspamd: If you spend some time reading
> the documentation on the rspamd website you might find that:
>
> 1. the weight of rules which classify messages as 'ham' or 'spam'
> i.e. those rules which rely on the 'training' of messages, does not
> have to be, in the overall context, critical. rspamd deploys a
> boatload of 'tests', by default, and even more can be enabled, and
> each of those can be assigned a score. hamminess or spamminess is
> just one 'test'.

+1. rspamd doesn't do badly even with little/no training for spam/ham.

It does have problems with certain mail, for example it likes to have various
MIME headers, so you may need to make some exemptions for things like
daily/security mail output, or mail from people who don't use MIME MUAs.

> 2. That the rspamd website specifically links to 'pre-built' ham
> and spam databases which you are free to download and use.

Definitely you would need to read documentation if using tools like
rspamd or spamassassin.

Re: the whole greylisting, spam filtering thing

[Peter N. M. Hansteen]

On 09/29/17 15:06, Markus Rosjat wrote:

> my boss is getting on my nerves that greylisting is basically out of
> date because of things like outlook.com and mails ending up delayed for
> ever. So the next logical step would be to deploy a tool like rspamd or
> spamassasin to examin mail content. These tools need to be trained and
> if you have a small mailserver with less accounts this could take a
> while I imagine.

It won't surprise anyone here that I disagree with the assertion that
greylisting is in any way outdated. Come back with that assertion when
the SMTP RFC is amended to drop the retry requirement.

But there are actors in the email market that do not particularly care
about standards compliance one way or the other, unfortunately (at least
for those of us below critical mass in terms of volume) is to use the
nospamd feature and not exposing those sending domains to greylisting at
all. My sedimentary nospamd file, built on discovering SPF info for
badly behaved domains, is available here
https://home.nuug.no/~peter/nospamd - I only started commenting entries
after a while, but it's a Works for me(tM) file. See man spamd for
examples of how to include that in your config. If you want to build and
maintain your own nospamd based on SPF records, Aaron Poffenberger's
spf_fetch is very well worth looking into (see
https://github.com/akpoff/spf_fetch)

> So my question is, is there some source that you could use to  train
> these kind of tools (like a database that you could connect to for
> training conntent ) or is every one here, that uses these tools, lucky
> enough to have a shit load of users that do the training for your systems?

Yes, you need content filtering too. As others have said, you won't be
able to totally avoid the training effort based on local preferences,
but with working greylisting in front of the content filtering, those
servers will run a lot cooler than without.

I suppose my long rant from a few years back is still relevant -
https://bsdly.blogspot.no/2014/02/effective-spam-and-malware.html, for
the fun parts of doing greytrapping see
https://bsdly.blogspot.no/2013/05/keep-smiling-waste-spammers-time.html
and
https://bsdly.blogspot.no/2013/04/maintaining-publicly-available.html
and of course
https://bsdly.blogspot.no/2012/05/in-name-of-sane-email-setting-up-spamd.html
might still be of some use.

- P

-- 
Peter N. M. Hansteen, member of the first RFC 1149 implementation team
http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
"Remember to set the evil bit on all malicious network traffic"
delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.

Re: the whole greylisting, spam filtering thing

[Markus Rosjat]

Hi Leo,



Am 29.09.2017 um 16:57 schrieb Leo Unglaub:

Hey,

On 09/29/17 15:06, Markus Rosjat wrote:
my boss is getting on my nerves that greylisting is basically out of 
date because of things like outlook.com and mails ending up delayed 
for ever. So the next logical step would be to deploy a tool like 
rspamd or spamassasin to examin mail content. These tools need to be 
trained and if you have a small mailserver with less accounts this 
could take a while I imagine


i assume that your boss is not an engineer and also not very familiar 
with how emails work. Greylisting it clearly NOT out of date at all. 
Greylisting simply makes use of stuff that is defined in the SMTP RFC. 
Every email server is allowed to temporary deny the delivery of an email 
and ask the sending server for another try.




well we use greylisting and I gave MS a free pass but sometimes it 
doesn't seem to work anyway but that's ok for me.


The problem in this case is clearly Microsoft who has no idea how email 
is supposed to work. You have two options here.




the customer will always complain no matter how often you explain the 
real problem :)


A: Simply don't care about Microsoft and just send customers to a 
website where you describe the problem and tell them to contact 
Microsoft in order to fix there stuff. This works very well, my Company 
hosts around 2,3 Million mailboxes and we use Greylisting and customers 
are okay with it.


B: You exclude the outlook.com outgoing servers from greylisting. 
Microsoft provides a list of IP addresses that they use for delivery:

https://mail.live.com/mail/ipspace.aspx


65.54.190.0/26
65.54.190.64/26
65.54.190.128/26
65.54.190.192/26
65.55.116.0/26
65.55.111.64/26
65.55.116.64/26
65.55.111.128/26
65.55.34.0/26
65.55.34.64/26
65.55.34.128/26
65.55.34.192/26
65.55.90.0/26
65.55.90.64/26
65.55.90.128/26
65.55.90.192/26
65.54.51.64/26
65.54.61.64/26
207.46.66.0/28
157.55.0.192/26
157.55.1.128/26
157.55.2.0/26
157.55.2.64/26


Greetings
Leo



I also check the spf record files of MS and added them too so we will 
see what's going to happen. I need to move to a more up to date setup so 
I just check my options what's used these days and yes greylisting works 
for me as long as no office 365 is involved but a lot of business 
partners of our customers moving to 365 and the email solution so it 
becomes a problem for me too. It's just fustrating to see a mail 
greylisted from 40 different ips ...


regards

--
Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de

G+H Webservice GbR Gorzolla, Herrmann
Königsbrücker Str. 70, 01099 Dresden

http://www.ghweb.de
fon: +49 351 8107220   fax: +49 351 8107227

Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before 
you print it, think about your responsibility and commitment to the 
ENVIRONMENT

Re: the whole greylisting, spam filtering thing

[Leo Unglaub]

Hey,

On 09/29/17 15:06, Markus Rosjat wrote:
my boss is getting on my nerves that greylisting is basically out of 
date because of things like outlook.com and mails ending up delayed for 
ever. So the next logical step would be to deploy a tool like rspamd or 
spamassasin to examin mail content. These tools need to be trained and 
if you have a small mailserver with less accounts this could take a 
while I imagine


i assume that your boss is not an engineer and also not very familiar 
with how emails work. Greylisting it clearly NOT out of date at all. 
Greylisting simply makes use of stuff that is defined in the SMTP RFC. 
Every email server is allowed to temporary deny the delivery of an email 
and ask the sending server for another try.


The problem in this case is clearly Microsoft who has no idea how email 
is supposed to work. You have two options here.


A: Simply don't care about Microsoft and just send customers to a 
website where you describe the problem and tell them to contact 
Microsoft in order to fix there stuff. This works very well, my Company 
hosts around 2,3 Million mailboxes and we use Greylisting and customers 
are okay with it.


B: You exclude the outlook.com outgoing servers from greylisting. 
Microsoft provides a list of IP addresses that they use for delivery:

https://mail.live.com/mail/ipspace.aspx


65.54.190.0/26
65.54.190.64/26
65.54.190.128/26
65.54.190.192/26
65.55.116.0/26
65.55.111.64/26
65.55.116.64/26
65.55.111.128/26
65.55.34.0/26
65.55.34.64/26
65.55.34.128/26
65.55.34.192/26
65.55.90.0/26
65.55.90.64/26
65.55.90.128/26
65.55.90.192/26
65.54.51.64/26
65.54.61.64/26
207.46.66.0/28
157.55.0.192/26
157.55.1.128/26
157.55.2.0/26
157.55.2.64/26


Greetings
Leo

Re: the whole greylisting, spam filtering thing

[Markus Rosjat]

Hi,

Am 29.09.2017 um 15:39 schrieb Larry Hynes:

Markus Rosjat  wrote:

my boss is getting on my nerves


It may be mutual.



of course but well :)


that greylisting is basically out of date because of things like
outlook.com and mails ending up delayed for ever. So the next logical
step would be to deploy a tool like rspamd or spamassasin to examin
mail content. These tools need to be trained and if you have a small
mailserver with less accounts this could take a while I imagine.


Specifically in relation to rspamd: If you spend some time reading
the documentation on the rspamd website you might find that:

1. the weight of rules which classify messages as 'ham' or 'spam'
i.e. those rules which rely on the 'training' of messages, does not
have to be, in the overall context, critical. rspamd deploys a
boatload of 'tests', by default, and even more can be enabled, and
each of those can be assigned a score. hamminess or spamminess is
just one 'test'.

2. That the rspamd website specifically links to 'pre-built' ham
and spam databases which you are free to download and use.



I'll check this out !

Thank you for the hint !!!

regards

--
Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de

G+H Webservice GbR Gorzolla, Herrmann
Königsbrücker Str. 70, 01099 Dresden

http://www.ghweb.de
fon: +49 351 8107220   fax: +49 351 8107227

Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before 
you print it, think about your responsibility and commitment to the 
ENVIRONMENT

Re: the whole greylisting, spam filtering thing

[tech-lists]

On Fri, Sep 29, 2017 at 03:06:29PM +0200, Markus Rosjat wrote:


So my question is, is there some source that you could use to  train
these kind of tools (like a database that you could connect to for
training conntent ) or is every one here, that uses these tools, lucky
enough to have a shit load of users that do the training for your systems?

some informations about this would be helpful


As far as I understand it, the bayesnian lerning system learns stuff
pertinent to your domain, so you have to train it. Spam at one domain
might not be spam at another.

My system has only a few email addresses, but I keep all my spam. I
make sa-learn --spam read the spam, and sa-learn --ham the stuff I've
filtered and examined that they are all not-spam. It doesn't take
long.

https://wiki.apache.org/spamassassin/BayesInSpamAssassin
--
J.

Re: the whole greylisting, spam filtering thing

[Larry Hynes]

Markus Rosjat  wrote:
> my boss is getting on my nerves

It may be mutual.

> that greylisting is basically out of date because of things like
> outlook.com and mails ending up delayed for ever. So the next logical
> step would be to deploy a tool like rspamd or spamassasin to examin
> mail content. These tools need to be trained and if you have a small
> mailserver with less accounts this could take a while I imagine.

Specifically in relation to rspamd: If you spend some time reading
the documentation on the rspamd website you might find that:

1. the weight of rules which classify messages as 'ham' or 'spam'
i.e. those rules which rely on the 'training' of messages, does not
have to be, in the overall context, critical. rspamd deploys a
boatload of 'tests', by default, and even more can be enabled, and
each of those can be assigned a score. hamminess or spamminess is
just one 'test'.

2. That the rspamd website specifically links to 'pre-built' ham
and spam databases which you are free to download and use.

the whole greylisting, spam filtering thing

[Markus Rosjat]

Hi there,

my boss is getting on my nerves that greylisting is basically out of 
date because of things like outlook.com and mails ending up delayed for 
ever. So the next logical step would be to deploy a tool like rspamd or 
spamassasin to examin mail content. These tools need to be trained and 
if you have a small mailserver with less accounts this could take a 
while I imagine.


So my question is, is there some source that you could use to  train 
these kind of tools (like a database that you could connect to for 
training conntent ) or is every one here, that uses these tools, lucky 
enough to have a shit load of users that do the training for your systems?


some informations about this would be helpful

regards

--
Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de

G+H Webservice GbR Gorzolla, Herrmann
Königsbrücker Str. 70, 01099 Dresden

http://www.ghweb.de
fon: +49 351 8107220   fax: +49 351 8107227

Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before 
you print it, think about your responsibility and commitment to the 
ENVIRONMENT