Re: the whole greylisting, spam filtering thing
On Mon, Oct 02, 2017 at 07:16:43AM +, rosjat wrote: > Hi there again, > > so I will try to ask the question about implementing rspam on a dedicated > machine oder at the mailsystem again because I don't know if it was lost in > the converstion :). How is you setup now? Do you do any analysis or antivirus checking at all? I would start to put it on the same machine, as it is designed to use less resources. If the machine bogs down, take another one and implement it there. Nobody knows your mail volume or your machines or your actual setup you're running now except from you ;). hth, Marc > > Is there some effort in NOT run rspamd on the same machine as the > mailsystem? I was just wondering because it could make some transitioning a > little easier but if the amount of "workarounds" to relays mails through > another instance is not worth it then I will go with spamfilterting on the > mailsystem. > > regards > > -- > Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de > > G+H Webservice GbR Gorzolla, Herrmann > Königsbrücker Str. 70, 01099 Dresden > > http://www.ghweb.de > fon: +49 351 8107220 fax: +49 351 8107227 > > Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you > print it, think about your responsibility and commitment to the ENVIRONMENT >
Re: the whole greylisting, spam filtering thing
Op Sun, 01 Oct 2017 22:11:27 +0200 schreef Rupert Gallagher : Spammers keep trying, from the same IPs, for days here, so graylisting is useless for us. All of them? On my end about 90% only try once. -- Gemaakt met Opera's e-mailprogramma: http://www.opera.com/mail/
Re: the whole greylisting, spam filtering thing
Hi there again, so I will try to ask the question about implementing rspam on a dedicated machine oder at the mailsystem again because I don't know if it was lost in the converstion :). Is there some effort in NOT run rspamd on the same machine as the mailsystem? I was just wondering because it could make some transitioning a little easier but if the amount of "workarounds" to relays mails through another instance is not worth it then I will go with spamfilterting on the mailsystem. regards -- Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de G+H Webservice GbR Gorzolla, Herrmann Königsbrücker Str. 70, 01099 Dresden http://www.ghweb.de fon: +49 351 8107220 fax: +49 351 8107227 Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you print it, think about your responsibility and commitment to the ENVIRONMENT
Re: the whole greylisting, spam filtering thing
Spammers keep trying, from the same IPs, for days here, so graylisting is useless for us. On SA and other things that require training, this is a nice story for you. A client received an average of 60 spam items per day on his own inbox alone. He trusted Kasperski, was confident on the accountant's ability to filter manually, and was happy to pay for everybody's waste of time in filtering their own daily dose of junk. This is a company with a cash-flow of 12M per year. One day, the accountant printed a fake bill: both archives and backups were encrypted by a ransomware. The boss paid for it, against the advice of the police, because funding international organised crime is a penal offence here. He obtained the antivirus, but it did not work. Sent from ProtonMail Mobile On Fri, Sep 29, 2017 at 3:06 PM, Markus Rosjat wrote: > Hi there, my boss is getting on my nerves that greylisting is basically out > of date because of things like outlook.com and mails ending up delayed for > ever. So the next logical step would be to deploy a tool like rspamd or > spamassasin to examin mail content. These tools need to be trained and if you > have a small mailserver with less accounts this could take a while I imagine. > So my question is, is there some source that you could use to train these > kind of tools (like a database that you could connect to for training > conntent ) or is every one here, that uses these tools, lucky enough to have > a shit load of users that do the training for your systems? some informations > about this would be helpful regards -- Markus Rosjat fon: +49 351 8107223 > mail: ros...@ghweb.de G+H Webservice GbR Gorzolla, Herrmann Königsbrücker > Str. 70, 01099 Dresden http://www.ghweb.de fon: +49 351 8107220 fax: +49 351 > 8107227 Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! > Before you print it, think about your responsibility and commitment to the > ENVIRONMENT
[rspamd and smtpd] (was: the whole greylisting, spam filtering thing)
By the way, does anyone has some instructions to use rspamd with the default smtpd ? Regards. -- thuban
Re: the whole greylisting, spam filtering thing
Hi Markus/all, On Fri, 29 Sep 2017 15:06:29 +0200 Markus Rosjat wrote: > ... greylisting ... like outlook.com and mails ending up delayed > for ever The 'ungrey-robins' tool automatically solves this problem for round-robin sending servers (Google, Outlook, Amazon, Yahoo, BT, etc.) Start with the README & see the logs directory for evidence: http://web.Britvault.Co.UK/products/ungrey-robins/ Otherwise;- simply set spamd's greylisting expire time to 4 days, not 4 hours. I ran servers this way for years - the mail does come through... Cheers, -- Craig Skinner | http://twitter.com/Craig_Skinner | http://linkd.in/yGqkv7
Re: the whole greylisting, spam filtering thing
On 09/30/17 04:39, Stuart Henderson wrote: >> It won't surprise anyone here that I disagree with the assertion that >> greylisting is in any way outdated. Come back with that assertion when >> the SMTP RFC is amended to drop the retry requirement. > > These senders do retry, but not always from the same source address. > Are you aware of any requirement in RFC5321 about source addresses > of retries? I didn't find any when I looked (or even a requirement that > retries are done over the same IP protocol version). We had hoped for a clarification of the relevant parts in that RFC update, but unfortunately the RFC still does not require retrying from the same IP address. Back when the original was written it may have been the default assumption that retries would come from the same host, but even then at least some site would have had more than one outgoing mail exchanger in place. > Greylisting still has its place, but with the way email operates today, > exemptions are unavoidable if you have a requirement to communicate > reliably with users of many email services. Especially with a strict > per-host greylisting implementation, where you don't get any benefit > from the common thing where senders often arrange to retry from within > the same v4 /24. Unfortunately some senders (IIRC outlook.com being one) don't necessarily stay within the same /24, even. That's why we need the nospamd trick. And this will become incrementally more fun (fsvo) as more of the traffic moves to IPv6. -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ "Remember to set the evil bit on all malicious network traffic" delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: the whole greylisting, spam filtering thing
I start greylisting on the firewall and thats ok but should I implement a dedicated system for rspamd and relay the "ok-Mails" from there to the mailsystem or simply run rspamd on the mailsystem und plug it front of the mailserver like postfix? aha so if you are using Postfix then there are plenty anti-spam features that truly reduces the amount of spam and almost wipes it all out **during the SMTP session**: `man 5 postconf` and search for those patterns (this is postfix 3.1). # NETWORK restrictions (smtpd_client_restrictions) check_policy_service unix:private/policy reject_unknown_client_hostname check_client_access hash:/etc/postfix/client_access reject_rbl_client ... reject_unauth_pipelining unknown_client_reject_code = 554 smtpd_data_restrictions = reject_unauth_pipelining # HELO/EHLO restrictions reject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_unknown_helo_hostname regexp:/etc/postfix/helo.regexp # MAIL FROM restrictions check_sender_access hash:/etc/postfix/sender_access, reject_non_fqdn_sender, reject_unknown_sender_domain # RCPT TO restrictions reject_non_fqdn_recipient, reject_unknown_recipient_domain unknown_address_reject_code = 554 if some spam comes through that, it is a pretty one (and even passed tru the SPF check). This already gets rid of 98% of the spam for me. Adding rspamd or whatever milter on top of that would clearly get you to 99%. No greylisting is needed. Eventually make sure STARTTLS is enabled so the MX talk through TLS, setup your SPF records for your domain and eventually setup DKIM.
Re: the whole greylisting, spam filtering thing
Hi, thank you all for the helpful input on that subject. I have one last thing to ask about it. What would be a good approach to implementing rspamd? I start greylisting on the firewall and thats ok but should I implement a dedicated system for rspamd and relay the "ok-Mails" from there to the mailsystem or simply run rspamd on the mailsystem und plug it front of the mailserver like postfix? Regards -- Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de G+H Webservice GbR Gorzolla, Herrmann Königsbrücker Str. 70, 01099 Dresden http://www.ghweb.de fon: +49 351 8107220 fax: +49 351 8107227 Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you print it, think about your responsibility and commitment to the ENVIRONMENT
Re: the whole greylisting, spam filtering thing
On 2017-09-29, Peter N. M. Hansteen wrote: > On 09/29/17 15:06, Markus Rosjat wrote: > >> my boss is getting on my nerves that greylisting is basically out of >> date because of things like outlook.com and mails ending up delayed for >> ever. So the next logical step would be to deploy a tool like rspamd or >> spamassasin to examin mail content. These tools need to be trained and >> if you have a small mailserver with less accounts this could take a >> while I imagine. > > It won't surprise anyone here that I disagree with the assertion that > greylisting is in any way outdated. Come back with that assertion when > the SMTP RFC is amended to drop the retry requirement. These senders do retry, but not always from the same source address. Are you aware of any requirement in RFC5321 about source addresses of retries? I didn't find any when I looked (or even a requirement that retries are done over the same IP protocol version). Greylisting still has its place, but with the way email operates today, exemptions are unavoidable if you have a requirement to communicate reliably with users of many email services. Especially with a strict per-host greylisting implementation, where you don't get any benefit from the common thing where senders often arrange to retry from within the same v4 /24. What you can do with rspamd is only greylist mail that looks spammy but isn't scored highly enough to block outright. (Or you could think of that as making an exemption for mail that doesn't look too spammy). This works quite well in my experience. Unfortunately it's a lot more complex to configure than spamd, though once you start adding scripts and trying to work out who to whitelist, the spamd setup doesn't seem quite so straightforward either. Most of the spam that reaches my mailbox is forwarded by a (high IP reputation) host that sits behind spamd. (I'm looking at you, Chinese state-owned enterprise trying to order a batch of fox fur from my @openbsd address! And others.) That's a lot trickier to block on my side without false positives..
Re: the whole greylisting, spam filtering thing
On 2017-09-29, Larry Hynes wrote: > Markus Rosjat wrote: >> my boss is getting on my nerves > > It may be mutual. > >> that greylisting is basically out of date because of things like >> outlook.com and mails ending up delayed for ever. So the next logical >> step would be to deploy a tool like rspamd or spamassasin to examin >> mail content. These tools need to be trained and if you have a small >> mailserver with less accounts this could take a while I imagine. > > Specifically in relation to rspamd: If you spend some time reading > the documentation on the rspamd website you might find that: > > 1. the weight of rules which classify messages as 'ham' or 'spam' > i.e. those rules which rely on the 'training' of messages, does not > have to be, in the overall context, critical. rspamd deploys a > boatload of 'tests', by default, and even more can be enabled, and > each of those can be assigned a score. hamminess or spamminess is > just one 'test'. +1. rspamd doesn't do badly even with little/no training for spam/ham. It does have problems with certain mail, for example it likes to have various MIME headers, so you may need to make some exemptions for things like daily/security mail output, or mail from people who don't use MIME MUAs. > 2. That the rspamd website specifically links to 'pre-built' ham > and spam databases which you are free to download and use. Definitely you would need to read documentation if using tools like rspamd or spamassassin.
Re: the whole greylisting, spam filtering thing
On 09/29/17 15:06, Markus Rosjat wrote: > my boss is getting on my nerves that greylisting is basically out of > date because of things like outlook.com and mails ending up delayed for > ever. So the next logical step would be to deploy a tool like rspamd or > spamassasin to examin mail content. These tools need to be trained and > if you have a small mailserver with less accounts this could take a > while I imagine. It won't surprise anyone here that I disagree with the assertion that greylisting is in any way outdated. Come back with that assertion when the SMTP RFC is amended to drop the retry requirement. But there are actors in the email market that do not particularly care about standards compliance one way or the other, unfortunately (at least for those of us below critical mass in terms of volume) is to use the nospamd feature and not exposing those sending domains to greylisting at all. My sedimentary nospamd file, built on discovering SPF info for badly behaved domains, is available here https://home.nuug.no/~peter/nospamd - I only started commenting entries after a while, but it's a Works for me(tM) file. See man spamd for examples of how to include that in your config. If you want to build and maintain your own nospamd based on SPF records, Aaron Poffenberger's spf_fetch is very well worth looking into (see https://github.com/akpoff/spf_fetch) > So my question is, is there some source that you could use to train > these kind of tools (like a database that you could connect to for > training conntent ) or is every one here, that uses these tools, lucky > enough to have a shit load of users that do the training for your systems? Yes, you need content filtering too. As others have said, you won't be able to totally avoid the training effort based on local preferences, but with working greylisting in front of the content filtering, those servers will run a lot cooler than without. I suppose my long rant from a few years back is still relevant - https://bsdly.blogspot.no/2014/02/effective-spam-and-malware.html, for the fun parts of doing greytrapping see https://bsdly.blogspot.no/2013/05/keep-smiling-waste-spammers-time.html and https://bsdly.blogspot.no/2013/04/maintaining-publicly-available.html and of course https://bsdly.blogspot.no/2012/05/in-name-of-sane-email-setting-up-spamd.html might still be of some use. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ "Remember to set the evil bit on all malicious network traffic" delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: the whole greylisting, spam filtering thing
Hi Leo, Am 29.09.2017 um 16:57 schrieb Leo Unglaub: Hey, On 09/29/17 15:06, Markus Rosjat wrote: my boss is getting on my nerves that greylisting is basically out of date because of things like outlook.com and mails ending up delayed for ever. So the next logical step would be to deploy a tool like rspamd or spamassasin to examin mail content. These tools need to be trained and if you have a small mailserver with less accounts this could take a while I imagine i assume that your boss is not an engineer and also not very familiar with how emails work. Greylisting it clearly NOT out of date at all. Greylisting simply makes use of stuff that is defined in the SMTP RFC. Every email server is allowed to temporary deny the delivery of an email and ask the sending server for another try. well we use greylisting and I gave MS a free pass but sometimes it doesn't seem to work anyway but that's ok for me. The problem in this case is clearly Microsoft who has no idea how email is supposed to work. You have two options here. the customer will always complain no matter how often you explain the real problem :) A: Simply don't care about Microsoft and just send customers to a website where you describe the problem and tell them to contact Microsoft in order to fix there stuff. This works very well, my Company hosts around 2,3 Million mailboxes and we use Greylisting and customers are okay with it. B: You exclude the outlook.com outgoing servers from greylisting. Microsoft provides a list of IP addresses that they use for delivery: https://mail.live.com/mail/ipspace.aspx 65.54.190.0/26 65.54.190.64/26 65.54.190.128/26 65.54.190.192/26 65.55.116.0/26 65.55.111.64/26 65.55.116.64/26 65.55.111.128/26 65.55.34.0/26 65.55.34.64/26 65.55.34.128/26 65.55.34.192/26 65.55.90.0/26 65.55.90.64/26 65.55.90.128/26 65.55.90.192/26 65.54.51.64/26 65.54.61.64/26 207.46.66.0/28 157.55.0.192/26 157.55.1.128/26 157.55.2.0/26 157.55.2.64/26 Greetings Leo I also check the spf record files of MS and added them too so we will see what's going to happen. I need to move to a more up to date setup so I just check my options what's used these days and yes greylisting works for me as long as no office 365 is involved but a lot of business partners of our customers moving to 365 and the email solution so it becomes a problem for me too. It's just fustrating to see a mail greylisted from 40 different ips ... regards -- Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de G+H Webservice GbR Gorzolla, Herrmann Königsbrücker Str. 70, 01099 Dresden http://www.ghweb.de fon: +49 351 8107220 fax: +49 351 8107227 Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you print it, think about your responsibility and commitment to the ENVIRONMENT
Re: the whole greylisting, spam filtering thing
Hey, On 09/29/17 15:06, Markus Rosjat wrote: my boss is getting on my nerves that greylisting is basically out of date because of things like outlook.com and mails ending up delayed for ever. So the next logical step would be to deploy a tool like rspamd or spamassasin to examin mail content. These tools need to be trained and if you have a small mailserver with less accounts this could take a while I imagine i assume that your boss is not an engineer and also not very familiar with how emails work. Greylisting it clearly NOT out of date at all. Greylisting simply makes use of stuff that is defined in the SMTP RFC. Every email server is allowed to temporary deny the delivery of an email and ask the sending server for another try. The problem in this case is clearly Microsoft who has no idea how email is supposed to work. You have two options here. A: Simply don't care about Microsoft and just send customers to a website where you describe the problem and tell them to contact Microsoft in order to fix there stuff. This works very well, my Company hosts around 2,3 Million mailboxes and we use Greylisting and customers are okay with it. B: You exclude the outlook.com outgoing servers from greylisting. Microsoft provides a list of IP addresses that they use for delivery: https://mail.live.com/mail/ipspace.aspx 65.54.190.0/26 65.54.190.64/26 65.54.190.128/26 65.54.190.192/26 65.55.116.0/26 65.55.111.64/26 65.55.116.64/26 65.55.111.128/26 65.55.34.0/26 65.55.34.64/26 65.55.34.128/26 65.55.34.192/26 65.55.90.0/26 65.55.90.64/26 65.55.90.128/26 65.55.90.192/26 65.54.51.64/26 65.54.61.64/26 207.46.66.0/28 157.55.0.192/26 157.55.1.128/26 157.55.2.0/26 157.55.2.64/26 Greetings Leo
Re: the whole greylisting, spam filtering thing
Hi, Am 29.09.2017 um 15:39 schrieb Larry Hynes: Markus Rosjat wrote: my boss is getting on my nerves It may be mutual. of course but well :) that greylisting is basically out of date because of things like outlook.com and mails ending up delayed for ever. So the next logical step would be to deploy a tool like rspamd or spamassasin to examin mail content. These tools need to be trained and if you have a small mailserver with less accounts this could take a while I imagine. Specifically in relation to rspamd: If you spend some time reading the documentation on the rspamd website you might find that: 1. the weight of rules which classify messages as 'ham' or 'spam' i.e. those rules which rely on the 'training' of messages, does not have to be, in the overall context, critical. rspamd deploys a boatload of 'tests', by default, and even more can be enabled, and each of those can be assigned a score. hamminess or spamminess is just one 'test'. 2. That the rspamd website specifically links to 'pre-built' ham and spam databases which you are free to download and use. I'll check this out ! Thank you for the hint !!! regards -- Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de G+H Webservice GbR Gorzolla, Herrmann Königsbrücker Str. 70, 01099 Dresden http://www.ghweb.de fon: +49 351 8107220 fax: +49 351 8107227 Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you print it, think about your responsibility and commitment to the ENVIRONMENT
Re: the whole greylisting, spam filtering thing
On Fri, Sep 29, 2017 at 03:06:29PM +0200, Markus Rosjat wrote: So my question is, is there some source that you could use to train these kind of tools (like a database that you could connect to for training conntent ) or is every one here, that uses these tools, lucky enough to have a shit load of users that do the training for your systems? some informations about this would be helpful As far as I understand it, the bayesnian lerning system learns stuff pertinent to your domain, so you have to train it. Spam at one domain might not be spam at another. My system has only a few email addresses, but I keep all my spam. I make sa-learn --spam read the spam, and sa-learn --ham the stuff I've filtered and examined that they are all not-spam. It doesn't take long. https://wiki.apache.org/spamassassin/BayesInSpamAssassin -- J.
Re: the whole greylisting, spam filtering thing
Markus Rosjat wrote: > my boss is getting on my nerves It may be mutual. > that greylisting is basically out of date because of things like > outlook.com and mails ending up delayed for ever. So the next logical > step would be to deploy a tool like rspamd or spamassasin to examin > mail content. These tools need to be trained and if you have a small > mailserver with less accounts this could take a while I imagine. Specifically in relation to rspamd: If you spend some time reading the documentation on the rspamd website you might find that: 1. the weight of rules which classify messages as 'ham' or 'spam' i.e. those rules which rely on the 'training' of messages, does not have to be, in the overall context, critical. rspamd deploys a boatload of 'tests', by default, and even more can be enabled, and each of those can be assigned a score. hamminess or spamminess is just one 'test'. 2. That the rspamd website specifically links to 'pre-built' ham and spam databases which you are free to download and use.
the whole greylisting, spam filtering thing
Hi there, my boss is getting on my nerves that greylisting is basically out of date because of things like outlook.com and mails ending up delayed for ever. So the next logical step would be to deploy a tool like rspamd or spamassasin to examin mail content. These tools need to be trained and if you have a small mailserver with less accounts this could take a while I imagine. So my question is, is there some source that you could use to train these kind of tools (like a database that you could connect to for training conntent ) or is every one here, that uses these tools, lucky enough to have a shit load of users that do the training for your systems? some informations about this would be helpful regards -- Markus Rosjatfon: +49 351 8107223mail: ros...@ghweb.de G+H Webservice GbR Gorzolla, Herrmann Königsbrücker Str. 70, 01099 Dresden http://www.ghweb.de fon: +49 351 8107220 fax: +49 351 8107227 Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you print it, think about your responsibility and commitment to the ENVIRONMENT