Re: Stan's List [was: free antivirus scanner ?]
On 2012-01-14 5:39 PM, Stan Hoeppner wrote: On 1/14/2012 6:40 AM, Charles Marcus wrote: I was more interested in what specific changes he made in order to use it as a HELO blacklist, and how and why it avoided false positives when it is used the way we have been using it It wouldn't really require any changes. You could use it with check_helo_access as is. The reason it avoids FPs in this usage is just what he stated: legit MTAs with generic rDNS are going to HELO with a real hostname, not the rDNS string. Thanks to you and Noel for the explanation... I'd also be curious to see comparisons of blocked traffic in these two different uses (again by a high volume setup)... -- Best regards, Charles
Re: Stan's List [was: free antivirus scanner ?]
On 1/14/2012 6:40 AM, Charles Marcus wrote: > I was more interested in what specific changes he made in order to use > it as a HELO blacklist, and how and why it avoided false positives when > it is used the way we have been using it It wouldn't really require any changes. You could use it with check_helo_access as is. The reason it avoids FPs in this usage is just what he stated: legit MTAs with generic rDNS are going to HELO with a real hostname, not the rDNS string. -- Stan
Re: Stan's List [was: free antivirus scanner ?]
On 1/14/2012 6:40 AM, Charles Marcus wrote: > I was more interested in what specific changes he made in order to > use it as a HELO blacklist, and how and why it avoided false > positives when it is used the way we have been using it > To use it as a HELO blacklist, you simply call it with check_helo_access pcre:/path/to/file This is less likely to have FPs since most mail admins will not use a dynamic-looking rDNS as their HELO hostname. Many bots apparently are coded to use the rDNS as their HELO; it will catch those.
Re: Stan's List [was: free antivirus scanner ?]
On 2012-01-13 6:05 PM, Stan Hoeppner wrote: On 1/13/2012 2:13 PM, email builder wrote: We use a modified version as a HELO blacklist. This avoids the false Interesting... can you provide specific details on what you mean by 'modified version'? I second that. I'm feeling convinced enough to use it as it was intended, BUT ideally, I don't desire rejecting even those stubborn people who insist on running their email server from their bedroom without relaying through their ISP. Used as intended fqrdns.pcre will definitely block the "bedroom email servers" doing direct-to-MX, because odds are high that his parents' PC or little sister's PC are infected with a spam bot, which again is the purpose of this table. Thus, if that is really your desire, you don't want to use this table. You're better off using Postscreen. I was more interested in what specific changes he made in order to use it as a HELO blacklist, and how and why it avoided false positives when it is used the way we have been using it -- Best regards, Charles
Re: Stan's List [was: free antivirus scanner ?]
On 13 jan. 2012, at 21:13, email builder wrote:
>>> We use a modified version as a HELO blacklist. This avoids the false
>>> positives we saw while testing it as a reverse DNS restriction but,
>>> because the use of the reverse hostname as the HELO string is a
>>> common pattern in spam attempts from compromised hosts, it's still
>>> very effective.
>>
>> Interesting... can you provide specific details on what you mean by
>> 'modified version'?
>
> I second that. I'm feeling convinced enough to use it as it was
> intended, BUT ideally, I don't desire rejecting even those stubborn
> people who insist on running their email server from their bedroom
> without relaying through their ISP.
>
> Do you have a script that modifies the list into whatever format your
> method requires?
>
> Does anyone have any comments on the efficacy of this method?
>
> I assume all it would take is for bots to change the way they
> create their HELO hostname to bypass this.
The modifications are rather basic, really.
We've commented out some lines that were giving us false positives,
and modified the REJECT message to match its context, as well as
adding the error code we use for post processing and the like.
Legitimate mail servers aren't an issue for us, since they do not
use the reverse DNS string as their HELO greeting, and therefore
they do not get rejected. They might get rejected for other reasons
(hello 'sbs2003.local'!) but that's not during this step.
It's currently maintained by hand, as automating it would take
more time that it'd save right now. Premature optimization etc.
As for bots changing their habits, I am not worried. New patterns
do emerge at times, but old habits die hard. If at some point it
turns out that it is no longer as effective, like after an upgrade
to 2.8 or higher, it will be reevaluated.
Cya,
Jona
--
P.S.: As for false positives, we had to comment out the following;
/^dd[1-9][0-9]{3,5}\.kasserver\.com$/
/^h[1-9][0-9]{3,7}\.stratoserver\.net$/
They are the default HELO strings for DS/VPS providers here in
Europe. The reverse DNS has often been updated to match the
domain name of the main website or whatever, so it tends to be
unique to our way of using the list.
We have tried in the past to get people to update their HELO, but
that turned out to be futile, and the amount of FPs we get from
it are higher than the spam attempts blocked.
Hence their removal from the list. YMMV, of course :-)
Re: Stan's List [was: free antivirus scanner ?]
On 1/13/2012 2:13 PM, email builder wrote: >>> We use a modified version as a HELO blacklist. This avoids the false >> Interesting... can you provide specific details on what you mean by >> 'modified version'? > > I second that. I'm feeling convinced enough to use it as it was > intended, BUT ideally, I don't desire rejecting even those stubborn > people who insist on running their email server from their bedroom > without relaying through their ISP. Used as intended fqrdns.pcre will definitely block the "bedroom email servers" doing direct-to-MX, because odds are high that his parents' PC or little sister's PC are infected with a spam bot, which again is the purpose of this table. Thus, if that is really your desire, you don't want to use this table. You're better off using Postscreen. -- Stan
Re: Stan's List [was: free antivirus scanner ?]
>> We use a modified version as a HELO blacklist. This avoids the false >> positives we saw while testing it as a reverse DNS restriction but, >> because the use of the reverse hostname as the HELO string is a >> common pattern in spam attempts from compromised hosts, it's still >> very effective. >> >> It's a 'check_helo_access' restriction in our >> 'smtpd_recipient_restrictions', and sits right before our RBL > checks, >> where it has blocked 17235 attempts so far this year, with zero false >> positives since we started using it, in November somewhere. > > Interesting... can you provide specific details on what you mean by > 'modified version'? I second that. I'm feeling convinced enough to use it as it was intended, BUT ideally, I don't desire rejecting even those stubborn people who insist on running their email server from their bedroom without relaying through their ISP. Do you have a script that modifies the list into whatever format your method requires? Does anyone have any comments on the efficacy of this method? I assume all it would take is for bots to change the way they create their HELO hostname to bypass this.
Re: Stan's List [was: free antivirus scanner ?]
On 1/13/2012 3:48 AM, DTNX/NGMX Postmaster wrote: > On 11 jan. 2012, at 16:12, email builder wrote: > >>> http://www.hardwarefreak.com/fqrdns.pcre <-- Stan's big list >> >> So who is using Stan's list? What do people have to say about >> it? What should I consider in regard to possibly implementing it? > > We use a modified version as a HELO blacklist. This avoids the false > positives we saw while testing it as a reverse DNS restriction but, because > the use of the reverse hostname as the HELO string is a common pattern in > spam attempts from compromised hosts, it's still very effective. > > It's a 'check_helo_access' restriction in our 'smtpd_recipient_restrictions', > and sits right before our RBL checks, where it has blocked 17235 attempts so > far this year, with zero false positives since we started using it, in > November somewhere. > > So another 'Thanks!' to Stan for providing something that saves us quite a > bit of time :-) You're welcome Jona. Glad it's useful for you in this manner. I believe there are also some users who replace all of the default actions with their own PREPEND for use in scoring systems. I'm glad you are taking advantage of "...As such, you are completely free to use it and modify it as you see fit, for your purposes, with absolutely no strings attached." Even better than GPL in many ways. I still wish to this day I knew the identity of the original author of the predecessor REGEXP table, so I could credit him in the docs, but he simply wished to remain anonymous. I feel kinda weird/guilty/? each time I receive thanks/praise/credit for something I did not create, but merely polished a little, expanded a little, maintain a little, and did a little word-of-mouth advertising of on some lists. The heavy lifting was done by others. So, thanks foes to the anonymous original author as well. -- Stan
Re: Stan's List [was: free antivirus scanner ?]
On 2012-01-13 4:48 AM, DTNX/NGMX Postmaster wrote: We use a modified version as a HELO blacklist. This avoids the false positives we saw while testing it as a reverse DNS restriction but, because the use of the reverse hostname as the HELO string is a common pattern in spam attempts from compromised hosts, it's still very effective. It's a 'check_helo_access' restriction in our 'smtpd_recipient_restrictions', and sits right before our RBL checks, where it has blocked 17235 attempts so far this year, with zero false positives since we started using it, in November somewhere. Interesting... can you provide specific details on what you mean by 'modified version'? I'm always looking for a safe (fp free) native postfix way to increase blocks before RBL checks... Thanks, -- Best regards, Charles
Re: Stan's List [was: free antivirus scanner ?]
On 11 jan. 2012, at 16:12, email builder wrote: >> http://www.hardwarefreak.com/fqrdns.pcre <-- Stan's big list > > So who is using Stan's list? What do people have to say about > it? What should I consider in regard to possibly implementing it? We use a modified version as a HELO blacklist. This avoids the false positives we saw while testing it as a reverse DNS restriction but, because the use of the reverse hostname as the HELO string is a common pattern in spam attempts from compromised hosts, it's still very effective. It's a 'check_helo_access' restriction in our 'smtpd_recipient_restrictions', and sits right before our RBL checks, where it has blocked 17235 attempts so far this year, with zero false positives since we started using it, in November somewhere. So another 'Thanks!' to Stan for providing something that saves us quite a bit of time :-) Cya, Jona
Re: Stan's List [was: free antivirus scanner ?]
On 1/12/2012 1:11 AM, Stan Hoeppner wrote: > On 1/11/2012 3:56 PM, email builder wrote: > http://www.hardwarefreak.com/fqrdns.pcre <-- Stan's big list > >> Noel, thank you for the thorough response. Thanks also to >> all the other responders. I'm definitely convinced. :) >> >> And of course, thanks to Stan! > > Of all days for me to be away from the KB most of the day... ;) > > Others have already hit the high points (thanks guys) so I'll simply try > to clarify a few points, confirm some things. > > It may be maintained and hosted in a "hobby" manner, but I'd say the > file gets as much, if not more, professional use than hobby use. I'll clarify that my "hobby" remark refer to the fact that the list is provided free to the world at your own expense, as a result of your own good will and personal interest. No commercial interests are involved. In no way does the hobby remark refer to the quality or usefulness of the list. Indeed, this is one more example of a pure-volunteer effort providing a useful high-quality tool. Obviously, organizations both large and small, private and commercial, benefit from this. Thanks! -- Noel Jones
Re: Stan's List [was: free antivirus scanner ?]
On 1/11/2012 3:56 PM, email builder wrote: http://www.hardwarefreak.com/fqrdns.pcre <-- Stan's big list > Noel, thank you for the thorough response. Thanks also to > all the other responders. I'm definitely convinced. :) > > And of course, thanks to Stan! Of all days for me to be away from the KB most of the day... ;) Others have already hit the high points (thanks guys) so I'll simply try to clarify a few points, confirm some things. It may be maintained and hosted in a "hobby" manner, but I'd say the file gets as much, if not more, professional use than hobby use. A small sample of hosts that download the file 'regularly'. Yes, the top two are Barracuda Networks, makers of anti-spam appliances. XS4ALL is a large ISP in the Netherlands. Xelerence is a Canadian manufacturer of DNSSEC security appliances, etc. 64.235.144.50: mail.ess.barracuda.com 64.235.150.204: mail.ess.barracuda.com 82.161.212.182: firehole.xs4all.nl 83.163.53.136: puzzled.xs4all.nl 193.110.157.188: mx.xelerance.com 83.139.103.70: dougie.bnet.hr 83.139.103.73: jimbo.bnet.hr 38.126.132.162: gateway.dclab.com 62.77.203.7: z.mx.invitel.net 67.217.170.17: ashburn1.wheelockweb.com 70.167.15.110: mailbox.dop.com 80.150.141.20: mail.ecka-granules.com 88.198.27.178: mx20.monline.it 173.220.103.45: oouser.polylogics.net 129.241.43.189: fender.itea.ntnu.no 147.215.1.189: cache.esiee.fr 147.32.9.2: nms.fjfi.cvut.cz 194.156.172.86: mail96.atlas.de 70.43.81.99: smtp.media-brokers.com 129.187.15.138: lxbsc02.ws.lrz.de 207.10.60.100: hide-inside.nemetschek.net Very little time goes into this. Maintenance simply consists of adding a expression to match a new or previously unknown rDNS string. This typically occurs when an ISP receives a new netblock from its RIR and puts it into service. Rarely, one ISP acquires another and renames their generic rDNS patterns to reflect the new company name. Adding or changing rDNS strings is typically a pretty major infrastructure change. Thus don't simply don't occur very often. It's akin to adding new area codes or prefixes to a telephone system. It primarily targets 'consumer' rDNS patterns, both dynamic and some static. Some static patterns are rejected, some have a prepend for use with SA and the like. Regarding Postscreen integration I don't see that as a win. Postscreen and the fqrdns.pcre table both target bot spam, but in different ways. Postscreen targets bot spam directly, fqrdns.pcre indirectly. Postscreen tends to stop most bot spam on it's own without need of dnsbls or this pcre table. Leaving fqrdns.prce in smtpd should kill any that make it through Postscreen, if any do. For those who don't know regular expressions, the table looks huge because we use fully qualified expressions in most cases, making them very long lines. And each expression has ISP specific optional rejection text, taking up even more space. As Noel mentioned you may get FPs if receiving mail from a host that sits on a residential or small biz line. If that occurs the proper way around it is to whitelist the client IP address, sending domain, or sender address, with an access table. See 'man 5 access'. Let us/me know how it works for you. Easiest way to check its performance is with something like: $ grep -i -c "please relay" /var/log/[your-mail-log-file] -- Stan
Re: Stan's List [was: free antivirus scanner ?]
>>> http://www.hardwarefreak.com/fqrdns.pcre <-- Stan's big list >> >> I've been curious about Stan's list of pcres. It looks massive, >> and Stan >> seems to be a regular expert contributer here. But I'm reluctant to >> start using a text file from a web site with nothing on it and only a >> bit of documentation in the file itself (not saying it's not >> sufficiently >> clear to implement, just that I don't feel like I have enough info to >> judge if the list will continue to be supported, if it's supported by >> more than one person, if it's supported just as a hobby or not, >> whether or not many other administrators are making use of it..) >> >> So who is using Stan's list? What do people have to say about >> it? What should I consider in regard to possibly implementing it? > > I use it. Stan occasionally updates it based his experience and > user feedback. I see the last update was 2011/08. Unlike a dnsbl, > this list does not require much updating; a few times a year is > adequate. > > I would consider the list a hobby like many other non-commercial > free antispam services. If Stan decides to stop supporting and/or > publishing it, it will still keep on working, and someone else might > volunteer "official" maintenance. Since it's basically a text > file, > any user is free to add/remove entries as they see fit. I expect > that even with zero updates the file would continue to be fairly > effective for years. > > As stated elsewhere, the intent of the file is to reject "consumer" > dynamic internet connections without the overhead of a DNS lookup. > Connections from these hosts are almost always spambots doing > direct-to-MX spamming. > > I would classify it as low risk of false positives, and fairly safe. > (but not 100% safe; few rules are. YMMV and such.) I've had a > couple of FP's from idiots that run their business mail servers on a > cablemodem with a dynamic rDNS name (their IP is static, but the > rDNS incorrectly says dynamic), so I added their IP to a local > whitelist. You may or may not run into the same easily-fixed problem. > > Use it like: > smtpd_client_restrictions = > permit_mynetworks > # uncomment next line if using SASL > # permit_sasl_authenticated > check_reverse_client_hostname_access pcre:/etc/postfix/fqrdns.pcre > > For testing, you can prepend warn_if_reject like this: > warn_if_reject reject_reverse_client... > This causes postfix to log a warning: message without actually > rejecting the mail. Then examine the log for anything interesting. > > It can alternately be used in smtpd_recipient_restrictions (or any > smtpd_*_restrictions sections), but be sure it's after > permit_mynetworks and permit_sasl_authenticated so you don't reject > your own authorized users. > > If you have an old postfix version that doesn't have the > check_reverse_client_hostname_access restriction, you'll need to use > check_client_access instead. The check_client_access will be a > little less effective, but doesn't affect safety. Noel, thank you for the thorough response. Thanks also to all the other responders. I'm definitely convinced. :) And of course, thanks to Stan!
Re: Stan's List [was: free antivirus scanner ?]
On Wed, 11 Jan 2012 07:12:15 -0800 (PST), email builder wrote: So who is using Stan's list? its blowing in the wind What do people have to say about it? good What should I consider in regard to possibly implementing it? ask for paypal account to pay Stan
Re: Stan's List [was: free antivirus scanner ?]
On Wednesday 11 January 2012 12:52:42 Mark Alan wrote: > I would also be interesting to be able to use a similar mechanism > earlier, from the postscreen_access_list (after permit_mynetworks > but before going outside to fetch the postscreen_dnsbl_* stuff): > > postscreen_access_list = permit_mynetworks, > check_reverse_client_hostname_access pcre:/etc/postfix/fqrdns.pcre Not possible, because postscreen does not look up PTR. > But http://www.postfix.org/postconf.5.html#postscreen_access_list > states: (This link also lists all possible actions, and check_reverse_client_hostname_access is not among them.) > "To discourage the use of hash, btree, etc. tables, there is no > support for substring matching like smtpd(8). Use CIDR tables > instead." But this is merely a dumbed-down version of check_client_access, so you're out of luck on this one. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
Re: Stan's List [was: free antivirus scanner ?]
On Wed, 11 Jan 2012 10:19:36 -0600, Noel Jones wrote: > I would classify it as low risk of false positives, and fairly safe. > (but not 100% safe; few rules are. YMMV and such.) I've had a > couple of FP's from idiots that run their business mail servers on a > cablemodem with a dynamic rDNS name (their IP is static, but the > rDNS incorrectly says dynamic), so I added their IP to a local > whitelist. You may or may not run into the same easily-fixed problem. > > Use it like: > smtpd_client_restrictions = > permit_mynetworks > # uncomment next line if using SASL > # permit_sasl_authenticated > check_reverse_client_hostname_access pcre:/etc/postfix/fqrdns.pcre I would also be interesting to be able to use a similar mechanism earlier, from the postscreen_access_list (after permit_mynetworks but before going outside to fetch the postscreen_dnsbl_* stuff): postscreen_access_list = permit_mynetworks, check_reverse_client_hostname_access pcre:/etc/postfix/fqrdns.pcre But http://www.postfix.org/postconf.5.html#postscreen_access_list states: "To discourage the use of hash, btree, etc. tables, there is no support for substring matching like smtpd(8). Use CIDR tables instead." M.
Re: Stan's List [was: free antivirus scanner ?]
On 1/11/2012 9:12 AM, email builder wrote: > > >>> I'm searching for a friend (who has very few money) an open source >>> antivirus scanner for email server that works with Postfix. >>> >>> Any infos/links/advices welcome >> >> One link, Google, would have easily found clamav. >> >> Info/advice: with postscreen(8), sane HELO restrictions, and good >> DNSBLs, clamav is not going to get much use. >> >> http://www.postfix.org/POSTSCREEN_README.html <-- Postfix 2.8 req'd >> http://readlist.com/lists/postfix.org/postfix-users/28/140973.html >> http://jimsun.linxnet.com/misc/postfix-anti-UCE.txt >> http://www.spamhaus.org/zen/ <-- worth the cost if not gratis for you >> http://www.spamhaus.org/whitepapers/effective_filtering.html >> http://barracudacentral.org/rbl <-- gratis but registration req'd >> http://www.hardwarefreak.com/fqrdns.pcre <-- Stan's big list > > I've been curious about Stan's list of pcres. It looks massive, and Stan > seems to be a regular expert contributer here. But I'm reluctant to > start using a text file from a web site with nothing on it and only a > bit of documentation in the file itself (not saying it's not sufficiently > clear to implement, just that I don't feel like I have enough info to > judge if the list will continue to be supported, if it's supported by > more than one person, if it's supported just as a hobby or not, > whether or not many other administrators are making use of it..) > > So who is using Stan's list? What do people have to say about > it? What should I consider in regard to possibly implementing it? I use it. Stan occasionally updates it based his experience and user feedback. I see the last update was 2011/08. Unlike a dnsbl, this list does not require much updating; a few times a year is adequate. I would consider the list a hobby like many other non-commercial free antispam services. If Stan decides to stop supporting and/or publishing it, it will still keep on working, and someone else might volunteer "official" maintenance. Since it's basically a text file, any user is free to add/remove entries as they see fit. I expect that even with zero updates the file would continue to be fairly effective for years. As stated elsewhere, the intent of the file is to reject "consumer" dynamic internet connections without the overhead of a DNS lookup. Connections from these hosts are almost always spambots doing direct-to-MX spamming. I would classify it as low risk of false positives, and fairly safe. (but not 100% safe; few rules are. YMMV and such.) I've had a couple of FP's from idiots that run their business mail servers on a cablemodem with a dynamic rDNS name (their IP is static, but the rDNS incorrectly says dynamic), so I added their IP to a local whitelist. You may or may not run into the same easily-fixed problem. Use it like: smtpd_client_restrictions = permit_mynetworks # uncomment next line if using SASL # permit_sasl_authenticated check_reverse_client_hostname_access pcre:/etc/postfix/fqrdns.pcre For testing, you can prepend warn_if_reject like this: warn_if_reject reject_reverse_client... This causes postfix to log a warning: message without actually rejecting the mail. Then examine the log for anything interesting. It can alternately be used in smtpd_recipient_restrictions (or any smtpd_*_restrictions sections), but be sure it's after permit_mynetworks and permit_sasl_authenticated so you don't reject your own authorized users. If you have an old postfix version that doesn't have the check_reverse_client_hostname_access restriction, you'll need to use check_client_access instead. The check_client_access will be a little less effective, but doesn't affect safety. -- Noel Jones
Re: Stan's List [was: free antivirus scanner ?]
On 2012-01-11 10:12 AM, email builder wrote: > So who is using Stan's list? What do people have to say about > it? What should I consider in regard to possibly implementing it? I am using it (for a while now)... This isn't really like a DNSBL, it simply rejects hosts that are 'spammy', meaning, on dynamic IPs - ie, botnets... There really is very little worry about FPs (false positives) now, and it doesn't need a lot of maintenance... even if Stan never updated it again, it would continue to be useful and with little to no FPs probably for many years to come... Try it, you'll be glad you did... ;) -- Best regards, Charles
