Re: Postscreen update
Stan Hoeppner wrote: For example: http://www.spamhaus.org/datafeed/ The Spamhaus DNSBL Datafeed is a service for users with professional DNSBL query requirements, such as corporate networks and ISPs. It offers both a Query service and an Rsync service (you can choose). The paid Query service mentioned above requires the Postfix feature you are asking about. It's an authentication mechanism. The Rsync service allows downloading the entire Spamhaus databases multiple times a day and hosting them on a local dns server or via an rbldnsd daemon on each MX. The latter is suitable for those such as big ISPs with massive mail flows, who cannot afford the latency of over the wire network based dnsbl queries. It's also a reasonable option due to cost; the paid query service is more expensive (at least at the level we were looking at here) compared to the rsync service. A remote dnsbl query can take anywhere from 20-200 milliseconds (or more) depending on number of hops and network conditions. A query to a local network dns server can take less than 1ms. A query to an rbldnsd daemon residing on the MX MTA host itself can occur in a few microseconds, as it is an interprocess communication occurring at the speed of system memory. This is the preferred method for some of the worlds busiest MTAs. All this performance comes at a cost: the rbldnsd method requires multiple gigabytes of system memory for the Spamhaus zone files alone. Hmm, no, less than 100M: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 28776 rbldns20 0 81740 65m 700 S0 3.3 118:49.42 rbldnsd And this with a modest local blacklist loaded in as well. The on-disk files for all of the lists total just over 100M. We just run the Spamhaus data on a non-public zone on our general resolvers (running dnscache) and we have yet to see any latency problems. The biggest sysadmin/network costs for the rsync service are in configuration (may need extra scripting to distribute the data to multiple rbldnsd instances, depending on how you want to arrange your DNS services - otherwise, it's set up once, let it run) and update bandwidth - currently they provide a script intended to be called once a minute to update the zone data source files. -kgd
Re: Postscreen update
Kris Deugau put forth on 9/29/2010 2:33 PM: Hmm, no, less than 100M: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 28776 rbldns20 0 81740 65m 700 S0 3.3 118:49.42 rbldnsd I was going by information I received from another list. I don't use the data feed service. Does this include the CBL data set within Zen? I would make an educated guess that the size of the CBL data set would be over 100MB alone. 25 million 32bit IP addresses (4 bytes) would be 100MB, if my math is correct. 25 million bot infected hosts around the world seems like a very conservative estimate. And this with a modest local blacklist loaded in as well. The on-disk files for all of the lists total just over 100M. We just run the Spamhaus data on a non-public zone on our general resolvers (running dnscache) and we have yet to see any latency problems. With fast resolvers and local GigE, performance should be fine for many sites as you state. It's also easier to manage than running rbldnsd on each MX as you have a single update point. I know of one site, coincidentally also in Canada, running two MX hosts. Each receives, IIRC, on an average day, ~50 million connection attempts, 100 million total. This is nowhere near the numbers of a good sized ISP obviously and tiny compared to a gorilla such as Gmail. The OP runs rbldnsd on each MX, with the full Spamhaus zones minus the CBL. Also incorporated into the rbldnsd instances are extensive local block lists, the Enemies List, the CBL data, and some other mirrored dnsbl data. This may be the multi gigabyte setup I was thinking of, which isn't just Spamhaus zones. Interestingly, this site doesn't reject any spam due to any hits against any list. After DATA, a 55x is returned to the client, but the entire message is saved for further anti spam heuristics processing. It's one of the most elaborate setups I've heard of. Then again, some of the most elaborate setups _no one_ will probably hear about. ;) The biggest sysadmin/network costs for the rsync service are in configuration (may need extra scripting to distribute the data to multiple rbldnsd instances, depending on how you want to arrange your DNS services - otherwise, it's set up once, let it run) and update bandwidth - currently they provide a script intended to be called once a minute to update the zone data source files. Yeah, running the Spamhaus zones on local rbldnsd instances on each MX would require some distribution magic, as you state. Never done this myself. I'd be more inclined to go the route you've taken, if I were ever in a position to manage such a thing. -- Stan
Re: Postscreen update
On 09/13/2010 10:55 PM, Wietse Venema wrote: Postscreen is a single Postfix 2.8 daemon that keeps spambots away from Postfix SMTP server processes, so that more Postfix server resources remain available for handling mail. It will hopefully become part of the next stable Postfix release. After adding DNSBL weights and filters two weeks ago, I rewrote the remainder of postscreen in the past 1+ week, and spent the past several days updating documentation so that people can actually use this thing. The re-born postscreen has been running on several sites since the beginning of the weekend. Postscreen now has a built-in SMTP protocol engine that allows it to log the helo/sender/recipient of rejected mail. With a few good DNSBL lists, this can dramatically reduce the load on Postfix SMTP servers (blocking mail without logging is not an option for everyone). One cautionary note: postscreen is meant to handle mail from MTAs not end-user clients. Its protocol tests are safe for properly- implemented MTAs, but they have not been tested with end-user systems. Of course end-user systems should connect to the submission port, not the port 25 that postscreen listens on... See http://www.porcupine.org/postfix-mirror/POSTSCREEN_README.html for an overview, configuration information and more. The last code drop was postfix-2.8-20100913, which is the same code as snapshot 20100912, but with a bunch of minor documentation fixes. Be sure to review the RELEASE_NOTES file if you are upgrading from an older postscreen version - the DNSBL implementation now reveals the DNSBL domain name in SMTP replies, so it needs to be censored to avoid disclosing ZEN etc. passwords. Wietse Hi Wietse, iam currently using the postfix snapshot with the older postscreen version which was still experimental (the first 2.8 snapshot with postscreen), in combination with greylisting my spam levels dropped so low that i can currently not train the content based spamfilter. So is there a need to update because the release which included postscreen before is experimental ? I read the readme about deep protocol inspection and of course i will use it as soon as its needed since i currently have no spam at all is there a need to upgrade due instability fixes etc ? thx Frank
Re: Postscreen update
* Matt Hayes domin...@slackadelic.com: Thanks for the update. I'm working on implementing this now, however, I'm a bit confused with the postscreen_dnsbl_reply_map option. Why? It's just for mapping RBL names. Unless you have a paid subscription with spamhaus.org, you don't need it. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Postscreen update
Frank Doege: On 09/13/2010 10:55 PM, Wietse Venema wrote: Postscreen is a single Postfix 2.8 daemon that keeps spambots away from Postfix SMTP server processes, so that more Postfix server resources remain available for handling mail. It will hopefully become part of the next stable Postfix release. After adding DNSBL weights and filters two weeks ago, I rewrote the remainder of postscreen in the past 1+ week, and spent the past several days updating documentation so that people can actually use this thing. The re-born postscreen has been running on several sites since the beginning of the weekend. Postscreen now has a built-in SMTP protocol engine that allows it to log the helo/sender/recipient of rejected mail. With a few good DNSBL lists, this can dramatically reduce the load on Postfix SMTP servers (blocking mail without logging is not an option for everyone). One cautionary note: postscreen is meant to handle mail from MTAs not end-user clients. Its protocol tests are safe for properly- implemented MTAs, but they have not been tested with end-user systems. Of course end-user systems should connect to the submission port, not the port 25 that postscreen listens on... See http://www.porcupine.org/postfix-mirror/POSTSCREEN_README.html for an overview, configuration information and more. The last code drop was postfix-2.8-20100913, which is the same code as snapshot 20100912, but with a bunch of minor documentation fixes. Be sure to review the RELEASE_NOTES file if you are upgrading from an older postscreen version - the DNSBL implementation now reveals the DNSBL domain name in SMTP replies, so it needs to be censored to avoid disclosing ZEN etc. passwords. Wietse Hi Wietse, iam currently using the postfix snapshot with the older postscreen version which was still experimental (the first 2.8 snapshot with postscreen), in combination with greylisting my spam levels dropped so low that i can currently not train the content based spamfilter. So is there a need to update because the release which included postscreen before is experimental ? I read the readme about deep protocol inspection and of course i will use it as soon as its needed since i currently have no spam at all is there a need to upgrade due instability fixes etc ? If there is any need to update any supported Postfix release then there will be an announcement. The last supported release is Postfix 2.4. Wieste
Postscreen update
Postscreen is a single Postfix 2.8 daemon that keeps spambots away from Postfix SMTP server processes, so that more Postfix server resources remain available for handling mail. It will hopefully become part of the next stable Postfix release. After adding DNSBL weights and filters two weeks ago, I rewrote the remainder of postscreen in the past 1+ week, and spent the past several days updating documentation so that people can actually use this thing. The re-born postscreen has been running on several sites since the beginning of the weekend. Postscreen now has a built-in SMTP protocol engine that allows it to log the helo/sender/recipient of rejected mail. With a few good DNSBL lists, this can dramatically reduce the load on Postfix SMTP servers (blocking mail without logging is not an option for everyone). One cautionary note: postscreen is meant to handle mail from MTAs not end-user clients. Its protocol tests are safe for properly- implemented MTAs, but they have not been tested with end-user systems. Of course end-user systems should connect to the submission port, not the port 25 that postscreen listens on... See http://www.porcupine.org/postfix-mirror/POSTSCREEN_README.html for an overview, configuration information and more. The last code drop was postfix-2.8-20100913, which is the same code as snapshot 20100912, but with a bunch of minor documentation fixes. Be sure to review the RELEASE_NOTES file if you are upgrading from an older postscreen version - the DNSBL implementation now reveals the DNSBL domain name in SMTP replies, so it needs to be censored to avoid disclosing ZEN etc. passwords. Wietse
Re: Postscreen update
On 09/13/2010 03:55 PM, Wietse Venema wrote: Postscreen is a single Postfix 2.8 daemon that keeps spambots away from Postfix SMTP server processes, so that more Postfix server resources remain available for handling mail. It will hopefully become part of the next stable Postfix release. After adding DNSBL weights and filters two weeks ago, I rewrote the remainder of postscreen in the past 1+ week, and spent the past several days updating documentation so that people can actually use this thing. The re-born postscreen has been running on several sites since the beginning of the weekend. Postscreen now has a built-in SMTP protocol engine that allows it to log the helo/sender/recipient of rejected mail. With a few good DNSBL lists, this can dramatically reduce the load on Postfix SMTP servers (blocking mail without logging is not an option for everyone). One cautionary note: postscreen is meant to handle mail from MTAs not end-user clients. Its protocol tests are safe for properly- implemented MTAs, but they have not been tested with end-user systems. Of course end-user systems should connect to the submission port, not the port 25 that postscreen listens on... See http://www.porcupine.org/postfix-mirror/POSTSCREEN_README.html for an overview, configuration information and more. The last code drop was postfix-2.8-20100913, which is the same code as snapshot 20100912, but with a bunch of minor documentation fixes. Be sure to review the RELEASE_NOTES file if you are upgrading from an older postscreen version - the DNSBL implementation now reveals the DNSBL domain name in SMTP replies, so it needs to be censored to avoid disclosing ZEN etc. passwords. Wietse Wietse, Thanks for the update. I'm working on implementing this now, however, I'm a bit confused with the postscreen_dnsbl_reply_map option. I know this is useful when you enabled the DEEP checks, which I plan on doing, but want to make sure I have the full concept behind the above parameter before I turn anything on. I've not had to use anything involving a DNSBL and a password before so just curious what I'm missing. Thanks for the hard work! -Matt
Re: Postscreen update
Matt Hayes: Thanks for the update. I'm working on implementing this now, however, I'm a bit confused with the postscreen_dnsbl_reply_map option. I know this is useful when you enabled the DEEP checks, which I plan on doing, but want to make sure I have the full concept behind the above parameter before I turn anything on. No, postscreen now always says rejected by $dnsbl when it rejects mail. I've not had to use anything involving a DNSBL and a password before so just curious what I'm missing. Then, you don't have to worry about it. Wietse
Re: Postscreen update
Hi Wietse, that sounds great. Is there a RHEL5 source package available for building postfix-2.8-20100913? Thanks Morten -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix- us...@postfix.org] On Behalf Of Wietse Venema Sent: Monday, September 13, 2010 9:55 PM To: Postfix users Subject: Postscreen update Postscreen is a single Postfix 2.8 daemon that keeps spambots away from Postfix SMTP server processes, so that more Postfix server resources remain available for handling mail. It will hopefully become part of the next stable Postfix release. After adding DNSBL weights and filters two weeks ago, I rewrote the remainder of postscreen in the past 1+ week, and spent the past several days updating documentation so that people can actually use this thing. The re-born postscreen has been running on several sites since the beginning of the weekend. Postscreen now has a built-in SMTP protocol engine that allows it to log the helo/sender/recipient of rejected mail. With a few good DNSBL lists, this can dramatically reduce the load on Postfix SMTP servers (blocking mail without logging is not an option for everyone). One cautionary note: postscreen is meant to handle mail from MTAs not end-user clients. Its protocol tests are safe for properly- implemented MTAs, but they have not been tested with end-user systems. Of course end-user systems should connect to the submission port, not the port 25 that postscreen listens on... See http://www.porcupine.org/postfix-mirror/POSTSCREEN_README.html for an overview, configuration information and more. The last code drop was postfix-2.8-20100913, which is the same code as snapshot 20100912, but with a bunch of minor documentation fixes. Be sure to review the RELEASE_NOTES file if you are upgrading from an older postscreen version - the DNSBL implementation now reveals the DNSBL domain name in SMTP replies, so it needs to be censored to avoid disclosing ZEN etc. passwords. Wietse
Re: Postscreen update
On 09/13/2010 07:37 PM, Wietse Venema wrote: Matt Hayes: Thanks for the update. I'm working on implementing this now, however, I'm a bit confused with the postscreen_dnsbl_reply_map option. I know this is useful when you enabled the DEEP checks, which I plan on doing, but want to make sure I have the full concept behind the above parameter before I turn anything on. No, postscreen now always says rejected by $dnsbl when it rejects mail. I've not had to use anything involving a DNSBL and a password before so just curious what I'm missing. Then, you don't have to worry about it. Wietse Works for me. Just trying to catch the concepts behind all the configuration changes before someone pops into #postfix on freenode and asks! Thanks! -Matt
Re: Postscreen update
On Mon, 2010-09-13 at 19:20:05 -0400, Matt Hayes wrote: I'm a bit confused with the postscreen_dnsbl_reply_map option. http://www.postfix.org/postconf.5.html#postscreen_dnsbl_reply_map I've not had to use anything involving a DNSBL and a password before so just curious what I'm missing. That is probably because you do not pay for a DNSBL datafeed. :) In such cases, the DNSBL domain name includes a secret key; dnsbl_reply_maps prevent disclosure of that key to SMTP clients. -- Sahil Tandon sa...@freebsd.org
Re: Postscreen update
On 09/13/2010 09:31 PM, Sahil Tandon wrote: On Mon, 2010-09-13 at 19:20:05 -0400, Matt Hayes wrote: I'm a bit confused with the postscreen_dnsbl_reply_map option. http://www.postfix.org/postconf.5.html#postscreen_dnsbl_reply_map I've not had to use anything involving a DNSBL and a password before so just curious what I'm missing. That is probably because you do not pay for a DNSBL datafeed. :) In such cases, the DNSBL domain name includes a secret key; dnsbl_reply_maps prevent disclosure of that key to SMTP clients. Well that makes total sense. Thank you for the clarification. -Matt
Re: Postscreen update
Sahil Tandon put forth on 9/13/2010 8:31 PM: On Mon, 2010-09-13 at 19:20:05 -0400, Matt Hayes wrote: I've not had to use anything involving a DNSBL and a password before so just curious what I'm missing. That is probably because you do not pay for a DNSBL datafeed. :) In such cases, the DNSBL domain name includes a secret key; dnsbl_reply_maps prevent disclosure of that key to SMTP clients. For example: http://www.spamhaus.org/datafeed/ The Spamhaus DNSBL Datafeed is a service for users with professional DNSBL query requirements, such as corporate networks and ISPs. It offers both a Query service and an Rsync service (you can choose). The paid Query service mentioned above requires the Postfix feature you are asking about. It's an authentication mechanism. The Rsync service allows downloading the entire Spamhaus databases multiple times a day and hosting them on a local dns server or via an rbldnsd daemon on each MX. The latter is suitable for those such as big ISPs with massive mail flows, who cannot afford the latency of over the wire network based dnsbl queries. A remote dnsbl query can take anywhere from 20-200 milliseconds (or more) depending on number of hops and network conditions. A query to a local network dns server can take less than 1ms. A query to an rbldnsd daemon residing on the MX MTA host itself can occur in a few microseconds, as it is an interprocess communication occurring at the speed of system memory. This is the preferred method for some of the worlds busiest MTAs. All this performance comes at a cost: the rbldnsd method requires multiple gigabytes of system memory for the Spamhaus zone files alone. -- Stan
postscreen update: DNSBL filters and weights
Postscreen is a single Postfix 2.8 daemon that keeps spambots away from Postfix SMTP server processes, so that more Postfix server resources remain available for handling mail. It will hopefully become part of the next stable Postfix release. Below is a quote from the release notes about new filter and weight support for postscreen DNSBL lookup results. This was released earlier as non-production snapshot. It has held up well, and after some polishing of documentation and comments, it should be ready. Next steps are a) support for different expiration times for different tests, b) a dummy SMTP engine (similar to the smtp-sink test program) to log the client/helo/sender/recipient for blocked mail, and c) a simple form of greylisting if time permits. Wietse Incompatibility with snapshot 20100830 == Use postfix reload after installing this code, otherwise the dnsblog(8) daemon may complain. The postscreen-to-dnsblog protocol had to be changed to support DNSBL query result filters. Major changes with snapshot 20100830 Postscreen DNSBL support is extended with optional fixed-string filters, with optional integral weight factors, and with an adjustable threshold to block SMTP clients with DNSBL score = that threshold. Support for wild-card patterns will be added later. The updated postscreen configuration syntax is: postscreen_dnsbl_sites = domain[=ipaddr][*weight] ... postscreen_dnsbl_threshold = score Elements inside [] are optional, ipaddr is an IPv4 address, and weight and score are integral numbers. The [] are not part of the postscreen_dnsbl_sites input. By default, weight and score are equal to 1, and entries without filter will match any non-error DNSBL reply. Use a negative weight value for whitelisting. Examples: To use example.com as a high-confidence blocklist, and to block mail with example.net and example.org only when both agree, use: postscreen_dnsbl_threshold = 2 postscreen_dnsbl_sites = example.com*2, example.net, example.org To filter only DNSBL replies containing 127.0.0.4, use: postscreen_dnsbl_sites = example.com=127.0.0.4 See also postconf(5) for the fine details.
