New Mailinglist Home
Hello people, I have to move the ML to a new host because the old one will soon no longer be available. For everyone who wants to keep track of changes or possibly new development the new location is at https://listen.jpberlin.de/mailman/listinfo/policyd-weight-users -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: policyd-weight as a tarpit
On Sun, Mar 21, 2010 at 11:20:48PM +0100, Gregor Glashüttner wrote: Hi! I??m using policyd-weight and think it does a great job in rejecting spam. I would like to do a little more and eat up the spammers resources by tarpitting them. My system doesn??t have too much traffic so i simply added sleep(5); in policyd-weight??s section parse and store results, do some cleanup, return results, right before the two lines return($EREJECTMSG.$RHSBLMSG.$RELAYMSG.$DYN_DNS_MSG);. This should eat 5 secs of the spammers time, right? This also eats your resources as your MTA has to spawn new smtpd(8) processes for new clients (the other smtpd-ones wait for policyd-weight's answer). This could also enable a DoS (one can circumvent the cache-rejects by changing the sender-envelope and thus trigger your sleep) Also there is smtpd_error_sleep_time (default 1s) See man 8 smtpd | less +/TARPIT -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: policyd-weight: blocked because of previous errors
On Tue, Dec 22, 2009 at 01:45:56PM +0100, Helga Mayer wrote: Hello list, I have a problem with rejects due to cache entries. We use policyd-weight-0.1.14-beta-17. This is the message found in the logfile: Dec 21 16:09:28 smtp2 postfix/smtpd[16364]: connect from mail-telecontrol.customer.solnet.ch[82.220.17.226] Dec 21 16:09:29 smtp2 postfix/policyd-weight[30193]: decided action=550 temporarily blocked because of previous errors - retrying too fast. penalty: 30 seconds x 0 retries.; client=82.220.17.226 helo=smtp.telecontrol.ch from=$sen...@telecontrol.ch to=$recipi...@uni-hohenheim.de; delay: 0s Dec 21 16:09:29 smtp2 postfix/smtpd[16364]: NOQUEUE: reject: RCPT from mail-telecontrol.customer.solnet.ch[82.220.17.226]: 550 5.7.1 $recipi...@uni-hohenheim.de: Recipient address rejected: temporarily blocked because of previous errors - retrying too fast. penalty: 30 seconds x 0 retries.; from=$sen...@telecontrol.ch to=$recipi...@uni-hohenheim.de proto=ESMTP helo=smtp.telecontrol.ch There are no other log entries for 82.220.17.226 during the last 8 days. The cache entry is: [bz]grep 82.220.17.226 /var/log/...log... results only in this snippet? policyd-weight -s|grep 82.220.17.226 blocked: 82.220.17.226 1 0 1261408171 1261408171 (UNIX) is the date of the first (and only) reject + 2 seconds : 1261408171 = Mon, 21 Dec 2009 15:09:31 GMT + 2 seconds indeed sounds strange but could be explained if the log is done in GMT (which would make it then a retry after 59:57 minutes). Is the policy service used by many machines or _only_ by localhost? As a workaround we whitelisted the particular IP. The headers of a mail received from this server are: Received: from smtp.telecontrol.ch (mail-telecontrol.customer.solnet.ch [82.220.17.226]) by smtp2.rz.uni-hohenheim.de (Postfix) with ESMTP for $recipi...@uni-hohenheim.de; Tue, 22 Dec 2009 12:23:13 +0100 (CET) Received: from PRISM.telecontrol.local ([192.168.30.11]) by PRISM.telecontrol.local ([192.168.30.11]) with mapi; Tue, 22 Dec 2009 12:23:18 +0100 Does lead to a reject, yes. SENDER % host telecontrol.ch telecontrol.ch has address 93.88.240.108 telecontrol.ch mail is handled by 5 mta-gw.infomaniak.ch. % host mta-gw.infomaniak.ch mta-gw.infomaniak.ch has address 84.16.68.126 mta-gw.infomaniak.ch has address 84.16.68.125 HELO % host smtp.telecontrol.ch smtp.telecontrol.ch is an alias for mail.infomaniak.ch. mail.infomaniak.ch has address 84.16.68.123 mail.infomaniak.ch has address 84.16.68.124 CLIENT % host mail-telecontrol.customer.solnet.ch mail-telecontrol.customer.solnet.ch has address 82.220.17.226 The client is in no relation (naming or subnet-wise) to sender or helo. Would the sender use a correct HELO, he wouldn't have this problem. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.15 devel-3
Hello, this is a 'scoring fix' with explicit ALPHA status. Fix/Changes: Policyd-weight didn't check whether the (verified) client hostname matches the sender domain. CL_HOSTNAME_MATCHES_FROM(DOMAIN) uses the score of @helo_ip_in_client_subnet as the context is similiar. Logging (client=) changed to also tell the client name provided by postfix. This affects users which try to communicate with microsoft. I myself stumpled about this today (registering with eopen). Log-Example before Fix: 12:01:14 info: weighted check: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 HELO_IP_IN_CL16_SUBNET=-0.41 RESOLVED_IP_IS_NOT_HELO=1.5 (check from: .microsoft. - helo: .internal.smtp.mscom.phx. - helo-domain: .phx.) FROM/MX_MATCHES_NOT_UNVR_HELO(DOMAIN)=1.6 RANDOM_SENDER=0.25 IN_PM_RFCI=3.975 IN_ABUSE_RFCI=3.975; client=207.46.22.101 helo=internal.smtp.mscom.phx.gbl from=cnfrm...@microsoft.com to=i...@kuttendreier.de; rate: 6.39 Log-Example after Fix: 14:47:56 info: weighted check: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 HELO_IP_IN_CL16_SUBNET=-0.41 (check from: .microsoft. - helo: .internal.smtp.mscom.phx. - helo-domain: .phx.) CL_HOSTNAME_MATCHES_FROM(DOMAIN)=-1.2 RANDOM_SENDER=0.25 IN_PM_RFCI=0.875 IN_ABUSE_RFCI=0.875 helo_ips: internal.smtp.mscom.phx.gbl 216.32.180.22 207.46.232.182 207.46.197.32; instance=207.46.22.101cnfrm...@microsoft.com client=delivery.smtp.microsoft.com[207.46.22.101] helo=internal.smtp.mscom.phx.gbl from=cnfrm...@microsoft.com to=; rate: -4.11 (FYI: HELO_IP_IN_CL16_SUBNET might irritate. This means that the client IP might also be in in the subnet of the _FROM_ addresses. (which is the case here)) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.15 devel-1
Hello, even though I officially don't maintain polw, I want to thank Jonas Genannt for his work to start on IPv6 support. 0.1.15 devel-x should be dedicated to stabilize/test IPv6. Other priorities would be: - evaluating of RBL A records, 127.0.0.1, 127.0.0.2, and so on The result of _one_ RBl with many records should be not much higher than the record with the highest value. Simple addition/multiplying/averaging leads to false results. Eg: 127.1 = 3.4 127.2 = 1.5 127.3 = 1.5 127.4 = 2.3 ~~~ ~4.2 127.1 = 3.4 127.2 = 2.3 --- ~4.1 - configurable dynamic/static-host heuristic via regexp-array defaulting to the current set -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: security: version update: version 0.1.14 beta-17
On Sat, Mar 29, 2008 at 02:12:20PM -0400, Whit Blauvelt wrote: Hi Robert, What's this: postfix/policyd-weight[18125]: warning: cache: err: cache: chdir /tmp/.policyd-weight/: No such file or directory at /usr/sbin/policyd-weight line 2948, GEN8330 line 100 A misguiding log-message. Following patch makes it not more clear, but more correct. --- old/policyd-weight Fri Mar 28 15:55:22 2008 +++ new/policyd-weight Sat Mar 29 20:50:44 2008 @@ -2945,7 +2945,8 @@ # change directory to $LOCKPATH in order to get some # coredumps just in case. -chdir $LOCKPATH/cores/cache or die cache: chdir $LOCKPATH: $!; +chdir $LOCKPATH/cores/cache or die +cache: chdir $LOCKPATH/cores/cache: $!; mylog(info='cache spawned'); Manually killing policyd-weight implies to kill the cache instance. The way to completely shut down policyd-weight is policyd-weight -k stop This doesn't work anymore if the directory has been deleted, in such cases you need to do a ps xauww | grep policyd-weight and kill the pids by verifying that it is a policyd-weight process. Policyd-weight needs completely to be shut down _before_ you change the $LOCKPATH config parameter. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: security: version update: version 0.1.14 beta-15 (was: Insecure lockfile creation - vulnerability report)
On Thu, Mar 27, 2008 at 11:52:17PM +0100, Andrej Kacian wrote:
On Tue, 25 Mar 2008 01:40:31 +0100
Robert Felber [EMAIL PROTECTED] wrote:
the new version addresses the issue below. Policyd-weight does now exit if
it
detects symlinks on directories or sockets at startup or directory creation.
Hello Robert,
I'm afraid 0.1.14.15 doesn't fix the issue reported.
By symlinking /tmp/.policyd-weight to /root and starting policyd-weight, I was
still able to change ownership of /root directory to user policyd-weight is
configured to run as.
Thanks for reporting.
This is weird, and I am a little bit confused:
# perl -wle 'if(-l /tmp/.policyd-weight){ print err }'
err
The question is now, why the same test in policyd-weight is
not resulting in a true value.
--
Robert Felber (PGP: 896CF30B)
Munich, Germany
Policyd-weight Mailinglist - http://www.policyd-weight.org/
security: version update: version 0.1.14 beta-17
Hello, policyd-weight still did not check the working directory correctly. 1st: I assumed [ -L /foo/bar ] is the same as [ -L /foo/bar/ ] because the -L tells the file test what to look for. But in the latter form it is checked with S_IFDIR. We normalize the path with File::Spec-canonpath as s,/+$,, is not sufficient. 2nd: policyd-weight didn't check the ownership of real directories which might have been resulted in a race attack. Policyd-weight once gets the stat/lstat and reuses that information in order to provide some sort of atomicity of the check_symlnk() sub-routine. MD5 (policyd-weight)= 68373b7cfeda52b78df6229ed658771e SHA256 (policyd-weight) = 4245495685e516e00a363a97aaa17456f48c51fcbdb4458989a9d68db64083bc MD5 (policyd-weight-0.1.14.17.tar.gz) = c90128d2442ba343e8127dc0dbdcfd9a SHA256 (policyd-weight-0.1.14.17.tar.gz)= c13bac397cbd8c018b41686da4e4ce9450fb045752d7f0ab518d9836b39dbf36 -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Thanks for fix, Website slightly confused
On Fri, Mar 28, 2008 at 12:34:05PM -0400, Whit Blauvelt wrote: Hi, At the moment policyd-weight Version: 0.1.14 beta-17 links to beta-15, Empty the cache, force a reload. It is 0.1.14 beta-17 for both (beta, devel) versions. You are encouraged to update to a version newer than 0.1.14 beta-17 is nicely proactive. Fixed. Sorry. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: errors regarding cache
On Tue, Mar 25, 2008 at 08:51:59AM +0100, Martin Monshausen wrote: Hello, we've reinstalled our server recently and since then we've got the following error: (in mail.log) postfix/policyd-weight[22860]: warning: cache: err: cache: chdir /tmp/.policyd-weight/: No such file or directory at /usr/local/bin/policyd-weight line 2938, STDIN line 24. This might be due to the version update. Delete the directory and let policyd-weight rebuild it. postfix/policyd-weight[22859]: warning: cache_query: $csock couln't be created: connect: No such file or directory, calling spawn_cache() Can someone please give some advice how to resolve the problem? kill all policyd-weight instances. rm -rf /tmp/.policyd-weight policyd-weight start The latter message warning: cache_query: $csock couln't be created is normal at startups. It should appear once after each startup but not during runtime. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Announce: I stop development of policyd-weight
Hello, to put it short: I don't develop policyd-weight any further, nor do I do any patching. I don't have the time and resources anymore. Real- and Work-Life keeps me way to busy (When I started policyd-weight in 2005 I was able to program up to 48 hours. Today, with family and so on, I only have 1 to 2 hours with interrupts - while I need up to 1 or 2 hours to become familiar with the code again). I do this step also because I realized I cannot provide, due to the time-constraints, the constant reliability required for such a project anymore. I think, it is my responsibility to stop, when the quality/reliability of development gets lower. Probably I should even have done this step earlier. Another reason is, if I receive patches, they often contain module dependencies. I don't like to reject such patches or to stress users to not use modules. You will know what I mean if you received the Nth patch containing yet another simple module and if you look one day at the memory footprint or if you find yourself debugging N foreign modules. If someone wants to continue with policyd-weight then it would also be nice, if he takes over the policyd-weight.org domain. I also appreciate maintainers-only. Those seeking an alternative to policyd-weight might want to have a look at postfwd - which makes the step to stop on policyd-weight easier. I will keep the ML running until a new maintainer has set up a new ML. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: RBL-based greylisting using Policyd-weight and Postgrey
On Wed, Jan 16, 2008 at 06:09:20PM +0100, fili wrote: Ok, a bug. Fixing appears troublesome (breaks lowest-resource-usage-policy). Not certain whether requests which will be answered with 'rc:' should generally not be cached (this wouldn't break cache-resources). $CACHESIZE=0; Thanks Rovert, I've got it up and running now using $CACHESIZE=0; Do you think that no-caching might result in higher loads on a mail-heavy server? Not load, but more smtpd processes waiting for a polw reply. I've read the release info of 0.1.14 beta-14, specificly: results with 'rc:' as action are not cached Is it useful for my current setup to update? Useful yes, required, not really. And should I then change $CACHESIZE back to the default value? You can delete it (with the latest version). On a different note, wouldn't it be a good idea to introduce a variable like: $BLOCK_RETRY_TTL = 30; 30 being the seconds in which retries will be temporarily blocked. If this value is set to 0, then Policyd-weight won't block retries at all. This is the job of $NTTL and $NTIME in concert $NTTL (default: 1) The client is penalized for that many retries. $NTIME (default: 30) The $NTTL counter will only be decremented if the client waits at least $NTIME seconds. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Bug#461124: policyd-weight init.d/script needs 1 fix
On Wed, Jan 16, 2008 at 09:36:30PM +0100, Jan Wagner wrote: On Wednesday 16 January 2008 20:47, Justin Piszcz wrote: Perhaps per-Robert, Make one for the cache, one for the daemon? e.g., when one 'stops' a daemon, shouldn't it kill all relevant processes it created? what if anybody comes to the idea to stop the cache via initscript, but don't stop the daemon? In this case it should be ensured, that the cache is starting or is started, when the daemon starts. I guess thats most of the users are confused by 2 init scripts and will not use it in the correct way. For example stoping the cache while the daemon runs or things like that. I would prefer to echo a hint when stoping/restarting the daemon, how to stop the cache. Any invents? Well to avoid discussions about correctness I'd be okay if we say rc.d/policyd-weight stop|start|restart|dstop|drestart stop|start|restart - affects all (cache, daemon) dstop|drestart - affects only daemon In the end, everyone expects the cache to be killed, too. Those that have read an updated documentation know, that the daemon can also be stop/[re]started by just issuing /usr/sbin/policyd-weight stop|start or using the appropriate init params if they are used to manage everything via init scripts. I would count it as collateral damage if some users frequently purge the cache. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Re: RBL-based greylisting using Policyd-weight and Postgrey
On Thu, Jan 10, 2008 at 07:42:10PM +0100, fili wrote: Should work. Depending on what you want to achieve. greylist clients which are on at least one RBL reject clients which are on too many rbls If possible, I would like to use greylisting -only- if client appears on too many RBLs. In all other situations clients should pass thru (no 550 reject, no greylisting). Then your setup should work. Maybe you should set REJECTLEVEL to an insane high value like 100 or so. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: policyd-weight feature request
On Sun, Dec 23, 2007 at 06:23:11AM -0500, Justin Piszcz wrote: Hi, Was wondering if support for whitelists would be made available in policyd-weight? For example, see: http://www.dnswl.org/ I add it in here: 'list.dnswl.org',0.00,-5.0, 'DNSWL', change this to 'list.dnswl.org',-5.0,0, 'DNSWL', The first score is added if the RBL/DNSWL has hit, i.e. the client is listed. If the HIT score is greater than 0 it is treated as a RBL hit, if the score is less than 0 (eg: -1) it is treated as a DNSWL hit. But it still counts as a 'bad' RBL, is there any chance of making a whitelist section where if X number of whitelist RBLs include a certain IP -or- the value is less than X it is allowed? This then leads to a second question, perhaps one wants to place emphasis or weight upon the trust level: Per: http://www.dnswl.org/tech Trustworthiness / Score (127.0.x.Y): * 0 = none - only avoid outright blocking (eg Hotmail, Yahoo mailservers, -0.1) * 1 = low - reduce chance of false positives (-1.0) * 2 = medium - make sure to avoid false positives but allow override for clear cases (-10.0) * 3 = high - avoid override (-100.0). So it would need to be something like: list.dnswl.org ret=127.0.0.0 -5.0 list.dnswl.org ret=127.0.0.1 -3.0 Just an idea.. But the main request is a @whitelist for RBL's to help reduce false positives. Justin. Policyd-weight Mailinglist - http://www.policyd-weight.org/ -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Problem with false positives
On Wed, Nov 28, 2007 at 03:26:11PM +0100, Til Obes wrote: Hello, after months of successfull use of the daemon, we are having more and more problems :( For example mails from yahoo or strato are getting rejected. Logs? Do i have to use always the latest devel version? Which one are you using? /path/to/policyd-weight -v It also seems, that sometimes pw is giving up with to many dns lookup errors. If a NS is unresponsive, polw rather lets pass them to keep the mail flowing. Currently this behaviour is not configurable. Maybe it would be nice, to let administrators decide, whether or not to 4xx clients with unresponsive DNS rather than accepting them. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: RBL-based greylisting using Policyd-weight and Postgrey
On Thu, Jan 10, 2008 at 04:51:52PM +0100, Fili wrote: Hello Policyd List, I'm trying to set up RBL-based greylisting using Policyd-weight and Postgrey on Debian Etch. After some intensive Google-ing I came to the conclusion that it should be possible. However, I couldn't find any concrete configuration examples. Would the following configuration work? --postfix: main.cf-- smtpd_restriction_classes = greylist greylist = check_policy_service inet:127.0.0.1:6 smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, check_policy_service inet:127.0.0.1:12525 --policyd-weight.conf-- $REJECTLEVEL = 4.25; $dnsbl_checks_only = 1; $MAXDNSBLHITS = 4; $MAXDNSBLMSG = 'rc:greylist'; $BIND_ADDRESS= 'all'; Should work. Depending on what you want to achieve. greylist clients which are on at least one RBL reject clients which are on too many rbls It is unclear to me if the 'rc:greylist' is supported on the Debian packaged version: 0.1.14-beta-6. It contains handling for rc: messages. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Another MacOS problem Launching after reboot
On Fri, Nov 09, 2007 at 03:04:01PM -0500, Vince Admin Account wrote: First, policyd-weight is working EXTREMELY well. I am running it in daemon mode. The only problem is starting it automatically after a reboot. I created a launchd plist and launchd does start the polw daemon, but on one of the machines (the busy one) it does not connect properly to the DNS server, causing all the DNS lookups to fail. Manually restarting polw makes everything work. grep '(warn|err|main)' /var/log/maillog does give what? (also, do you see other signs of unusual log activity aside of warn, err?) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: HELO_NUMERIC
On Wed, Nov 07, 2007 at 04:15:37PM +, Riaan Kok wrote:
(another one of these!)
the test around line 2340 or so looks at whether there's any number or
closing square bracket at the end of a helo and then fires HELO_NUMERIC?
Say a server incorrectly identifies itself as EHLO server3, it already
gets penalised in the right places, and it might not be the intention to
fire off HELO_NUMERIC as well then?
Well, yes, the RE could also be expressed as
/^\d+\.\d+\.\d+\.\d+\]*$/
I stumbled onto a decent perl-regexp for identifying IP addresses on some
PHP site a while ago, if you think it useful here instead of the current
test:
/^\[(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])(\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])){3}\]$/
(square brackets added to fit this case)
Serious checks whether something is a valid IP should be done with
inet_* functions.
We don't need to be that serious and are fine with simple checks (if they are
not too simple as in /[\d|\]]$/ )
--
Robert Felber (PGP: 896CF30B)
Munich, Germany
Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: FROM_MULTIPARTED?
On Thu, Nov 01, 2007 at 01:45:32PM +, Riaan Kok wrote: Hi Robert and list, It seems that this test looks whether there is two or more full stops in the domain of a sender.. Am I correct? Shouldn't it be three or more? Yes. Should be sanitized. Otherwise you'd penalise blah.org.fr (for example).. and raise the false alarm rate for many normal international sender addresses. thanks, Riaan -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Policyd-weight questions
On Tue, Oct 16, 2007 at 06:33:28PM +0300, Henrik Krohns wrote: On Tue, Oct 16, 2007 at 05:17:38PM +0200, Robert Felber wrote: On Tue, Oct 16, 2007 at 11:47:53AM +0100, Riaan Kok wrote: On 16/10/2007, Robert Felber [EMAIL PROTECTED] wrote: On Tue, Oct 16, 2007 at 08:36:11AM +0200, Andreas Fuchs wrote: 2. in the log i have quite often the following entry Oct 16 08:30:53 schilt postfix/policyd[20148]: decided action=DUNNO NULL () Sender; delay: 0s I don't know exactly how to debug them, the process number is repeating quite often, any ideas? That are NULL-sender (mainly generated by DSN). You MUST let pass them. If the NULL Sender try to deliver to non-existing users then reject all mail for non-existent users. Why MUST they pass? How to ensure that DSN arrive from hosts to which you have sent mail but which are listed or otherwise penalized by policyd-weight? If the host happens to be penalized and is legimate, don't you have more to worry about than losing some DSN? :) Ok, a (ham) scoring for NULL sender will be done. The default will be to let pass NULL sender unscored, though. I am currently trying to make polw run on a mail system with 17 mil mails (loadbalanced) per day (196 mails per sec). With polw the smtpd porcesses grow gen sky (1000) Without polw postfix has around 200 processes open (per box). So, NULL sender scoring would certainly add a neat side-effect while being 'extremely' effective ;-) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: various patches against latest devel version
On Mon, Oct 01, 2007 at 11:49:57AM +0100, Riaan Kok wrote: Robert, On 25/09/2007, Robert Felber [EMAIL PROTECTED] wrote: On Tue, Sep 25, 2007 at 11:47:28AM +0100, Giles Westwood i wrote: I think it's a bit silly to score countries in the context of what policyd-weight does. It weights helo/dns/etc with scoring tuned specifically for it. If you add something like this to the mix, it gets pretty badly off-balanced I think? I think I've already stated that such changes (i.e. scoring by nationality, race, sex, age, opinion, religion) will be only available as inofficial patch which I do not host or give support for. I also recalled that I even have troubles with scoring OS/MTAs. People told me, that it is not up to me what to score but to give the possibility to score. Which is partly true. I think it is ok for people who want to setup a denial rampart stage to implement such possibilities themselves. Policyd-weight however does not want to be zero tolerant and a denial rampart. Policyd-weight does only want to enforce some configuration and does get a little help by RBLs (I've already stated, that I would love to get rid of RBLs, too). I admit, that the random sender check breaks this philosophy. The random sender check may even cause false positives. However, the random sender can be reconfigured - and the defaults score only high if DNSBL listed. The success of viruses and phishing is not only the fault of people who click on everything - it is more the fault of administrators who accept any faulty configuration (permitted by RFCs). I sometimes have the feeling that phishers and viruses point to the RFCs saying see, look at the RFCs, you must accept me, nelsonHaa Haaa/nelson or look at all the admins which accept such SMTP crap even though the RFCs permit them to reject such stuff, He He. My combination of postgrey and policyd with my corporate related tweaks works great though and we're considering removing dspam as it's hardly needed. I'm afraid that I use policyd unmodified on a different server with lots of unrelated clients but I had to set reject levels very high because genuine mail was rejected. Policyd-weight is designed to enforce a even more precise MTA configuration for dialup users. I.e. people who want to run a MTA on a dialup should setup every piece correctly and preferably sign up for a free DynDNS MX host. Whereas people from foreign countries do not really have a chance. Except sign up for a different country -- which is more of a burden and not free. Note: I mail sometimes from home with a DUL listed dialup through ek-muc and the home MTA must pass polw. This does only fail if I get a spamhaus listed IP - which is resolved by reconnecting automatically. This all does not mean that the patch is completely rejected, I haven't read everything yet. This is all actually useful to know for us who - use policyd-weight, - want to make constructive suggestions, - and/or want to improve or build on policyd-weight, but the website doesn't quite make it all clear.. I think it would be nice, when you've got the time, to add a Vision or For Potential Developers section to the website where you explain what policyd-weight IS and what it IS NOT and what kinds of contributions would be useful and welcome. I have - probably external hooks at certain stages on www.policyd-weight.org/todo.txt Which means, that at such stages people can hook in own rules/scorings/user-sql-lookups and the like. This, and maybe a long with the poissibility to play around with scoring like for instance http://postfwd.jpkessler.de/ I also have troubles with a feature-rich policy server. Feature-rich is Amavis/SpamAssassin. We all know how huge one Amavis Process is. This is the reason, why Amavis/SA should be an after-queue processor with an own queue. With this in mind I'd like to not exceed the 10mb mark - at least not as long as we don't have a full policy/DNS multiplex aware server. This thread suggests to me that there is a need out there to modify/customise policyd-weight, and although patches that take the program to areas where it was not intended to be is not useful for default inclusion, patches that makes it easier to customise or add modules to PW would be welcome? Currently I'd more like resource/portability/stability patches. Along with stuff from the todo.txt. Enabling different kinds of usage, perhaps even grouping its operation (RBLs, RFC stuff, regional, anti-spam, experimental, etc.).. Also enabling you to get rid of RBLs in the situation where you want to do it, but keep other folks happy that desire RBLs to stay.. Anyway, my 2 cents.. Riaan Policyd-weight Mailinglist - http://www.policyd-weight.org/ -- Robert Felber (PGP: 896CF30B) Munich
Re: 4xx or 5xx: just a matter of taste?
On Mon, Oct 01, 2007 at 01:41:56PM +0100, Riaan Kok wrote: On 01/10/2007, Robert Felber [EMAIL PROTECTED] wrote: On Mon, Oct 01, 2007 at 11:47:50AM +0100, Riaan Kok wrote: Fair enough, about default intentions, but the default operation of policyd-weight does not adhere to this. As low scores are more likely to be good and high scores are more likely to be bad, most of your false positives will sit in the score range just above the REJECTLEVEL.. And by default, everything above REJECTLEVEL and below DEFER_LEVEL gets deferred Not everything but clients whose log-line match DEFER_STRING. Which is SPAMCOP (a temporarily issue) and BOGUS_MX (a testing safety). ok, whoops, my misinterpretation then: it was not very clear to me from the .conf's notes that DEFER_LEVEL and DEFER_STRING are related. I'd still be interested if you or anybody have a rough idea what difference policyd-weight's cache makes on a system where PW already uses a caching DNS server on localhost.. Negative answeres reach their time to life rather quickly. Also this overrules RRs with a short TTL. In addition: 1 unix socket lookup should be quicker than ~ 15 DNS lookups. especially because the latest patch at Version 0.1.14beta-10 disables the PW cache for whomever wants to use 421's as their primary go-away action. thanks, Riaan -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: various patches against latest devel version
On Tue, Sep 25, 2007 at 11:47:28AM +0100, Giles Westwood i wrote: I think it's a bit silly to score countries in the context of what policyd-weight does. It weights helo/dns/etc with scoring tuned specifically for it. If you add something like this to the mix, it gets pretty badly off-balanced I think? I think I've already stated that such changes (i.e. scoring by nationality, race, sex, age, opinion, religion) will be only available as inofficial patch which I do not host or give support for. I also recalled that I even have troubles with scoring OS/MTAs. People told me, that it is not up to me what to score but to give the possibility to score. Which is partly true. I think it is ok for people who want to setup a denial rampart stage to implement such possibilities themselves. Policyd-weight however does not want to be zero tolerant and a denial rampart. Policyd-weight does only want to enforce some configuration and does get a little help by RBLs (I've already stated, that I would love to get rid of RBLs, too). I admit, that the random sender check breaks this philosophy. The random sender check may even cause false positives. However, the random sender can be reconfigured - and the defaults score only high if DNSBL listed. The success of viruses and phishing is not only the fault of people who click on everything - it is more the fault of administrators who accept any faulty configuration (permitted by RFCs). I sometimes have the feeling that phishers and viruses point to the RFCs saying see, look at the RFCs, you must accept me, nelsonHaa Haaa/nelson or look at all the admins which accept such SMTP crap even though the RFCs permit them to reject such stuff, He He. My combination of postgrey and policyd with my corporate related tweaks works great though and we're considering removing dspam as it's hardly needed. I'm afraid that I use policyd unmodified on a different server with lots of unrelated clients but I had to set reject levels very high because genuine mail was rejected. Policyd-weight is designed to enforce a even more precise MTA configuration for dialup users. I.e. people who want to run a MTA on a dialup should setup every piece correctly and preferably sign up for a free DynDNS MX host. Whereas people from foreign countries do not really have a chance. Except sign up for a different country -- which is more of a burden and not free. Note: I mail sometimes from home with a DUL listed dialup through ek-muc and the home MTA must pass polw. This does only fail if I get a spamhaus listed IP - which is resolved by reconnecting automatically. This all does not mean that the patch is completely rejected, I haven't read everything yet. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: what penalty?
On Wed, Sep 19, 2007 at 09:56:48AM +0100, Riaan Kok wrote: Hi there, On 18/09/2007, Robert Felber [EMAIL PROTECTED] wrote: On Tue, Sep 18, 2007 at 04:37:14PM +0200, Michael Mertel wrote: - decided action=550 temporarily blocked because of previous errors - retrying too fast. penalty: 30 seconds x 1 retries. (multirecipient mail) - I checked the log and was not able to find the previous errors. What event can cause this message This message is caused when a previous validation of the client,sender already lead to a reject. So, this message is seen when a cached negative client crosses a threshold of attempts within a time period? Is it right to use a 550-don't-try-again action here with no regard to the actions that has been applied to this client before? No it is not. Has been changed in 0.1.14 beta-10. Clients which were rejected with 4xx|DEFER_* are not cached anymore. You need to restart the cache if you have clients which were 4xx deferred in the cache. Note: polw users which made use of DEFER_ACTION, DEFER_LEVEL and DEFER_STRING do not need to worry. For those the results where not cached. I.e. this affects only users which said DEFER_* or 4xx in REJECTMSG and MAXDNSBLMSG. However - the OP didn't provide all logs regarding the rejected IP so we are left assuming. What's the thinking behind choosing between 450s and 550s in policyd-weight? Be more specific? -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: too many local DNS Errors
On Thu, Sep 13, 2007 at 10:49:54AM +0200, Jan Scholten wrote: Guten Tag policyd-weight-list, I installed policyd-weight and it works pretty well, reducing spams by roughly 50%-75% But i noticed that most of the spam that ist passed has following characteristics: Sep 13 08:24:01 svr postfix/smtpd[24881]: connect from unknown[90.188.9.6] Sep 13 08:24:56 svr postfix/policyd-weight[24843]: decided action=PREPEND X-policyd-weight: passed - too many local DNS-errors in HELO MX lookups for 9.6] The anaylzing of [n.n.n.n] HELO arguments is not yet completely programmed. However, I have not seen such HELOs producing local DNS errors. I have (from your private post) following result: echo helo_name=[85.110.88.63] client_address=85.110.88.63 [EMAIL PROTECTED] request=smtpd_access_policy | /path/to/policyd-weight -d 11:29:54 info: weighted check: IN_DYN_PBL_SPAMHAUS=3.25 NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 CL_IP_NE_HELO=4.75 RESOLVED_IP_IS_NOT_HELO=1.5 (check from: .bpop. - helo: .[85.110.88.63]. - helo-domain: .63].) FROM_NOT_FAILED_HELO(DOMAIN)=6.25 helo_ips: 64.178.213.17 64.178.214.6; [EMAIL PROTECTED] client=85.110.88.63 helo=[85.110.88.63] [EMAIL PROTECTED] to=; rate: 11.25 Can you please post following: echo helo_name=[85.110.88.63] client_address=85.110.88.63 [EMAIL PROTECTED] request=smtpd_access_policy | /path/to/policyd-weight -d polw-debug.txt -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: default configuration blocks legitimate mail (0.1.14 beta-6)
On Thu, Sep 13, 2007 at 05:56:07AM -0400, Justin Piszcz wrote: On Thu, 13 Sep 2007, Francis Galiegue wrote: Le jeudi 13 septembre 2007, Justin Piszcz a écrit : Aug 20 18:23:36 l2 postfix/smtpd[11969]: NOQUEUE: reject: RCPT from smtp2.netcabo.pt[212.113.174.29]: 550 5.7.1 [EMAIL PROTECTED]: Recipient address rejected: Mail appeared to be SPAM or forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settings or to get removed from DNSBLs; MTA helo: exch01smtp09.hdi.tvcabo, MTA hostname: smtp2.netcabo.pt[212.113.174.29] (helo/hostname mismatch); from=[EMAIL PROTECTED] to=[EMAIL PROTECTED] proto=ESMTP helo=exch01smtp09.hdi.tvcabo # VERSION: 0.1.14 beta-6 We see here that the ISP is tvcabo in Portugal but that they are going through an exchange server and it passed all of my postfix checks no errors, but it died here. Can we make either an exception for exch* (exhcange hosts) or somehow be more careful with this type of mail? I am not sure of the best approach but just reporting this and asking for suggestions. Thanks! This should be done at the Postfix level. A good way of doing this is to add in smtpd_recipient_restrictions the following: smtpd_recipient_restrictions = whatever, check_client_access hash:/etc/postfix/client_exceptions, whatever Be sure to add the check_client_access BEFORE policyd. In /etc/postfix/client_exceptions, put: the.ip.address OK # You can put a hostname instead of an IP address if you wish and compile the map with: postmap /etc/postfix/client_exceptions Once it's done, reload postfix (a restart is NOT needed). Yes this is what I ended up doing but I wish it had not been rejected in the first place, always have to lose that first e-mail. :( you could set up a pcre map smtpd_recipient_restrictions = ... reject_unauth_destinaion ... check_client_access pcre:/etc/postfix/exchange_exceptions.pcre check_policy_service ... /etc/postfix/exchange_exceptions.pcre /[^.]*(exch|smtp).*\..*\../ OK This won't help with postfix' unknown clients, though (I think). Also, you want to make exceptions based on a _failing_ HELO, so you would have to do a smtpd_recipient_restrictions = ... reject_unauth_destinaion ... check_helo_access pcre:/etc/postfix/exchange_exceptions.pcre check_policy_service ... with the file like above in order to allow broken exchanges, or clients which act like broken exchanges. This would mean, anyone who says HELO exchange.blah.nonresolving will not be handed to policyd-weight -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: 0.1.14 beta-7 (non official/fix-testing)
Hello, This version is not official because I haven't found the time to stress-test and package it. Thanks to everyone who fixes/reports critical bugs in order to release a next official beta. Changes from 0.1.14 beta-5 to 0.1.14 beta-7: 0.1.14 beta-7 (non-offficial as per Fri Sep 14 03:23:55 CEST 2007) - (fix) Header was not included if $dnsbl_checks_only = 1; and $ADD_X_HEADER = 1; - Thanks to J. Genannt - (fix) Corrected handling of [n.n.n.n] HELOs and address-literals as sender (long standing issue) - (change)Introduced @dnsbl_checks_only_regexps in order to skip DNS checks for certain client hostnames 0.1.14 beta-6 (non-offficial as per Fri Sep 14 03:23:55 CEST 2007) - (change)Added -D (Don't detach) switch for daemon-tools/runit users - (change)Added signals handlers for most of signals so that they are at least logged, also, provide a perl backtrace. - (change)prerequisite steps for providing coredumps (build coredump directories, chdir) - coredumps are non-trivial: we start as root, change uid. At this moment coredumps are denied by kernel in order to protect root-data. The only workaround would be, to start cache and master via system() after changing uid - (change)In daemon mode wrongly crafted policy requests don't lead to a child-exit anymore, only the connection is closed - (change)log-facilities other than 'info' are now mentioned in log-lines - (change)SMTP information such as client, helo, sender and to are now logged in each log-message. If $DEBUG is set this also logs the instance variable. - (fix) rbl_lookup used sometimes 65536 as packet id which appeared to cause problems - (fix) Check for syslog absence. If syslog is not available then log temporarily to $LOCKPATH/polw-emergency.log - (tmpfix)Introduced $TRY_BALANCE which closes connections to smtpds after they got their response in order to avoid too many established smtpd-policyd-weight (child) connections. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: add header and rbl checks only
On Thu, Sep 13, 2007 at 08:00:21PM +0200, Jonas Genannt wrote: Hello, when I'm set $dnsbl_checks_only to 1 and I would like to add the policyd-weight headers to the mail with $ADD_X_HEADER= 1; policyd ignores the add_x_header statement. Only when I set $dnsbl_checks_only to 0, policyd will add the headers to the mail. bug or feature? Rather a bug. Fixed in 0.1.14 beta-7 (non official) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Thu, Aug 02, 2007 at 01:00:04PM -0700, Paul B. Henson wrote: On Wed, 1 Aug 2007, Robert Felber wrote: I.e: 10831 had 8 SMTPD clients. If 1 of those is served, all others must wait. So - the 8th one has to wait a long time - but not always, depending on whether all other smtpd are active and how long the requests take. I guess I misunderstood the policyd-weight architecture? I thought each child process served one and only one request at a time, which is why you recommended that the configured number of children match the configured number of postfix processes? How does one child end up with multiple established connections? You might want to try out the current devel (Fri Aug 03 09:02:20 CEST 2007), I have updated it to close connections to smtpd clients in order to avoid too many established connections to a single policyd-weight child. You need to set $TRY_BALANCE = 1; in your policyd-weight.conf Warning: this requires testing and is only a temporarily workaround -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Serious probelm with policyd-weight
On Thu, Aug 02, 2007 at 11:33:04AM +0200, Robert Felber wrote: On Thu, Aug 02, 2007 at 09:11:05AM +0100, Thomas Krieger wrote: The master die()d because Sys::Syslog throw a die() when the master attempted to use syslog(). This appears to be a race condition when syslog is not available for log-message submission. I am afraid I cannot resolve this cleanly. Postgrey had the same problem, simple fix. Upgrade Sys::Syslog. http://lists.ee.ethz.ch/postgrey/msg01815.html .. and policyd-weight needs to add nofatal to openlog call too. Well the Sys::Syslog in Debian Sarge is quit old. It's perl 5.8.4 and the Sys::Syslog version is 0.05 I think. I will update to the newest version of Sys::Syslog. Robert will you change the openlog accordingly? nofatal cannot be made default as per configuration (well, it can but this would add perl exception-code to the standard config). But Henrik sent me the postgrey way of avoiding this. At least this way the master doesn't die - but the information which should be logged won't be logged. I think I will try an eval loop and if that fails after some runs we log to the console. Don't know whether that is cleanly portable, though. I have updated policyd-weight devel to work around syslog absence. In cases of syslog absence it logs an polw-emergency.log to $LOCKPATH I haven't used /var/log stuff here because policyd-weight children are not root, and the file shall only be present if syslog absent. It is up to the administrator the check for this file and remove/cycle it. It is not expected to exist or grow continuously. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: policyd-weight RPM Packages
On Wed, Jul 25, 2007 at 07:41:17PM +0200, Thomas Krieger wrote: Hello, is anyone out there maintaining rpm packages of policyd-weight for Fedora and RedHat Enterprise Linux? If not please let me know. I can do this. Hello Tom, I haven't seen RedHat RPMs yet. If you can do this, thanks! P.s.: sorry for not responding to your earlier private mail. I've forgotten it. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Fri, Jul 20, 2007 at 12:22:48PM -0700, Paul B. Henson wrote:
On Wed, 18 Jul 2007, Robert Felber wrote:
65536 appears to be problematic. I have fixed this now in 0.1.14.6
Was it a simple fix? Any chance of a small patch I can apply to my running
0.1.14.5?
--- policyd-weight Thu May 10 12:01:41 2007
+++ policyd-weight-0.1.14.5-p1 Sat Jul 21 09:21:20 2007
@@ -2953,7 +2953,7 @@
my $query = shift(@bu);
my $rtype = shift(@bu);
-my $oid = 1 + int(rand(65536));
+my $oid = 1 + int(rand(65535));
$rtype = 'A' unless ($rtype $RTYPES{$rtype});
--
Robert Felber (PGP: 896CF30B)
Munich, Germany
Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Thu, Jul 19, 2007 at 07:49:55AM +0200, Robert Felber wrote: On Wed, Jul 18, 2007 at 04:16:30PM -0700, Paul B. Henson wrote: On Mon, 16 Jul 2007, Paul B. Henson wrote: I have upgraded to 0.1.14.5 today, will make the suggested configuration changes, and see what happens. After upgrading to 0.1.14.5, I am continuing to receive timeout errors: 61: postfix/smtpd: warning: problem talking to server 127.0.0.1:12525: Connection timed out 35: postfix/smtpd: warning: timeout on 127.0.0.1:12525 while reading input attribute name I now require more verbose debugging of the system environment at such errors. Can you write a logwrapper which calls lsof, netstat and ps to see how much process are up, which state, for how long, which queue-fills, etc. Also, I'm still getting weird rbl_lookup errors, but now with more detail: 3: policyd-weight: rbl_lookup: unknown error: out:145.20.15.204.sbl-xbl.spamhaus.org, in:145.20.15.204.sbl-xbl.spamhaus.org, out-id:65536, in-id:0 3: policyd-weight: rbl_lookup: unknown error: out:5.200.113.208.list.dsbl.org, in:5.200.113.208.list.dsbl.org, out-id:65536, in-id:0 3: policyd-weight: rbl_lookup: unknown error: out:65.135.144.24.sbl-xbl.spamhaus.org, in:65.135.144.24.sbl-xbl.spamhaus.org, out-id:65536, in-id:0 3: policyd-weight: rbl_lookup: unknown error: out:mx185.technologygrouptwentytwo.com.abuse.rfc-ignorant.org, in:mx185.technologygrouptwentytwo.com.abuse.rfc-ignorant.org, out-id:65536, in-id:0 Which DNS servers are in between? Polw sent 65536 as packet-id out, and got 0 in return (which I see the first time). 65536 appears to be problematic. I have fixed this now in 0.1.14.6 Although I assume your timeouts will continue, thus I am still interested in system states at such timeouts. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Wed, Jul 18, 2007 at 08:04:12AM +0200, Gerald Holl wrote: Robert Felber wrote: On Thu, Jul 12, 2007 at 02:20:41PM +0200, Gerald Holl wrote: Robert Felber wrote: This could happen if all policyd-weight processes are hogged up. Should be logged with MAX_PROX NN reached. How many policyd-weight childs do you have at such moments? Alternatively, what is your kernel setting for somaxconn? If it is 128 then you should increase it to 1024 or some higher value (this is a general recommendation for any server). This isn't being logged by policyd-weight, as this cannot be detected by polw. Hello Robert, The net.core.somaxconn is set to 128. I'll try to increase it to 1024 any report any changes. In thread [EMAIL PROTECTED] you recommend the 0.1.14.5 from policyd-weight.org. Is the timeout bug fixed in that version? I have re-read Jul 11 18:06:23 jimbo postfix/smtpd[31532]: warning: timeout on 127.0.0.1:12525 while reading input attribute name Jul 11 18:06:23 jimbo postfix/smtpd[31532]: warning: problem talking to server 127.0.0.1:12525: Connection timed out Jul 11 18:06:23 jimbo postfix/smtpd[31532]: NOQUEUE: reject: RCPT from unknown[61.142.35.204]: 451 4.3.5 Server configuration problem; from=[EMAIL PROTECTED] to=[EMAIL PROTECTED] proto=ESMTP helo=204.35.142.61.broad.dg.gd.dynamic.163data.com.cn And the connection timed out suggests rather, that this is a out of socket problem but it could also be, that there is another bug in 0.1.14 which might result in timeouts. Although I increased net.core.somaxconn to 1024 I got a timeout this morning: Jul 18 05:14:57 postfix/smtpd[317]: warning: timeout on 127.0.0.1:12525 while reading input attribute name Jul 18 05:14:57 postfix/smtpd[317]: warning: problem talking to server 127.0.0.1:12525: Connection timed out Assuming that you are using 0.1.14: - Update to 0.1.14.5 (or even 0.1.14.6, both should be ok) - check whether policyd-weight's $MAX_PROC reflect postfix' smtpd MAX_PROC Probably the notification about exceeded policyd-weight MAX_PROCs was logrotated out. This notification is only announced once. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Wed, Jul 18, 2007 at 04:16:30PM -0700, Paul B. Henson wrote: On Mon, 16 Jul 2007, Paul B. Henson wrote: I have upgraded to 0.1.14.5 today, will make the suggested configuration changes, and see what happens. After upgrading to 0.1.14.5, I am continuing to receive timeout errors: 61: postfix/smtpd: warning: problem talking to server 127.0.0.1:12525: Connection timed out 35: postfix/smtpd: warning: timeout on 127.0.0.1:12525 while reading input attribute name I now require more verbose debugging of the system environment at such errors. Can you write a logwrapper which calls lsof, netstat and ps to see how much process are up, which state, for how long, which queue-fills, etc. Also, I'm still getting weird rbl_lookup errors, but now with more detail: 3: policyd-weight: rbl_lookup: unknown error: out:145.20.15.204.sbl-xbl.spamhaus.org, in:145.20.15.204.sbl-xbl.spamhaus.org, out-id:65536, in-id:0 3: policyd-weight: rbl_lookup: unknown error: out:5.200.113.208.list.dsbl.org, in:5.200.113.208.list.dsbl.org, out-id:65536, in-id:0 3: policyd-weight: rbl_lookup: unknown error: out:65.135.144.24.sbl-xbl.spamhaus.org, in:65.135.144.24.sbl-xbl.spamhaus.org, out-id:65536, in-id:0 3: policyd-weight: rbl_lookup: unknown error: out:mx185.technologygrouptwentytwo.com.abuse.rfc-ignorant.org, in:mx185.technologygrouptwentytwo.com.abuse.rfc-ignorant.org, out-id:65536, in-id:0 Which DNS servers are in between? Polw sent 65536 as packet-id out, and got 0 in return (which I see the first time). -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Fri, Jul 13, 2007 at 07:17:18PM -0700, Paul B. Henson wrote: On Wed, 11 Jul 2007, Robert Felber wrote: Which version? 0.1.14.5, it looks like. Any warnings, error messages in advance? The only error messages I recall seeing are: policyd-weight[812]: rbl_lookup: unknown error This error should happen only with versions prior to 0.1.14.2. There was a DNS nonce bug which sometimes was 0 and thus treated wrong - leading to above error. policyd-weight[9910]: warning: ignoring garbage: 1 hm. They don't seem to correlate with the timeouts... The first probably not, but the second I am not certain. Would be interesting what caused this. I currently cannot imagine a scenario which would send 1 to policyd-weight. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Fri, Jul 13, 2007 at 07:44:08PM -0700, Paul B. Henson wrote: On Wed, 11 Jul 2007, Robert Felber wrote: This could happen if all policyd-weight processes are hogged up. Should be logged with MAX_PROX NN reached. How many policyd-weight childs do you have at such moments? There are no instances of that message in my logs. I currently have the maximum number of processes set to 100. Alternatively, what is your kernel setting for somaxconn? If it is 128 then you should increase it to 1024 or some higher value (this is a general recommendation for any server). This isn't being logged by policyd-weight, as this cannot be detected by polw. somaxconn is currently the default, which I believe is 128. You should really increase this. I will update the setup howto as well. This level has caused many problems in the past. I currently have the maximum number of postfix smtp processes set to 300, so the theory here is that all 100 policyd-weight processes are busy, 128 postfix processes are attempting to connect and sitting in the listen queue, and then the 129th+ processes get connection timed out? Yes because policyd-weight childrens all are in a accept state. If the kernel doesnt provide a socket-descriptor due to somaxconn issues the policyd-weight returns to accept() on its listen socket. At some time postfix will timeout. But that doesn't make sense, because shouldn't policyd-weight log a notification when it tried to start the 101st process which would have exceeded the maximum? Yes. How many policyd-weight instances are up at this time? The only way the queue backlog should exceed 128 is if that many connections are made without policyd-weight doing an accept? Or not being able to do a sane accept(). -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Thu, Jul 12, 2007 at 02:20:41PM +0200, Gerald Holl wrote: Robert Felber wrote: This could happen if all policyd-weight processes are hogged up. Should be logged with MAX_PROX NN reached. How many policyd-weight childs do you have at such moments? Alternatively, what is your kernel setting for somaxconn? If it is 128 then you should increase it to 1024 or some higher value (this is a general recommendation for any server). This isn't being logged by policyd-weight, as this cannot be detected by polw. Hello Robert, The net.core.somaxconn is set to 128. I'll try to increase it to 1024 any report any changes. In thread [EMAIL PROTECTED] you recommend the 0.1.14.5 from policyd-weight.org. Is the timeout bug fixed in that version? I have re-read Jul 11 18:06:23 jimbo postfix/smtpd[31532]: warning: timeout on 127.0.0.1:12525 while reading input attribute name Jul 11 18:06:23 jimbo postfix/smtpd[31532]: warning: problem talking to server 127.0.0.1:12525: Connection timed out Jul 11 18:06:23 jimbo postfix/smtpd[31532]: NOQUEUE: reject: RCPT from unknown[61.142.35.204]: 451 4.3.5 Server configuration problem; from=[EMAIL PROTECTED] to=[EMAIL PROTECTED] proto=ESMTP helo=204.35.142.61.broad.dg.gd.dynamic.163data.com.cn And the connection timed out suggests rather, that this is a out of socket problem but it could also be, that there is another bug in 0.1.14 which might result in timeouts. The timeout-bug which I mentioned was introduced in 0.1.14.2 and corrected at several places along 0.1.14.4 and 0.1.14.5 For further developing and bug-finding I rather require reports of the latest beta release as there were many changes/fixes from 0.1.14 to 0.1.14.5 -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Wed, Jul 11, 2007 at 12:42:38PM -0700, Paul B. Henson wrote: I'm afraid I don't read your language :(, but I believe I'm getting the same problem you are: postfix/smtpd: warning: problem talking to server 127.0.0.1:12525: Connection timed out postfix/smtpd: warning: timeout on 127.0.0.1:12525 while reading input attribute name The rejection is a temporary failure, the same we give for gray listing, so I haven't been overly concerned. Usually there are only a couple of dozen instances of this problem per day, and I write it off as collateral damage. Every now and again, the server seems to completely go out to lunch and there are thousands if not tens of thousands of instances of this problem. On those days my motivation to fix it increases, but I still haven't followed up on it. Which version? Any warnings, error messages in advance? -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Wed, Jul 11, 2007 at 06:52:58PM -0700, Paul B. Henson wrote: There are two different errors: Connection timed out, which seems to be an error at the actual TCP level where the connection is never established, This could happen if all policyd-weight processes are hogged up. Should be logged with MAX_PROX NN reached. How many policyd-weight childs do you have at such moments? Alternatively, what is your kernel setting for somaxconn? If it is 128 then you should increase it to 1024 or some higher value (this is a general recommendation for any server). This isn't being logged by policyd-weight, as this cannot be detected by polw. If I understand correctly, this parameter would only apply for a policy server named policy which is being spawned out of master.cf, not the case for policyd-weight. I don't think this parameter would have any effect on a policyd-weight configuration. I did find a different parameter: smtpd_policy_service_timeout (default: 100s) The time limit for connecting to, writing to or receiving from a delegated SMTPD policy server. Possibly increasing this might reduce the number of timeouts when the server doesn't respond quickly enough, but 100 seconds seems like an awfully long time. Ideally policyd-weight shouldn't take nearly that long to do its job. I wouldn't really increase this. 100s are quite a long time and anything taking longer than that should be considered a bug in policyd-weight. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: policyd-weight crashes
On Wed, Jun 13, 2007 at 12:08:18PM +0200, Robert Felber wrote: Currently I can think only of some situations where a crash without logging could happen: - Signals like KILL, INT, TERM, QUIT, PIPE (and other sigs which cause termination) are not logged. Try http://www.policyd-weight.org/policyd-weight-devel This is still an unofficial version (0.1.14 beta-6) I have updated http://www.policyd-weight.org/policyd-weight-devel The SIG handlers provide now a perl backtrace. Coredumps still are not possible due to changing UID from root to policyd user. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Strange scoring with 0.1.14 beta-5
On Fri, May 18, 2007 at 10:18:41PM +0200, [EMAIL PROTECTED] wrote: @dnsbl_score = ( 'sa-hil.habeas.com', 8.00, 0,'HIL-HABEAS', 'sa-hul.habeas.com', -1.00, 0,'HUL-HABEAS', 'sa-trusted.bondedsender.org', -4.25, 0, 'TRUSTED-BONDESENDER', 'sa-other.bondedsender.org', -4.25, 0, 'OTHER-BONDESENDER', 'wl.trusted-forwarder.org', -0.50, 0,'T-FWL-DNSWL', 'list.dnswl.org',-0.50, 0,'DNSWL', 'white.dnsbl.securityplanet.nl', -0.70, 0, 'SECURITYPLANETWL', 'exemptions.ahbl.org', -1.00, 0,'EXEMPTIONS-AHBL', 'ch.countries.nerd.dk', -1.00, 0,'NERD-CH', 'se.countries.nerd.dk', -1.00, 0,'NERD-SE', 'us.countries.nerd.dk', 2.044, 0,'NERD-US', This has a hit. And - the client meets ~ 4 conditions for CLIENT_NOT_MX/A_FROM_DOMAIN CLIENT/24_NOT_MX/A_FROM_DOMAIN From the code: ## client == MX/A FROM domain # if( ($mx_ok != 1) ( ($do_client_from_check) ($dnsbl_hits 0) ) ) $mx_ok wasn't 1 do_client from check was 1 because helo (domains) didn't appear to be responsible for sender domain (Arguments and sender MX results) $dnsbl_hits was greater 0 Subnets of the client didn't match sender A/MX subnets Solution, lower the score for us.countries.nerd.dk With 1.044 the client passes here with -0.732 -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Suggested whitelist entries?
On Wed, Apr 04, 2007 at 10:49:30AM -0400, Francisco Reyes wrote: Would it be ok to set $REJECTLEVEL = 2; instead of 1? Yes. Others even use values around 4 to 5. Note: *simple* forged szenarios as helo microsoft.com mail from: [EMAIL PROTECTED] are possible with REJECTLEVEL greater than 1. (viruses like sobig.f do it this way) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: An idea for discussion
On Sat, Mar 31, 2007 at 07:47:50AM +0200, Thomas Krieger wrote: I'm using policyd-weight and it does a great job for me. A few days ago I had a discussion with people using our mailserver. They mentioned that they want to have a possibility to decide, whether polidyd-weight should decline mails or mark mails as spam. The main argument was that for firms it is very important to receive all the mails regardless they are spam or not. My idea was that policyd-weight should either decline the email after the header checks or mark it as spam with an entry in the mail header like spamassassin does. The best solution would be to switch the behaviour due to the mailbox the mail goes to. Perhaps the lookup can be done against an MySQL database, an LDAP or another data source. As policyd-weight has no user based configuration this is not possible. One possible solution: Useres who want flagged mail should opt-out for policyd-weight. This can be done with postfix restrictions classes + SQL. This requires an after queue filter which has a user-based configuration an can flag and deliver on user's demand. If only domain based decisions are required then you may look at http://blog.waja.info/2006/09/20/running-different-policyd-weight-instances/ You could create 2 instances, one for blocking, one for tagging and maintain a restrcition class for which rcpt domain shall use which policyd-weight instance (you can issue -f /path/file to use a special configfile). -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Suggestion and request for sample configs
On Tue, Feb 20, 2007 at 10:42:26PM +0100, Robert Felber wrote: My verifications for RL 1 are: the home dialup link (DUL listed, dialup-triggering hostname, but using DynDNS, and sometimes the current hostname as HELO) Ford (mailwatch) servers which are broken and inconsistent GM servers dito Sorbs Virus characteristics Ouch, I meant Sober characteristics. This appears to be a freudian slip. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: version update: version 0.1.14 beta-3
On Tue, Feb 27, 2007 at 08:57:40AM -0500, Francisco Reyes wrote: Robert Felber writes: The FreeBSD port has been committed and should appear soon on the CVS mirrors. http://www.freebsd.org/cgi/query-pr.cgi?pr=109592 Did you have a chance to change the rc.d script? Yes, as you can see on the link above and in changes.txt. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Suggestion and request for sample configs
On Tue, Feb 20, 2007 at 12:47:45PM -0500, Francisco Reyes wrote: From the default I would change sbl-xbl.spamhaus.org to zen.spamhaus.org. Until a mechanism to distinguish between the various zen results has been implemented I'd suggest to use following compination to make use of 'zen' - 'dynablock.njabl.org',3.25, 0,'DYN_NJABL', + 'pbl.spamhaus.org', 3.25, 0,'DYN_SPAMHAUS_PBL', you should leave the sbl-xbl entry intact. If you want to use zen directly in postfix you should know that you have to configure it appropriate there too, otherwise you start to block users from dialups. If you don't want to block dialups (for instance if you want to allow dyndns users) but still want to use spamhaus as a 0|1 decision in postfix then you should use reject_rbl_client zen.spamhaus.org=127.0.0.2 reject_rbl_client zen.spamhaus.org=127.0.0.4 reject_rbl_client zen.spamhaus.org=127.0.0.5 reject_rbl_client zen.spamhaus.org=127.0.0.6 reject_rbl_client zen.spamhaus.org=127.0.0.7 reject_rbl_client zen.spamhaus.org=127.0.0.8 Which exlcudes pbl (+dynablock) Or - to keep it simple: reject_rbl_client sbl-xbl.spamhaus.org -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Suggestion and request for sample configs
On Tue, Feb 20, 2007 at 10:01:30PM -0500, Francisco Reyes wrote: Performance wise would it make sense to leave in postfix any RBLS that one uses as a single point of rejection? Depends. With a DNS cache which caches negative responses too, yes. In other words if I use zen.spamhaus.org to block any email listed in it, and I set the level high enough in policyd-weight to reject mails by been in that one RBL, would it be faster to have the RBL in postfix? I cannot make a statement because I haven't compared it, but I'd guess: yes. Also, you should ask yourself whether you would use dynablock.njabl.org for outright blocking (this is a DUL list (dial up)), and included into zen. Basically you give the decision to accept DynDNS MX users away to their ISP. Along the same lines.. does policyd-weight stops the second it sees that the email RBL score hit the MAXDNSBLSCORE or does it continue to do additional tests? $MAXDNSBLHITS = 2; # If Client IP is listed in MORE # DNSBLS than this var, it gets # REJECTed immediately $MAXDNSBLSCORE = 8; # alternatively, if the score of # DNSBLs is ABOVE this # level, reject immediately immediately is what you are looking for, yes. Which makes me wonder.. are the RBL tests done first or after the other tests? In the order as they appear in the logging string. RBLs are done first, RHSBL are done last, and only if the REJECTLEVEL has not been exceeded. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: FROM/MX_MATCHES_NOT_HELO
On Tue, Feb 20, 2007 at 11:55:02PM -0500, Francisco Reyes wrote: Looking at the man page don't see a definition for: FROM/MX_MATCHES_NOT_HELO The header had: FROM/MX_MATCHES_NOT_HELO(DOMAIN)=1.938 client=87.118.194.148 helo=host-148.universumbit.87.118.194.0.0xff00.macomnet.net [EMAIL PROTECTED] I see how the from doesn't match the helo domain, but what is the MX part of that test? This is from a spam that got through Also don't see it in the policyd-weight.conf Is this some type of composite score made from other scores? The score setting for this is $from_match_regex_verified_helo This can have following log-entries: FROM/MX_MATCHES_HELO(DOMAIN) FROM/MX_MATCHES_NOT_HELO(DOMAIN) FROM/MX_MATCHES_UNVR_HELO(DOMAIN) FROM/MX_MATCHES_NOT_UNVR_HELO(DOMAIN) MAIL_SEEMS_FORGED FROM_NOT_FAILED_HELO(DOMAIN) It does check: a) whether the sender (domain) matches stringwise the HELO (domain) by comparing .example1. vs .example2. b) whether the MX for the sender domain matches stringwise the HELO domain a and b in conjunction with the results of HELO/SENDER vs IP comparisons. The score is influenced by the scores of RBL results / 4 plus $bogus_mx_penalty * $bogus_mx_penalty You do not want to tweak with this (unless you want to lower its score). This is not documented yet as this is for suicide people only and might change anyway due to the general beta status. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Request for MTA Help Site
Hello, as I have some google strings for Mail appeared to be SPAM or forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settings in my logs I thought I setup a site which gives some basic SMTP explanations to the admins. The URL will be http://www.policyd-weight.org/mtahelp.html I am asking for help to either a) complete the page or b) give additional info or links for configuring the major MTAs properly. The page should give intuitive help for the qmail, exim, sendmail, exchange and of course postfix admins in terms of - what obviously caused this message, which program is responsible for this - how to setup a correct HELO (MTA, DNS (requirements)) - how to do in the lack of a reverse DNS or being a dynamic host (speaking of DynDNS) - what are bogus MX records, how to verify for their own HELO/SENDER domain, and so on - what to do in case of RBL listings (contacting the postmaster who uses policyd-weight) - ... In short, the basic principles for each MTA. I know there was some site around which already explained MTA principles but from my memory that site was rather non-intuitive and lost in unstructured detail and also not complete. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Request for MTA Help Site
On Wed, Jan 31, 2007 at 01:46:21PM -0600, Paul Schmehl wrote: The DNS stuff is generic, so perhaps it should have its own section. Basics are: An entry in the zone file that looks like this: @ IN MX 10 hostname.domain.tld. An A record in the zone file that looks like this: hostname IN A IP.AD.DR.ESS A PTR (reverse) record in the reverse file that looks like this: IP.AD.DR.ESS IN PTR hostname.domain.tld To test it: dig -x IP.AD.DR.ESS must return hostname.domain.tld E.g. ;; ANSWER SECTION: ESS.DR.AD.IP.in-addr.arpa. 21600 IN PTR hostname.domain.tld. dig -t MX domain.tld must return hostname.domain.tld E.g. ;; ANSWER SECTION: domain.tld. 28800 IN MX 10 hostname.domain.tld. Yes, someone who takes the challenge to write and update a pretty, down to the point, non-overloaded shiny site would be appreciated. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Request for MTA Help Site
On Wed, Jan 31, 2007 at 02:46:03PM -0500, Larry Ludwig wrote: Oh we also use this page to help track any stats on bocked emails. If a bounced email gets clicked on we know about it as this page is not linked anywhere (until now :-) ) I see. Well, the users who look for that response on google usually end up on the policyd-weight site and look at some config file or the script itself. :-/ -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Future and the end of RBLs
Hello, Most may have noticed it, dynablock.njabl.org closes. Users are urged to use spamhaus' zen list which includes pbl. This raises following issues: - The entropy of reputative RBLs shrinks. Which is generally bad. Even though spamhaus is a very good RBL it is not good to rely only on one good RBL. In suche a case policyd-weight may become obsolete. - We do not parse returned IP addresses of RBL lookups as of now. Neither do we have a simple configuration sheme for such a method. Thus it makes it harder to detect dialup links now. I do expect more good RBLs to disappear in favor of spamhaus. I won't follow this route. One reason may be that spamhaus is not free. Just because nearly everyone uses it for free _now_ (and I guess 90% don't respect their usage policies) doesn't mean, that they do not start to enforce their usage policy later (when everyone got addicted). I don't want to discredit spamhaus, I just mention my pragmatic doubts. I'll implement Henrik's p0f idea as soon as my employer and family allows it. Personally I think, that the long term route must lead to an interaction with an after queue content filter and a removal of RBL lookups et all. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: [Patch] p0f and selective greylisting
On Wed, Jan 03, 2007 at 04:13:03PM +0200, Henrik Krohns wrote: Hi, I whipped up a patch for policyd-weight-devel. It adds p0f scoring support and greylisting (to be exact, user defined postfix action) by some rules. Thanks. Looks very interesting. I will dive in. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Happy new year.
Hello, at first I wish everyone a Merry X-Mas, also those who celebrate X-Mas around 6/7th January. Also I wish everyone a happy new year and the best for their families. I'll have holidays from today to 8th January, so I'll have little to no time to check for e-mails. I hope your and our systems survive and won't suffer from bugs :-) See you next year. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: polw falling behind?
On Sat, Dec 02, 2006 at 07:17:03AM -0600, Len Conrad wrote: What are the average delays from a client connecting to the smtpd and an answer from policyd-weight? Where do I find that info? This can only be guessed by lines like: Dec 4 00:07:40 fpsvr1z150 postfix/smtpd[31885]: connect from unknown[85.137.11.234] Dec 4 00:07:44 fpsvr1z150 postfix/policyd-weight[65787]: weighted check: IN_SBL_XBL_SPAMHAUS=4.35 NOT_IN_BL_NJABL=-1.5 NOT_IN_SPAMCOP=-1.5 CL_IP_NE_HELO=5.85 RESOLVED_IP_IS_NOT_HELO=1.5 (check from: .01-flash-web-templates. - helo: .fran-r0pedpv5aw. - helo-domain: .fran-r0pedpv5aw.) FROM_NOT_FAILED_HELO(DOMAIN)=7.35 client=85.137.11.234 helo=fran-r0pedpv5aw [EMAIL PROTECTED] [EMAIL PROTECTED], rate: 16.05 In above case it took 4 seconds to answer. What's your kernel setting for so_maxconn (linux: sysctl net.core.somaxconn) kern.ipc.somaxconn: 128 You should increase this value to 512 or 1024. From your Recv-Q entries it seems to run ok. It seems that your machine runs out of tcp slots. Thus increase kern.ipc.somaxconn. As of the moment policyd-weight cannot do asynchronous DNS lookups. Thus one policyd-weight instance cannot handle 2 smtpd clients at once but must process them serial. Doing that async with one process is a very hard task and I fear that it will impossible with the current design. However, if a policyd-weight instance is busy then a new smtpd client will cause policyd-weight to create a new instance, including required sockets. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: [PATCH] rbl_lookup
On Sun, Dec 03, 2006 at 07:47:02PM +0100, Francis Galiegue wrote: As a bonus, it gives information in the logs such as these on lookups: Dec 3 19:40:22 ns35774 postfix/policyd-weight[10602]: Checking 185.234.168.208 TXT against dynablock.njabl.org Dec 3 19:40:22 ns35774 postfix/policyd-weight[10602]: Checking 185.234.168.208 TXT against zen.spamhaus.org Dec 3 19:40:22 ns35774 postfix/policyd-weight[10602]: Checking 185.234.168.208 TXT against bl.spamcop.net Dec 3 19:40:22 ns35774 postfix/policyd-weight[10602]: Checking 185.234.168.208 TXT against dnsbl.njabl.org Dec 3 19:40:22 ns35774 postfix/policyd-weight[10602]: Checking 185.234.168.208 TXT against list.dsbl.org Currently most are happy if such entries are only visible in debug mode to do not flood the logs. As such we log the results rather in the response. Thanks for the patch. I will look into it and see how we can make use of it. Your second patch wasn't attached. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: polw falling behind?
On Fri, Dec 01, 2006 at 12:23:08PM -0600, Len Conrad wrote: For today: mx1# egrep -ic ^... .. 10.*problem talking to server 127.0.0.1:12525 /var/log/maillog 3849 ... stop/start policyd-weight, then mx1# egrep -ic ^... .. 11.*problem talking to server 127.0.0.1:12525 /var/log/maillog 0 What's going on? No one can tell by that line of log. How many smtpd processes exist at such times, how many policyd-weight process exist at such times? What's you operating system? What are the average delays from a client connecting to the smtpd and an answer from policyd-weight? Where there warning:s or err:s before? What's your kernel setting for so_maxconn (linux: sysctl net.core.somaxconn) whats the output of bsd:netstat -na | grep '.12525' linux: netstat -pna | grep '.12525' at such times. If you see in the netstat output above a process with a huge Recv-Q like: PROTO Recv-Q Send-Q Local Address Foreign AddressState PID/Program name tcp1234567 0 127.0.0.1.12525127.0.0.1.55710 ESTABLISHED 24216/policyd-weight then I would appreciate an strace -fp $PID output where $PID is the PID of the process with the huge Recv-Q Please cosnider that there is alaways a different approach when using different OSes, so please tell allways your OS and kernel version (uname -a). The Perl version is handy, too! P.s.: I am busy and not available until monday. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: problems starting policyd-weight from master.cf
On Tue, Nov 28, 2006 at 12:34:46PM +0100, Peter Fischer wrote: Hi list! If I start policyd-weight from master.cf like this: # policyd-weight perl policy daemon for postfix 127.0.0.1:2527 inetn n n - 1 spawn ^^^ no good Nov 28 01:59:23 hostname postfix/smtpd[27455]: warning: timeout on 127.0.0.1:2527 while reading input attribute name after some 1,5 hours of policyd running. Policyd-weight attempts to guarantee to answer within 100s (smtpd_policy_service_timeout) for *one* request. If 3 requests are in queue because you limit to *one* spawn process then you will exceed that 100s sooner or later (DNS is sometimes slow and a pain). As such, remove that '1'. ( When I ran policyd-weight from master.cf like this before # policyd-weight perl policy daemon for postfix 127.0.0.1:2527 inetn n n - - spawn and after the first concurent smtpds: hostname ~ # ps -ef| grep pol polw 17466 1 0 00:43 ?00:00:00 policyd-weight (cache) postfix 24296 17399 0 01:18 ?00:00:00 spawn -n 127.0.0.1:2527 -t inet user=polw argv=/usr/bin/perl /usr/lib/postfix/policyd-weight postfix 24971 17399 0 01:27 ?00:00:00 spawn -n 127.0.0.1:2527 -t inet user=polw argv=/usr/bin/perl /usr/lib/postfix/policyd-weight polw 24972 24971 0 01:27 ?00:00:00 policyd-weight (master) polw 25698 24296 1 01:36 ?00:00:00 policyd-weight (master) that is why I restricted policyd-weight to 1 maxproc in master.cf. ) Bad idea :-) (This may work with fast policy services such as greylisters, though). -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: problems starting policyd-weight from master.cf
On Tue, Nov 28, 2006 at 07:10:36PM +0100, Peter Fischer wrote: Hi Robert Phillip! Robert Felber schrieb: [...] # policyd-weight perl policy daemon for postfix 127.0.0.1:2527 inetn n n - 1 spawn ^^^ no good [...] So, multiple policyd-weight (master) processes are not a bad thing per se, and you tell me there are no such things like race conditions, locking issues while accessing the (cache) process, etc? We avoid races by using lock directories. The cache is (intentionally) a single threaded, low profile process. (child) processes are only spawned in daemon mode. Actually that you see (master) processes in master.cf mode is an unfortunate naming accident. Will be changed. (p.s. the list has intentionally no reply-to: set. I've tried to find a HOWTO to convience thunderbird to handle mailinglists appropriate (i.e. to not send off-list but on-list), but I failed, if you find something, please let me know.) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: timeout while reading input attribute name
On Tue, Nov 14, 2006 at 01:50:24PM +0100, Urban Hillebrand wrote: Hello list, we have an intermittent problem here. We saw several times that postfix starts (temporarily) bouncing mails with 450 server configuration error. All we see in our logs is Nov 14 00:22:54 postfix/smtpd[5456]: connect from pool-68-237-243-52.ny325.east.verizon.net[68.237.243.52] Nov 14 00:24:37 postfix/smtpd[5456]: warning: timeout on 127.0.0.1:12525 while reading input attribute name Nov 14 00:24:37 postfix/smtpd[5456]: warning: problem talking to server 127.0.0.1:12525: Connection timed out Nov 14 00:26:18 postfix/smtpd[5456]: NOQUEUE: reject: RCPT from pool-68-237-243-52.ny325.east.verizon.net[68.237.243.52]: 450 Server configuration problem; from=[EMAIL PROTECTED] to=[EMAIL PROTECTED] proto=ESMTP helo=friend Until now, we suspected either linux limits (number of filehandles etc.), or a too small number of $MAX_PROC for policyd-weight - without having any evidence. This changed today. We experienced severe performance problems with the DNS server, which acts as forwarder for our caching only bind9 on our machine. One sideeffect of those performance problems was exactly the error described above - until those issues were resolved, all mails were bounced with the 450 error. It would be my understanding that in case of DNS errors policyd-weight should return DUNNO after $MAXDNSERR queries, right? Right. Obviously the alarm call doesn't interrupt the recv() call appropriate. Any ideas on this? Anything we could do to debug this? You could set $DEBUG = 1; In your logs you should then see a line like: warning: rbl_lookup: timeout: nask1.2.3.4 or similar If my timeout implentation does not work on your box then this should be the last line of that policyd-weight PID. It should then hang around forever, without logging. A quickfix would be to say $USE_NET_DNS = 1; in the config file. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
[EMAIL PROTECTED]: Re: timeout while reading input attribute name]
On Tue, Nov 14, 2006 at 03:44:30PM +0100, Urban Hillebrand wrote: A quickfix would be to say $USE_NET_DNS = 1; in the config file. I will try that, thanks. Is the assumption that $MAX_PROC should match maximum number of smtpd processes correct? It's prefered. But it should be possible with less, too. Actually policyd-weight should never ever spawn as much instances as smtps (at least not in daemon mode). If that happens then something is broken. Can you live with a pre-beta? I have a 0.1.14 pre1-beta2 here on which I fix the timeout implementation, as far as I can see I am done with it. This pre beta also tries to make better use of the cache and has a Mac OS X fix. However, I'd release it as 0.1.14 beta-2 if that would fix your issue, too. (Note: you should remove the $USE_NET_DNS entry from you config in order to test) It includes following changes: cache efficiency - store only IP of too much DNSBL listed hosts (tested) store only IP if client matches not helo, or is a dyn client store ip-[EMAIL PROTECTED] in all other cases this saves from dictionary attacks while still avoiding cache poisoning Mac OS X/SuSe- Privilege dropping sanitized, shouldn't run in taint (tested) Mode anymore SuSe - timeout implementations not reliable, fix attempt (testing) (should not affect $USE_NET_DNS users (perl 5.6)) RBLs - Avoid RBLs which cause too much errors for a certain amount of subsequent errors. Currently the default for testing is $BL_ERROR_SKIP = 2 - skip RBLs which had 2 errors, $BL_SKIP_RELEASE = 10 - skip them for that many times. The value of the RBL's good score is applied in skip cases. Version can be downloaded from http://www.policyd-weight.org/policyd-weight-0.1.14-pre1-beta2 MD5 (policyd-weight-0.1.14-pre1-beta2) = 5de95929eb831b2c00fa0649d23f8333 -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Newest -devel command time limit exceeded.
On Thu, Nov 02, 2006 at 04:41:20AM -0500, Justin Piszcz wrote: Nov 1 17:50:29 box postfix/spawn[24371]: warning: /usr/bin/perl: process id 24373: command time limit exceeded Sure this is from policyd-weight? AFAIK it runs as spawn mode -- at least that's what it looks like. Yes. You should switch to daemon mode. However, the warning ought to be only a warning. unix:private/rbl_policy,reject_unverified_sender, To avoid such warnings you may set rbl_policy_time_limit = 3600 outside of smtpd_recipient_restrictions. This tells postfix to keep that spawned process alive for 3600 seconds. The default value postfix uses is 1000. If an smtpd lives longer than 1000 seconds, then spawn will exit the spawned policy process for resource reasons. After all, this should have nothing to do with upgrading policyd-weight versions. Rather a unfortunate timing constellation of connecting smtp clients which causes smtpds to live longer than 1000 seconds. (see also http://www.postfix.org/SMTPD_POLICY_README.html#client_config) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Problem starting daemon on Mac OS X 10.4
On Wed, Nov 01, 2006 at 04:43:01PM +0200, Vladimir Vladimirov wrote:
Hi everyone
I've installing policyd-weight on Mac OS X 10.4 and discovered
problem. Here is debug:
policyd-weight version: 0.1.14 beta, CacheVer: 3
System: Darwin pagraphics 8.8.0 Darwin Kernel Version 8.8.0: Fri Sep 8
17:18:57
PDT 2006; root:xnu-792.12.6.obj~1/RELEASE_PPC Power Macintosh powerpc
Perl version: 5.008006
Net::DNS version: 0.59
config: default settings
debug: using port 12526
debug: USER: spamd
debug: GROUP: postfix
debug: issuing user: root
debug: issuing group: root
08:46:57 debug: startup: using default settings
08:46:57 warning: err: setruid() not implemented at ./policyd-weight line
2342.
I've found solution here:
http://groups.google.com/group/comp.lang.perl.misc/browse_thread/thread/9beeaa638ba83213/dba793905626ca75%23dba793905626ca75
And have made a dirty hack to make script working - see below (also attached)
871,873c871,878
$( = $gname; if($!) { die ($)($) set GID to
$gname: $!; }
$) = $gname $gname; if($!) { die set EGID to $gname: $!; }
Unfortunately that is not portable either, also, this does not empty the groups
you are in.
Please try following:
perl -wle '
($(,$)) = (80, 80 80); die $! if $!;
($,$) = (80, 80); die $! if $!;
print ruid: $\neuid: $\nrgid: $(\negid: $)\n '
This must output following:
ruid: 80
euid: 80
rgid: 80 80
egid: 80 80
I do NOT expect that to be portable, after all, set*id - regardless which
approach one uses, is non portable, even using use POSIX is non portable.
However, I'll first use the above version, check for errors and then use the
current approach. If nothing works, then the OS/perl on that plattform is
highly broken.
The patch your provided may work on Mac, but outputs here:
linux redhat 7.1, kernel 2.4.25, perl 5.6:
root# perl -wle '
$(=$) = 80; die $! if $!;
$=$ = 80; die $! if $!;
print ruid: $\neuid: $\nrgid: $(\negid: $)\n '
ruid: 80
euid: 80
rgid: 80 10 6 4 3 2 1 0
egid: 80 10 6 4 3 2 1 0
^^ Danger!
FreeBSD 6.1, perl 5.8.8:
root# perl -wle '
quote $(=$) = 80; die $! if $!;
quote $=$ = 80; die $! if $!;
quote print ruid: $\neuid: $\nrgid: $(\negid: $)\n '
Operation not permitted at -e line 3.
No comment.
--
Robert Felber (PGP: 896CF30B)
Munich, Germany
Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Problem starting daemon on Mac OS X 10.4
On Wed, Nov 01, 2006 at 06:21:29PM +0200, Vladimir Vladimirov wrote: Thank you for pointing me Robert I've tried code you've sent me: sh-2.05b# perl -wle ' ($(,$)) = (80, 80 80); die $! if $!; ($,$) = (80, 80); die $! if $!; print ruid: $\neuid: $\nrgid: $(\negid: $)\n ' ruid: 80 euid: 80 rgid: 80 80 egid: 80 80 it works Good to know. Thanks! The diff you provided changed other stuff, too. However, I take it as suggestion on how to implement it. Thanks again. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.14 beta
changes: - This is a version bumb from 0.1.13 beta-16 to 0.1.14 beta. For a full set of changes from 0.1.12 beta-4 to 0.1.14 beta please read http://www.policyd-weight.org/releases/policyd-weight-0.1.14/changes.txt NOTE: the changes do list 0.1.12 beta-4 to 0.1.14 beta because we skipped a 0.1.13 beta release to Operating Systems. The 0.1.14 beta update has been reported to the FreeBSD ports and should be updated soon. Packagers may use http://www.policyd-weight.org/releases/policyd-weight-0.1.14.tar.gz MD5 (policyd-weight-0.1.14.tar.gz) = fb4829a57c8b805fe981ee949a145042 -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Other RBLs and scoring..
On Sat, Oct 28, 2006 at 08:38:36PM +0530, Dhawal Doshy wrote: Robert Felber wrote: On Sat, Oct 28, 2006 at 04:35:49PM +0530, Dhawal Doshy wrote: Ignore the scores in the above list as that is where i need your assistance. What ought to be the basis for scoring of these RBLs? Aggressive/Effective RBLs may add a Good score, i.e. -1.5 or similiar RBLs which are not that effective but have also zero false positives may have a high BAD score and a 0 GOOD score RBLs which are not really trustworthy should have a low BAD score and 0 GOOD score. Another thing, how does one use multi-lookup rbls? say one composite RBL returning different codes for different listing reasons. We do not score for listing reasons. If the RBL returns 127.0.0.[1|2|3|and so on] we count it as one hit and apply the score of the RBL. Aynthing else would go too far (we try to keep things as simple as possible - i.e. less is more). Thanks, btw i am creating a small init script for redhat / clones, would you be interested in a contrib? Yes. But I do prefer following scripts: - should be provided by a package maintainer for policyd-weight of the related OS - If there is no package for your OS yet, then consider making one - The URL for the package listing in $OS should be provided, for instance: http://www.freebsd.org/cgi/cvsweb.cgi/ports/mail/postfix-policyd-weight/ - If your OS does not have a package repository then of course your contact address must appear within that script. - If that init script works only for that particular $OS release such as Coolnix 2.1 but not Coolnix 2.0 then I'd like to know that (better yet: make it compatible with both versions.) I hope that's not too strict. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Questions about score perl taint checking
On Wed, Oct 25, 2006 at 05:44:26PM +0200, Urban Hillebrand wrote: (3) This is probably to early to report, as I have not yet been able to reproduce this problem. Has anyone had problems with policyd-weight and perl taint checking? Well, ok, the thing is, we did not change your script in any way to turn it on! I believe that now. Could some error in the config file have led to this? Right now they are like this: Yes. You must have something in your configfile which makes perl choke on that file. I have figured out what exactly causes this behaviour: Background: some versions of suse didn't implement the perl POSIX module correctly. As such we switched from dropping privileges via POSIX module to $, $, $( and $). Now this raises another issue (mainly due to a not obvious documented perl feature): If we set $ then perl turns on taint mode. We could do $=$=n - but that is non-portable. I am about to fix that taint issue by making it taint-safe. Stay tuned, I will release a 0.1.13 beta-16 today. (meanwhile you can double check your config file for syntax errors or bogus characters). -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Questions about score perl taint checking
On Wed, Oct 25, 2006 at 03:34:07PM +0200, Urban Hillebrand wrote: Hello list, we are trying to implement policyd-weight (0.1.13 beta-14) on 2 medium-sized mailgateways (~150.000 Mails / day). Please allow me some newbie questions: (1) We only want to use RBLs for the time being, so we set $dnsbl_checks_only = 1 in the conffile. We see lots of blocked hosts (Your MTA is listed in too many DNSBLs; check http://...;), but from time to time we also reject mails with this message: Mail appeared to be SPAM or forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settings or to get removed from DNSBLs (multirecipient mail) Note the multirecipient mail. Policyd-weight checks before RBL checks whether the multirecipient mail has already been reject. i.e.: RCPT TO: [EMAIL PROTECTED] 5xx too many rbls RCPT TO: [EMAIL PROTECTED] 5xx $REJECTMSG with multirecipient statement (2) Probably related: Which scores are computed for $REJECTLEVEL? HELO + RHSBL? Too many to name: SENDER (and subdomain) A/MX records vs client IP (subnets) HELO (and subdomain) A/MX records vs client IP (subnets) if SENDER failed Client PTR vs SENDER/HELO domains (and parent domains) if HELO and SENDER failed (alone this made up 36 possibilities of verification the last time I tried to form some evaluation scenario) Numeric HELO Random Sender localpart Bogus Sender MX records (empty, private networks) Anonymous Sender localpart (nobody|anonymous) RHSBL influenced by previous checks And more (this is a quick answer, more remains for the documentation) (3) This is probably to early to report, as I have not yet been able to reproduce this problem. Has anyone had problems with policyd-weight and perl taint checking? DO _NOT_ use taint (yet), DO _NOT_ use -w Perl is too chatty on STDERR. Postfix reads also STDERR messages (at least this goes for the master.cf mode). Thus we have to be very strict on what we output to STDERR|STDOUT or manage without ill module hacks that STDERR gets redirected to mylog(). And taint leads to unexpected exits. (NOTE: personally I avoid modules as much as possible for single point of failure reasons.) Some (intermittent) errors we saw: postfix/policyd-weight[20585]: child: err: Bad file descriptor Insecure dependency in eval while running with -T switch at /usr/libexec/postfix/policyd-weight line 2754, GEN5000 line 540. This is when read in the config file. The code is supposed to be perl code and must be executed - taint breaks this of course. Thus the config file must be writeable only by root - which policyd-weight checks. or postfix/policyd-weight[20340]: cache: err: Insecure dependency in eval while running with -T switch at /usr/libexec/postfix/policyd-weight line 2754, GEN6 line 1 Same thing as above. After that: postfix/smtpd[31437]: warning: premature end-of-input on 127.0.0.1:12525 while reading input attribute name postfix/smtpd[31437]: warning: problem talking to server 127.0.0.1:12525: Success All Mails were bounced with 450 Server configuration problem from that point on :( Perl warnings and taints are nice sometimes. Sometimes they break things. I will try to make policyd-weight -t compatible in 0.1.15 devel - but I do not have good feelings. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Hostname verification like in postfix?
On Tue, Oct 24, 2006 at 02:49:56PM +0200, Thomas Bange wrote: Hello, while reading through my logfiles I found this: Oct 22 05:51:41 mail postfix/smtpd[10412]: warning: 60.52.87.69: hostname tm.net.my verification failed: Name or service not known Oct 22 05:51:41 mail postfix/smtpd[10412]: connect from unknown[60.52.87.69] Oct 22 05:51:57 mail postfix/policyd-weight[20658]: weighted check: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 IX_MANITU=ERR CL_IP_NE_HELO=1.5 REV_IP_EQ_HELO=-1.25 (check from: .verbe. - helo: .tm.net.) FROM_MATCHES_NOT_UNVR_HELO=1.6 client=60.52.87.69 helo=tm.net.my [EMAIL PROTECTED] [EMAIL PROTECTED], rate: -2.65 For postfix it's a unknown host, but for policyd-weight it is not. As far as I understand it, postfix does a forward lookup on the sender IP and a reverse lookup on the result. If they match, everything is fine. If not, hostname verification fails. Is that right? policyd-weight seems to use just the reverse lookup on the IP for its helo-checks. Wouldn't it be better to do it the postfix way? Actually unknown host to postfix means CL_IP_NE_HELO in policyd-weight. But as this postfix restriction is troublesome we weight it. If you trust reject_unknown_client then place it into main.cf. An alternative would be to set #BAD(failure) #GOOD(success) @client_ip_eq_helo_score = (1.5, -1.25 ); to @client_ip_eq_helo_score = (2.5, -1.25 ); or similiar via your policyd-weight.conf. I do not suggest to do so (neither reject_unknown_* nor rejecting them with too high scores). Policyd-weight uses reverse records as a last resort to get some client/helo/sender DNS correlation to avoid rejecting not fully misconfigured MTAs/DNS. Btw, I seem to get IX_MANITU=ERR very often. Is that normal for that zone? Or does IX_MANITU=ERR mean NOT_IN_IX_MANITU (which I've never seen)? ERR means that there was some timeout or no responible servers could be reached. The IX DNSBL seems to have some troubles. You may read: http://www.heise.de/ix/foren/go.shtml?read=1msg_id=11442568forum_id=48292 And wait for some answer of Bert Ungerer. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Hostname verification like in postfix?
On Tue, Oct 24, 2006 at 03:56:14PM +0200, Thomas Bange wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Robert Felber Sent: Tuesday, October 24, 2006 3:48 PM To: policyd-weight-list@ek-muc.de Subject: Re: Hostname verification like in postfix? [...] I will rethink about it, as it wouldn't require extra DNS lookups. (currently I see no proper way to include it, but that doesn't mean anything). Well, that explains why CL_IP_NE_HELO is not always scored, even with a (postfix) unknown host: Oct 22 15:41:40 mail postfix/smtpd[4257]: warning: 158.75.241.14: hostname host14.smgr.pl verification failed: Name or service not known Oct 22 15:41:40 mail postfix/smtpd[4257]: connect from unknown[158.75.241.14] Oct 22 15:41:56 mail postfix/policyd-weight[20658]: weighted check: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 HELO_IP_IN_CL_SUBNET=-1.2 (check from: .valleyesp. - helo: .host14.smgr.) FROM_MATCHES_NOT_HELO=1 client=158.75.241.14 helo=host14.smgr.pl [EMAIL PROTECTED] [EMAIL PROTECTED], rate: -4.7 Right, lets investigate this client: % host host14.smgr.pl host14.smgr.pl does not exist, try again % host 158.75.241.14 Name: host14.smgr.pl % host smgr.pl smgr.pl A 158.75.241.6 I.e. the client is in the /24 subnet of 158.75.241. Thus we totally ignore the fact that it is an postfix unknown client but lay more attention on whether what the client tells us with his HELO is true or partially misconfigured. Unfortunately that is a good example of may be legitime - may be spam - not even an evaluation of unknown client would tell us, whether it is spam or not. But - if we would score unknown clients against RBLs in *certain* other constellations then making use of unknown clients *might* help. Allthough (currently) it is not clear to me in *which* constellations we should score unknown against RBLs. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: policyd-weight mysteriously dying
On Mon, Oct 16, 2006 at 01:30:11PM -0400, Marshal Newrock wrote: I've been using policyd-weight on a busy server for a week with no problems. Last night, while checking the logs, I saw a small handful of messages (about five) saying connection refused by policyd-weight, indicating to me that I had used up the number of available processes. You should have a log-message main: MAX_PROC (150) reached. However, 100 is huge. What's the average mail/hour (or mail/minute) this server has to handle? What is the average delay when a client connects to an smtpd and policyd-weight answers? So I increased $MAX_PROCS from 150 to 200, and $MIN_PROCS from 10 to 20, and issued policyd-weight reload. At the same time, I increased the maximum number of smtpd processes for postfix from 200 to 300, as it also had indicated it had run out of available processes. After this, policyd-weight no longer works. [...] When it is started, and 'ps aux | grep policyd-weight' run, sometimes it simply doesn't show up at all, or only the cache will show up. Sometimes the master will show up with no child processes. While it is running, if it manages to start, sometimes it will successfully work, but frequently postfix reports warning: premature end-of-input on 127.0.0.1:12525 while reading input attribute name. I have tried restoring $MAX_PROCS and $MIN_PROCS to their default values, but there is no change. I also set $DEBUG=1 but got no extra logging. As you are running Gentoo, which log files did you check. Gentoo might send debug messages to /var/log/debug.log # date debug-pr.txt 21 # /path/to/policyd-weight -k restart debug-pr.txt 21 # echo /path/to/policyd-weight -k restart -- done debug-pr.txt 21 # bash # grep weight\[ /var/log/mail.log /var/log/debug.log debug-pr.txt 21 # exit (that bash call is to make sure that no zsh is used. zsh does sometimes so some weird globbing or escapes meta-chars where it shouldn't escape them) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: policyd-weight mysteriously dying
On Mon, Oct 16, 2006 at 09:22:09PM +0200, Robert Felber wrote: running, if it manages to start, sometimes it will successfully work, but frequently postfix reports warning: premature end-of-input on 127.0.0.1:12525 while reading input attribute name. I have tried restoring $MAX_PROCS and $MIN_PROCS to their default values, but there is no change. I also set $DEBUG=1 but got no extra logging. As you are running Gentoo, which log files did you check. Gentoo might send debug messages to /var/log/debug.log # date debug-pr.txt 21 # /path/to/policyd-weight -k restart debug-pr.txt 21 # echo /path/to/policyd-weight -k restart -- done debug-pr.txt 21 # bash # grep weight\[ /var/log/mail.log /var/log/debug.log debug-pr.txt 21 # exit This might be a bit too verbose and big-sized, instead use following: # datedebug-pr.txt 21 # sysctl net.core.somaxconn debug-pr.txt 21 # lsof -i -n | wc -l debug-pr.txt 21 # /path/to/policyd-weight -k restart debug-pr.txt 21 wait some seconds or send some mails to yourself # echo /path/to/policyd-weight -k restart -- done debug-pr.txt 21 # bash # grep weight\[ /var/log/mail.* | grep -v decided\|weighted check debug-pr.txt 21 # date debug-pr.txt 21 (notice: grep /var/log/mail.* -- just encountered: warnings are sent to /var/log/mail.warn) -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Security update: version 0.1.13 beta-14
changes: core: - mylog() does now pass all arguments with a preceding %s format-string to avoid, that remote format string may be executed. See NOTEs. NOTES: If $DEBUG = 1; then policyd-weight may be vulnerable to a remote and local sprintf vulnerability if old Sys::Syslog versions are in place. A syslog call for the cache queries was handed straight to the syslog() routine without preceding a format-string %s. If the MTA did not deny illegal characters then a remote user may send a special format string as sender which would execute at that debug message: mylog(info=cache_query: $query $ip $sender $rate) if $DEBUG; I cannot find information in which Sys::Syslog version this issue has been fixed, as such I release this security fix. It is highly recommended to update Sys::Syslog, too. This issue is rather old and might be read here: http://news.perl-foundation.org/2005/12/updated_perl_modules_alleviate.html You may check whether you are vulnerable following way: a) look if you have $DEBUG = 1; set if so: b) telnet localhost 12525 (or the port under which policyd-weight runs) [EMAIL PROTECTED] helo_name=somedomain.com\n client_address=1.2.3.4\n request=smtpd_access_policy\n \n \n check your log if you have following entry: Oct 13 12:15:05 devil postfix/policyd-weight[14219]: cache_query: nask 1.2.3.4 [EMAIL PROTECTED] if you see a 0 before somedomain.com you are vulnerable, if you see an %i you are not vulnerable. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.13 beta-12
changes: core: - Childs didn't die() verbose in all cases, fixed (hopefully). - Call close only on the RBL socket if it exists and is connect, _obviously_ this was the reason for a silent death of childs. Scoring: - BAD_MX and BOGUS_MX sometimes computed and used twice - fixed. Number formats: - myrnd() subroutine implemented which should sanitize floating point output. Unfortunately perl does not store/use floats in a sane way. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: version update: version 0.1.13 beta-13
On Thu, Oct 12, 2006 at 12:15:55PM -0600, Gary V wrote: Should policyd-weight.conf have a defined return? I'm not a Perl person, but I have seen a defined return recommended for other external files like this. [...stuff...] 1; # return This is generally true for packages/modules, files used via require() and so on. We use (for scope and 'use strict'-reasons) a completely different approach: we slurp it into a string, and then eval that string. Perl reports eval errors in @$. So there is no need in checking the last execution result of the external file. Or from http://perldoc.perl.org/functions/require.html: The file must return true as the last statement to indicate successful execution of any initialization code, so it's customary to end such a file with 1; unless you're sure it'll return true otherwise. But it's better just to put the 1; , in case you add more statements. We do not use require() but eval() which is described here: http://perldoc.perl.org/functions/eval.html If there was no error, $@ is guaranteed to be a null string. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Watch your Logs today [Humor]
Hello, Current Version: 0.1.13 Current Beta:13 Current Day: Friday, 13th Be prepared! -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Interessting rate
On Wed, Oct 11, 2006 at 02:37:13PM +0200, Thomas Bange wrote: Hello, nothing really important, but a little bit out of the ordinary: Oct 11 14:14:29 www postfix/policyd-weight[7552]: weighted check: NOT_IN_BL_NJABL=-1.5 NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 CL_IP_EQ_HELO_IP=-2 (check from: .amuro. - helo: .kyonggi.ac.) FROM_MATCHES_NOT_HELO=1 IN_DSN_RFCI=6.3 client=203.249.3.202 helo=sniper.kyonggi.ac.kr [EMAIL PROTECTED] [EMAIL PROTECTED], rate: 0.801 ^ Looks like a rounding error somehow. Actually we don't call any rounding-magic and let perl do its stuff. Thus some divisions will look nasty. As soon as I have 0.1.14 beta out I will do all the remaining stuff (including rounding) in 0.1.15 devel (the userbase is growing and reports more frequent, which wasn't the case before). 0.1.14 beta must wait though - there is still a known unresolved issue when handling /etc/hosts NS servers which are not available. Policyd-weight freaked out on a SuSe because of this condition. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Interessting rate
On Wed, Oct 11, 2006 at 09:45:35AM -0500, Paul Schmehl wrote: --On Wednesday, October 11, 2006 15:54:13 +0200 Robert Felber [EMAIL PROTECTED] wrote: Actually we don't call any rounding-magic and let perl do its stuff. Thus some divisions will look nasty. As soon as I have 0.1.14 beta out I will do all the remaining stuff (including rounding) in 0.1.15 devel (the userbase is growing and reports more frequent, which wasn't the case before). 0.1.14 beta must wait though - there is still a known unresolved issue when handling /etc/hosts NS servers which are not available. Policyd-weight freaked out on a SuSe because of this condition. Perhaps the solution to that is to include at least one, hard-coded fallback NS that is known to be reliably available? That wouldn't solve the problem entirely, because, if the client is processing mail and a router upstream begins flapping, *nothing* is going to work, but it should at least ameliorate the problem. Perhaps a time out with a soft failure would be appropriate? Maybe policyd could queue messages until it can reach an NS? Actually the intended behavior is to let pass the mail if policyd-weight has DNS errors. I have to narrow down why and when exactly it is freaking out. It's the first and only time I have seen such behavior. Even when I stresstested policyd-weight here and had turned off the DNS servers it didn't freak out that way, instead it had let pass it with too many local DNS errors. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Probably /etc/resolv.conf NS issues. Was: Re: Interessting rate
On Wed, Oct 11, 2006 at 02:23:22PM -0500, Paul Schmehl wrote: --On Wednesday, October 11, 2006 18:21:05 +0200 Robert Felber [EMAIL PROTECTED] wrote: Actually the intended behavior is to let pass the mail if policyd-weight has DNS errors. I have to narrow down why and when exactly it is freaking out. A misconfigured nsswitch.conf file? Err, my bad. I meant /etc/resolv.conf. Howver, I am trying to reproduce it at home. From the reporter I got such line: Oct 4 21:30:23 dhrhserver postfix/policyd-weight[20162]: Daemon terminated. [start] Oct 5 08:16:05 dhrhserver postfix/policyd-weight[2302]: could not open RBL Lookup Socket: IO::Socket::INET: connect: Network is unreachable Network is unreachable Oct 5 08:16:05 dhrhserver postfix/policyd-weight[2683]: policyd-weight 0.1.13 beta-11 started and daemonized using default settings, $USER=polw, EUID: 1010, UID: 1010 [then the first connect:] Oct 5 08:30:19 dhrhserver postfix/smtpd[3886]: connect from host.example.tld[xx.xx.xx.xx] [And then the spinoff which goes on like hell:] Oct 5 08:30:19 dhrhserver postfix/policyd-weight[3890]: child: spawned Oct 5 08:30:19 dhrhserver postfix/policyd-weight[3891]: child: spawned Oct 5 08:30:19 dhrhserver postfix/policyd-weight[3892]: child: spawned Oct 5 08:30:19 dhrhserver postfix/policyd-weight[3893]: child: spawned Oct 5 08:30:19 dhrhserver postfix/policyd-weight[3894]: child: spawned Oct 5 08:30:19 dhrhserver postfix/policyd-weight[3895]: child: spawned Oct 5 08:30:19 dhrhserver postfix/policyd-weight[2683]: child 3890 exited Oct 5 08:30:19 dhrhserver postfix/policyd-weight[2683]: child 3891 exited Oct 5 08:30:19 dhrhserver postfix/policyd-weight[2683]: child 3893 exited When I set at home an unreachable NS (say 10.1.1.1) it simply does let pass the mail with too many local DNS errors. I am still not certain whether it is an OS issue, a design issue, or whatever. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
release delay
Hello list, 0.1.14 beta was scheduled to be released today but over the weekend a new issue with SuSe arised which means, that at some (for me unknown) point policyd-weight spawns childs like mad which exit immediately again. I do not have full logs of this, as such I was not able to track it down. Please stand by. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: BOGUS_MX counted twice
On Mon, Oct 09, 2006 at 12:29:29PM +0200, Thomas Bange wrote: Hello, I just found this in my maillog: Oct 7 01:46:58 www postfix/policyd-weight[7552]: weighted check: NOT_IN_BL_NJABL=-1.5 NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 BOGUS_MX=2.1 BOGUS_MX=2.1 ^ CL_IP_EQ_HELO_IP=-2 (check from: .is-asp. - helo: .mailrelay1fra.is-teledata.) FROM_MATCHES_NOT_HELO=5.41 client=217.11.193.159 helo=mailrelay1fra.is-teledata.com [EMAIL PROTECTED] [EMAIL PROTECTED], rate: 3.11 Oct 7 01:46:58 www postfix/policyd-weight[7552]: decided action=450 Mail appeared to be SPAM or forged. Ask your Mail/DNS- Administrator to correct HELO and DNS MX settings or to get removed from DNSBLs Why is BOGUS_MX counted twice? Which version? What is the un-obfuscated sender domain? When I check against xxx.is-asp.com I get only one BOGUS_MX: info: weighted check: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_BL_NJABL=-1.5 NOT_IN_SPAMCOP=-1.5 BOGUS_MX=2.1 CL_IP_EQ_HELO_IP=-2 (check from: .is-asp. - helo: .mailrelay1fra.is-teledata.) FROM_MATCHES_NOT_HELO=5.41 client=217.11.193.159 helo=mailrelay1fra.is-teledata.com [EMAIL PROTECTED] to= helo_ips: 217.11.194.67 217.11.194.155 217.11.192.4 217.11.193.159, rate: 1.01 -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.13 beta-10
changes: core: - BAD_MX was asigned twice. Also BAD_MX gets now only logged if its value is non-zero. The difference between BAD_MX and BOGUS_MX is: BAD_MX: MX lookups of the FQDN of the sender argument gave an empty or bad entry. This is only harmfull if DNSBL listed as the FQDN might be a hostname and an A lookup might succeed. BAD_MX gets asigned in case of a hostname thoug, I haven't found a way to separate machine123.example.tld from subdomain.example.tld. That's the reason for using this Penalty _only_ if DNSBL listed. BOGUS_MX: Both, A and MX lookups of the FQDN sender argument gave an bogus or empty entry. Penalization regardless of DNSBL hits. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.13 beta-11
changes: core: - Policyd-weight exits now with an informative errormessage if the syslog-socket couldn't be set up. scoring: - RHSBL checks are now half-dnsbl influenced (yahoo groups still pass even if spamcop listed) - i.e. should catch more spam. Note: I hope this was the last change in this 0.1.13 beta line. If nothing serious is found 0.1.13 beta-11 will be released as 0.1.14 beta on 10/09/2006. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.13 beta-9
changes: core: - UID/EUID/GID/EGID is not set via the perl POSIX module anymore but by perl vars accordingly to`'man perlvar | less +/UID' SuSe 9.0 does have a bug with POSIX::setuid - thus the change which *should* work on all plattforms. Policyd-weight reports UID/EUIDs with its syslog startup message. - -f option added to hand policyd-weight a configuration file scoring: - BOGUS_MX still was a bit too aggressive. We penalize empty/bogus MX against DNSBL. Also: if both, MX and A record of the sender domain are bogus/empty it gets full penalization. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
System upgrad
Due to an upgrading from FreeBSD 5.3 to 6.1 there are some problems with majordomo. If you experience problems, please notify me at [EMAIL PROTECTED] -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: FROM_MATCHES_NOT_HELO and
On Wed, Sep 20, 2006 at 09:47:39AM +0200, Jan Wagner wrote: Hi, I did update from 0.1.11 beta-2 to latest verion ... since then I have two problems. Sep 19 08:50:45 gandalf postfix/policyd-weight[14407]: weighted check: NOT_IN_BL_NJABL=-1.5 NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 BOGUS_MX=2.1 CL_IP_EQ_HELO_IP=-2 (check from: .alicedsl. - helo: .backup.tmt.) FROM_MATCHES_NOT_HELO=5.41 client=217.145.99.58 helo=backup.tmt.de [EMAIL PROTECTED] [EMAIL PROTECTED], rate: 1.01 Lower @bogus_mx_score from: @bogus_mx_score = (2.1,0); to @bogus_mx_score = (2.0,0); The other thing is ... why I got a 450? I only redefined @dnsbl_score in /etc/policyd-weight.conf, so shouldn?t there go the default $REJECTMSG, which is 550 ...? Thats due the BOGUS_MX hit which may be a temporary issue and can be controlled with: $DEFER_STRING = 'IN_SPAMCOP= BOGUS_MX='; If that string is found in the return message, and the score iss lower than $DEFER_LEVEL (default 5) then the client is temporarily rejected as it might be a DNS issue. However, I will update the defaults, or check rather for bogus MX/A records instead of solely MX records. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: unreachable check_policy_service
On Mon, Sep 18, 2006 at 04:19:01PM +0200, Jan Wagner wrote: To make sure that policyd-weight always runs you could use some sort of daemon-tools (http://cr.yp.to/daemontools.html) Normaly I use ps-watcher (http://ps-watcher.sourceforge.net/) :-) But this wouldn?t help if there is policyd-weight running on a dedicated server and this one is unreachable (crash/hardware failure/network trouble/ ...) Indeed. This should be probably handled by postfix, meaning, that one should be able to decide whether to DUNNO the check_policy_service in case of connection problems. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: Problem with @rhsbl?
On Thu, Sep 07, 2006 at 03:26:48PM -0500, Paul Schmehl wrote: I discovered that I was rejected legitimate emails from sbcglobal.net because they're listed in rfc-ignorant.org for both postmaster@ and [EMAIL PROTECTED] (Both addresses are valid now, but apparently no one had notified rfc-ignorant.org.) I didn't want to whitelist sbcglobal.net, but I wanted the email to come through, so I changed the penalty score for rhsbl from 3.3 to 3.1 (just enough to get in under the wire unless there are other problems.) However, I *thought* I could correct the problem by adding a GOOD score to PM_RFCI and/or ABUSE_RFCI. When I tried that, it didn't work. Is this a bug? No bug. Not yet implemented as the priority for ham-scores in rhsbl results was _very_ low. Fixing that soon. Meanwhile you should instead give postmaster and abuse rhsbls very low scores (0.1 or so) or even remove it. I'm still about to throw those two lists out. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: GeoIP patch
On Sun, Sep 03, 2006 at 11:50:55PM +0200, Boris Hajduk wrote: Please understand that I cannot use the patch and as such not really evaluate it's usefullness, as this server hosts an international ML. No problem. Maybe just put a pointer to the patch for those running very small/personnal mail servers. That's an idea which I had on my radar, too. I think I will add a patch section on the main site with contact information of the patch author. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Solaris test
Hello, I've now tried to fix the Solaris bug for the INET/Daemon mode. This is a fix which remains to be tested across the Solaris Versions. Please download the current devel version at: http://www.policyd-weight.org/policyd-weight-devel Please also test this version on other platforms than solaris. If this version proves to run as intended it becomes officially beta-7 Background: Perl on Solaris obviously treats socketpair() created IPC-channels different than on other OSs. $sock-send fails on socketpair created channels, we use now print $sock foo\n; for sending and $sock-recv(my $ans, 1024); for receiving which hopefully works across all plattforms -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: GeoIP patch
On Fri, Sep 01, 2006 at 02:47:53PM +0200, Boris Hajduk wrote: Hi, 60% of the spam I receive originates from China and Korea. I used to drop SMTP packets from these countries using a netfilter patch I wrote on my linux server. I included a few examples in policyd-weight.conf.GeoIP.patch , just make sure you use ISO -3166 country codes. You can find the official list here : http://www.iso.org/iso/fr/prods-services/iso3166ma/02iso-3166-code-lists/list-en1.html Caution : it has only been tested on a low-volume mail server, yet. Don't put it on a big production server without test. Give it a try, remarks and critics are welcome. I've looked at the patch. Looks ok so far. (I also noticed that the rather unusual style has been followed, which makes things easier, thanks). I am willing to include such patches. However, the addition of country-based scores raises issues which some might say it is not good to hand such ill weapons to users. I think it is ok in after queue situations, though. Also note, I don't know what Geo::IP exactly does, which delays it adds which issues arise and so forth. See [1]. Suggestion, you should weight it against DNSBL and/or RHSBL or probably other checks which seem appropriate (cannot give a hint at the moment). Please understand that I cannot use the patch and as such not really evaluate it's usefullness, as this server hosts an international ML. [1] If it is OK with you, I'll add it in the next 0.1.14 devel branch (which takes some time, as I still don't have that Solaris Bug fixed, nor continued at the 0.1.13 man-pages). -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.13 beta-6
changes: - (fix)CL_IP_NE_HELO string was built wrong which was leading to scores such as 1.23.25 in DNSBL score strings - (change) Scores for each check were computed twice, once for the overall-score and once for the log-string which wasted unnecessary CPU cycles. Is now computed only once. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: warning: cache: could not open policyd-weight.conf:
On Thu, Aug 31, 2006 at 11:02:47AM -0500, Paul Schmehl wrote: The FreeBSD box is running error-free, but the CentOS box keeps generating this error in the logs: warning: cache: could not open policyd-weight.conf: It is not trying to open /etc/policyd-weight.conf but policyd-weight.conf Can you issue a ./policyd-weight -k restart and post the log entries of the startup process? Also the entries of the could not open line *after* the restart. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: version update: version 0.1.13 beta-6
On Thu, Aug 31, 2006 at 12:44:48PM -0600, Gary V wrote: Robert, since there is only one sample config, and it may or may not change with new revisions, on each revision would you be so kind to tell us whether the sample config has also changed or not (so I can keep them synchronized with a minimal amount of work)? With the next version bumb I won't provide an example config (except for packages). Instead use policyd-weight defaults file then. This saves me from keeping two files in sync. I will announce it when default config values change and mark it, though. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
***SPAM*** Re: spam with perfect rDNS/HELO/sender match
On Mon, Aug 28, 2006 at 09:37:24AM -0500, /dev/rob0 wrote: Then ... the theory is that rDNS/HELO/sender match with the domain NS found in that list would be strong evidence, perhaps even proof, of spam. Working hypotheses are that the list of NS IP addresses will be very small, and that the overlap will be very high. quick an dirty check from policyd-weight logs, I expect, then we can utilize amavis results, too: Questionable (i.e. false positives?) entries: dns1.de ? dns2.de ? hichina.com ? registerfly.com ? register.com? How would we separate the men from the boys? # bzgrep \( IN_SURB\| IN_AHB\) /var/log/mail/maillog* | perl -pse 's/.*from=.*?@(.*?).*/$1/' | sort -u | perl -pse 'system(host -t NS $_)' | grep name server 0-0.com name server ns1.floridaserver.com. 0-0.com name server ns2.floridaserver.com. 001f.com name server dns1.hichina.com. 001f.com name server dns2.hichina.com. 010-101.com name server dns1.010-101.com. 010-101.com name server ns2.010-101.com. 0451.com name server sdns.hl.cninfo.net. 0668.com name server dns1.hichina.com. 0668.com name server dns2.hichina.com. 0668.com name server dns3.hichina.com. 09pia.com name server ns3.cypack.com. 09pia.com name server ns4.cypack.com. 1-800-patches.com name server ns1.inetcentrex.com. 1-800-patches.com name server ns2.inetcentrex.com. 1-call.com name server echidna.aswell.net. 1-call.com name server wally.aswell.net. 163.com name server ns.nease.net. 163.com name server ns3.nease.net. 19pinkgirl.info name server dns3.registerfly.com. 19pinkgirl.info name server dns2.registerfly.com. 19pinkgirl.info name server dns1.registerfly.com. 4diamonds19.info name server dns3.registerfly.com. 4diamonds19.info name server dns2.registerfly.com. 4diamonds19.info name server dns1.registerfly.com. dramotolado.com name server ns1.dzot.net. dramotolado.com name server ns2.dzot.net. healthnewstoday.net name server dns10.register.com. healthnewstoday.net name server dns9.register.com. huntfortreasure.net name server ns1.vendaregroup.com. huntfortreasure.net name server ns2.vendaregroup.com. huntfortreasure.net name server vns2.vendaregroup.com. limpokolako.com name server ns1.dzot.net. limpokolako.com name server ns2.dzot.net. mail15.com name server ns1.pochta.ru. mail15.com name server ns2.pochta.ru. mail15.com name server ns3.pochta.ru. mail15.com name server ns4.pochta.ru. megastopodo.com name server ns1.dzot.net. megastopodo.com name server ns2.dzot.net. premiumreifen.de name server dns.dns1.de. premiumreifen.de name server dns.dns2.de. rungirl19.info name server dns3.registerfly.com. rungirl19.info name server dns2.registerfly.com. rungirl19.info name server dns1.registerfly.com. sirapukato.com name server ns1.000y.net. sirapukato.com name server ns2.000y.net. slamdunkfan.com name server park25.secureserver.net. slamdunkfan.com name server park26.secureserver.net. stabikomnod.com name server ns1.000y.net. stabikomnod.com name server ns2.000y.net. stopokoinod.com name server ns1.000y.net. stopokoinod.com name server ns2.000y.net. strogdordd.com name server ns1.dzot.net. strogdordd.com name server ns2.dzot.net. velotrafokot.com name server ns1.000y.net. velotrafokot.com name server ns2.000y.net. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
Re: spam with perfect rDNS/HELO/sender match
On Tue, Aug 29, 2006 at 08:56:44AM +0200, Robert Felber wrote: On Mon, Aug 28, 2006 at 09:37:24AM -0500, /dev/rob0 wrote: Then ... the theory is that rDNS/HELO/sender match with the domain NS found in that list would be strong evidence, perhaps even proof, of spam. Working hypotheses are that the list of NS IP addresses will be very small, and that the overlap will be very high. quick an dirty check from policyd-weight logs, I expect, then we can utilize amavis results, too: Questionable (i.e. false positives?) entries: dns1.de ? dns2.de ? hichina.com ? registerfly.com ? register.com? How would we separate the men from the boys? # bzgrep \( IN_SURB\| IN_AHB\) /var/log/mail/maillog* | perl -pse 's/.*from=.*?@(.*?).*/$1/' | sort -u | perl -pse 'system(host -t NS $_)' | grep name server This gives at least a close figure. I have following in my head: Table of terms: SPAM Domain:registered domain to RECEIVE possible spam-answers i.e. Sender Domain DNS Domain:DNS Service who handles SPAM Domain Considerations: We can not use MX records of SPAM Domain as they may point to a legitime domain (exploitation of concept). I thought, we could also check to which domain the MX points to get the NS Server of that MX domain, but these pointers can be intentionally abused. We rely on the NS record, specifically the NS-hostname, of DNS Domain solely. The overhead of looking up their IPs is way to heavy, the NS-hostname is considered sufficient (for now). For an automation in policyd-weight we can solely use results of RHSBLs which list spam-domains, such as AHBL or SURBL. Using other indicators is not appropriate as the spammer can set a legitime domain to get them blacklisted by us, this goes also for spamhaus and other RBL listings. To make safe use of an automated blacklisting mechanism for sender domains whose domain is hosted by a spammers-paradise DNS domain we need more than one argument to blacklist them. I made up three arguments: number of SPAM Domains listed by RHSBLs number of SPAMs sent. number of own SPAM-Domain detections AB C D or: dzot.net:20:4:2 which means, the DNS Provider dzot.net hosts 4 SPAM Domains which were detected by AHBL or SURBL. 20 mails were received by domains of dzot.net 2 SPAM-domains were automatically added. Automatic addition of domains happens, when a configurable ratio-level of B,C and D exceeded. we return for a check SPAM_DNS a score of: ((B*(C+D)*(C+D)/100) * ((C+D)*(C+D))/100) * 100 In the above case: # echo scale=3; ( (20*(4+2)*(4+2)/100) * ((4+2)*(4+2))/100) * 100 | bc # 259.200 which will result in a immediately reject, D will be increased if it is a not RHSBL listed SPAM-Domain i.e. dzot.net:21:4:3 if the result exceeds 1000 we won't increase counters anymore. if it would be foo.tld:10:1:0 # echo scale=3; ((10*(1+0)*(1+0)/100)* ((1+0)*(1+0))/100) * 100 | bc # 0.100 No immediately reject, but B will be increased i.e. foo.tld:11:1:0 if it would be bar.tld:10:2:0 # echo scale=3; ((10*(2+0)*(2+0)/100)* ((2+0)*(2+0))/100) * 100 | bc # 1.600 No immediately reject, but B will be increased, also D will be increased if it is not a RHSBL listed SPAM-Domain i.e. bar.tld:11:2:1 This is so far what I can come up with, and must be further investigated. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
homepage fixes
Setup HOWTO as well as FAQs contained a typo/error: check_policy_service 127.0.0.1:12525 which caused a Aug 27 22:50:56 machine postfix/smtpd[72739]: fatal: invalid transport name: 127.0.0.1 in service: 127.0.0.1:12525 corrected it to check_policy_service inet:127.0.0.1:12525 Thanks to B. Hajduk for reporting. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
version update: version 0.1.13 beta-5
changes: scoring: - CL_IP_NE_HELO was not DNSBL influenced. When the IP of the HELO argument couldnt be verified against the Client IP we just gave the bad score of @client_ip_eq_helo_score without adding eventually RBL scores. -- Robert Felber (PGP: 896CF30B) Munich, Germany Policyd-weight Mailinglist - http://www.policyd-weight.org/
