Re: spamd greylisting: false positives
On 2012-05-28, David Diggles da...@elven.com.au wrote: So there you have it. Don't use spamd with greytrapping if your secondary MX is going to deliver a bounce. It will confuse SMTP servers into giving up. well, that doesn't just apply to spamd.. you are better off not listing a secondary MX unless it's A) working and B) has equalyl strong or better spam controls as the primary MX
Re: spamd greylisting: false positives
On 2012-05-27, David Diggles da...@elven.com.au wrote: From: Stuart Henderson stu () spacehopper ! org Date: 2012-05-27 22:29:50 On 2012-05-27, David Diggles da...@elven.com.au wrote: Bummer, I have forgotten to pflog the spamd connections to lo0 So this breaks spamlogd which means servers will expire from the greylist even if they mail you regularly.. Do you mean this pf rule pass in log on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state breaks spamlogd? Would you mind explaining why, and how I can un-break it? Ah sorry I misread, I thought you had written that you weren't logging the SMTP connections, not the spamd connections..
Re: spamd greylisting: false positives
David Diggles da...@elven.com.au writes: So there you have it. Don't use spamd with greytrapping if your secondary MX is going to deliver a bounce. It will confuse SMTP servers into giving up. Secondary MXes that are not set up to actually receive mail for your domain is one thing (annoying, but just a simple misconfiguration), another thing you need to do is make sure the secondaries have the same or equivalent level of spam and malware protection. That's where things like spamd's syncronization options come in handy. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: spamd greylisting: false positives
* David Diggles da...@elven.com.au [2012-05-28 02:44]: Why shouldn't I? These guys do in their example. https://calomel.org/spamd_config.html that alone is a reason to not do it. really, everything on calomel.org is garbage. you are best off to ignore it. i wish somebody would track this guy don, explain hom how his garbage hurts the community, and makes him remove it. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: spamd greylisting: false positives
In response to various tidbits that popped up in this thread, I put together some notes on setting up a sane email system, in a works for me article: http://bsdly.blogspot.com/2012/05/in-name-of-sane-email-setting-up-spamd.html -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: spamd greylisting: false positives
-Ursprungligt meddelande-
Fren: owner-m...@openbsd.org [mailto:owner-m...@openbsd.org] Fvr David
Diggles
Skickat: den 27 maj 2012 02:53
Till: misc@openbsd.org
Dmne: Re: spamd greylisting: false positives
This may seem like a dead horse to some by now, but I am disappointed no one
replied to the msg, I supplied the detailed event information with
timestamps, regarding lists.openbsd.org mails not being whitelisted by spamd
when run in greylist mode.
RFC282, 4.5.4.1 Sending Strategy:
The sender MUST delay retrying a particular destination after one
attempt has failed. In general, the retry interval SHOULD be at
least 30 minutes; however, more sophisticated and variable strategies
will be beneficial when the SMTP client can determine the reason for
non-delivery.
Retries continue until the message is transmitted or the sender gives
up; the give-up time generally needs to be at least 4-5 days. The
parameters to the retry algorithm MUST be configurable.
Yet I have been advised not to mess with the default timings with -G option.
It looks to me like the retry intervals of lists.openbsd.org are not
sufficient to get it whitelisted by spamd.
I am well beyond assuming anything, and prepared to learn / accept any
constructive advice.
Can anyone confirm they have the following scenario?
* A clean installed OpenBSD 5.1 configured as a primary MX
* Clean spamd settings, clean /var/db/spamd
* Default spamd with no options
* Default spamlogd with no options
* The pf.conf uses spamd entries from the example pf.conf from etc.tgz
* No manual whitelist entry for lists.openbsd.org
* Incoming from lists.openbsd.org is eventually whitelisted by spamd
I am just trying to learn the cause, and I have been fully prepared to wear
egg on my face if my own configuration is causing the problem. I have not
yet proven this is the case.
I believe I have checked everything anyone suggested to check.
I really don't want my next check be to roll back to 4.9 and see if
lists.openbsd.org will auto whitelist like it previously did.
In hope,
David
On Sat, May 26, 2012 at 01:19:38PM +1000, David Diggles wrote:
Ok I am still not getting emails from lists.openbsd.org (so
please if you reply, cc to me).
I restarted spamd at this time after deleting /var/db/spamd and
clearing the bypass tables in pf at this time:
2012-05-26 02:13:12 # /usr/libexec/spamd
Here is the last message to make it to sendmail from misc:
fgrep from= /var/log/maillog|fgrep owner-misc|tail -1|awk '{print
$1,$2,$3}'
May 26 01:54:35
The pf rules for spamd I have are taken from the default pf.conf:
pass in on egress inet proto tcp from any to any port = 25 flags S/SA
rdr-to 127.0.0.1 port 8025 pass in on egress proto tcp from nospamd
to any port = 25 flags S/SA pass in log on egress proto tcp from
spamd-white to any port = 25 flags S/SA pass out log on egress proto
tcp from any to any port = 25 flags S/S
It is currently Sat May 26 12:54:31 EST 201
Times of passed smtp connections for May 26:
tcpdump -n -e -ttt -r /var/log/pflog 21|fgrep .25:|\ fgrep 'May
26'|awk '{print $3}'
01:14:53.793995
04:17:11.846707
05:00:19.443080
05:15:01.487277
07:17:51.114440
09:35:58.120098
10:14:21.444822
11:53:33.611903
So I will skip the first entry when I grep for the ip addresses, with
a tail +2 because it occurred
*before* I reset everything.
tcpdump -n -e -ttt -r /var/log/pflog 21|fgrep .25:|\ fgrep 'May
26'|awk '{print $10}'|tail +2|\ awk -F. '{print
$1.$2.$3.$4}'|sort -n
17.254.6.112
74.125.82.47
113.172.232.215
129.21.208.44
202.58.38.80
203.59.1.110
206.46.252.115
I have the following tables.
pfctl -s Tables
nospamd
spamd-white
Confirming against the spamd-white table
pfctl -t spamd-white -Ts
17.254.6.112
74.125.82.47
113.172.232.215
129.21.208.44
202.58.38.80
203.59.1.110
206.46.252.115
lists.openbsd.org = 192.43.244.163
So nothing from misc has made it to sendmail since I emptied nospamd
and spamd-white on pf.conf
These are all the attempts from lists.openbsd.org since I cleared the
spamdb and pf tables.
fgrep 192.43.244.163 /var/log/spamd|fgrep 'May 26'
May 26 02:53:48 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 02:54:00 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 03:00:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 03:00:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 04:41:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 04:41:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:04:19 skitL spamd[25502]: 192.43.244.163: connected (2/1)
May 26 05:04:31 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:15:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:15:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:19:36 skitL spamd[25502
Re: spamd greylisting: false positives
Hi again David. If all the spamd settings are back to default, I would recommend trying to pinpoint where the problem is. Just to check if it could be something wrong with the syntax of your pf rules regarding spamd, just comment them out. pfctl -f /etc/pf.conf and run for a while and see if you receive any mails. /Hasse I am running spamd in blacklist mode now, so I am once again receiving the mailing list. I think the default spamd timings do not give lists.openbsd.org enough time to retransmit in the whitelist window. It would be nice if someone else had the time to attempt reproducing this. This one you sent me earlier has some advice about tuning the timings, https://calomel.org/spamd_config.html In this section: We suggest setting the pass time to as high as you are comfortable with. Use a time between 10 and 55 minutes. You are welcome to set it as low as 2 minutes, but it is possible that some spammers might get white listed. After setting up spamd take some time, go through the logs and look for patterns. Adjust the pass time as necessary. ... I realise I have been advised in the list here not to mess around with the timings :P
Re: spamd greylisting: false positives
What do you mean by running in blacklist mode ? Which settings are different from Grey trapping ? Are Openbsd mailing list the only list or mail you have problems with ? /Hasse By blacklist mode, I mean this: spamd -b spamd-setup -b pf.conf: table spamd persist pass in on egress proto tcp from spamd to any port smtp \ rdr-to 127.0.0.1 port spamd The OpenBSD mailing list was not the only smtp server I was having problems with. Others included: 1. test email from my work account 2. business email from a wholesaler I have been organising purchase with For sake of 2. I need to play safe for the time being, and suffer a few extra incoming spams a day. Once I have that sorted out, I will be ready to try greylisting again.
Re: spamd greylisting: false positives
Hi David, On 2012-05-27 11.51, David Diggles wrote: Hi again David. If all the spamd settings are back to default, I would recommend trying to pinpoint where the problem is. Just to check if it could be something wrong with the syntax of your pf rules regarding spamd, just comment them out. pfctl -f /etc/pf.conf and run for a while and see if you receive any mails. /Hasse I am running spamd in blacklist mode now, so I am once again receiving the mailing list. I think the default spamd timings do not give lists.openbsd.org enough time to retransmit in the whitelist window. It would be nice if someone else had the time to attempt reproducing this. No, I really don't think that's your problem. The default settings give spamd a four-hour window for retransmittions to occur. Judging from the log excerpt you included in one of your earlier mails the misc list server does a number of retries in that time and should easily make it. What does spamdb | fgrep 192.43.244.163 say? Could you add the '-v' (verbose) flag to your spamd startup options? That enables spamd to log the source and destination address for any mails it greylists, so you can check that you're actually getting the same tuples for each retry. Remember that greylisting is based on the combination of three entities, the source IP, the source mail address and the destination mail address (what's called a tuple in greylisting terminology). All three of those must be the same between retransmit attempts for the greylisting to recognize the attempt. Otherwise it will just add another entry to spamdb and mark it GREY, pending either timeout or another connect with the same tuple (in which case it either does nothing if the attempt was too soon or lets it through and changes the entry to WHITE in spamdb). Also remember that it actually takes three tries to get the mail through. First the initial attempt, which is rejected with a 451 temporary error. Then the next attempt, which gives the IP address clearance and adds it to the spamd-white PF table, but still results in a 451 temporary error back to the originating mail server. The third attempt will be routed by PF directly to your real SMTP server and will never be seen by spamd. I often find it helpful, when debugging mail problems such as yours, to simulate a sending SMTP server by using telnet your.smtp.server smtp and entering SMTP commands to the server manually, like this: $ telnet mailgate.internetlabbet.net 25 Trying 217.75.101.10... Connected to mailgate.internetlabbet.net. Escape character is '^]'. 220 mailgate.internetlabbet.net ESMTP spamd IP-based SPAM blocker; Sun May 27 12:36:26 2012 HELO lofgren.biz 250 Hello, spam sender. Pleased to be wasting your time. MAIL FROM: bl-li...@lofgren.biz 250 You are about to try to deliver spam. Your time will be spent, for nothing. RCPT TO: bl-li...@lofgren.biz 250 This is hurting you more than it is hurting me. DATA 451 Temporary failure, please try again later. Connection closed by foreign host. $ _ The above results in these lines in my syslog: May 27 12:36:26 fw1 spamd[836]: 66.7.199.108: connected (2/1) May 27 12:37:06 fw1 spamd[836]: (GREY) 66.7.199.108: bl-li...@lofgren.biz - bl-li...@lofgren.biz May 27 12:37:16 fw1 spamd[836]: 66.7.199.108: disconnected after 50 seconds. And the following in spamdb: # spamdb | fgrep 66.7.199.108 GREY|66.7.199.108|lofgren.biz|bl-li...@lofgren.biz|bl-li...@lofgren.biz|1338115026|1338201426|1338201426|1|0 Then I wait passtime minutes and retry the same thing, and check my logs and spamdb. Then I try again to verify that I'm this time indeed talking to my sendmail (or your smtp server of choice). Regards, /Benny This one you sent me earlier has some advice about tuning the timings, https://calomel.org/spamd_config.html In this section: We suggest setting the pass time to as high as you are comfortable with. Use a time between 10 and 55 minutes. You are welcome to set it as low as 2 minutes, but it is possible that some spammers might get white listed. After setting up spamd take some time, go through the logs and look for patterns. Adjust the pass time as necessary. ... I realise I have been advised in the list here not to mess around with the timings :P -- internetlabbet.se / work: +46 8 551 124 80 / Words must Benny Lofgren/ mobile: +46 70 718 11 90 / be weighed, / fax:+46 8 551 124 89/not counted. /email: benny -at- internetlabbet.se
Re: spamd greylisting: false positives
On 2012-05-27, David Diggles da...@elven.com.au wrote: What do you mean by running in blacklist mode ? Which settings are different from Grey trapping ? Are Openbsd mailing list the only list or mail you have problems with ? /Hasse By blacklist mode, I mean this: spamd -b spamd-setup -b pf.conf: table spamd persist pass in on egress proto tcp from spamd to any port smtp \ rdr-to 127.0.0.1 port spamd The OpenBSD mailing list was not the only smtp server I was having problems with. Others included: 1. test email from my work account 2. business email from a wholesaler I have been organising purchase with For sake of 2. I need to play safe for the time being, and suffer a few extra incoming spams a day. Once I have that sorted out, I will be ready to try greylisting again. Greylisting behaves a bit differently when used on a low-volume mail server compared to a large campus/organisation mail system where most of the big providers will be constantly auto-whitelisted due to the volume of mail they're sending (i.e. big sites get a lot more inbound mail so the problem mailhosts tend to stay in the database at most times). For the smaller setups I've built when I used greylisting I often needed to maintain separate whitelists. The only actively-maintained public whitelist I know of (dnswl.org) is only usable via DNS queries unless you pay for high volume access (and spamd doesn't support doing this via DNS lookups), so you are stuck with an outdated public list, and/or manual listings. Not sure if these particular ones are still a problem, but here are examples of some senders I had problems with at various times in the past (excessive delays or retries which were too slow to get past greylisting) : amazon, aol, bigfish, ebay, gmail, messagelabs, orange, paypal, postini, yahoo. Now I either use dnswl to exempt known problem hosts from greylistikg (with some manual whitelists on top; not done via spamd) - or in several cases just stopped using greylisting altogether.
Re: spamd greylisting: false positives
Hi everyone, sorry about the whiney tone. I am really appreciating all the help. On Sunday 27 May 2012, David Diggles wrote: This may seem like a dead horse to some by now, but I am disappointed
Re: spamd greylisting: false positives
Just made a minor change to pf.conf, to modulate state all tcp and keep state all udp: I am getting tired, it is late here. Hope I have not made any silly mistakes in this :D #--- # defaults #--- set loginterface egress match in all scrub (no-df max-mss 1440) antispoof quick for egress pass pass proto tcp modulate state pass proto udp keep state block in log on egress #--- # ssh #--- table ssh-black persist file /etc/pf/ssh-black table ssh-white persist file /etc/pf/ssh-white pass in on egress inet proto tcp from ssh-white to egress port ssh \ modulate state pass in on egress inet proto tcp from !ssh-black to egress port ssh \ modulate state \ (max-src-conn-rate 1/30, overload ssh-black flush) #--- # authpf #--- table authpf_users persist pass in on egress from authpf_users pass in on egress proto tcp from authpf_users modulate state pass in on egress proto udp from authpf_users keep state #--- # spamd - greylist mode #--- table spamd-white persist table nospamd persist file /etc/mail/nospamd pass in on egress proto tcp from any to egress port smtp \ rdr-to 127.0.0.1 port spamd pass in on egress proto tcp from nospamd to egress port smtp \ modulate state pass in log on egress proto tcp from spamd-white to egress port smtp \ modulate state pass out log on egress proto tcp to any port smtp modulate state #--- There is one GREY entry from lists.openbsd.org so far. root@skitL:~:0# spamdb|fgrep 192.43.244.163 GREY|192.43.244.163|shear.ucar.edu|owner-misc+M122933=david=elven.com...@openbsd.org|da...@elven.com.au|1338127686|1338142086|1338142086|1|0 root@skitL:~:0# date Mon May 28 00:44:18 EST 2012 root@skitL:~:0# date -r 1338127686 Mon May 28 00:08:06 EST 2012 I need to go sleep now, so I will check again in the morning before I go to work. Cheers, .d.d.
Re: spamd greylisting: false positives
After sleeping on it 6 hours, this is what I can report from the logs. root@skitL:log:0# cat spamd|fgrep 192.43.244.163|fgrep May 28 May 28 00:07:55 skitL spamd[21325]: 192.43.244.163: connected (1/0) May 28 00:08:06 skitL spamd[21325]: (GREY) 192.43.244.163: owner-misc+M122933=david=elven.com...@openbsd.org - da...@elven.com.au May 28 00:08:07 skitL spamd[21325]: 192.43.244.163: disconnected after 12 seconds. May 28 00:49:51 skitL spamd[20306]: 192.43.244.163: connected (1/0) May 28 00:50:03 skitL spamd[20306]: (GREY) 192.43.244.163: owner-misc+M122934=david=elven.com...@openbsd.org - da...@elven.com.au May 28 00:50:03 skitL spamd[20306]: 192.43.244.163: disconnected after 12 seconds. root@skitL:log:0# spamdb WHITE|202.58.38.80|||1338136570|1338140183|1341250605|2|0 TRAPPED|106.79.132.74|1338226638 TRAPPED|180.215.141.229|1338226988 GREY|186.206.211.111|baced36f.virtua.com.br|packer8...@reb.com|d...@elven.com.au|1338143338|1338157738|1338157738|1|0 GREY|95.180.252.146|59.167.212.41|and...@bb-dsh.org|d...@elven.com.au|1338152111|1338166511|1338166511|1|0 TRAPPED|64.20.227.133|1338241213 TRAPPED|217.149.28.204|1338241498 TRAPPED|174.123.14.196|1338232031 TRAPPED|83.169.61.34|1338235874 TRAPPED|95.180.252.146|1338238511 Bummer, I have forgotten to pflog the spamd connections to lo0 root@skitL:log:0# tcpdump -n -e -r /var/log/pflog port spamd tcpdump: WARNING: snaplen raised from 116 to 160 root@skitL:log:0# tcpdump -n -e -r /var/log/pflog port smtp tcpdump: WARNING: snaplen raised from 116 to 160 01:00:38.572058 rule 16/(match) pass out on xl0: 172.25.101.7.33057 66.49.254.25.25: S 3802061083:3802061083(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 2973717195[|tcp] (DF) 01:30:37.983151 rule 17/(match) pass out on xl0: 172.25.101.7.23127 66.49.254.25.25: S 3663599646:3663599646(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 862970203[|tcp] (DF) 04:36:24.378104 rule 16/(match) pass in on xl0: 202.58.38.80.25350 172.25.101.7.25: S 1021603063:1021603063(0) win 16384 mss 1420,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop 04:36:31.105838 rule 17/(match) pass out on xl0: 172.25.101.7.3605 173.194.79.27.25: S 2304184706:2304184706(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 451645464[|tcp] (DF) So I have just loaded a new pf.conf with logging turned on for spamd, this is what I have running now. #--- # defaults #--- set loginterface egress set skip on lo match in all scrub (no-df max-mss 1440) antispoof quick for egress pass pass proto tcp modulate state pass proto udp keep state block in log on egress #--- # ssh #--- table ssh-black persist file /etc/pf/ssh-black table ssh-white persist file /etc/pf/ssh-white pass in on egress inet proto tcp from ssh-white to egress \ port ssh modulate state pass in on egress inet proto tcp from !ssh-black to egress \ port ssh modulate state \ (max-src-conn-rate 1/30, overload ssh-black flush) #--- # squid #--- table squid-white persist file /etc/pf/squid-white pass in on egress inet proto tcp from squid-white to egress \ port 3128 modulate state #--- # authpf #--- table authpf_users persist pass in on egress from authpf_users pass in on egress proto tcp from authpf_users modulate state pass in on egress proto udp from authpf_users keep state #--- # spamd - greylist mode #--- table spamd-white persist table nospamd persist file /etc/mail/nospamd pass in log on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state pass in on egress proto tcp from nospamd to egress \ port smtp synproxy state pass in log on egress proto tcp from spamd-white to egress \ port smtp synproxy state pass out log on egress proto tcp to any port smtp modulate state #---
Re: spamd greylisting: false positives
On 2012-05-27, David Diggles da...@elven.com.au wrote: Bummer, I have forgotten to pflog the spamd connections to lo0 So this breaks spamlogd which means servers will expire from the greylist even if they mail you regularly..
Re: spamd greylisting: false positives
From: Stuart Henderson stu () spacehopper ! org Date: 2012-05-27 22:29:50 On 2012-05-27, David Diggles da...@elven.com.au wrote: Bummer, I have forgotten to pflog the spamd connections to lo0 So this breaks spamlogd which means servers will expire from the greylist even if they mail you regularly.. Do you mean this pf rule pass in log on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state breaks spamlogd? Would you mind explaining why, and how I can un-break it?
Re: spamd greylisting: false positives
Or did you mean, this breaks spamlogd, rather? pass in on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state This is what it was. The logging is on now. On Mon, May 28, 2012 at 08:53:09AM +1000, David Diggles wrote: From: Stuart Henderson stu () spacehopper ! org Date: 2012-05-27 22:29:50 On 2012-05-27, David Diggles da...@elven.com.au wrote: Bummer, I have forgotten to pflog the spamd connections to lo0 So this breaks spamlogd which means servers will expire from the greylist even if they mail you regularly.. Do you mean this pf rule pass in log on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state breaks spamlogd? Would you mind explaining why, and how I can un-break it?
Re: spamd greylisting: false positives
David Diggles da...@elven.com.au writes: Or did you mean, this breaks spamlogd, rather? pass in on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state This is what it was. The logging is on now. The important ones to log are the rules that pass smtp traffic from the members of the spamd-white table (and nospamd if you're using that) plus the one that passes smtp traffic from your real mail server to elsewhere. See the spamd and spamlogd man pages, it's explained there. But why are you synproxying for spamd? - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: spamd greylisting: false positives
On Sun, May 27, 2012 at 7:19 PM, Peter N. M. Hansteen pe...@bsdly.net wrote: David Diggles da...@elven.com.au writes: Or did you mean, this breaks spamlogd, rather? pass in on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state This is what it was. The logging is on now. The important ones to log are the rules that pass smtp traffic from the members of the spamd-white table (and nospamd if you're using that) plus the one that passes smtp traffic from your real mail server to elsewhere. See the spamd and spamlogd man pages, it's explained there. But why are you synproxying for spamd? - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds. Perhaps off-topic, but I've been working under the assumption that synproxy (or some portion thereof) has been broken for the last few releases. Whether or not it was actually useful to my configuration, enabling synproxy shouldn't hurt anything, right? Because switching from synproxy to modulate state seemed to fix intermittent download issues for Windows machines sitting behind my pf NAT. Never had an issue on the OpenBSD device itself, FWIW. Can anyone confirm with better-than-anecdotal knowledge? --david
Re: spamd greylisting: false positives
List: openbsd-misc Subject:Re: spamd greylisting: false positives From: peter () bsdly ! net (Peter N ! M ! Hansteen) Date: 2012-05-27 23:19:47 Message-ID: 87sjel43fw.fsf () deeperthought ! bsdly ! net [Download message RAW] Or did you mean, this breaks spamlogd, rather? pass in on egress proto tcp from any to egress \ port smtp rdr-to 127.0.0.1 port spamd synproxy state This is what it was. The logging is on now. The important ones to log are the rules that pass smtp traffic from the members of the spamd-white table (and nospamd if you're using that) plus the one that passes smtp traffic from your real mail server to elsewhere. See the spamd and spamlogd man pages, it's explained there. Ok, I was doing this. I just started logging the rdr-to spamd rule too. But why are you synproxying for spamd? Why shouldn't I? These guys do in their example. https://calomel.org/spamd_config.html delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds. It's cool to see an on-topic sig. .d.d.
Re: spamd greylisting: false positives
It amazes me that nobody has yet given you the calomel warning. Not the best source of clues. That is the most polite comment you will see about that website. On Mon, 28 May 2012 10:43:08 +1000, David Diggles wrote: These guys do in their example. https://calomel.org/spamd_config.html delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds. It's cool to see an on-topic sig. *** NOTE *** Please DO NOT CC me. I am subscribed to the list. Mail to the sender address that does not originate at the list server is tarpitted. The reply-to: address is provided for those who feel compelled to reply off list. Thankyou. Rod/ --- This life is not the real thing. It is not even in Beta. If it was, then OpenBSD would already have a man page for it.
Re: spamd greylisting: false positives
But why are you synproxying for spamd? Why shouldn't I? These guys do in their example. https://calomel.org/spamd_config.html don't ever recommend calomel on a openbsd mailing list, search the archives for why. here's a hint: they work spectacularly
Re: spamd greylisting: false positives
David Diggles da...@elven.com.au writes: But why are you synproxying for spamd? Why shouldn't I? The synproxy was added way back as a way to protect back ends that were less intelligent about connection setup and IIRC even had one or more known SYN-related vulnerabilities, so we had a way to only pass valid, completed connections. In relation to spamd, it doesn't add any security, but carries with it the slight overhead of the syn proxying. These guys do in their example. https://calomel.org/spamd_config.html I'd ask them the same question. It rarely if ever makes sense to pile on options just because they're available. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: spamd greylisting: false positives
Ok, I took synproxy out. What about modulate state? The pf.conf example in spamd(8) does not include it. I think I can guess, the answer will be: not needed Oh, thanks for the heads up about calomel.org. Someone else on list recommended it to me.
Re: spamd greylisting: false positives
Ok, I searched calomel and had a good laugh. smells like calomel
Re: spamd greylisting: false positives
-Ursprungligt meddelande- Fren: owner-m...@openbsd.org [mailto:owner-m...@openbsd.org] Fvr David Diggles Skickat: den 28 maj 2012 03:54 Till: misc@openbsd.org Dmne: Re: spamd greylisting: false positives Ok, I searched calomel and had a good laugh. smells like calomel Grow up ! I recommended Calomel because that site gave me some good advice and understanding of spamd. First I tried Peters site, with no up to dates rules. Therefore I don't recommend it. Otherwise Peter is my kind of Guru but a bit focused on selling his books, and therefore dont Want to give away the full recipe for free. As always, you can not expect copy and paste. But a site that realy made a difference for me and my use of spamd : http://www.benzedrine.cx/relaydb.html /hasse
Re: spamd greylisting: false positives
Solved! I caused the cause of the problem with misconfigured DNS. I had a secondary MX defined in DNS for elven.com.au that is not yet configured to receive for elven.com.au. I tested again from work, and got this error: - The following addresses had permanent fatal errors - da...@elven.com.au (reason: 550 relay not permitted) - Transcript of session follows - ... while talking to mx.bincrow.net.: DATA 451 Temporary failure, please try again later. ... while talking to smtp.free2air.net.: DATA 550 relay not permitted 550 5.1.1 da...@elven.com.au... User unknown 503 valid RCPT command must precede DATA First attempt goes to the primary MX, mx.bincrow.net (where I have spamd running). Gets the 451, then retries on the secondary MX, gets 550 user unknown, then gives up (I verified this by watching the spamdb output and the counter in tuples of problem emails stayed at 1) I have removed the secondary MX from the DNS for elven.com.au then lists.openbsd.org did the expected retries and became whitelisted.
Re: spamd greylisting: false positives
So there you have it. Don't use spamd with greytrapping if your secondary MX is going to deliver a bounce. It will confuse SMTP servers into giving up. On Mon, May 28, 2012 at 03:38:16PM +1000, David Diggles wrote: I had a secondary MX defined in DNS for elven.com.au that is not yet configured to receive for elven.com.au.
Re: spamd greylisting: false positives
On 2012-05-25, David Diggles da...@elven.com.au wrote: I wasn't receiving email, from lists.openbsd.org and also from my work email address, until I added the respective smtp servers to the whitelist table in pf. do you have spamlogd running? Seriously though, if I have to keep manually adding smtp servers to a whitelist, I will run in blacklist only mode. yes, you do, various large sites use either pools of senders with a shared queue, or senders behind large nats, or bad retry cycles etc. you really need something like the dnswl list (only available by dns lookup for the mos part). one thing that can help is to restrict spamd to only affecting windows hosts (using 'from any os windows' in pf rules).
Re: spamd greylisting: false positives
This may seem like a dead horse to some by now, but I am disappointed
no one replied to the msg, I supplied the detailed event information with
timestamps, regarding lists.openbsd.org mails not being whitelisted by
spamd when run in greylist mode.
RFC282, 4.5.4.1 Sending Strategy:
The sender MUST delay retrying a particular destination after one
attempt has failed. In general, the retry interval SHOULD be at
least 30 minutes; however, more sophisticated and variable strategies
will be beneficial when the SMTP client can determine the reason for
non-delivery.
Retries continue until the message is transmitted or the sender gives
up; the give-up time generally needs to be at least 4-5 days. The
parameters to the retry algorithm MUST be configurable.
Yet I have been advised not to mess with the default timings with -G option.
It looks to me like the retry intervals of lists.openbsd.org are not
sufficient to get it whitelisted by spamd.
I am well beyond assuming anything, and prepared to learn / accept
any constructive advice.
Can anyone confirm they have the following scenario?
* A clean installed OpenBSD 5.1 configured as a primary MX
* Clean spamd settings, clean /var/db/spamd
* Default spamd with no options
* Default spamlogd with no options
* The pf.conf uses spamd entries from the example pf.conf from etc.tgz
* No manual whitelist entry for lists.openbsd.org
* Incoming from lists.openbsd.org is eventually whitelisted by spamd
I am just trying to learn the cause, and I have been fully prepared to wear
egg on my face if my own configuration is causing the problem. I have not
yet proven this is the case.
I believe I have checked everything anyone suggested to check.
I really don't want my next check be to roll back to 4.9 and see if
lists.openbsd.org will auto whitelist like it previously did.
In hope,
David
On Sat, May 26, 2012 at 01:19:38PM +1000, David Diggles wrote:
Ok I am still not getting emails from
lists.openbsd.org (so please if you reply, cc to me).
I restarted spamd at this time after deleting /var/db/spamd and
clearing the bypass tables in pf at this time:
2012-05-26 02:13:12 # /usr/libexec/spamd
Here is the last message to make it to sendmail from misc:
fgrep from= /var/log/maillog|fgrep owner-misc|tail -1|awk '{print $1,$2,$3}'
May 26 01:54:35
The pf rules for spamd I have are taken from the default pf.conf:
pass in on egress inet proto tcp from any to any port = 25 flags S/SA rdr-to
127.0.0.1 port 8025
pass in on egress proto tcp from nospamd to any port = 25 flags S/SA
pass in log on egress proto tcp from spamd-white to any port = 25 flags S/SA
pass out log on egress proto tcp from any to any port = 25 flags S/S
It is currently Sat May 26 12:54:31 EST 201
Times of passed smtp connections for May 26:
tcpdump -n -e -ttt -r /var/log/pflog 21|fgrep .25:|\
fgrep 'May 26'|awk '{print $3}'
01:14:53.793995
04:17:11.846707
05:00:19.443080
05:15:01.487277
07:17:51.114440
09:35:58.120098
10:14:21.444822
11:53:33.611903
So I will skip the first entry when I grep for the
ip addresses, with a tail +2 because it occurred
*before* I reset everything.
tcpdump -n -e -ttt -r /var/log/pflog 21|fgrep .25:|\
fgrep 'May 26'|awk '{print $10}'|tail +2|\
awk -F. '{print $1.$2.$3.$4}'|sort -n
17.254.6.112
74.125.82.47
113.172.232.215
129.21.208.44
202.58.38.80
203.59.1.110
206.46.252.115
I have the following tables.
pfctl -s Tables
nospamd
spamd-white
Confirming against the spamd-white table
pfctl -t spamd-white -Ts
17.254.6.112
74.125.82.47
113.172.232.215
129.21.208.44
202.58.38.80
203.59.1.110
206.46.252.115
lists.openbsd.org = 192.43.244.163
So nothing from misc has made it to sendmail since I emptied
nospamd and spamd-white on pf.conf
These are all the attempts from lists.openbsd.org since
I cleared the spamdb and pf tables.
fgrep 192.43.244.163 /var/log/spamd|fgrep 'May 26'
May 26 02:53:48 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 02:54:00 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 03:00:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 03:00:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 04:41:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 04:41:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:04:19 skitL spamd[25502]: 192.43.244.163: connected (2/1)
May 26 05:04:31 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:15:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:15:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:19:36 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:19:48 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:26:38 skitL spamd[25502]: 192.43.244.163: connected
Re: spamd greylisting: false positives
/var/log/spamd spamd[11000]: queueing deletion of x.x.x.x mx1.example.com f...@example.com da...@elven.com.au spamd[11000]: queueing deletion of y.y.y.y mx2.example.com f...@example.com da...@elven.com.au Both of these emails I wished to receive, as I corresponded with them yesterday. :( I am now trying spamd with the following: /usr/libexec/spamd -d -G5:1:864 On Fri, May 25, 2012 at 11:26:19AM +0530, Girish Venkatachalam wrote: Dear David, I don't understand how spamd deletes any message. It never does. Can you clarify? I am certain your observation is wrong; it is something else. I wish to know more. Thanks. -Girish On Fri, May 25, 2012 at 10:56 AM, David Diggles da...@elven.com.au wrote: Since upgrading from 4.9 to 5.1, I am getting a lot of false positives with spamd running in greylisting mode, from email addresses I previously did not. A number of false negatives are still getting through, too. Eg: I needed to add lists.openbsd.org to /etc/mail/nospamd to receive messages from this list, or spamdb would delete the messages. How can I tune this to make spamd more forgiving? ?The default -G options in spamd(8) do not seem to be taking effect. ?I emailed someone yesterday, received a reply. Then today, they emailed me again from the same address and their messages were deleted by spamdb. What conditions cause an address to move from grey to white, or grey to black? Trying to get my head around this, and losing emails all the while.. .d.d. -- Gayatri Hitech http://gayatri-hitech.com
Re: spamd greylisting: false positives
Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message?
Re: spamd greylisting: false positives
Here are the logs for my failed attempts at joining the misc mailing list. All with default spamd settings. Like I said, it did not succeed until I added lists.openbsd.org to the /etc/mail/nospamd and reloaded the pf rule. May 15 23:48:58 mx spamd[6698]: new entry 192.43.244.163 from owner-majord...@openbsd.org to da...@elven.com.au, helo shear.ucar.edu May 15 23:48:58 mx spamd[6698]: new entry 192.43.244.163 from owner-majordomo+tf691-ac35-2...@openbsd.org to da...@elven.com.au, helo shear.ucar.edu May 16 03:49:42 mx spamd[11000]: queueing deletion of 192.43.244.163 shear.ucar.edu owner-majord...@openbsd.org da...@elven.com.au May 16 03:49:42 mx spamd[11000]: queueing deletion of 192.43.244.163 shear.ucar.edu owner-majordomo+tf691-ac35-2...@openbsd.org da...@elven.com.au May 16 06:17:03 mx spamd[6698]: new entry 192.43.244.163 from owner-majordomo+t7fa9-267c-0...@openbsd.org to da...@elven.com.au, helo shear.ucar.edu May 16 06:17:03 mx spamd[6698]: new entry 192.43.244.163 from owner-majord...@openbsd.org to da...@elven.com.au, helo shear.ucar.edu May 16 06:23:34 mx spamd[6698]: new entry 192.43.244.163 from owner-majordomo+t380a-276c-0...@openbsd.org to da...@elven.com.au, helo shear.ucar.edu May 16 06:28:40 mx spamd[6698]: new entry 192.43.244.163 from owner-majordomo+t26ad-c7af-0...@openbsd.org to da...@elven.com.au, helo shear.ucar.edu May 16 10:17:46 mx spamd[11000]: queueing deletion of 192.43.244.163 shear.ucar.edu owner-majordomo+t7fa9-267c-0...@openbsd.org da...@elven.com.au May 16 10:17:46 mx spamd[11000]: queueing deletion of 192.43.244.163 shear.ucar.edu owner-majord...@openbsd.org da...@elven.com.au May 16 10:23:47 mx spamd[11000]: queueing deletion of 192.43.244.163 shear.ucar.edu owner-majordomo+t380a-276c-0...@openbsd.org da...@elven.com.au May 16 10:28:47 mx spamd[11000]: queueing deletion of 192.43.244.163 shear.ucar.edu owner-majordomo+t26ad-c7af-0...@openbsd.org da...@elven.com.au On Fri, May 25, 2012 at 11:26:19AM +0530, Girish Venkatachalam wrote: Dear David, I don't understand how spamd deletes any message. It never does. Can you clarify? I am certain your observation is wrong; it is something else. I wish to know more. Thanks. -Girish On Fri, May 25, 2012 at 10:56 AM, David Diggles da...@elven.com.au wrote: Since upgrading from 4.9 to 5.1, I am getting a lot of false positives with spamd running in greylisting mode, from email addresses I previously did not. A number of false negatives are still getting through, too. Eg: I needed to add lists.openbsd.org to /etc/mail/nospamd to receive messages from this list, or spamdb would delete the messages. How can I tune this to make spamd more forgiving? ?The default -G options in spamd(8) do not seem to be taking effect. ?I emailed someone yesterday, received a reply. Then today, they emailed me again from the same address and their messages were deleted by spamdb. What conditions cause an address to move from grey to white, or grey to black? Trying to get my head around this, and losing emails all the while.. .d.d. -- Gayatri Hitech http://gayatri-hitech.com
Re: spamd greylisting: false positives
On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? A qualified yes. The message isn't dropped if the sending server fails to resend before greyexp hours, it is dropped the first time delivery is attempted; if other attempts to deliver occur before passtime minutes pass, or after greyexp hours, the message will continue to be dropped. You reduced the whitelisting interval from (25 minutes, 240 minutes] to (5 minutes, 60 minutes], a pretty big cut. Perhaps that is your problem. -- Matthew Weigel hacker unique idempot . ent
Re: spamd greylisting: false positives
On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? It's dropped when it's first received, and it will continue to get dropped until passtime minutes have passed. If it is then received before greyexp hours have passed, it will be delivered and the remote host will be whitelisted for sending mail. If greyexp hours pass without seeing that tuple again, the tuple is deleted and it's back to the beginning for that host. You reduced greyexp to 1 hour, which may well be causing your problems. -- Matthew Weigel hacker unique idempot . ent
Re: spamd greylisting: false positives
On Thu, May 24, 2012 at 11:09 PM, David Diggles da...@elven.com.au wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? Yes, but that is entirely up to the sending mailserver. If you do not receive a message that was initially greylisted and then subsequently whitelisted by hand (spamdb -a x.x.x.x) it is because the sending mailserver failed to re-send the message, not because spamd dropped it. spamd does not delete e-mail, it refuses to accept e-mail. Read spamdb(8); I cannot find immediate references, but I am pretty sure those log messages refer to the deletion of entries from the database. Also read the GREYTRAPPING section of spamd(8), spamd.alloweddomains may be of some use to you.
Re: spamd greylisting: false positives
Like I said, it was in default mode when this behavior started. Now I am messin with the timings trying to overcome this dropping of messages. Are you saying I should be increasing this from 25 minutes? On Fri, May 25, 2012 at 02:03:03AM -0500, Matthew Weigel wrote: On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? A qualified yes. The message isn't dropped if the sending server fails to resend before greyexp hours, it is dropped the first time delivery is attempted; if other attempts to deliver occur before passtime minutes pass, or after greyexp hours, the message will continue to be dropped. You reduced the whitelisting interval from (25 minutes, 240 minutes] to (5 minutes, 60 minutes], a pretty big cut. Perhaps that is your problem. -- Matthew Weigel hacker unique idempot . ent
Re: spamd greylisting: false positives
Oh, so if I am relying on remote mailservers being configured to resend after a temporary failure, how do I second guess the time intervals they are configured with? If they even resend at all? Eg: lists.openbsd.org failed with default grey settings in spamd. I guess I don't have the skills to run a greytrappin spamd, and still receive the mails I want to receive, and ought to run in blacklist mode instead ;-) On Fri, May 25, 2012 at 02:03:03AM -0500, Matthew Weigel wrote: On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? A qualified yes. The message isn't dropped if the sending server fails to resend before greyexp hours, it is dropped the first time delivery is attempted; if other attempts to deliver occur before passtime minutes pass, or after greyexp hours, the message will continue to be dropped. You reduced the whitelisting interval from (25 minutes, 240 minutes] to (5 minutes, 60 minutes], a pretty big cut. Perhaps that is your problem. -- Matthew Weigel hacker unique idempot . ent
Re: spamd greylisting: false positives
I am now trying it with -G120:6:864 Although I can't think how to reproduce the problem in a controlled way, other than wait and see what emails I don't get :/ On Fri, May 25, 2012 at 02:07:33AM -0500, Matthew Weigel wrote: On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? It's dropped when it's first received, and it will continue to get dropped until passtime minutes have passed. If it is then received before greyexp hours have passed, it will be delivered and the remote host will be whitelisted for sending mail. If greyexp hours pass without seeing that tuple again, the tuple is deleted and it's back to the beginning for that host. You reduced greyexp to 1 hour, which may well be causing your problems. -- Matthew Weigel hacker unique idempot . ent
Re: spamd greylisting: false positives
* David Diggles da...@elven.com.au [2012-05-25 09:18]: Like I said, it was in default mode when this behavior started. Now I am messin with the timings trying to overcome this dropping of messages. Are you saying I should be increasing this from 25 minutes? the defaults are fine, afaict almost everybody is running with them. that is not your problem, something in your setup is very wrong. first sanity check would be the clock, tho I have a hard time seeing how it could jump repeatedly. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: spamd greylisting: false positives
On Fri, 25 May 2012 17:22:04 +1000 David Diggles wrote: Eg: lists.openbsd.org failed with default grey settings in spamd. I find it hard to believe lists.openbsd.org isn't RFC compliant. I guess you have another problem. If you send me an address privately. I'll send a mail from Yahoo. I know Yahoo mails get through tighter than default settings.
Re: spamd greylisting: false positives
-Ursprungligt meddelande- Fren: owner-m...@openbsd.org [mailto:owner-m...@openbsd.org] Fvr David Diggles Skickat: den 25 maj 2012 11:14 Till: misc@openbsd.org Dmne: Re: spamd greylisting: false positives I am now trying it with -G120:6:864 Although I can't think how to reproduce the problem in a controlled way, other than wait and see what emails I don't get :/ On Fri, May 25, 2012 at 02:07:33AM -0500, Matthew Weigel wrote: On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? It's dropped when it's first received, and it will continue to get dropped until passtime minutes have passed. If it is then received before greyexp hours have passed, it will be delivered and the remote host will be whitelisted for sending mail. If greyexp hours pass without seeing that tuple again, the tuple is deleted and it's back to the beginning for that host. You reduced greyexp to 1 hour, which may well be causing your problems. -- Matthew Weigel hacker unique idempot . ent Ahh... Just struck me Please check the syntax of your pf rules This is what's working for me : table spamd-white persist pass in log on egress proto tcp from spamd-white rdr-to 127.0.0.1 port smtp pass in log on egress proto tcp from !spamd-white rdr-to 127.0.0.1 port spamd /Hasse
Re: spamd greylisting: false positives
-Ursprungligt meddelande- Fren: owner-m...@openbsd.org [mailto:owner-m...@openbsd.org] Fvr David Diggles Skickat: den 25 maj 2012 11:14 Till: misc@openbsd.org Dmne: Re: spamd greylisting: false positives I am now trying it with -G120:6:864 Although I can't think how to reproduce the problem in a controlled way, other than wait and see what emails I don't get :/ On Fri, May 25, 2012 at 02:07:33AM -0500, Matthew Weigel wrote: On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? It's dropped when it's first received, and it will continue to get dropped until passtime minutes have passed. If it is then received before greyexp hours have passed, it will be delivered and the remote host will be whitelisted for sending mail. If greyexp hours pass without seeing that tuple again, the tuple is deleted and it's back to the beginning for that host. You reduced greyexp to 1 hour, which may well be causing your problems. -- Matthew Weigel hacker unique idempot . ent Hello Not a behavior I can recognize. I would recommend to start over the configuration from the beginning, after checking the obvious system settings. Standard settings should be fine as a starter. Later on, adjust to your likings. You can find some good instructions (explainations) here : http://www.pantz.org/software/spamd/configspamd.html https://calomel.org/spamd_config.html Regards Hasse
Re: spamd greylisting: false positives
The spamd pf.conf rules I have are: table spamd-white persist table nospamd persist file /etc/mail/nospamd pass in on egress proto tcp from any to any port smtp \ rdr-to 127.0.0.1 port spamd pass in on egress proto tcp from nospamd to any port smtp pass in log on egress proto tcp from spamd-white to any port smtp pass out log on egress proto tcp to any port smtp Henning, the clock seems fine. Ntpd is not complaining about losing time. I will return all the spamd options to default. spamd-setup is running from cron, 13 mins after every hour. On 15th of May, I upgraded to 5.1 with a clean install. Maybe the problem is not spamd, but my configuration of sendmail. On Fri, May 25, 2012 at 12:20:45PM +0200, obsd wrote: -Ursprungligt meddelande- Fren: owner-m...@openbsd.org [mailto:owner-m...@openbsd.org] Fvr David Diggles Skickat: den 25 maj 2012 11:14 Till: misc@openbsd.org Dmne: Re: spamd greylisting: false positives I am now trying it with -G120:6:864 Although I can't think how to reproduce the problem in a controlled way, other than wait and see what emails I don't get :/ On Fri, May 25, 2012 at 02:07:33AM -0500, Matthew Weigel wrote: On 25.05.2012 01:09, David Diggles wrote: Can messages get dropped if mail servers fail to resend within time interval, after receiving the initial temporary failure message? It's dropped when it's first received, and it will continue to get dropped until passtime minutes have passed. If it is then received before greyexp hours have passed, it will be delivered and the remote host will be whitelisted for sending mail. If greyexp hours pass without seeing that tuple again, the tuple is deleted and it's back to the beginning for that host. You reduced greyexp to 1 hour, which may well be causing your problems. -- Matthew Weigel hacker unique idempot . ent Ahh... Just struck me Please check the syntax of your pf rules This is what's working for me : table spamd-white persist pass in log on egress proto tcp from spamd-white rdr-to 127.0.0.1 port smtp pass in log on egress proto tcp from !spamd-white rdr-to 127.0.0.1 port spamd /Hasse
Re: spamd greylisting: false positives
David Diggles wrote: I am now trying it with -G120:6:864 Although I can't think how to reproduce the problem in a controlled way, other than wait and see what emails I don't get :/ Stop playing with those settings, you are freaking out about log entries that don't mean what you think they mean. spamd seems to have a queue for updates to the database. This is based off of a 2 minute look at the code in question, but it seems to use the to do all changes from a scan run at once, so it can do all the necessary changes in one burst. What you are seeing are messages about it deleting stuff from its own database, not email. spamd doesn't accept email. The email never gets to your smtp daemon until it's gotten through spamd's delays. Are you actually not receiving email? Or assuming there is a problem based upon your panicked look at the logs? If emails aren't getting through, they may either need to be explicitly whitelisted (providers like gmail do things that mean they may never get through graylisting), or you have misconfigured your setup. --Kurt
Re: spamd greylisting: false positives
I wasn't receiving email, from lists.openbsd.org and also from my work email address, until I added the respective smtp servers to the whitelist table in pf. I could see them in the greylist when I typed spamdb. Yes. I did misunderstand the spamd log entry about deletion. Though I would not bother playing with settings if the expected emails were being received. I will go ahead and flush the spamdb database, and the pf tables and start over with default everything, no whitelist pf entries. This time I will sit on my hands and wait. Maybe I was not being patient enough. When you say I misconfigured my setup, what do you mean? What else is there to spamd other than adding it to rc.conf.local and a few pf rules? As for gmail; I have not had this issue sending email from gmail to spamd. Seriously though, if I have to keep manually adding smtp servers to a whitelist, I will run in blacklist only mode. On Fri, May 25, 2012 at 11:07:12AM -0400, Kurt Mosiejczuk wrote: David Diggles wrote: I am now trying it with -G120:6:864 Although I can't think how to reproduce the problem in a controlled way, other than wait and see what emails I don't get :/ Stop playing with those settings, you are freaking out about log entries that don't mean what you think they mean. spamd seems to have a queue for updates to the database. This is based off of a 2 minute look at the code in question, but it seems to use the to do all changes from a scan run at once, so it can do all the necessary changes in one burst. What you are seeing are messages about it deleting stuff from its own database, not email. spamd doesn't accept email. The email never gets to your smtp daemon until it's gotten through spamd's delays. Are you actually not receiving email? Or assuming there is a problem based upon your panicked look at the logs? If emails aren't getting through, they may either need to be explicitly whitelisted (providers like gmail do things that mean they may never get through graylisting), or you have misconfigured your setup. --Kurt
Re: spamd greylisting: false positives
On 25.05.2012 10:50, David Diggles wrote: I wasn't receiving email, from lists.openbsd.org and also from my work email address, until I added the respective smtp servers to the whitelist table in pf. I could see them in the greylist when I typed spamdb. In the greylist, or in the whitelist (both are stored in /var/db/spamdb)? I'm wondering now whether your /var/db/spamdb got wiped out when you upgraded. If that happened, then all pre-existing whitelist entries would be gone, and emails would have to go through greylisting again. Also, if your standard procedure when making changes was as below (wiping out spamdb), you would be pretty much guaranteed to drop a lot of mail on the floor given exponential back off. I will go ahead and flush the spamdb database, and the pf tables and start over with default everything, no whitelist pf entries. Presumably you have at least some whitelist entries there, and some mail in transit that you would like to eventually receive. Flushing the database now would mean that anything currently greylisted is very unlikely to be whitelisted, and anything whitelisted will be greylisted next time it tries to deliver mail. This time I will sit on my hands and wait. Maybe I was not being patient enough. With default settings, you need to be patient for 4 hours. Past 4 hours, the chances are close to nil that you'll get that mail. Until 4 hours have passed, though, it's completely possible you'll still receive the mail. As for gmail; I have not had this issue sending email from gmail to spamd. You will. Seriously though, if I have to keep manually adding smtp servers to a whitelist, I will run in blacklist only mode. It's pretty straightforward to script pulling SPF records from Google and whitelisting them. Facebook is another company that sends a lot of mail through many servers, but documents those servers in SPF records you can poll (say, on a weekly basis). There are very few other mail server clusters that have that behavior, so once you identify those two, and script it, the problem is basically solved. For example, you could move your current nospamd file to /etc/mail/nospamd.constant, and then do the following in /etc/weekly.local: next_part Whitelisting Google mail servers /usr/sbin/dig _spf.google.com TXT + short | tr \ \n | grep ip4: \ | cut -d: -f2 | sort -n /etc/mail/nospamd.dynamic cat /etc/mail/nospamd.constant /etc/mail/nospamd.dynamic /etc/mail/nospamd /sbin/pfctl -t gmail-white -T replace -f /etc/mail/nospamd 21 \ | grep -v 'no changes' That's very close to something someone else shared on misc@ many moons ago, I don't remember who. -- Matthew Weigel hacker unique idempot . ent
Re: spamd greylisting: false positives
On Sat, May 26, 2012 at 01:50:40AM +1000, David Diggles wrote: I will go ahead and flush the spamdb database, and the pf tables and start over with default everything, no whitelist pf entries. spamd acts up for me occasionally. In such cases I just /etc/rc.d/spamd stop rm /var/db/spamd /etc/rc.d/spamd start When you say I misconfigured my setup, what do you mean? He might have assumed that because you described your problem more with narration than with logs or copy/paste, and when you did copy and paste, it was incomplete. For example: I am now trying it with -G120:6:864 Is that from rc.conf.local? The command line? If it was the command line, were you calling spamd directly, or using rc.d? Is that the entire argument list passed? Only what you thought was relevant to the issue? We don't know. And when you say spamd was deleting email... that's just not possible because spamd returns a 4xy/5xy after DATA. There's no reason to narrate when logs tell a better story. Nicolai
Re: spamd greylisting: false positives
Thanks for also replying directly. Since I cleared nospamd override table in pf, I am no longer receiving emails from misc. I wasn't receiving email, from lists.openbsd.org and also from my work email address, until I added the respective smtp servers to the whitelist table in pf. I could see them in the greylist when I typed spamdb. And how many attempts were listed? Was there any whitelist entries when you checked? How long had it been since the email was sent and you still hadn't received it? Some emails did not get through at all. I didn't check for whitelist entries or number of attempts at the time. Yes. I did misunderstand the spamd log entry about deletion. Though I would not bother playing with settings if the expected emails were being received. The default graylist settings are almost certainly the only thing definitively not causing your problems. And the way you tweaked them were more likely to completely break things. Ok, I have returned the greylist settings to default now and I will go through the pf log and try find out why lists.openbsd.org is once again not making it to my inbox. I will go ahead and flush the spamdb database, and the pf tables and start over with default everything, no whitelist pf entries. That almost certainly will *not* help. The only real entries that cause problems are TRAPPED entries. GREY entries mean the sender is partway through getting whitelisted. If you delete the entries, you set them back to square one. And since they have been backing off on trying, the odds are much greater that the sending side will give up. I am not trying to help, I am reproducing the problem by reverting to the original conditions. Now it's the weekend, I have time to spend on solving it. This time I will sit on my hands and wait. Maybe I was not being patient enough. With greylisting, that is usually the case. Once it has been running for a while, most things have no delays, but until a sender has been whitelisted by getting through the greylisting process, you will have probably an hour delay. I restarted spamd and flushed all the pf tables almost 10 hours ago now. Still nothing from lists.openbsd.org. When you say I misconfigured my setup, what do you mean? What else is there to spamd other than adding it to rc.conf.local and a few pf rules? The pf rules are important. You get them wrong and it doesn't work. The spamd pf rules I am using are simply uncommented from the default OpenBSD pf.conf file. I will audit pf rules and ensure I don't have other rules interfering with spamd. Seriously though, if I have to keep manually adding smtp servers to a whitelist, I will run in blacklist only mode. Generally you don't need to, but there *are* misbehaving setups out there, and if you never monitor your email server, they won't get through. A quick google search turns up http://www.greylisting.org/whitelisting.shtml, which can help you pre-populate entries if you can't be bothered. Thanks for googling that one for me. I must have a misconfiguration. My apologies for dumping on the list, wasting peoples time etc.
Re: spamd greylisting: false positives
Ok I am still not getting emails from
lists.openbsd.org (so please if you reply, cc to me).
I restarted spamd at this time after deleting /var/db/spamd and
clearing the bypass tables in pf at this time:
2012-05-26 02:13:12 # /usr/libexec/spamd
Here is the last message to make it to sendmail from misc:
fgrep from= /var/log/maillog|fgrep owner-misc|tail -1|awk '{print $1,$2,$3}'
May 26 01:54:35
The pf rules for spamd I have are taken from the default pf.conf:
pass in on egress inet proto tcp from any to any port = 25 flags S/SA rdr-to
127.0.0.1 port 8025
pass in on egress proto tcp from nospamd to any port = 25 flags S/SA
pass in log on egress proto tcp from spamd-white to any port = 25 flags S/SA
pass out log on egress proto tcp from any to any port = 25 flags S/S
It is currently Sat May 26 12:54:31 EST 201
Times of passed smtp connections for May 26:
tcpdump -n -e -ttt -r /var/log/pflog 21|fgrep .25:|\
fgrep 'May 26'|awk '{print $3}'
01:14:53.793995
04:17:11.846707
05:00:19.443080
05:15:01.487277
07:17:51.114440
09:35:58.120098
10:14:21.444822
11:53:33.611903
So I will skip the first entry when I grep for the
ip addresses, with a tail +2 because it occurred
*before* I reset everything.
tcpdump -n -e -ttt -r /var/log/pflog 21|fgrep .25:|\
fgrep 'May 26'|awk '{print $10}'|tail +2|\
awk -F. '{print $1.$2.$3.$4}'|sort -n
17.254.6.112
74.125.82.47
113.172.232.215
129.21.208.44
202.58.38.80
203.59.1.110
206.46.252.115
I have the following tables.
pfctl -s Tables
nospamd
spamd-white
Confirming against the spamd-white table
pfctl -t spamd-white -Ts
17.254.6.112
74.125.82.47
113.172.232.215
129.21.208.44
202.58.38.80
203.59.1.110
206.46.252.115
lists.openbsd.org = 192.43.244.163
So nothing from misc has made it to sendmail since I emptied
nospamd and spamd-white on pf.conf
These are all the attempts from lists.openbsd.org since
I cleared the spamdb and pf tables.
fgrep 192.43.244.163 /var/log/spamd|fgrep 'May 26'
May 26 02:53:48 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 02:54:00 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 03:00:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 03:00:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 04:41:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 04:41:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:04:19 skitL spamd[25502]: 192.43.244.163: connected (2/1)
May 26 05:04:31 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:15:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:15:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:19:36 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:19:48 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:26:38 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:26:50 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:31:10 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:31:22 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:37:54 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:38:06 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 05:43:38 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 05:43:50 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 06:32:55 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 06:33:08 skitL spamd[25502]: 192.43.244.163: disconnected after 13
seconds.
May 26 07:00:31 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 07:00:43 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 07:29:59 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 07:30:11 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 07:53:46 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 07:53:58 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 08:26:24 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 08:26:36 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 09:14:32 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 09:14:44 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 10:12:59 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 10:13:10 skitL spamd[25502]: 192.43.244.163: disconnected after 11
seconds.
May 26 11:44:37 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 11:44:49 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
May 26 11:54:40 skitL spamd[25502]: 192.43.244.163: connected (1/0)
May 26 11:54:52 skitL spamd[25502]: 192.43.244.163: disconnected after 12
seconds.
spamd greylisting: false positives
Since upgrading from 4.9 to 5.1, I am getting a lot of false positives with spamd running in greylisting mode, from email addresses I previously did not. A number of false negatives are still getting through, too. Eg: I needed to add lists.openbsd.org to /etc/mail/nospamd to receive messages from this list, or spamdb would delete the messages. How can I tune this to make spamd more forgiving? The default -G options in spamd(8) do not seem to be taking effect. I emailed someone yesterday, received a reply. Then today, they emailed me again from the same address and their messages were deleted by spamdb. What conditions cause an address to move from grey to white, or grey to black? Trying to get my head around this, and losing emails all the while.. .d.d.
