Re: spamd extension
Hello! On Wed, Oct 26, 2005 at 09:12:34AM -0400, Frank Bax wrote: spamd only delays the *first* message between the two parties. After that there is no delay - as long as sender continues to use the same SMTP server. And there's no mailout pool with shared queue involved, and if the envelope sender address is always the same (i.e. no VERP, no SES, no self-signed SRS, no SRS-enabled forwards, etc.). Have you tried whitelisting these servers: http://greylisting.org/whitelisting.shtml Is there an underlying assumption in your question that spamd is the actual problem? During the initial weeks of using spamd on my server, half of the complaints about undelivered email were not the fault of spamd. So the other half *was* the fault of spamd? Kind regards, Hannah.
Re: spamd extension
From: Hannah Schroeter [EMAIL PROTECTED] And there's no mailout pool with shared queue involved, and if the envelope sender address is always the same (i.e. no VERP, no SES, no self-signed SRS, no SRS-enabled forwards, etc.). Surprisingly few. problem? During the initial weeks of using spamd on my server, half of the complaints about undelivered email were not the fault of spamd. So the other half *was* the fault of spamd? Oh, you floccinaucinihilipilificatrix, you! The very paranoid among us will read the disclaimers involved in greylisting and never get round to implementing it. The braver souls will just do it and see what happens. It turns out that it is an *extremely* valuable tool - far more so than simple content filters, no matter how good they are - and it is well worth having. And I say that as someone who started off at the paranoid end of the spectrum and who implemented greylisting a lot sooner than planned solely because a new CIO had used it at his previous site and insisted we put it up. Yes, there are a few teething troubles, but they mostly get taken care of in the first month where you're monitoring everything closely anyway. There were only two systematic problems we had: 1) some sites issue an RSET, before the RSET code was in spamd. 2) People using older installations of Cisco PIX firewalls had SMTP masking enabled (visible by connecting to their server and seeing stars (***) where text should be.) Asking them to turn off this useless and broken misfeature fixed the problem, or if they weren't willing to do that, have them mask only incoming connections, not outgoing ones. At our University we have some very demanding faculty with a low tolerance for email glitches. Despite this the greylisting not only went without complaints, it has generated more praise for the IT dept than any other measure in the last year (which is probably a bit galling to the guys working on the hard stuff ;-) ) My advice, just do it. Graham
Re: spamd extension
At 02:22 PM 10/28/05, Hannah Schroeter wrote: On Wed, Oct 26, 2005 at 09:12:34AM -0400, Frank Bax wrote: During the initial weeks of using spamd on my server, half of the complaints about undelivered email were not the fault of spamd. So the other half *was* the fault of spamd? Sorry, spamd was not at fault - let me rewrite that sentence. During the initial weeks of using spamd on my server, half of the complaints about undelivered email had nothing to do with spamd.
Re: spamd extension
On Tue, 25 Oct 2005 20:57:15 -0500 James Harless [EMAIL PROTECTED] wrote: What I'm looking for is a way to whitelist them based on user input.. before their initial email has been sent. In this somewhat typical scenario, the user has contacted me and said I don't want mail from [EMAIL PROTECTED] to be delayed... whitelist them, please. Sure, it can be done as long as you can figure out what server [EMAIL PROTECTED] will use to send their email and that's not as easy as it may initially seem. xxx might not always send using the same provider, the provider may have multiple outbound relays, he/she may be using a friends computer, he/she may use a wifi hotspot etc etc. Bottom line is that there's no reliable way to determine this ahead of time. Just whitelisting email addresses themselves deafeats the purpose of spamd. --- Lars Hansson Message from: Lars Hansson [EMAIL PROTECTED]
Re: spamd extension
Chad, I appreciate the insight. I do realize it's a difficult problem but, I think that there's a solution (albeit possibly from someone smarter than I). I do have variables that are known (the sender email address and the recipient email address). The problem is tying them to the IP Address of the MTA when it's seen @ spamd. It may be that there isn't a solution without direct modification of spamd. If that's the case, then I hope the developer(s) will consider this suggestion. I definitely won't be disabling spamd ;). I would have a minor revolution on my hands if my users suddenly had spam again...heh. OpenBSD greylisting has been very effective for us thus far. --James On 10/26/05, Chad M Stewart [EMAIL PROTECTED] wrote: James, The more I think about this one, the more I think there is no solution to your issue. Well okay there are two choices, either use spamd or not. :) You would have to have ESP to know from which IP address a particular sender would be sending. If I'm sitting in a hotel and using their WiFi then it is very probable that my message will be coming from their SMTP server, not that which I use normally. Given only my mail address you have no way of determining for sure, which server I use to send mail. The server I submit a message to does not have to be the server that eventually connects to the recipients server in DNS. You can't provide an email address to spamd as the redirection happens before spamd, rather with PF. The default is to send the packets to spamd. Once the connection gets rdr to spamd, I'm not aware of anyway to say, redirect again to your real MTA. That brings us back to knowing the connecting servers IP address. You could disable spamd protection and see how long it takes for your users to complain about the amount of spam they are getting. :) -Chad On Oct 25, 2005, at 9:57 PM, James Harless wrote: I appreciate the suggestions, but, not quite what I'm looking for yet. Either of these would allow me to whitelist someone AFTER they had been greylisting. What I'm looking for is a way to whitelist them based on user input.. before their initial email has been sent. In this somewhat typical scenario, the user has contacted me and said I don't want mail from [EMAIL PROTECTED] to be delayed... whitelist them, please. --James -- What would Bilano do?
Re: spamd extension
--On 26 October 2005 08:21 -0500, James Harless wrote: I do have variables that are known (the sender email address and the recipient email address). The problem is tying them to the IP Address of the MTA when it's seen @ spamd. It may be that there isn't a solution without direct modification of spamd. By design, spamd can't do this. It neither accepts mail itself, nor proxies to the real backend server. It always sends a tempfail result code, and if it's the second time it's seen client_ip|src|dest, it adds to a table at the same time, so that on the third attempt the real mailserver is hit instead. I definitely won't be disabling spamd ;) The type of functionality you're looking for needs something with hooks directly into the mail server itself, there's no way with spamd to avoid delaying a connection unless you /already/ know the IP address. Maybe milter-greylist or postgrey already do what you're looking for, or if not they'll likely be easier to adapt.
Re: spamd extension
At 09:57 PM 10/25/05, James Harless wrote: I appreciate the suggestions, but, not quite what I'm looking for yet. Either of these would allow me to whitelist someone AFTER they had been greylisting. What I'm looking for is a way to whitelist them based on user input.. before their initial email has been sent. In this somewhat typical scenario, the user has contacted me and said I don't want mail from [EMAIL PROTECTED] to be delayed... whitelist them, please. spamd only delays the *first* message between the two parties. After that there is no delay - as long as sender continues to use the same SMTP server. Have you tried whitelisting these servers: http://greylisting.org/whitelisting.shtml Is there an underlying assumption in your question that spamd is the actual problem? During the initial weeks of using spamd on my server, half of the complaints about undelivered email were not the fault of spamd.
Re: spamd extension
On 10/26/05, Frank Bax [EMAIL PROTECTED] wrote: At 09:57 PM 10/25/05, James Harless wrote: I appreciate the suggestions, but, not quite what I'm looking for yet. Either of these would allow me to whitelist someone AFTER they had been greylisting. What I'm looking for is a way to whitelist them based on user input.. before their initial email has been sent. In this somewhat typical scenario, the user has contacted me and said I don't want mail from [EMAIL PROTECTED] to be delayed... whitelist them, please. spamd only delays the *first* message between the two parties. After that there is no delay - as long as sender continues to use the same SMTP server. My experience is that greylisting requires at least 2 failed attempts. Maybe my pf.conf isn't setup properly. But, there's always 1 'extra' failure that seems to me should pass through. Have you tried whitelisting these servers: http://greylisting.org/whitelisting.shtml Is there an underlying assumption in your question that spamd is the actual problem? During the initial weeks of using spamd on my server, half of the complaints about undelivered email were not the fault of spamd. I do whitelist the servers on greylisting.org http://greylisting.org. There's no real doubt that greylisting is part of my 'issue'. It's not unmanageable, by any means, but, I'm just wondering if there isn't a way to correct the problem. Greylisting is 99% of the time not a problem. But, sometimes, the client is on the phone with a customer or in some other situation where they need to receive the email quickly. With my current greylisting setups, I can't guarantee any time when they'll receive the first email from a contact other than 'will take at least 5 mins and can take much longer depending on how their mail server is configured'. In any case, it's not unmanageable. I just set expectations with customers and they're not wanting to move away from greylisting. But, it does *feel* like a 'solvable problem'. --James -- What would Bilano do?
Re: spamd extension
If you are using spamlogd correctly, so that it is whitelisting the destination addresses of target mailservers, I find the actual need for this to be near zero, since most people send mail to [EMAIL PROTECTED] and as soon as they do the server is whitelisted for the reply - this is not the case with some big sites where their inbound mx differs from the ip their outbound mail comes from, but it works to speed up the process most of the time. - and when it doesn't the email is delayed a half hour or a little more. Basically, the correct answer is suck it up princess, in pathological cases someone's email might be delayed by a short while getting to you in normal cases it won't. Usually users ask for this when you tell them what you are doing and they don't understand that in 95% of the cases they never see a delay. -Bob * James Harless [EMAIL PROTECTED] [2005-10-25 20:09]: I appreciate the suggestions, but, not quite what I'm looking for yet. Either of these would allow me to whitelist someone AFTER they had been greylisting. What I'm looking for is a way to whitelist them based on user input.. before their initial email has been sent. In this somewhat typical scenario, the user has contacted me and said I don't want mail from [EMAIL PROTECTED] to be delayed... whitelist them, please. --James On 10/25/05, Bob Beck [EMAIL PROTECTED] wrote: spamdb -a `spamdb | grep '[EMAIL PROTECTED]|[EMAIL PROTECTED]' | cut -d '|' -f 2` -Bob * James Harless [EMAIL PROTECTED] [2005-10-25 15:50]: I would like some advice on extending spamd functionality. I'm not sure the best approach to this problem. Problem: I administer several independent mail gateway / firewall devices that greylist for their networks. I've done a fair job of educating users about how greylisting will affect their email but, inevitably a user will contact me to request that an incoming email be whitelisted. The only information they have is 1) sending email address and 2) receiving email address. Of course, spamd only deals in IP addresses and it may be difficult to find the ip address of the sending mail server. Additionally, I'd like to provide some method to the users where they could whitelist someone themselves without requesting directly from me. What I envision: A script or extension to spamd that would allow me to input a 'from' and 'rcpt to' address. Then, the next time that combo is seen, from any IP address...it gets whitelisted automatically. I envision this only happening one time and then returning to greylisting as normal. I understand that there's a chance of someone sending spam through in that window with the proper from/to combo .. but, it's small enough to accept. Thoughts? Does this sound feasible? Is this a reasonable solution? If so, what direction would you recommend for implementation? (I'm no programmer.. but, not afraid of diving in, nonetheless.) --James -- What would Bilano do?
Re: spamd extension
On Wed, 2005-10-26 at 09:06:11 -0600, Bob Beck proclaimed... Basically, the correct answer is suck it up princess, in pathological cases someone's email might be delayed by a short while getting to you in normal cases it won't. Usually users ask for this when you tell them what you are doing and they don't understand that in 95% of the cases they never see a delay. Hell, I usualy just blame the other ISP and by the time the customer argues, the mail is re-sent and waiting for them :-)
Re: spamd extension
My experience is that greylisting requires at least 2 failed attempts. Maybe my pf.conf isn't setup properly. But, there's always 1 'extra' failure that seems to me should pass through. James is right, it's a design flaw of spamd that two failed attempts are required. This is what happens: 1) first attempt, goes to spamd, is logged. 2) second attempt, goes to spamd, is marked as good ... *BUT* it still went to spamd. spamd is not an application relay, so it has no way of passing that currently-active second attempt through to the true MTA, so ... 3) third attempt, redirected to true MTA The only fix for this is a *major* redesign of spamd (or equivalently incorporating spamd's greylisting code into a spamfilter which *does* relay connections at the IP level to an MTA - which is actually what I'm working on at the moment) One of the pre-requisites (in my opinion) for a filter which relays connections (rather than routing them through) is full transparency, i.e. the MTA sees the IP of the original caller, not the IP of the relay. This is so that the MTA continues to do third-party relay rejection and does not require you to duplicate that logic in your relay host. Fortunately for us, OpenBSD+pf have exactly the facilities needed to transparently forward at the TCP/IP session level, albeit not a common or easy thing to do. Graham
Re: spamd extension
On 10/26/05, James Harless [EMAIL PROTECTED] wrote: Chad, I appreciate the insight. I do realize it's a difficult problem but, I think that there's a solution (albeit possibly from someone smarter than I). Nope there's just not. I do have variables that are known (the sender email address and the recipient email address). The problem is tying them to the IP Address of the MTA when it's seen @ spamd. It may be that there isn't a solution without direct modification of spamd. If that's the case, then I hope the developer(s) will consider this suggestion. How would you find an unknown ip of an unknown machine? About the only *chance* you have is doing MX lookup's and hoping that email comes from that same server. If their organization uses various relays and proxies to send, you are out of luck. There's no way to get that information without a previously harvested email and looking at the message headers. --Bryan
Re: spamd extension
On Oct 26, 2005, at 11:54 AM, Graham Toal wrote: My experience is that greylisting requires at least 2 failed attempts. Maybe my pf.conf isn't setup properly. But, there's always 1 'extra' failure that seems to me should pass through. James is right, it's a design flaw of spamd that two failed attempts are required. This is what happens: 1) first attempt, goes to spamd, is logged. 2) second attempt, goes to spamd, is marked as good ... *BUT* it still went to spamd. spamd is not an application relay, so it has no way of passing that currently-active second attempt through to the true MTA, so ... 3) third attempt, redirected to true MTA I agree this is how things work. I disagree that this is a design flaw. Instead this is the fundamental thing that makes spamd so great at what it does. Maybe I'm a little too RFC biased, but if the standards say XYZ MUST be done, then if the sending MTA is not playing by the rules, I don't want their mail. Though I'm happy to talk and work with them to get their servers fixed. The side effect being that all those spammer zombie machines don't get a message into my servers. :) spamd is ensuring that MTAs are following the standards. The standards say that a sending MTA must wait 30 minutes before attempting a retry, thus the default passtime for spamd is 25 minutes, which I think is a good buffer. If MTAs should retry in say 15 minutes, I don't know what spamd does, I've not tested that scenario. I would hope that maybe spamd would update the initial time to the most recent attempt and wait to put the IP in the whitelist pool until passtime has passed between retries. I often see delays of either an hour or two when first getting a message via a new MTA. Which makes sense to me, and I think is tolerable. Email is not instant messaging. If it absolutely has to be there NOW, then use something else. :) 00:00 -- first connection attempted 00:30 -- second connection attempted 00:31 -- IP now whitelisted I've found that some MTAs will try make a 3rd attempt 60 minutes from the first attempt, while others seem to wait 60 minutes or more from the 2nd attempt. -Chad
Re: spamd extension
How would you find an unknown ip of an unknown machine? About the only *chance* you have is doing MX lookup's and hoping that email comes from that same server. If their organization uses various relays and proxies to send, you are out of luck. There's no way to get that information without a previously harvested email and looking at the message headers. Well, that's exactly the point... you don't find the ip. You put in a temporal entry that says 'whitelist the next ip address that connects attempting to send mail from $sender to $rcpt'. After that, the entry expires. It's been pointed out here that it just isn't possible, currently. I'm ok with that. The issue is smaller than the problem that it solves (removing most of the spam from my networks). Thanks for all the input. --James
Re: spamd extension
Graham Toal wrote: The only fix for this is a *major* redesign of spamd (or equivalently incorporating spamd's greylisting code into a spamfilter which *does* relay connections at the IP level to an MTA - which is actually what I'm working on at the moment) Why start from scratch ? There are enough seasoned, full featured MTA's around that will allow you to incorparate greylisting. And you get all the other stuff like STARTTLS, AUTH etc gratis. I'd either accept spamd's few limitiations or incorparate greylisting into a MTA. Just my thoughts. Hans
Re: spamd extension
--On 26 October 2005 09:12 -0400, Frank Bax wrote: Have you tried whitelisting these servers: http://greylisting.org/whitelisting.shtml That list by policy only includes 'shared queue' servers on blocks larger than /24 (the greylisting software written by the list compiler usually masks the last byte of the address anyway). If your spamd box regularly receives mail from users at large sites that use different machines for outbound and inbound mail, where a shared queue is involved, and don't have enough users yourself to ensure that the most common of these are already whitelisted, greylisting software other than spamd might be a better choice. As luck would have it these are also often the sites with crappy retry cycles delaying mail multiple hours. But then, I wouldn't want to run a full mta on the small hardware I usually run spamd on sitting in front of mail servers, and larger sites that are less affected by this problem probably don't want to devote full mta resources to their spam senders either, so it's good that there are both lightweight and more featureful choices.
Re: spamd extension
At 11:05 AM 10/26/05, James Harless wrote: On 10/26/05, Frank Bax [EMAIL PROTECTED] wrote: spamd only delays the *first* message between the two parties. After that there is no delay - as long as sender continues to use the same SMTP server. My experience is that greylisting requires at least 2 failed attempts. Maybe my pf.conf isn't setup properly. But, there's always 1 'extra' failure that seems to me should pass through. Correct. One *message* - two (or more) failed attempts before delivery. Extra failed attempts can sometimes happen - it depends on sender's retry frequency compared to spamd_flags values.
Re: spamd extension
Stuart Henderson wrote: --On 26 October 2005 08:21 -0500, James Harless wrote: I do have variables that are known (the sender email address and the recipient email address). The problem is tying them to the IP Address of the MTA when it's seen @ spamd. It may be that there isn't a solution without direct modification of spamd. By design, spamd can't do this. It neither accepts mail itself, nor proxies to the real backend server. It always sends a tempfail result code, and if it's the second time it's seen client_ip|src|dest, it adds to a table at the same time, so that on the third attempt the real mailserver is hit instead. I definitely won't be disabling spamd ;) The type of functionality you're looking for needs something with hooks directly into the mail server itself, there's no way with spamd to avoid delaying a connection unless you /already/ know the IP address. Maybe milter-greylist or postgrey already do what you're looking for, or if not they'll likely be easier to adapt. Not to venture off topic, but it's at this point that I would suggest you look at qpsmtpd (http://smtpd.develooper.com) for your anti-spam needs. It's an SMTP server written entirely in perl and is incredibly extensible (easy to do so as well.) It's nice and speedy: apache.org and perl.org receive all of their mail through it. It can tie into Postfix and qmail, and there is an experimental SMTP proxy function as well. I hope to getting around to creating an interface to sendmail as well. Its connections can be managed by an internal polling server (using epoll or kqueue under linux/bsd if available), a forkserver model, tcpserver (with speedy-cgi/pperl/forkserver), or apache2 (via mod_perl). It is my current perl love, and I would highly recommend at least a peek at it. For a quick summary by one of the main developers, see: http://www.oreillynet.com/pub/a/sysadmin/2005/09/15/qpsmtpd.html
Re: spamd extension
The only fix for this is a *major* redesign of spamd (or equivalently incorporating spamd's greylisting code into a spamfilter which *does* relay connections at the IP level to an MTA - which is actually what I'm working on at the moment) Why start from scratch ? There are enough seasoned, full featured MTA's around that will allow you to incorparate greylisting. And you get all the other stuff like STARTTLS, AUTH etc gratis. I'd either accept spamd's few limitiations or incorparate greylisting into a MTA. Just my thoughts. There *are* several greylisting implementations using MTAs if that is what you want. The attractive feature of spamd+openbsd/pf is that it is MTA-agnostic. After it does its thing it simply routes your connection through to the real MTA at the IP level. Anyway, it's not starting from scratch for me - I have a mature pseudo-transparent SMTP filter that works well and has been in service for over a year - it's just that I have not publicised it much because in its current form it requires configuration, such as telling it what domains you accept mail for, which IPs are local, etc. I needed to learn about transparent bridging first and recode the I/O so that the filtering is not visible at the IP level. Which I now have, mostly. My filter uses spamassassin plus spamprobe plus uvscan plus clamav, with some automatic detection of spamtrap addresses thrown in. I haven't yet added greylisting to it, and indeed our deployment at the University where I work has an openbsd running spamd sitting in front of my filter sitting in front of the real MTA! By incorporating the logic from spamd into my code, I can remove one piece of hardware. And improve spamd while I'm at it, because with thi sarchitecture I can forward that second connection attempt to the MTA, and avoid having two delays rather than one. Graham
Re: spamd extension
On 10/26/05, James Harless [EMAIL PROTECTED] wrote: Chad, I appreciate the insight. I do realize it's a difficult problem but, I think that there's a solution (albeit possibly from someone smarter than I). Nope there's just not. There is, but not with spamd as currently implemented. The fix would involve this: 1) accept the connection, remember the target IP 2) go through the rcpt from/mail to protocol, and when you have the information, check it in your whitelist. If it is present, open a connection with the original target, repeat the rcpt/mail exchange (not forgetting the HELO) and then sit back and transparently proxy the rest of the connection. It's doable, it's just not easy. That plus a lot more is what the filter I was talking about in the other thread does; maybe if it's not too difficult, I'll do a shorter version which doesn't have the majority of my code, but just adds the logic above to spamd, if there's any interest? It does require spamd to be running in a transparent bridge. *NOT* a NAT gateway, which is the most common configuration. By the way, the other improvement I'd make in spamd if I had my druthers, is that it would have the option of accepting the initial email and returning the tempfail code at the end of the data exchange rather than before it as it currently does. This would allow proper QA on the rejected mails. You'ld need to create a signature of an email and when the mail went through successfully on the second attempt, locate the original copy using the signature and remove it from the cache; mails which never retried would remain in the cache, and would be swept after an appropriate time out, giving you a good record of rejected mails. You could either use this info to generate stats, or you could run the mails through a traditional spam filter as a consistency check, to try to detect genuine connections that had been inadvertently blocked. Or if you're sure all the rejects were genuinely spam, you could feed the saved copies into spam filter training, or to a cooperative net project like Vipul. Lots of scope there for new features. Graham
spamd extension
I would like some advice on extending spamd functionality. I'm not sure the best approach to this problem. Problem: I administer several independent mail gateway / firewall devices that greylist for their networks. I've done a fair job of educating users about how greylisting will affect their email but, inevitably a user will contact me to request that an incoming email be whitelisted. The only information they have is 1) sending email address and 2) receiving email address. Of course, spamd only deals in IP addresses and it may be difficult to find the ip address of the sending mail server. Additionally, I'd like to provide some method to the users where they could whitelist someone themselves without requesting directly from me. What I envision: A script or extension to spamd that would allow me to input a 'from' and 'rcpt to' address. Then, the next time that combo is seen, from any IP address...it gets whitelisted automatically. I envision this only happening one time and then returning to greylisting as normal. I understand that there's a chance of someone sending spam through in that window with the proper from/to combo .. but, it's small enough to accept. Thoughts? Does this sound feasible? Is this a reasonable solution? If so, what direction would you recommend for implementation? (I'm no programmer.. but, not afraid of diving in, nonetheless.) --James
Re: spamd extension
spamdb -a `spamdb | grep '[EMAIL PROTECTED]|[EMAIL PROTECTED]' | cut -d '|' -f 2` -Bob * James Harless [EMAIL PROTECTED] [2005-10-25 15:50]: I would like some advice on extending spamd functionality. I'm not sure the best approach to this problem. Problem: I administer several independent mail gateway / firewall devices that greylist for their networks. I've done a fair job of educating users about how greylisting will affect their email but, inevitably a user will contact me to request that an incoming email be whitelisted. The only information they have is 1) sending email address and 2) receiving email address. Of course, spamd only deals in IP addresses and it may be difficult to find the ip address of the sending mail server. Additionally, I'd like to provide some method to the users where they could whitelist someone themselves without requesting directly from me. What I envision: A script or extension to spamd that would allow me to input a 'from' and 'rcpt to' address. Then, the next time that combo is seen, from any IP address...it gets whitelisted automatically. I envision this only happening one time and then returning to greylisting as normal. I understand that there's a chance of someone sending spam through in that window with the proper from/to combo .. but, it's small enough to accept. Thoughts? Does this sound feasible? Is this a reasonable solution? If so, what direction would you recommend for implementation? (I'm no programmer.. but, not afraid of diving in, nonetheless.) --James
