Re: : : Purpose of spamd-setup in greylisting mode?
On Tue, Aug 12, 2008 at 01:25:15PM +0200, Raimo Niskanen wrote: : I (and others) use variations on a slightly different approach... : I can publish the scripts if anyone is interested. http://www.erlang.org/~raimo/greytrap/ -- / Raimo Niskanen, Erlang/OTP, Ericsson AB -- / Raimo Niskanen, Erlang/OTP, Ericsson AB
Re: Purpose of spamd-setup in greylisting mode?
On Tue, 2008-08-12 at 09:46 +0200, Morgan Wesstrvm wrote: information Google turned up. A general reflection is that it's a little hard to grasp from the man pages how all the components work together (spamd, spamlogd, spamd-setup, spamdb, pf) especially when you're only used to blacklisting like I am. But everything seems to be working just I just had to do the same (on OpenBSD, though). I do find the man pages quite complete, to be honest. It may be not a step-through guide on all topics, but once I took myself the time to actually read it all through, it became quite clea.: fine now. However, I'm confused about the purpose of spamd-setup in greylisting mode. * There is no longer a spamd table to fill with blacklisted IP addresses. Correct. Because spamd takes care of blacklisted IPs and no longer pf. * Addresses being whitelisted in spamdb are automatically moved to spamd-white even if I don't run spamd-setup. yes, as explained in spamd(8): spamd regularly scans the /var/db/spamd database and configures all whitelist addresses as the pf(4) spamd-white table This is required so that whitelisted IPs can bypass spamd entirely using pf. * spamdb does NOT get populated with any blacklisted IP addresses when I run spamd-setup. No, because you don't need it. Everything that is not in the state of GREY, TRAP, SPAMTRAP or WHITE is obviously blacklisted. So exactly what does spamd-setup do in greylisting mode? Do I need it? spamd-setup(8) The spamd-setup utility sends blacklist data to spamd(8), as well as configuring mail rejection messages for blacklist entries. One more question. If I want to blacklist an IP address manually I assume I use spamdb -T -a ip-address. That creates a SPAMTRAP record in the database at least. Is this the same as blacklisting? If it is, No, if you'd like to blacklist an IP manually, you can either do so by using a custom pf table (e.g. mywhite) that omits redirection from that table to spamd. Or, the way we do it, we compile our own list and configure/load it in spamd.conf(5): The spamd.conf file is read by spamd-setup(8) to configure blacklists for spamd(8). why doesn't the various blacklists in spamd.conf show up here then? How Well, you have to configure them somehere. are those lists handled in greylisting mode? They are loaded using spamd-setup and fed into spamd(8). Hope that helps, -- Stephan A. Rickauer --- Institute of Neuroinformatics Tel +41 44 635 30 50 University / ETH Zurich Sec +41 44 635 30 52 Winterthurerstrasse 190 Fax +41 44 635 30 53 CH-8057 ZurichWebwww.ini.uzh.ch
Re: Purpose of spamd-setup in greylisting mode?
Stephan A. Rickauer wrote: On Tue, 2008-08-12 at 09:46 +0200, Morgan Wesstrvm wrote: information Google turned up. A general reflection is that it's a little hard to grasp from the man pages how all the components work together (spamd, spamlogd, spamd-setup, spamdb, pf) especially when you're only used to blacklisting like I am. But everything seems to be working just I just had to do the same (on OpenBSD, though). I do find the man pages quite complete, to be honest. It may be not a step-through guide on all topics, but once I took myself the time to actually read it all through, it became quite clea.: Thank you for your answers Stephan. Please bear with me because I come from the land of blacklisting only :-) In general I'm very pleased with the BSD man pages but when I try to learn new functionality I find myself looking for details I probably only can get if I read the actual source code and I understand its place is not in the man pages. * There is no longer a spamd table to fill with blacklisted IP addresses. Correct. Because spamd takes care of blacklisted IPs and no longer pf. Yes, but what does that mean? Does spamd keep an internal list of blacklisted IP addresses and why is it not in the spamd database in that case (which seems a natural place for it)? Can I see it somewhere and manipulate it manually? yes, as explained in spamd(8): spamd regularly scans the /var/db/spamd database and configures all whitelist addresses as the pf(4) spamd-white table Ok, that is of course obvious now when I read it :-) Still curious of _how_ it's actually done. Does this somehow has to do with the fdescfs filesystem that has to be mounted? I believe I read somewhere that the interaction with the tables are done through this filesystem. Why not use the API that pfctl use? I really feel like a jerk asking these stupid questions... :-) No, because you don't need it. Everything that is not in the state of GREY, TRAP, SPAMTRAP or WHITE is obviously blacklisted. I understand the states GREY and WHITE but what exactly is TRAP and SPAMTRAP if they're not the blacklisted addresses? I read the section GREYTRAPPING in spamd(8): When running spamd in default mode, it may be useful to define spamtrap destination addresses to catch spammers as they send mail from greylisted hostssnip I haven't slept tonight so I simply don't understand what this paragraph is saying or what its purpose is? Can I enter fake email addresses here and if a GREY host happens to send a mail to this fake address, that host gets blacklisted? How big is the chance that it would try a fake random address I enter here...? (LOL, I can imagine you have a good laugh by now but I really like to learn :-) ) spamd-setup(8) The spamd-setup utility sends blacklist data to spamd(8), as well as configuring mail rejection messages for blacklist entries. Yes :-) Blacklist data - is that the IP addresses collected from the blacklists defined in spamd.conf? No, if you'd like to blacklist an IP manually, you can either do so by using a custom pf table (e.g. mywhite) that omits redirection from that table to spamd. Or, the way we do it, we compile our own list and configure/load it in spamd.conf(5): I can still maintain a local blacklist file that I load in spamd.conf as I have done since I started using spamd then? I was excited for a while that I could drop that file and enter the IP addresses permanently in the spamd database instead but the local file is fine with me since I know how it works. I just feel uncomfortable not being able to see the list anywhere... The spamd.conf file is read by spamd-setup(8) to configure blacklists for spamd(8). why doesn't the various blacklists in spamd.conf show up here then? How Well, you have to configure them somehere. My confusion comes from the fact that I'm still told to use spamd.conf and spamd-setup as I've always done while using blacklisting only mode, but now I'm missing spamd and the rules that refer to that table and I can't see anywhere that the blacklists are actually parsed and used... Sorry again for being a thick jerk and being stuck in the old thinking but I really appreciate any insight into this because I'm completely set on learning this. :-) Kind regards Morgan
Re: Purpose of spamd-setup in greylisting mode?
Morgan Wesstrvm [EMAIL PROTECTED] writes: I haven't slept tonight so I simply don't understand what this paragraph is saying or what its purpose is? Can I enter fake email addresses here and if a GREY host happens to send a mail to this fake address, that host gets blacklisted? How big is the chance that it would try a fake random address I enter here...? (LOL, I can imagine you have a good laugh by now but I really like to learn :-) ) This is where you may find a major source of entertainment. Yes, you can enter bogus addresses in the traplist. Yes, the easiest way to decide what to put in your traplist is to harvest from the joejob-generated bounce messages that keep piling up. For good measure, you can publish your list of spamtraps on the web and sit back and laugh at tail -f /var/log/spamd. My spammer traps are at http://www.bsdly.net/~peter/traplist.shtml, a series of blog posts about this very topic starts with the post http://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html and my newest spamd entertainment can be found at my still-fresh Name and Shame Robot page, http://www.bsdly.net/~peter/nameandshame.html. The name and shame part means essentially now that we have a list of IP addresses that have verifiably tried to deliver mail to our spamtraps, it is trivial to extract the log data of each of those addresses' actions from our spamd log. To the extent that the admins involved actually can be bothered to look, it's also a bit clearer evidence that they may have, er, potential for improvement than if we were just using the raw list of IP addresses. Socially responsible sysadmins, y'know. I infer from certain characters in your name and message headers that you're .se based, so my almost-done Norwegian writeup about the name and shame robot at http://www.bsdly.net/~peter/nameandshame_no.txt might provide some entertainment despite the funny spelling (I'm pondering compressing this to a third of the size for publication in one of the IT rags, and yes, I may even rewrite this in English given enough round tuits). - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: Purpose of spamd-setup in greylisting mode?
Does this somehow has to do with the fdescfs filesystem that has to be mounted? Are you by any chance using this on a non-OpenBSD OS? Yes, FreeBSD. I remember when I upgraded spamd once during it's 3.x era, it suddenly started to complain about missing fdescfs and refused to start so I had to mount that in FreeBSD even though I only ran spamd in blacklisting mode only. This time I didn't even try to run it without fdescfs... still interested to know how fdescfs is used in combination with spamd or is that referred to in grey.c? Get some sleep then, it should be clearer. A couple of caffeine pills and a glass of coke makes wonders. Dropped the coffee 10 years ago because it killed my stomach ;-) Can I enter fake email addresses here and if a GREY host happens to send a mail to this fake address, that host gets blacklisted? Yes. How big is the chance that it would try a fake random address I enter here...? High when you use an address that spammers already know about. This piece fell into place now thanks to Peter Hansteen's post. I love it already... When you greylist, you just need to redirect all traffic from addresses not in spamd-white to spamd. PF doesn't need to know whether that's currently subject to greylisting or on a blacklist Yes, this was the first mental obstacle to overcome but I managed to do that. I was just curious to know what happened then, when spamd was hit by requests from both black- and grey-listed hosts as well as unknown hosts. But I understand now that spamd tells them apart with the help of that internal list I can't see anywhere... :-) Thanks again all of you who have answered my questions. /Morgan
Re: Purpose of spamd-setup in greylisting mode?
Peter N. M. Hansteen wrote: This is where you may find a major source of entertainment. Yes, you can enter bogus addresses in the traplist. Yes, the easiest way to decide what to put in your traplist is to harvest from the joejob-generated bounce messages that keep piling up. For good measure, you can publish your list of spamtraps on the web and sit back and laugh at tail -f /var/log/spamd. Simply lovely. The creativity of the developers never stops to amaze me. :-) And yes, I'm .se. I'll make sure to read the links you referred to. Still have 3 days left on my vacation so they will come in handy... thanks again for your help! /Morgan
Re: : Purpose of spamd-setup in greylisting mode?
On Tue, Aug 12, 2008 at 11:46:29AM +0200, Peter N. M. Hansteen wrote: Morgan Wesstrvm [EMAIL PROTECTED] writes: I haven't slept tonight so I simply don't understand what this paragraph is saying or what its purpose is? Can I enter fake email addresses here and if a GREY host happens to send a mail to this fake address, that host gets blacklisted? How big is the chance that it would try a fake random address I enter here...? (LOL, I can imagine you have a good laugh by now but I really like to learn :-) ) This is where you may find a major source of entertainment. Yes, you can enter bogus addresses in the traplist. Yes, the easiest way to decide what to put in your traplist is to harvest from the joejob-generated bounce messages that keep piling up. For good measure, you can publish your list of spamtraps on the web and sit back and laugh at tail -f /var/log/spamd. I (and others) use variations on a slightly different approach... When spammers apparently started to generate their target addresses from parts a'la: AnastasiabeetRansom AnastasiacartonGrover : SavannahenthusiastGrover SavannahkobayashiRansom i found the SPAMTRAP mechanism too simple since it uses exact matches of the addresses and the spammers generated addresses had too much variation. A greyscanner script has the possibility to be more intelligent. In my case I use a modified greyscanner script [original]: http://www.ualberta.ca/~beck/greyscanner/ my modifications are extended DNS checks and mail address checking using an address pattern file. I use newsyslog to make the maillog rotation process /var/log/maillog.0 to find User unknown lines. Hosts mailing to unknown addresses are removed from spamd-white. Unknown addresses are saved in a sort -u file. The saved unknown addresses are then processed to find address prefixes and postfixes into a file a'la: ^Anastasia ^Savannah Grover$ Ransom$ My modified greyscanner script then use these prefixes and postfixes for address validation when processing the spamdb database. I can publish the scripts if anyone is interested. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB
Re: : Purpose of spamd-setup in greylisting mode?
Raimo Niskanen wrote: On Tue, Aug 12, 2008 at 11:46:29AM +0200, Peter N. M. Hansteen wrote: Morgan Wesstrvm [EMAIL PROTECTED] writes: I haven't slept tonight so I simply don't understand what this paragraph is saying or what its purpose is? Can I enter fake email addresses here and if a GREY host happens to send a mail to this fake address, that host gets blacklisted? How big is the chance that it would try a fake random address I enter here...? (LOL, I can imagine you have a good laugh by now but I really like to learn :-) ) This is where you may find a major source of entertainment. Yes, you can enter bogus addresses in the traplist. Yes, the easiest way to decide what to put in your traplist is to harvest from the joejob-generated bounce messages that keep piling up. For good measure, you can publish your list of spamtraps on the web and sit back and laugh at tail -f /var/log/spamd. I (and others) use variations on a slightly different approach... When spammers apparently started to generate their target addresses from parts a'la: AnastasiabeetRansom AnastasiacartonGrover : SavannahenthusiastGrover SavannahkobayashiRansom i found the SPAMTRAP mechanism too simple since it uses exact matches of the addresses and the spammers generated addresses had too much variation. A greyscanner script has the possibility to be more intelligent. In my case I use a modified greyscanner script [original]: http://www.ualberta.ca/~beck/greyscanner/ my modifications are extended DNS checks and mail address checking using an address pattern file. I use newsyslog to make the maillog rotation process /var/log/maillog.0 to find User unknown lines. Hosts mailing to unknown addresses are removed from spamd-white. Unknown addresses are saved in a sort -u file. The saved unknown addresses are then processed to find address prefixes and postfixes into a file a'la: ^Anastasia ^Savannah Grover$ Ransom$ My modified greyscanner script then use these prefixes and postfixes for address validation when processing the spamdb database. I can publish the scripts if anyone is interested. I'm interested to see your scripts
Re: Purpose of spamd-setup in greylisting mode?
Stuart Henderson wrote: On 2008-08-12, Morgan Wesstrvm [EMAIL PROTECTED] wrote: Correct. Because spamd takes care of blacklisted IPs and no longer pf. Yes, but what does that mean? Does spamd keep an internal list of blacklisted IP addresses yes and why is it not in the spamd database in that case (which seems a natural place for it)? Can I see it somewhere and manipulate it manually? it's transient fast-changing information, there isn't a lot of point writing it to disk... I need to ask a follow-up question here cause I may have completely misunderstood some basic concept. When I blacklist a host it stay blacklisted forever... or at least until they come crawling back and beg me to be removed from the blacklist - there's nothing transiet or fast-changing about it. How would I handle the hosts that have been dynamically blacklisted during the computer's uptime if I have to reboot it? I wouldn't want to loose those addresses and begin from scratch. The spamd-white table is also dynamically populated but I can save the contents of that table with pfctl and a cronjob. Is there a mechanism to correctly deal with this that I have missed? /Morgan
Re: : Purpose of spamd-setup in greylisting mode?
Raimo Niskanen [EMAIL PROTECTED] writes: I can publish the scripts if anyone is interested. Those script sound very interesting. I'd love to see them. - P -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: Purpose of spamd-setup in greylisting mode?
On 2008-08-12, Morgan Wesstrvm [EMAIL PROTECTED] wrote: and why is it not in the spamd database in that case (which seems a natural place for it)? Can I see it somewhere and manipulate it manually? it's transient fast-changing information, there isn't a lot of point writing it to disk... I need to ask a follow-up question here cause I may have completely misunderstood some basic concept. When I blacklist a host it stay blacklisted forever... or at least until they come crawling back and beg me to be removed from the blacklist - there's nothing transiet or fast-changing about it. That sounds like something you want to put in a file-based blacklist that spamd-setup reads. I don't think you really want to be removing and re-adding tens of thousands of /var/db/spamd entries from a network-based blacklist once an hour. How would I handle the hosts that have been dynamically blacklisted during the computer's uptime if I have to reboot it? Dynamically, what do you mean, by greylisting? They *are* in /var/db/spamd as TRAPPED entries. Same for whitelisted entries (spamd-white).
Re: Purpose of spamd-setup in greylisting mode?
I don't think you really want to be removing and re-adding tens of thousands of /var/db/spamd entries from a network-based blacklist once an hour. How would I handle the hosts that have been dynamically blacklisted during the computer's uptime if I have to reboot it? Dynamically, what do you mean, by greylisting? They *are* in /var/db/spamd as TRAPPED entries. Same for whitelisted entries (spamd-white). Please correct me if I have got the basic concepts wrong but this is how I understand it. Blacklisting. Blacklists are initially read from spamd.conf and those lists are refreshed by spamd-setup once every hour or whatever you set the cron job to. These are not my concern. Spammers not included in those blacklist will connect to me and will be unknown to spamd at their first connect. They will be greylisted and if they _don't_ try to resend the same mail again within 4 hours they will be blacklisted. The practical difference seems to be minor though. Greylisted hosts are stuttered for 10 seconds while blacklisted hosts are stuttered indefinetely. Those dynamically added hosts, added outside the static blacklists, I would like to save them so I can blacklist them immediately after next reboot without having them go through greylisting again. Whitelisting. Greylisted hosts that _do_ resend their mail within 4 hours will be automatically whitelisted and put in spamd-white. Mail-servers contacted by me, as logged by pf, will be whitelisted immediately and put in spamd-white. I would like to save those entries too, so those hosts won't have to go through greylisting again either after a reboot. It's maybe not a big deal since all hosts will get their true status pretty quickly again I guess. It just feels logically correct to keep their status across a reboot. Regards Morgan
