re: (graylisting) better spam filter for postfix
I treid grey listng and don't use it because too many servers were not re-sending the e-mail back asap. Alot did and there was no problem. But some took up to a day to retry the message. I remeber reading about DPSAM. Also going to look at amavisd-new and assp. I like the idea of calling it a engine when using mutiple applications. Thanks for the sugguestions, Josh -- This message has been scanned for viruses and dangerous content by Mychoice, and is believed to be clean.
Re: Better spam filter for postfix
Zitat von Henrik K h...@hege.li: On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Regards Andreas smime.p7s Description: S/MIME Signatur
Re: Better spam filter for postfix
On 16 juil. 2010, at 09:27, lst_ho...@kwsoft.de wrote: In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Before-queue content filtering is great. It demands more CPU because the filtering must be very fast, but it works. In France, we have a similar limitation: it's illegal to destroy a communication (mail, email…). With BQCF spam is rejected, not destroyed. Patrick PRONIEWSKI -- Administrateur Système - SENTIER - Université Lumière Lyon 2 smime.p7s Description: S/MIME cryptographic signature
Re: Better spam filter for postfix
Am 16.07.2010 09:27, schrieb lst_ho...@kwsoft.de: Zitat von Henrik K h...@hege.li: On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Regards Andreas why not use spamass-milter drops spam during smtp income stage this is allowed anyway, also clamav-milter with sanesecurity works nice this way, bouncing mail after recieve by whatever reason may produce backscatter, so it isnt a good idea in every case or country, normally you only flag spam and pass it and/or hold it ( for human postmaster inspection ) i. if use amavis with after queue filter , mail always needs daily support, and companies who stopped filtering in germany ( i dont know one ) have mostly a problem with helpless admins ignorant managers/users etc, not with law or existing antispam solutions so its mostly a human problem -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: Better spam filter for postfix
Zitat von Robert Schetterer rob...@schetterer.org: Am 16.07.2010 09:27, schrieb lst_ho...@kwsoft.de: Zitat von Henrik K h...@hege.li: On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Regards Andreas why not use spamass-milter drops spam during smtp income stage this is allowed anyway, also clamav-milter with sanesecurity works nice this way, bouncing mail after recieve by whatever reason may produce backscatter, so it isnt a good idea in every case or country, normally you only flag spam and pass it and/or hold it ( for human postmaster inspection ) i. if use amavis with after queue filter , mail always needs daily support, and companies who stopped filtering in germany ( i dont know one ) have mostly a problem with helpless admins ignorant managers/users etc, not with law or existing antispam solutions so its mostly a human problem The point is - Before-Queue content filter is expansive and must be combined with cheap reject techologies anyway if you have non negliable load - Tagging spam is nearly useless because no user like to poke through the dustbin to search for potential lost mail - Spam-Bouncing is no option at all - In general the false positive rate is a higher and more difficult to find out with content filter compared to a sane set of reputation based filters So the most reasonable approch is to ditch content filter at all and use a sane set of reputation based decisions and maybe greylisting to reject spam at earliest possible stage. I don't speak about or even recommend to not use spam filtering, but content filter is sometimes the bigger problem compared to some slipping through spams. Regards Andreas smime.p7s Description: S/MIME Signatur
Re: Better spam filter for postfix
Am 16.07.2010 10:15, schrieb lst_ho...@kwsoft.de: Zitat von Robert Schetterer rob...@schetterer.org: Am 16.07.2010 09:27, schrieb lst_ho...@kwsoft.de: Zitat von Henrik K h...@hege.li: On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Regards Andreas why not use spamass-milter drops spam during smtp income stage this is allowed anyway, also clamav-milter with sanesecurity works nice this way, bouncing mail after recieve by whatever reason may produce backscatter, so it isnt a good idea in every case or country, normally you only flag spam and pass it and/or hold it ( for human postmaster inspection ) i. if use amavis with after queue filter , mail always needs daily support, and companies who stopped filtering in germany ( i dont know one ) have mostly a problem with helpless admins ignorant managers/users etc, not with law or existing antispam solutions so its mostly a human problem The point is - Before-Queue content filter is expansive and must be combined with cheap reject techologies anyway sorry explain cheap if you have non negliable load - Tagging spam is nearly useless because no user like to poke through the dustbin to search for potential lost mail i dont understand, as you always need support mail, its no problem to solve user questions, only the rate of questions should be handable by the corosponding number of postmaster and/or supporters - Spam-Bouncing is no option at all why ?, a bounce is no thing of evil, there will be bounces by several reasons ever - In general the false positive rate is a higher and more difficult to find out with content filter compared to a sane set of reputation based filters i have false postive under 0,1 promille no problem here So the most reasonable approch is to ditch content filter at all and use a sane set of reputation based decisions and maybe greylisting to reject spam at earliest possible stage. you should always use all usefull antispam technics which make sense anyway ( specially that ones that are native in postfix ) greylisting is one of them , but in a few cases on my site simply does not work anymore defending bots so antispam is always a filter chain, the real antispam filter such as spamassassin should always be one of the last I don't speak about or even recommend to not use spam filtering, but content filter is sometimes the bigger problem compared to some slipping through spams. maybe, thats individual, like spam always is, competent postmaster should choose the right way in the right case Regards Andreas no need to flame, i have no problem with supporting ca 10 mailservers with antispam enabled up to 1 mail addresses some spam always slipping trough,always some false positives , thats the nature of the beast, the goal is keeping that rate low in my case spam filtering is no such problem , as mailservers that have buggy dns setups are in rbls etc, after all, one of the biggest problems are false tagging to antispam filters in mail clients i.e outlook which produces more questions then server side filters, as most users dont understand their mail client settings -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: Better spam filter for postfix
Zitat von Robert Schetterer rob...@schetterer.org: Am 16.07.2010 10:15, schrieb lst_ho...@kwsoft.de: Zitat von Robert Schetterer rob...@schetterer.org: Am 16.07.2010 09:27, schrieb lst_ho...@kwsoft.de: Zitat von Henrik K h...@hege.li: On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Regards Andreas why not use spamass-milter drops spam during smtp income stage this is allowed anyway, also clamav-milter with sanesecurity works nice this way, bouncing mail after recieve by whatever reason may produce backscatter, so it isnt a good idea in every case or country, normally you only flag spam and pass it and/or hold it ( for human postmaster inspection ) i. if use amavis with after queue filter , mail always needs daily support, and companies who stopped filtering in germany ( i dont know one ) have mostly a problem with helpless admins ignorant managers/users etc, not with law or existing antispam solutions so its mostly a human problem The point is - Before-Queue content filter is expansive and must be combined with cheap reject techologies anyway sorry explain cheap cheap as opposed to expansive in resource usage (CPU/RAM/Connection Slots). You surely don't want to hammer all the spam boots at your content-filter . - Tagging spam is nearly useless because no user like to poke through the dustbin to search for potential lost mail i dont understand, as you always need support mail, its no problem to solve user questions, only the rate of questions should be handable by the corosponding number of postmaster and/or supporters The problem is that the user after some time abandon to look in the spam folders and therefore fals positives are lost after tagging. - Spam-Bouncing is no option at all why ?, a bounce is no thing of evil, there will be bounces by several reasons ever I was speaking of bouncing by content filter detected spam. The sender address is faked anyway so bouncing spam *is* evil. You maybe confused bouncing with rejecting?? - In general the false positive rate is a higher and more difficult to find out with content filter compared to a sane set of reputation based filters i have false postive under 0,1 promille no problem here It as always a matter of preference. The more you try to achieve 100% spam free the more false positives you have to accept. As said the evil is not merely the rate but the possibility to get lost without notice. So the most reasonable approch is to ditch content filter at all and use a sane set of reputation based decisions and maybe greylisting to reject spam at earliest possible stage. you should always use all usefull antispam technics which make sense anyway ( specially that ones that are native in postfix ) greylisting is one of them , but in a few cases on my site simply does not work anymore defending bots so antispam is always a filter chain, the real antispam filter such as spamassassin should always be one of the last I don't speak about or even recommend to not use spam filtering, but content filter is sometimes the bigger problem compared to some slipping through spams. maybe, thats individual, like spam always is, competent postmaster should choose the right way in the right case Amen Regards Andreas no need to flame, i have no problem with supporting ca 10 mailservers with antispam enabled up to 1 mail addresses some spam always slipping trough,always some false positives , thats the nature of the beast, the goal is keeping that rate low in my case spam filtering is no such problem , as mailservers that have buggy dns setups are in rbls etc, after all, one of the biggest problems are false tagging to antispam filters in mail clients i.e outlook which produces more questions then server side filters, as most users dont understand their mail client settings no flame intended at all. Simply have to say that the conclusion (not from you but from earlier posts in this thread) that one *must* use content filter is plain wrong because
Re: Better spam filter for postfix
Original-Nachricht Datum: Fri, 16 Jul 2010 11:03:27 +0200 Von: Robert Schetterer rob...@schetterer.org An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix Am 16.07.2010 10:15, schrieb lst_ho...@kwsoft.de: Zitat von Robert Schetterer rob...@schetterer.org: Am 16.07.2010 09:27, schrieb lst_ho...@kwsoft.de: Zitat von Henrik K h...@hege.li: On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Regards Andreas why not use spamass-milter drops spam during smtp income stage this is allowed anyway, also clamav-milter with sanesecurity works nice this way, bouncing mail after recieve by whatever reason may produce backscatter, so it isnt a good idea in every case or country, normally you only flag spam and pass it and/or hold it ( for human postmaster inspection ) i. if use amavis with after queue filter , mail always needs daily support, and companies who stopped filtering in germany ( i dont know one ) have mostly a problem with helpless admins ignorant managers/users etc, not with law or existing antispam solutions so its mostly a human problem The point is - Before-Queue content filter is expansive and must be combined with cheap reject techologies anyway sorry explain cheap Content filtering where you process the WHOLE message is considered as expensive. Just processing a bunch of headers or checking the client against DNSBL/RHWL/DNSWL/etc or checking the client IP reputation or checking things like proper HELO/EHLO or or or is considered as cheep. if you have non negliable load - Tagging spam is nearly useless because no user like to poke through the dustbin to search for potential lost mail i dont understand, as you always need support mail, its no problem to solve user questions, only the rate of questions should be handable by the corosponding number of postmaster and/or supporters - Spam-Bouncing is no option at all why ?, a bounce is no thing of evil, there will be bounces by several reasons ever - In general the false positive rate is a higher and more difficult to find out with content filter compared to a sane set of reputation based filters i have false postive under 0,1 promille no problem here So the most reasonable approch is to ditch content filter at all and use a sane set of reputation based decisions and maybe greylisting to reject spam at earliest possible stage. you should always use all usefull antispam technics which make sense anyway ( specially that ones that are native in postfix ) greylisting is one of them , Greylisting is NOT native to Postfix! but in a few cases on my site simply does not work anymore defending bots so antispam is always a filter chain, the real antispam filter such as spamassassin should always be one of the last I don't speak about or even recommend to not use spam filtering, but content filter is sometimes the bigger problem compared to some slipping through spams. maybe, thats individual, like spam always is, competent postmaster should choose the right way in the right case Regards Andreas no need to flame, i have no problem with supporting ca 10 mailservers with antispam enabled up to 1 mail addresses some spam always slipping trough,always some false positives , thats the nature of the beast, the goal is keeping that rate low in my case spam filtering is no such problem , as mailservers that have buggy dns setups are in rbls etc, after all, one of the biggest problems are false tagging to antispam filters in mail clients i.e outlook which produces more questions then server side filters, as most users dont understand their mail client settings -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria -- GMX DSL: Internet-, Telefon- und Handy-Flat ab 19,99 EUR/mtl. Bis zu 150 EUR Startguthaben inklusive! http://portal.gmx.net/de/go/dsl
Re: Better spam filter for postfix
Am 16.07.2010 13:10, schrieb Steve: Original-Nachricht Datum: Fri, 16 Jul 2010 11:03:27 +0200 Von: Robert Schetterer rob...@schetterer.org An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix Am 16.07.2010 10:15, schrieb lst_ho...@kwsoft.de: Zitat von Robert Schetterer rob...@schetterer.org: Am 16.07.2010 09:27, schrieb lst_ho...@kwsoft.de: Zitat von Henrik K h...@hege.li: On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. In Germany many companies have given up on content filtering because it is not allowed to drop mail after accepting, if there is a chance that private mail *could* be involved. So with content filter your only choice would be to tag spam and let the user sort out, which lead to no advantage for using content filter at all. So content filter are mostly a selling point and not a favorable solution. Regards Andreas why not use spamass-milter drops spam during smtp income stage this is allowed anyway, also clamav-milter with sanesecurity works nice this way, bouncing mail after recieve by whatever reason may produce backscatter, so it isnt a good idea in every case or country, normally you only flag spam and pass it and/or hold it ( for human postmaster inspection ) i. if use amavis with after queue filter , mail always needs daily support, and companies who stopped filtering in germany ( i dont know one ) have mostly a problem with helpless admins ignorant managers/users etc, not with law or existing antispam solutions so its mostly a human problem The point is - Before-Queue content filter is expansive and must be combined with cheap reject techologies anyway sorry explain cheap Content filtering where you process the WHOLE message is considered as expensive. Just processing a bunch of headers or checking the client against DNSBL/RHWL/DNSWL/etc or checking the client IP reputation or checking things like proper HELO/EHLO or or or is considered as cheep. if you have non negliable load - Tagging spam is nearly useless because no user like to poke through the dustbin to search for potential lost mail i dont understand, as you always need support mail, its no problem to solve user questions, only the rate of questions should be handable by the corosponding number of postmaster and/or supporters - Spam-Bouncing is no option at all why ?, a bounce is no thing of evil, there will be bounces by several reasons ever - In general the false positive rate is a higher and more difficult to find out with content filter compared to a sane set of reputation based filters i have false postive under 0,1 promille no problem here So the most reasonable approch is to ditch content filter at all and use a sane set of reputation based decisions and maybe greylisting to reject spam at earliest possible stage. you should always use all usefull antispam technics which make sense anyway ( specially that ones that are native in postfix ) greylisting is one of them , Greylisting is NOT native to Postfix! i dont meant that, sorry for eventual missunderstoods whatever i think its all said happy sunny weekend but in a few cases on my site simply does not work anymore defending bots so antispam is always a filter chain, the real antispam filter such as spamassassin should always be one of the last I don't speak about or even recommend to not use spam filtering, but content filter is sometimes the bigger problem compared to some slipping through spams. maybe, thats individual, like spam always is, competent postmaster should choose the right way in the right case Regards Andreas no need to flame, i have no problem with supporting ca 10 mailservers with antispam enabled up to 1 mail addresses some spam always slipping trough,always some false positives , thats the nature of the beast, the goal is keeping that rate low in my case spam filtering is no such problem , as mailservers that have buggy dns setups are in rbls etc, after all, one of the biggest problems are false tagging to antispam filters in mail clients i.e outlook which produces more questions then server side filters, as most users dont understand their mail client settings -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: Better spam filter for postfix
Steve wrote: [big snip] So you have made your point. You prefer (or are required) to have user in control. Yes. The big problem is that no solution out there is 100% accurate for all users. So the only way to make the user happy is to delegate the control to him. Can't speek for all users. But I have the impression that users don't want to go through piles of spam and take action. They just expect the damn spam filter to work by itself. At least our users expect this :-) Mikael
Re: Better spam filter for postfix
On Fri, Jul 16, 2010 at 02:55:17PM +0200, Mikael Bak wrote: Steve wrote: [big snip] So you have made your point. You prefer (or are required) to have user in control. Yes. The big problem is that no solution out there is 100% accurate for all users. So the only way to make the user happy is to delegate the control to him. Can't speek for all users. But I have the impression that users don't want to go through piles of spam and take action. They just expect the damn spam filter to work by itself. At least our users expect this :-) Mikael Hi, Speaking for our environment, we use DSPAM with a pre-trained base so that when a user starts initially, they get reasonably good spam filter/false positive rates. This means that instead of piles of spam they have just a few mistakes and the accuracy increases quickly from there to the point that the vast majority of users have to train perhaps a couple of messages a month. The initial pretraining is good enough relative to other systems that many never train at all. Rule based filtering, on the other hand, was very labor intensive for the users and fraught with false-positive and negatives. As a member of the support team, we have many fewer problems regarding spam E-mail since we changed to DSPAM from a purely filter-based approach. We use SpamAssassin via amavisd-new and statistical filtering such as that provided by DSPAM, CRM114, and others do a much better job with much less maintenance. Cheers, Ken
Re: Better spam filter for postfix
Original-Nachricht Datum: Fri, 16 Jul 2010 14:55:17 +0200 Von: Mikael Bak mik...@t-online.hu An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix Steve wrote: [big snip] So you have made your point. You prefer (or are required) to have user in control. Yes. The big problem is that no solution out there is 100% accurate for all users. So the only way to make the user happy is to delegate the control to him. Can't speek for all users. But I have the impression that users don't want to go through piles of spam and take action. They just expect the damn spam filter to work by itself. At least our users expect this :-) Mine do the same. At least the bigger part of them. Learning a Anti-Spam filter is something that they consider black magic and anyway they have no time to do that. They want just good mails to arrive and bad to never reach their box. Often they them self don't know 100% what good and what bad is. But regardless they expect the Anti-Spam filter to know for them. So this is the reason I use something that allows me to train one dataset which is then merged at runtime with the individual user data. And since I know to code I have no problem to add additional small code that does the training automatically for them by using various techniques. So far most of my users NEVER see a spam mail in months. I have accounts that are 100% Spam free for over a year. And I have accounts that have never complained about false positive for ages. But then I have accounts that are not so easy to handle. One customer is in the steel trading business. Boy, boy, boy... some of those steel producing companies from eastern Europe or from Asia are always at least on one or more blacklists, have bad HELO/EHLO, no reverse DNS entries, failing on SPF and and and... Using something like greylisting is no option either because that damn steel price can change a bunch of cents in minutes and then multiply that with a gazillion of kilos a ship can transport and there you are: a lot of money can be lost by holding back a mail for 2 Minutes. Getting such a domain Spam free is a challenge. And so far only statistical Anti-Spam filters where capable to handle that for me. Forget SpamAssassin, forget the cheep tools that you can put in front of Postifx, etc. They all fail. Some more, some less. So you need to be very creative and thinking out of the normal border to get your job done when filtering such a domain. Mikael Steve -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
Re: Better spam filter for postfix
Original-Nachricht Datum: Fri, 16 Jul 2010 08:09:54 -0500 Von: Kenneth Marshall k...@rice.edu An: Mikael Bak mik...@t-online.hu CC: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix On Fri, Jul 16, 2010 at 02:55:17PM +0200, Mikael Bak wrote: Steve wrote: [big snip] So you have made your point. You prefer (or are required) to have user in control. Yes. The big problem is that no solution out there is 100% accurate for all users. So the only way to make the user happy is to delegate the control to him. Can't speek for all users. But I have the impression that users don't want to go through piles of spam and take action. They just expect the damn spam filter to work by itself. At least our users expect this :-) Mikael Hi, Speaking for our environment, we use DSPAM with a pre-trained base so that when a user starts initially, they get reasonably good spam filter/false positive rates. This means that instead of piles of spam they have just a few mistakes and the accuracy increases quickly from there to the point that the vast majority of users have to train perhaps a couple of messages a month. The initial pretraining is good enough relative to other systems that many never train at all. Rule based filtering, on the other hand, was very labor intensive for the users and fraught with false-positive and negatives. As a member of the support team, we have many fewer problems regarding spam E-mail since we changed to DSPAM from a purely filter-based approach. We use SpamAssassin via amavisd-new and statistical filtering such as that provided by DSPAM, CRM114, and others do a much better job with much less maintenance. And if I am not wrong that you have a large DSPAM installation. Right? Could you tell us how many users you have? Is the whole university campus using DSPAM? Cheers, Ken Steve -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
Re: Better spam filter for postfix
On Fri, Jul 16, 2010 at 08:06:11PM +0200, Steve wrote: Original-Nachricht Datum: Fri, 16 Jul 2010 08:09:54 -0500 Von: Kenneth Marshall k...@rice.edu An: Mikael Bak mik...@t-online.hu CC: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix On Fri, Jul 16, 2010 at 02:55:17PM +0200, Mikael Bak wrote: Steve wrote: [big snip] So you have made your point. You prefer (or are required) to have user in control. Yes. The big problem is that no solution out there is 100% accurate for all users. So the only way to make the user happy is to delegate the control to him. Can't speek for all users. But I have the impression that users don't want to go through piles of spam and take action. They just expect the damn spam filter to work by itself. At least our users expect this :-) Mikael Hi, Speaking for our environment, we use DSPAM with a pre-trained base so that when a user starts initially, they get reasonably good spam filter/false positive rates. This means that instead of piles of spam they have just a few mistakes and the accuracy increases quickly from there to the point that the vast majority of users have to train perhaps a couple of messages a month. The initial pretraining is good enough relative to other systems that many never train at all. Rule based filtering, on the other hand, was very labor intensive for the users and fraught with false-positive and negatives. As a member of the support team, we have many fewer problems regarding spam E-mail since we changed to DSPAM from a purely filter-based approach. We use SpamAssassin via amavisd-new and statistical filtering such as that provided by DSPAM, CRM114, and others do a much better job with much less maintenance. And if I am not wrong that you have a large DSPAM installation. Right? Could you tell us how many users you have? Is the whole university campus using DSPAM? We currently use DSPAM for our campus and have about 15k email accounts. The average number of tokens is less than 20k per person thanks to the pre-trained corpus. Cheers, Ken
Re: Better spam filter for postfix
Steve: Original-Nachricht Datum: Fri, 16 Jul 2010 16:44:23 -0400 Von: Charles Marcus cmar...@media-brokers.com An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix Steve, I request that you end this thread. Wietse
Better spam filter for postfix
As most of you guys know. I use mailscanner. I would like recomendations of what else to use. I prefer a all in one package like what mailscanner does. It also utilizes clamav and spamassion. The problem is most of the information I find on the net is outdated or for projects that stops. Seems like everybody has there way of dealing wiht spam filterting. So This is a ask of what you guys find the most usefull. I'm hosting mutiple domains (virtual via mysql) so I cannot be sepecific to each one. Also I'm using postini with some but not all the domains. Thanks, Josh -- This message has been scanned for viruses and dangerous content by Mychoice, and is believed to be clean.
Re: Better spam filter for postfix
Use greylisting, eg postgrey and set it up to work before amavisd-new or mailscanner. 2010/7/15 Josh Cason joc...@mychoice.cc As most of you guys know. I use mailscanner. I would like recomendations of what else to use. I prefer a all in one package like what mailscanner does. It also utilizes clamav and spamassion. The problem is most of the information I find on the net is outdated or for projects that stops. Seems like everybody has there way of dealing wiht spam filterting. So This is a ask of what you guys find the most usefull. I'm hosting mutiple domains (virtual via mysql) so I cannot be sepecific to each one. Also I'm using postini with some but not all the domains. Thanks, Josh -- This message has been scanned for viruses and dangerous content by Mychoice, and is believed to be clean.
Re: Better spam filter for postfix
Original-Nachricht Datum: Thu, 15 Jul 2010 19:37:48 +0200 Von: Ralf Hildebrandt ralf.hildebra...@charite.de An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix * Josh Cason joc...@mychoice.cc: As most of you guys know. I use mailscanner. I would like recomendations of what else to use. I prefer a all in one package like what mailscanner does. It also utilizes clamav and spamassion. So does amavisd-new If you looking for something that is beyond just being better then I recommend CRM114 or DSPAM or OSBF-Lua. If you insist in having the AV included in the Anti-Spam tool then use something like DSPAM. I use all of the above mentioned and all of them are fast and accurate. DSPAM is the one that is the easiest to scale and DSPAM is the one using the lowest amount of memory (DSPAM alone uses on my setup less then 10MB of memory for hundreds of domains having thousands of users in total). From a algorithm viewpoint CRM114 is a insane tool. It offers you a lot of algorithms and is virtually expendable to anything you like (it includes it's own language). If you used SA in the past then any of the above will surprise you in terms of speed, memory consumption and accuracy. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
Re: Better spam filter for postfix
Or sqlgrey, a fork of postgrey. http://sqlgrey.sourceforge.net/ On Jul 15, 2010, at 11:59 AM, Kai Krakow wrote: Use greylisting, eg postgrey and set it up to work before amavisd-new or mailscanner. 2010/7/15 Josh Cason joc...@mychoice.cc As most of you guys know. I use mailscanner. I would like recomendations of what else to use. I prefer a all in one package like what mailscanner does. It also utilizes clamav and spamassion. The problem is most of the information I find on the net is outdated or for projects that stops. Seems like everybody has there way of dealing wiht spam filterting. So This is a ask of what you guys find the most usefull. I'm hosting mutiple domains (virtual via mysql) so I cannot be sepecific to each one. Also I'm using postini with some but not all the domains. Thanks, Josh -- This message has been scanned for viruses and dangerous content by Mychoice, and is believed to be clean.
Re: Better spam filter for postfix
Original-Nachricht Datum: Thu, 15 Jul 2010 12:03:17 -0700 Von: Bradley Giesbrecht bradley.giesbre...@gmail.com An: postfix-users postfix-users@postfix.org Betreff: Re: Better spam filter for postfix Or sqlgrey, a fork of postgrey. http://sqlgrey.sourceforge.net/ Or GROSS (the only greylisting application that I know working with a bloom filter (http://en.wikipedia.org/wiki/Bloom_filter)). http://code.google.com/p/gross/ On Jul 15, 2010, at 11:59 AM, Kai Krakow wrote: Use greylisting, eg postgrey and set it up to work before amavisd-new or mailscanner. 2010/7/15 Josh Cason joc...@mychoice.cc As most of you guys know. I use mailscanner. I would like recomendations of what else to use. I prefer a all in one package like what mailscanner does. It also utilizes clamav and spamassion. The problem is most of the information I find on the net is outdated or for projects that stops. Seems like everybody has there way of dealing wiht spam filterting. So This is a ask of what you guys find the most usefull. I'm hosting mutiple domains (virtual via mysql) so I cannot be sepecific to each one. Also I'm using postini with some but not all the domains. Thanks, Josh -- This message has been scanned for viruses and dangerous content by Mychoice, and is believed to be clean. -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
Re: Better spam filter for postfix
On 07/15/2010 12:29 PM, Steve wrote: Or GROSS (the only greylisting application that I know working with a bloom filter (http://en.wikipedia.org/wiki/Bloom_filter)). http://code.google.com/p/gross/ Thanks for the link, what I see there is very interesting - I'll check this out... Joe
Re: Better spam filter for postfix
On Thu, Jul 15, 2010 at 09:02:52PM +0200, Steve wrote: Original-Nachricht Datum: Thu, 15 Jul 2010 19:37:48 +0200 Von: Ralf Hildebrandt ralf.hildebra...@charite.de An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix * Josh Cason joc...@mychoice.cc: As most of you guys know. I use mailscanner. I would like recomendations of what else to use. I prefer a all in one package like what mailscanner does. It also utilizes clamav and spamassion. So does amavisd-new If you looking for something that is beyond just being better then I recommend CRM114 or DSPAM or OSBF-Lua. If you insist in having the AV included in the Anti-Spam tool then use something like DSPAM. I'd consider those as engines. You can run one or all of them if you really want. MailScanner, Amavisd-new, Mimedefang and even SA (as a framework) are some of the glues that might utilize them. Also ClamAV isn't just an AV tool. It's a lot more of an Anti-Spam tool when used with Sanesecurity signatures etc. There are a million combinations of glues, engines and other general anti-spam methods. You need to be very clear on your needs to get a meaningful answer (and maybe not even then). I use all of the above mentioned and all of them are fast and accurate. DSPAM is the one that is the easiest to scale and DSPAM is the one using the lowest amount of memory (DSPAM alone uses on my setup less then 10MB of memory for hundreds of domains having thousands of users in total). From a algorithm viewpoint CRM114 is a insane tool. It offers you a lot of algorithms and is virtually expendable to anything you like (it includes it's own language). If you used SA in the past then any of the above will surprise you in terms of speed, memory consumption and accuracy. Generally DSPAM etc require user interaction/learning. SA does not, since it's a framework of rules and plugins and can autolearn Bayes if you want to - or even do the same for DSPAM etc if you use them as SA plugins. Let's not forget that DSPAM etc also require a database backend, which might require lots of memory and/or disk, so it's not exactly free either. Accuracy depends heavily on configuration of all the components and other voodoo. There are no easy answers.
Re: Better spam filter for postfix
Original-Nachricht Datum: Thu, 15 Jul 2010 23:54:22 +0300 Von: Henrik K h...@hege.li An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix On Thu, Jul 15, 2010 at 09:02:52PM +0200, Steve wrote: Original-Nachricht Datum: Thu, 15 Jul 2010 19:37:48 +0200 Von: Ralf Hildebrandt ralf.hildebra...@charite.de An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix * Josh Cason joc...@mychoice.cc: As most of you guys know. I use mailscanner. I would like recomendations of what else to use. I prefer a all in one package like what mailscanner does. It also utilizes clamav and spamassion. So does amavisd-new If you looking for something that is beyond just being better then I recommend CRM114 or DSPAM or OSBF-Lua. If you insist in having the AV included in the Anti-Spam tool then use something like DSPAM. I'd consider those as engines. You can run one or all of them if you really want. MailScanner, Amavisd-new, Mimedefang and even SA (as a framework) are some of the glues that might utilize them. Well those so called engines can run on their own. They don't need to be wrapped inside any of the glues you mention. Especially not when those glues are memory hogs. Also ClamAV isn't just an AV tool. It's a lot more of an Anti-Spam tool when used with Sanesecurity signatures etc. There are a million combinations of glues, engines and other general anti-spam methods. You need to be very clear on your needs to get a meaningful answer (and maybe not even then). I use all of the above mentioned and all of them are fast and accurate. DSPAM is the one that is the easiest to scale and DSPAM is the one using the lowest amount of memory (DSPAM alone uses on my setup less then 10MB of memory for hundreds of domains having thousands of users in total). From a algorithm viewpoint CRM114 is a insane tool. It offers you a lot of algorithms and is virtually expendable to anything you like (it includes it's own language). If you used SA in the past then any of the above will surprise you in terms of speed, memory consumption and accuracy. Generally DSPAM etc require user interaction/learning. So does CRM114 and OSBF-Lua. But you are wrong in thinking that they need an insane amount of training/learning. SA does not, since it's a framework of rules and plugins and can autolearn Bayes if you want to - or even do the same for DSPAM etc if you use them as SA plugins. Let's not forget that DSPAM etc also require a database backend, You are WRONG. DSPAM does NOT require a database backend. I don't know where you have that from? DSPAM MIGHT use a database backend but can run well without one (using the Hash driver). which might require lots of memory and/or disk, so it's not exactly free either. Accuracy depends heavily on configuration of all the components and other voodoo. What? Voodoo? Yeah right. There is less voodoo in CRM114, OSBF-Lua and DSPAM then in SA. I explain a user the following: * you get mail and if it is wrongly classified by the Anti-Spam filter then you correct it and the filter will learn. * the wrong classification is done based on YOUR prior classification you have feed to the Anti-Spam filter. * if you feed wrong data to the Anti-Spam filter then the filter will make errors. * the more you correct the higher the accuracy gets and you need less and less to correct errors. That's easy to understand. IMHO it is easier to explain then telling the user: * there is an army of rule writers out there that is writing rules for SA where THEY are telling what is spam and what is ham. And if the user asks me: what rules are that? Then I would need to say that there are a gazillion of rules that I can not explain in detail without taking much of his time to go throw all the rules one by one. Anyway... For me the three mentioned products are all better then SA because they have a smaller memory footprint then SA and are way faster then SA and properly set up require less maintenance and are way more accurate then SA. And regarding the training: DSPAM and CRM114 offers features where you can pre-learn so that your users are having from day one already a high accuracy (generally above 95%) and if they re-classify the first bunch of errors then their accuracy jumps easy over 98.x%/99.x%. In DSPAM that kind of setup is accomplished with merged groups or classification groups or shared groups. In CRM114 you can at run time allocate and merge as many CSS files (one pre-trained should be enough) as you like. There are no easy answers. And this is generally the field where Anti-Spam tools that do not depend on pre-made rules are shining, because they are very adaptive. -- GMX DSL: Internet-, Telefon- und Handy-Flat ab 19,99 EUR/mtl. Bis zu 150 EUR Startguthaben inklusive! http://portal.gmx.net/de/go/dsl
Re: Better spam filter for postfix
On Thu, Jul 15, 2010 at 11:16:43PM +0200, Steve wrote: If you looking for something that is beyond just being better then I recommend CRM114 or DSPAM or OSBF-Lua. If you insist in having the AV included in the Anti-Spam tool then use something like DSPAM. I'd consider those as engines. You can run one or all of them if you really want. MailScanner, Amavisd-new, Mimedefang and even SA (as a framework) are some of the glues that might utilize them. Well those so called engines can run on their own. They don't need to be wrapped inside any of the glues you mention. Especially not when those glues are memory hogs. Can you be more specific? Maybe you are addressing SA memory usage, which might only matter on some cases. Servers have lots of memory these days, and good MTA checks might reduce scanning needs greatly. Generally DSPAM etc require user interaction/learning. So does CRM114 and OSBF-Lua. But you are wrong in thinking that they need an insane amount of training/learning. That's what I meant with etc. I did use DSPAM exclusively for few months in the past, but for my personal use I saw no benefits from it. SA does not, since it's a framework of rules and plugins and can autolearn Bayes if you want to - or even do the same for DSPAM etc if you use them as SA plugins. Let's not forget that DSPAM etc also require a database backend, You are WRONG. DSPAM does NOT require a database backend. I don't know where you have that from? DSPAM MIGHT use a database backend but can run well without one (using the Hash driver). So you don't consider the CSS Hash driver a database backend? It requires disk, memory and CPU to store and retrieve tokens. Whatever.. which might require lots of memory and/or disk, so it's not exactly free either. Accuracy depends heavily on configuration of all the components and other voodoo. What? Voodoo? Yeah right. There is less voodoo in CRM114, OSBF-Lua and DSPAM then in SA. I explain a user the following: * you get mail and if it is wrongly classified by the Anti-Spam filter then you correct it and the filter will learn. * the wrong classification is done based on YOUR prior classification you have feed to the Anti-Spam filter. * if you feed wrong data to the Anti-Spam filter then the filter will make errors. * the more you correct the higher the accuracy gets and you need less and less to correct errors. That's easy to understand. IMHO it is easier to explain then telling the user: * there is an army of rule writers out there that is writing rules for SA where THEY are telling what is spam and what is ham. And if the user asks me: what rules are that? Then I would need to say that there are a gazillion of rules that I can not explain in detail without taking much of his time to go throw all the rules one by one. Anyway... So you have made your point. You prefer (or are required) to have user in control. I guess you don't use ANY other methods (blacklists etc) than users own statistical input, since you might have to tell your users that THEY though your mail was spam? For me the three mentioned products are all better then SA because they have a smaller memory footprint then SA and are way faster then SA and properly set up require less maintenance and are way more accurate then SA. Good for you. Naturally resource usage is lower, the less stuff you do. One has to balance needs against that. But let's forget the accuracy bs, there are too many variables for such generic claims to be made. You can achieve happy users with pretty much any tool out there if used right. I'm in a happy position to be able to reject/quarantine spam for 1000+ users without ever bothering them with it, and very rarely get any questions about mail. If I had to do it the ISP way, I might consider DSPAM, then again I see nothing against using SA (or any other tool out there). And regarding the training: DSPAM and CRM114 offers features where you can pre-learn so that your users are having from day one already a high accuracy (generally above 95%) and if they re-classify the first bunch of errors then their accuracy jumps easy over 98.x%/99.x%. In DSPAM that kind of setup is accomplished with merged groups or classification groups or shared groups. In CRM114 you can at run time allocate and merge as many CSS files (one pre-trained should be enough) as you like You make it sound like statistical filters are invincible against different mail flows and pure user stupidity. There are no easy answers. And this is generally the field where Anti-Spam tools that do not depend on pre-made rules are shining, because they are very adaptive. Right, like SA for example only depends on pre-made rules and doesn't have any statistical or realtime capabilities.. I think continuing this is pointless and a bit off-topic.
Re: Better spam filter for postfix
Original-Nachricht Datum: Fri, 16 Jul 2010 02:09:43 +0300 Von: Henrik K h...@hege.li An: postfix-users@postfix.org Betreff: Re: Better spam filter for postfix On Thu, Jul 15, 2010 at 11:16:43PM +0200, Steve wrote: If you looking for something that is beyond just being better then I recommend CRM114 or DSPAM or OSBF-Lua. If you insist in having the AV included in the Anti-Spam tool then use something like DSPAM. I'd consider those as engines. You can run one or all of them if you really want. MailScanner, Amavisd-new, Mimedefang and even SA (as a framework) are some of the glues that might utilize them. Well those so called engines can run on their own. They don't need to be wrapped inside any of the glues you mention. Especially not when those glues are memory hogs. Can you be more specific? Maybe you are addressing SA memory usage, which might only matter on some cases. Servers have lots of memory these days, and good MTA checks might reduce scanning needs greatly. Yes. Servers have a lot of memory those days but not enough memory to waste it. My point is not only memory. My biggest problem with tools such as SA is that it is very slow compared to other solutions out there. I in general can say that I classify x messages per second with filter XYZ while I in general would say that SpamAssassin needs x seconds per message. All the test in the past I have done with SpamAssassin confirm that statement. And for me system resources are important. Be it memory, CPU cycles, throughput etc... Generally DSPAM etc require user interaction/learning. So does CRM114 and OSBF-Lua. But you are wrong in thinking that they need an insane amount of training/learning. That's what I meant with etc. I did use DSPAM exclusively for few months in the past, but for my personal use I saw no benefits from it. Okay. SA does not, since it's a framework of rules and plugins and can autolearn Bayes if you want to - or even do the same for DSPAM etc if you use them as SA plugins. Let's not forget that DSPAM etc also require a database backend, You are WRONG. DSPAM does NOT require a database backend. I don't know where you have that from? DSPAM MIGHT use a database backend but can run well without one (using the Hash driver). So you don't consider the CSS Hash driver a database backend? It requires disk, memory and CPU to store and retrieve tokens. Whatever.. Well... it has a structure but I would not consider it a database in the classical way. If the CSS file is a database then a XML file is a database too and I personally don't consider a XML file to be a database. which might require lots of memory and/or disk, so it's not exactly free either. Accuracy depends heavily on configuration of all the components and other voodoo. What? Voodoo? Yeah right. There is less voodoo in CRM114, OSBF-Lua and DSPAM then in SA. I explain a user the following: * you get mail and if it is wrongly classified by the Anti-Spam filter then you correct it and the filter will learn. * the wrong classification is done based on YOUR prior classification you have feed to the Anti-Spam filter. * if you feed wrong data to the Anti-Spam filter then the filter will make errors. * the more you correct the higher the accuracy gets and you need less and less to correct errors. That's easy to understand. IMHO it is easier to explain then telling the user: * there is an army of rule writers out there that is writing rules for SA where THEY are telling what is spam and what is ham. And if the user asks me: what rules are that? Then I would need to say that there are a gazillion of rules that I can not explain in detail without taking much of his time to go throw all the rules one by one. Anyway... So you have made your point. You prefer (or are required) to have user in control. Yes. The big problem is that no solution out there is 100% accurate for all users. So the only way to make the user happy is to delegate the control to him. I guess you don't use ANY other methods (blacklists etc) than users own statistical input, since you might have to tell your users that THEY though your mail was spam? No. I use other methods. A lot of them. I even developed my own stuff based on research papers from Anti-Spam researchers/companies. My setup is made that way that I have made many defense rings around Postfix. Each ring has it's own techniques and the father the ring is from Postfix the less resources it uses. However... each domain owner and/or user has control over the rings. He/she can turn them on/off, depending on their needs. I preset which are on and which are off but at the end each one of them is controllable by the end-user (or domain owner, which precedes user rules). Some stuff however is not controllable by the end user or domain
Re: Better spam filter for postfix
Steve put forth on 7/15/2010 4:16 PM: * if you feed wrong data to the Anti-Spam filter then the filter will make errors. Content (header/body) filters have always been error prone and always will be. The key to success is if the error rate is acceptable. For users to train them, they have to be run in post-queue mode. For performance reasons, most OPs run them in post-queue mode anyway. And by doing this you're unnecessarily eating b/w on your internet link(s). There are plenty of good methods available to drop spam connections at SMTP time without ever having to accept the spam for content analysis. I use many such methods, and I don't use content filters. Never have. I probably spend more time fighting spam than other OPs do. Using content filters such as SA can definitely cut down on mail OP time spent fighting spam. Which method is more effective depends on one's priorities, and thus this subject can be debated ad infinitum. I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. YMMV, etc. -- Stan
Re: Better spam filter for postfix
On Thu, Jul 15, 2010 at 11:06:44PM -0500, Stan Hoeppner wrote: I will say generically that for an OP who has the time, avoiding content filters and using SMTP time blocking methods is probably more effective in the long run and makes more efficient use of network and server resources. You always have time to advertise content filters being bad, so I just have to make a pointless rebuttal.. Can you tell me any big public service (not a one man server) that doesn't use content filtering at all? By public I don't mean a site that has the ability to block freemailers, universities, etc hacked accounts.. I'm sure any serious site uses lots of SMTP time rejects, but you _need_ some sort of content filtering for the rest. Unless you bear the burden on clients MUA. PS. I think I've spent maybe an hour or two maintaining our mail server in the last few months, and it's still running fine.. how is that not efficient? My work time costs much more than the imaginary network and server resources.
