Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Justin Mason wrote: > > Matt, any chance you could open a bz feature request for this? > Done.. also added to it so there's now 3 warning level options. http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5710 > --j. >
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Matt Kettler writes: > Daryl C. W. O'Shea wrote: > > Justin Mason wrote: > >> OK, we really need to figure out some way to kill these FAQs off. Every > >> week, someone asks a question about why SpamAssassin is killing their > >> server, and most of the time the answer is "stop using blacklist.cf and > >> blacklist-uri.cf". If 1 person is asking the question, chances are > >> there's another 10 people who aren't asking, and who are just ditching > >> SpamAssassin entirely. :( > > > >> I think I'll add a new question right on the top of the FAQ list > >> about this... > >> > >> What else can we do? > > > > Has anyone asked Bill to stop distributing the blacklist in a format > > suitable for direct use with SpamAssassin? That, to me, seems to be > > the most effective and sensible way to deal with it. > I'd agree there. > > Modifying the software, as has been discussed, seems a little > > extreme to me. > I dono, I think that having some --lint warnings generated when the > overall config is really absurdly large seems useful for this kind of > problem in general. A basic "um, dude, that's a lot of config, are you > sure your server can handle this" might be a good thing. You never know > when someone else might make a sa-blacklist, or some tool that > auto-generates rules might get popular and get out-of-control > sometimes.. etc.. Matt, any chance you could open a bz feature request for this? --j.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Matt Kettler wrote: > Daniel J McDonald wrote: > >> On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote: >> >> >>> Justin Mason wrote: >>> >>> What else can we do? >>> Add code to generate a lint warning any time a .cf file over 1mb is read >>> unless a config option is set to silence it? >>> >>> >> But people don't read logs, or they would know... I'd suggest die-ing >> instead. >> > > -1 on die-ing. There's no precedent for that in any of SA's existing > behavior, even under the most severe config errors. This is largely done > because it could screw over someones mail queue following an upgrade. > > And in this case, even if they read the logs, or ran a --lint, they > wouldn't know anything other than "it's slow" and maybe "it eats memory > like a hog". At least this way we can reach the ones that actually check > the basics. > > > > another example: machine reboots and for some reason, a new big file is there. nobody to read logs. I don't think there is now a need to have a general solution for large files. the problem of the black*.cf can be solved by - rename the file to something like READWARNINGINSIDEFILE-black*.cf - create empty black*.cf there's no reason to punish those who do efforts to listen to advice. those who blindly download files with RDJ or other will notice that their system is faster and that some spam may be missed. not a critical issue.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
jdow wrote: > From: "Matt Kettler" <[EMAIL PROTECTED]> > Sent: Friday, 2007, October 26 20:19 > >> I dono, I think that having some --lint warnings generated when the >> overall config is really absurdly large seems useful for this kind of >> problem in general. A basic "um, dude, that's a lot of config, are you >> sure your server can handle this" might be a good thing. You never know >> when someone else might make a sa-blacklist, or some tool that >> auto-generates rules might get popular and get out-of-control >> sometimes.. etc.. >> >> However, the whole idea of having it kill SA is way out-of-bounds, IMHO. >> SA won't even do that if you feed it a conf file full of output from >> /dev/random... > > The problem there, Matt, is that the definition of absurdly large varies > greatly with application. > True, which is why I wanted the warning level to be a configurable option. That way folks can see the warning, decide if there machine is big enough, and silence the warning if desired.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
From: "Matt Kettler" <[EMAIL PROTECTED]> Sent: Friday, 2007, October 26 20:19 Daryl C. W. O'Shea wrote: Justin Mason wrote: OK, we really need to figure out some way to kill these FAQs off. Every week, someone asks a question about why SpamAssassin is killing their server, and most of the time the answer is "stop using blacklist.cf and blacklist-uri.cf". If 1 person is asking the question, chances are there's another 10 people who aren't asking, and who are just ditching SpamAssassin entirely. :( I think I'll add a new question right on the top of the FAQ list about this... What else can we do? Has anyone asked Bill to stop distributing the blacklist in a format suitable for direct use with SpamAssassin? That, to me, seems to be the most effective and sensible way to deal with it. I'd agree there. Modifying the software, as has been discussed, seems a little extreme to me. I dono, I think that having some --lint warnings generated when the overall config is really absurdly large seems useful for this kind of problem in general. A basic "um, dude, that's a lot of config, are you sure your server can handle this" might be a good thing. You never know when someone else might make a sa-blacklist, or some tool that auto-generates rules might get popular and get out-of-control sometimes.. etc.. However, the whole idea of having it kill SA is way out-of-bounds, IMHO. SA won't even do that if you feed it a conf file full of output from /dev/random... The problem there, Matt, is that the definition of absurdly large varies greatly with application. {^_-}
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
> > > But people don't read logs, or they would know... I'd suggest die-ing > > > instead. > On Fri, 26 Oct 2007, Duane Hill wrote: > > Why not make it a configurable option in local.cf defaulting to > > die. That way for those of us who create custom .cf files that > > have the system resources can do so and not have to split them up > > into more than one file. On 26.10.07 09:43, John D. Hardin wrote: > No, the size-to-die-at should be configurable, not whether you die or > warn. If you *want* to support large custom config files, then up the > limit. ...and some people would be surprised why is SA dying after they added one line to their config file, instead of being a bit slower. No, I don't think this is a good idea. -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Boost your system's speed by 500% - DEL C:\WINDOWS\*.*
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Add code to generate a lint warning any time a .cf file over 1mb is read unless a config option is set to silence it? Simple solution, assuming someone has control of the machine with blacklist.cf on it: Replace it with a single blank line. No complaints about "SA stopped working!". Just suddenly things run faster. Loren
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Daryl C. W. O'Shea wrote: > Justin Mason wrote: >> OK, we really need to figure out some way to kill these FAQs off. Every >> week, someone asks a question about why SpamAssassin is killing their >> server, and most of the time the answer is "stop using blacklist.cf and >> blacklist-uri.cf". If 1 person is asking the question, chances are >> there's another 10 people who aren't asking, and who are just ditching >> SpamAssassin entirely. :( > >> I think I'll add a new question right on the top of the FAQ list >> about this... >> >> What else can we do? > > Has anyone asked Bill to stop distributing the blacklist in a format > suitable for direct use with SpamAssassin? That, to me, seems to be > the most effective and sensible way to deal with it. I'd agree there. > Modifying the software, as has been discussed, seems a little > extreme to me. I dono, I think that having some --lint warnings generated when the overall config is really absurdly large seems useful for this kind of problem in general. A basic "um, dude, that's a lot of config, are you sure your server can handle this" might be a good thing. You never know when someone else might make a sa-blacklist, or some tool that auto-generates rules might get popular and get out-of-control sometimes.. etc.. However, the whole idea of having it kill SA is way out-of-bounds, IMHO. SA won't even do that if you feed it a conf file full of output from /dev/random...
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Justin Mason wrote: OK, we really need to figure out some way to kill these FAQs off. Every week, someone asks a question about why SpamAssassin is killing their server, and most of the time the answer is "stop using blacklist.cf and blacklist-uri.cf". If 1 person is asking the question, chances are there's another 10 people who aren't asking, and who are just ditching SpamAssassin entirely. :( I think I'll add a new question right on the top of the FAQ list about this... What else can we do? Has anyone asked Bill to stop distributing the blacklist in a format suitable for direct use with SpamAssassin? That, to me, seems to be the most effective and sensible way to deal with it. Modifying the software, as has been discussed, seems a little extreme to me. Daryl
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
On 10/26/07 6:59 PM, "Matt Kettler" <[EMAIL PROTECTED]> wrote: > Daniel J McDonald wrote: >> On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote: >> >>> Justin Mason wrote: >>> What else can we do? >>> Add code to generate a lint warning any time a .cf file over 1mb is read >>> unless a config option is set to silence it? >>> >> >> But people don't read logs, or they would know... I'd suggest die-ing >> instead. > > -1 on die-ing. There's no precedent for that in any of SA's existing > behavior, even under the most severe config errors. This is largely done > because it could screw over someones mail queue following an upgrade. > > And in this case, even if they read the logs, or ran a --lint, they > wouldn't know anything other than "it's slow" and maybe "it eats memory > like a hog". At least this way we can reach the ones that actually check > the basics. > For what it's worth: I stopped using blacklist.cf back around 2.1.3..haven't seen much of a difference with or without itone of the companies I work with has: File messages : from Oct 1 00:01:37 to Oct 26 19:28:48 Total number of emails processed by the spam filter : 74552 Number of spams : 69518 ( 93.25%) Number of clean messages: 5034 ( 6.75%) Average message analysis time : 6.73 seconds Average spam analysis time : 6.61 seconds Average clean message analysis time : 8.29 seconds Average message score : 21.62 Average spam score : 23.80 Average clean message score : -6.44 Total spam volume : 345 Mbytes Total clean volume :92 Mbytes We are well pleased with SA without blacklist.cf :) James
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Daniel J McDonald wrote: > On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote: > >> Justin Mason wrote: >> >>> What else can we do? >>> >>> >> Add code to generate a lint warning any time a .cf file over 1mb is read >> unless a config option is set to silence it? >> > > But people don't read logs, or they would know... I'd suggest die-ing > instead. -1 on die-ing. There's no precedent for that in any of SA's existing behavior, even under the most severe config errors. This is largely done because it could screw over someones mail queue following an upgrade. And in this case, even if they read the logs, or ran a --lint, they wouldn't know anything other than "it's slow" and maybe "it eats memory like a hog". At least this way we can reach the ones that actually check the basics.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
On Fri, 26 Oct 2007, Nigel Frankcom wrote: > On Fri, 26 Oct 2007 09:43:37 -0700 (PPT), "John D. Hardin" > <[EMAIL PROTECTED]> wrote: > > >On Fri, 26 Oct 2007, Duane Hill wrote: > > > >> > But people don't read logs, or they would know... I'd suggest die-ing > >> > instead. > >> > >> Why not make it a configurable option in local.cf defaulting to > >> die. That way for those of us who create custom .cf files that > >> have the system resources can do so and not have to split them up > >> into more than one file. > > > >No, the size-to-die-at should be configurable, not whether you die or > >warn. If you *want* to support large custom config files, then up the > >limit. > > Perhaps a little more info about each rule would be helpful? I've > ended up with mine through a variety of trial and error and list post > comments and suggestions. Huh? We're discussing adding a capability for limiting rules file sizes, so that things like blacklist.cf can be made obviously painful. This isn't about individual rules - though I suppose if you tried hard enough you could write a 50kb RE... Is that what you were commenting on? Here's my topical comment: in addition to globally upping the limit, perhaps an explicit per-filename size limit bypass as well? CONFIG_FILE_SIZE_LIMIT 100kb ACCEPT_LARGE_CONFIG_FILE generated_rules_01.cf -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED] key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- ...the Fates notice those who buy chainsaws... -- www.darwinawards.com --- 5 days until Halloween
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
On Fri, 26 Oct 2007 09:43:37 -0700 (PPT), "John D. Hardin" <[EMAIL PROTECTED]> wrote: >On Fri, 26 Oct 2007, Duane Hill wrote: > >> > But people don't read logs, or they would know... I'd suggest die-ing >> > instead. >> >> Why not make it a configurable option in local.cf defaulting to >> die. That way for those of us who create custom .cf files that >> have the system resources can do so and not have to split them up >> into more than one file. > >No, the size-to-die-at should be configurable, not whether you die or >warn. If you *want* to support large custom config files, then up the >limit. Perhaps a little more info about each rule would be helpful? I've ended up with mine through a variety of trial and error and list post comments and suggestions. I run SA on a dedicated machine and it has had problems in the past, though admittedly some of those could have been attributed to a combination of remote DNS and remote MySQL. Still, some explanation regarding the caveats (which _are_ included in some rules info) could help the process some? Just my 2p worth. Kind regards Nigel. BTW - 5 days to Halloween and the little buggers are knocking my door already - some things American should remain American! :-D
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
At 04:52 26-10-2007, Justin Mason wrote: What else can we do? Add a warning at startup if a sample message takes more than X seconds to be scanned. Whether people will actually read that warning is another question. Regards, -sm
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
On Fri, 26 Oct 2007 09:43:37 -0700 (PPT) "John D. Hardin" <[EMAIL PROTECTED]> confabulated: > On Fri, 26 Oct 2007, Duane Hill wrote: > > > > But people don't read logs, or they would know... I'd suggest > > > die-ing instead. > > > > Why not make it a configurable option in local.cf defaulting to > > die. That way for those of us who create custom .cf files that > > have the system resources can do so and not have to split them up > > into more than one file. > > No, the size-to-die-at should be configurable, not whether you die or > warn. If you *want* to support large custom config files, then up the > limit. Yes. That would be better. -- _|_ (_| |
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
On Fri, 26 Oct 2007, Duane Hill wrote: > > But people don't read logs, or they would know... I'd suggest die-ing > > instead. > > Why not make it a configurable option in local.cf defaulting to > die. That way for those of us who create custom .cf files that > have the system resources can do so and not have to split them up > into more than one file. No, the size-to-die-at should be configurable, not whether you die or warn. If you *want* to support large custom config files, then up the limit. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED] key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- ...the Fates notice those who buy chainsaws... -- www.darwinawards.com --- 5 days until Halloween
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
On Fri, 26 Oct 2007 07:35:13 -0500 Daniel J McDonald <[EMAIL PROTECTED]> confabulated: > On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote: > > Justin Mason wrote: > > > > > > What else can we do? > > > > > Add code to generate a lint warning any time a .cf file over 1mb is > > read unless a config option is set to silence it? > > But people don't read logs, or they would know... I'd suggest die-ing > instead. Why not make it a configurable option in local.cf defaulting to die. That way for those of us who create custom .cf files that have the system resources can do so and not have to split them up into more than one file. -- _|_ (_| |
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Matt Kettler writes: > Justin Mason wrote: > > OK, we really need to figure out some way to kill these FAQs off. Every > > week, someone asks a question about why SpamAssassin is killing their > > server, and most of the time the answer is "stop using blacklist.cf and > > blacklist-uri.cf". If 1 person is asking the question, chances are > > there's another 10 people who aren't asking, and who are just ditching > > SpamAssassin entirely. :( > > > > > > RDJ folks -- can you zero out, or remove, those two files from the list > > entirely? It doesn't seem to matter if we say "don't use them" on > > our websites, people will set up RDJ to download everything anyway > > it seems. > > > That will help a little. However, a lot of folks using RDJ are using > really old versions. Remember how many folks started posting to the list > after I modified antidrug.cf to generate errors? This happened long > after I got them to modify RDJ to not include it. > > There also seem to be a some sites out there that have copies of RDJ > which aren't recent. For example, Fortress Systems (fsl.com, commercial > MailScanner) still has an old copy on their "resources" site that still > supports antidrug,cf and if enabled will to download antidrug.cf from > comcast. (The updated version lives on sandgnat.com) They're default > config doesn't have it in the trusted rulesets, but they really > shouldn't have support for it in their script at all anymore. > > (In other news, I finally canceled the comcast account on Tuesday. So > that one is now completely out of my control. Fortress Systems, are you > listening? I'll try posting on the MailScanner list later..) > > > I think I'll add a new question right on the top of the FAQ list > > about this... > > > > What else can we do? > > > Add code to generate a lint warning any time a .cf file over 1mb is read > unless a config option is set to silence it? > > Possibly even have this as as: > warn_conffile_maxsize (speced in KB, default 1024) > > Users that want to use absurdly large files can just raise the number.. > > We could do the same with a warning based on rule count, and/or > white/blacklist entries. > > Of course, we might need to do a little research as to what's > reasonable, but certainly the numbers in the blacklist files are a good > example of what's not reasonable.. +1, this is a good idea. Any one .cf file should not be that large. --j.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote: > Justin Mason wrote: > > > > What else can we do? > > > Add code to generate a lint warning any time a .cf file over 1mb is read > unless a config option is set to silence it? But people don't read logs, or they would know... I'd suggest die-ing instead. > > Possibly even have this as as: > warn_conffile_maxsize (speced in KB, default 1024) > > Users that want to use absurdly large files can just raise the number.. +1 -- Daniel J McDonald, CCIE #2495, CISSP #78281, CNX Austin Energy http://www.austinenergy.com
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Quoting Matt Kettler <[EMAIL PROTECTED]>: > Justin Mason wrote: > > OK, we really need to figure out some way to kill these FAQs off. Every > > week, someone asks a question about why SpamAssassin is killing their > > server, and most of the time the answer is "stop using blacklist.cf and > > blacklist-uri.cf". If 1 person is asking the question, chances are > > there's another 10 people who aren't asking, and who are just ditching > > SpamAssassin entirely. :( > > > > > > RDJ folks -- can you zero out, or remove, those two files from the list > > entirely? It doesn't seem to matter if we say "don't use them" on > > our websites, people will set up RDJ to download everything anyway > > it seems. +1 > Add code to generate a lint warning any time a .cf file over 1mb is read > unless a config option is set to silence it? > > Possibly even have this as as: > warn_conffile_maxsize (speced in KB, default 1024) > > Users that want to use absurdly large files can just raise the number.. > > We could do the same with a warning based on rule count, and/or +1 Jeff C.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Justin Mason wrote: > OK, we really need to figure out some way to kill these FAQs off. Every > week, someone asks a question about why SpamAssassin is killing their > server, and most of the time the answer is "stop using blacklist.cf and > blacklist-uri.cf". If 1 person is asking the question, chances are > there's another 10 people who aren't asking, and who are just ditching > SpamAssassin entirely. :( > > > RDJ folks -- can you zero out, or remove, those two files from the list > entirely? It doesn't seem to matter if we say "don't use them" on > our websites, people will set up RDJ to download everything anyway > it seems. > That will help a little. However, a lot of folks using RDJ are using really old versions. Remember how many folks started posting to the list after I modified antidrug.cf to generate errors? This happened long after I got them to modify RDJ to not include it. There also seem to be a some sites out there that have copies of RDJ which aren't recent. For example, Fortress Systems (fsl.com, commercial MailScanner) still has an old copy on their "resources" site that still supports antidrug,cf and if enabled will to download antidrug.cf from comcast. (The updated version lives on sandgnat.com) They're default config doesn't have it in the trusted rulesets, but they really shouldn't have support for it in their script at all anymore. (In other news, I finally canceled the comcast account on Tuesday. So that one is now completely out of my control. Fortress Systems, are you listening? I'll try posting on the MailScanner list later..) > I think I'll add a new question right on the top of the FAQ list > about this... > > What else can we do? > Add code to generate a lint warning any time a .cf file over 1mb is read unless a config option is set to silence it? Possibly even have this as as: warn_conffile_maxsize (speced in KB, default 1024) Users that want to use absurdly large files can just raise the number.. We could do the same with a warning based on rule count, and/or white/blacklist entries. Of course, we might need to do a little research as to what's reasonable, but certainly the numbers in the blacklist files are a good example of what's not reasonable..