Re: [Razor-users] Razor2 spam detection rate (bayes poison/e4)
At 03:55 PM 12/15/2003, [EMAIL PROTECTED] wrote: Hash the rendered (via links -dump) html ? They already strip HTML.. The problem is what if the bayes poison IS renderable, which is true of a lot of current bayes poison.. --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On 12/15, Matt Kettler wrote: > If they are doing random poison inserted throughout the entire message, and > they re-generate the random poison multiple times during the spam run, > there's nothing e4 (or any other hash system) can do to detect it. Hash the rendered (via links -dump) html ? -- "You may be recognized soon. Hide." http://www.ChaosReigns.com --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
At 12:11 PM 12/15/2003, Nels Lindquist wrote: John's correct. The huge increase in hitrate is observed *after* I report; Oh, well.. of course you're going to have a massive hitrate increase in that case. It looks like you get a lot of spam which is sufficiently unique to fool razor... Either that or spammers bayes poison methods are also hurting e4 (either intentionally or accidentally)... If they are doing random poison inserted throughout the entire message, and they re-generate the random poison multiple times during the spam run, there's nothing e4 (or any other hash system) can do to detect it. --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
At 11:21 PM 12/14/2003, Dale Britt wrote: This may sound really stupid, but I'm guessing the mail must stay on the server so this can be done, is this correct? Well, you need a copy of the message in RFC 822 format.. Or is it as simple as forwarding a spam email? (something is telling me no) No, you can't "forward" spam email to razor.. most mail clients mangle the bejezus out of the body when you forward mail, making it useless to razor. --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On 14 Dec 2003 at 20:14, John Andersen wrote: > On Sunday 14 December 2003 13:39, Matt Kettler wrote: > > Neils mentioned that he has a low hit rate when mail is delivered, but when > > he re-scans the SAME MESSAGES a few days later, the hit rate is three times > > higher. > > No, Neils said he had a 60% hit rate after RESCANNING mail he > submitted already. You would expect that, especially if Neils had > "good carma" at Razor. But the point was that Razor, left to its own > devices has a 15% hit rate. John's correct. The huge increase in hitrate is observed *after* I report; I generally do a razor-check immediately prior to reporting for comparison purposes. Generally speaking, there *is* a slight increase in razor-detected spam between initial reception and my pre- reporting razor-check, but it's nothing compared to what I see after I've reported. Even after a week, during which I might have expected *lots* of people to report, the pre- and post-reporting checks are dramatically different. It just seems odd to me that my individual reporting could make such a difference! I guess I don't really understand the algorithm by which an individual message's spam-threshold-according-to-razor is determined. I should probably consult some documentation and see if I can get a better handle on it. :-) One possibility that occurred to me is that a greater proportion of the spam being sent out nowadays is randomized for each recipient. I don't mean the content the spammer wants you to read--I'm talking about all the fake HTML tags and bayes poison sprinkled throughout many messages. If the messages *are* unique (or close to it) then presumably my reporting a message body which is unique to me (or say, ten recipients) won't actually have a big impact for other razor users. Nels Lindquist <*> Information Systems Manager Morningstar Air Express Inc. --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On Sunday 14 December 2003 13:39, Matt Kettler wrote: > Neils mentioned that he has a low hit rate when mail is delivered, but when > he re-scans the SAME MESSAGES a few days later, the hit rate is three times > higher. No, Neils said he had a 60% hit rate after RESCANNING mail he submitted already. You would expect that, especially if Neils had "good carma" at Razor. But the point was that Razor, left to its own devices has a 15% hit rate. You might blame it on timeing, because Neils said he only submits once a week or so, but the point remains that the OTHER spam engines DID detect it wher razor failed. -- _ John Andersen --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
Hi Matt Thank you for your reply If I'm understanding you correctly, I send spam to razor-report This may sound really stupid, but I'm guessing the mail must stay on the server so this can be done, is this correct? Or is it as simple as forwarding a spam email? (something is telling me no) Sorry for the dumb questions Dale - Original Message - From: "Matt Kettler" <[EMAIL PROTECTED]> To: "Dale Britt" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Sunday, December 14, 2003 2:41 PM Subject: Re: [Razor-users] Razor2 spam detection rate > At 12:53 PM 12/13/2003, Dale Britt wrote: > >I have email addresses that have been around over 6 years and receive a ton > >of unwanted emails which could help the Razor system. > > > >So if someone could explain to a layman, the exact procedure and process to > >report spam, I'm sure I can be a productive contributor to this community > >which benefits us all. > > First create a user ID (used to track your report source so that a > confidence can be assigned based on your reporting accuracy. ) > > razor-admin --register > > Then report messages: > > razor-report > > > > > > --- > This SF.net email is sponsored by: IBM Linux Tutorials. > Become an expert in LINUX or just sharpen your skills. Sign up for IBM's > Free Linux Tutorials. Learn everything from the bash shell to sys admin. > Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click > ___ > Razor-users mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/razor-users --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
At 12:53 PM 12/13/2003, Dale Britt wrote: I have email addresses that have been around over 6 years and receive a ton of unwanted emails which could help the Razor system. So if someone could explain to a layman, the exact procedure and process to report spam, I'm sure I can be a productive contributor to this community which benefits us all. First create a user ID (used to track your report source so that a confidence can be assigned based on your reporting accuracy. ) razor-admin --register Then report messages: razor-report --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
At 12:29 AM 12/13/2003, John Andersen wrote: But wait a minute here... His spam detection is already working, and he is routeing this spam to his spam folder. Its just NOT RAZOR that is detecting the stuff as spam. He said: Wait a minute.. Please re-read my post AND the post I was replying to, I think you massively misinterpreted me. I was not focusing on why razor has a low and/or high hitrate, or how to fix it. Neils mentioned that he has a low hit rate when mail is delivered, but when he re-scans the SAME MESSAGES a few days later, the hit rate is three times higher. I was strictly posting an explanation of WHY this jump in hitrate happens when you wait a while between runs of razor-check So feeding this already-automatically-detected-spam to razor is going to do NOTHING for his over-all spam detection, I was not attempting to tell him how to fix anything. I was not telling him to even DO anything.. I was telling him WHY he sees the behaviors that he's observing. --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
Hi One of the reasons I had Razor installed on our server was because of the community feeding spam to the Razor system that wasn't caught by other means. As an NON Technical person, I don't understand how to submit my spam to the Razor system. I have email addresses that have been around over 6 years and receive a ton of unwanted emails which could help the Razor system. So if someone could explain to a layman, the exact procedure and process to report spam, I'm sure I can be a productive contributor to this community which benefits us all. Thanks Dale - Original Message - From: "PieterB" <[EMAIL PROTECTED]> To: "Theo Van Dinter" <[EMAIL PROTECTED]> Cc: "Nels Lindquist" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Saturday, December 13, 2003 2:08 AM Subject: Re: [Razor-users] Razor2 spam detection rate > On Thu, Dec 11, 2003 at 07:47:12PM -0500, Theo Van Dinter wrote: > > oh, forgot to mention. with DCC I'm seeing ~76% hit rate, SpamCop is > > ~75%, and SpamAssassin is ~99.8%. :) > For what is worth... My recent hitrates are: > > RAZOR: ~31%, PYZOR: ~45%, BAYES_99 from SpamAsssasin: >92% > Both RAZOR and PYZOR: ~7% > > So most of my spam is catched by SpamAssassin (with BAYES enabled), > but the combining RAZOR, PYZOR increases the overall hitratio (the > reported spam for Razor and Pyzor seems quite disjunct). > > I'll give DCC and SpamBayes a try as well, later this month or next > year. > > Regards, > > Pieter > > -- > http://zwiki.org/PieterB > > > --- > This SF.net email is sponsored by: IBM Linux Tutorials. > Become an expert in LINUX or just sharpen your skills. Sign up for IBM's > Free Linux Tutorials. Learn everything from the bash shell to sys admin. > Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click > ___ > Razor-users mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/razor-users --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On Thu, Dec 11, 2003 at 07:47:12PM -0500, Theo Van Dinter wrote: > oh, forgot to mention. with DCC I'm seeing ~76% hit rate, SpamCop is > ~75%, and SpamAssassin is ~99.8%. :) For what is worth... My recent hitrates are: RAZOR: ~31%, PYZOR: ~45%, BAYES_99 from SpamAsssasin: >92% Both RAZOR and PYZOR: ~7% So most of my spam is catched by SpamAssassin (with BAYES enabled), but the combining RAZOR, PYZOR increases the overall hitratio (the reported spam for Razor and Pyzor seems quite disjunct). I'll give DCC and SpamBayes a try as well, later this month or next year. Regards, Pieter -- http://zwiki.org/PieterB --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On Friday 12 December 2003 22:09, Kelson Vibber wrote: > On Friday 12 December 2003 9:29 pm, John Andersen wrote: > > So feeding this already-automatically-detected-spam to razor is > > going to do NOTHING for his over-all spam detection, because all he is > > doing is training razor to recognize spam that has already been > > recognized as spam by some other Spamassassin tests. Its not going to > > help him at all. > > But it *is* going to help *other* people who are relying on Razor and don't > use SpamAssassin - or who have SA configured differently. Perhaps true, but that was not the focus of the Oringnal Posters question which had to do with the low detection rate of Razor. One has to ask why bother with razor if it must be fed by the more successful spam detection agents. Those agents will always be better than razor if razor relies on them for its reporting. The original intent of Razor was that someone would look at each piece of spam and report it thru razor if it was indeed spam. Vipul specifically discouraged automated means of detection. Yet I suspect automated reporting is now the norm. Is this low 15% detection rate perhaps caused by the automated reporting? -- _ John Andersen --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On Friday 12 December 2003 9:29 pm, John Andersen wrote: > So feeding this already-automatically-detected-spam to razor is > going to do NOTHING for his over-all spam detection, because all he is > doing is training razor to recognize spam that has already been recognized > as spam by some other Spamassassin tests. Its not going to help him > at all. But it *is* going to help *other* people who are relying on Razor and don't use SpamAssassin - or who have SA configured differently. -- Kelson Vibber SpeedGate Communications, www.speed.net --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On Thursday 11 December 2003 15:00, Matt Kettler wrote: > At 06:49 PM 12/11/2003, Nels Lindquist wrote: > >Could it be that my razor setup isn't optimal? Is there some > >explanation for what I'm seeing? > > what you're seeing is VERY easy to explain. > > Razor lists mail based on many people reporting it. > > It takes time before enough reports come in to list a message as spam. > > If you get the spam and razor-check it very early in this cycle, it may not > hit. Running your check much later, after all the reports are in, is very > likely to increase your hitrate. But wait a minute here... His spam detection is already working, and he is routeing this spam to his spam folder. Its just NOT RAZOR that is detecting the stuff as spam. He said: >I'm running SpamAssassin + Razor via a sendmail milter, so spam >detection is done at the earliest possible moment from the >perspective of our network. A copy of anything scoring over 5.0 is >saved on our mail server So feeding this already-automatically-detected-spam to razor is going to do NOTHING for his over-all spam detection, because all he is doing is training razor to recognize spam that has already been recognized as spam by some other Spamassassin tests. Its not going to help him at all. -- _ John Andersen --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
Re: [Razor-users] Razor2 spam detection rate
On Thu, Dec 11, 2003 at 07:45:39PM -0500, Theo Van Dinter wrote: > I used to get about that, but now am seeing around ~35% hit rate. :( oh, forgot to mention. with DCC I'm seeing ~76% hit rate, SpamCop is ~75%, and SpamAssassin is ~99.8%. :) -- Randomly Generated Tagline: "What you end up with, after running an operating system concept through these many marketing coffee filters, is something not unlike plain hot water." (By Matt Welsh) pgp0.pgp Description: PGP signature
Re: [Razor-users] Razor2 spam detection rate
On Thu, Dec 11, 2003 at 04:49:05PM -0700, Nels Lindquist wrote: > keep seeing mentions of detection rates on the order of 80% or > better, but I'm not seeing anything nearly that good. I used to get about that, but now am seeing around ~35% hit rate. :( -- Randomly Generated Tagline: I bet Einstein turned himself all sorts of colors before he invented the lightbulb. -- Homer Simpson Bart the Genius pgp0.pgp Description: PGP signature
Re: [Razor-users] Razor2 spam detection rate
At 06:49 PM 12/11/2003, Nels Lindquist wrote: Could it be that my razor setup isn't optimal? Is there some explanation for what I'm seeing? what you're seeing is VERY easy to explain. Razor lists mail based on many people reporting it. It takes time before enough reports come in to list a message as spam. If you get the spam and razor-check it very early in this cycle, it may not hit. Running your check much later, after all the reports are in, is very likely to increase your hitrate. --- This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click ___ Razor-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/razor-users
