what we're trying to come up with is a way to have the GA (or somehow!) calculate a good SA score for the Razor rules. The way we calculate scores for the other rules is to feed a bunch of spam and nonspam messages through SA and see how often the rules are triggered. For time-invariant rules, this is pretty straightforward and works great. For Razor (and other net checks to a lesser extent) this doesn't work so good. We do have ages of the various messages we're checking, so I suppose we could do some kind of weighting to treat more recent mails as more significant for the purposes of getting input to the GA for Razor -- it would help us a bunch if we knew what the attack/decay curve looked like on razor reports though -- Do you have such data, and if so, can you share it? What I mean by attack/decay curves is basically something like
For attack: How many reports/time period happen on average for a particular signature, tracked over time. So eg for a typical message, starting the clock at time-of-first-report, how many reports in minute 1, minute 2, minute 3, etc (or whatever the timescale ought to be)
For decay: From time of first report, what is the average time to expiration. Actually, we really could better use a histogram, eg 12% of sigs expire after 90 days, 38% after 120 days, 75% after 150 days, 98% after 180 days, or something like that.
Or do you have any suggestions for a better way to handle this?
C
On Tuesday, January 14, 2003, at 08:22 AM, Jordan Ritter wrote:
When old signatures are deprecated, the system tends to recompensate
rather quickly for SPAM that is seen and reported in real time. In
other words, we really won't care about blocking SPAM that hasn't been
seen for a few months, because it's not a problem for our users.
However, in this case we're in the process of migrating to a new DB
model, so the circumstances are a little different. Given the size of
SpamNet, though, it doesn't take long at all for the body of knowledge
the network holds to reconstitute the signatures it is missing, by way
of what people are actually receiving (versus what you're storing for
safe-keeping).
If you'd like, go ahead and resubmit your spambox, but I think we'd
argue that the test below isn't useful for the purpose of gauging
accuracy, since SpamNet's design and model are fundamentally different
from SA (and any rules-based system really).
--jordan
On Tue, Jan 14, 2003 at 02:00:29PM +0000, Justin Mason wrote:
#
# Razor folks --
#
# Dan (below) just tested a load of mail from the last few months against
# many network tests, including Razor2. The number of hits against R2
# has just dropped dramatically.
#
# have you just done a big expiration of old hashes? What is the "typical"
# length of time a message's hashes remain in the dbs?
#
# --j.
#
# ------- Forwarded Message
#
# Date: 13 Jan 2003 23:06:53 -0800
# From: Daniel Quinlan <[EMAIL PROTECTED]>
# To: <[EMAIL PROTECTED]>
# Subject: [SAdev] Razor2 slacking off, broken, or what?
#
# a network mass-check started at Fri Jan 10 09:24:44 UTC 2003:
#
# OVERALL% SPAM% HAM% S/O RANK SCORE NAME
# 20084 8138 11946 0.405 0.00 0.00 (all messages)
# 100.000 40.5198 59.4802 0.405 0.00 0.00 (all messages as %)
# 9.893 24.2197 0.1339 0.995 0.95 3.91 RAZOR2_CHECK
#
# a network mass-check started at Tue Jan 14 04:16:48 UTC 2003:
#
# OVERALL% SPAM% HAM% S/O RANK SCORE NAME
# 20114 8140 11974 0.405 0.00 0.00 (all messages)
# 100.000 40.4693 59.5307 0.405 0.00 0.00 (all messages as %)
# 0.423 1.0319 0.0084 0.992 0.91 3.91 RAZOR2_CHECK
#
# Is RAZOR2 expiring message digests faster, did their database get
# whacked, or did something else change?
#
# The newest email in my corpus is 4 days old and I have spam going back
# to the end of March 2002, so it's fairly recent stuff.
#
# Thanks to the new date code in mass-check, I can tell you the average
# age of RAZOR2_CHECK spam hits on my Jan 10 run was about 133 days which
# is almost exactly the same as the average age of messages in my spam
# corpus -- 135 days old.
#
# However, the average age of RAZOR2_CHECK hits is now only 64 days, but
# what concerns me more is that the SPAM% is 24x lower now. If I look at
# the last 10% of my ham and spam, it's a bit better, but not great:
#
# OVERALL% SPAM% HAM% S/O RANK SCORE NAME
# 2011 814 1197 0.405 0.00 0.00 (all messages)
# 100.000 40.4774 59.5226 0.405 0.00 0.00 (all messages as %)
# 2.685 6.5111 0.0835 0.987 0.89 3.91 RAZOR2_CHECK
#
# or messages newer than 28 days:
#
# OVERALL% SPAM% HAM% S/O RANK SCORE NAME
# 544 426 118 0.783 0.00 0.00 (all messages)
# 100.000 78.3088 21.6912 0.783 0.00 0.00 (all messages as %)
# 9.007 11.2676 0.8475 0.930 0.74 3.91 RAZOR2_CHECK
#
# Note: the sample size (that's only 1 ham hit) is probably too small to
# pay any attention to the S/O ratio here.
#
# Daniel
#
# --
# Daniel Quinlan Linux, open source, and
# http://www.pathname.com/~quinlan/ anti-spam consulting
#
#
# -------------------------------------------------------
# This SF.NET email is sponsored by: FREE SSL Guide from Thawte
# are you planning your Web Server Security? Click here to get a FREE
# Thawte SSL guide and find the answers to all your SSL security issues.
# http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0026en
# _______________________________________________
# Spamassassin-devel mailing list
# [EMAIL PROTECTED]
# https://lists.sourceforge.net/lists/listinfo/spamassassin-devel
#
#
#
# ------- End of Forwarded Message
#
#
#
# -------------------------------------------------------
# This SF.NET email is sponsored by: FREE SSL Guide from Thawte
# are you planning your Web Server Security? Click here to get a FREE
# Thawte SSL guide and find the answers to all your SSL security issues.
# http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0026en
# _______________________________________________
# Razor-users mailing list
# [EMAIL PROTECTED]
# https://lists.sourceforge.net/lists/listinfo/razor-users
<mime-attachment>
-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving your online business a competitive advantage. Test-drive a Thawte SSL certificate - our easy online guide will show you how. Click here to get started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
Razor-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/razor-users
