On Friday, August 6, 2004, 11:37:13 AM, Scott wrote:
SF> I'll hijack this discussion from the Imail forum into the sniffer forum...
SF> -Mad,
SF> I enjoyed the discussion about putting some math into the weighting of spam tests.
SF> I've run my July numbers through your ((SS-SH)/(SS+SH)) formula.
SF> I think the formula works well in the very effective and the
SF> ineffective tests, but it's the moderately effective tests (maybe
SF> 5 to 15% false positives) that weighted too highly in my thoughts.
SF> You example of FIVETEN-SRC, (SS=20621, SH=3305) coming in
SF> with 72% of the weight value is a high for my weighting. Call me
SF> conservative, but I couldn't weight at 72 (100 points in my
SF> testing world) a test that was false 14% of the time.
If I'm reading you right, you may have missed a step. The original
theory was that you would set a constraint that at least 2 tests are
required to reach your hold weight.
If your hold weight is 100, then you would divide that by 2 before
applying the accuracy estimates to this target. That is, your Wct
(Constrained Target Weight) would be 50.
In that case FIVETEN-SRC would be weighted to 72% of your constrained
target (50) so you would get: 36.
After doing this you would adjust your hold weight for the appropriate
sensitivity on your system. This is a linear model.
--- That's one way to slice it.
SF> I've got plenty of tests that I wrangle over the weights:
SF> Declude's badheaders, spamheader, AHBL-domains, lots of the RFCI tests.
SF> Do you have any more formulas in the mad scientist lair that are more conservative?
One other thing you can do with the linear method is to adjust the
constraint. Set your Wct (see above) to a lower value. Think about Wct
this way:
N = Number of "perfect" tests required to reach the threshold.
Wt = Threshold weight (lets say 100).
Wct = Wt/N.
So if you require 2.5 perfect tests then you end up with:
Wct = 100/2.5
= 40
So, your .72 test resolves this way:
W = A * Wct
= .72 * 40
= 28.8
~ 29
--- Now, this is more conservative, but you will get more false
negatives this way too because the model is linear - so more test must
fire to reach the threshold. On many systems this just won't cut it.
Here's another that truly punishes the poorer tests. Using the theory
that your hold weight be reached by any single test that is 100%
accurate:
Let A = the test's estimated accuracy.
Let W = the weight of a test.
Let Wt = your hold threshold (let's say 100).
For each test, W = A^2*Ht
In this case, a test with an estimated accuracy of .72 would result in
a weight of:
W = (.72)^2 * 100
= (.72)*(.72)*100
= .5184 * 100
= 51.84
~ So we call it 52.
This is a nonlinear model based on "the square of the distance to
target". It can be tuned by adjusting the exponent (in this case 2).
If you want to further separate poor tests from good tests then you
adjust the exponent upward. For example what about 3.
W = (.72)^3 * 100
= (.72)(.72)(.72) * 100
= .373248 * 100
= 37.3
~ call it 37.
* You don't have to use integers for exponents but the math is easier
to explain that way...
* The variable names I'm using up here are made up on the spot so
don't hold me to them, I might not remember ;-)
* Yes, there are more ways to define test weights, and even more ways
to combine weights (though that's not supported in Declude)... I
haven't explored them all. These models should give you lots of ideas
- and remember that you can always tweak them once you get the model
that works for you.
* Picking the right model probably depends on a lot of factors - your
tolerances for FP and FN, the tests you have selected, other
configuration policies, etc... I'm particularly fond of the nonlinear
method above - but the proof will be out there with you folks. What
works in the lab and what works in the real world are often very
different.
* The models above are only basic models. The real equations are more
complex. For example, the single test weight probably includes a local
"trust" factor (based on your experiences) and a linear bias. So we
end up with something more like:
g = "Gain, or amplification"
T = Trust
B = Bias
W = (A^g * Wt * T) + B
--- Then there are also some mixed models that might be used, but when
it comes down to it, we're probably splitting hairs to get that
complex here.
Hope this helps,
_M
This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html
