From: "Matt Kettler" <[EMAIL PROTECTED]>
jdow wrote:
And it is scored LESS than BAYES_95 by default. That's a clear signal
that the theory behind the scoring system is a little skewed and needs
some rethinking.
No.. It does not mean there's a problem with the scoring system. It
means you're trying to apply a simple linear model to something which is
inherently not linear, nor simple. This is a VERY common misconception.
Please bear with me for a minute as I explain some things.
This is more-or-less the same misconception as expecting rules with
higher S/O's to always score higher than those with lower S/O's.
Generally this is true, but there's more to consider that can cause the
opposite to be true.
The score of a rule in SA is not a function of the performance of that
one rule, nor should it be. The score of a SA rule is a function of what
combinations of rules it matches in conjunction with. This creates a
"real world fit" of a complex set of rules against real-world behavior.
This complex interaction between rules results in most of the "problems"
people see. People inherently expect simple linearity. However, consider
that SA scoring is a function of several hundred variable equation
attempting to perform an approximation of optimal fit to a sampling of
human behavior. Why, based on that, would you ever expect the score two
of those hundreds of variables to be linear as a function of spam hit rate?
It is perfectly reasonable to assume that most of the mail matching
BAYES_99 also matches a large number of the stock spam rules that SA
comes with. These highly-obvious mails are the model after which most SA
rules are made in the first place. Thus, these mails need less score
boost, as they already have a lot of score from other rules in the ruleset.
However, mails matching BAYES_95 are more likely to be "trickier", and
are likely to match fewer other rules. These messages are more likely to
require an extra boost from BAYES_95's score than those which match
BAYES_99.
Matt, I understand the model. I believe it is the wrong model to apply.
Experience indicates this is very much the case. And I must remind you
that an ounce of actual experience is worth a neutron star worth of
theory. When I raise the score of BAYES_99 and 95 to be monotonically
increasing with 99 at or very near to 5.0 I demonstrably get far fewer
escaped spams at a cost of VERY few (low enough to be unnoticed)
caught hams. When experience disagrees with the model some extra thought
is required with regards to the model.
As far as I can see the perceptron does not handle single factors that
are exceptionally good at catching spam with exceptionally few false
alarms AND is often the ONLY marker for actual spam that is caught. This
latter is very often the case here with regards to BAYES_99. (The logged
hams caught as spam are escaped spams or else cases that are impossible
to catch correctly without complex meta rules, such as LKML or other
technical code, patch, and diff bearing mailing lists that also do not
adequately filter being relayed through. For these lists I have actually
had to artificially rescore all the BAYES scores using meta rules. I am
fine tuning these alterations at the moment. I've had some spams escape.
My OWN number of mismarked hams has become vanishingly small. Loren does
not have these rules yet. If he wants 'em I'll give them to him quickly.)
Note the "goodness" of BAYES_99 here - stats including me and Loren over
80,000 messages total.
1 BAYES_99 20156 4.88 25.08 91.61 0.07
1 BAYES_00 46107 15.54 57.36 0.07 78.98
The BAYES_99's *I* have seen on "ham" are running exclusively to spams
that managed to fire a negative scoring rule for mailing lists. LKML and
FreeBSD are the two lists so affected.
Now, in the last two days I have had some ham come in as spam, not due
to BAYES_9x at all. It was a political discussion that happened to
trigger a lot of the mortgage spam rules. "Cain't do much about that!"
(At least not without giving Yahoo Groups an utterly unwarranted negative
score.)
Based on *MY* experience the perceptron performance model was not the
appropriate model to choose.
{^_^}
