Re: [sniffer] MDLP Tests

2005-04-02 Thread Pete McNeil
On Saturday, April 2, 2005, 4:09:31 PM, Jay wrote:

JSHNL Hello - 
 
JSHNL I am reviewing your MDLP report at
JSHNL http://www.sortmonster.com/MDLP/MDLP-Example-Long.html, and find some
JSHNL tests that are seemingly quite effective that I'm not familiar with.  If
JSHNL anyone has any informaiton about these tests, please let me know:

JSHNL - FABEL (is this the same as FABELSOURCES at
JSHNL http://www.declude.com/Articles.asp?ID=97Redirected=Y?)

FABEL   ip4rspamsources.fabel.dk127.0.0.2

JSHNL - MXRATE-*

MXRATE-BLACKip4rpub.mxrate.net  127.0.0.2
MXRATE-WHITEip4rpub.mxrate.net  127.0.0.3
MXRATE-SUSP ip4rpub.mxrate.net  127.0.0.4

JSHNL - UCEPROTEC*

UCEPROTECRDOip4rdnsbl-1.uceprotect.net  127.0.0.2
UCEPROTECCMUL   ip4rdnsbl-2.uceprotect.net  127.0.0.2
UCEPROTECCVIR   ip4rdnsbl-3.uceprotect.net  127.0.0.2

JSHNL Also, perhaps I am misunderstanding the data, but SNIFFER has a SQ of
JSHNL .802 - isn't that relatively bad ?

Actually, that's the hyper-accuracy penalty at work. I wrote a bunch
about that on the MDLP page. What's going on is that SNF frequently
catches spam that virtually no other tests are catching yet and as a
result the total weight never reaches the threshold. Every one of
those events shows up counting against it.

We research these periodically (we used to look at them constantly)
and with very rare exceptions we find that these are not false
positives.

In fact, on our systems last year SNF had fewer than 10 FP. (several
of those were messages from customers that actually contained examples
of spam, malware, or logs with spammy URI). Of course, our numbers are
a more than bit skewed because the vast majority of traffic on our
system is spam... so we can't use that to calculate a false positive
rate that has any real meaning.

The closest we can really get to an indication of false positive rates
from SNF is to point at our FP rate page:

http://www.sortmonster.com/MessageSniffer/Performance/FalseReportsRates.jsp

This page shows counts of all false positives reported to us on a
daily basis for all of our customers. At least two of these systems
are service providers with 10 or more licenses which submit false
positives automatically as they are reported from their customers.

So anyway, the short answer is that the SA and SQ values on the
SNIFFER tests are skewed by the hyper-accuracy penalty inherent in how
MDLP develops these scores. The true accuracy values are very much
higher and this is regularly confirmed by both hard reviews of the
data and anecdotal evidence from our customers.

Hope this helps,

_M




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


RE: [sniffer] MDLP Tests

2005-04-02 Thread Jay Sudowski - Handy Networks LLC
Ahh, that makes more sense now.  ham is just what does not pass the
spam threshold. In this light, if Sniffer is hyper accurate and
catches more real spam than all others, it will appear less accurate
overall because of the deficienes in the other tests.  For some reason,
I was thinking that ham was being calculated differently.

Thanks for the tests, as well.

-Jay

PS - I did read your stuff about hyper-accuracy, but everything wasn't
meshing for me, hence my question :)

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil
Sent: Saturday, April 02, 2005 4:43 PM
To: Jay Sudowski - Handy Networks LLC
Subject: Re: [sniffer] MDLP Tests

On Saturday, April 2, 2005, 4:09:31 PM, Jay wrote:

JSHNL Hello -
 
JSHNL I am reviewing your MDLP report at 
JSHNL http://www.sortmonster.com/MDLP/MDLP-Example-Long.html, and find 
JSHNL some tests that are seemingly quite effective that I'm not 
JSHNL familiar with.  If anyone has any informaiton about these tests,
please let me know:

JSHNL - FABEL (is this the same as FABELSOURCES at
JSHNL http://www.declude.com/Articles.asp?ID=97Redirected=Y?)

FABEL   ip4rspamsources.fabel.dk127.0.0.2

JSHNL - MXRATE-*

MXRATE-BLACKip4rpub.mxrate.net  127.0.0.2
MXRATE-WHITEip4rpub.mxrate.net  127.0.0.3
MXRATE-SUSP ip4rpub.mxrate.net  127.0.0.4

JSHNL - UCEPROTEC*

UCEPROTECRDOip4rdnsbl-1.uceprotect.net  127.0.0.2
UCEPROTECCMUL   ip4rdnsbl-2.uceprotect.net  127.0.0.2
UCEPROTECCVIR   ip4rdnsbl-3.uceprotect.net  127.0.0.2

JSHNL Also, perhaps I am misunderstanding the data, but SNIFFER has a 
JSHNL SQ of
JSHNL .802 - isn't that relatively bad ?

Actually, that's the hyper-accuracy penalty at work. I wrote a bunch
about that on the MDLP page. What's going on is that SNF frequently
catches spam that virtually no other tests are catching yet and as a
result the total weight never reaches the threshold. Every one of those
events shows up counting against it.

We research these periodically (we used to look at them constantly) and
with very rare exceptions we find that these are not false positives.

In fact, on our systems last year SNF had fewer than 10 FP. (several of
those were messages from customers that actually contained examples of
spam, malware, or logs with spammy URI). Of course, our numbers are a
more than bit skewed because the vast majority of traffic on our system
is spam... so we can't use that to calculate a false positive rate
that has any real meaning.

The closest we can really get to an indication of false positive rates
from SNF is to point at our FP rate page:

http://www.sortmonster.com/MessageSniffer/Performance/FalseReportsRates.
jsp

This page shows counts of all false positives reported to us on a daily
basis for all of our customers. At least two of these systems are
service providers with 10 or more licenses which submit false positives
automatically as they are reported from their customers.

So anyway, the short answer is that the SA and SQ values on the SNIFFER
tests are skewed by the hyper-accuracy penalty inherent in how MDLP
develops these scores. The true accuracy values are very much higher and
this is regularly confirmed by both hard reviews of the data and
anecdotal evidence from our customers.

Hope this helps,

_M




This E-Mail came from the Message Sniffer mailing list. For information
and (un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


RE: [sniffer] MDLP Tests

2005-04-02 Thread Colbeck, Andrew
Jay, here's more web information on the mxrate tests:

http://www.mxrate.com/lookup/dns.htm


Andrew 8)

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil
Sent: Saturday, April 02, 2005 1:43 PM
To: Jay Sudowski - Handy Networks LLC
Subject: Re: [sniffer] MDLP Tests


On Saturday, April 2, 2005, 4:09:31 PM, Jay wrote:

JSHNL Hello -
 
JSHNL I am reviewing your MDLP report at 
JSHNL http://www.sortmonster.com/MDLP/MDLP-Example-Long.html, and find 
JSHNL some tests that are seemingly quite effective that I'm not 
JSHNL familiar with.  If anyone has any informaiton about these tests, 
JSHNL please let me know:

JSHNL - FABEL (is this the same as FABELSOURCES at
JSHNL http://www.declude.com/Articles.asp?ID=97Redirected=Y?)

FABEL   ip4rspamsources.fabel.dk127.0.0.2

JSHNL - MXRATE-*

MXRATE-BLACKip4rpub.mxrate.net  127.0.0.2
MXRATE-WHITEip4rpub.mxrate.net  127.0.0.3
MXRATE-SUSP ip4rpub.mxrate.net  127.0.0.4

JSHNL - UCEPROTEC*

UCEPROTECRDOip4rdnsbl-1.uceprotect.net  127.0.0.2
UCEPROTECCMUL   ip4rdnsbl-2.uceprotect.net  127.0.0.2
UCEPROTECCVIR   ip4rdnsbl-3.uceprotect.net  127.0.0.2

JSHNL Also, perhaps I am misunderstanding the data, but SNIFFER has a 
JSHNL SQ of .802 - isn't that relatively bad ?

Actually, that's the hyper-accuracy penalty at work. I wrote a bunch
about that on the MDLP page. What's going on is that SNF frequently
catches spam that virtually no other tests are catching yet and as a
result the total weight never reaches the threshold. Every one of those
events shows up counting against it.

We research these periodically (we used to look at them constantly) and
with very rare exceptions we find that these are not false positives.

In fact, on our systems last year SNF had fewer than 10 FP. (several of
those were messages from customers that actually contained examples of
spam, malware, or logs with spammy URI). Of course, our numbers are a
more than bit skewed because the vast majority of traffic on our system
is spam... so we can't use that to calculate a false positive rate
that has any real meaning.

The closest we can really get to an indication of false positive rates
from SNF is to point at our FP rate page:

http://www.sortmonster.com/MessageSniffer/Performance/FalseReportsRates.
jsp

This page shows counts of all false positives reported to us on a daily
basis for all of our customers. At least two of these systems are
service providers with 10 or more licenses which submit false positives
automatically as they are reported from their customers.

So anyway, the short answer is that the SA and SQ values on the SNIFFER
tests are skewed by the hyper-accuracy penalty inherent in how MDLP
develops these scores. The true accuracy values are very much higher and
this is regularly confirmed by both hard reviews of the data and
anecdotal evidence from our customers.

Hope this helps,

_M




This E-Mail came from the Message Sniffer mailing list. For information
and (un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html

This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html