On Monday, November 29, 2004, 3:43:03 PM, Steinar wrote: SR> Hi!
SR> I want to try a new rule strength with Sniffer. SR> Can I change this myself or is this done by SortMonster? I've responded to this off list. However, for the benefit of everyone... We will be offering expanded rulebases as part of a "pro" license at some point in the future at a higher price. However, in the short term we are offering the expanded rulebase on a case by case basis. If you would like to experiment with a more sensitive setting on your rulebase file, then please send a note to us at support@ and we will make the adjustment for you. Along the way please consider the following and remember the above ;-) The normal rule strength threshold is 1.0. This setting removes rules from the rulebase file once they become ineffective. The effectiveness of the rules is measured on a logarithmic scale as "Rule Strength". This number is derived from the relative number of messages that were tagged by a given rule over the previous 45 days. You can reference the current rule strength numbers on this page: <http://www.sortmonster.com/MessageSniffer/Performance/RuleStrengths.jsp> The combined number of messages captured by rules below a strength of 1.0 is currently 0.21% of the total number of messages tagged by sniffer as reported by systems that send us their log files. This is the point of diminishing returns. Below a rule strength number of 1.0, the number of rules included increases and the size of the rulebase file increase dramatically for each additional message that can be captured. Rulebases that have their rule strength threshold set at 0.1 (the most sensitive setting) may be as much as 5 times the size of a standard rulebase. This is a significant increase in the bandwidth required to transmit and process the rulebase file. The newest Persistent Instance technology (and the coming plugin DLL for MDaemon) largely mitigate the additional system resources needed at the client end since the underlying pattern matching engine in Message Sniffer is extremely efficient. However, the costs of bandwidth and resources to compile these rulebases increases significantly (thus the coming price increase for "pro" licenses). It's a good idea to keep a perspective on how the rule strength number is related to capturing messages. This relationship is constantly changing as the system learns and grows, however currently the following is true based on approximately 130 reported log files covering a "window" of about 45 days: A rule with a strength of 5.0 will trap 5,762,592 messages. A rule with a strength of ~4.5 will trap ~1,190,887 messages. A rule with a strength of ~4.0 will trap ~ 251,257 messages. A rule with a strength of ~3.5 will trap ~ 53,991 messages. A rule with a strength of ~3.0 will trap ~ 11,374 messages. A rule with a strength of ~2.5 will trap ~ 2,398 messages. A rule with a strength of ~2.0 will trap ~ 505 messages. A rule with a strength of ~1.5 will trap ~ 97 messages. A rule with a strength of ~1.0 will trap ~ 19 messages. A rule with a strength of ~0.5 will trap ~ 4 messages. At 0.1 (the most sensitive) we reach numbers down to 1 hit in 130 systems over the past 45 days. Most of the rules in this lower range (below 1.0) are in transition either into or out of active status. That is, the spammers have either abandoned the patterns... or they may have started them up again. If you think about it, when it only takes 19 hits to bring a rule up into the "normal" range it's not likely that a rule will stay there for long. However, spammers now frequently command many thousands of hijacked sysetms for delivering their content - so when they do have a way through the filtering system they can make quite an impact before the hole is closed. For this reason we are continuing to work on ways to speed up the response time and effectiveness of our system. There are a number of complex effects associated with how the Message Sniffer system is tuned. Due to recent changes in the techniques used by spammers I am going to re-calibrate the default rulebase settings over the next few weeks. In the mean time, if anyone wants to use a more sensitive rule strength setting then we will probably not go below 0.5 except in a few rare cases where systems see a lot of traffic. Systems which have very high traffic levels and the most sensitive rule strength settings will act has a kind of "advanced guard" for the rest of us by quickly increasing the rule strength on rules that are "becoming active". This will ensure that the other systems quickly see these rules become activated, without having all systems carry the burden of the entire active rulebase. This tuning is complex... and everyone is busy, so I've tried not to get too technical in this note, but I do hope that I've shed some light on the subject. It has become quite popular lately ;-) Thanks to everyone for all of your help. Best, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
