Pete,
Our Hold range has returned to more normal territory on Thursday. Here's the stats from
<snip/>
One of my thoughts regarding minimum rule strengths and grace periods is that all groups aren't necessarily the same. For instance Nigerian scams are low volume and sporadic, and my system performs the worst on these things. Maybe lower rule strengths and longer grace periods makes much more sense for the Phishing category than it does for many other categories for instance. Is that possible?
These are definitely some things to look at - great food for new research projects.
There is a great diversity - luckily the scanning engine has a huge amount of headroom so most of the time we don't need to tune things very precisely. In any of the categories you mention we see some rules die immediately, and others seem to live on forever - often without a great deal of reason for either case.
The fact that your hold range returned after we adjusted the rule strength calculation window is a good indication that the relevant tuning parameter is minimum rule strength. I noted that the previous adjustment (changing the window from 45 to 35 days) happened precisely one month ago. This strongly suggested that we were seeing a "wave front" of sorts pass through the tuning system - so on a hunch I put it back to 45. Your report helps to support this conjecture.
The grace period value has the greatest effect early on in a rule's life cycle and probably shouldn't be extended beyond about 10 days. The design of the grace period feature is that it gives a new rule time for it's rule strength to rise to the minimum threshold. After that it's all about the performance of the rule. This sets up a competitive environment in the system. Reaching a threshold of 1.0 currently requires that at least 19 messages fail on that rule within the analysis window and on one of the systems that are providing logs for analysis. With about 110 logs being consistently reported there are plenty of chances for 19 hits to happen.
[ an "ordinary" reporting system processes about 1300 messages per hour with sniffer spending about 190ms of computing time per message (or about 7% of the available computing time). In 5 days a rule has about 17160000 opportunities to "kill" a message. To stay alive, a rule need only achieve a kill about .00011655% (one ten thousandth of a percent) of the time. Of course, these numbers are a lot like the average US family having 2.3 kids - ever seen .3 of a kid? --- but the scale of the numbers seems right. ]
It could be argued that if a rule can't account for at least that many hits across 110 systems in 5 days then it's not going to be missed... The counter to this argument is that the spammers are driving toward diversity to make filtering systems of all types difficult to train and maintain -- as you noted, half of the active rules in the default configuration are in this very low strength range.
I also looked up the rule strengths on your site and found that about 50%, or maybe more, have a strength below 1, and maybe lowering that is worth testing out so long as I don't massively increase the number of records. I do think though that I would like to test out extending the grace period. Most of my false positives are not on things that this would affect, and that might give niche sources a little extra coverage if I understand things correctly.
Possibly - but I think an adjustment in the minimum rule strength will probably suffice given the sensitivity at that range. For example, if you adjust your minimum rule strength to 0.8 then on 10 credited kills would be required over a period of 5 days on 110 systems in order to push the rule above the strength threshold. Thereafter it would remain in place for at least 45 days (with the current settings) --- each of those days providing another opportunity to increase or maintain it's strength...
There is also another mechanism at work here --- our core system scans every presumed ham message one more time with every rule in the system (min rule strength 0). The log from this scan is injected into the normal analysis so that if a message matching a deactivated rule reaches our system through any path the strength for that rule will be raised above 0.
The second stage of the reactivation process then kicks in because our system normally scans messages with a minimum rule strength of 0.1 - so any messages that were being missed will continue to rise in strength if they are seen in any volume in our spam traps or submitted spam.
Once we see 20 instances every system will begin using the reactivated rule... Some systems will begin even before that because they are using more sensitive settings in their rulebases - this fact helps to accelerate the process.
Anyway, a long story short - I think the first thing to try is adjusting the Minimum Rule Strength. This is by far the most sensitive setting - though the two do interact dynamically - especially if more reporting systems use more sensitive ruelbases. If there is reason to believe that a new rule might not meet this threshold within 5 days then you might consider increasing that value but I don't think I'd do it for any other reason. Just consider the implications.
If we add 300+ rules per day then 5 days guarantees more than 1500 additional rules ongoing - and every one of them would likely be a low performer. Throw in a few spam storms though and it could add up quickly. During the period just before our adjustments we experienced a few bad weeks where 5 days worth of new rules could account for 10-20% of the active ruleset - this was a lot of data!
Then again - since the scanning engine is very efficient (particularly with version 2-3) these kinds of numbers are "in the noise" --- with one exception: bandwidth. Adding rules causes the rulebase file to expand and when we multiply that out across all of our users each with several updates per day it can add up to a lot of bandwidth.
[ Current rulebase files typically carry 35K-40K active heuristics so 1.5K is small. ]
[ We are considering a new format for the rulebase files in version 3 which will be guaranteed to reduce the rulebase file sizes by 25%. We are also considering upgrades to the scanning engine that will improve it's efficiency by 30% or more (measured in operations per byte scanned). Finally in a later version of 2x we will be introducing IPDB which will offload many of the IP based rules in the system to a more space efficient format. ]
[ Delivery Bandwidth issues have been largely mitigated with the implementation of mod_gzip on the web server, an adjustment to some automated schedules that used to cause a large number of systems to converge at common times, and an automated pacing delivery system that spreads rulebase updates throughout the day and reduces noise in the queue. In addition we have secured additional hosting facilities with 100MB of burstable bandwidth available which we will be bringing on line as needed. ]
I'll follow your directions and contact you directly regarding any affirmative changes, but I thought it might be beneficial to keep this discussion public since some other stats hounds might find this information to be of use :)
All Good.
I recommend we try adjusting your rule strength threshold to 0.8 and leave the grace period as it is for now. Let me know if this is what you would like to do.
If you can glean anything from the numbers that I gave you, please add your thoughts.
They are interesting - but I can't reach any conclusions from them --- just that my intuition tells me something happened on Tuesday, and that the system appears to be "ringing" a bit... but there's not enough data to get past intuition ;-)
_M
