[sniffer] Re: Significant increase in false positives

[Matt]

There is no doubt that having Declude handle xhdr files would be optimal.  I might add that an option to exclude the header on non-hits would also be wise.  David Barker appears open to some feature requests of late, and I would think that you could make this happen.  Not everyone has capacity limitations, so the internal functionality would probably suit the needs of many of your users also, and cover all non-SA systems instead of just Declude. Regarding this rule, the binary segment is non-searchable.  My only solution would be to write some _vbscript_ that parsed the Sniffer log for hits and move the files from my CopyFile directory back into Declude's Proc.  I'm guessing that someone could also do some grepping for this, but that ain't a strength of mine.  I could do this in minutes though if I had headers to search on.  Thankfully this rule only hit about 1,000 times this weekend as a final match (I'm ignoring those that weren't final matches since those would have hit anyway).  My gateway gets rid of most image spams, so I would expect a comparably higher rate for others. Regarding false positives in general.  I don't expect Sniffer to be perfect due to the way that rules are generated, but I have two suggestions. 1) One would be to test all new rules on a small sub-set of E-mail that covers the most common patterns such as attachments and E-mail/webmail clients with various formats including forwards and replies.  This would likely stop the worst of the worst in terms of FP issues like the one earlier this year that was hitting on most base64 code.  I envision hundreds of test messages and not thousands, so this should be practical. 2) The second suggestion is one that I have mentioned many times before in private involving being able to tag messages on multiple types of hits for a stronger result.  The separation would need to be on the type rule so that all rule types would be isolated from one another.  For instance, phrase, pattern, IP and domain rules could be put in different codes and allowed to be scored in combination.  It would also be equally as important to treat rules from user submissions different from those generated from spam traps since these rules are not nearly as universal.  I currently average just under 3 matches per message that Sniffer hits, and I would imagine that there is a lot of mixing between these types.  This would allow many that are scoring Sniffer lower than our block weight to then score these multiple classification hits higher.  This wouldn't be useful though unless it was seperated by types like I listed since I often find multiple hits under the current rulebase format. Thanks, Matt Pete McNeil wrote: Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: > Pete, Would you please clarify this a bit.  Declude of course doesn't record the rule in the headers, so this is difficult to figure out.  Knowing the pattern may help identify the problematic messages.  Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name  image spam binary segment as text !1AQaq"2 Created  2006-10-14 Source  !1AQaq"2 Hidden  false Blocked  false Origin  Spam Trap Type  Simple Text Created By  [EMAIL PROTECTED] Owner  [EMAIL PROTECTED] Strength  3.20638481603822 False Reports  11 From Users  7 Rule belongs to following groups [252] Problematic I removed the rule as soon as we began receiving reports - about mid-day today. > I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own.  I believe the D* file may be editable when the external app is launched.  That would make recovery of this so much easier for me (minutes instead of hours of work). I have discussed this with Declude and I am hopeful that we will have better integration w/ Declude some time in the future. In the mean time, our next version will include a feature to inject headers into message files. Understand, however, that this is an expensive feature that will substantially increase the I/O requirements on any mail server. Injecting headers requires that the entire message file must be written to disk an additional time. This is not a small consideration-- Where once most spam were tiny text/html files (often less than 5K) today's image spam variants are frequently 5 to 10 times the size of the old spam we used to know. Also- note that this kind of thing can be very buggy on Winx systems -- sometimes changes to files are not reflected immediately between processes. For example, rename operations are not atomic - so when the old message file is deleted and the new version is renamed from it's temp file to the original message file name, other Winx processes that depend on that file may not respond reliably. For all of these reasons and more I've probably not thought of - this feature will be a "use at your own risk / YMMV" option. All that said, there is an existing option in the current version of SNF to produce a .xhdr file for each message. This option is frequently used in *nix systems that use SNF. It would be possible to write a short utility (perhaps even a script) that would modify quarantined messages out-of-band to include the contents of the .xhdr file as X- headers. Such a utility is not currently on our development list, however, and I hallucinate that such a device would tend to evolve into something somewhat system specific. The best option would be for Declude to add a feature that picks up x-headers created by external programs (perhaps in files named <message-file-name>.xhdr) so that they can be added in a single message rewrite along with the headers that Declude already adds. This would solve the I/O problems and standardize the mechanism for any other external programs that might wish to add headers. Hope this helps, _M --  Pete McNeil Chief Scientist, Arm Research Labs, LLC. ############################################################# This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>