[sniffer] Re: Significant increase in false positives
That's where the rule was, I believe. It started over the weekend, and Pete removed the rule morning or mid-day yesterday. Darin. - Original Message - From: "Greg Evanitsky" <[EMAIL PROTECTED]> To: "Message Sniffer Community" Sent: Tuesday, October 17, 2006 9:35 AM Subject: [sniffer] Re: Significant increase in false positives On Oct 16, 2006, at 5:17 PM, Darin Cox wrote: > Anyone else seeing a sudden increase in FPs? We normally report a > few each day, but we're seeing a 10x increase in FPs for the past > three days. What particular group, if any, are you seeing them in? The experimental-abstract (61) category is my main fp problem lately. Curious, Greg # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
On Oct 16, 2006, at 5:17 PM, Darin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. What particular group, if any, are you seeing them in? The experimental-abstract (61) category is my main fp problem lately. Curious, Greg # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Dear Pete, Thank you for your reply. I can confirm that based on our initial review, there were few, (if any) false positives as a result of the bad rule. It's great to have your support as well as that of the Snifffer community. Sincerely, Michael SteinComputer House - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Tuesday, October 17, 2006 8:42 AM Subject: [sniffer] Re: Significant increase in false positives Hello Computer, Monday, October 16, 2006, 11:09:03 PM, you wrote: > Dear Pete, Sniffer blocked 35,000 messages today, and roughly 7200 of them were blocked by the 1174356 rule. Do you think many of these were false positives? Do you know a way of searching through 35,000 Imail messages to find the FP's ? What would you suggest in this situation. This was not a bad-rule alert or rule-panic situation. Most of these messages were probably NOT false positives. The rule does have a higher rate than is acceptable (so it was dropped), but it doesn't catch every message with an image, and it does catch primarily image spam. If I felt strongly about researching this there would be 7200 to look through (not 35000) and I would probably only look through those that failed no other tests or were below some very low weight threshold otherwise - that would probably bring the number down into a range < 100 messages (based on what I've seen reported). [ Educated guess items: > 80% of content is usually spam. On weekends this number is higher. This weekend there were some new, aggressive image spam campaigns - so the number of spam captured by a rule like this would be higher than normal rather than lower. The rule was essentially in place only during the weekend and only received FP reports late Sun through early Mon and some systems have reported no discernable increase in false positives during this period. 20% of 7200 is close to 150, so the conservative number likely not to be spam in that group is less than that (due to the weekend) so approximately 100 seems reasonable. If there are FPs then it is likely they failed no other tests. ] Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Hello Darin, Monday, October 16, 2006, 11:19:32 PM, you wrote: > Hi Pete, Can you clarify what this .xhdr option is and how we can enable it? I don't remember anything in the documentation that describes it. I think there were references to the config file previously, but there was never anything about it in mine. If you could give an example of how to enable and use the info it would be greatly appreciated. In your snf .cfg file, un-comment the .xhdr option line: X-Headers # XHeader File Output - When set to On the engine will create a new file with # each message scanned with the name scanfilename.xhdr that contains x-header # information that should be added to the message. XHeaderData: X-SortMonster-MessageSniffer-Rules XHeaderFinal: X-SortMonster-MessageSniffer-Result A quick, ok-but-not-very-correct way to add these headers to the message is simply to pre-pend them (copy msgfile.xhdr + msgfile to newmsgfile, delete msgfile, rename newmsgfile msgfile). Technically, X-headers can go anywhere in the header section. By convention they go just before the body. A utility would seek out the first empty line in the message (\r\n\r\n) and emit the contents of the .xhdr file there. Oddly enough we occasionally get support questions about "what are all these .xhdr files building up in my spool directory" when folks accidentally turn on this option without knowing what it will do ;-) Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Hi Pete, You're exactly right, but we often get spoiled by the high quality of your detection rate. It's easy to expect perfection when it means less work for us . Thanks for all you do to keep the quality so high. Darin. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Tuesday, October 17, 2006 8:42 AM Subject: [sniffer] Re: Significant increase in false positives Hello Computer, Monday, October 16, 2006, 11:09:03 PM, you wrote: > Dear Pete, Sniffer blocked 35,000 messages today, and roughly 7200 of them were blocked by the 1174356 rule. Do you think many of these were false positives? Do you know a way of searching through 35,000 Imail messages to find the FP's ? What would you suggest in this situation. This was not a bad-rule alert or rule-panic situation. Most of these messages were probably NOT false positives. The rule does have a higher rate than is acceptable (so it was dropped), but it doesn't catch every message with an image, and it does catch primarily image spam. If I felt strongly about researching this there would be 7200 to look through (not 35000) and I would probably only look through those that failed no other tests or were below some very low weight threshold otherwise - that would probably bring the number down into a range < 100 messages (based on what I've seen reported). [ Educated guess items: > 80% of content is usually spam. On weekends this number is higher. This weekend there were some new, aggressive image spam campaigns - so the number of spam captured by a rule like this would be higher than normal rather than lower. The rule was essentially in place only during the weekend and only received FP reports late Sun through early Mon and some systems have reported no discernable increase in false positives during this period. 20% of 7200 is close to 150, so the conservative number likely not to be spam in that group is less than that (due to the weekend) so approximately 100 seems reasonable. If there are FPs then it is likely they failed no other tests. ] Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Hello Computer, Monday, October 16, 2006, 11:09:03 PM, you wrote: > Dear Pete, Sniffer blocked 35,000 messages today, and roughly 7200 of them were blocked by the 1174356 rule. Do you think many of these were false positives? Do you know a way of searching through 35,000 Imail messages to find the FP's ? What would you suggest in this situation. This was not a bad-rule alert or rule-panic situation. Most of these messages were probably NOT false positives. The rule does have a higher rate than is acceptable (so it was dropped), but it doesn't catch every message with an image, and it does catch primarily image spam. If I felt strongly about researching this there would be 7200 to look through (not 35000) and I would probably only look through those that failed no other tests or were below some very low weight threshold otherwise - that would probably bring the number down into a range < 100 messages (based on what I've seen reported). [ Educated guess items: > 80% of content is usually spam. On weekends this number is higher. This weekend there were some new, aggressive image spam campaigns - so the number of spam captured by a rule like this would be higher than normal rather than lower. The rule was essentially in place only during the weekend and only received FP reports late Sun through early Mon and some systems have reported no discernable increase in false positives during this period. 20% of 7200 is close to 150, so the conservative number likely not to be spam in that group is less than that (due to the weekend) so approximately 100 seems reasonable. If there are FPs then it is likely they failed no other tests. ] Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
I'm attaching an old message to this list which may come in
handy. It's from my perspective, which is using Declude and IMail, with
the spam messages in d:\imail\spool\spam and needing to be moved to
d:\imail\spool to be re-scanned. Now that I use a newer version of
Declude, my paths are d:\imail\spool\spam for the source and
d:\imail\spool\proc for the destination.
Replace "828931" with "1174356" in the gawk
line.
Replace the date embedded in the sniffer log file name
wildcard with today's date. I went through the 15th, 16th and 17th to be
safe.
If you're archiving your logs, you'll of course have
to unpack them first. And if you don't rotate your logs often,
you may not need the wildcard on the log filename at
all.
I think I had 267 hits in my msgids.txt
file.
Andrew 8)
From: Message Sniffer Community
[mailto:[EMAIL PROTECTED] On Behalf Of Computer House
SupportSent: Monday, October 16, 2006 8:09 PMTo: Message
Sniffer CommunitySubject: [sniffer] Re: Significant increase in
false positives
Dear Pete,
Sniffer blocked 35,000 messages today, and
roughly 7200 of them were blocked by the 1174356 rule.
Do you think many of these were false
positives? Do you know a way of searching through 35,000
Imail messages to find the FP's ?
What would you suggest in this
situation.
Thank you,
Michael SteinComputer House
- Original Message -
From:
Pete McNeil
To: Message Sniffer Community
Sent: Monday, October 16, 2006 8:46
PM
Subject: [sniffer] Re: Significant
increase in false positives
Hello Darin,
Monday, October 16, 2006, 5:17:26 PM, you wrote:
>
Anyone else seeing a sudden increase in FPs?
We normally report a few each day, but we're seeing a 10x
increase in FPs for the past three
days.
Not sure if this is it, but there was an image segment rule that went in
over the weekend and resulted in an unusual number of false positives today.
The rule was removed. IIRC the rule id was: 1174356
Hope this helps,
_M
--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.
#
This message is sent to you because you are subscribed to
the mailing list .
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
--- Begin Message ---
Goran, this is pretty much what I did to get to
re-queuing:gawk "$0 ~ /Final\t828931/ {print
substr($3,2,16)}" gxamq2kt.log.20060207* >msgids.txtThe
file msgids.txt will now contain just the GUID part of the D[guid].SMD from
column 3 in the tab delimited Message Sniffer log files.I then used a
batch file I had previously created called qm.cmd (for queue and move).
Note that the folders I specify are for Declude 1.x, which has an overflow
folder. I use the overflow folder so that Declude will re-analyze the
message:Rem this is the qm.cmd file
listingmove d:\imail\spool\spam\d%1.smd u:\imail\spool\ >nulmove
d:\imail\spool\spam\q%1.smd u:\imail\spool\overflow\ >nulI
then issued from the command line:for /F %i in
(msgids.txt) do @qm.cmd %iThat takes of re-queuing all the held
messages. I am using a move instead of a copy because I want Declude to be
able to move a message it deems spam to the spam folder. If I used a copy,
it would fail to do the move because the file is already in the spam folder, and
Declude would then pass control back to Imail, which would then deliver the spam
inbound.After my queue went back to normal, I then set to work on my
dec0207.log file to determine if the entirety of the message was spam or ham
based on whether it was held or not (which is the simple scenario I
have).I hope that helps,Andrew 8)
p.s. Another re-posting in HTML so as to
preserve the line breaks. Sorry for the duplication,
folks.
> -Original
Message-> From: [EMAIL PROTECTED]> [mailto:[EMAIL PROTECTED]] On Behalf
Of Goran Jovanovic> Sent: Tuesday, February 07, 2006 5:39 PM> To:
sniffer@SortMonster.com> Subject: RE: Re[4]: [sniffer] Bad Rule -
828931>> I just ran the grep command on my log and I got 850
hits.>> Now is there a way to take the output of the grep command
and> use it pull out the total weight of corresponding message>
from the declude log file, or maybe the subject?>> Goran
Jovanovic> Omega Network Solutions>> >>
> -Original Message-> > From:
[EMAIL PROTECTED]> [mailto:[EMAIL PROTECTED]]>
> On Beha
[sniffer] Re: Significant increase in false positives
There is no doubt that having Declude handle xhdr files would be optimal. I might add that an option to exclude the header on non-hits would also be wise. David Barker appears open to some feature requests of late, and I would think that you could make this happen. Not everyone has capacity limitations, so the internal functionality would probably suit the needs of many of your users also, and cover all non-SA systems instead of just Declude. Regarding this rule, the binary segment is non-searchable. My only solution would be to write some _vbscript_ that parsed the Sniffer log for hits and move the files from my CopyFile directory back into Declude's Proc. I'm guessing that someone could also do some grepping for this, but that ain't a strength of mine. I could do this in minutes though if I had headers to search on. Thankfully this rule only hit about 1,000 times this weekend as a final match (I'm ignoring those that weren't final matches since those would have hit anyway). My gateway gets rid of most image spams, so I would expect a comparably higher rate for others. Regarding false positives in general. I don't expect Sniffer to be perfect due to the way that rules are generated, but I have two suggestions. 1) One would be to test all new rules on a small sub-set of E-mail that covers the most common patterns such as attachments and E-mail/webmail clients with various formats including forwards and replies. This would likely stop the worst of the worst in terms of FP issues like the one earlier this year that was hitting on most base64 code. I envision hundreds of test messages and not thousands, so this should be practical. 2) The second suggestion is one that I have mentioned many times before in private involving being able to tag messages on multiple types of hits for a stronger result. The separation would need to be on the type rule so that all rule types would be isolated from one another. For instance, phrase, pattern, IP and domain rules could be put in different codes and allowed to be scored in combination. It would also be equally as important to treat rules from user submissions different from those generated from spam traps since these rules are not nearly as universal. I currently average just under 3 matches per message that Sniffer hits, and I would imagine that there is a lot of mixing between these types. This would allow many that are scoring Sniffer lower than our block weight to then score these multiple classification hits higher. This wouldn't be useful though unless it was seperated by types like I listed since I often find multiple hits under the current rulebase format. Thanks, Matt Pete McNeil wrote: Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: > Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name image spam binary segment as text !1AQaq"2 Created 2006-10-14 Source !1AQaq"2 Hidden false Blocked false Origin Spam Trap Type Simple Text Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength 3.20638481603822 False Reports 11 Fr
[sniffer] Re: Significant increase in false positives
Hi Pete, Can you clarify what this .xhdr option is and how we can enable it? I don't remember anything in the documentation that describes it. I think there were references to the config file previously, but there was never anything about it in mine. If you could give an example of how to enable and use the info it would be greatly appreciated. Darin. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Monday, October 16, 2006 11:13 PM Subject: [sniffer] Re: Significant increase in false positives Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: > Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name image spam binary segment as text !1AQaq"2 Created 2006-10-14 Source !1AQaq"2 Hidden false Blocked false Origin Spam Trap Type Simple Text Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength 3.20638481603822 False Reports 11 From Users 7 Rule belongs to following groups [252] Problematic I removed the rule as soon as we began receiving reports - about mid-day today. > I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work). I have discussed this with Declude and I am hopeful that we will have better integration w/ Declude some time in the future. In the mean time, our next version will include a feature to inject headers into message files. Understand, however, that this is an expensive feature that will substantially increase the I/O requirements on any mail server. Injecting headers requires that the entire message file must be written to disk an additional time. This is not a small consideration-- Where once most spam were tiny text/html files (often less than 5K) today's image spam variants are frequently 5 to 10 times the size of the old spam we used to know. Also- note that this kind of thing can be very buggy on Winx systems -- sometimes changes to files are not reflected immediately between processes. For example, rename operations are not atomic - so when the old message file is deleted and the new version is renamed from it's temp file to the original message file name, other Winx processes that depend on that file may not respond reliably. For all of these reasons and more I've probably not thought of - this feature will be a "use at your own risk / YMMV" option. All that said, there is an existing option in the current version of SNF to produce a .xhdr file for each message. This option is frequently used in *nix systems that use SNF. It would be possible to write a short utility (perhaps even a script) that would modify quarantined messages out-of-band to include the contents of the .xhdr file as X- headers. Such a utility is not currently on our development list, however, and I hallucinate that such a device would tend to evolve into something somewhat system specific. The best option would be for Declude to add a feature that picks up x-headers created by external programs (perhaps in files named .xhdr) so that they can be added in a single message rewrite along with the headers that Declude already adds. This would solve the I/O problems and standardize the mechanism for any other external programs that might wish to add headers. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscrib
[sniffer] Re: Significant increase in false positives
Hi Matt, I know Pete has requested this in the past, but Declude hasn't been willing to make the change necessary for this to make it in the headers. But I totally agree with you, I'd love to see this in the headers so tracking down the rule isn't such a pain. Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Monday, October 16, 2006 10:03 PM Subject: [sniffer] Re: Significant increase in false positives Pete,Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help.I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work).Thanks,MattPete McNeil wrote: Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: > Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC.# This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: > Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name image spam binary segment as text !1AQaq"2 Created 2006-10-14 Source !1AQaq"2 Hidden false Blocked false Origin Spam Trap Type Simple Text Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength 3.20638481603822 False Reports 11 From Users 7 Rule belongs to following groups [252] Problematic I removed the rule as soon as we began receiving reports - about mid-day today. > I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work). I have discussed this with Declude and I am hopeful that we will have better integration w/ Declude some time in the future. In the mean time, our next version will include a feature to inject headers into message files. Understand, however, that this is an expensive feature that will substantially increase the I/O requirements on any mail server. Injecting headers requires that the entire message file must be written to disk an additional time. This is not a small consideration-- Where once most spam were tiny text/html files (often less than 5K) today's image spam variants are frequently 5 to 10 times the size of the old spam we used to know. Also- note that this kind of thing can be very buggy on Winx systems -- sometimes changes to files are not reflected immediately between processes. For example, rename operations are not atomic - so when the old message file is deleted and the new version is renamed from it's temp file to the original message file name, other Winx processes that depend on that file may not respond reliably. For all of these reasons and more I've probably not thought of - this feature will be a "use at your own risk / YMMV" option. All that said, there is an existing option in the current version of SNF to produce a .xhdr file for each message. This option is frequently used in *nix systems that use SNF. It would be possible to write a short utility (perhaps even a script) that would modify quarantined messages out-of-band to include the contents of the .xhdr file as X- headers. Such a utility is not currently on our development list, however, and I hallucinate that such a device would tend to evolve into something somewhat system specific. The best option would be for Declude to add a feature that picks up x-headers created by external programs (perhaps in files named .xhdr) so that they can be added in a single message rewrite along with the headers that Declude already adds. This would solve the I/O problems and standardize the mechanism for any other external programs that might wish to add headers. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Dear Pete, Sniffer blocked 35,000 messages today, and roughly 7200 of them were blocked by the 1174356 rule. Do you think many of these were false positives? Do you know a way of searching through 35,000 Imail messages to find the FP's ? What would you suggest in this situation. Thank you, Michael SteinComputer House - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Monday, October 16, 2006 8:46 PM Subject: [sniffer] Re: Significant increase in false positives Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: > Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work). Thanks, Matt Pete McNeil wrote: Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: > Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Hi Pete, I haven't looked at the Sniffer logs, as cross referencing from the Declude logs is a bit of a pain, but many of the FPs did have images, so that probably accounts for most of them if it was an Experimental rule. Darin. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Monday, October 16, 2006 8:46 PM Subject: [sniffer] Re: Significant increase in false positives Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: > Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: > Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Since we have almost all business users and they do a lot of intl biz we just mark the subject as "Probable SPAM:" so no email is deleted. Oh well, I am off topic anyway, thanks for the feedback all. Herb Robert Grosshandler wrote: That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:sniffer@sortmonster.com] On Behalf Of Herb Guenther Sent: Monday, October 16, 2006 4:35 PM To: Message Sniffer Community Subject: [sniffer] Re: Significant increase in false positives Hi Darin; Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue? Herb Darin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
We're seeing it with the latest and greatest gateway version. Again, not a problem. Since it's above our delete weight, always, we just delete them. Users never see them. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Darin CoxSent: Monday, October 16, 2006 5:12 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Ahh... good. The first thing they'll probably tell you is to update to the latest 4.x version, see if the problem persists, then re-report it. Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:51 PM Subject: [sniffer] Re: Significant increase in false positives Not sure, this is what my declude diags.txt saysDeclude 4.1.0 DiagnosticsCompilation Platform: SmarterMailCopyright (c) 2000-2005 Declude, Inc.HerbDarin Cox wrote: We see this occasionally with Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.-- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Here's something I found in our log: 10:16 00:14 SMTP-(f6cb00f3e742) 451 VS5-MF Excessive unknown recipients - possible Open Relay http://help.yahoo.com/help/us/mail/spam/spam-18.html (#4.4.5) [EMAIL PROTECTED] Bill Foresman Matrosity Hosting www.matrosity.com 850.656.2644 From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Matrosity HostingSent: Monday, October 16, 2006 6:17 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Anyone having issues getting email to Yahoo today? Thanks, Bill Foresman Matrosity Hosting www.matrosity.com 850.656.2644 From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Darin CoxSent: Monday, October 16, 2006 6:15 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives For us, it doesn't calculate the proper weight when this happens, and only acts on the weight seen in the topmost headers. One of these years I'll finally exercise the right to use our 4.x license, I just don't have time for new problems at this point. Darin. - Original Message - From: Robert Grosshandler To: Message Sniffer Community Sent: Monday, October 16, 2006 5:57 PM Subject: [sniffer] Re: Significant increase in false positives That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Herb GuentherSent: Monday, October 16, 2006 4:35 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Anyone having issues getting email to Yahoo today? Thanks, Bill Foresman Matrosity Hosting www.matrosity.com 850.656.2644 From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Darin CoxSent: Monday, October 16, 2006 6:15 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives For us, it doesn't calculate the proper weight when this happens, and only acts on the weight seen in the topmost headers. One of these years I'll finally exercise the right to use our 4.x license, I just don't have time for new problems at this point. Darin. - Original Message - From: Robert Grosshandler To: Message Sniffer Community Sent: Monday, October 16, 2006 5:57 PM Subject: [sniffer] Re: Significant increase in false positives That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Herb GuentherSent: Monday, October 16, 2006 4:35 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
For us, it doesn't calculate the proper weight when this happens, and only acts on the weight seen in the topmost headers. One of these years I'll finally exercise the right to use our 4.x license, I just don't have time for new problems at this point. Darin. - Original Message - From: Robert Grosshandler To: Message Sniffer Community Sent: Monday, October 16, 2006 5:57 PM Subject: [sniffer] Re: Significant increase in false positives That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Herb GuentherSent: Monday, October 16, 2006 4:35 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Ahh... good. The first thing they'll probably tell you is to update to the latest 4.x version, see if the problem persists, then re-report it. Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:51 PM Subject: [sniffer] Re: Significant increase in false positives Not sure, this is what my declude diags.txt saysDeclude 4.1.0 DiagnosticsCompilation Platform: SmarterMailCopyright (c) 2000-2005 Declude, Inc.HerbDarin Cox wrote: We see this occasionally with Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.-- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Herb GuentherSent: Monday, October 16, 2006 4:35 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Not sure, this is what my declude diags.txt says Declude 4.1.0 Diagnostics Compilation Platform: SmarterMail Copyright (c) 2000-2005 Declude, Inc. Herb Darin Cox wrote: We see this occasionally with Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin; Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue? Herb Darin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
We see this occasionally with Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Hi Darin; Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue? Herb Darin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
