[sniffer] Significant increase in false positives
Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin.
[sniffer] Re: Significant increase in false positives
Hi Darin; Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue? Herb Darin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
We see this occasionallywith Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Not sure, this is what my declude diags.txt says Declude 4.1.0 Diagnostics Compilation Platform: SmarterMail Copyright (c) 2000-2005 Declude, Inc. Herb Darin Cox wrote: We see this occasionallywith Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin; Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue? Herb Darin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Herb GuentherSent: Monday, October 16, 2006 4:35 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Declude header not modified correctly
Ping them on the Declude list for the lack of response, and CC David Barker for a response. He seem tobe the best means ot getting results these days. What version are you running? Understandably you'll only get a response if you're running the latest 3.x or 4.x, as older versions are no longer supported. Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:58 PM Subject: [sniffer] Re: Declude header not modified correctly It is frustrating because sniffer is catching them and they are not getting marked so they still end up in the ol inbox. Have opened some tickets at declude a few times and never got a response. So no one has a magic bullet on this one?HerbKami Razvan wrote: We see that a lot too.. we run 2.14 Kami From: Message Sniffer Community [mailto:sniffer@sortmonster.com] On Behalf Of Darin CoxSent: Monday, October 16, 2006 5:44 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives We see this occasionallywith Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?Herb-- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Ahh... good. The first thing they'll probably tell you is to update to the latest 4.x version, see if the problem persists, then re-report it. Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:51 PM Subject: [sniffer] Re: Significant increase in false positives Not sure, this is what my declude diags.txt saysDeclude 4.1.0 DiagnosticsCompilation Platform: SmarterMailCopyright (c) 2000-2005 Declude, Inc.HerbDarin Cox wrote: We see this occasionallywith Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.-- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
For us, it doesn't calculate the proper weight when this happens, and only acts on the weight seen in the topmost headers. One of these years I'll finally exercise the right to use our 4.x license, I just don't have time for new problems at this point. Darin. - Original Message - From: Robert Grosshandler To: Message Sniffer Community Sent: Monday, October 16, 2006 5:57 PM Subject: [sniffer] Re: Significant increase in false positives That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Herb GuentherSent: Monday, October 16, 2006 4:35 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Anyone having issues getting email to Yahoo today? Thanks, Bill Foresman Matrosity Hosting www.matrosity.com 850.656.2644 From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Darin CoxSent: Monday, October 16, 2006 6:15 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives For us, it doesn't calculate the proper weight when this happens, and only acts on the weight seen in the topmost headers. One of these years I'll finally exercise the right to use our 4.x license, I just don't have time for new problems at this point. Darin. - Original Message - From: Robert Grosshandler To: Message Sniffer Community Sent: Monday, October 16, 2006 5:57 PM Subject: [sniffer] Re: Significant increase in false positives That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Herb GuentherSent: Monday, October 16, 2006 4:35 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
We're seeing it with the latest and greatest gateway version. Again, not a problem. Since it's above our delete weight, always, we just delete them. Users never see them. Rob From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Darin CoxSent: Monday, October 16, 2006 5:12 PMTo: Message Sniffer CommunitySubject: [sniffer] Re: Significant increase in false positives Ahh... good. The first thing they'll probably tell you is to update to the latest 4.x version, see if the problem persists, then re-report it. Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:51 PM Subject: [sniffer] Re: Significant increase in false positives Not sure, this is what my declude diags.txt saysDeclude 4.1.0 DiagnosticsCompilation Platform: SmarterMailCopyright (c) 2000-2005 Declude, Inc.HerbDarin Cox wrote: We see this occasionallywith Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin;Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue?HerbDarin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.-- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Since we have almost all business users and they do a lot of intl biz we just mark the subject as "Probable SPAM:" so no email is deleted. Oh well, I am off topic anyway, thanks for the feedback all. Herb Robert Grosshandler wrote: That's been a problem for a long time, but for us, it still treats that e-mail as spam, with the appropriate weight. 100% of the time if Declude does that, the e-mail is beyond our delete weight. Rob From: Message Sniffer Community [mailto:sniffer@sortmonster.com] On Behalf Of Herb Guenther Sent: Monday, October 16, 2006 4:35 PM To: Message Sniffer Community Subject: [sniffer] Re: Significant increase in false positives Hi Darin; Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue? Herb Darin Cox wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Darin. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Declude header not modified correctly
Me either, I guess I will have to call them in the AM as it seems to be a general problem. As an aside, I am largely happy with the product but this one has been a long term issue and seems from my experience to be getting exploited by spammers. Andy Schmidt wrote: What's the magic trick to OPENING a ticket on Declude's site. I logged into the customer area, and see no way to open a ticket. But, ifI go to the support page, it specifically instructs me to log into the customer area to open a ticket? Best Regards Andy Schmidt Phone: +1 201 934-3414 x20 (Business) Fax: +1 201 934-9206 From: Message Sniffer Community [mailto:sniffer@sortmonster.com] On Behalf Of Herb Guenther Sent: Monday, October 16, 2006 05:58 PM To: Message Sniffer Community Subject: [sniffer] Re: Declude header not modified correctly It is frustrating because sniffer is catching them and they are not getting marked so they still end up in the ol inbox. Have opened some tickets at declude a few times and never got a response. So no one has a magic bullet on this one? Herb Kami Razvan wrote: We see that a lot too.. we run 2.14 Kami From: Message Sniffer Community [mailto:sniffer@sortmonster.com] On Behalf Of Darin Cox Sent: Monday, October 16, 2006 5:44 PM To: Message Sniffer Community Subject: [sniffer] Re: Significant increase in false positives We see this occasionallywith Declude 1.82. What version are you running? Darin. - Original Message - From: Herb Guenther To: Message Sniffer Community Sent: Monday, October 16, 2006 5:35 PM Subject: [sniffer] Re: Significant increase in false positives Hi Darin; Not seeing a lot of false pos messages, but there are lots of spam messages sneaking through our system because declude is not modifying the header correctly. It is adding a header stub to the bottom of the message so that users mail client filters which look for the modified subject line is not working. Anyone else having that issue? Herb -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way. -- Herb Guenther Lanex, LLC www.lanex.com (262)789-0966x102 Office (262)780-0424 Direct This e-mail is confidential and is for the use of the intended recipient(s)only. If you are not an intended recipient please advise us of our error by return e-mail then delete this e-mail and any attached files. You may not copy, disclose or use the contents in any way.
[sniffer] Re: Significant increase in false positives
Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Significant increase in false positives
Hi Pete, I haven't looked at the Sniffer logs, as cross referencing from the Declude logs is a bit of a pain, but many of the FPs did have images, so that probably accounts for most of them if it was an Experimental rule. Darin. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Monday, October 16, 2006 8:46 PM Subject: [sniffer] Re: Significant increase in false positives Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Significant increase in false positives
Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work). Thanks, Matt Pete McNeil wrote: Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Significant increase in false positives
Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name image spam binary segment as text !1AQaq"2 Created 2006-10-14 Source !1AQaq"2 Hidden false Blocked false Origin Spam Trap Type Simple Text Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength 3.20638481603822 False Reports 11 From Users 7 Rule belongs to following groups [252] Problematic I removed the rule as soon as we began receiving reports - about mid-day today. I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work). I have discussed this with Declude and I am hopeful that we will have better integration w/ Declude some time in the future. In the mean time, our next version will include a feature to inject headers into message files. Understand, however, that this is an expensive feature that will substantially increase the I/O requirements on any mail server. Injecting headers requires that the entire message file must be written to disk an additional time. This is not a small consideration-- Where once most spam were tiny text/html files (often less than 5K) today's image spam variants are frequently 5 to 10 times the size of the old spam we used to know. Also- note that this kind of thing can be very buggy on Winx systems -- sometimes changes to files are not reflected immediately between processes. For example, rename operations are not atomic - so when the old message file is deleted and the new version is renamed from it's temp file to the original message file name, other Winx processes that depend on that file may not respond reliably. For all of these reasons and more I've probably not thought of - this feature will be a "use at your own risk / YMMV" option. All that said, there is an existing option in the current version of SNF to produce a .xhdr file for each message. This option is frequently used in *nix systems that use SNF. It would be possible to write a short utility (perhaps even a script) that would modify quarantined messages out-of-band to include the contents of the .xhdr file as X- headers. Such a utility is not currently on our development list, however, and I hallucinate that such a device would tend to evolve into something somewhat system specific. The best option would be for Declude to add a feature that picks up x-headers created by external programs (perhaps in files named message-file-name.xhdr) so that they can be added in a single message rewrite along with the headers that Declude already adds. This would solve the I/O problems and standardize the mechanism for any other external programs that might wish to add headers. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Significant increase in false positives
Hi Matt, I know Pete has requested this in the past, but Declude hasn't been willing to make the change necessary for this to make it in the headers. But I totally agree with you, I'd love to see this in the headers so tracking down the rule isn't such a pain. Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Monday, October 16, 2006 10:03 PM Subject: [sniffer] Re: Significant increase in false positives Pete,Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help.I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work).Thanks,MattPete McNeil wrote: Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC.# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Significant increase in false positives
Hi Pete, Can you clarify what this .xhdr option is and how we can enable it? I don't remember anything inthe documentationthat describes it. I think there were references to the config file previously, but there was never anything about it in mine. If you could give an example of how to enable and use the info it would be greatly appreciated. Darin. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Monday, October 16, 2006 11:13 PM Subject: [sniffer] Re: Significant increase in false positives Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name image spam binary segment as text !1AQaq"2 Created 2006-10-14 Source !1AQaq"2 Hidden false Blocked false Origin Spam Trap Type Simple Text Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength 3.20638481603822 False Reports 11 From Users 7 Rule belongs to following groups [252] Problematic I removed the rule as soon as we began receiving reports - about mid-day today. I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work). I have discussed this with Declude and I am hopeful that we will have better integration w/ Declude some time in the future. In the mean time, our next version will include a feature to inject headers into message files. Understand, however, that this is an expensive feature that will substantially increase the I/O requirements on any mail server. Injecting headers requires that the entire message file must be written to disk an additional time. This is not a small consideration-- Where once most spam were tiny text/html files (often less than 5K) today's image spam variants are frequently 5 to 10 times the size of the old spam we used to know. Also- note that this kind of thing can be very buggy on Winx systems -- sometimes changes to files are not reflected immediately between processes. For example, rename operations are not atomic - so when the old message file is deleted and the new version is renamed from it's temp file to the original message file name, other Winx processes that depend on that file may not respond reliably. For all of these reasons and more I've probably not thought of - this feature will be a "use at your own risk / YMMV" option. All that said, there is an existing option in the current version of SNF to produce a .xhdr file for each message. This option is frequently used in *nix systems that use SNF. It would be possible to write a short utility (perhaps even a script) that would modify quarantined messages out-of-band to include the contents of the .xhdr file as X- headers. Such a utility is not currently on our development list, however, and I hallucinate that such a device would tend to evolve into something somewhat system specific. The best option would be for Declude to add a feature that picks up x-headers created by external programs (perhaps in files named message-file-name.xhdr) so that they can be added in a single message rewrite along with the headers that Declude already adds. This would solve the I/O problems and standardize the mechanism for any other external programs that might wish to add headers. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To
[sniffer] Re: Significant increase in false positives
There is no doubt that having Declude handle xhdr files would be optimal. I might add that an option to exclude the header on non-hits would also be wise. David Barker appears open to some feature requests of late, and I would think that you could make this happen. Not everyone has capacity limitations, so the internal functionality would probably suit the needs of many of your users also, and cover all non-SA systems instead of just Declude. Regarding this rule, the binary segment is non-searchable. My only solution would be to write some _vbscript_ that parsed the Sniffer log for hits and move the files from my CopyFile directory back into Declude's Proc. I'm guessing that someone could also do some grepping for this, but that ain't a strength of mine. I could do this in minutes though if I had headers to search on. Thankfully this rule only hit about 1,000 times this weekend as a final match (I'm ignoring those that weren't final matches since those would have hit anyway). My gateway gets rid of most image spams, so I would expect a comparably higher rate for others. Regarding false positives in general. I don't expect Sniffer to be perfect due to the way that rules are generated, but I have two suggestions. 1) One would be to test all new rules on a small sub-set of E-mail that covers the most common patterns such as attachments and E-mail/webmail clients with various formats including forwards and replies. This would likely stop the worst of the worst in terms of FP issues like the one earlier this year that was hitting on most base64 code. I envision hundreds of test messages and not thousands, so this should be practical. 2) The second suggestion is one that I have mentioned many times before in private involving being able to tag messages on multiple types of hits for a stronger result. The separation would need to be on the type rule so that all rule types would be isolated from one another. For instance, phrase, pattern, IP and domain rules could be put in different codes and allowed to be scored in combination. It would also be equally as important to treat rules from user submissions different from those generated from spam traps since these rules are not nearly as universal. I currently average just under 3 matches per message that Sniffer hits, and I would imagine that there is a lot of mixing between these types. This would allow many that are scoring Sniffer lower than our block weight to then score these multiple classification hits higher. This wouldn't be useful though unless it was seperated by types like I listed since I often find multiple hits under the current rulebase format. Thanks, Matt Pete McNeil wrote: Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name image spam binary segment as text !1AQaq"2 Created 2006-10-14 Source !1AQaq"2 Hidden false Blocked false Origin Spam Trap Type Simple Text Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength 3.20638481603822 False Reports 11 From Users
