[sniffer] Bad Rule - 828931
Hello Sniffer folks, I'm sorry to report that another bad rule got past us today. The rule has been removed (was in from about 1200-1500), but it may be in some of your rulebases. To avoid a problem with this rule you can enter a rule-panic entry in your .cfg file for rule id: 828931 If it is not already, the rule will be gone from your rulebase after your next update. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) Chief Scientist (www.armresearch.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bad Rule - 828931
Dear Pete, In the future, please let us know immediately when you become aware of this. As it is, I will spend the next 3 hours picking out the fales positives from the mailbox and forwarding them to the clients. If I could have put the rulepanic in place an hour ago it would have saved me a lot of work and confused customers. Thank you, Michael Stein Computer House - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: sniffer@sortmonster.com Sent: Tuesday, February 07, 2006 4:07 PM Subject: [sniffer] Bad Rule - 828931 Hello Sniffer folks, I'm sorry to report that another bad rule got past us today. The rule has been removed (was in from about 1200-1500), but it may be in some of your rulebases. To avoid a problem with this rule you can enter a rule-panic entry in your .cfg file for rule id: 828931 If it is not already, the rule will be gone from your rulebase after your next update. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) Chief Scientist (www.armresearch.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[2]: [sniffer] Bad Rule - 828931
I do most humbly apologize, It was my intention to do it immediately, however I became embroiled in related support issues and was delayed. I don't expect more of these, but I will make announcing their discovery the next event after removing them from the system. Thanks, _M On Tuesday, February 7, 2006, 4:19:24 PM, Computer wrote: CHS Dear Pete, CHS In the future, please let us know immediately when you become aware of this. CHS As it is, I will spend the next 3 hours picking out the fales positives from CHS the mailbox and forwarding them to the clients. If I could have put the CHS rulepanic in place an hour ago it would have saved me a lot of work and CHS confused customers. CHS Thank you, CHS Michael Stein CHS Computer House CHS - Original Message - CHS From: Pete McNeil [EMAIL PROTECTED] CHS To: sniffer@sortmonster.com CHS Sent: Tuesday, February 07, 2006 4:07 PM CHS Subject: [sniffer] Bad Rule - 828931 CHS Hello Sniffer folks, CHS I'm sorry to report that another bad rule got past us today. The CHS rule has been removed (was in from about 1200-1500), but it may be CHS in some of your rulebases. CHS To avoid a problem with this rule you can enter a rule-panic entry CHS in your .cfg file for rule id: 828931 CHS If it is not already, the rule will be gone from your rulebase after CHS your next update. CHS Thanks, CHS _M CHS Pete McNeil (Madscientist) CHS President, MicroNeil Research Corporation CHS Chief SortMonster (www.sortmonster.com) CHS Chief Scientist (www.armresearch.com) CHS This E-Mail came from the Message Sniffer mailing list. For information and CHS (un)subscription instructions go to CHS http://www.sortmonster.com/MessageSniffer/Help/Help.html CHS This E-Mail came from the Message Sniffer mailing list. For CHS information and (un)subscription instructions go to CHS http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Bad Rule - 828931
Dear Pete, Please excuse my previous E-mail if it seemed a bit harsh. I guess I am so used to your great service, that on the rare occasion when this happens, I panic. Thanks for being there to walk me through the procedure. Sincerely, Michael Stein Computer House - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: Computer House Support sniffer@SortMonster.com Sent: Tuesday, February 07, 2006 4:24 PM Subject: Re[2]: [sniffer] Bad Rule - 828931 I do most humbly apologize, It was my intention to do it immediately, however I became embroiled in related support issues and was delayed. I don't expect more of these, but I will make announcing their discovery the next event after removing them from the system. Thanks, _M On Tuesday, February 7, 2006, 4:19:24 PM, Computer wrote: CHS Dear Pete, CHS In the future, please let us know immediately when you become aware of this. CHS As it is, I will spend the next 3 hours picking out the fales positives from CHS the mailbox and forwarding them to the clients. If I could have put the CHS rulepanic in place an hour ago it would have saved me a lot of work and CHS confused customers. CHS Thank you, CHS Michael Stein CHS Computer House CHS - Original Message - CHS From: Pete McNeil [EMAIL PROTECTED] CHS To: sniffer@sortmonster.com CHS Sent: Tuesday, February 07, 2006 4:07 PM CHS Subject: [sniffer] Bad Rule - 828931 CHS Hello Sniffer folks, CHS I'm sorry to report that another bad rule got past us today. The CHS rule has been removed (was in from about 1200-1500), but it may be CHS in some of your rulebases. CHS To avoid a problem with this rule you can enter a rule-panic entry CHS in your .cfg file for rule id: 828931 CHS If it is not already, the rule will be gone from your rulebase after CHS your next update. CHS Thanks, CHS _M CHS Pete McNeil (Madscientist) CHS President, MicroNeil Research Corporation CHS Chief SortMonster (www.sortmonster.com) CHS Chief Scientist (www.armresearch.com) CHS This E-Mail came from the Message Sniffer mailing list. For information and CHS (un)subscription instructions go to CHS http://www.sortmonster.com/MessageSniffer/Help/Help.html CHS This E-Mail came from the Message Sniffer mailing list. For CHS information and (un)subscription instructions go to CHS http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Downloads are slow.
I'm not showing this from my location and the server looks ok. I just downloaded a few rulebases, each in under 3 seconds. Please provide a traceroute -- that should show us where the issue is (if it is still there). Thanks, _M On Tuesday, February 7, 2006, 4:39:35 PM, Chuck wrote: CS Download speeds from your server are running 17 kbps at my location. CS Chuck Schick CS Warp 8, Inc. CS (303)-421-5140 CS www.warp8.com CS This E-Mail came from the Message Sniffer mailing list. For CS information and (un)subscription instructions go to CS http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Downloads are slow.
Agreed, my last report showed pretty slow times. All today were slower now that I look at them. I normally see up to 1.3M with overall times around 800-900K. John C 0K .. .. .. .. .. 36.79 KB/s 50K .. .. .. .. .. 11.51 KB/s 100K .. .. .. .. .. 19.76 KB/s 150K .. .. .. .. .. 11.98 KB/s 200K .. .. .. .. .. 37.20 KB/s 250K .. .. .. .. .. 10.60 KB/s 300K .. .. .. .. .. 16.00 KB/s 350K .. .. .. .. .. 19.05 KB/s 400K .. .. .. .. .. 22.22 KB/s 450K .. .. .. .. .. 10.32 KB/s 500K .. .. .. .. .. 13.50 KB/s 550K .. .. .. .. ..2.74 KB/s 600K .. .. .. .. ..8.40 KB/s 650K .. .. .. .. ..6.00 KB/s 700K .. .. .. .. ..9.97 KB/s 750K .. .. .. .. ..6.07 KB/s 800K .. .. .. .. ..5.89 KB/s 850K .. .. .. .. ..9.20 KB/s 900K .. .. .. .. ..6.46 KB/s 950K .. .. .. .. ..4.94 KB/s 1000K .. .. .. .. ..7.67 KB/s 1050K .. .. .. .. ..9.97 KB/s 1100K .. .. .. .. .. 13.28 KB/s 1150K .. .. .. .. .. 24.61 KB/s 1200K .. .. .. .. .. 12.36 KB/s 1250K .. .. .. .. .. 31.06 KB/s 1300K .. .. .. .. ..4.87 KB/s 1350K .. .. .. .. .. 34.77 KB/s 1400K .. .. .. .. .. 14.29 KB/s 1450K .. . .. .. .. 16.24 KB/s 1500K .. .. .. .. .. 33.33 KB/s 1550K .. . .. .. .. 21.48 KB/s 1600K .. .. .. .. .. 23.19 KB/s 1650K .. .. .. .. .. 27.34 KB/s 1700K .. .. .. .. .. 14.68 KB/s 1750K .. .. .. .. .. 47.76 KB/s 1800K .. .. .. .. .. 15.17 KB/s 1850K .. .. .. .. .. 16.17 KB/s 1900K .. .. .. .. .. 18.39 KB/s 1950K .. .. .. .. .. 74.40 KB/s 2000K .. .. .. .. .. 14.10 KB/s 2050K .. .. .. .. .. 12.70 KB/s 2100K .. .. .. .. .. 29.36 KB/s 2150K .. .. .. .. .. 16.58 KB/s 2200K .. .. .. .. .. 21.62 KB/s 2250K .. .. .. .. .. 17.49 KB/s 2300K .. .. .. .. .. 11.00 KB/s 2350K .. .. .. .. .. 21.20 KB/s 2400K .. .. .. .. .. 31.69 KB/s 2450K .. .. .. .. .. 20.12 KB/s 2500K .. .. .. .. .. 57.14 KB/s 2550K .. .. .. 13.94 KB/s 15:52:29 (12.45 KB/s) - `.new.gz' saved [2646653] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, February 07, 2006 4:46 PM To: Chuck Schick Subject: Re: [sniffer] Downloads are slow. I'm not showing this from my location and the server looks ok. I just downloaded a few rulebases, each in under 3 seconds. Please provide a traceroute -- that should show us where the issue is (if it is still there). Thanks, _M On Tuesday, February 7, 2006, 4:39:35 PM, Chuck wrote: CS Download speeds from your server are running 17 kbps at my location. CS Chuck Schick CS Warp 8, Inc. CS (303)-421-5140 CS www.warp8.com CS This E-Mail came from the Message Sniffer mailing list. For CS information and (un)subscription instructions go to CS http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing
Re[2]: [sniffer] Downloads are slow.
Somebody please tell me I'm doing something wrong here. I use this expression in Baregrep Final\t828931 and it yields 22,055 matching lines across 3 of my 4 license's log files. Since this is set to my hold weight, I'm assuming that means I've had 22,055 holds on this rule? -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Bad Rule - 828931
Sorry, wrong thread on the last post. Add'l question. Pete, what is the content of the rule? Tuesday, February 7, 2006, 6:05:53 PM, you wrote: DS Somebody please tell me I'm doing something wrong here. I use this DS expression in Baregrep Final\t828931 and it yields 22,055 matching DS lines across 3 of my 4 license's log files. DS Since this is set to my hold weight, I'm assuming that means I've had DS 22,055 holds on this rule? -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[2]: [sniffer] Bad Rule - 828931
Hello Matt, Tuesday, February 7, 2006, 6:27:25 PM, you wrote: M rule number, and I don't have the tools set up or the knowledge of grep M yet to do a piped query of Sniffer's logs to extract the spool file names. http://www.baremetalsoft.com/ is a great grep'er for windows. In BSD I always used .* to represent any number of characters, white space or non, but that didn't seem to work with baregrep. That's why I was trying to confirm with anyone on the list my regex of Final\t828931 was an accurate regex to find every message that 'finaled' on that rule. I'm praying that I screwed up the expression and I don't have 22,055 messages held by that rule. M BTW, David, it is generally better not to hold or block on one single M test, especially one that automates such listings (despite whatever M safeguards there might be). I know, shame on me. I guess I'm used to the days that we used to be able to hold on sniffer alone. We have some safeguards in place now and are transitioning our rule methodologies but hadn't gotten to this one yet as this always seems to hit back-burner. This is also why I'd really like to see the content of the rule to see how it made it passed our safeguards. -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] Bad Rule - 828931
Don't know about the proper syntax for baregrep, but for the standard UNIX grep for Win32, the following would give you an accurate count: grep -c Final.*828931 c:\imail\declude\sniffer\logfile.log Bill -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Sullivan Sent: Tuesday, February 07, 2006 4:12 PM To: sniffer@SortMonster.com Subject: Re[2]: [sniffer] Bad Rule - 828931 Hello Matt, Tuesday, February 7, 2006, 6:27:25 PM, you wrote: M rule number, and I don't have the tools set up or the knowledge of M grep yet to do a piped query of Sniffer's logs to extract the spool file names. http://www.baremetalsoft.com/ is a great grep'er for windows. In BSD I always used .* to represent any number of characters, white space or non, but that didn't seem to work with baregrep. That's why I was trying to confirm with anyone on the list my regex of Final\t828931 was an accurate regex to find every message that 'finaled' on that rule. I'm praying that I screwed up the expression and I don't have 22,055 messages held by that rule. M BTW, David, it is generally better not to hold or block on one single M test, especially one that automates such listings (despite whatever M safeguards there might be). I know, shame on me. I guess I'm used to the days that we used to be able to hold on sniffer alone. We have some safeguards in place now and are transitioning our rule methodologies but hadn't gotten to this one yet as this always seems to hit back-burner. This is also why I'd really like to see the content of the rule to see how it made it passed our safeguards. -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] Bad Rule - 828931
Final\t828931 and Final.*828931 both found 850 entries in my current log using Baregrep. John C -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Sullivan Sent: Tuesday, February 07, 2006 6:12 PM To: sniffer@SortMonster.com Subject: Re[2]: [sniffer] Bad Rule - 828931 Hello Matt, Tuesday, February 7, 2006, 6:27:25 PM, you wrote: M rule number, and I don't have the tools set up or the knowledge of M grep yet to do a piped query of Sniffer's logs to extract the spool file names. http://www.baremetalsoft.com/ is a great grep'er for windows. In BSD I always used .* to represent any number of characters, white space or non, but that didn't seem to work with baregrep. That's why I was trying to confirm with anyone on the list my regex of Final\t828931 was an accurate regex to find every message that 'finaled' on that rule. I'm praying that I screwed up the expression and I don't have 22,055 messages held by that rule. M BTW, David, it is generally better not to hold or block on one single M test, especially one that automates such listings (despite whatever M safeguards there might be). I know, shame on me. I guess I'm used to the days that we used to be able to hold on sniffer alone. We have some safeguards in place now and are transitioning our rule methodologies but hadn't gotten to this one yet as this always seems to hit back-burner. This is also why I'd really like to see the content of the rule to see how it made it passed our safeguards. -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bad Rule - 828931
On Tuesday, February 7, 2006, 6:15:13 PM, David wrote: DS Sorry, wrong thread on the last post. DS Add'l question. Pete, what is the content of the rule? The rule info is: Rule - 828931 NameC%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Created 2006-02-07 Source C%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Hidden false Blocked false Origin User Submission TypeManual Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength3.84258274153269 False Reports 0 From Users 0 Rule belongs to following groups [252] Problematic The rule was an attempt to build an abstract matching two ed pill names (you can see them in there) while compensating for heavy obfuscation. The mistake was in using %+ through the rule. The rule would match the intended spam (and there was a lot of it, so 22,055 most likely includes mostly spam. Unfortunately it would also match messages containing the listed capital letters in that order throughout the message. Essentially, if the text is long enough then it will probably match. A greater chance of FP match if the text of the message is in all caps. Also if there is a badly coded base64 segment and file attachment (badly coded base64 might not be decoded... raw base64 will contain many of these letters in mixed case and therefore increase the probability of matching them all). Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Date/time stamp in logs
I don't get into the sniffer logs like I should, but just noticed this. It is 2/7/06 6:42 CST here, but my logs show 20060208004243, which would indicate +6 hours off of Zulu, Greenwich, Coordinated Universal Time, or whatever we are calling these days. Is that right, sniffer doesn't stamp local time? John C This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[2]: [sniffer] Downloads are slow.
I've had an internal note that our colo provider is working on a networking problem. That's probably what you're seeing. Apparently it doesn't effect all paths to the 'net equally and/or it may be solved by now. _M On Tuesday, February 7, 2006, 5:53:35 PM, John wrote: JC Agreed, my last report showed pretty slow times. All today were slower now JC that I look at them. I normally see up to 1.3M with overall times around JC 800-900K. JC John C JC 0K .. .. .. .. .. 36.79 KB/s JC50K .. .. .. .. .. 11.51 KB/s JC 100K .. .. .. .. .. 19.76 KB/s JC 150K .. .. .. .. .. 11.98 KB/s JC 200K .. .. .. .. .. 37.20 KB/s JC 250K .. .. .. .. .. 10.60 KB/s JC 300K .. .. .. .. .. 16.00 KB/s JC 350K .. .. .. .. .. 19.05 KB/s JC 400K .. .. .. .. .. 22.22 KB/s JC 450K .. .. .. .. .. 10.32 KB/s JC 500K .. .. .. .. .. 13.50 KB/s JC 550K .. .. .. .. ..2.74 KB/s JC 600K .. .. .. .. ..8.40 KB/s JC 650K .. .. .. .. ..6.00 KB/s JC 700K .. .. .. .. ..9.97 KB/s JC 750K .. .. .. .. ..6.07 KB/s JC 800K .. .. .. .. ..5.89 KB/s JC 850K .. .. .. .. ..9.20 KB/s JC 900K .. .. .. .. ..6.46 KB/s JC 950K .. .. .. .. ..4.94 KB/s JC 1000K .. .. .. .. ..7.67 KB/s JC 1050K .. .. .. .. ..9.97 KB/s JC 1100K .. .. .. .. .. 13.28 KB/s JC 1150K .. .. .. .. .. 24.61 KB/s JC 1200K .. .. .. .. .. 12.36 KB/s JC 1250K .. .. .. .. .. 31.06 KB/s JC 1300K .. .. .. .. ..4.87 KB/s JC 1350K .. .. .. .. .. 34.77 KB/s JC 1400K .. .. .. .. .. 14.29 KB/s JC 1450K .. . .. .. .. 16.24 KB/s JC 1500K .. .. .. .. .. 33.33 KB/s JC 1550K .. . .. .. .. 21.48 KB/s JC 1600K .. .. .. .. .. 23.19 KB/s JC 1650K .. .. .. .. .. 27.34 KB/s JC 1700K .. .. .. .. .. 14.68 KB/s JC 1750K .. .. .. .. .. 47.76 KB/s JC 1800K .. .. .. .. .. 15.17 KB/s JC 1850K .. .. .. .. .. 16.17 KB/s JC 1900K .. .. .. .. .. 18.39 KB/s JC 1950K .. .. .. .. .. 74.40 KB/s JC 2000K .. .. .. .. .. 14.10 KB/s JC 2050K .. .. .. .. .. 12.70 KB/s JC 2100K .. .. .. .. .. 29.36 KB/s JC 2150K .. .. .. .. .. 16.58 KB/s JC 2200K .. .. .. .. .. 21.62 KB/s JC 2250K .. .. .. .. .. 17.49 KB/s JC 2300K .. .. .. .. .. 11.00 KB/s JC 2350K .. .. .. .. .. 21.20 KB/s JC 2400K .. .. .. .. .. 31.69 KB/s JC 2450K .. .. .. .. .. 20.12 KB/s JC 2500K .. .. .. .. .. 57.14 KB/s JC 2550K .. .. .. 13.94 KB/s JC 15:52:29 (12.45 KB/s) - `.new.gz' saved [2646653] JC -Original Message- JC From: [EMAIL PROTECTED] JC [mailto:[EMAIL PROTECTED] JC On Behalf Of Pete McNeil JC Sent: Tuesday, February 07, 2006 4:46 PM JC To: Chuck Schick JC Subject: Re: [sniffer] Downloads are slow. JC I'm not showing this from my location and the server looks ok. JC I just downloaded a few rulebases, each in under 3 seconds. JC Please provide a traceroute -- that should show us where the issue
Re[2]: [sniffer] Bad Rule - 828931
Hello Pete, Tuesday, February 7, 2006, 7:43:52 PM, you wrote: PM The rule would match the intended spam (and there was a lot of it, so PM 22,055 most likely includes mostly spam. On spot check I'm seeing about 30-40% of the messages are valid. PM Unfortunately it would also match messages containing the listed PM capital letters in that order throughout the message. Essentially, if PM the text is long enough then it will probably match. A greater chance PM of FP match if the text of the message is in all caps. Also if there PM is a badly coded base64 segment and file attachment (badly coded PM base64 might not be decoded... raw base64 will contain many of these PM letters in mixed case and therefore increase the probability of PM matching them all). Not sure, can anyone think of a way to cross check this? What if I put all the released messages back through sniffer? -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Bad Rule - 828931
So, in my terms (simple), this rule only catches msg if the two drug names are in that order and in all capitals, but not necessarily one immediately following the other? John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, February 07, 2006 6:44 PM To: David Sullivan Subject: Re: [sniffer] Bad Rule - 828931 On Tuesday, February 7, 2006, 6:15:13 PM, David wrote: DS Sorry, wrong thread on the last post. DS Add'l question. Pete, what is the content of the rule? The rule info is: Rule - 828931 NameC%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Created 2006-02-07 Source C%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Hidden false Blocked false Origin User Submission TypeManual Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength3.84258274153269 False Reports 0 From Users 0 Rule belongs to following groups [252] Problematic The rule was an attempt to build an abstract matching two ed pill names (you can see them in there) while compensating for heavy obfuscation. The mistake was in using %+ through the rule. The rule would match the intended spam (and there was a lot of it, so 22,055 most likely includes mostly spam. Unfortunately it would also match messages containing the listed capital letters in that order throughout the message. Essentially, if the text is long enough then it will probably match. A greater chance of FP match if the text of the message is in all caps. Also if there is a badly coded base64 segment and file attachment (badly coded base64 might not be decoded... raw base64 will contain many of these letters in mixed case and therefore increase the probability of matching them all). Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Date/time stamp in logs
On Tuesday, February 7, 2006, 7:48:05 PM, John wrote: JC I don't get into the sniffer logs like I should, but just noticed this. It JC is 2/7/06 6:42 CST here, but my logs show 20060208004243, which would JC indicate +6 hours off of Zulu, Greenwich, Coordinated Universal Time, or JC whatever we are calling these days. Is that right, sniffer doesn't stamp JC local time? That's right. Sniffer stamps GMT so that all sniffer logs from all systems can be coordinated easily. Similarly, system events (like the last update on a rulebase) are recorded/represented here in GMT. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[5]: [sniffer] Bad Rule - 828931
On Tuesday, February 7, 2006, 8:14:53 PM, David wrote: DS Hello Pete, DS Tuesday, February 7, 2006, 8:11:50 PM, you wrote: DS Not sure, can anyone think of a way to cross check this? What if I put DS all the released messages back through sniffer? PM That would be good -- new rules were added to correctly capture the PM bad stuff. I almost suggested something more complex. DS That said...anyone know specifics of reprocessing messages through DS Declude on Imail? I know that in 1.x Declude would drop some kind of DS marker so that q/d's copied into spool would not be reprocessed but I DS don't remember what it was and don't know if it works same in 3.x. DS Posted question on Declude JM list but no answer so far. IIRC messages in the spool under scan would be locked until declude was done with them. After that, placing the Q and D files into the spool would mean that normal IMail processes would deliver them on the next sweep. The way around this was to place the messages back in the overflow folder (I'm not sure which parts - I think the Q goes in overflow and the D stays in spool -- someone will know for sure). The theory there is that messages sent to the overflow folder are sent there before they are scanned in order to backlog the extra processing load. So, messages coming out of the overflow folder would naturally be scanned ( for the first time - thinks the robot ). _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bad Rule - 828931
Pete, Gotcha. Basically anything that I trapped that is over 10 KB may have failed this (because that would be indicative of having an attachment in base64). It is much less likely to have hit on things without attachments, but it of course would be possible, and the bigger it was, the more likely that it could have failed. I also searched my Sniffer logs for the rule number and found no hits. It appears that I missed the bad rulebase. Thanks, Matt Pete McNeil wrote: On Tuesday, February 7, 2006, 6:15:13 PM, David wrote: DS Sorry, wrong thread on the last post. DS Add'l question. Pete, what is the content of the rule? The rule info is: Rule - 828931 NameC%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Created 2006-02-07 Source C%+I%+A%+L%+I%+S%+V%+I%+A%+G%+R%+A Hidden false Blocked false Origin User Submission TypeManual Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength3.84258274153269 False Reports 0 From Users 0 Rule belongs to following groups [252] Problematic The rule was an attempt to build an abstract matching two ed pill names (you can see them in there) while compensating for heavy obfuscation. The mistake was in using %+ through the rule. The rule would match the intended spam (and there was a lot of it, so 22,055 most likely includes mostly spam. Unfortunately it would also match messages containing the listed capital letters in that order throughout the message. Essentially, if the text is long enough then it will probably match. A greater chance of FP match if the text of the message is in all caps. Also if there is a badly coded base64 segment and file attachment (badly coded base64 might not be decoded... raw base64 will contain many of these letters in mixed case and therefore increase the probability of matching them all). Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bad Rule - 828931
Pete, The overflow directory disappeared when 3.x was introduced. I posted a follow up on the Declude list about how to do this. Matt Pete McNeil wrote: On Tuesday, February 7, 2006, 8:14:53 PM, David wrote: DS Hello Pete, DS Tuesday, February 7, 2006, 8:11:50 PM, you wrote: DS Not sure, can anyone think of a way to cross check this? What if I put DS all the released messages back through sniffer? PM That would be good -- new rules were added to correctly capture the PM bad stuff. I almost suggested something more complex. DS That said...anyone know specifics of reprocessing messages through DS Declude on Imail? I know that in 1.x Declude would drop some kind of DS marker so that q/d's copied into spool would not be reprocessed but I DS don't remember what it was and don't know if it works same in 3.x. DS Posted question on Declude JM list but no answer so far. IIRC messages in the spool under scan would be locked until declude was done with them. After that, placing the Q and D files into the spool would mean that normal IMail processes would deliver them on the next sweep. The way around this was to place the messages back in the overflow folder (I'm not sure which parts - I think the Q goes in overflow and the D stays in spool -- someone will know for sure). The theory there is that messages sent to the overflow folder are sent there before they are scanned in order to backlog the extra processing load. So, messages coming out of the overflow folder would naturally be scanned ( for the first time - thinks the robot ). _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[4]: [sniffer] Bad Rule - 828931
David Drop the q/d files back into the \spool\proc directory. Declude will reprocess them. If you put them in just the \spool, queue manager will send them out in the next queue run, bypassing Declude. John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Sullivan Sent: Tuesday, February 07, 2006 7:15 PM To: Pete McNeil Subject: Re[4]: [sniffer] Bad Rule - 828931 Hello Pete, Tuesday, February 7, 2006, 8:11:50 PM, you wrote: DS Not sure, can anyone think of a way to cross check this? What if I DS put all the released messages back through sniffer? PM That would be good -- new rules were added to correctly capture the PM bad stuff. I almost suggested something more complex. That said...anyone know specifics of reprocessing messages through Declude on Imail? I know that in 1.x Declude would drop some kind of marker so that q/d's copied into spool would not be reprocessed but I don't remember what it was and don't know if it works same in 3.x. Posted question on Declude JM list but no answer so far. -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[4]: [sniffer] Bad Rule - 828931
I just ran the grep command on my log and I got 850 hits. Now is there a way to take the output of the grep command and use it pull out the total weight of corresponding message from the declude log file, or maybe the subject? Goran Jovanovic Omega Network Solutions -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Sullivan Sent: Tuesday, February 07, 2006 7:47 PM To: Landry, William (MED US) Subject: Re[4]: [sniffer] Bad Rule - 828931 Hello William, Tuesday, February 7, 2006, 7:39:05 PM, you wrote: LWMU grep -c Final.*828931 c:\imail\declude\sniffer\logfile.log That's what I tried. Just figured out I forgot to capitalize the F. It works. Confirmed - 22,055 I'm writing a program now to parse the sniffer log file, extract the file ID, lookup the id in sql server, determine quarantine location, extract q/d pair from quarantine and send to user. -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[6]: [sniffer] Bad Rule - 828931
Hello John, Tuesday, February 7, 2006, 8:30:57 PM, you wrote: JC Drop the q/d files back into the \spool\proc directory. Declude will JC reprocess them. If you put them in just the \spool, queue manager will send JC them out in the next queue run, bypassing Declude. Perfect, thanks. I just dropped the q/d from known spam into \proc and it reprocessed and HELD again. -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Bad Rule - 828931
Thanks for the update, Pete. I also appreciate that you expanded on how that rule went wild. I can see that the intent was good but the unintended consequences were not so good. Here's how it played out on my server: How many messages hit the FP rules: 2,042 How many messages Declude decided were ham anyway: 1,093 How many messages Declude decided were viruses: 0 How many messages Declude decided were spam: 949 Of the spam, when re-queued, how many were ham: 583 Of the spam, when re-queued, how many were still spam: 366 So, in total: How many messages hit the bad 828931 rule: 2,042 How many were indeed spam: 366 How many were false positives: 1,676 Andrew 8) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Bad Rule - 828931
Thanks for the update, Pete.I also appreciate that you expanded on how that rule went wild. I can see that the intent was good but the unintended consequences were not so good.Here's how it played out on my server:How many messages hit the FP rules: 2,042How many messages Declude decided were ham anyway: 1,093How many messages Declude decided were viruses: 0How many messages Declude decided were spam: 949Of the spam, when re-queued, how many were ham: 583Of the spam, when re-queued, how many were still spam: 366So, in total:How many messages hit the bad 828931 rule: 2,042How many were indeed spam: 366How many were false positives: 1,676Andrew 8)p.s. Re-posted in HTML so that I don't have to explain the line breaks that were eaten in the plain text version post.
RE: Re[4]: [sniffer] Bad Rule - 828931
OK to answer my own question. Run the following commands grep -U Final.828931 snf.log 1.txt cut -b26-41 1.txt 2.txt grep -U -f2.txt d:\spool\dec0207.log 3.txt egrep -U \smd Tests failed|\smd Subject 3.txt 4.txt notepad 4.txt Now I have to read my 4.txt and figure out what I am going to do about it. Goran Jovanovic Omega Network Solutions -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Goran Jovanovic Sent: Tuesday, February 07, 2006 8:39 PM To: sniffer@SortMonster.com Subject: RE: Re[4]: [sniffer] Bad Rule - 828931 I just ran the grep command on my log and I got 850 hits. Now is there a way to take the output of the grep command and use it pull out the total weight of corresponding message from the declude log file, or maybe the subject? Goran Jovanovic Omega Network Solutions -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Sullivan Sent: Tuesday, February 07, 2006 7:47 PM To: Landry, William (MED US) Subject: Re[4]: [sniffer] Bad Rule - 828931 Hello William, Tuesday, February 7, 2006, 7:39:05 PM, you wrote: LWMU grep -c Final.*828931 c:\imail\declude\sniffer\logfile.log That's what I tried. Just figured out I forgot to capitalize the F. It works. Confirmed - 22,055 I'm writing a program now to parse the sniffer log file, extract the file ID, lookup the id in sql server, determine quarantine location, extract q/d pair from quarantine and send to user. -- Best regards, Davidmailto:[EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
