[sniffer] Re: RulePanic on 3741490
On 1/7/2011 10:19 AM, Darin Cox wrote: Hi guys, We're seeing a lot of FPs on 3741490 this morning. I've added a RulePanic for it in our systems. The rule was created at 0539 and removed at 0853 when it was detected by our early warning system. It codes for a binary segment found in some image files. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: RulePanic on 3741490
Hmmm... so 70 minutes after the rule was released we were notified of the rule update for auto-update of rulebase, but at 10:11ET we still hadn't gotten the update for the 8:53am removal. Anything we can do to speed up the rulebase update notifications? Also, for rules identified as problematic and removed, what about an automated email so we can remove it immediately via RulePanic. For peak times like beginning of the business day, that would be very helpful. An hour could save a lot of headaches for both us and our customers. Or are there so many of those that we would be swamped with notifications? Just trying to figure out a way to avoid this as much as possible in the future. It cost me a half hour this morning, and, more importantly, delayed over 150 legitimate messages to our customers. Thanks in advance for anything you can do. Darin. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Friday, January 07, 2011 11:27 AM Subject: [sniffer] Re: RulePanic on 3741490 On 1/7/2011 10:19 AM, Darin Cox wrote: Hi guys, We're seeing a lot of FPs on 3741490 this morning. I've added a RulePanic for it in our systems. The rule was created at 0539 and removed at 0853 when it was detected by our early warning system. It codes for a binary segment found in some image files. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: RulePanic on 3741490
On 1/7/2011 12:33 PM, Darin Cox wrote: Hmmm... so 70 minutes after the rule was released we were notified of the rule update for auto-update of rulebase, but at 10:11ET we still hadn't gotten the update for the 8:53am removal. Anything we can do to speed up the rulebase update notifications? Update notifications happen as soon as the rulebase compilers have created a new rulebase. We are in the process of reworking our compiler cluster to improve it's performance and further shorten update times. Also, for rules identified as problematic and removed, what about an automated email so we can remove it immediately via RulePanic. For peak times like beginning of the business day, that would be very helpful. An hour could save a lot of headaches for both us and our customers. Or are there so many of those that we would be swamped with notifications? We have features on the short list to automatically render removed rules inert in near real-time (within seconds). Just trying to figure out a way to avoid this as much as possible in the future. It cost me a half hour this morning, and, more importantly, delayed over 150 legitimate messages to our customers. We are constantly improving our process to minimize these cases, increase the speed with which we can detect and correct these, and add features to automate and expedite the process. Thanks in advance for anything you can do. Thanks very much for your feedback! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: RulePanic on 3741490
H Update notifications happen as soon as the rulebase compilers have created a new rulebase. I don't know what your internal processes are, but if I understand this correctly the rule was created at 5:39am ET, and was compiled into the rulebase somewhere just before 8:53am ET, at which point update notifications were sent. From the customer point of view, when the rule was created or removed doesn't really matter, and those times are meaningless to us. What matters is when the rulebases that include them are published/updated, as that is what we key off of for updates. We have features on the short list to automatically render removed rules inert in near real-time (within seconds) Sounds good. That would definitely be better than notifications for us to be able to put in RulePanics, assuming there's no negative effect to overall performance from checking each rule for active/inactive state. I assume some sort of push mechanism to all subscribers, to notify their systems that a rule is no longer valid, is what you're planning here. Best. Darin. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Friday, January 07, 2011 1:43 PM Subject: [sniffer] Re: RulePanic on 3741490 On 1/7/2011 12:33 PM, Darin Cox wrote: Hmmm... so 70 minutes after the rule was released we were notified of the rule update for auto-update of rulebase, but at 10:11ET we still hadn't gotten the update for the 8:53am removal. Anything we can do to speed up the rulebase update notifications? Update notifications happen as soon as the rulebase compilers have created a new rulebase. We are in the process of reworking our compiler cluster to improve it's performance and further shorten update times. Also, for rules identified as problematic and removed, what about an automated email so we can remove it immediately via RulePanic. For peak times like beginning of the business day, that would be very helpful. An hour could save a lot of headaches for both us and our customers. Or are there so many of those that we would be swamped with notifications? We have features on the short list to automatically render removed rules inert in near real-time (within seconds). Just trying to figure out a way to avoid this as much as possible in the future. It cost me a half hour this morning, and, more importantly, delayed over 150 legitimate messages to our customers. We are constantly improving our process to minimize these cases, increase the speed with which we can detect and correct these, and add features to automate and expedite the process. Thanks in advance for anything you can do. Thanks very much for your feedback! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: RulePanic on 3741490
On 1/7/2011 2:00 PM, Darin Cox wrote: H "Update notifications happen as soon as the rulebase compilers have created a new rulebase." I don't know what your internal processes are, butif I understand this correctlythe rule was created at 5:39am ET, and was compiled into the rulebase somewhere just before 8:53am ET, at which point update notifications were sent. From the customer point of view, when the rule was created or removed doesn't really matter,and those times are meaningless to us. What matters is when the rulebases that include them are published/updated, as that is what we key off of for updates. The rulebase compiler that was responsible for your update would have performed it's queries to collect rules for folding. For some time after that the compilers would crunch on that data in order to create the folded token matrix. Both the query and folding operations take time. There are many compilers and many, many rulebase files. The system is adaptive so the times required to perform these operations changes constantly. All that by way of saying that you are essentially correct, but also that there is no single answer we can give that would describe the case for every customer. The rule was removed at about 0853. Some customers rulebase compilation began immediately after that event, others before. All took some time to complete. Part of our rework is to reduce the time required for both phases of this operation so that on average less time is required for a change to appear in active SNF systems. The goal is ultimately to make that as close to instantaneous as possible without incurring performance, availability, scalability, and reliability penalties. "We have features on the short list to automatically render removed rules inert in near real-time (within seconds)" Sounds good. That would definitely be better than notifications for us to be able to put in RulePanics, assuming there's no negative effect to overall performance from checking each rule for active/inactive state. I assume some sort of push mechanism to all subscribers, to notify their systems that a rule is no longer valid,is what you're planning here. The implementation that is currently planned would operate something like our system making temporary entries into your rule-panic system. Those entries would survive long enough to ensure that your active rulebase contains those changes before the entries expire. Wherever possible we like to leverage tried and true mechanisms (like the rule-panic entry system) when producing new functionality. Also, wherever possible we like to engineer facilities that can be leveraged in multiple ways in future. It's a planning heavy process, but one that pays off in better reliability and greater overall flexibility. (IMO). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
