Darin Cox wrote:
We had a lot... 534 hits between 3:26 and 4:41pm ET, which is when we
added the rule panic. It appears the rule was added in a rulebase
that was automatically updated at 3:26pm ET.
Pete? Status?
Here is a preliminary report on the bad rule:
The rule was coded in error -- the highlighted section flipped just as
the content was copied into the rule generator. As a result unintended
content was coded. The intended content was a link of similar size and
structure so it was not immediately obvious that an error had occurred.
Storm conditions were high so the error was not noticed immediately by
Adam -- recently coded rules quickly scrolled off screen and out of sight.
A rulebase update when out and the error was caught. The rule was
removed immediately upon detection (actually -- no rules are ever
removed. The bad rule remains in place to prevent it ever being coded
again).
The rule appears to have been in place for one or two updates depending
upon variable update rates and timing.
I note that several of the FP reports actually came in after the rule
had already been removed. There is also evidence that the rule was
autopanic'd on many systems rendering it inert.
---
We are reviewing procedures to minimize this possibility and to improve
upon detection times.
Circumstances were unusual for this event as we were two people down
during a (approximate) 45 minute window at the moment this error
occurred and we were also under heavy storm conditions. This combination
is very rare, but was unavoidable today.
Under normal circumstances It is very likely we would have caught the
error before it went out if the error had occurred at all. Normally
rules are under nearly continuous review be at least one additional pair
of eyes. (Minimum two brains).
We are very sorry about the trouble.
Best,
_M
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[email protected]>.
To unsubscribe, E-mail to: <[email protected]>
To switch to the DIGEST mode, E-mail to <[email protected]>
To switch to the INDEX mode, E-mail to <[email protected]>
Send administrative queries to <[email protected]>
