On Tue, Oct 30, 2012 at 8:33 PM, Alan Robertson <[email protected]> wrote:
I'm not sure I know what kinds of things would qualify for this
"extremely hard to recover from" criteria. I must have missed something
- brain broken - something ;-). Could you kindly help me out here?
Bear in mind this is open source, so idiots can contribute ill thought
out features as well as bugs.
I have some experience with that. I've learned a little from it - but
I'm hardly infallible ;-)
Examples of "harder than the original install" to upgrade out of:
* Network traffic O(hosts^2) due to accidental full mesh reporting,
* Memory footprint O(hosts log network-hops) due to over zealous ECMP,
* Alert flood due to conditions never being able to pass as working,
* Priority inversion, lack of backoff, when eg localhost is icmp unreachable.
OK. This is good to keep in mind. I could put some limits into the
system that would cause it to refuse new heartbeats past a certain
number, or would have it complain or restart if memory usage climbed
past - let's say double the initial hour's memory usage. I should
probably plan on a graceful self-restart capability.
I planned on having an audit of the heartbeat topology in the automated
tests, but wasn't planning on it for the manual tests.
There are some cases where I should have some backoff of alerts - in
particular for receiving heartbeats from unknown machines.
I have seen two or three scheduler bugs where my code has not gotten any
CPU for a while - several minutes. I'm not sure how localhost can
become unreachable, but I understand about backing off on alerts - but
not sure it applies in the case of heartbeats, etc.
*
**More details would be helpful to make sure I'm getting the message
through my skull...*
Just to calibrate expectations...
This first release will be pretty modest. I've called it the "toy
release". Most of these will likely go into a set of features for
future releases. But the ideas of backoff and maximal resource usage,
and planning automated restarts in "oops" cases seem worth planning on.
There is a second class of problems if you assume that a user has an
old fleet management system and, after Assim has proven itself,
decided to stop running two systems and turned down the old one. It
then becomes embarrassing if the fleet reporting functions of Assim
are (ahem) inaccurate after an upgrade. Partly because it becomes
difficult to _find_ the machines that need to be upgraded with the
repaired codebase.
Hopefully, our discovery features will allow us to find any machines we
lose track of. I haven't implemented the full ARP listening agent - but
I will...
Also, if machines keep heartbeating their neighbors and aren't known to
the system, we'll get these "martian" errors I know I need to throttle
down. Obviously not to zero, but still, reporting them once an hour is
enough...
This is a general problem - and that seems to just imply better
testing. Did you have something specific in mind?
Thanks again for your time and thoughtful reply!
-- Alan Robertson
[email protected]
_______________________________________________
Assimilation mailing list - Discovery-Driven Monitoring
[email protected]
http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation
http://assimmon.org/
