19.03.2011 19:10, Dan Frincu:
Even if that is set, we need to verify that the resources are,
indeed,
NOT running where they shouldn't be; remember, it is our job
to ensure
that the configured policy is enforced. So, we probe them
everywhere to
ensure they are indeed not around, and stop them if we find them.
Again, WHY do you need to verify things which cannot happen by
setup? If some resource cannot, REALLY CANNOT exist on a node, and
administrator can confirm this, why rely on network, cluster
stack, resource agents, electricity in power outlet, etc. to
verify that 2+2 is still 4?
Don't want to step on any toes or anything, mainly because me stepping
on somebody's toes without the person wearing a pair of steel-toe cap
boots would leave them toeless, but I've been hearing the ranting go
on and on and just felt like maybe something's missing from the
picture, specifically, an example for why checking for resources on
passive nodes is a good thing, which I haven't seen thus far.
...
Ok, so far it sounds perfect, but what happens if on the
secondary/passive node, someone starts the service, by user error, by
upgrading the software and thus activating its automatic startup at
the given runlevel and restarting the secondary node (common practice
when performing upgrades in a cluster environment), etc. If Pacemaker
were not to check all the nodes for the service being active or not =>
epic fail. Its state-based model, where it maintains a state of the
resources and performs the necessary actions to bring the cluster to
that state is what saves us from the "epic fail" moment.
Surely you are right. Resources must be monitored on standby nodes to
prevent such a scenario. You can screw your setup by many other ways,
howewer. And pacemaker (1.0.10, at least) does not execute recurring
monitor on passive node, so you may start your service by hands, and it
will be unnoticed for quite some time.
What I am talking about is monitoring (probing) of a resource on a node
where this resource cannot be exist. For example, if you have five nodes
in your cluster and a DRBD resource, which can, by it's nature, work on
no more than two nodes. Then, other three of your nodes will be
occasionally probed for that resource. If that action fails, the
resource will be restarted everywhere. If that node cannot be fenced,
the resource will be dead.
There is still at least one case when such a failure may happen even if
RA is perfect: misbehaving or highly overloaded node may cause RA
timeout. And bugs or configuration errors may, of course.
A resource should not depend on unrelated things, such as nodes which
have no connections to the resource. Then the resource will be more stable.
I'm trying to be impartial here, although I may be biased by my
experience to rule in favor of Pacemaker, but here's a thought, it's a
free world, we all have the freedom of speech, which I'm also
exercising at the moment, want something done, do it yourself, patches
are being accepted, don't have the time, ask people for their help, in
a polite manner, wait for them to reply, kindly ask them again (and
prayers are heard, Steven Dake released >>
http://www.mail-archive.com/openais@lists.linux-foundation.org/msg06072.html
<< a patch for automatic redundant ring recovery, thank you Steven),
want something done fast, pay some developers to do it for you, say
the folks over at www.linbit.com <http://www.linbit.com> wouldn't mind
some sponsorship (and I'm not affiliated with them in any way, believe
it or not, I'm actually doing this without external incentives, from
the kindness of my heart so to speak).
My goal for now is to make the problem clear to the team. It is doubtful
that such a patch will be accepted without that, given current reaction.
Moreover, it is not clear how to fix the problem to the best advantage.
This cluster stack is brilliant. It's a pity to see how it fails to keep
a resource running while it is relatively simple to avoid unneeded downtime.
Thank you for participating.
P.S. There is a crude workaround: op monitor interval="0" timeout="10"
on_fail="nothing". Obvoiusly, it has own deficiencies.
--
Pavel Levshin
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
