On 05/13/2016 06:00 AM, Ulrich Windl wrote: >>>> Dejan Muhamedagic <deja...@fastmail.fm> schrieb am 13.05.2016 um 12:16 in > Nachricht <20160513101626.GA12493@walrus.homenet>: >> Hi, >> >> On Fri, May 13, 2016 at 09:05:54AM +0200, Ulrich Windl wrote: >>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 12.05.2016 um 16:41 in >>>>>> Nachricht >>> <57349629.40...@redhat.com>: >>>> On 05/12/2016 02:56 AM, Ulrich Windl wrote: >>>>> Hi! >>>>> >>>>> I have a question regarding an RA written by myself and pacemaker >>>> 1.1.12-f47ea56 (SLES11 SP4): >>>>> >>>>> During "probe" all resources' "monitor" actions are executed (regardless >>>>> of >> >>>> any ordering constraints). Therefore my RA considers a parameter as >>>> invalid >> >>>> ("file does not exist") (the file will be provided once some supplying >>>> resource is up) and returns rc=2. >>>>> OK, this may not be optimal, but pacemaker makes it worse: It does not >>>> repeat the probe once the resource would start, but keeps the state, >>>> preventing a resource start: >>>>> >>>>> primitive_monitor_0 on h05 'invalid parameter' (2): call=73, >>>> status=complete, exit-reason='none', last-rc-change='Wed May 11 17:03:39 >> 2016', >>>> queued=0ms, exec=82ms >>>> >>>> Correct, OCF_ERR_CONFIGURED is a "fatal" error: >>>> >>>> >> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explai >> >> >>>> ned/index.html#_how_are_ocf_return_codes_interpreted >>> >>> I think the mistake here is assuming that OCF_ERR_CONFIGURED only depends >>> on >> the RA parameters, when in fact the validity of RA params may depend on the >> environment found at the time of checking. And as we all now the environment >> changes, especially when resources are started and stopped.
It's not a mistake; it's the definition of OCF_ERR_CONFIGURED. The various documentation could be more clear, but the intent is: the resource's configuration *in the cluster configuration* is inherently invalid, and could never work, regardless of what system it was on. Examples listed include a required parameter not being specified, or a nonnumeric value supplied for an integer parameter. OCF_ERR_CONFIGURED is the *only* fatal error (fatal = the operation will not be retried anywhere). Of course, as you mentioned later, here you actually have OCF_ERR_ARGS (2) -- I saw "invalid parameter" and mistakenly thought OCF_ERR_CONFIGURED ... >>>>> So you would say that monitor may only return "success" or "not running", >>>> but I feel the RA should detect the condition that the resource could not >> run >>>> at all at the present state. >>>> >>>> OCF_ERR_CONFIGURED is meant to indicate that the resource could not >>>> possibly run *as configured*, regardless of the system's current state. >>> >>> But how do you handle parameters that describe file or host names (which >>> may >> exist or not independently of a change in the param's value)? >> >> The RA should've exited with OCF_ERR_INSTALLED. That's the code >> which means that there's something wrong with the environment on >> this node, but that the resource could be started on another one. > > Really, besides implementation, I don't see why OCF_ERR_INSTALLED is less > permanent than OCF_ERR_ARGS. > >> >>>> So for example, a required parameter is missing or invalid. >>>> >>>> You could possibly use OCF_ERR_ARGS in this case (a "hard" error that >>>> bans the particular node, and means that the resource's configuration is >>>> not valid on this particular node). >>> >>> ("rc=2" _is_ OCF_ERR_ARGS) OCF_ERR_ARGS and OCF_ERR_INSTALLED are both hard errors (hard = the operation will not be retried on this node). Looking at the code, OCF_ERR_INSTALLED is treated differently for probes (it's not recorded as a failed op), but I think the node would still get banned from running the resource. I totally understand your point now: The probe may hit one of these conditions only because it is called before depended-on resources are up, but the cluster doesn't really care -- it just wants to know whether the resource is running. Using ocf_is_probe to mask these errors works, but could be considered a less-than-ideal workaround. Now, it rings a bell: Last year, we looked into implementing "ordered probes", which is essentially what you originally suggested, regarding re-probing. See order_probes() in pengine/allocate.c; basically, there are difficult implementation issues that haven't been figured out. Bottom line, using ocf_is_probe is the best approach currently. >>>> But, I suspect the right answer here is simply an order constraint >>>> between the supplying resource and this resource. This resource's start >>> >>> As I said before the problem is probes: They are all started immediately >> when a node comes up. And if a probe fails, no start is ever attempted later. >>> AFAIK probes ignore any colocation or ordering constraints. >> >> Try with ocf_is_probe. On probes, if the file's missing, exit >> with OCF_NOT_RUNNING. > > I did that in the meantime, but I consider this as a work-around. > Spontaneously I feel each RA should have a separate "probe" action with the > following semantics: > IF parameters seem valid AND the resource is up RETURN success > ELSE return not running. > Specifically this means any invalid parameter will result in "NOT RUNNING". > > I'm not saying having a separate "probe" will make things better; it's just > clear that probes may be called from "invalid context" (I name it). > >> >>>> action, not monitor, should be the one that checks for the existence of >>>> the supplied file. >>> >>> So it all depends what an invalid parameter (in the sense of validate-all) >> actually is. Maybe the documentation should be more clear abou that. >>> (As a matter of fact, the only RA docs I found yesterday are still hosted >>> at >> linux-ha (from linbit)) >> >> The developer's guide does have a note about probes and >> ocf_is_probe and even some sample code (see the monitor and >> validate-all actions). >> >>>>> Shouldn't pacemaker reprobe resources before it tries to start them? >>>> >>>> Probes are meant to check whether the resource is already active >>>> anywhere. The decision of whether and where to start the resource takes >>>> into account the result of the probes, so it doesn't make sense to >>>> re-probe -- that's what the initial probe was for. >>> >>> In case a _probe_ returned an error indicating it was not able to decide, >>> it >> would make sense to reprobe after colocation and ordering constraiints are >> fulfilled. >>> >>> One of the first lessions I've learned with HA was that the state of a >> resource is not Boolean (up or down), but multistate (I wrote on that after >> I >> had started to use pacemaker): >>> A resource can be "down", "starting", "started/up", "stopping", >> "down/stopped" and "undecided" (the case whare retries can make sense). >> During states "starting" and "stopping" it does not make sense to enforce a >> Boolean result like "up" or "down", because the resource may fail to perform >> the transition, and the resource may hang in the state of "starting" or >> "stopping" (wich both are definitely not "up"). >>> If you need a specific directory to decide whether a specific resource is >>> up >> or down, and that directory does not exist, you cannot really make a >> statement about the resource's state: Most likely it won't be "up", but you >> cannot say for sure that it is down (e.g. you have a state file on NFS and >> NFS server is down at the time of checking). >>> >> >> The transitioning period (i.e. start, stop, promote, demote) for >> the resource may take a while, depending on the resource and the >> actual deployment, which is why setting the timeouts properly is >> important. Note that probes are meant to run mostly on node >> startup, that is to establish the initial state. >> >> If the RA, on probe, cannot decide the state the resource is in, >> then it should return OCF_RA_GENERIC and let the resource manager >> cleanup (by invoking stop). Furthermore, the RA must take into >> account that some of its dependencies may not be running. > > But if the problem is due to "invalid calling context", then action "stop" > will have the same problem (and return an error as well). STONITH deathmatch > then? > > Regards, > Ulrich > [...] > > > _______________________________________________ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > _______________________________________________ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org