Re: [ClusterLabs] Oracle resource agent failure

[Muhammad Sharfuddin]

On 04/05/2017 09:53 AM, SAYED, MAJID ALI SYED AMJAD ALI wrote:


Hello,

We are trying to build a 2 node Active/Passive  Linux HA cluster using 
pacemaker and corosync on RHEL 7.2 using shared storage (SAN) that 
will be used for oracle database


We have build LVM resource, Filesystem resource and Virtual IP.

The database administrator has build  couple of  oracle db and oracle 
listener resources.


All these resources are in group

Resource Group: oraclegrp

 vgres  (ocf::heartbeat:LVM): Started krplporacle001

 Cluster_ip (ocf::heartbeat:IPaddr2):   Started krplporacle001

 u02 (ocf::heartbeat:Filesystem):Started krplporacle001

 u03 (ocf::heartbeat:Filesystem):Started krplporacle001

 u04 (ocf::heartbeat:Filesystem):Started krplporacle001

 ls_testdbc (ocf::heartbeat:oralsnr):   Started krplporacle001

 testdbc (ocf::heartbeat:oracle):Stopped

 ls_samtest (ocf::heartbeat:oralsnr):   Stopped

 samtest (ocf::heartbeat:oracle):Stopped

For  testing purpose he has purposely corrupted testdbc oracle 
resource and transferred the resource to passive node.


But  we are not sure if one oracle resource in group why the other is 
not starting after transferring the service


Any help would be much appreciated

The logs only specify that oracle resource for testdbc is not running 
, but it does not even attempt to start samtest?


resources in the resource group started in the sequence, so in your case 
if testdbc is stopped(for any reason) cluster wouldn't even attempt to 
start the "ls_samtest" and "samtest", because they are dependent upon 
successful startup of 'testdbc'.


Hope this helps.

--
Regards,

Muhammad Sharfuddin
Cell: +92-3332144823 | UAN: +92(21) 111-111-142 ext: 112 | NDS.COM.PK 

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Oracle resource agent failure

[SAYED, MAJID ALI SYED AMJAD ALI]

Hello,
We are trying to build a 2 node Active/Passive  Linux HA cluster using 
pacemaker and corosync on RHEL 7.2 using shared storage (SAN) that will be used 
for oracle database
We have build LVM resource, Filesystem resource and Virtual IP.
The database administrator has build  couple of  oracle db and oracle listener 
resources.
All these resources are in group

Resource Group: oraclegrp
 vgres  (ocf::heartbeat:LVM):   Started krplporacle001
 Cluster_ip (ocf::heartbeat:IPaddr2):   Started krplporacle001
 u02(ocf::heartbeat:Filesystem):Started krplporacle001
 u03(ocf::heartbeat:Filesystem):Started krplporacle001
 u04(ocf::heartbeat:Filesystem):Started krplporacle001
 ls_testdbc (ocf::heartbeat:oralsnr):   Started krplporacle001
 testdbc(ocf::heartbeat:oracle):Stopped
 ls_samtest (ocf::heartbeat:oralsnr):   Stopped
 samtest(ocf::heartbeat:oracle):Stopped

For  testing purpose he has purposely corrupted testdbc oracle resource and 
transferred the resource to passive node.
But  we are not sure if one oracle resource in group why the other is not 
starting after transferring the service
Any help would be much appreciated
The logs only specify that oracle resource for testdbc is not running , but it 
does not even attempt to start samtest?





MAJID SAYED
HPC System Administrator.
King Abdullah International Medical Research Centre
Phone:+9661801(Ext:40631)
Email:sayed...@ngha.med.sa



This Email and any files transmitted may contain confidential and/or privileged 
information and is intended solely for the addressee(s) named. If you have 
received this information in error, or are being posted by accident, please 
notify the sender by return Email, do not redistribute this email message, 
delete it immediately and keep no copies of it. All opinions and/or views 
expressed in this email are solely those of the author and do not necessarily 
represent those of NGHA. Any purchase order, purchase advice or legal 
commitment is only valid once backed by the signed hardcopy by the authorized 
person from NGHA.
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] STONITH not communicated back to initiator until token expires

[Ken Gaillot]

On 03/13/2017 10:43 PM, Chris Walker wrote:
> Thanks for your reply Digimer.
> 
> On Mon, Mar 13, 2017 at 1:35 PM, Digimer  > wrote:
> 
> On 13/03/17 12:07 PM, Chris Walker wrote:
> > Hello,
> >
> > On our two-node EL7 cluster (pacemaker: 1.1.15-11.el7_3.4; corosync:
> > 2.4.0-4; libqb: 1.0-1),
> > it looks like successful STONITH operations are not communicated from
> > stonith-ng back to theinitiator (in this case, crmd) until the STONITHed
> > node is removed from the cluster when
> > Corosync notices that it's gone (i.e., after the token timeout).
> 
> Others might have more useful info, but my understanding of a lost node
> sequence is this;
> 
> 1. Node stops responding, corosync declares it lost after token timeout
> 2. Corosync reforms the cluster with remaining node(s), checks if it is
> quorate (always true in 2-node)
> 3. Corosync informs Pacemaker of the membership change.
> 4. Pacemaker invokes stonith, waits for the fence agent to return
> "success" (exit code of the agent as per the FenceAgentAPI
> [https://docs.pagure.org/ClusterLabs.fence-agents/FenceAgentAPI.md]
> ).
> If
> the method fails, it moves on to the next method. If all methods fail,
> it goes back to the first method and tries again, looping indefinitely.
> 
> 
> That's roughly my understanding as well for the case when a node
> suddenly leaves the cluster (e.g., poweroff), and this case is working
> as expected for me.  I'm seeing delays when a node is marked for STONITH
> while it's still up (e.g., after a stop operation fails).  In this case,
> what I expect to see is something like:
> 1.  crmd requests that stonith-ng fence the node
> 2.  stonith-ng (might be a different stonith-ng) fences the node and
> sends a message that it has succeeded
> 3.  stonith-ng (the original from step 1) receives this message and
> communicates back to crmd that the node has been fenced
> 
> but what I'm seeing is
> 1.  crmd requests that stonith-ng fence the node
> 2.  stonith-ng fences the node and sends a message saying that it has
> succeeded
> 3.  nobody hears this message
> 4.  Corosync eventually realizes that the fenced node is no longer part
> of the config and broadcasts a config change
> 5.  stonith-ng finishes the STONITH operation that was started earlier
> and communicates back to crmd that the node has been STONITHed

In your attached log, bug1 was DC at the time of the fencing, and bug0
takes over DC after the fencing. This is what I expect is happening
(logs from bug1 would help confirm):

1. crmd on the DC (bug1) runs pengine which sees the stop failure and
schedules fencing (of bug1)

2. stonithd on bug1 sends a query to all nodes asking who can fence bug1

3. Each node replies, and stonithd on bug1 chooses bug0 to execute the
fencing

4. stonithd on bug0 fences bug1. At this point, it would normally report
the result to the DC ... but that happens to be bug1.

5. Once crmd on bug0 takes over DC, it can decide that the fencing
succeeded, but it can't take over DC until it sees that the old DC is
gone, which takes a while because of your long token timeout. So, this
is where the delay is coming in.

I'll have to think about whether we can improve this, but I don't think
it would be easy. There are complications if for example a fencing
topology is used, such that the result being reported in step 4 might
not be the entire result.

> I'm less convinced that the sending of the STONITH notify in step 2 is
> at fault; it also seems possible that a callback is not being run when
> it should be.
> 
> 
> The Pacemaker configuration is not important (I've seen this happen on
> our production clusters and on a small sandbox), but the config is:
> 
> primitive bug0-stonith stonith:fence_ipmilan \
> params pcmk_host_list=bug0 ipaddr=bug0-ipmi action=off
> login=admin passwd=admin \
> meta target-role=Started
> primitive bug1-stonith stonith:fence_ipmilan \
> params pcmk_host_list=bug1 ipaddr=bug1-ipmi action=off
> login=admin passwd=admin \
> meta target-role=Started
> primitive prm-snmp-heartbeat snmptrap_heartbeat \
> params snmphost=bug0 community=public \
> op monitor interval=10 timeout=300 \
> op start timeout=300 interval=0 \
> op stop timeout=300 interval=0
> clone cln-snmp-heartbeat prm-snmp-heartbeat \
> meta interleave=true globally-unique=false ordered=false
> notify=false
> location bug0-stonith-loc bug0-stonith -inf: bug0
> location bug1-stonith-loc bug1-stonith -inf: bug1
> 
> The corosync config might be more interesting:
> 
> totem {
> version: 2
> crypto_cipher: none
> crypto_hash: none
> secauth: off
> rrp_mode: passive
> transport: udpu
> token: 24
> consensus: 1000
> 
> interface {
> ringnumber 0
>  

Re: [ClusterLabs] to change resource id - how?

[Dejan Muhamedagic]

On Mon, Apr 03, 2017 at 09:26:04AM -0500, Ken Gaillot wrote:
> On 04/03/2017 06:35 AM, lejeczek wrote:
> > hi
> > I'm sroogling and reading but cannot find any info - how to
> > (programmatically) change resources ids? In other words: how to rename
> > these entities?
> > many thanks
> > L
> 
> As far as I know, higher-level tools don't support this directly -- you

At least crmsh does. There's "crm cfg rename".

Thanks,

Dejan

> have to edit the XML. The basic process is:
> 
> 1. Save a copy of the live CIB to a file.
> 2. Edit that file, and do a search-and-replace on the desired name (so
> you change it in constraints, etc., as well as the resource definition).
> 3. Push the configuration section of that file to the live CIB.
> 
> The higher-level tools have commands to do that, but at the low level,
> it would be something like:
> 
> 1. cibadmin -Q --scope configuration > tmp.cib
> 2. vim tmp.cib
> 3. cibadmin -x tmp.cib --replace --scope configuration
> 
> The cluster will treat it as a completely new resource, so it will stop
> the old one and start the new one.
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Coming in Pacemaker 1.1.17: Per-operation fail counts

[Ken Gaillot]

On 04/04/2017 01:18 AM, Ulrich Windl wrote:
 Ken Gaillot  schrieb am 03.04.2017 um 17:00 in 
 Nachricht
> :
>> Hi all,
>>
>> Pacemaker 1.1.17 will have a significant change in how it tracks
>> resource failures, though the change will be mostly invisible to users.
>>
>> Previously, Pacemaker tracked a single count of failures per resource --
>> for example, start failures and monitor failures for a given resource
>> were added together.
> 
> That is "per resource operation", not "per resource" ;-)

I mean that there was only a single number to count failures for a given
resource; before this change, failures were not remembered separately by
operation.

>> In a thread on this list last year[1], we discussed adding some new
>> failure handling options that would require tracking failures for each
>> operation type.
> 
> So the existing set of operations failures was restricted to 
> start/stop/monitor? How about master/slave featuring two monitor operations?

No, both previously and with the new changes, all operation failures are
counted (well, except metadata!). The only change is whether they are
remembered per resource or per operation.

>> Pacemaker 1.1.17 will include this tracking, in preparation for adding
>> the new options in a future release.
>>
>> Whereas previously, failure counts were stored in node attributes like
>> "fail-count-myrsc", they will now be stored in multiple node attributes
>> like "fail-count-myrsc#start_0" and "fail-count-myrsc#monitor_1"
>> (the number distinguishes monitors with different intervals).
> 
> Wouldn't it be thinkable to store is as (transient) resource attribute, 
> either local to a node (LRM) or including the node attribute (CRM)?

Failures are specific to the node the failure occurred on, so it makes
sense to store them as transient node attributes.

So, to be more precise, we previously recorded failures per
node+resource combination, and now we record them per
node+resource+operation+interval combination.

>> Actual cluster behavior will be unchanged in this release (and
>> backward-compatible); the cluster will sum the per-operation fail counts
>> when checking against options such as migration-threshold.
>>
>> The part that will be visible to the user in this release is that the
>> crm_failcount and crm_resource --cleanup tools will now be able to
>> handle individual per-operation fail counts if desired, though by
>> default they will still affect the total fail count for the resource.
> 
> Another thing to think about would be "fail count" vs. "fail rate": Currently 
> there is a fail count, and some reset interval, which allows to build some 
> failure rate from it. Maybe many users just have the requirement that some 
> resource shouldn't fail again and again, but with long uptimes (and then the 
> operatior forgets to reset fail counters), occasional failures (like once in 
> two weeks) shouldn't prevent a resource from running.

Yes, we discussed that a bit in the earlier thread. It would be too much
of an incompatible change and add considerable complexity to start
tracking the failure rate.

Failure clearing hasn't changed -- failures can only be cleared by
manual commands, the failure-timeout option, or a restart of cluster
services on a node.

For the example you mentioned, a high failure-timeout is the best answer
we have. You could set a failure-timeout of 24 hours, and if the
resource went 24 hours without any failures, any older failures would be
forgotten.

>> As an example, if "myrsc" has one start failure and one monitor failure,
>> "crm_failcount -r myrsc --query" will still show 2, but now you can also
>> say "crm_failcount -r myrsc --query --operation start" which will show 1.
> 
> Would accumulated monitor failures ever prevent a resource from starting, or 
> will it force a stop of the resource?

As of this release, failure recovery behavior has not changed. All
operation failures are added together to produce a single fail count per
resource, as was recorded before. The only thing that changed is how
they're recorded.

Failure recovery is controlled by the resource's migration-threshold and
the operation's on-fail. By default, on-fail=restart and
migration-threshold=INFINITY, so a monitor failure would result in
1,000,000 restarts before being banned from the failing node.

> Regards,
> Ulrich
> 
>>
>> Additionally, crm_failcount --delete previously only reset the
>> resource's fail count, but it now behaves identically to crm_resource
>> --cleanup (resetting the fail count and clearing the failure history).
>>
>> Special note for pgsql users: Older versions of common pgsql resource
>> agents relied on a behavior of crm_failcount that is now rejected. While
>> the impact is limited, users are recommended to make sure they have the
>> latest version of their pgsql resource agent before upgrading to
>> pacemaker 1.1.17.
>>
>> [1] 

Re: [ClusterLabs] Antw: Re: to change resource id - how?

[Les Green]

If you put the cluster in maintenance mode first, then restart with the
new config, resources should stay up right?

On 04.04.2017 08:07, Ulrich Windl wrote:
 Ken Gaillot  schrieb am 03.04.2017 um 16:26 in 
 Nachricht
> :
>> On 04/03/2017 06:35 AM, lejeczek wrote:
>>> hi
>>> I'm sroogling and reading but cannot find any info - how to
>>> (programmatically) change resources ids? In other words: how to rename
>>> these entities?
>>> many thanks
>>> L
>>
>> As far as I know, higher-level tools don't support this directly -- you
>> have to edit the XML. The basic process is:
>>
>> 1. Save a copy of the live CIB to a file.
>> 2. Edit that file, and do a search-and-replace on the desired name (so
>> you change it in constraints, etc., as well as the resource definition).
>> 3. Push the configuration section of that file to the live CIB.
>>
>> The higher-level tools have commands to do that, but at the low level,
>> it would be something like:
>>
>> 1. cibadmin -Q --scope configuration > tmp.cib
>> 2. vim tmp.cib
>> 3. cibadmin -x tmp.cib --replace --scope configuration
>>
>> The cluster will treat it as a completely new resource, so it will stop
>> the old one and start the new one.
> 
> In my experience it's very much advisable to stop any resource that is to be 
> renamed _before_ doing so. Maybe the software became smarter since I suffered 
> from problems, but it seems wise anyway...
> 
> Regards,
> Ulrich
> 
>>
>> ___
>> Users mailing list: Users@clusterlabs.org 
>> http://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> Project Home: http://www.clusterlabs.org 
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>> Bugs: http://bugs.clusterlabs.org 
> 
> 
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Antw: Coming in Pacemaker 1.1.17: Per-operation fail counts

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 03.04.2017 um 17:00 in 
>>> Nachricht
:
> Hi all,
> 
> Pacemaker 1.1.17 will have a significant change in how it tracks
> resource failures, though the change will be mostly invisible to users.
> 
> Previously, Pacemaker tracked a single count of failures per resource --
> for example, start failures and monitor failures for a given resource
> were added together.

That is "per resource operation", not "per resource" ;-)

> 
> In a thread on this list last year[1], we discussed adding some new
> failure handling options that would require tracking failures for each
> operation type.

So the existing set of operations failures was restricted to 
start/stop/monitor? How about master/slave featuring two monitor operations?

> 
> Pacemaker 1.1.17 will include this tracking, in preparation for adding
> the new options in a future release.
> 
> Whereas previously, failure counts were stored in node attributes like
> "fail-count-myrsc", they will now be stored in multiple node attributes
> like "fail-count-myrsc#start_0" and "fail-count-myrsc#monitor_1"
> (the number distinguishes monitors with different intervals).

Wouldn't it be thinkable to store is as (transient) resource attribute, either 
local to a node (LRM) or including the node attribute (CRM)?

> 
> Actual cluster behavior will be unchanged in this release (and
> backward-compatible); the cluster will sum the per-operation fail counts
> when checking against options such as migration-threshold.
> 
> The part that will be visible to the user in this release is that the
> crm_failcount and crm_resource --cleanup tools will now be able to
> handle individual per-operation fail counts if desired, though by
> default they will still affect the total fail count for the resource.

Another thing to think about would be "fail count" vs. "fail rate": Currently 
there is a fail count, and some reset interval, which allows to build some 
failure rate from it. Maybe many users just have the requirement that some 
resource shouldn't fail again and again, but with long uptimes (and then the 
operatior forgets to reset fail counters), occasional failures (like once in 
two weeks) shouldn't prevent a resource from running.

> 
> As an example, if "myrsc" has one start failure and one monitor failure,
> "crm_failcount -r myrsc --query" will still show 2, but now you can also
> say "crm_failcount -r myrsc --query --operation start" which will show 1.

Would accumulated monitor failures ever prevent a resource from starting, or 
will it force a stop of the resource?

Regards,
Ulrich

> 
> Additionally, crm_failcount --delete previously only reset the
> resource's fail count, but it now behaves identically to crm_resource
> --cleanup (resetting the fail count and clearing the failure history).
> 
> Special note for pgsql users: Older versions of common pgsql resource
> agents relied on a behavior of crm_failcount that is now rejected. While
> the impact is limited, users are recommended to make sure they have the
> latest version of their pgsql resource agent before upgrading to
> pacemaker 1.1.17.
> 
> [1] http://lists.clusterlabs.org/pipermail/users/2016-September/004096.html 
> -- 
> Ken Gaillot 
> 
> ___
> Users mailing list: Users@clusterlabs.org 
> http://lists.clusterlabs.org/mailman/listinfo/users 
> 
> Project Home: http://www.clusterlabs.org 
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
> Bugs: http://bugs.clusterlabs.org 





___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org