[ClusterLabs] Antw: ocf:lvm2:VolumeGroup Probe Issue

[Ulrich Windl]

>>> Marc Smith  schrieb am 08.11.2016 um 17:37 in Nachricht
:
> Hi,
> 
> First, I realize ocf:lvm2:VolumeGroup comes from the LVM2 package and
> not resource-agents, but I'm hoping someone on this list is familiar
> with this RA and can provide some insight.
> 
> In my cluster configuration, I'm using ocf:lvm2:VolumeGroup to manage
> my LVM VG's, and I'm using the cluster to manage DLM and CLVM. I have
> my constraints in place and everything seems to be working mostly,
> except I'm hitting a glitch with ocf:lvm2:VolumeGroup and the initial
> probe operation.
> 
> On startup, a probe operation (monitor) is issued for all of the
> resources, but ocf:lvm2:VolumeGroup is returning OCF_ERR_GENERIC in
> VolumeGroup_status() (via VolumeGroup_monitor()) since clvmd hasn't
> started yet... this line in VolumeGroup_status() is the trouble:
> 
> VGOUT=`vgdisplay -v $OCF_RESKEY_volgrpname 2>&1` || exit $OCF_ERR_GENERIC
> 
> When clvmd is not running, 'vgdisplay -v name' will always return
> something like this:

Hi!

We also use cLVM and VGs in SLES11 SP4, but we do not have that problem. 
Independent of that I think that vgsisplay, should simply say that any cVG is 
not active if clvmd is not up and running (vgchange -a y will fail then, of 
course, but it may give an appropriate error message). Maybe for vgdisplay the 
problem is the "-v". Unfortunately vgdisplay dos not have an "Active yes/no" 
status output (like HP-UX had).

I wonder whether a simpler solution is to check for a properly named symbolic 
link in /dev/mapper...

What I see here is easier:
LVM(prm_LVM_CFS_VMs)[4971]: WARNING: LVM Volume CFS_VMs is not available 
(stopped)
But we get a problem from the filesystem probe:
Filesystem(prm_CFS_VMs_fs)[5031]: WARNING: Couldn't find device 
[/dev/CFS_VMs/xen]. Expected /dev/??? to exist

Then start DLM, clvmd, ...

Ulrich

> 
> --snip--
>   connect() failed on local socket: No such file or directory
>   Internal cluster locking initialisation failed.
>   WARNING: Falling back to local file-based locking.
>   Volume Groups with the clustered attribute will be inaccessible.
> VG name on command line not found in list of VGs: biggie
>   Volume group "biggie" not found
>   Cannot process volume group biggie
> --snip--
> 
> And exits with a status of 5. So, my question is, do I patch the RA?
> Or is there some cluster constraint I can add so a probe/monitor
> operation isn't performed for the VolumeGroup resource until CLVM has
> been started?
> 
> Any other advice? Is ocf:heartbeat:LVM or ocf:lvm2:VolumeGroup the
> more popular RA for managing LVM VG's? Any comments from other users
> on experiences using either (good, bad)? Both appear to achieve the
> same function, just a bit differently.
> 
> 
> Thanks,
> 
> Marc
> 
> ___
> Users mailing list: Users@clusterlabs.org 
> http://clusterlabs.org/mailman/listinfo/users 
> 
> Project Home: http://www.clusterlabs.org 
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
> Bugs: http://bugs.clusterlabs.org 





___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Resources wont start on new node unless it is the only active node

[Ryan Anstey]

I had a feeling it was something to do with that. It was confusing because
I could use the move command to move between my three original hosts, just
not the fourth. Then there was the device not found errors which added to
the confusion.

I have the resource stickiness set because I have critical things running
that I don't want to move around (such as a windows kvm) and I'd rather not
see any downtime from the reboots. (It tries to actually migrate kvm's but
it doesn't work, lxc vm's are stopped and started.)

I moved them to specific servers because I thought I could do a better job
at balancing the cluster on my own (based on knowing what my VM's were
capable of). I'm not sure if my methods are a good idea or not, just
sounded right to me at the time.

On Tue, Nov 8, 2016 at 2:00 PM Ken Gaillot  wrote:

> On 11/08/2016 12:54 PM, Ryan Anstey wrote:
> > I've been running a ceph cluster with pacemaker for a few months now.
> > Everything has been working normally, but when I added a fourth node it
> > won't work like the others, even though their OS is the same and the
> > configs are all synced via salt. I also don't understand pacemaker that
> > well since I followed a guide for it. If anyone could steer me in the
> > right direction I would greatly appreciate it. Thank you!
> >
> > - My resources only start if the new node is the only active node.
> > - Once started on new node, if they are moved back to one of the
> > original nodes, it won't go back to the new one.
> > - My resources work 100% if I start them manually (without pacemaker).
> > - (In the logs/configs below, my resources are named "unifi",
> > "rbd_unifi" being the main one that's not working.)
>
> The key is all the location constraints starting with "cli-" in your
> configuration. Such constraints were added automatically by command-line
> tools, rather than added by you explicitly.
>
> For example, Pacemaker has no concept of "moving" a resource. It places
> all resources where they can best run, as specified by the
> configuration. So, to move a resource, command-line tools add a location
> constraint making the resource prefer a different node.
>
> The downside is that the preference doesn't automatically go away. The
> resource will continue to prefer the other node until you explicitly
> remove the constraint.
>
> Command-line tools that add such constraints generally provide some way
> to clear them. If you clear all those constraints, resources will again
> be placed on any node equally.
>
> Separately, you also have a default resource stickiness of 100. That
> means that even after you remove the constraints, resources that are
> already running will tend to stay where they are. But if you stop and
> start a resource, or add a new resource, it could start on a different
> node.
>
> 
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Resources wont start on new node unless it is the only active node

[Ken Gaillot]

On 11/08/2016 12:54 PM, Ryan Anstey wrote:
> I've been running a ceph cluster with pacemaker for a few months now.
> Everything has been working normally, but when I added a fourth node it
> won't work like the others, even though their OS is the same and the
> configs are all synced via salt. I also don't understand pacemaker that
> well since I followed a guide for it. If anyone could steer me in the
> right direction I would greatly appreciate it. Thank you!
> 
> - My resources only start if the new node is the only active node.
> - Once started on new node, if they are moved back to one of the
> original nodes, it won't go back to the new one.
> - My resources work 100% if I start them manually (without pacemaker).
> - (In the logs/configs below, my resources are named "unifi",
> "rbd_unifi" being the main one that's not working.)

The key is all the location constraints starting with "cli-" in your
configuration. Such constraints were added automatically by command-line
tools, rather than added by you explicitly.

For example, Pacemaker has no concept of "moving" a resource. It places
all resources where they can best run, as specified by the
configuration. So, to move a resource, command-line tools add a location
constraint making the resource prefer a different node.

The downside is that the preference doesn't automatically go away. The
resource will continue to prefer the other node until you explicitly
remove the constraint.

Command-line tools that add such constraints generally provide some way
to clear them. If you clear all those constraints, resources will again
be placed on any node equally.

Separately, you also have a default resource stickiness of 100. That
means that even after you remove the constraints, resources that are
already running will tend to stay where they are. But if you stop and
start a resource, or add a new resource, it could start on a different node.

> 
> Log when running cleaning up the resource on the NEW node:
> 
> Nov 08 09:25:20 h4 Filesystem(fs_unifi)[18044]: WARNING: Couldn't find
> device [/dev/rbd/rbd/unifi]. Expected /dev/??? to exist
> Nov 08 09:25:20 h4 lrmd[3564]: notice: lxc_unifi_monitor_0:18018:stderr
> [ unifi doesn't exist ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation lxc_unifi_monitor_0:
> not running (node=h4, call=484, rc=7, cib-update=390, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: h4-lxc_unifi_monitor_0:484 [
> unifi doesn't exist\n ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation fs_unifi_monitor_0: not
> running (node=h4, call=480, rc=7, cib-update=391, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation rbd_unifi_monitor_0:
> not running (node=h4, call=476, rc=7, cib-update=392, confirmed=true)
> 
> Log when running cleaning up the resource on the OLD node:  
> 
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: notice: State transition S_IDLE ->
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 cib[11389]: warning: A-Sync reply to crmd failed: No
> message of desired type
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:22 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:25 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:25 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   rbd_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   fs_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   lxc_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 119:
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 119
> (ref=pe_calc-dc-1478625686-648) derived from
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 12: monitor
> rbd_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 9: monitor
> 

Re: [ClusterLabs] pacemaker after upgrade from wheezy to jessie

[Ken Gaillot]

On 11/07/2016 09:08 AM, Toni Tschampke wrote:
> We managed to change the validate-with option via workaround (cibadmin
> export & replace) as setting the value with cibadmin --modify doesn't
> write the changes to disk.
>
> After experimenting with various schemes (xml is correctly interpreted
> by crmsh) we are still not able to communicate with local crmd.
> 
> Can someone please help to determine why the local crmd is not
> responding (we disabled our other nodes to eliminate possible corosync
> related issues) and runs into errors/timeouts when issuing crmsh or
> cibadmin related commands.

It occurs to me that wheezy used corosync 1. There were major changes
from corosync 1 to 2 ... 1 relied on a "plugin" to provide quorum for
pacemaker, whereas 2 has quorum built-in.

Did your upgrade documentation describe how to update the corosync
configuration, and did that go well? crmd may be unable to function due
to lack of quorum information.

> examples for not working local commands
> 
> timeout when running cibadmin: (strace attachment)
>> cibadmin --upgrade --force
>> Call cib_upgrade failed (-62): Timer expired
> 
> error when running a crm resource cleanup
>> crm resource cleanup $vm
>> Error signing on to the CRMd service
>> Error performing operation: Transport endpoint is not connected
> 
> I attached the strace log from running cib_upgrade, does this help to
> find the cause of the timeout issue?
> 
> Here is the corosync dump when locally starting pacemaker:
> 
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [MAIN  ] main.c:1256
>> Corosync Cluster Engine ('2.3.6'): started and ready to provide service.
>> Nov 07 16:01:59 [24339] nebel1 corosync info[MAIN  ] main.c:1257
>> Corosync built-in features: dbus rdma monitoring watchdog augeas
>> systemd upstart xmlconf qdevices snmp pie relro bindnow
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [TOTEM ]
>> totemnet.c:248 Initializing transport (UDP/IP Multicast).
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [TOTEM ]
>> totemcrypto.c:579 Initializing transmit/receive security (NSS) crypto:
>> none hash: none
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [TOTEM ]
>> totemnet.c:248 Initializing transport (UDP/IP Multicast).
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [TOTEM ]
>> totemcrypto.c:579 Initializing transmit/receive security (NSS) crypto:
>> none hash: none
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [TOTEM ]
>> totemudp.c:671 The network interface [10.112.0.1] is now up.
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [SERV  ] service.c:174
>> Service engine loaded: corosync configuration map access [0]
>> Nov 07 16:01:59 [24339] nebel1 corosync info[QB]
>> ipc_setup.c:536 server name: cmap
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [SERV  ] service.c:174
>> Service engine loaded: corosync configuration service [1]
>> Nov 07 16:01:59 [24339] nebel1 corosync info[QB]
>> ipc_setup.c:536 server name: cfg
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [SERV  ] service.c:174
>> Service engine loaded: corosync cluster closed process group service
>> v1.01 [2]
>> Nov 07 16:01:59 [24339] nebel1 corosync info[QB]
>> ipc_setup.c:536 server name: cpg
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [SERV  ] service.c:174
>> Service engine loaded: corosync profile loading service [4]
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [SERV  ] service.c:174
>> Service engine loaded: corosync resource monitoring service [6]
>> Nov 07 16:01:59 [24339] nebel1 corosync info[WD] wd.c:669
>> Watchdog /dev/watchdog is now been tickled by corosync.
>> Nov 07 16:01:59 [24339] nebel1 corosync warning [WD] wd.c:625
>> Could not change the Watchdog timeout from 10 to 6 seconds
>> Nov 07 16:01:59 [24339] nebel1 corosync warning [WD] wd.c:464
>> resource load_15min missing a recovery key.
>> Nov 07 16:01:59 [24339] nebel1 corosync warning [WD] wd.c:464
>> resource memory_used missing a recovery key.
>> Nov 07 16:01:59 [24339] nebel1 corosync info[WD] wd.c:581 no
>> resources configured.
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [SERV  ] service.c:174
>> Service engine loaded: corosync watchdog service [7]
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [SERV  ] service.c:174
>> Service engine loaded: corosync cluster quorum service v0.1 [3]
>> Nov 07 16:01:59 [24339] nebel1 corosync info[QB]
>> ipc_setup.c:536 server name: quorum
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [TOTEM ]
>> totemudp.c:671 The network interface [10.110.1.1] is now up.
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [TOTEM ]
>> totemsrp.c:2095 A new membership (10.112.0.1:348) was formed. Members
>> joined: 1
>> Nov 07 16:01:59 [24339] nebel1 corosync notice  [MAIN  ] main.c:310
>> Completed service synchronization, ready to provide service.
>> Nov 07 16:01:59 [24341] nebel1 pacemakerd:   notice: main: 
>> Starting Pacemaker 1.1.15 | build=e174ec8 features: 

Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?

[Ken Gaillot]

On 11/04/2016 01:57 PM, CART Andreas wrote:
> Hi
>  
> I have a basic 2 node active/passive cluster with Pacemaker (1.1.14 ,
> pcs: 0.9.148) / CMAN (3.0.12.1) / Corosync (1.4.7) on RHEL 6.8.
> This cluster runs NFS on top of DRBD (8.4.4).
>  
> Basically the system is working on both nodes and I can switch the
> resources from one node to the other.
> But switching resources to the other node does not work, if I try to
> move just one resource and have the others follow due to the location
> constraints.
>  
> From the logged messages I see that in this “failure case” there is NO
> attempt to demote/promote the DRBD clone resource.
>  
> Here is my setup:
> ==
> Cluster Name: clst1
> Corosync Nodes:
> ventsi-clst1-sync ventsi-clst2-sync
> Pacemaker Nodes:
> ventsi-clst1-sync ventsi-clst2-sync
>  
> Resources:
> Resource: IPaddrNFS (class=ocf provider=heartbeat type=IPaddr2)
>   Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=24
>   Operations: start interval=0s timeout=20s (IPaddrNFS-start-interval-0s)
>   stop interval=0s timeout=20s (IPaddrNFS-stop-interval-0s)
>   monitor interval=5s (IPaddrNFS-monitor-interval-5s)
> Resource: NFSServer (class=ocf provider=heartbeat type=nfsserver)
>   Attributes: nfs_shared_infodir=/var/lib/nfsserversettings/
> nfs_ip=xxx.xxx.xxx.xxx nfsd_args="-H xxx.xxx.xxx.xxx"
>   Operations: start interval=0s timeout=40 (NFSServer-start-interval-0s)
>   stop interval=0s timeout=20s (NFSServer-stop-interval-0s)
>   monitor interval=10s timeout=20s
> (NFSServer-monitor-interval-10s)
> Master: DRBDClone
>   Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true
>   Resource: DRBD (class=ocf provider=linbit type=drbd)
>Attributes: drbd_resource=nfsdata
>Operations: start interval=0s timeout=240 (DRBD-start-interval-0s)
>promote interval=0s timeout=90 (DRBD-promote-interval-0s)
>demote interval=0s timeout=90 (DRBD-demote-interval-0s)
>stop interval=0s timeout=100 (DRBD-stop-interval-0s)
>monitor interval=1s timeout=5 (DRBD-monitor-interval-1s)
> Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)
>   Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4
>   Operations: start interval=0s timeout=60
> (DRBD_global_clst-start-interval-0s)
>   stop interval=0s timeout=60
> (DRBD_global_clst-stop-interval-0s)
>   monitor interval=20 timeout=40
> (DRBD_global_clst-monitor-interval-20)
>  
> Stonith Devices:
> Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1
>   Operations: monitor interval=60s (ipmi-fence-clst1-monitor-interval-60s)
> Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1
>   Operations: monitor interval=60s (ipmi-fence-clst2-monitor-interval-60s)
> Fencing Levels:
>  
> Location Constraints:
>   Resource: ipmi-fence-clst1
> Disabled on: ventsi-clst1-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)
>   Resource: ipmi-fence-clst2
> Disabled on: ventsi-clst2-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)
> Ordering Constraints:
>   start IPaddrNFS then start NFSServer (kind:Mandatory)
> (id:order-IPaddrNFS-NFSServer-mandatory)
>   promote DRBDClone then start DRBD_global_clst (kind:Mandatory)
> (id:order-DRBDClone-DRBD_global_clst-mandatory)
>   start DRBD_global_clst then start IPaddrNFS (kind:Mandatory)
> (id:order-DRBD_global_clst-IPaddrNFS-mandatory)
> Colocation Constraints:
>   NFSServer with IPaddrNFS (score:INFINITY)
> (id:colocation-NFSServer-IPaddrNFS-INFINITY)
>   DRBD_global_clst with DRBDClone (score:INFINITY)
> (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)

It took me a while to notice it, it's easily overlooked, but the above
constraint is the problem. It says DRBD_global_clst must be located
where DRBDClone is running ... not necessarily where DRBDClone is
master. This constraint should be created like this:

pcs constraint colocation add DRBD_global_clst with master DBRDClone

>   IPaddrNFS with DRBD_global_clst (score:INFINITY)
> (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)
>  
> Resources Defaults:
> resource-stickiness: INFINITY
> Operations Defaults:
> timeout: 10s
>  
> Cluster Properties:
> cluster-infrastructure: cman
> dc-version: 1.1.14-8.el6-70404b0
> have-watchdog: false
> last-lrm-refresh: 1478277432
> no-quorum-policy: ignore
> stonith-enabled: true
> symmetric-cluster: true
> 

Re: [ClusterLabs] Live migration not working on shutdown

[Ken Gaillot]

On 11/04/2016 05:51 AM, IT Nerb GmbH wrote:
> Zitat von Klaus Wenninger :
> 
>> On 11/02/2016 06:32 PM, Ken Gaillot wrote:
>>> On 10/26/2016 06:12 AM, Rainer Nerb wrote:
 Hello all,

 we're currently testing a 2-node-cluster with 2 vms and live migration
 on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
 migration via ssh-method.

 Live migration works, if we issue "pcs resource move ...", "pcs cluster
 standby", "pcs cluster stop" and even "systemctl rescue".
 The latter only worked, after adding the following additional
 dependencies to pacemaker.service and leaving the management of those
 services to systemd:

   * After/Requires=systemd-machined.service
   * After/Requires=systemd-machine-id-commit.service
   * After/Requires=remote-fs.target
   * After/Requires=libvirtd.service
   * After/Requires=iscsi.service
   * After/Requires=iscsid.service
   * After/Requires=sshd.service
>>> This makes sense when clustered resources depend on services that aren't
>>> themselves managed by the cluster. It's dependent on your situation, so
>>> it's not something that pacemaker can solve generically.
> First approach was to use systemd-resources as there are no ocf:
> resource-agents for iSCSI-Initiators or libvirtd in our distribution.
> But then migration failed even on "systemctl rescue".
>>>
>>> You may already be aware, but the easiest way to add such requirements
>>> is to put them in a systemd unit override, e.g.
>>> /etc/systemd/system/pacemaker.service.d/dependencies.conf.
> Yes, that's how we implemented the additional dependencies.
>>>
 When shutting down or rebooting migration fails and not even the
 regular
 shutdown of the vms succeeds. Systemd seems to tear down the vms by
 terminating something they depend on.

 Is this a known issue? Did we miss any further dependencies?
>>> There was a shutdown issue when using systemd-class cluster resources
>>> (systemd: instead of ocf:), but I believe that was fixed in the package
>>> you're using, and it's probably not relevant here anyway.
>> Speaking of
>> https://github.com/ClusterLabs/pacemaker/pull/887/commits/6aae8542abedc755b90c8c49aa5c429718fd12f1?
>>
>>
>> It shouldn't be in Centos 7.2 but I agree unless there are no
>> systemd-resources involved it wouldn't matter.
>>
>>>
>>> It does sound like there's another dependency, but I don't know what.
>>>
>>> What log messages do you see on the failure?
> See attached log files.

The line that stands out to me is:

Nov  4 11:11:39 kvm02 systemd: Stopping Virtual Machine qemu-2-samba2.

Systemd is stopping the VM before pacemaker is able to migrate it. I'm
guessing that is due to this line in the libvirt unit:

Before=libvirt-guests.service

It appears systemd feels free to do that part in parallel, even though
libvirt itself has to wait until pacemaker finishes stopping. Try adding
libvirt-guests to your pacemaker override.

>>>
 Tia
 Rainer


 

 IT Nerb GmbH
 Lessingstraße 8
 85098 Großmehring

 Telefon : +49 700 ITNERBGMBH
 Telefax : +49 8407 939 284
 email : i...@it-nerb.de
 Internet : www.it-nerb.de 
 Geschäftsführer:Rainer Nerb
 Handelsregister:HRB 2592
 HR-Gericht:Ingolstadt

 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Preventing switchover in case of failing ping node

[Ken Gaillot]

On 11/03/2016 08:49 AM, Detlef Gossrau wrote:
> Hi all,
> 
> is it possible to prevent a switchover in a active/passive cluster if a
> ping node completely fails ?
> 
> Situation:
> 
> A ping node is put into maintenance and not reachable for a certain
> time. The cluster nodes getting the information about the failing ping
> node at different times. For example, node 1 - which is the passive node
> - is getting the information earlier than node 2. As a result the
> resources are moved to node 2. But in the case the ping node is not
> available at all, no switchover should be executed. Is it possible to
> prevent this ?

There's no way for one node to predict that all other nodes will also be
unable to see the target. The only workarounds I see are:

* Use a second ping target on a different path, if available

* If the maintenance is planned, you could disable the relevant
constraints during the window

> 
> Thanks for any help !
> 
> Best regards,
> Detlef


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How Pacemaker reacts to fast changes of the same parameter in configuration

[Dejan Muhamedagic]

On Tue, Nov 08, 2016 at 12:54:10PM +0100, Klaus Wenninger wrote:
> On 11/08/2016 11:40 AM, Kostiantyn Ponomarenko wrote:
> > Hi,
> >
> > I need a way to do a manual fail-back on demand.
> > To be clear, I don't want it to be ON/OFF; I want it to be more like
> > "one shot".
> > So far I found that the most reasonable way to do it - is to set
> > "resource stickiness" to a different value, and then set it back to
> > what it was. 
> > To do that I created a simple script with two lines:
> >
> > crm configure rsc_defaults resource-stickiness=50
> > crm configure rsc_defaults resource-stickiness=150
> >
> > There are no timeouts before setting the original value back.
> > If I call this script, I get what I want - Pacemaker moves resources
> > to their preferred locations, and "resource stickiness" is set back to
> > its original value. 
> >
> > Despite it works, I still have few concerns about this approach.
> > Will I get the same behavior under a big load with delays on systems
> > in cluster (which is truly possible and a normal case in my environment)?
> > How Pacemaker treats fast change of this parameter?
> > I am worried that if "resource stickiness" is set back to its original
> > value to fast, then no fail-back will happen. Is it possible, or I
> > shouldn't worry about it?
> 
> AFAIK pengine is interrupted when calculating a more complicated transition
> and if the situation has changed a transition that is just being executed
> is aborted if the input from pengine changed.
> So I would definitely worry!
> What you could do is to issue 'crm_simulate -Ls' in between and grep for
> an empty transition.
> There might be more elegant ways but that should be safe.

crmsh has an option (-w) to wait for the PE to settle after
committing configuration changes.

Thanks,

Dejan
> 
> > Thank you,
> > Kostia
> >
> >
> > ___
> > Users mailing list: Users@clusterlabs.org
> > http://clusterlabs.org/mailman/listinfo/users
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] ocf:lvm2:VolumeGroup Probe Issue

[Marc Smith]

Perfect, thanks for the quick reply.

--Marc

On Tue, Nov 8, 2016 at 12:00 PM, Ken Gaillot  wrote:
> On 11/08/2016 10:37 AM, Marc Smith wrote:
>> Hi,
>>
>> First, I realize ocf:lvm2:VolumeGroup comes from the LVM2 package and
>> not resource-agents, but I'm hoping someone on this list is familiar
>> with this RA and can provide some insight.
>>
>> In my cluster configuration, I'm using ocf:lvm2:VolumeGroup to manage
>> my LVM VG's, and I'm using the cluster to manage DLM and CLVM. I have
>> my constraints in place and everything seems to be working mostly,
>> except I'm hitting a glitch with ocf:lvm2:VolumeGroup and the initial
>> probe operation.
>>
>> On startup, a probe operation (monitor) is issued for all of the
>> resources, but ocf:lvm2:VolumeGroup is returning OCF_ERR_GENERIC in
>> VolumeGroup_status() (via VolumeGroup_monitor()) since clvmd hasn't
>> started yet... this line in VolumeGroup_status() is the trouble:
>>
>> VGOUT=`vgdisplay -v $OCF_RESKEY_volgrpname 2>&1` || exit $OCF_ERR_GENERIC
>>
>> When clvmd is not running, 'vgdisplay -v name' will always return
>> something like this:
>>
>> --snip--
>>   connect() failed on local socket: No such file or directory
>>   Internal cluster locking initialisation failed.
>>   WARNING: Falling back to local file-based locking.
>>   Volume Groups with the clustered attribute will be inaccessible.
>> VG name on command line not found in list of VGs: biggie
>>   Volume group "biggie" not found
>>   Cannot process volume group biggie
>> --snip--
>>
>> And exits with a status of 5. So, my question is, do I patch the RA?
>> Or is there some cluster constraint I can add so a probe/monitor
>> operation isn't performed for the VolumeGroup resource until CLVM has
>> been started?
>
> Ordered probes is a desired feature for Pacemaker, but the
> implementation is much trickier than it appears at first glance, so
> there's no timeline for when it might arrive.
>
> In the meantime, patching the RA to exit with "not running" in this
> situation is probably the best workaround. There is a library function
> ocf_is_probe you can use to avoid messing with "real" monitors.
>
>> Any other advice? Is ocf:heartbeat:LVM or ocf:lvm2:VolumeGroup the
>> more popular RA for managing LVM VG's? Any comments from other users
>> on experiences using either (good, bad)? Both appear to achieve the
>> same function, just a bit differently.
>>
>>
>> Thanks,
>>
>> Marc
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] ocf:lvm2:VolumeGroup Probe Issue

[Marc Smith]

Hi,

First, I realize ocf:lvm2:VolumeGroup comes from the LVM2 package and
not resource-agents, but I'm hoping someone on this list is familiar
with this RA and can provide some insight.

In my cluster configuration, I'm using ocf:lvm2:VolumeGroup to manage
my LVM VG's, and I'm using the cluster to manage DLM and CLVM. I have
my constraints in place and everything seems to be working mostly,
except I'm hitting a glitch with ocf:lvm2:VolumeGroup and the initial
probe operation.

On startup, a probe operation (monitor) is issued for all of the
resources, but ocf:lvm2:VolumeGroup is returning OCF_ERR_GENERIC in
VolumeGroup_status() (via VolumeGroup_monitor()) since clvmd hasn't
started yet... this line in VolumeGroup_status() is the trouble:

VGOUT=`vgdisplay -v $OCF_RESKEY_volgrpname 2>&1` || exit $OCF_ERR_GENERIC

When clvmd is not running, 'vgdisplay -v name' will always return
something like this:

--snip--
  connect() failed on local socket: No such file or directory
  Internal cluster locking initialisation failed.
  WARNING: Falling back to local file-based locking.
  Volume Groups with the clustered attribute will be inaccessible.
VG name on command line not found in list of VGs: biggie
  Volume group "biggie" not found
  Cannot process volume group biggie
--snip--

And exits with a status of 5. So, my question is, do I patch the RA?
Or is there some cluster constraint I can add so a probe/monitor
operation isn't performed for the VolumeGroup resource until CLVM has
been started?

Any other advice? Is ocf:heartbeat:LVM or ocf:lvm2:VolumeGroup the
more popular RA for managing LVM VG's? Any comments from other users
on experiences using either (good, bad)? Both appear to achieve the
same function, just a bit differently.


Thanks,

Marc

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: What is the logic when two node are down at the same time and needs to be fenced

[Niu Sibo]

On 11/8/2016 5:08 PM, Ulrich Windl wrote:

Niu Sibo  schrieb am 07.11.2016 um 16:59 in

Nachricht <5820a4cc.9030...@linux.vnet.ibm.com>:

Hi Ken,

Thanks for the clarification. Now I have another real problem that needs
your advise.

The cluster consists of 5 nodes and one of the node got a 1 second
network failure which resulted in one of the VirtualDomain resources to
start on two nodes at the same time. The cluster property
no_quorum_policy is set to stop.

At 16:13:34, this happened:
16:13:34 zs95kj attrd[133000]:  notice: crm_update_peer_proc: Node
zs93KLpcs1[5] - state is now lost (was member)
16:13:34 zs95kj corosync[132974]:  [CPG   ] left_list[0]
group:pacemakerd\x00, ip:r(0) ip(10.20.93.13) , pid:28721
16:13:34 zs95kj crmd[133002]: warning: No match for shutdown action on 5

Usually the node would be fenced now. In the meantime the node might _try_ to 
stop the resources.
In my case, this 1 sec network lost of zs93KLpcs1 didn't result in a  
fence action on zs93KLpcs1.


Which node might  _try_ to stop the resource? The DC or the node that 
lost the connection to the cluster?



16:13:34 zs95kj attrd[133000]:  notice: Removing all zs93KLpcs1
attributes for attrd_peer_change_cb
16:13:34 zs95kj corosync[132974]:  [CPG   ] left_list_entries:1
16:13:34 zs95kj crmd[133002]:  notice: Stonith/shutdown of zs93KLpcs1
not matched
...
16:13:35 zs95kj attrd[133000]:  notice: crm_update_peer_proc: Node
zs93KLpcs1[5] - state is now member (was (null))

Where are the logs from the other node? I don't see where resources are 
_started_.


From the log of zs93kl where 110187 guest is started:

11:41:19 [28727] zs93kl   crmd: (   lrm.c:2392  )  notice: 
process_lrm_event:Operation zs95kjg110187_res_start_0: ok 
(node=zs93KLpcs1, call=1249, rc=0, cib-update=837, confirmed=true)


This guest has been running on zs93KL since 11:41 until 16:13:34 when 
zs93KL lost the connection to the cluster for 1 second.  After this 
short break, the DC decides to start this guest on another node zs90kp 
which you can see from the log below. However, on zs93KL, it still has 
the following log:


16:14:17 [180373] zs93kl   crmd: ( utils.c:1942  )   debug: 
create_operation_update: do_update_resource: Updating resource 
zs95kjg110187_res after monitor op complete (interval=0)
16:14:17 [180373] zs93kl   crmd: (   lrm.c:2392  )  notice: 
process_lrm_event:   Operation zs95kjg110187_res_monitor_0: ok 
(node=zs93KLpcs1, call=1655, rc=0, cib-update=216, confirmed=true)
16:14:17 [180373] zs93kl   crmd: (   lrm.c:196   )   debug: 
update_history_cache:Updating history for 'zs95kjg110187_res' 
with monitor op



  From the DC:
[root@zs95kj ~]# crm_simulate --xml-file
/var/lib/pacemaker/pengine/pe-input-3288.bz2 |grep 110187
   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Started
zs93KLpcs1 <--This is the baseline that everything works normal

[root@zs95kj ~]# crm_simulate --xml-file
/var/lib/pacemaker/pengine/pe-input-3289.bz2 |grep 110187
   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped
<--- Here the node zs93KLpcs1 lost it's network for 1 sec and
resulted in this state.

[root@zs95kj ~]# crm_simulate --xml-file
/var/lib/pacemaker/pengine/pe-input-3290.bz2 |grep 110187
   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped

[root@zs95kj ~]# crm_simulate --xml-file
/var/lib/pacemaker/pengine/pe-input-3291.bz2 |grep 110187
   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped


  From the DC's pengine log, it has:
16:05:01 zs95kj pengine[133001]:  notice: Calculated Transition 238:
/var/lib/pacemaker/pengine/pe-input-3288.bz2
...
16:13:41 zs95kj pengine[133001]:  notice: Start
zs95kjg110187_res#011(zs90kppcs1)
...
16:13:41 zs95kj pengine[133001]:  notice: Calculated Transition 239:
/var/lib/pacemaker/pengine/pe-input-3289.bz2

  From the DC's CRMD log, it has:
Sep  9 16:05:25 zs95kj crmd[133002]:  notice: Transition 238
(Complete=48, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-3288.bz2): Complete
...
Sep  9 16:13:42 zs95kj crmd[133002]:  notice: Initiating action 752:
start zs95kjg110187_res_start_0 on zs90kppcs1
...
Sep  9 16:13:56 zs95kj crmd[133002]:  notice: Transition 241
(Complete=81, Pending=0, Fired=0, Skipped=172, Incomplete=341,
Source=/var/lib/pacemaker/pengine/pe-input-3291.bz2): Stopped

Here I do not see any log about pe-input-3289.bz2 and pe-input-3290.bz2.
Why is this?

  From the log on zs93KLpcs1 where guest 110187 was running, i do not see
any message regarding stopping this resource after it lost its
connection to the cluster.

Any ideas where to look for possible cause?

On 11/3/2016 1:02 AM, Ken Gaillot wrote:

On 11/02/2016 11:17 AM, Niu Sibo wrote:

Hi all,

I have a general question regarding the fence login in pacemaker.

I have setup a three nodes cluster with Pacemaker 1.1.13 and cluster
property no_quorum_policy set to 

Re: [ClusterLabs] How Pacemaker reacts to fast changes of the same parameter in configuration

[Klaus Wenninger]

On 11/08/2016 11:40 AM, Kostiantyn Ponomarenko wrote:
> Hi,
>
> I need a way to do a manual fail-back on demand.
> To be clear, I don't want it to be ON/OFF; I want it to be more like
> "one shot".
> So far I found that the most reasonable way to do it - is to set
> "resource stickiness" to a different value, and then set it back to
> what it was. 
> To do that I created a simple script with two lines:
>
> crm configure rsc_defaults resource-stickiness=50
> crm configure rsc_defaults resource-stickiness=150
>
> There are no timeouts before setting the original value back.
> If I call this script, I get what I want - Pacemaker moves resources
> to their preferred locations, and "resource stickiness" is set back to
> its original value. 
>
> Despite it works, I still have few concerns about this approach.
> Will I get the same behavior under a big load with delays on systems
> in cluster (which is truly possible and a normal case in my environment)?
> How Pacemaker treats fast change of this parameter?
> I am worried that if "resource stickiness" is set back to its original
> value to fast, then no fail-back will happen. Is it possible, or I
> shouldn't worry about it?

AFAIK pengine is interrupted when calculating a more complicated transition
and if the situation has changed a transition that is just being executed
is aborted if the input from pengine changed.
So I would definitely worry!
What you could do is to issue 'crm_simulate -Ls' in between and grep for
an empty transition.
There might be more elegant ways but that should be safe.

> Thank you,
> Kostia
>
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] How Pacemaker reacts to fast changes of the same parameter in configuration

[Kostiantyn Ponomarenko]

Hi,

I need a way to do a manual fail-back on demand.
To be clear, I don't want it to be ON/OFF; I want it to be more like "one
shot".
So far I found that the most reasonable way to do it - is to set "resource
stickiness" to a different value, and then set it back to what it was.
To do that I created a simple script with two lines:

crm configure rsc_defaults resource-stickiness=50
crm configure rsc_defaults resource-stickiness=150

There are no timeouts before setting the original value back.
If I call this script, I get what I want - Pacemaker moves resources to
their preferred locations, and "resource stickiness" is set back to its
original value.

Despite it works, I still have few concerns about this approach.
Will I get the same behavior under a big load with delays on systems in
cluster (which is truly possible and a normal case in my environment)?
How Pacemaker treats fast change of this parameter?
I am worried that if "resource stickiness" is set back to its original
value to fast, then no fail-back will happen. Is it possible, or I
shouldn't worry about it?

Thank you,
Kostia
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] permissions under /etc/corosync/qnetd

[Jan Friesse]

Ferenc Wágner napsal(a):

Jan Friesse  writes:


Ferenc Wágner napsal(a):


Have you got any plans/timeline for 2.4.2 yet?


Yep, I'm going to release it in few minutes/hours.


Man, that was quick.  I've got a bunch of typo fixes queued..:) Please
consider announcing upcoming releases a couple of days in advance; as a
packager, I'd much appreciate it.  Maybe even tag release candidates...


We are tagging RC of big releases. This release was super small (eventho 
breaking compatibility). And actually, it's better to be released rather 
sooner than later. I will definitively think about RC of smaller 
releases (2.3.x -> 2.4.x).




Anyway, I've got a question concerning corosync-qnetd.  I run it as
user and group coroqnetd.  Is granting it read access to cert8.db and
key3.db enough for proper operation?  corosync-qnetd-certutil gives


Should be, but it's not tested.


write access to group coroqnetd to everything, which seems unintuitive


Yep. Idea is to allow scenario of qnetd administrator role. So basically 
regular (non-root) user within coroqnetd group without root passwd 
knowledge/sudo administering qnetd.




to me.  Please note that I've got zero experience with NSS.  But I don't
expect the daemon to change the certificate database.  Should I?


Nope it shouldn't.

Regards,
  Honza






___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Antw: Re: What is the logic when two node are down at the same time and needs to be fenced

[Ulrich Windl]

>>> Niu Sibo  schrieb am 07.11.2016 um 16:59 in
Nachricht <5820a4cc.9030...@linux.vnet.ibm.com>:
> Hi Ken,
> 
> Thanks for the clarification. Now I have another real problem that needs 
> your advise.
> 
> The cluster consists of 5 nodes and one of the node got a 1 second 
> network failure which resulted in one of the VirtualDomain resources to 
> start on two nodes at the same time. The cluster property 
> no_quorum_policy is set to stop.
> 
> At 16:13:34, this happened:
> 16:13:34 zs95kj attrd[133000]:  notice: crm_update_peer_proc: Node 
> zs93KLpcs1[5] - state is now lost (was member)
> 16:13:34 zs95kj corosync[132974]:  [CPG   ] left_list[0] 
> group:pacemakerd\x00, ip:r(0) ip(10.20.93.13) , pid:28721
> 16:13:34 zs95kj crmd[133002]: warning: No match for shutdown action on 5

Usually the node would be fenced now. In the meantime the node might _try_ to 
stop the resources.

> 16:13:34 zs95kj attrd[133000]:  notice: Removing all zs93KLpcs1 
> attributes for attrd_peer_change_cb
> 16:13:34 zs95kj corosync[132974]:  [CPG   ] left_list_entries:1
> 16:13:34 zs95kj crmd[133002]:  notice: Stonith/shutdown of zs93KLpcs1 
> not matched
> ...
> 16:13:35 zs95kj attrd[133000]:  notice: crm_update_peer_proc: Node 
> zs93KLpcs1[5] - state is now member (was (null))

Where are the logs from the other node? I don't see where resources are 
_started_.


> 
>  From the DC:
> [root@zs95kj ~]# crm_simulate --xml-file 
> /var/lib/pacemaker/pengine/pe-input-3288.bz2 |grep 110187
>   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Started 
> zs93KLpcs1 <--This is the baseline that everything works normal
> 
> [root@zs95kj ~]# crm_simulate --xml-file 
> /var/lib/pacemaker/pengine/pe-input-3289.bz2 |grep 110187
>   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped 
> <--- Here the node zs93KLpcs1 lost it's network for 1 sec and 
> resulted in this state.
> 
> [root@zs95kj ~]# crm_simulate --xml-file 
> /var/lib/pacemaker/pengine/pe-input-3290.bz2 |grep 110187
>   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped
> 
> [root@zs95kj ~]# crm_simulate --xml-file 
> /var/lib/pacemaker/pengine/pe-input-3291.bz2 |grep 110187
>   zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped
> 
> 
>  From the DC's pengine log, it has:
> 16:05:01 zs95kj pengine[133001]:  notice: Calculated Transition 238: 
> /var/lib/pacemaker/pengine/pe-input-3288.bz2
> ...
> 16:13:41 zs95kj pengine[133001]:  notice: Start 
> zs95kjg110187_res#011(zs90kppcs1)
> ...
> 16:13:41 zs95kj pengine[133001]:  notice: Calculated Transition 239: 
> /var/lib/pacemaker/pengine/pe-input-3289.bz2
> 
>  From the DC's CRMD log, it has:
> Sep  9 16:05:25 zs95kj crmd[133002]:  notice: Transition 238 
> (Complete=48, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
> Source=/var/lib/pacemaker/pengine/pe-input-3288.bz2): Complete
> ...
> Sep  9 16:13:42 zs95kj crmd[133002]:  notice: Initiating action 752: 
> start zs95kjg110187_res_start_0 on zs90kppcs1
> ...
> Sep  9 16:13:56 zs95kj crmd[133002]:  notice: Transition 241 
> (Complete=81, Pending=0, Fired=0, Skipped=172, Incomplete=341, 
> Source=/var/lib/pacemaker/pengine/pe-input-3291.bz2): Stopped
> 
> Here I do not see any log about pe-input-3289.bz2 and pe-input-3290.bz2. 
> Why is this?
> 
>  From the log on zs93KLpcs1 where guest 110187 was running, i do not see 
> any message regarding stopping this resource after it lost its 
> connection to the cluster.
> 
> Any ideas where to look for possible cause?
> 
> On 11/3/2016 1:02 AM, Ken Gaillot wrote:
>> On 11/02/2016 11:17 AM, Niu Sibo wrote:
>>> Hi all,
>>>
>>> I have a general question regarding the fence login in pacemaker.
>>>
>>> I have setup a three nodes cluster with Pacemaker 1.1.13 and cluster
>>> property no_quorum_policy set to ignore. When two nodes lost their NIC
>>> corosync is running on at the same time, it looks like the two nodes are
>>> getting fenced one by one, even I have three fence devices defined for
>>> each of the node.
>>>
>>> What should I be expecting in the case?
>> It's probably coincidence that the fencing happens serially; there is
>> nothing enforcing that for separate fence devices. There are many steps
>> in a fencing request, so they can easily take different times to complete.
>>
>>> I noticed if the node rejoins the cluster before the cluster starts the
>>> fence actions, some resources will get activated on 2 nodes at the
>>> sametime. This is really not good if the resource happens to be
>>> VirutalGuest.  Thanks for any suggestions.
>> Since you're ignoring quorum, there's nothing stopping the disconnected
>> node from starting all resources on its own. It can even fence the other
>> nodes, unless the downed NIC is used for fencing. From that node's point
>> of view, it's the other two nodes that are lost.
>>
>> Quorum is the only solution I know of to prevent that. Fencing will
>> correct the situation, but it won't prevent it.
>>
>> See the 

[ClusterLabs] Antw: Re: Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 07.11.2016 um 16:15 in 
>>> Nachricht
:
> On 11/07/2016 01:41 AM, Ulrich Windl wrote:
> Ken Gaillot  schrieb am 04.11.2016 um 22:37 in 
> Nachricht
>> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>:
>>> On 11/04/2016 02:29 AM, Ulrich Windl wrote:
>>> Ken Gaillot  schrieb am 03.11.2016 um 17:08 in
> * The new ocf:pacemaker:attribute resource agent sets a node attribute
> according to whether the resource is running or stopped. This may be
> useful in combination with attribute-based rules to model dependencies
> that simple constraints can't handle.

 I don't quite understand this: Isn't the state of a resource in the CIB 
>>> status
 section anyway? If not, why not add it? So it would be readily available 
 for
 anyone (rules, constraints, etc.).
>>>
>>> This (hopefully) lets you model more complicated relationships.
>>>
>>> For example, someone recently asked whether they could make an ordering
>>> constraint apply only at "start-up" -- the first time resource A starts,
>>> it does some initialization that B needs, but once that's done, B can be
>>> independent of A.
>> 
>> Is "at start-up" before start of the resource, after start of the resource, 
> or parallel to the start of the resource ;-)
>> Probably a "hook" in the corresponding RA is the better approach, unless you 
> can really model all of the above.
>> 
>>>
>>> For that case, you could group A with an ocf:pacemaker:attribute
>>> resource. The important part is that the attribute is not set if A has
>>> never run on a node. So, you can make a rule that B can run only where
>>> the attribute is set, regardless of the value -- even if A is later
>>> stopped, the attribute will still be set.
>> 
>> If a resource is not running on a node,, it is "stopped"; isn't it?
> 
> Sure, but what I mean is, if resource A has *never* run on a node, then
> the corresponding node attribute will be *unset*. But if A has ever
> started and/or stopped on a node, the attribute will be set to one value
> or the other. So, a rule can be used to check whether the attribute is
> set, to determine whether A has *ever* been run on the node, regardless
> of whether it is currently running.

What I wanted to say is this: To be usable the cluster framework should be 
clever enough to set the proper attribute to "not running" even if a resource 
never ran. It's easier to implement it once in the framework rather than 
implementing it in every tool.

> 
>>> Another possible use would be for a cron that needs to know whether a
>>> particular resource is running, and an attribute query is quicker and
>>> easier than something like parsing crm_mon output or probing the service.
>> 
>> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so besides 
> of lacking options and inefficient implementation, why should one be faster 
> than the other?
>> 
>>>
>>> It's all theoretical at this point, and I'm not entirely sure those
>>> examples would be useful :) but I wanted to make the agent available for
>>> people to experiment with.
>> 
>> A good product manager should resist the attempt to provide any feature the 
> customers ask for, avoiding bloat-ware. That is to protect the customer from 
> their own bad decisions. In most cases there is a better, more universal 
> solution to the specific problem.
> 
> Sure, but this is a resource agent -- it adds no overhead to anyone not
> using it, and since we don't have any examples or walk-throughs using
> it, users would have to investigate and experiment to see whether it's
> of any use in their environment.
> 
> Hopefully, this will turn out to be a general-purpose tool of value to
> multiple problem scenarios.
> 
> * Pacemaker's existing "node health" feature allows resources to move
> off nodes that become unhealthy. Now, when using
> node-health-strategy=progressive, a new cluster property
> node-health-base will be used as the initial health score of newly
> joined nodes (defaulting to 0, which is the previous behavior). This
> allows cloned and multistate resource instances to start on a node even
> if it has some "yellow" health attributes.

 So the node health is more or less a "node score"? I don't understand the 
>>> last
 sentence. Maybe give an example?
>>>
>>> Yes, node health is a score that's added when deciding where to place a
>>> resource. It does get complicated ...
>>>
>>> Node health monitoring is optional, and off by default.
>>>
>>> Node health attributes are set to red, yellow or green (outside
>>> pacemaker itself -- either by a resource agent, or some external
>>> process). As an example, let's say we have three node health attributes
>>> for CPU usage, CPU temperature, and SMART error count.
>>>
>>> With a progressive strategy, red and yellow are assigned some negative