Re: [ClusterLabs] pacemaker after upgrade from wheezy to jessie

[Ken Gaillot]

On 11/03/2016 05:51 AM, Toni Tschampke wrote:
> Hi,
> 
> we just upgraded our nodes from wheezy 7.11 (pacemaker 1.1.7) to jessie
> (pacemaker 1.1.15, corosync 2.3.6).
> During the upgrade pacemaker was removed (rc) and reinstalled after from
> jessie-backports, same for crmsh.
> 
> Now we are encountering multiple problems:
> 
> First I checked the configuration on a single node running pacemaker &
> corosync which dropped a strange error, followed by multiple lines
> stating syntax is wrong. crm configure show then showed up a mixed view
> of xml and crmsh singleline syntax.
> 
>> ERROR: Cannot read schema file
> '/usr/share/pacemaker/pacemaker-1.1.rng': [Errno 2] No such file or
> directory: '/usr/share/pacemaker/pacemaker-1.1.rng'

pacemaker-1.1.rng was renamed to pacemaker-next.rng in Pacemaker 1.1.12,
as it was used to hold experimental new features rather than as the
actual next version of the schema. So, the schema skipped to 1.2.

I'm going to guess you were using the experimental 1.1 schema as the
"validate-with" at the top of /var/lib/pacemaker/cib/cib.xml. Try
changing the validate-with to pacemaker-next or pacemaker-1.2 and see if
you get better results. Don't edit the file directly though; use the
cibadmin command so it signs the end result properly.

After changing the validate-with, run:

  crm_verify -x /var/lib/pacemaker/cib/cib.xml

and fix any errors that show up.

> When we looked into that folder there was pacemaker-1.0.rng, 1.2 and so
> on. As a quick try we symlinked the 1.2 to 1.1 and the syntax errors
> were gone. When running crm resource show, all resources showed up, when
> running crm_mon -1fA the output was unexpected as it showed all nodes
> offline, with no DC elected:
> 
>> Stack: corosync
>> Current DC: NONE
>> Last updated: Thu Nov  3 11:11:16 2016
>> Last change: Thu Nov  3 09:54:52 2016 by root via cibadmin on nebel1
>>
>>  *** Resource management is DISABLED ***
>>  The cluster will not attempt to start, stop or recover services
>>
>> 3 nodes and 73 resources configured:
>> 5 resources DISABLED and 0 BLOCKED from being started due to failures
>>
>> OFFLINE: [ nebel1 nebel2 nebel3 ]
> 
> we tried to manually change dc-version
> 
> when issuing a simple cleanup command I got the following error:
> 
>> crm resource cleanup DrbdBackuppcMs
>> Error signing on to the CRMd service
>> Error performing operation: Transport endpoint is not connected
> 
> which looks like crmsh is not able to communicate with crmd and nothing
> is logged in this case in corosync.log
> 
> we experimented with multiple config changes (corosync.conf: pacemaker
> ver 0 > 1)
> cib-bootstrap-options: cluster-infrastructure from openais to corosync
> 
>> Package versions:
>> cman 3.1.8-1.2+b1
>> corosync 2.3.6-3~bpo8+1
>> crmsh 2.2.0-1~bpo8+1
>> csync2 1.34-2.3+b1
>> dlm-pcmk 3.0.12-3.2+deb7u2
>> libcman3 3.1.8-1.2+b1
>> libcorosync-common4:amd64 2.3.6-3~bpo8+1
>> munin-libvirt-plugins 0.0.6-1
>> pacemaker 1.1.15-2~bpo8+1
>> pacemaker-cli-utils 1.1.15-2~bpo8+1
>> pacemaker-common 1.1.15-2~bpo8+1
>> pacemaker-resource-agents 1.1.15-2~bpo8+1
> 
>> Kernel: #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
> 
> I attached our cib before upgrade and after, as well as the one with the
> mixed syntax and our corosync.conf.
> 
> When we tried to connect a second node to the cluster, pacemaker starts
> it's deamons, starts corosync and dies after 15 tries with following in
> corosync log:
> 
>> crmd: info: crm_timer_popped: Wait Timer (I_NULL) just popped (2000ms)
>> crmd: info: do_cib_control: Could not connect to the CIB service:
>> Transport endpoint is not connected
>> crmd:  warning: do_cib_control:
>> Couldn't complete CIB registration 15 times... pause and retry
>> attrd: error: attrd_cib_connect: Signon to CIB failed:
>> Transport endpoint is not connected (-107)
>> attrd: info: main: Shutting down attribute manager
>> attrd: info: qb_ipcs_us_withdraw: withdrawing server sockets
>> attrd: info: crm_xml_cleanup: Cleaning up memory from libxml2
>> crmd: info: crm_timer_popped: Wait Timer (I_NULL) just popped (2000ms)
>> pacemakerd:  warning: pcmk_child_exit:
>> The attrd process (12761) can no longer be respawned,
>> shutting the cluster down.
>> pacemakerd: notice: pcmk_shutdown_worker: Shutting down Pacemaker
> 
> A third node joins without above error, but crm_mon still shows all
> nodes as offline.
> 
> Thanks for any advice how to solve this, I'm out of ideas now.
> 
> Regards, Toni
> 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] packmaker: After migrate a resource, changing resource non-unique param lead to resoruce restart

[Ken Gaillot]

On 11/03/2016 02:01 AM, 李清硕 wrote:
> Hi everyone,
> I'm testing pacemaker resoruce live migration,  in a simple test
> environment with two virtual machines.
> The resource is something encapsulated kvm, when i perfrom migrate, for
> example, form node1 to node2, 
> i notice, the pacemaker invoke migrate_to, a one-off monitor on node1,
> like this:
>   (migrate_to:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN
>(migrate_to:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 0
>   (monitor:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN  
>   (monitor:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 7
> invoke two one-off monitor on node2 like this:
>   (monitor:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN
>  (monitor:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 0
>(monitor:rgavhcnP-iOgMD4-Bdht)  ACTION BEDIN
>  (monitor:rgavhcnP-iOgMD4-Bdht) ACTION EDN WITH 0
> When i perfrom crm resource operations resoruce_name(actually i don't
> what this is, maybe lead to the resource restart) , the output is: 
>rgavhcnP-iOgMD4-Bdh (ocf::heartbeat:fronvm) Started:
> rgavhcnP-iOgMD4-Bdht_monitor_0 (node=h122, call=41, rc=0,
> last-rc-change=Thu Nov  3 14:33:25 2016, exec=232ms): complete
>rgavhcnP-iOgMD4-Bdht (ocf::heartbeat:fronvm): Started:
> rgavhcnP-iOgMD4-Bdht_monitor_0 (node=h122, call=41, rc=0,
> last-rc-change=Thu Nov  3 14:33:25 2016, exec=232ms): complete
>  
> rgavhcnP-iOgMD4-Bdht (ocf::heartbeat:fronvm): 
> Started:rgavhcnP-iOgMD4-Bdht_monitor_12
> (node=h122, call=42, rc=0, last-rc-change=Thu Nov  3 14:33:34
> 2016,exec=197ms):complete
>   rgavhcnP-iOgMD4-Bdht (ocf::heartbeat:fronvm): Started:
> rgavhcnP-iOgMD4-Bdht_monitor_0 (node=h123, call=35, rc=7,
> last-rc-change=Thu Nov  3 14:34:25 2016, exec=213ms): complete
> The last i change resource params "config" (config is a param name),
> would lead to resource restart, which i expect reload, not restart.
> Is that a normal or a bug or  i do something wrong. if you need more
> detail, please tell me, Any hint will help

Pacemaker should reload a resource after a parameter change if:

* unique=0 in the parameter's metadata in the resource agent ()

* the resource agent advertises support for the reload operation in its
metadata ()

* no unique parameter was also changed at the same time

Are all those true in this case?

> Thank you！
> 
> The pacemaker config:
> node 1084759930: h122
> node 1084759931: h123
> primitive fence stonith:fence_agent \
> params pcmk_host_check=none action=reboot \
> op monitor interval=3600s \
> meta target-role=Started
> primitive rgavhcnP-iOgMD4-Bdht fronvm \
> params config="/mnt/m6dd63be9650f737da8c65d6caa00bc00"
> vm_name=rgavhcnP-iOgMD4-Bdht \
> op start timeout=600s interval=0 \
> op stop timeout=180s interval=0 \
> op migrate_to timeout=1800s interval=0 \
> op migrate_from timeout=60s interval=0 \
> op monitor interval=120s timeout=60s on-fail=ignore \
> meta target-role=Started allow-migrate=true
> clone fence-clone fence \
> meta target-role=Started
> location loc-fence-h122 fence-clone 100: h122
> location loc-fence-h123 fence-clone 100: h123
> location loc-paPupXaC-Ns5mnb-Xdfe-rgavhcnP-iOgMD4-Bdht
> rgavhcnP-iOgMD4-Bdht 1: h122
> location loc-tEJVJGWl-ifcTSp-Ta1v-rgavhcnP-iOgMD4-Bdht
> rgavhcnP-iOgMD4-Bdht 2: h123
> property cib-bootstrap-options: \
> stonith-enabled=true \
> stonith-action=reboot \
> stonith-timeout=120s \
> no-quorum-policy=stop \
> symmetric-cluster=false \
> crmd-transition-delay=5s \
> start-failure-is-fatal=FALSE \
> have-watchdog=false \
> dc-version=1.1.13-10.el7.centos-44eb2dd \
> cluster-infrastructure=corosync \
> cluster-name=OQSQoxu5XrsUcds8 \
> last-lrm-refresh=1478154860
> rsc_defaults rsc-options: \
> resource-stickiness=1024 \
> migration-threshold=3
> 
> The log:
>  Nov 03 14:34:29 [87769] h123cib: (  cib_file.c:293   )info:
> cib_file_backup:Archived previous version as
> /var/lib/pacemaker/cib/cib-83.raw
> Nov 03 14:34:29 [87769] h123cib: (  cib_file.c:423   )info:
> cib_file_write_with_digest:Wrote version 0.494.0 of the CIB to disk
> (digest: 97a2f450d62400802a2e04cf20ae010c)
> Nov 03 14:34:29 [87769] h123cib: (  cib_file.c:442   )info:
> cib_file_write_with_digest:Reading cluster configuration file
> /var/lib/pacemaker/cib/cib.lTaDlA (digest:
> /var/lib/pacemaker/cib/cib.IlPpJD)
> Nov 03 14:34:34 [87769] h123cib: (  messages.c:239   )info:
> cib_process_ping:Reporting our current digest to h122:
> 01ebcac1c89e4493808c256c6c4b86a4 for 0.494.0 (0x2265720 0)
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1660  )info:
> cib_perform_op:Diff: --- 0.494.0 2
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1662  )info:
> cib_perform_op:Diff: +++ 0.494.1 (null)
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1728  )info:
> cib_perform_op:+  /cib:  @num_updates=1
> Nov 03 14:34:34 [87769] h123cib: (   xml.c:1684  )info:
> cib_perform_op:++
> /cib/status/node_state[@id='1084759930']/lrm[@id='1084759930

[ClusterLabs] Fix in Pacemaker 1.1.15 retroactively assigned CVE-2016-7797

[Ken Gaillot]

Hello all,

Pacemaker 1.1.15, released earlier this year, contained a fix for a
potential denial-of-service vulnerability in pacemaker_remote. This
vulnerability has been retroactively assigned the Common Vulnerabilities
and Exposures identifier CVE-2016-7797.

This was mentioned in the 1.1.15 release notes, but is being raised
again for anyone interested in the CVE ID, such as distribution packagers.

Before Pacemaker 1.1.15, an unprivileged user able to attempt connection
to the IP address and port used for an active Pacemaker Remote
connection could trivially force the connection to drop. The
vulnerability only affects clusters with Pacemaker Remote nodes.

For details, see:

  http://bugs.clusterlabs.org/show_bug.cgi?id=5269

-- 
Ken Gaillot 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [SECURITY] CVE-2016-7035 - pacemaker - improper IPC guarding

[Ken Gaillot]

On 11/03/2016 06:03 AM, Jan Pokorný wrote:
> Following issue is being publicly disclosed today; more information
> regarding the release process will arrive later today and also this
> is an opportunity to announce http://clusterlabs.org/wiki/Security
> page that was intoduced to help keeping track of security issues
> (any fellow project is welcome to use that as well, Andrew or Ken
> can make and account on the wiki on your behalf).
> 
> It was discovered that at some not so uncommon circumstances, some
> pacemaker daemons could be talked to, via libqb-facilitated IPC, by
> unprivileged clients due to flawed authorization decision.  Depending
> on the capabilities of affected daemons, this might equip unauthorized
> user with local privilege escalation or up to cluster-wide remote
> execution of possibly arbitrary commands when such user happens to
> reside at standard or remote/guest cluster node, respectively.
> 
> The original vulnerability was introduced in an attempt to allow
> unprivileged IPC clients to clean up the file system materialized
> leftovers in case the server (otherwise responsible for the lifecycle
> of these files) crashes.  While the intended part of such behavior is
> now effectively voided (along with the unintended one), a best-effort
> fix to address this corner case systemically at libqb is coming along
> (https://github.com/ClusterLabs/libqb/pull/231).
> 
> Affected versions:  1.1.10-rc1 (2013-04-17) - 1.1.15 (2016-06-21)
> Impact: Important
> CVSSv3 ranking: 8.8 : AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H
> 
> Credits for independent findings, in chronological order:
>   Jan "poki" Pokorný, of Red Hat
>   Alain Moulle, of ATOS/BULL
> 
> 
> Patch for the issue, which is applicable on all affected versions:
> https://github.com/ClusterLabs/pacemaker/pull/1166/commits/5a20855d6054ebaae590c09262b328d957cc1fc2

More details:

A fix has been applied to Pacemaker's upstream master branch. Also, we
are starting the release process for 1.1.16 today, so the fix is
included in the 1.1 branch as well. It is additionally attached in patch
format to this message.

Anyone who has built one of the affected Pacemaker versions from source
is strongly encouraged to apply the patch, or rebuild from a current
branch. Popular distributions are expected to have patched packages
available soon (some already released today).

>From edba45825a11513661840fbc8c4b81f607fed18b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jan=20Pokorn=C3=BD?= 
Date: Tue, 23 Aug 2016 18:09:49 +0200
Subject: [PATCH] High: libcrmcommon: fix CVE-2016-7035 (improper IPC guarding)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

It was discovered that at some not so uncommon circumstances, some
pacemaker daemons could be talked to, via libqb-facilitated IPC, by
unprivileged clients due to flawed authorization decision.  Depending
on the capabilities of affected daemons, this might equip unauthorized
user with local privilege escalation or up to cluster-wide remote
execution of possibly arbitrary commands when such user happens to
reside at standard or remote/guest cluster node, respectively.

The original vulnerability was introduced in an attempt to allow
unprivileged IPC clients to clean up the file system materialized
leftovers in case the server (otherwise responsible for the lifecycle
of these files) crashes.  While the intended part of such behavior is
now effectively voided (along with the unintended one), a best-effort
fix to address this corner case systemically at libqb is coming along.

Affected versions:  1.1.10-rc1 (2013-04-17) - 1.1.15 (2016-06-21)
Impact: Important
CVSSv3 ranking: 8.8 : AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H

Credits for independent findings, in chronological order:
  Jan "poki" Pokorný, of Red Hat
  Alain Moulle, of ATOS/BULL
---
 lib/common/ipc.c | 14 +++---
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/lib/common/ipc.c b/lib/common/ipc.c
index 6d6d3cd..9f63dfe 100644
--- a/lib/common/ipc.c
+++ b/lib/common/ipc.c
@@ -293,7 +293,6 @@ crm_client_disconnect_all(qb_ipcs_service_t *service)
 crm_client_t *
 crm_client_new(qb_ipcs_connection_t * c, uid_t uid_client, gid_t gid_client)
 {
-static uid_t uid_server = 0;
 static gid_t gid_cluster = 0;
 
 crm_client_t *client = NULL;
@@ -304,7 +303,6 @@ crm_client_new(qb_ipcs_connection_t * c, uid_t uid_client, gid_t gid_client)
 }
 
 if (gid_cluster == 0) {
-uid_server = getuid();
 if(crm_user_lookup(CRM_DAEMON_USER, NULL, &gid_cluster) < 0) {
 static bool have_error = FALSE;
 if(have_error == FALSE) {
@@ -314,16 +312,10 @@ crm_client_new(qb_ipcs_connection_t * c, uid_t uid_client, gid_t gid_client)
 }
 }
 
-if(gid_cluster != 0 && gid_client != 0) {
-uid_t best_uid = -1; /* Passing -1 to chown(2) means don't change */
-
-if(uid_client == 0 || uid_server == 0) { /* Someone is priv

Re: [ClusterLabs] Coming in 1.1.16: versioned resource parameters

[Ken Gaillot]

> With a new feature in the current master branch, which will be part of
> the next Pacemaker release, you will be able to specify different
> resource parameters to be used with different versions of a resource agent.

FYI, this feature is being held back to 1.1.17, due to a planned
reimplementation that adds functionality and handles rolling upgrades
better. It will still be available in the master branch, but not the 1.1
branch.
-- 
Ken Gaillot 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

ClusterLabs is happy to announce the first release candidate for
Pacemaker version 1.1.16. Source code is available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc1

The most significant enhancements in this release are:

* rsc-pattern may now be used instead of rsc in location constraints, to
allow a single location constraint to apply to all resources whose names
match a regular expression. Sed-like %0 - %9 backreferences let
submatches be used in node attribute names in rules.

* The new ocf:pacemaker:attribute resource agent sets a node attribute
according to whether the resource is running or stopped. This may be
useful in combination with attribute-based rules to model dependencies
that simple constraints can't handle.

* Pacemaker's existing "node health" feature allows resources to move
off nodes that become unhealthy. Now, when using
node-health-strategy=progressive, a new cluster property
node-health-base will be used as the initial health score of newly
joined nodes (defaulting to 0, which is the previous behavior). This
allows cloned and multistate resource instances to start on a node even
if it has some "yellow" health attributes.

* Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
properly passed to multistate resources with notification enabled. This
has been fixed. To help resource agents detect when the fix is
available, the CRM feature set has been incremented. (Whenever the
feature set changes, mixed-version clusters are supported only during
rolling upgrades -- nodes with an older version will not be allowed to
rejoin once they shut down.)

* Watchdog-based fencing using sbd now works on remote nodes.

* The build process now takes advantage of various compiler features
(RELRO, PIE, as-needed linking, etc.) that enhance security and start-up
performance. See the "Hardening flags" comments in the configure.ac file
for more details.

* Python 3 compatibility: The Pacemaker project now targets
compatibility with both python 2 (versions 2.6 and later) and python 3
(versions 3.2 and later). All of the project's python code now meets
this target, with the exception of CTS, which is still python 2 only.

* The Pacemaker coding guidelines have been replaced by a more
comprehensive addition to the documentation set, "Pacemaker
Development". It is intended for developers working on the Pacemaker
code base itself, rather than external code such as resource agents. A
copy is viewable at
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Development/index.html

As usual, the release includes many bugfixes, including a fix for a
serious security vulnerability (CVE-2016-7035). For a more detailed list
of changes, see the change log:

https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog

Everyone is encouraged to download, compile and test the new release. We
do many regression tests and simulations, but we can't cover all
possible use cases, so your feedback is important and appreciated. Due
to the security fix, I intend to keep this release cycle short, so quick
testing feedback is especially appreciated.

Many thanks to all contributors of source code to this release,
including Andrew Beekhof, Bin Liu, Christian Schneider, Christoph Berg,
David Shane Holden, Ferenc Wágner, Yan Gao, Hideo Yamauchi, Jan Pokorný,
Ken Gaillot, Klaus Wenninger, Kostiantyn Ponomarenko, Kristoffer
Grönlund, Lars Ellenberg, Masatake Yamato, Michal Koutný, Nakahira
Kazutomo, Nate Clark, Nishanth Aravamudan, Oyvind Albrigtsen, Ruben
Kerkhof, Tim Bishop, Vladislav Bogdanov and Yusuke Iida. Apologies if I
have overlooked anyone.
-- 
Ken Gaillot 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/04/2016 02:53 AM, Jan Pokorný wrote:
> On 04/11/16 08:29 +0100, Ulrich Windl wrote:
>> Ken Gaillot  schrieb am 03.11.2016 um 17:08 in
>> Nachricht <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>:
>>> ClusterLabs is happy to announce the first release candidate for
>>> Pacemaker version 1.1.16. Source code is available at:
>>>
>>> https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc1 
>>>
>>> The most significant enhancements in this release are:
>>>
>>> [...]
>>>
>>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
>>> properly passed to multistate resources with notification enabled. This
>>> has been fixed. To help resource agents detect when the fix is
>>> available, the CRM feature set has been incremented. (Whenever the
>>> feature set changes, mixed-version clusters are supported only during
>>> rolling upgrades -- nodes with an older version will not be allowed to
>>> rejoin once they shut down.)
>>
>> Where can I find a description of current "CRM feature sets"?
> 
> It's originally internal-only versioning for cluster to know which
> node has the oldest software and hence is predestined to be DC
> (in rolling update scenario).
> 
> Ken recently mapped these versions (together with LRMD protocol
> versions relevant in the context of pacemaker remote communication)
> to proper release versions: http://clusterlabs.org/wiki/ReleaseCalendar

There is also a description of how the cluster uses the CRM feature set
in the new upgrade documentation:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_rolling_node_by_node



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/04/2016 02:29 AM, Ulrich Windl wrote:
>>>> Ken Gaillot  schrieb am 03.11.2016 um 17:08 in
> Nachricht
> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>:
>> ClusterLabs is happy to announce the first release candidate for
>> Pacemaker version 1.1.16. Source code is available at:
>>
>> https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc1 
>>
>> The most significant enhancements in this release are:
>>
>> * rsc-pattern may now be used instead of rsc in location constraints, to
>> allow a single location constraint to apply to all resources whose names
>> match a regular expression. Sed-like %0 - %9 backreferences let
>> submatches be used in node attribute names in rules.
>>
>> * The new ocf:pacemaker:attribute resource agent sets a node attribute
>> according to whether the resource is running or stopped. This may be
>> useful in combination with attribute-based rules to model dependencies
>> that simple constraints can't handle.
> 
> I don't quite understand this: Isn't the state of a resource in the CIB status
> section anyway? If not, why not add it? So it would be readily available for
> anyone (rules, constraints, etc.).

This (hopefully) lets you model more complicated relationships.

For example, someone recently asked whether they could make an ordering
constraint apply only at "start-up" -- the first time resource A starts,
it does some initialization that B needs, but once that's done, B can be
independent of A.

For that case, you could group A with an ocf:pacemaker:attribute
resource. The important part is that the attribute is not set if A has
never run on a node. So, you can make a rule that B can run only where
the attribute is set, regardless of the value -- even if A is later
stopped, the attribute will still be set.

Another possible use would be for a cron that needs to know whether a
particular resource is running, and an attribute query is quicker and
easier than something like parsing crm_mon output or probing the service.

It's all theoretical at this point, and I'm not entirely sure those
examples would be useful :) but I wanted to make the agent available for
people to experiment with.

>> * Pacemaker's existing "node health" feature allows resources to move
>> off nodes that become unhealthy. Now, when using
>> node-health-strategy=progressive, a new cluster property
>> node-health-base will be used as the initial health score of newly
>> joined nodes (defaulting to 0, which is the previous behavior). This
>> allows cloned and multistate resource instances to start on a node even
>> if it has some "yellow" health attributes.
> 
> So the node health is more or less a "node score"? I don't understand the last
> sentence. Maybe give an example?

Yes, node health is a score that's added when deciding where to place a
resource. It does get complicated ...

Node health monitoring is optional, and off by default.

Node health attributes are set to red, yellow or green (outside
pacemaker itself -- either by a resource agent, or some external
process). As an example, let's say we have three node health attributes
for CPU usage, CPU temperature, and SMART error count.

With a progressive strategy, red and yellow are assigned some negative
score, and green is 0. In our example, let's say yellow gets a -10 score.

If any of our attributes are yellow, resources will avoid the node
(unless they have higher positive scores from something like stickiness
or a location constraint).

Normally, this is what you want, but if your resources are cloned on all
nodes, maybe you don't care if some attributes are yellow. In that case,
you can set node-health-base=20, so even if two attributes are yellow,
it won't prevent resources from running (20 + -10 + -10 = 0).

There is nothing in node-health-base that is specific to clones; that's
just the most likely use case.

>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
>> properly passed to multistate resources with notification enabled. This
>> has been fixed. To help resource agents detect when the fix is
>> available, the CRM feature set has been incremented. (Whenever the
>> feature set changes, mixed-version clusters are supported only during
>> rolling upgrades -- nodes with an older version will not be allowed to
>> rejoin once they shut down.)
> 
> Where can I find a description of current "CRM feature sets"?
>
> Ulrich
> 
>>
>> * Watchdog-based fencing using sbd now works on remote nodes.
>>
>> * The build process now takes advantage of various compiler features
>> (RELRO, PIE, as-needed linking, etc.) that 

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/07/2016 01:41 AM, Ulrich Windl wrote:
>>>> Ken Gaillot  schrieb am 04.11.2016 um 22:37 in 
>>>> Nachricht
> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>:
>> On 11/04/2016 02:29 AM, Ulrich Windl wrote:
>>>>>> Ken Gaillot  schrieb am 03.11.2016 um 17:08 in
>>>> * The new ocf:pacemaker:attribute resource agent sets a node attribute
>>>> according to whether the resource is running or stopped. This may be
>>>> useful in combination with attribute-based rules to model dependencies
>>>> that simple constraints can't handle.
>>>
>>> I don't quite understand this: Isn't the state of a resource in the CIB 
>> status
>>> section anyway? If not, why not add it? So it would be readily available for
>>> anyone (rules, constraints, etc.).
>>
>> This (hopefully) lets you model more complicated relationships.
>>
>> For example, someone recently asked whether they could make an ordering
>> constraint apply only at "start-up" -- the first time resource A starts,
>> it does some initialization that B needs, but once that's done, B can be
>> independent of A.
> 
> Is "at start-up" before start of the resource, after start of the resource, 
> or parallel to the start of the resource ;-)
> Probably a "hook" in the corresponding RA is the better approach, unless you 
> can really model all of the above.
> 
>>
>> For that case, you could group A with an ocf:pacemaker:attribute
>> resource. The important part is that the attribute is not set if A has
>> never run on a node. So, you can make a rule that B can run only where
>> the attribute is set, regardless of the value -- even if A is later
>> stopped, the attribute will still be set.
> 
> If a resource is not running on a node,, it is "stopped"; isn't it?

Sure, but what I mean is, if resource A has *never* run on a node, then
the corresponding node attribute will be *unset*. But if A has ever
started and/or stopped on a node, the attribute will be set to one value
or the other. So, a rule can be used to check whether the attribute is
set, to determine whether A has *ever* been run on the node, regardless
of whether it is currently running.

>> Another possible use would be for a cron that needs to know whether a
>> particular resource is running, and an attribute query is quicker and
>> easier than something like parsing crm_mon output or probing the service.
> 
> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so besides 
> of lacking options and inefficient implementation, why should one be faster 
> than the other?
> 
>>
>> It's all theoretical at this point, and I'm not entirely sure those
>> examples would be useful :) but I wanted to make the agent available for
>> people to experiment with.
> 
> A good product manager should resist the attempt to provide any feature the 
> customers ask for, avoiding bloat-ware. That is to protect the customer from 
> their own bad decisions. In most cases there is a better, more universal 
> solution to the specific problem.

Sure, but this is a resource agent -- it adds no overhead to anyone not
using it, and since we don't have any examples or walk-throughs using
it, users would have to investigate and experiment to see whether it's
of any use in their environment.

Hopefully, this will turn out to be a general-purpose tool of value to
multiple problem scenarios.

>>>> * Pacemaker's existing "node health" feature allows resources to move
>>>> off nodes that become unhealthy. Now, when using
>>>> node-health-strategy=progressive, a new cluster property
>>>> node-health-base will be used as the initial health score of newly
>>>> joined nodes (defaulting to 0, which is the previous behavior). This
>>>> allows cloned and multistate resource instances to start on a node even
>>>> if it has some "yellow" health attributes.
>>>
>>> So the node health is more or less a "node score"? I don't understand the 
>> last
>>> sentence. Maybe give an example?
>>
>> Yes, node health is a score that's added when deciding where to place a
>> resource. It does get complicated ...
>>
>> Node health monitoring is optional, and off by default.
>>
>> Node health attributes are set to red, yellow or green (outside
>> pacemaker itself -- either by a resource agent, or some external
>> process). As an example, let's say we have three node health attributes
>> for CPU usage, CPU temperature, and SMART erro

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/07/2016 03:47 AM, Klaus Wenninger wrote:
> On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote:
>> On Mon, 7 Nov 2016 10:12:04 +0100
>> Klaus Wenninger  wrote:
>>
>>> On 11/07/2016 08:41 AM, Ulrich Windl wrote:
>>>>>>> Ken Gaillot  schrieb am 04.11.2016 um 22:37 in
>>>>>>> Nachricht  
>>>> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>:  
>>>>> On 11/04/2016 02:29 AM, Ulrich Windl wrote:  
>>>>>>>>> Ken Gaillot  schrieb am 03.11.2016 um 17:08 in  
>>>>>> Nachricht
>>>>>> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>:  
>> ...
>>>>> Another possible use would be for a cron that needs to know whether a
>>>>> particular resource is running, and an attribute query is quicker and
>>>>> easier than something like parsing crm_mon output or probing the service. 
>>>>>  
>>>> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so
>>>> besides of lacking options and inefficient implementation, why should one
>>>> be faster than the other?  
>>> attrd_updater doesn't go for the CIB
>> AFAIK, attrd_updater actually goes to the CIB, unless you set "--private"
>> since 1.1.13:
>> https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177
> That prevents values being stored in the CIB. attrd_updater should
> always talk to attrd as I got it ...

It's a bit confusing: Both crm_attribute and attrd_updater will
ultimately affect both attrd and the CIB in most cases, but *how* they
do so is different. crm_attribute modifies the CIB, and lets attrd pick
up the change from there; attrd_updater notifies attrd, and lets attrd
modify the CIB.

The difference is subtle.

With corosync 2, attrd only modifies "transient" node attributes (which
stay in effect till the next reboot), not "permanent" attributes. So
crm_attribute must be used if you want to set a permanent attribute.
crm_attribute also has the ability to modify cluster properties and
resource defaults, as well as node attributes.

On the other hand, by contacting attrd directly, attrd_updater can
change an attribute's "dampening" (how often it is flushed to the CIB),
and it can (as mentioned above) set "private" attributes that are never
written to the CIB (and thus never cause the cluster to re-calculate
resource placement).

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/07/2016 12:03 PM, Jehan-Guillaume de Rorthais wrote:
> On Mon, 7 Nov 2016 09:31:20 -0600
> Ken Gaillot  wrote:
> 
>> On 11/07/2016 03:47 AM, Klaus Wenninger wrote:
>>> On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote:  
>>>> On Mon, 7 Nov 2016 10:12:04 +0100
>>>> Klaus Wenninger  wrote:
>>>>  
>>>>> On 11/07/2016 08:41 AM, Ulrich Windl wrote:  
>>>>>>>>> Ken Gaillot  schrieb am 04.11.2016 um 22:37 in
>>>>>>>>> Nachricht
>>>>>> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>:
>>>>>>> On 11/04/2016 02:29 AM, Ulrich Windl wrote:
>>>>>>>>>>> Ken Gaillot  schrieb am 03.11.2016 um 17:08
>>>>>>>>>>> in
>>>>>>>> Nachricht
>>>>>>>> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>:
>>>> ...  
>>>>>>> Another possible use would be for a cron that needs to know whether a
>>>>>>> particular resource is running, and an attribute query is quicker and
>>>>>>> easier than something like parsing crm_mon output or probing the
>>>>>>> service.
>>>>>> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so
>>>>>> besides of lacking options and inefficient implementation, why should one
>>>>>> be faster than the other?
>>>>> attrd_updater doesn't go for the CIB  
>>>> AFAIK, attrd_updater actually goes to the CIB, unless you set "--private"
>>>> since 1.1.13:
>>>> https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177  
>>> That prevents values being stored in the CIB. attrd_updater should
>>> always talk to attrd as I got it ...  
>>
>> It's a bit confusing: Both crm_attribute and attrd_updater will
>> ultimately affect both attrd and the CIB in most cases, but *how* they
>> do so is different. crm_attribute modifies the CIB, and lets attrd pick
>> up the change from there; attrd_updater notifies attrd, and lets attrd
>> modify the CIB.
>>
>> The difference is subtle.
>>
>> With corosync 2, attrd only modifies "transient" node attributes (which
>> stay in effect till the next reboot), not "permanent" attributes.
> 
> So why "--private" is not compatible with corosync 1.x as attrd_updater only 
> set
> "transient" attributes anyway?

Corosync 1 does not support certain reliability guarantees required by
the current attrd, so when building against the corosync 1 libraries,
pacemaker will install "legacy" attrd instead. The difference is mainly
that the current attrd can guarantee atomic updates to attribute values.
attrd_updater actually can set permanent attributes when used with
legacy attrd.

> How and where private attributes are stored?

They are kept in memory only, in attrd. Of course, attrd is clustered,
so they are kept in sync across all nodes.

>> So crm_attribute must be used if you want to set a permanent attribute.
>> crm_attribute also has the ability to modify cluster properties and
>> resource defaults, as well as node attributes.
>>
>> On the other hand, by contacting attrd directly, attrd_updater can
>> change an attribute's "dampening" (how often it is flushed to the CIB),
>> and it can (as mentioned above) set "private" attributes that are never
>> written to the CIB (and thus never cause the cluster to re-calculate
>> resource placement).
> 
> Interesting, thank you for the clarification.
> 
> As I understand it, it resumes to:
> 
>   crm_attribute -> CIB <-(poll/notify?) attrd
>   attrd_updater -> attrd -> CIB

Correct. On startup, attrd registers with CIB to be notified of all changes.

> Just a quick question about this, is it possible to set a "dampening" high
> enough so attrd never flush it to the CIB (kind of private attribute too)?

I'd expect that to work, if the dampening interval was higher than the
lifetime of the cluster being up.

It's also possible to abuse attrd to create a kind of private attribute
by using a node name that doesn't exist and never will. :) This ability
is intentionally allowed, so you can set attributes for nodes that the
current partition isn't aware of, or nodes that are planned to be added
later, but only attributes for known nodes will be written to the CIB.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] ocf:lvm2:VolumeGroup Probe Issue

[Ken Gaillot]

On 11/08/2016 10:37 AM, Marc Smith wrote:
> Hi,
> 
> First, I realize ocf:lvm2:VolumeGroup comes from the LVM2 package and
> not resource-agents, but I'm hoping someone on this list is familiar
> with this RA and can provide some insight.
> 
> In my cluster configuration, I'm using ocf:lvm2:VolumeGroup to manage
> my LVM VG's, and I'm using the cluster to manage DLM and CLVM. I have
> my constraints in place and everything seems to be working mostly,
> except I'm hitting a glitch with ocf:lvm2:VolumeGroup and the initial
> probe operation.
> 
> On startup, a probe operation (monitor) is issued for all of the
> resources, but ocf:lvm2:VolumeGroup is returning OCF_ERR_GENERIC in
> VolumeGroup_status() (via VolumeGroup_monitor()) since clvmd hasn't
> started yet... this line in VolumeGroup_status() is the trouble:
> 
> VGOUT=`vgdisplay -v $OCF_RESKEY_volgrpname 2>&1` || exit $OCF_ERR_GENERIC
> 
> When clvmd is not running, 'vgdisplay -v name' will always return
> something like this:
> 
> --snip--
>   connect() failed on local socket: No such file or directory
>   Internal cluster locking initialisation failed.
>   WARNING: Falling back to local file-based locking.
>   Volume Groups with the clustered attribute will be inaccessible.
> VG name on command line not found in list of VGs: biggie
>   Volume group "biggie" not found
>   Cannot process volume group biggie
> --snip--
> 
> And exits with a status of 5. So, my question is, do I patch the RA?
> Or is there some cluster constraint I can add so a probe/monitor
> operation isn't performed for the VolumeGroup resource until CLVM has
> been started?

Ordered probes is a desired feature for Pacemaker, but the
implementation is much trickier than it appears at first glance, so
there's no timeline for when it might arrive.

In the meantime, patching the RA to exit with "not running" in this
situation is probably the best workaround. There is a library function
ocf_is_probe you can use to avoid messing with "real" monitors.

> Any other advice? Is ocf:heartbeat:LVM or ocf:lvm2:VolumeGroup the
> more popular RA for managing LVM VG's? Any comments from other users
> on experiences using either (good, bad)? Both appear to achieve the
> same function, just a bit differently.
> 
> 
> Thanks,
> 
> Marc

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Pacemaker 1.1.16 - Release Candidate 1

[Ken Gaillot]

On 11/08/2016 03:02 AM, Ulrich Windl wrote:
>>>> Ken Gaillot  schrieb am 07.11.2016 um 16:15 in 
>>>> Nachricht
>>>>>> * Pacemaker's existing "node health" feature allows resources to move
>>>>>> off nodes that become unhealthy. Now, when using
>>>>>> node-health-strategy=progressive, a new cluster property
>>>>>> node-health-base will be used as the initial health score of newly
>>>>>> joined nodes (defaulting to 0, which is the previous behavior). This
>>>>>> allows cloned and multistate resource instances to start on a node even
>>>>>> if it has some "yellow" health attributes.
>>>>>
>>>>> So the node health is more or less a "node score"? I don't understand the 
>>>> last
>>>>> sentence. Maybe give an example?
>>>>
>>>> Yes, node health is a score that's added when deciding where to place a
>>>> resource. It does get complicated ...
>>>>
>>>> Node health monitoring is optional, and off by default.
>>>>
>>>> Node health attributes are set to red, yellow or green (outside
>>>> pacemaker itself -- either by a resource agent, or some external
>>>> process). As an example, let's say we have three node health attributes
>>>> for CPU usage, CPU temperature, and SMART error count.
>>>>
>>>> With a progressive strategy, red and yellow are assigned some negative
>>>> score, and green is 0. In our example, let's say yellow gets a -10 score.
>>>>
>>>> If any of our attributes are yellow, resources will avoid the node
>>>> (unless they have higher positive scores from something like stickiness
>>>> or a location constraint).
>>>>
>>>
>>> I understood so far.
>>>
>>>> Normally, this is what you want, but if your resources are cloned on all
>>>> nodes, maybe you don't care if some attributes are yellow. In that case,
>>>> you can set node-health-base=20, so even if two attributes are yellow,
>>>> it won't prevent resources from running (20 + -10 + -10 = 0).
>>>
>>> I don't understand that: "node-health-base" is a global setting, but what 
>>> you 
>> want is an exception for some specific (clone) resource.
>>> To me the more obvious solution would be to provide an exception rule for 
>> the resource, not a global setting for the node.
>>
>> The main advantage of node-health-base over other approaches -- such as
>> defining a constant #health-base attribute for all nodes, or defining
>> positive location constraints for each resource on each node -- is that
>> node-health-base applies to all resources and nodes, present and future.
>> If someone adds a node to the cluster, it will automatically get
>> node-health-base when it joins, whereas any other approach requires
>> additional configuration changes (which leaves a window where the value
>> is not applied).
> 
> So the node-health-base is a default value for the node until it will be 
> explicitly set? Do you try to handle the problem "all nodes are to be assumed 
> bad until proven to be good"? Are we maybe fighting a completely different 
> problem (with some RAs)?

node-health-base is a sort of default health value, but node-health is
never explicitly set -- it's the sum of node-health-base and the
adjustments for each health attribute.

node-health-base could be used for the "assumed bad" approach: you could
set node-health-base to a negative value, and set green to a positive
value (rather than 0, which is its default). Then, each green attribute
would eat away at the deficit.

>>
>> It also simplifies the configuration the more nodes/resources you have,
>> and is less prone to accidental configuration mistakes.
>>
>> The idea is straightforward: instead of each node starting with a health
>> score of 0 (which means any negative health attribute will push all
>> resources away), start each node with a positive health score, so that
>> health has to drop below a certain point before affecting resources.
> 
> I don't see the difference between "starting at 0, substracting a small 
> score" and "staring at some positive, subtracting a large score": You are 
> saying that any negative score will move all resources away? I thought it 
> only happens on -INFINITY.

Pacemaker always combines scores from all sources and uses the final
value to decide resource placement

Re: [ClusterLabs] What is the logic when two node are down at the same time and needs to be fenced

[Ken Gaillot]

On 11/07/2016 09:59 AM, Niu Sibo wrote:
> Hi Ken,
> 
> Thanks for the clarification. Now I have another real problem that needs
> your advise.
> 
> The cluster consists of 5 nodes and one of the node got a 1 second
> network failure which resulted in one of the VirtualDomain resources to
> start on two nodes at the same time. The cluster property
> no_quorum_policy is set to stop.
> 
> At 16:13:34, this happened:
> 16:13:34 zs95kj attrd[133000]:  notice: crm_update_peer_proc: Node
> zs93KLpcs1[5] - state is now lost (was member)
> 16:13:34 zs95kj corosync[132974]:  [CPG   ] left_list[0]
> group:pacemakerd\x00, ip:r(0) ip(10.20.93.13) , pid:28721
> 16:13:34 zs95kj crmd[133002]: warning: No match for shutdown action on 5
> 16:13:34 zs95kj attrd[133000]:  notice: Removing all zs93KLpcs1
> attributes for attrd_peer_change_cb
> 16:13:34 zs95kj corosync[132974]:  [CPG   ] left_list_entries:1
> 16:13:34 zs95kj crmd[133002]:  notice: Stonith/shutdown of zs93KLpcs1
> not matched
> ...
> 16:13:35 zs95kj attrd[133000]:  notice: crm_update_peer_proc: Node
> zs93KLpcs1[5] - state is now member (was (null))
> 
> From the DC:
> [root@zs95kj ~]# crm_simulate --xml-file
> /var/lib/pacemaker/pengine/pe-input-3288.bz2 |grep 110187
>  zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Started
> zs93KLpcs1 <--This is the baseline that everything works normal
> 
> [root@zs95kj ~]# crm_simulate --xml-file
> /var/lib/pacemaker/pengine/pe-input-3289.bz2 |grep 110187
>  zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped
> <--- Here the node zs93KLpcs1 lost it's network for 1 sec and
> resulted in this state.

This is unexpected. Can you open a bug report at
http://bugs.clusterlabs.org/ and attach the full logs from around this
time, and maybe the above two pe-input files?

> [root@zs95kj ~]# crm_simulate --xml-file
> /var/lib/pacemaker/pengine/pe-input-3290.bz2 |grep 110187
>  zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped
> 
> [root@zs95kj ~]# crm_simulate --xml-file
> /var/lib/pacemaker/pengine/pe-input-3291.bz2 |grep 110187
>  zs95kjg110187_res  (ocf::heartbeat:VirtualDomain): Stopped
> 
> 
> From the DC's pengine log, it has:
> 16:05:01 zs95kj pengine[133001]:  notice: Calculated Transition 238:
> /var/lib/pacemaker/pengine/pe-input-3288.bz2
> ...
> 16:13:41 zs95kj pengine[133001]:  notice: Start
> zs95kjg110187_res#011(zs90kppcs1)
> ...
> 16:13:41 zs95kj pengine[133001]:  notice: Calculated Transition 239:
> /var/lib/pacemaker/pengine/pe-input-3289.bz2
> 
> From the DC's CRMD log, it has:
> Sep  9 16:05:25 zs95kj crmd[133002]:  notice: Transition 238
> (Complete=48, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-3288.bz2): Complete
> ...
> Sep  9 16:13:42 zs95kj crmd[133002]:  notice: Initiating action 752:
> start zs95kjg110187_res_start_0 on zs90kppcs1
> ...
> Sep  9 16:13:56 zs95kj crmd[133002]:  notice: Transition 241
> (Complete=81, Pending=0, Fired=0, Skipped=172, Incomplete=341,
> Source=/var/lib/pacemaker/pengine/pe-input-3291.bz2): Stopped
> 
> Here I do not see any log about pe-input-3289.bz2 and pe-input-3290.bz2.
> Why is this?
> 
> From the log on zs93KLpcs1 where guest 110187 was running, i do not see
> any message regarding stopping this resource after it lost its
> connection to the cluster.
> 
> Any ideas where to look for possible cause?
> 
> On 11/3/2016 1:02 AM, Ken Gaillot wrote:
>> On 11/02/2016 11:17 AM, Niu Sibo wrote:
>>> Hi all,
>>>
>>> I have a general question regarding the fence login in pacemaker.
>>>
>>> I have setup a three nodes cluster with Pacemaker 1.1.13 and cluster
>>> property no_quorum_policy set to ignore. When two nodes lost their NIC
>>> corosync is running on at the same time, it looks like the two nodes are
>>> getting fenced one by one, even I have three fence devices defined for
>>> each of the node.
>>>
>>> What should I be expecting in the case?
>> It's probably coincidence that the fencing happens serially; there is
>> nothing enforcing that for separate fence devices. There are many steps
>> in a fencing request, so they can easily take different times to
>> complete.
>>
>>> I noticed if the node rejoins the cluster before the cluster starts the
>>> fence actions, some resources will get activated on 2 nodes at the
>>> sametime. This is really not good if the resource happens to be
>>> VirutalGuest.  Thanks for any suggestions.
>> Since you're ignoring quorum, there's nothing stopping the disconnected
>> node from starting all resou

Re: [ClusterLabs] Preventing switchover in case of failing ping node

[Ken Gaillot]

On 11/03/2016 08:49 AM, Detlef Gossrau wrote:
> Hi all,
> 
> is it possible to prevent a switchover in a active/passive cluster if a
> ping node completely fails ?
> 
> Situation:
> 
> A ping node is put into maintenance and not reachable for a certain
> time. The cluster nodes getting the information about the failing ping
> node at different times. For example, node 1 - which is the passive node
> - is getting the information earlier than node 2. As a result the
> resources are moved to node 2. But in the case the ping node is not
> available at all, no switchover should be executed. Is it possible to
> prevent this ?

There's no way for one node to predict that all other nodes will also be
unable to see the target. The only workarounds I see are:

* Use a second ping target on a different path, if available

* If the maintenance is planned, you could disable the relevant
constraints during the window

> 
> Thanks for any help !
> 
> Best regards,
> Detlef


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Live migration not working on shutdown

[Ken Gaillot]

On 11/04/2016 05:51 AM, IT Nerb GmbH wrote:
> Zitat von Klaus Wenninger :
> 
>> On 11/02/2016 06:32 PM, Ken Gaillot wrote:
>>> On 10/26/2016 06:12 AM, Rainer Nerb wrote:
>>>> Hello all,
>>>>
>>>> we're currently testing a 2-node-cluster with 2 vms and live migration
>>>> on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
>>>> migration via ssh-method.
>>>>
>>>> Live migration works, if we issue "pcs resource move ...", "pcs cluster
>>>> standby", "pcs cluster stop" and even "systemctl rescue".
>>>> The latter only worked, after adding the following additional
>>>> dependencies to pacemaker.service and leaving the management of those
>>>> services to systemd:
>>>>
>>>>   * After/Requires=systemd-machined.service
>>>>   * After/Requires=systemd-machine-id-commit.service
>>>>   * After/Requires=remote-fs.target
>>>>   * After/Requires=libvirtd.service
>>>>   * After/Requires=iscsi.service
>>>>   * After/Requires=iscsid.service
>>>>   * After/Requires=sshd.service
>>> This makes sense when clustered resources depend on services that aren't
>>> themselves managed by the cluster. It's dependent on your situation, so
>>> it's not something that pacemaker can solve generically.
> First approach was to use systemd-resources as there are no ocf:
> resource-agents for iSCSI-Initiators or libvirtd in our distribution.
> But then migration failed even on "systemctl rescue".
>>>
>>> You may already be aware, but the easiest way to add such requirements
>>> is to put them in a systemd unit override, e.g.
>>> /etc/systemd/system/pacemaker.service.d/dependencies.conf.
> Yes, that's how we implemented the additional dependencies.
>>>
>>>> When shutting down or rebooting migration fails and not even the
>>>> regular
>>>> shutdown of the vms succeeds. Systemd seems to tear down the vms by
>>>> terminating something they depend on.
>>>>
>>>> Is this a known issue? Did we miss any further dependencies?
>>> There was a shutdown issue when using systemd-class cluster resources
>>> (systemd: instead of ocf:), but I believe that was fixed in the package
>>> you're using, and it's probably not relevant here anyway.
>> Speaking of
>> https://github.com/ClusterLabs/pacemaker/pull/887/commits/6aae8542abedc755b90c8c49aa5c429718fd12f1?
>>
>>
>> It shouldn't be in Centos 7.2 but I agree unless there are no
>> systemd-resources involved it wouldn't matter.
>>
>>>
>>> It does sound like there's another dependency, but I don't know what.
>>>
>>> What log messages do you see on the failure?
> See attached log files.

The line that stands out to me is:

Nov  4 11:11:39 kvm02 systemd: Stopping Virtual Machine qemu-2-samba2.

Systemd is stopping the VM before pacemaker is able to migrate it. I'm
guessing that is due to this line in the libvirt unit:

Before=libvirt-guests.service

It appears systemd feels free to do that part in parallel, even though
libvirt itself has to wait until pacemaker finishes stopping. Try adding
libvirt-guests to your pacemaker override.

>>>
>>>> Tia
>>>> Rainer
>>>>
>>>>
>>>> 
>>>>
>>>> IT Nerb GmbH
>>>> Lessingstraße 8
>>>> 85098 Großmehring
>>>>
>>>> Telefon : +49 700 ITNERBGMBH
>>>> Telefax : +49 8407 939 284
>>>> email : i...@it-nerb.de
>>>> Internet : www.it-nerb.de <http://www.it-nerb.de>
>>>> Geschäftsführer:Rainer Nerb
>>>> Handelsregister:HRB 2592
>>>> HR-Gericht:Ingolstadt
>>>>
>>>> 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?

[Ken Gaillot]

On 11/04/2016 01:57 PM, CART Andreas wrote:
> Hi
>  
> I have a basic 2 node active/passive cluster with Pacemaker (1.1.14 ,
> pcs: 0.9.148) / CMAN (3.0.12.1) / Corosync (1.4.7) on RHEL 6.8.
> This cluster runs NFS on top of DRBD (8.4.4).
>  
> Basically the system is working on both nodes and I can switch the
> resources from one node to the other.
> But switching resources to the other node does not work, if I try to
> move just one resource and have the others follow due to the location
> constraints.
>  
> From the logged messages I see that in this “failure case” there is NO
> attempt to demote/promote the DRBD clone resource.
>  
> Here is my setup:
> ==
> Cluster Name: clst1
> Corosync Nodes:
> ventsi-clst1-sync ventsi-clst2-sync
> Pacemaker Nodes:
> ventsi-clst1-sync ventsi-clst2-sync
>  
> Resources:
> Resource: IPaddrNFS (class=ocf provider=heartbeat type=IPaddr2)
>   Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=24
>   Operations: start interval=0s timeout=20s (IPaddrNFS-start-interval-0s)
>   stop interval=0s timeout=20s (IPaddrNFS-stop-interval-0s)
>   monitor interval=5s (IPaddrNFS-monitor-interval-5s)
> Resource: NFSServer (class=ocf provider=heartbeat type=nfsserver)
>   Attributes: nfs_shared_infodir=/var/lib/nfsserversettings/
> nfs_ip=xxx.xxx.xxx.xxx nfsd_args="-H xxx.xxx.xxx.xxx"
>   Operations: start interval=0s timeout=40 (NFSServer-start-interval-0s)
>   stop interval=0s timeout=20s (NFSServer-stop-interval-0s)
>   monitor interval=10s timeout=20s
> (NFSServer-monitor-interval-10s)
> Master: DRBDClone
>   Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true
>   Resource: DRBD (class=ocf provider=linbit type=drbd)
>Attributes: drbd_resource=nfsdata
>Operations: start interval=0s timeout=240 (DRBD-start-interval-0s)
>promote interval=0s timeout=90 (DRBD-promote-interval-0s)
>demote interval=0s timeout=90 (DRBD-demote-interval-0s)
>stop interval=0s timeout=100 (DRBD-stop-interval-0s)
>monitor interval=1s timeout=5 (DRBD-monitor-interval-1s)
> Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)
>   Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4
>   Operations: start interval=0s timeout=60
> (DRBD_global_clst-start-interval-0s)
>   stop interval=0s timeout=60
> (DRBD_global_clst-stop-interval-0s)
>   monitor interval=20 timeout=40
> (DRBD_global_clst-monitor-interval-20)
>  
> Stonith Devices:
> Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1
>   Operations: monitor interval=60s (ipmi-fence-clst1-monitor-interval-60s)
> Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1
>   Operations: monitor interval=60s (ipmi-fence-clst2-monitor-interval-60s)
> Fencing Levels:
>  
> Location Constraints:
>   Resource: ipmi-fence-clst1
> Disabled on: ventsi-clst1-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)
>   Resource: ipmi-fence-clst2
> Disabled on: ventsi-clst2-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)
> Ordering Constraints:
>   start IPaddrNFS then start NFSServer (kind:Mandatory)
> (id:order-IPaddrNFS-NFSServer-mandatory)
>   promote DRBDClone then start DRBD_global_clst (kind:Mandatory)
> (id:order-DRBDClone-DRBD_global_clst-mandatory)
>   start DRBD_global_clst then start IPaddrNFS (kind:Mandatory)
> (id:order-DRBD_global_clst-IPaddrNFS-mandatory)
> Colocation Constraints:
>   NFSServer with IPaddrNFS (score:INFINITY)
> (id:colocation-NFSServer-IPaddrNFS-INFINITY)
>   DRBD_global_clst with DRBDClone (score:INFINITY)
> (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)

It took me a while to notice it, it's easily overlooked, but the above
constraint is the problem. It says DRBD_global_clst must be located
where DRBDClone is running ... not necessarily where DRBDClone is
master. This constraint should be created like this:

pcs constraint colocation add DRBD_global_clst with master DBRDClone

>   IPaddrNFS with DRBD_global_clst (score:INFINITY)
> (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)
>  
> Resources Defaults:
> resource-stickiness: INFINITY
> Operations Defaults:
> timeout: 10s
>  
> Cluster Properties:
> cluster-infrastructure: cman
> dc-version: 1.1.14-8.el6-70404b0
> have-watchdog: false
> last-lrm-refresh: 1478277432
> no-quorum-policy: ignore
> stonith-enabled: true
> symmetric-cluster: true
> =

Re: [ClusterLabs] pacemaker after upgrade from wheezy to jessie

[Ken Gaillot]

t; If this is the case something is wrong, greping for validate gives the
>>  > old string back.
>>
>> We found some strange behavior when setting "validate-with" via
>> cibadmin, corosync.log shows the successful transaction, issuing
>> cibadmin --query gives the correct value but it is NOT written into
>> cib.xml.
>>
>> We restarted pacemaker and value is reset to pacemaker-1.1
>> If signatures for the cib.xml are generated from pacemaker/cib, which
>> algorithm is used? looks like md5 to me.
>>
>> Would it be possible to manual edit the cib.xml and generate a valid
>> cib.xml.sig to get one step further in debugging process?
>>
>> Regards, Toni
>>
>> -- 
>> Mit freundlichen Grüßen
>>
>> Toni Tschampke | t...@halle.it
>> bcs kommunikationslösungen
>> Inh. Dipl. Ing. Carsten Burkhardt
>> Harz 51 | 06108 Halle (Saale) | Germany
>> tel +49 345 29849-0 | fax +49 345 29849-22
>> www.b-c-s.de | www.halle.it | www.wivewa.de
>>
>>
>> EINFACH ADRESSEN, TELEFONATE UND DOKUMENTE VERWALTEN - MIT WIVEWA -
>> IHREM WISSENSVERWALTER FUER IHREN BETRIEB!
>>
>> Weitere Informationen erhalten Sie unter www.wivewa.de
>>
>> Am 03.11.2016 um 16:39 schrieb Toni Tschampke:
>>>  > I'm going to guess you were using the experimental 1.1 schema as the
>>>  > "validate-with" at the top of /var/lib/pacemaker/cib/cib.xml. Try
>>>  > changing the validate-with to pacemaker-next or pacemaker-1.2 and
>>> see if
>>>  > you get better results. Don't edit the file directly though; use the
>>>  > cibadmin command so it signs the end result properly.
>>>  >
>>>  > After changing the validate-with, run:
>>>  >
>>>  >crm_verify -x /var/lib/pacemaker/cib/cib.xml
>>>  >
>>>  > and fix any errors that show up.
>>>
>>> strange, the location of our cib.xml differs from your path, our cib is
>>> located in /var/lib/heartbeat/crm/
>>>
>>> running cibadmin --modify --xml-text '>> validate-with="pacemaker-1.2"/>'
>>>
>>> gave no output but was logged to corosync:
>>>
>>> cib: info: cib_perform_op:-- >> validate-with="pacemaker-1.1"/>
>>> cib: info: cib_perform_op:++ >> num_updates="1" validate-with="pacemaker-1.2" crm_feature_set="3.0.6"
>>>   have-quorum="1" cib-last-written="Thu Nov  3 10:05:52 2016"
>>> update-origin="nebel1" update-client="cibadmin" update-user="root"/>
>>>
>>> I'm guessing this change should be instantly written into the xml file?
>>> If this is the case something is wrong, greping for validate gives the
>>> old string back.
>>>
>>> >> validate-with="pacemaker-1.1" crm_feature_set="3.0.6" have-quorum="1"
>>> cib-last-written="Thu Nov  3 16:19:51 2016" update-origin="nebel1"
>>> update-client="cibadmin" update-user="root">
>>>
>>> pacemakerd --features
>>> Pacemaker 1.1.15 (Build: e174ec8)
>>> Supporting v3.0.10:
>>>
>>> Should the crm_feature_set be updated this way too? I'm guessing this is
>>> done when "cibadmin --upgrade" succeeds?
>>>
>>> We just get an timeout error when trying to upgrade it with cibadmin:
>>> Call cib_upgrade failed (-62): Timer expired
>>>
>>> Do have permissions changed from 1.1.7 to 1.1.15? when looking at our
>>> quite big /var/lib/heartbeat/crm/ folder some permissions changed:
>>>
>>> -rw--- 1 hacluster root  80K Nov  1 16:56 cib-31.raw
>>> -rw-r--r-- 1 hacluster root   32 Nov  1 16:56 cib-31.raw.sig
>>> -rw--- 1 hacluster haclient  80K Nov  1 18:53 cib-32.raw
>>> -rw--- 1 hacluster haclient   32 Nov  1 18:53 cib-32.raw.sig
>>>
>>> cib-31 was before upgrading, cib-32 after starting upgraded pacemaker
>>>
>>>
>>> -- 
>>> Mit freundlichen Grüßen
>>>
>>> Toni Tschampke | t...@halle.it
>>> bcs kommunikationslösungen
>>> Inh. Dipl. Ing. Carsten Burkhardt
>>> Harz 51 | 06108 Halle (Saale) | Germany
>>> tel +49 345 29849-0 | fax +49 345 29849-22
>>> www.b-c-s.de | www.halle.it | www.wivewa.de
>>>
>>>
>>> EINFACH ADRESSEN, TELEFONATE UND DOKUMENTE VERWALTEN - MIT WIVEWA -
>>> IHREM WISSENSVE

Re: [ClusterLabs] Resources wont start on new node unless it is the only active node

[Ken Gaillot]

On 11/08/2016 12:54 PM, Ryan Anstey wrote:
> I've been running a ceph cluster with pacemaker for a few months now.
> Everything has been working normally, but when I added a fourth node it
> won't work like the others, even though their OS is the same and the
> configs are all synced via salt. I also don't understand pacemaker that
> well since I followed a guide for it. If anyone could steer me in the
> right direction I would greatly appreciate it. Thank you!
> 
> - My resources only start if the new node is the only active node.
> - Once started on new node, if they are moved back to one of the
> original nodes, it won't go back to the new one.
> - My resources work 100% if I start them manually (without pacemaker).
> - (In the logs/configs below, my resources are named "unifi",
> "rbd_unifi" being the main one that's not working.)

The key is all the location constraints starting with "cli-" in your
configuration. Such constraints were added automatically by command-line
tools, rather than added by you explicitly.

For example, Pacemaker has no concept of "moving" a resource. It places
all resources where they can best run, as specified by the
configuration. So, to move a resource, command-line tools add a location
constraint making the resource prefer a different node.

The downside is that the preference doesn't automatically go away. The
resource will continue to prefer the other node until you explicitly
remove the constraint.

Command-line tools that add such constraints generally provide some way
to clear them. If you clear all those constraints, resources will again
be placed on any node equally.

Separately, you also have a default resource stickiness of 100. That
means that even after you remove the constraints, resources that are
already running will tend to stay where they are. But if you stop and
start a resource, or add a new resource, it could start on a different node.

> 
> Log when running cleaning up the resource on the NEW node:
> 
> Nov 08 09:25:20 h4 Filesystem(fs_unifi)[18044]: WARNING: Couldn't find
> device [/dev/rbd/rbd/unifi]. Expected /dev/??? to exist
> Nov 08 09:25:20 h4 lrmd[3564]: notice: lxc_unifi_monitor_0:18018:stderr
> [ unifi doesn't exist ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation lxc_unifi_monitor_0:
> not running (node=h4, call=484, rc=7, cib-update=390, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: h4-lxc_unifi_monitor_0:484 [
> unifi doesn't exist\n ]
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation fs_unifi_monitor_0: not
> running (node=h4, call=480, rc=7, cib-update=391, confirmed=true)
> Nov 08 09:25:20 h4 crmd[3567]: notice: Operation rbd_unifi_monitor_0:
> not running (node=h4, call=476, rc=7, cib-update=392, confirmed=true)
> 
> Log when running cleaning up the resource on the OLD node:  
> 
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838209
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 crmd[11394]: notice: State transition S_IDLE ->
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838210
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838212
> Nov 08 09:21:18 h3 cib[11389]: warning: A-Sync reply to crmd failed: No
> message of desired type
> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:22 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:24 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:25 h3 crmd[11394]: notice: Notifications disabled
> Nov 08 09:21:25 h3 crmd[11394]: warning: No match for shutdown action on
> 167838211
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   rbd_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   fs_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Start   lxc_unifi(h3)
> Nov 08 09:21:26 h3 pengine[11393]: notice: Calculated Transition 119:
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Processing graph 119
> (ref=pe_calc-dc-1478625686-648) derived from
> /var/lib/pacemaker/pengine/pe-input-739.bz2
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 12: monitor
> rbd_unifi_monitor_0 on h4
> Nov 08 09:21:26 h3 crmd[11394]: notice: Initiating action 9: monitor
> rbd_unifi_mon

Re: [ClusterLabs] Antw: Resources wont start on new node unless it is the only active node

[Ken Gaillot]

On 11/09/2016 02:33 AM, Ulrich Windl wrote:
 Ryan Anstey  schrieb am 08.11.2016 um 19:54 in 
 Nachricht
>> Log when running cleaning up the resource on the OLD node:
>>
>> Nov 08 09:21:18 h3 crmd[11394]: warning: No match for shutdown action on
>> 167838209
> 
> This indicates a node communication problem!

Not necessarily; it's fairly routine. In 1.1.15, we changed the wording
to "No reason to expect node to be down", and in 1.1.16, we only log it
if there's a problem.

It indicates that Pacemaker checked whether a node was expected to be
down. It does that routinely for certain normal events to react
appropriately, so most of the time it's unimportant. It's only a problem
if the node went down, so since 1.1.16, it's only logged in that situation.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?

[Ken Gaillot]

onitor-interval-10)
> 
> Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)
> 
>   Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4
> 
>   Operations: start interval=0 timeout=60
> (DRBD_global_clst-start-interval-0)
> 
>   stop interval=0 timeout=60 (DRBD_global_clst-stop-interval-0)
> 
>   monitor interval=20 timeout=40
> (DRBD_global_clst-monitor-interval-20)
> 
> Resource: NFS_global_clst (class=ocf provider=heartbeat type=Filesystem)
> 
>   Attributes: device=xxx.xxx.xxx.xxx:/drbdmnts/global_clst/nfs
> directory=/global/nfs fstype=nfs
> 
>   Operations: start interval=0 timeout=60 (NFS_global_clst-start-interval-0)
> 
>   stop interval=0 timeout=60 (NFS_global_clst-stop-interval-0)
> 
>   monitor interval=20 timeout=40
> (NFS_global_clst-monitor-interval-20)
> 
> Resource: BIND_global_clst (class=ocf provider=heartbeat type=Filesystem)
> 
>   Attributes: device=/drbdmnts/global_clst/nfs directory=/global/nfs
> fstype=none options=bind
> 
>   Operations: start interval=0 timeout=60
> (BIND_global_clst-start-interval-0)
> 
>   stop interval=0 timeout=60 (BIND_global_clst-stop-interval-0)
> 
>   monitor interval=20 timeout=40
> (BIND_global_clst-monitor-interval-20)
> 
>  
> 
> Stonith Devices:
> 
> Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)
> 
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1
> 
>   Operations: monitor interval=60 (ipmi-fence-clst1-monitor-interval-60)
> 
> Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)
> 
>   Attributes: lanplus=1 login=foo passwd=bar action=reboot
> ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list
> pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1
> 
>   Operations: monitor interval=60 (ipmi-fence-clst2-monitor-interval-60)
> 
> Fencing Levels:
> 
>  
> 
> Location Constraints:
> 
>   Resource: DRBD_global_clst
> 
> Disabled on: ventsi-clst1-sync (score:-INFINITY) (role: Started)
> (id:cli-ban-DRBD_global_clst-on-ventsi-clst1-sync)
> 
>   Resource: ipmi-fence-clst1
> 
> Disabled on: ventsi-clst1-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)
> 
>   Resource: ipmi-fence-clst2
> 
> Disabled on: ventsi-clst2-sync (score:-INFINITY)
> (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)
> 
> Ordering Constraints:
> 
>   start IPaddrNFS then start NFSServer (kind:Mandatory)
> (id:order-IPaddrNFS-NFSServer-mandatory)
> 
>   promote DRBDClone then start DRBD_global_clst (kind:Mandatory)
> (id:order-DRBDClone-DRBD_global_clst-mandatory)
> 
>   start DRBD_global_clst then start IPaddrNFS (kind:Mandatory)
> (id:order-DRBD_global_clst-IPaddrNFS-mandatory)
> 
>   start NFSServer then start NFS_global_clst (kind:Mandatory)
> (id:order-NFSServer-NFS_global_clst-mandatory)
> 
>   start NFSServer then start BIND_global_clst (kind:Mandatory)
> (id:order-NFSServer-BIND_global_clst-mandatory)
> 
> Colocation Constraints:
> 
>   NFSServer with IPaddrNFS (score:INFINITY)
> (id:colocation-NFSServer-IPaddrNFS-INFINITY)
> 
>   IPaddrNFS with DRBD_global_clst (score:INFINITY)
> (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)
> 
>   NFS_global_clst with NFSServer (score:-INFINITY)
> (id:colocation-NFS_global_clst-NFSServer--INFINITY)
> 
>   BIND_global_clst with NFSServer (score:INFINITY)
> (id:colocation-BIND_global_clst-NFSServer-INFINITY)
> 
>   DRBD_global_clst with DRBDClone (score:INFINITY) (rsc-role:Started)
> (with-rsc-role:Master) (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)
> 
>  
> 
> Resources Defaults:
> 
> resource-stickiness: INFINITY
> 
> Operations Defaults:
> 
> timeout: 10s
> 
>  
> 
> Cluster Properties:
> 
> cluster-infrastructure: cman
> 
> dc-version: 1.1.14-8.el6-70404b0
> 
> have-watchdog: false
> 
> last-lrm-refresh: 1478703150
> 
> no-quorum-policy: ignore
> 
> stonith-enabled: true
> 
> symmetric-cluster: true
> 
> Node Attributes:
> 
> ventsi-clst1-sync: PostgresSon-data-status=DISCONNECT
> 
> ventsi-clst2-sync: PostgresSon-data-status=DISCONNECT
> 
>  
> 
>  
> 
> Kind regards
> 
> Andi
> 
>  
> 
> -Original Message-
> From: Ken Gaillot [mailto:kgail...@redhat.com]
> Sent: Dienstag, 8. November 2016 22:29
> To: users@clusterlabs.org
> Subject: Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix

[ClusterLabs] Pacemaker 1.1.16 - Release Candidate 2

[Ken Gaillot]

The second release candidate for Pacemaker version 1.1.16 is now
available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc2

It contains only minor fixes compared to 1.1.16-rc1. If there are no
significant issues found in the next couple of weeks, I will release
this as the final 1.1.16. Any feedback is appreciated.
-- 
Ken Gaillot 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: late stop of dependency?

[Ken Gaillot]

On 11/17/2016 02:46 AM, Ulrich Windl wrote:
> Hi!
> 
> I have a question:
> When having dependencies like "A has to start before B" and "A has to start 
> before C". Now when shutting down, B and C are shut down before A, as 
> requested.
> Now when B takes a long time to stop, C is stopped early. I wonder whether I 
> can tell pacemaker to stop C late, but I don't want to add a dependency like 
> "C has to stop before B" (I don't want a functional dependency between B and 
> C, because there is none).
> If that's too abstract: Think A is a resource B needs, and C is a performance 
> monitor for A.
> 
> Regards,
> Ulrich

I'd make "stop B then stop C" an asymmetrical, optional constraint.

asymmetrical = it only applies when stopping, not starting

optional = only apply the order if they happen to be stopping in the
same transition

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Locate resource with functioning member of clone set?

[Ken Gaillot]

On 11/17/2016 11:37 AM, Israel Brewster wrote:
> I have a resource that is set up as a clone set across my cluster,
> partly for pseudo-load balancing (If someone wants to perform an action
> that will take a lot of resources, I can have them do it on a different
> node than the primary one), but also simply because the resource can
> take several seconds to start, and by having it already running as a
> clone set, I can failover in the time it takes to move an IP resource -
> essentially zero down time.
> 
> This is all well and good, but I ran into a problem the other day where
> the process on one of the nodes stopped working properly. Pacemaker
> caught the issue, and tried to fix it by restarting the resource, but
> was unable to because the old instance hadn't actually exited completely
> and was still tying up the TCP port, thereby preventing the new instance
> that pacemaker launched from being able to start.
> 
> So this leaves me with two questions: 
> 
> 1) is there a way to set up a "kill script", such that before trying to
> launch a new copy of a process, pacemaker will run this script, which
> would be responsible for making sure that there are no other instances
> of the process running?

Sure, it's called a resource agent :)

When recovering a failed resource, Pacemaker will call the resource
agent's stop action first, then start. The stop should make sure the
service has exited completely. If it doesn't, the agent should be fixed
to do so.

> 2) Even in the above situation, where pacemaker couldn't launch a good
> copy of the resource on the one node, the situation could have been
> easily "resolved" by pacemaker moving the virtual IP resource to another
> node where the cloned resource was running correctly, and notifying me
> of the problem. I know how to make colocation constraints in general,
> but how do I do a colocation constraint with a cloned resource where I
> just need the virtual IP running on *any* node where there clone is
> working properly? Or is it the same as any other colocation resource,
> and pacemaker is simply smart enough to both try to restart the failed
> resource and move the virtual IP resource at the same time?

Correct, a simple colocation constraint of "resource R with clone C"
will make sure R runs with a working instance of C.

There is a catch: if *any* instance of C restarts, R will also restart
(even if it stays in the same place), because it depends on the clone as
a whole. Also, in the case you described, pacemaker would first try to
restart both C and R on the same node, rather than move R to another
node (although you could set on-fail=stop on C to force R to move).

If that's not sufficient, you could try some magic with node attributes
and rules. The new ocf:pacemaker:attribute resource in 1.1.16 could help
there.

> As an addendum to question 2, I'd be interested in any methods there may
> be to be notified of changes in the cluster state, specifically things
> like when a resource fails on a node - my current nagios/icinga setup
> doesn't catch that when pacemaker properly moves the resource to a
> different node, because the resource remains up (which, of course, is
> the whole point), but it would still be good to know something happened
> so I could look into it and see if something needs fixed on the failed
> node to allow the resource to run there properly.

Since 1.1.15, Pacemaker has alerts:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm47975782138832

Before 1.1.15, you can use the ocf:pacemaker:ClusterMon resource to do
something similar.

> 
> Thanks!
> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Bug in ocf-shellfuncs, ocf_local_nodename function?

[Ken Gaillot]

On 11/17/2016 11:59 AM, Israel Brewster wrote:
> This refers specifically to build version
> 5434e9646462d2c3c8f7aad2609d0ef1875839c7 of the ocf-shellfuncs file, on
> CentOS 6.8, so it might not be an issue on later builds (if any) or
> different operating systems, but it would appear that the
> ocf_local_nodename function can have issues with certain configurations.
> Specially, I was debugging an issue I was having with a resource agent
> that I traced down to that function returning the FQDN of the machine
> rather than the actual node name, which in my case was a short name.
> 
> In looking at the code, I see that the function is looking for a
> pacemaker version greater than 1.1.8, in which case it uses crm_node
> (which works), otherwise it just uses "uname -n", which returns the FQDN
> (at least in my configuration). To get the current version, it runs the
> command:
> 
> local version=$(pacemakerd -$ | grep "Pacemaker .*" | awk '{ print $2 }')
> 
> Which on CentOS 6.8 returns (as of today, at least):
> 
> 1.1.14-8.el6_8.1
> 
> Unfortunately, when that string is passed to the ocf_version_cmp
> function to compare against 1.1.8, it returns 3, for "bad format", and
> so falls back to using "uname -n", even though the version *is* greater
> than 1.1.8, and crm_node would return the proper value.
> 
> Of course, if you always set up your cluster to use the FQDN of the
> servers as the node name, or more specifically always set them up such
> that the output of uname -n is the node name, then there isn't an issue
> other than perhaps a undetectably slight loss of efficiency. However, as
> I accidentally proved by doing otherwise, there is no actual requirement
> when setting up a cluster that the node names match uname -n (although
> perhaps it is considered "best practice"?), as long as they resolve to
> an IP.

Yes, it is considered a "best practice" (or at least a "safer
practice"), because issues like this tend to pop up periodically. :(

I'd recommend filing a bug against the resource-agents package, so the
version comparison can be made more intelligent.

> 
> I've worked around this in my installation by simply modifying the
> resource agent to call crm_node directly (since I know I am running on a
> version greater than 1.1.8), but I figured I might mention it, since I
> don't get any results when trying to google the issue.
> ---
> Israel Brewster
> Systems Analyst II
> Ravn Alaska
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7293
> ---

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Query about resource stickiness

[Ken Gaillot]

On 11/17/2016 06:41 PM, phanidhar prattipati wrote:
> Good Morning All,
> 
> I have configured HA on 3 nodes and in order to disable automatic fail
> over i need to set resource stickiness value and not sure how to
> calculate it. Currently i set it o INFINITY which i believe is not the
> right way of doing it. Any pointers how to calculate based on the
> environment set up will be really great help?
> 
> 
> -- 
> Thanks,
> Phanidhar

INFINITY is fine -- many people use that for stickiness.

It's simply a matter of preference. How it matters is when weighing
against other scores in your configuration.

For example, let's say you have a resource R with a location constraint
preferring node N, and you have resource stickiness.

If N crashes or is shut down, R will move to another node. When N comes
back, R will stay on the other node, if the resource stickiness is
higher than the location constraint's score; it will move back to N, if
the location constraint's score is higher.

A score of INFINITY means never move back, as long as the new node stays up.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> Hi folks,
> 
> Is there a way to set a node attribute to the "status" section for few
> nodes at the same time?
> 
> In my case there is a node attribute which allows some resources to
> start in the cluster if it is set.
> If I set this node attribute for say two nodes in a way - one and then
> another, than these resources are not distributed equally between these
> two nodes. That because Pacemaker picks the first node to with this
> attribute is set and immediately starts all allowed resources on it. And
> this is not the behavior i would like to get.
> 
> Thank you,
> Kostia

Not that I know of, but it would be a good feature to add to
attrd_updater and/or crm_attribute.

You can probably hack it with a dampening value of a few seconds. If
your rule checks for a particular value of the attribute, set all the
nodes to a different value first, which will write that value and start
the dampening timer. Then set all the attributes to the desired value,
and they will get written out together when the timer expires.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD Insufficient Privileges Error

[Ken Gaillot]

On 11/20/2016 01:58 PM, Jasim Alam wrote:
> Hi,
> 
>  
> 
> I am trying to setup  two node H/A cluster with DRBD. Following is my
> configuration
> 
>  
> 
> /[root@node-1 ~]# pcs config/
> 
> /Cluster Name: Cluster-1/
> 
> /Corosync Nodes:/
> 
> /node-1 node-2 /
> 
> /Pacemaker Nodes:/
> 
> /node-1 node-2 /
> 
> / /
> 
> /Resources: /
> 
> / Resource: vip (class=ocf provider=heartbeat type=IPaddr2)/
> 
> /  Attributes: ip=103.9.185.211 cidr_netmask=32 /
> 
> /  Operations: start interval=0s timeout=20s (vip-start-interval-0s)/
> 
> /  stop interval=0s timeout=20s (vip-stop-interval-0s)/
> 
> /  monitor interval=30s (vip-monitor-interval-30s)/
> 
> /Resource: apache (class=ocf provider=heartbeat type=apache)/
> 
> /  Attributes: configfile=/etc/httpd/conf/httpd.conf
> statusurl=http://localhost/server-status /
> 
> /  Operations: start interval=0s timeout=40s (apache-start-interval-0s)/
> 
> /  stop interval=0s timeout=60s (apache-stop-interval-0s)/
> 
> /  monitor interval=1min (apache-monitor-interval-1min)/
> 
> /Master: StorageClone/
> 
> /  Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true /
> 
> /  Resource: storage (class=ocf provider=linbit type=drbd)/
> 
> /   Attributes: drbd_resource=drbd0 /
> 
> /   Operations: start interval=0s timeout=240 (storage-start-interval-0s)/
> 
> /   promote interval=0s timeout=90
> (storage-promote-interval-0s)/
> 
> /   demote interval=0s timeout=90 (storage-demote-interval-0s)/
> 
> /   stop interval=0s timeout=100 (storage-stop-interval-0s)/
> 
> /   monitor interval=60s (storage-monitor-interval-60s)/
> 
> / /
> 
> /Stonith Devices: /
> 
> /Fencing Levels: /
> 
> / /
> 
> /Location Constraints:/
> 
> /  Resource: apache/
> 
> /Enabled on: node-1 (score:50) (id:location-apache-node-1-50)/
> 
> /Ordering Constraints:/
> 
> /  start vip then start apache (kind:Mandatory)
> (id:order-vip-apache-mandatory)/
> 
> /Colocation Constraints:/
> 
> /  vip with apache (score:INFINITY) (id:colocation-vip-apache-INFINITY)/
> 
> / /
> 
> /Resources Defaults:/
> 
> /No defaults set/
> 
> /Operations Defaults:/
> 
> /No defaults set/
> 
> / /
> 
> /Cluster Properties:/
> 
> /cluster-infrastructure: corosync/
> 
> /cluster-name: Cluster-1/
> 
> /dc-version: 1.1.13-10.el7_2.4-44eb2dd/
> 
> /have-watchdog: false/
> 
> /no-quorum-policy: ignore/
> 
> /stonith-enabled: false/
> 
>  
> 
> The problem is I am getting insufficient privilege error on second node
> 
>  
> 
> /[root@node-1 ~]# pcs status/
> 
> /Cluster name: PSD-1/
> 
> /Last updated: Mon Nov 21 01:44:52 2016  Last change: Mon Nov 21
> 01:19:17 2016 by root via cibadmin on node-1/
> 
> /Stack: corosync/
> 
> /Current DC: node-1 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with
> quorum/
> 
> /2 nodes and 4 resources configured/
> 
> / /
> 
> /Online: [ node-1 node-2 ]/
> 
> / /
> 
> /Full list of resources:/
> 
> / /
> 
> /vip(ocf::heartbeat:IPaddr2):   Started node-1/
> 
> /apache (ocf::heartbeat:apache):Started node-1/
> 
> /Master/Slave Set: StorageClone [storage]/
> 
> / storage(ocf::linbit:drbd): FAILED node-2 (unmanaged)/
> 
> / Masters: [ node-1 ]/
> 
> / /
> 
> /Failed Actions:/
> 
> /* storage_stop_0 on node-2 'insufficient privileges' (4): call=16,
> status=complete, exitreason='none',/
> 
> /last-rc-change='Mon Nov 21 01:19:17 2016', queued=0ms, exec=2ms/
> 
> / /
> 
> / /
> 
> /PCSD Status:/
> 
> /  node-1: Online/
> 
> /  node-2: Online/
> 
> / /
> 
> /Daemon Status:/
> 
> /  corosync: active/disabled/
> 
> /  pacemaker: active/disabled/
> 
> /  pcsd: active/enabled/
> 
>  
> 
> but DRBD seems okay for both nodes
> 
>  
> 
> /  [root@node-1 ~]# drbd-overview /
> 
> / 0:drbd0/0  Connected Primary/Secondary UpToDate/UpToDate/
> 
> //
> 
> / [root@node-2 ~]# drbd-overview /
> 
> / 0:drbd0/0  Connected Secondary/Primary UpToDate/UpToDate/
> 
>  
> 
> Log of node2
> 
>  
> 
> /[root@node-2 ~]# tail -n 10 /var/log/messages/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:  notice: State transition S_NOT_DC
> -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> origin=do_election_count_vote ]/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:  notice: State transition S_PENDING
> -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE
> origin=do_cl_join_finalize_respond ]/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:   error: Failed to retrieve
> meta-data for ocf:linbit:drbd/

Pacemaker is unable to get the metadata for the resource agent. Try
getting it manually:

 OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/linbit/drbd meta-data

If that works, check whether selinux is blocking it.

> 
> /Nov 21 01:19:17 node-2 crmd[4060]: warning: No metadata found for
> drbd::ocf:linbit: Input/output error (-5)/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:   error: No metadata for
> linbit::ocf:drbd/
> 
> /Nov 21 01:19:17 node-2 crmd[4060]:  notice: Operation
> storage_monitor_

Re: [ClusterLabs] Reliable check for "is starting" state of a resource

[Ken Gaillot]

On 11/22/2016 10:53 AM, Kostiantyn Ponomarenko wrote:
> Hi folks,
> 
> I am looking for a good way of checking if a resource is in "starting"
> state.
> The thing is - I need to issue a command and I don't want to issue that
> command when this particular resource is starting. This resource start
> can take up to a few min. 
> As a note, I am OK with issuing that command if a resource is stopped
> (before it starts).
> 
> The best I can think of, is to check for an ongoing state transition of
> a resource in a loop with "crm_simulate -Ls" command.
> But with this approach I need to put the command into a loop, cut
> "Transition Summary" part, and then grep for the needed resource.
> And even that I have a question if this way is reliable?
> 
> Maybe be there is a better way of achieving the same result.  
> 
> Thank you,
> Kostia

Probably the cleanest way is to set record-pending=true (either as an
operation default, or just on the start operation you're interested in).
Then your command (or a wrapper) could check the CIB for a pending start.

A simpler approach would be "crm_resource --wait", which blocks until
the cluster is idle. The downside is you might wait for other actions
that you don't care about.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/22/2016 04:39 AM, Kostiantyn Ponomarenko wrote:
> Using "shadow cib" in crmsh looks like a good idea, but it doesn't work
> with node attributes set into "status" section of Pacemaker config.
> I wonder it it is possible to make it work that way.
> 
> Ken,
>>> start dampening timer
> Could you please elaborate more on this. I don't get how I can set this
> timer.
> Do I need to set this timer for each node?

Dampening is per attribute, so it applies to all nodes. You can set it
when you first create the attribute:

  attrd_updater -n $NAME --update $VALUE --delay $SECONDS

With dampening (delay), the attribute daemon will wait that long between
writes to the CIB. The goal is to reduce I/O activity for frequently
changing attributes, but it could also be handy here.


The --delay will be ignored if the above command is run after the
attribute already exists. You can change it for an already existing
attribute with

  attrd_updater -n $NAME --update-delay --delay $SECONDS

or

  attrd_updater -n $NAME --update-both $VALUE --delay $SECONDS


It's intentionally more trouble to set it on an already-created
attribute, because repeatedly changing the delay will make it useless
(each delay change requires an immediate write). Having a separate
command makes it less likely to be accidental.

> 
> 
> Thank you,
> Kostia
> 
> On Mon, Nov 21, 2016 at 9:30 AM, Ulrich Windl
>  <mailto:ulrich.wi...@rz.uni-regensburg.de>> wrote:
> 
> >>> Ken Gaillot mailto:kgail...@redhat.com>>
> schrieb am 18.11.2016 um 16:17 in Nachricht
>  <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>>:
> > On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> >> Hi folks,
> >>
> >> Is there a way to set a node attribute to the "status" section for few
> >> nodes at the same time?
> >>
> >> In my case there is a node attribute which allows some resources to
> >> start in the cluster if it is set.
> >> If I set this node attribute for say two nodes in a way - one and then
> >> another, than these resources are not distributed equally between these
> >> two nodes. That because Pacemaker picks the first node to with this
> >> attribute is set and immediately starts all allowed resources on it. 
> And
> >> this is not the behavior i would like to get.
> >>
> >> Thank you,
> >> Kostia
> >
> > Not that I know of, but it would be a good feature to add to
> > attrd_updater and/or crm_attribute.
> 
> With crm (shell) you don't have transactions for node attributes,
> but for the configuration. So if you add a location restriction
> preventing any resources on your nodes, then enable the nodes, and
> then delete the location restrictions in one transaction, you might
> get what you want. It's not elegant, but itt ill do.
> 
> To the crm shell maintainer: Is is difficult to build transactions
> to node status changes? The problem I see is this: For configuration
> you always have transactions (requiring "commit), but for nodes you
> traditionally have non (effects are immediate). So you'd need a
> thing like "start transaction" which requires a "commit" or some
> kind of abort later.
> 
> I also don't know whether a "shadow CIB" would help for the original
> problem.
> 
> Ulrich
> 
> >
> > You can probably hack it with a dampening value of a few seconds. If
> > your rule checks for a particular value of the attribute, set all the
> > nodes to a different value first, which will write that value and
> start
> > the dampening timer. Then set all the attributes to the desired value,
> > and they will get written out together when the timer expires.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/22/2016 04:39 AM, Kostiantyn Ponomarenko wrote:
> Using "shadow cib" in crmsh looks like a good idea, but it doesn't work
> with node attributes set into "status" section of Pacemaker config.
> I wonder it it is possible to make it work that way.

Forgot to mention -- the shadow CIB is probably the best way to do this.
I don't know if there's a way to do it in crmsh, but you can use it with
the low-level commands crm_shadow and crm_attribute --lifetime=reboot.

> Ken,
>>> start dampening timer
> Could you please elaborate more on this. I don't get how I can set this
> timer.
> Do I need to set this timer for each node?
> 
> 
> Thank you,
> Kostia
> 
> On Mon, Nov 21, 2016 at 9:30 AM, Ulrich Windl
>  <mailto:ulrich.wi...@rz.uni-regensburg.de>> wrote:
> 
> >>> Ken Gaillot mailto:kgail...@redhat.com>>
> schrieb am 18.11.2016 um 16:17 in Nachricht
>  <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>>:
> > On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> >> Hi folks,
> >>
> >> Is there a way to set a node attribute to the "status" section for few
> >> nodes at the same time?
> >>
> >> In my case there is a node attribute which allows some resources to
> >> start in the cluster if it is set.
> >> If I set this node attribute for say two nodes in a way - one and then
> >> another, than these resources are not distributed equally between these
> >> two nodes. That because Pacemaker picks the first node to with this
> >> attribute is set and immediately starts all allowed resources on it. 
> And
> >> this is not the behavior i would like to get.
> >>
> >> Thank you,
> >> Kostia
> >
> > Not that I know of, but it would be a good feature to add to
> > attrd_updater and/or crm_attribute.
> 
> With crm (shell) you don't have transactions for node attributes,
> but for the configuration. So if you add a location restriction
> preventing any resources on your nodes, then enable the nodes, and
> then delete the location restrictions in one transaction, you might
> get what you want. It's not elegant, but itt ill do.
> 
> To the crm shell maintainer: Is is difficult to build transactions
> to node status changes? The problem I see is this: For configuration
> you always have transactions (requiring "commit), but for nodes you
> traditionally have non (effects are immediate). So you'd need a
> thing like "start transaction" which requires a "commit" or some
> kind of abort later.
> 
> I also don't know whether a "shadow CIB" would help for the original
> problem.
> 
> Ulrich
> 
> >
> > You can probably hack it with a dampening value of a few seconds. If
> > your rule checks for a particular value of the attribute, set all the
> > nodes to a different value first, which will write that value and
> start
> > the dampening timer. Then set all the attributes to the desired value,
> > and they will get written out together when the timer expires.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Locate resource with functioning member of clone set?

[Ken Gaillot]

On 11/22/2016 02:28 PM, Israel Brewster wrote:
> On Nov 17, 2016, at 4:04 PM, Ken Gaillot  <mailto:kgail...@redhat.com>> wrote:
>>
>> On 11/17/2016 11:37 AM, Israel Brewster wrote:
>>> I have a resource that is set up as a clone set across my cluster,
>>> partly for pseudo-load balancing (If someone wants to perform an action
>>> that will take a lot of resources, I can have them do it on a different
>>> node than the primary one), but also simply because the resource can
>>> take several seconds to start, and by having it already running as a
>>> clone set, I can failover in the time it takes to move an IP resource -
>>> essentially zero down time.
>>>
>>> This is all well and good, but I ran into a problem the other day where
>>> the process on one of the nodes stopped working properly. Pacemaker
>>> caught the issue, and tried to fix it by restarting the resource, but
>>> was unable to because the old instance hadn't actually exited completely
>>> and was still tying up the TCP port, thereby preventing the new instance
>>> that pacemaker launched from being able to start.
>>>
>>> So this leaves me with two questions: 
>>>
>>> 1) is there a way to set up a "kill script", such that before trying to
>>> launch a new copy of a process, pacemaker will run this script, which
>>> would be responsible for making sure that there are no other instances
>>> of the process running?
>>
>> Sure, it's called a resource agent :)
>>
>> When recovering a failed resource, Pacemaker will call the resource
>> agent's stop action first, then start. The stop should make sure the
>> service has exited completely. If it doesn't, the agent should be fixed
>> to do so.
> 
> Ah, gotcha. I wasn't thinking along those lines in this case because the
> resource in question doesn't have a dedicated resource agent - it's a
> basic system service type resource. So then the proper approach would be
> to modify the init.d script such that when "stop" is called, it makes
> sure to completely clean up any associated processes - even if the PID
> file disappears or gets changed.

Correct. You could also use the init script as a starting point for a
proper OCF agent if you wanted the flexibility to add more capabilities
in the future. If you're interested, see:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#ap-ocf

>>
>>> 2) Even in the above situation, where pacemaker couldn't launch a good
>>> copy of the resource on the one node, the situation could have been
>>> easily "resolved" by pacemaker moving the virtual IP resource to another
>>> node where the cloned resource was running correctly, and notifying me
>>> of the problem. I know how to make colocation constraints in general,
>>> but how do I do a colocation constraint with a cloned resource where I
>>> just need the virtual IP running on *any* node where there clone is
>>> working properly? Or is it the same as any other colocation resource,
>>> and pacemaker is simply smart enough to both try to restart the failed
>>> resource and move the virtual IP resource at the same time?
>>
>> Correct, a simple colocation constraint of "resource R with clone C"
>> will make sure R runs with a working instance of C.
>>
>> There is a catch: if *any* instance of C restarts, R will also restart
>> (even if it stays in the same place), because it depends on the clone as
>> a whole. Also, in the case you described, pacemaker would first try to
>> restart both C and R on the same node, rather than move R to another
>> node (although you could set on-fail=stop on C to force R to move).
> 
> It *looked* like Pacemaker was continually trying to restart the cloned
> resource in this case - I think the issue being that from Pacemakers
> perspective the service *did* start successfully, it just failed again
> moments later (when it tried to bind to the port, and, being unable to,
> bailed out). So under the "default" configuration, Pacemaker would try
> restarting the service for quite a while before marking it as failed on
> that node. As such, it sounds like under the current configuration, the
> IP resource would never move (at least not in a reasonable time frame),
> as Pacemaker would simply continue to try restarting on the same node.
> 
> So to get around this, I'm thinking I could set the migration-threshold
> property on the clustered resource to something low, like two or three,
> perhaps co

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/23/2016 11:27 AM, Kostiantyn Ponomarenko wrote:
> Maybe I am doing something wrong, but I cannot set "status" section node
> attributes to a shadow cib, cluster applies them immediately.
> To try it out I do in a console:
> 
> crm_shadow --create test
> crm_attribute --type nodes --node node-0 --name my-attribute
> --update 1 --lifetime=reboot
> 
> And this attribute is set to the live cluster configuration immediately.
> What am I doing wrong?

Strange ... if you use --lifetime=forever, it's recorded in the shadow
CIB, but if you use --lifetime=reboot, it's recorded in the live CIB.
That looks like a bug to me; I'll make a note to investigate.

> Thank you,
> Kostia
> 
> On Tue, Nov 22, 2016 at 11:33 PM, Kostiantyn Ponomarenko
>  <mailto:konstantin.ponomare...@gmail.com>> wrote:
> 
> Ken,
> Thank you for the explanation.
> I will try this low-level way of shadow cib creation tomorrow.
> PS: I will sleep much better with this excellent news/idea. =)
> 
> Thank you,
> Kostia
> 
> On Tue, Nov 22, 2016 at 10:53 PM, Ken Gaillot  <mailto:kgail...@redhat.com>> wrote:
> 
> On 11/22/2016 04:39 AM, Kostiantyn Ponomarenko wrote:
> > Using "shadow cib" in crmsh looks like a good idea, but it doesn't 
> work
> > with node attributes set into "status" section of Pacemaker config.
> > I wonder it it is possible to make it work that way.
> 
> Forgot to mention -- the shadow CIB is probably the best way to
> do this.
> I don't know if there's a way to do it in crmsh, but you can use
> it with
> the low-level commands crm_shadow and crm_attribute
> --lifetime=reboot.
> 
> > Ken,
> >>> start dampening timer
> > Could you please elaborate more on this. I don't get how I can set 
> this
> > timer.
> > Do I need to set this timer for each node?
> >
> >
> > Thank you,
> > Kostia
>     >
> > On Mon, Nov 21, 2016 at 9:30 AM, Ulrich Windl
> >  <mailto:ulrich.wi...@rz.uni-regensburg.de>
> > <mailto:ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>>> wrote:
> >
> > >>> Ken Gaillot  <mailto:kgail...@redhat.com> <mailto:kgail...@redhat.com
> <mailto:kgail...@redhat.com>>>
> > schrieb am 18.11.2016 um 16:17 in Nachricht
> >  <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>
> > <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com
> <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>>>:
> > > On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> > >> Hi folks,
> > >>
> > >> Is there a way to set a node attribute to the "status"
> section for few
> > >> nodes at the same time?
> > >>
> > >> In my case there is a node attribute which allows some
> resources to
> > >> start in the cluster if it is set.
> > >> If I set this node attribute for say two nodes in a way
> - one and then
> > >> another, than these resources are not distributed
> equally between these
> > >> two nodes. That because Pacemaker picks the first node
> to with this
> > >> attribute is set and immediately starts all allowed
> resources on it. And
> > >> this is not the behavior i would like to get.
> > >>
> > >> Thank you,
> > >> Kostia
> > >
> > > Not that I know of, but it would be a good feature to add to
> > > attrd_updater and/or crm_attribute.
> >
> > With crm (shell) you don't have transactions for node
> attributes,
> > but for the configuration. So if you add a location
> restriction
> > preventing any resources on your nodes, then enable the
> nodes, and
> > then delete the location restrictions in one transaction,
> you might
> > get what you want. It's not elegant, but itt ill do.
> >
> > To

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/24/2016 05:24 AM, Kostiantyn Ponomarenko wrote:
> Attribute dampening doesn't work for me also.
> To test that I have a script:
> 
> attrd_updater -N node-0 -n my-attr --update false --delay 20
> sleep 3
> attrd_updater -N node-0 -n my-attr --update true
> sleep 7
> attrd_updater -N node-1 -n my-attr --update true

This sequence works for me -- the attributes are not written to the live
CIB until the end of the delay, when both are written at the same time.

The remaining issue must be with the policy engine. You could look at
the detail log on the DC when these changes were made; you should see
info-level messages with the CIB change with both values together (lines
with "cib_perform_op:   ++" and the attribute values), then "Transition
aborted" with "Transient attribute change", then a bunch of "pengine:"
lines saying what the cluster wants to do with each resource.

There should be some information about the scores used to place the
resources.

> 
> All my resources have this rule in Pacemaker config:
> 
> crm configure location res1-location-rule res1 \
> rule 0: my-attr eq true \
> rule -inf: my-attr ne true
> 
> On a working two-node cluster I remove "my-attr" from both nodes. 
> Then run my script. And all resources start on node-0.
> Am I doing something wrong?
> Or maybe my understanding of an attribute dampening is not correct?
> 
> My Pacemaker version is 1.1.13. (heh, not the last one, but it is what
> it is ...)
> 
> Thank you,
> Kostia
> 
> On Wed, Nov 23, 2016 at 7:27 PM, Kostiantyn Ponomarenko
>  <mailto:konstantin.ponomare...@gmail.com>> wrote:
> 
> Maybe I am doing something wrong, but I cannot set "status" section
> node attributes to a shadow cib, cluster applies them immediately.
> To try it out I do in a console:
> 
> crm_shadow --create test
> crm_attribute --type nodes --node node-0 --name my-attribute
> --update 1 --lifetime=reboot
> 
> And this attribute is set to the live cluster configuration immediately.
> What am I doing wrong?
> 
> Thank you,
> Kostia
> 
> On Tue, Nov 22, 2016 at 11:33 PM, Kostiantyn Ponomarenko
>  <mailto:konstantin.ponomare...@gmail.com>> wrote:
> 
> Ken,
> Thank you for the explanation.
>         I will try this low-level way of shadow cib creation tomorrow.
> PS: I will sleep much better with this excellent news/idea. =)
> 
> Thank you,
> Kostia
> 
> On Tue, Nov 22, 2016 at 10:53 PM, Ken Gaillot
> mailto:kgail...@redhat.com>> wrote:
> 
> On 11/22/2016 04:39 AM, Kostiantyn Ponomarenko wrote:
> > Using "shadow cib" in crmsh looks like a good idea, but it 
> doesn't work
> > with node attributes set into "status" section of Pacemaker 
> config.
> > I wonder it it is possible to make it work that way.
> 
> Forgot to mention -- the shadow CIB is probably the best way
> to do this.
> I don't know if there's a way to do it in crmsh, but you can
> use it with
> the low-level commands crm_shadow and crm_attribute
> --lifetime=reboot.
> 
> > Ken,
> >>> start dampening timer
> > Could you please elaborate more on this. I don't get how I can 
> set this
> > timer.
> > Do I need to set this timer for each node?
>         >
> >
> > Thank you,
> > Kostia
> >
> > On Mon, Nov 21, 2016 at 9:30 AM, Ulrich Windl
> >  <mailto:ulrich.wi...@rz.uni-regensburg.de>
> > <mailto:ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>>> wrote:
> >
> > >>> Ken Gaillot  <mailto:kgail...@redhat.com> <mailto:kgail...@redhat.com
> <mailto:kgail...@redhat.com>>>
> > schrieb am 18.11.2016 um 16:17 in Nachricht
> >  <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>
> >   
>  <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com
> <mailto:d6f449da-64f8-12ad-00be-e772d8e38...@redhat.com>>>:
> > > On 11/18/2016 08:55 AM, Kostiantyn Ponomarenko wrote:
> > >> Hi folks,
>   

Re: [ClusterLabs] [cluster Labs] standby and unstandby commands

[Ken Gaillot]

On 11/25/2016 04:33 PM, Omar Jaber wrote:
> Hi all ,
> 
> I have cluster contains three  nodes  with different sore  for location 
> constrain and  I have  group resource 
> 
> Running  on the  node  the  have  the highest score  for   location
> constrain when I  try to move  the  resource  from the  node  that have 
> the highest sore
> 
> To other  node by run command  "pcs cluster standby  node that have the  highest  location constrain score>"  the  resource 
> stop in the  node  and  fail in new node(the resource still start-fail
> stat periodically ) 
> 
> I thought at the first the  problem  is from different sore but  I
> changed it and  the  problem still exist   
> 
>  
> 
> And  when I run "pcs  status "  I  see  there  is action failed :
> 
> resource_monitor_1 on hostname for  the new node'not running' (7):
> call=268, status=complete, exitreason='none',
> 
> last-rc-change='Sat Nov 26 00:27:00 2016', queued=0ms, exec=0ms

When Pacemaker starts a resource, it also schedules any recurring
monitor configured for it, immediately after the start returns.

So, it is essential that the resource agent does not return from "start"
until the service is able to pass a "monitor" call. It's possible that
the "start" is returning too soon, and the service still has some
start-up time before it responds to requests. In that case, you'll need
to modify the resource agent.

Other possibilities are that the resource agent is returning success
from start even though the start failed, or that the service is starting
successfully but then immediately crashing.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Get rid of reload altogether

[Ken Gaillot]

On 11/27/2016 10:14 PM, Nikhil Utane wrote:
> Hi,
> 
> I understand the whole concept of reload and how to define parameters
> with unique=0 so that pacemaker can call the reload operation of the OCF
> script instead of stopping and starting the resource.
> Now my problem is that I have 100s of parameters and I don't want to
> specify each of those with unique=0.
> 
> Is there any other way to completely stop the whole business of reload?
> Basically if any of the resource parameters are changed, don't do anything.
> 
> -Thanks
> Nikhil

No, a full restart is the default behavior. If the resource parameters
change, pacemaker has to notify the resource agent somehow.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Pacemaker 1.1.16 released

[Ken Gaillot]

ClusterLabs is proud to announce the latest release of the Pacemaker
cluster resource manager, version 1.1.15. The source code is available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16
The most significant enhancements in this release are:

* rsc-pattern may now be used instead of rsc in location constraints, to
allow a single location constraint to apply to all resources whose names
match a regular expression. Sed-like %0 - %9 backreferences let
submatches be used in node attribute names in rules.

* The new ocf:pacemaker:attribute resource agent sets a node attribute
according to whether the resource is running or stopped. This may be
useful in combination with attribute-based rules to model dependencies
that simple constraints can't handle.

* Pacemaker's existing "node health" feature allows resources to move
off nodes that become unhealthy. Now, when using
node-health-strategy=progressive, a new cluster property
node-health-base will be used as the initial health score of newly
joined nodes (defaulting to 0, which is the previous behavior). This
allows a node to be treated as "healthy" even if it has some "yellow"
health attributes, which can be useful to allow clones to run on such nodes.

* Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
properly passed to multistate resources with notification enabled. This
has been fixed. To help resource agents detect when the fix is
available, the CRM feature set has been incremented. (Whenever the
feature set changes, mixed-version clusters are supported only during
rolling upgrades -- nodes with an older version will not be allowed to
rejoin once they shut down.)

* Watchdog-based fencing using sbd now works better on remote nodes.
This capability still likely has some limitations, however.

* The build process now takes advantage of various compiler features
(RELRO, PIE, as-needed linking, etc.) that enhance security and start-up
performance. See the "Hardening flags" comments in the configure.ac file
for more details.

* Python 3 compatibility: The Pacemaker project now targets
compatibility with both python 2 (versions 2.6 and later) and python 3
(versions 3.2 and later). All of the project's python code now meets
this target, with the exception of CTS, which is still python 2 only.

* The Pacemaker coding guidelines have been replaced by a more
comprehensive addition to the documentation set, "Pacemaker
Development". It is intended for developers working on the Pacemaker
code base itself, rather than external code such as resource agents. A
copy is viewable at
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Development/

As usual, the release includes many bugfixes, including a fix for a
serious security vulnerability (CVE-2016-7035). For a more detailed list
of changes, see the change log:

https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog

Many thanks to all contributors of source code to this release,
including Andrew Beekhof, Bin Liu, Christian Schneider, Christoph Berg,
David Shane Holden, Ferenc Wágner, Yan Gao, Hideo Yamauchi, Jan Pokorný,
Ken Gaillot, Klaus Wenninger, Kostiantyn Ponomarenko, Kristoffer
Grönlund, Lars Ellenberg, Masatake Yamato, Michal Koutný, Nakahira
Kazutomo, Nate Clark, Nishanth Aravamudan, Oyvind Albrigtsen, Ruben
Kerkhof, Tim Bishop, Vladislav Bogdanov and Yusuke Iida. Apologies if I
have overlooked anyone.
-- 
Ken Gaillot 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 11/30/2016 11:31 AM, Kostiantyn Ponomarenko wrote:
> Hi Ken,
> 
> I didn't look into the logs, but I experimented with it for a while.
> Here is what I found.
> 
> It worked for you because this attribute - "my-attr" - has not ever been
> set before in that cluster.
> 
> So if you set an attribute, then remove it, and then set it with
> "--delay", like:
> 
> # attrd_updater -N node-0 -n my-attr --update false --delay 20
> 
> , this delay (dampening) won't work.

Once set, attributes are not truly deleted -- only their values are
cleared. And --delay has no effect with --update if the attribute
already exists, which is what you see above.

To set a delay on an already existing attribute, you have to use
attrd_updater --update-delay or --update-both.

> Moreover, when you delete this attribute the actual remove will be
> delayed by that "--delay" which was used when the attribute was set.
> 
> 
> Thank you,
> Kostia
> 
> On Tue, Nov 29, 2016 at 1:08 AM, Ken Gaillot  <mailto:kgail...@redhat.com>> wrote:
> 
> On 11/24/2016 05:24 AM, Kostiantyn Ponomarenko wrote:
> > Attribute dampening doesn't work for me also.
> > To test that I have a script:
> >
> > attrd_updater -N node-0 -n my-attr --update false --delay 20
> > sleep 3
> > attrd_updater -N node-0 -n my-attr --update true
> > sleep 7
> > attrd_updater -N node-1 -n my-attr --update true
> 
> This sequence works for me -- the attributes are not written to the live
> CIB until the end of the delay, when both are written at the same time.
> 
> The remaining issue must be with the policy engine. You could look at
> the detail log on the DC when these changes were made; you should see
> info-level messages with the CIB change with both values together (lines
> with "cib_perform_op:   ++" and the attribute values), then "Transition
> aborted" with "Transient attribute change", then a bunch of "pengine:"
> lines saying what the cluster wants to do with each resource.
> 
> There should be some information about the scores used to place the
> resources.
> 
> >
> > All my resources have this rule in Pacemaker config:
> >
> > crm configure location res1-location-rule res1 \
> > rule 0: my-attr eq true \
> > rule -inf: my-attr ne true
> >
> > On a working two-node cluster I remove "my-attr" from both nodes.
> > Then run my script. And all resources start on node-0.
> > Am I doing something wrong?
> > Or maybe my understanding of an attribute dampening is not correct?
> >
> > My Pacemaker version is 1.1.13. (heh, not the last one, but it is what
> > it is ...)
> >
> > Thank you,
> > Kostia
> >
> > On Wed, Nov 23, 2016 at 7:27 PM, Kostiantyn Ponomarenko
> >  <mailto:konstantin.ponomare...@gmail.com>
> > <mailto:konstantin.ponomare...@gmail.com
> <mailto:konstantin.ponomare...@gmail.com>>> wrote:
> >
> > Maybe I am doing something wrong, but I cannot set "status" section
> > node attributes to a shadow cib, cluster applies them immediately.
> > To try it out I do in a console:
> >
> > crm_shadow --create test
> > crm_attribute --type nodes --node node-0 --name my-attribute
> > --update 1 --lifetime=reboot
> >
> > And this attribute is set to the live cluster configuration 
> immediately.
> > What am I doing wrong?
> >
> > Thank you,
> > Kostia
> >
> > On Tue, Nov 22, 2016 at 11:33 PM, Kostiantyn Ponomarenko
> >  <mailto:konstantin.ponomare...@gmail.com>
> > <mailto:konstantin.ponomare...@gmail.com
> <mailto:konstantin.ponomare...@gmail.com>>> wrote:
> >
> > Ken,
> > Thank you for the explanation.
> > I will try this low-level way of shadow cib creation tomorrow.
> > PS: I will sleep much better with this excellent news/idea. =)
> >
> > Thank you,
> > Kostia
> >
> > On Tue, Nov 22, 2016 at 10:53 PM, Ken Gaillot
> > mailto:kgail...@redhat.com>
> <mailto:kgail...@redhat.com <mailto:kgail...@redhat.com>>> wrote:
> >
> > On

Re: [ClusterLabs] Deleting a variable

[Ken Gaillot]

On 12/01/2016 01:15 AM, Ulrich Windl wrote:
>>>> Ken Gaillot  schrieb am 30.11.2016 um 21:39 in 
>>>> Nachricht
> <62cb811f-4396-ff36-ec03-67000b4ed...@redhat.com>:
> 
> [...]
>> Once set, attributes are not truly deleted -- only their values are
>> cleared. And --delay has no effect with --update if the attribute
>> already exists, which is what you see above.
> 
> Is there a difference between a "deleted" variable and a defined variable 
> that has an empty string as value? I feel there should be!
> 
> [...]
> 
> Ulrich

Not currently


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Set a node attribute for multiple nodes with one command

[Ken Gaillot]

On 12/01/2016 06:04 AM, Kostiantyn Ponomarenko wrote:
> OK, now I see. I still have a few questions.
> 1. Is there a good reason to not remove the attribute totally if it
> is "deleted"?

We do this for two reasons:

1. You can "delete" an attribute for just one node, while leaving it on
other nodes. Attrd needs to remember the attribute as a whole, and just
delete the value for the one node. Even deleting an attribute on all
nodes is done by separate deletes for each node, so this matters even then.

2. Information about the attribute needs to continue to exist in order
to reliably process changes from different nodes. I forget the exact
details, but I remember looking into it before.

These limitations could possibly be addressed by keeping the attribute
but setting a "deleted" flag, and changing how those would be reported,
but it's not on the radar right now.

> 2. Does "attrd_updater" sets attributes to "status" configuration
> section only?

Yes (when using corosync 2).

crm_attribute modifies the CIB directly, so there is no way to batch its
changes with a delay.

> 3. Do I need to modify/set "--delay" to 0 before removing or
> changing the attribute? Because now I see that previously set delay
> works when I delete the attribute (--delete).

It depends on what you want. The point of the delay is to write changes
out to the CIB only every so often, to save disk I/O. If you're
potentially changing attribute values several times per second, this
would be a huge performance boost. The delay affects all changes,
including deletes.

If you want a specific change to be written to the CIB immediately, then
yes, you have to set the delay to 0. You can either change the delay
beforehand with attrd_updater --update-delay or change the delay and
value together with --update-both.

> 4. Does a delay set only one time work until it's unset (set to 0)?

Yes

> Thank you,
> Kostia
> 
> On Wed, Nov 30, 2016 at 10:39 PM, Ken Gaillot  <mailto:kgail...@redhat.com>> wrote:
> 
> On 11/30/2016 11:31 AM, Kostiantyn Ponomarenko wrote:
> > Hi Ken,
> >
> > I didn't look into the logs, but I experimented with it for a while.
> > Here is what I found.
> >
> > It worked for you because this attribute - "my-attr" - has not ever been
> > set before in that cluster.
> >
> > So if you set an attribute, then remove it, and then set it with
> > "--delay", like:
> >
> > # attrd_updater -N node-0 -n my-attr --update false --delay 20
> >
> > , this delay (dampening) won't work.
> 
> Once set, attributes are not truly deleted -- only their values are
> cleared. And --delay has no effect with --update if the attribute
> already exists, which is what you see above.
> 
> To set a delay on an already existing attribute, you have to use
> attrd_updater --update-delay or --update-both.
> 
> > Moreover, when you delete this attribute the actual remove will be
> > delayed by that "--delay" which was used when the attribute was set.
> >
> >
> > Thank you,
> > Kostia
> >
> > On Tue, Nov 29, 2016 at 1:08 AM, Ken Gaillot  <mailto:kgail...@redhat.com>
> > <mailto:kgail...@redhat.com <mailto:kgail...@redhat.com>>> wrote:
> >
> > On 11/24/2016 05:24 AM, Kostiantyn Ponomarenko wrote:
> > > Attribute dampening doesn't work for me also.
> > > To test that I have a script:
> > >
> > > attrd_updater -N node-0 -n my-attr --update false --delay 20
> > > sleep 3
> > > attrd_updater -N node-0 -n my-attr --update true
> > > sleep 7
> > > attrd_updater -N node-1 -n my-attr --update true
> >
> > This sequence works for me -- the attributes are not written
> to the live
> > CIB until the end of the delay, when both are written at the
> same time.
> >
> > The remaining issue must be with the policy engine. You could
> look at
> > the detail log on the DC when these changes were made; you
> should see
> > info-level messages with the CIB change with both values
> together (lines
> > with "cib_perform_op:   ++" and the attribute values), then
> "Transition
> > aborted" with "Transient attribute change", then a bunch of
> "pengine:"
> > lines saying what the cluster wants to do with each

Re: [ClusterLabs] Pacemaker 1.1.16 released

[Ken Gaillot]

On 12/01/2016 10:13 AM, Jehan-Guillaume de Rorthais wrote:
> On Wed, 30 Nov 2016 14:05:19 -0600
> Ken Gaillot  wrote:
> 
>> ClusterLabs is proud to announce the latest release of the Pacemaker
>> cluster resource manager, version 1.1.15.
> 
> 1.1.6 I guess ;)

Whoops!

Well now I don't feel so bad since the correction is wrong too ;)

> But congrats anyway !
> 
>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
>> properly passed to multistate resources with notification enabled. This
>> has been fixed. To help resource agents detect when the fix is
>> available, the CRM feature set has been incremented. (Whenever the
>> feature set changes, mixed-version clusters are supported only during
>> rolling upgrades -- nodes with an older version will not be allowed to
>> rejoin once they shut down.)
> 
>   * how could we get the "CRM feature set" version from the RA?
>   * when this "CRM feature set" has been introduced in Pacemaker?
> 
> Regards,
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] clone resource - pacemaker remote

[Ken Gaillot]

On 12/02/2016 07:08 AM, philipp.achmuel...@arz.at wrote:
> hi,
> 
> what is best way to prevent clone resource trying to run on remote/guest
> nodes?

location constraints with a negative score:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_deciding_which_nodes_a_resource_can_run_on


you can even use a single constraint with a rule based on #kind ne
cluster, so you don't need a separate constraint for each node:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_node_attribute_expressions


alternatively, you can set symmetric-cluster=false and use positive
constraints for cluster nodes only

> 
> ...
> node 167873318: lnx0223a \
> attributes maintenance=off
> node 167873319: lnx0223b \
> attributes maintenance=off
> ...
> /primitive vm-lnx0107a VirtualDomain \/
> /params hypervisor="qemu:///system"
> config="/etc/kvm/lnx0107a.xml" \/
> /meta remote-node=lnx0107a238 \/
> /utilization cpu=1 hv_memory=4096/
> /primitive remote-lnx0106a ocf:pacemaker:remote \/
> /params server=xx.xx.xx.xx \/
> /meta target-role=Started/
> /group base-group dlm clvm vg1/
> /clone base-clone base-group \/
> /meta interleave=true target-role=Started/
> /.../
> 
> /Dec  1 14:32:57 lnx0223a crmd[9826]:   notice: Initiating start
> operation dlm_start_0 on lnx0107a238/
> /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: executing -
> rsc:dlm action:start call_id:7/
> /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: finished -
> rsc:dlm action:start call_id:7  exit-code:5 exec-time:16ms queue-time:0ms/
> /Dec  1 14:32:58 lnx0223b crmd[9328]:error: Result of start
> operation for dlm on lnx0107a238: Not installed/
> /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
> on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
> /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
> on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
> /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
> start for dlm:2 on lnx0107a238: not installed (5)/
> /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
> start for dlm:2 on lnx0107a238: not installed (5)/
> /.../
> /Dec  1 14:32:49 lnx0223a pengine[9824]:   notice: Start  
> dlm:3#011(remote-lnx0106a)/
> /Dec  1 14:32:49 lnx0223a crmd[9826]:   notice: Initiating monitor
> operation dlm_monitor_0 locally on remote-lnx0106a/
> /Dec  1 14:32:50 lnx0223a crmd[9826]:error: Result of probe
> operation for dlm on remote-lnx0106a: Not installed/
> /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
> on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
> /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
> on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
> /.../
> 
> ---
> env: pacemaker-1.1.15-19.15.x86_64
> 
> thank you!

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] hawk - pacemaker remote

[Ken Gaillot]

On 12/02/2016 07:38 AM, philipp.achmuel...@arz.at wrote:
> Hi,
> 
> pacemaker remote nodes do not show up in hawk gui.
> regarding documentation, this should work - any hints to activate this?
> 
> thank you!
> 
> env: (SLES12.2)
> pacemaker-1.1.15-19.15.x86_64
> hawk2-2.0.0+git.1468428505.0135e38-24.32.x86_64

I'm not familiar with hawk but based on earlier conversations with
someone who uses it, it may only see nodes in the  CIB section,
so try setting any permanent node attribute on the remote nodes to make
them show up

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker 1.1.16 released

[Ken Gaillot]

On 12/01/2016 11:58 AM, Jehan-Guillaume de Rorthais wrote:
> 
> 
> Le 1 décembre 2016 17:39:45 GMT+01:00, Ken Gaillot  a 
> écrit :
>> On 12/01/2016 10:13 AM, Jehan-Guillaume de Rorthais wrote:
>>> On Wed, 30 Nov 2016 14:05:19 -0600
>>> Ken Gaillot  wrote:
>>>
>>>> ClusterLabs is proud to announce the latest release of the Pacemaker
>>>> cluster resource manager, version 1.1.15.
>>>
>>> 1.1.6 I guess ;)
>>
>> Whoops!
>>
>> Well now I don't feel so bad since the correction is wrong too ;)
> 
> Argh !! :-D 
> 
> What about my questions bellow BTW ? :-P 
> 
>>> But congrats anyway !
>>>
>>>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not
>>>> properly passed to multistate resources with notification enabled.
>>>> This has been fixed. To help resource agents detect when the fix is
>>>> available, the CRM feature set has been incremented. (Whenever the
>>>> feature set changes, mixed-version clusters are supported only during
>>>> rolling upgrades -- nodes with an older version will not be allowed to
>>>> rejoin once they shut down.)
>>>
>>>   * how could we get the "CRM feature set" version from the RA?

it's passed as an environment variable

>>>   * when this "CRM feature set" has been introduced in Pacemaker?

always, see http://wiki.clusterlabs.org/wiki/ReleaseCalendar


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] need some help with failing resources

[Ken Gaillot]

On 12/05/2016 09:30 AM, Darko Gavrilovic wrote:
> On 12/5/2016 10:17 AM, Ken Gaillot wrote:
>> On 12/03/2016 05:19 AM, Darko Gavrilovic wrote:
>>> Here is the output for that resource.. edited
>>>
>>> primitive svc-mysql ocf:heartbeat:mysql \
>>> params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf"
>>> datadir="/var/lib/mysql" user="mysql" group="mysql"
>>> log="/var/log/mysqld.log" pid="/var/run/mysqld/mysqld.pid"
>>> socket="/var/lib/mysql/mysql.sock" test_table="***" test_user="***"
>>> test_passwd="" \
>>> op monitor interval="30s" timeout="60s" OCF_CHECK_LEVEL="5" \
>>> op start interval="0" timeout="120s" \
>>> op stop interval="0" timeout="120s" \
>>> meta target-role="Started" migration-threshold="2"
>>>
>>> ...skipping
>>> order mysql-before-httpd inf: svc-mysql:start svc-httpd:start
>>> order mysql-before-ssh inf: svc-mysql:start svc-ssh:start
>>> property $id="cib-bootstrap-options" \
>>> dc-version="1.0.6-f709c638237cdff7556cb6ab615f32826c0f8c06" \
>>> cluster-infrastructure="openais" \
>>> expected-quorum-votes="2" \
>>> last-lrm-refresh="1480762389" \
>>> no-quorum-policy="ignore" \
>>> stonith-enabled="true" \
>>> ms-drbd0="Master"
>>>
>>>
>>> dg
>>>
>>>
>>> On 12/3/2016 1:25 AM, Kostiantyn Ponomarenko wrote:
>>>> I assume that you are using crmsh.
>>>> If so, I suggest to post an output from "crm configure show" command
>>>> here.
>>>>
>>>> Thank you,
>>>> Kostia
>>>>
>>>> On Sat, Dec 3, 2016 at 5:54 AM, Darko Gavrilovic
>>>> mailto:da...@chass.utoronto.ca>> wrote:
>>>>
>>>> Hello, I have a two node cluster running that seems to be
>>>> failing to
>>>> start resources.
>>>>
>>>>  Resource Group: services
>>>>  svc-mysql  (ocf::heartbeat:mysql) Stopped
>>>>  svc-httpd  (ocf::heartbeat:apache) Stopped
>>>>  svc-ssh(lsb:sshd-virt) Stopped
>>>>  svc-tomcat6(lsb:tomcat6) Stopped
>>>>  svc-plone  (lsb:plone) Stopped
>>>>  svc-bacula (lsb:bacula-fd-virt) Stopped
>>>>
>>>> When I run crm resource start services the service group does not
>>>> start.
>>>>
>>>> I also tried starting the first resource in the group.
>>>> crm resource start svc-mysql
>>>>
>>>> It does not start either.
>>>>
>>>> The error I am seeing is:
>>>> Dec  2 21:59:43  pengine: [25829]: WARN: native_color: Resource
>>>> svc-mysql cannot run anywhere
>>>> Dec  2 22:00:26  pengine: [25829]: WARN: native_color: Resource
>>>> svc-mysql cannot run anywhere
>>
>> The most common reasons for the above message are:
>>
>> * Location or order constraints don't leave any place for the resource
>> to run
>>
>> * The resource has failed the maximum number of times on all nodes.
>> (Does "crm_mon" show any failures?)
> 
> crm_mon does not list any failures for this service group from what I
> can see.
> 
>>
>>>>
>>>> 4b4f-a239-8a10dad40587, cib=0.3857.2) : Resource op removal
>>>> Dec  2 21:59:32 server1 crmd: [25830]: info: te_rsc_command:
> 
> 
>>>> Initiating action 56: monitor svc-mysql_monitor_0 on server2
>>>> Dec  2 21:59:33 server1 crmd: [25830]: WARN: status_from_rc: Action
>>>> 56 (svc-mysql_monitor_0) on server2 failed (target: 7 vs. rc: 0):
>>>> Error
>>
>> The above error indicates that mysql is running on server2 but the
>> cluster didn't start it there. (The "_monitor_0" is called a "probe";
>> it's used to check the status of the service before the cluster starts
>> it. The "target: 7" means it expects the service to be stopped. The "rc:
>> 0" means the service is actually running.)
>>
>> Make sure you're not starting mysql at boot or by any o

Re: [ClusterLabs] Antwort: Re: clone resource - pacemaker remote

[Ken Gaillot]

On 12/05/2016 09:20 AM, philipp.achmuel...@arz.at wrote:
> Ken Gaillot  schrieb am 02.12.2016 19:27:09:
> 
>> Von: Ken Gaillot 
>> An: users@clusterlabs.org
>> Datum: 02.12.2016 19:32
>> Betreff: Re: [ClusterLabs] clone resource - pacemaker remote
>>
>> On 12/02/2016 07:08 AM, philipp.achmuel...@arz.at wrote:
>> > hi,
>> >
>> > what is best way to prevent clone resource trying to run on remote/guest
>> > nodes?
>>
>> location constraints with a negative score:
>>
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> Pacemaker_Explained/index.html#_deciding_which_nodes_a_resource_can_run_on
>>
>>
>> you can even use a single constraint with a rule based on #kind ne
>> cluster, so you don't need a separate constraint for each node:
>>
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> Pacemaker_Explained/index.html#_node_attribute_expressions
>>
>>
>> alternatively, you can set symmetric-cluster=false and use positive
>> constraints for cluster nodes only
>>
> 
> set constraint to single primitive, group, or on clone resource?
> are there any advantages/disadvantages using one of these methods?

When a resource is cloned, you want to refer to the clone name in any
constraints, rather than the primitive name.

For a group, it doesn't really matter, but it's simplest to use the
group name in constraints -- mainly that keeps you from accidentally
setting conflicting constraints on different members of the group. And
of course group members are automatically ordered/colocated with each
other, so you don't need individual constraints for that.

> 
>> >
>> > ...
>> > node 167873318: lnx0223a \
>> > attributes maintenance=off
>> > node 167873319: lnx0223b \
>> > attributes maintenance=off
>> > ...
>> > /primitive vm-lnx0107a VirtualDomain \/
>> > /params hypervisor="qemu:///system"
>> > config="/etc/kvm/lnx0107a.xml" \/
>> > /meta remote-node=lnx0107a238 \/
>> > /utilization cpu=1 hv_memory=4096/
>> > /primitive remote-lnx0106a ocf:pacemaker:remote \/
>> > /params server=xx.xx.xx.xx \/
>> > /meta target-role=Started/
>> > /group base-group dlm clvm vg1/
>> > /clone base-clone base-group \/
>> > /meta interleave=true target-role=Started/
>> > /.../
>> >
>> > /Dec  1 14:32:57 lnx0223a crmd[9826]:   notice: Initiating start
>> > operation dlm_start_0 on lnx0107a238/
>> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: executing -
>> > rsc:dlm action:start call_id:7/
>> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice: finished -
>> > rsc:dlm action:start call_id:7  exit-code:5 exec-time:16ms
> queue-time:0ms/
>> > /Dec  1 14:32:58 lnx0223b crmd[9328]:error: Result of start
>> > operation for dlm on lnx0107a238: Not installed/
>> > /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
>> > on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
>> > /Dec  1 14:32:58 lnx0223a crmd[9826]:  warning: Action 31 (dlm_start_0)
>> > on lnx0107a238 failed (target: 0 vs. rc: 5): Error/
>> > /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
>> > start for dlm:2 on lnx0107a238: not installed (5)/
>> > /Dec  1 14:34:07 lnx0223a pengine[9824]:  warning: Processing failed op
>> > start for dlm:2 on lnx0107a238: not installed (5)/
>> > /.../
>> > /Dec  1 14:32:49 lnx0223a pengine[9824]:   notice: Start  
>> > dlm:3#011(remote-lnx0106a)/
>> > /Dec  1 14:32:49 lnx0223a crmd[9826]:   notice: Initiating monitor
>> > operation dlm_monitor_0 locally on remote-lnx0106a/
>> > /Dec  1 14:32:50 lnx0223a crmd[9826]:error: Result of probe
>> > operation for dlm on remote-lnx0106a: Not installed/
>> > /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
>> > on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
>> > /Dec  1 14:32:50 lnx0223a crmd[9826]:  warning: Action 5 (dlm_monitor_0)
>> > on remote-lnx0106a failed (target: 7 vs. rc: 5): Error/
>> > /.../
>> >
>> > ---
>> > env: pacemaker-1.1.15-19.15.x86_64
>> >
>> > thank you!


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Error performing operation: Argument list too long

[Ken Gaillot]

On 12/05/2016 02:29 PM, Shane Lawrence wrote:
> I'm experiencing a strange issue with pacemaker. It is unable to check
> the status of a systemd resource.
> 
> systemctl shows that the service crashed:
> [root@xx ~]# systemctl status rsyslog
> ● rsyslog.service - System Logging Service
>Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
> vendor preset: enabled)
>Active: inactive (dead) since Mon 2016-12-05 07:41:11 UTC; 12h ago
>  Docs: man:rsyslogd(8)
>http://www.rsyslog.com/doc/
>  Main PID: 22703 (code=exited, status=0/SUCCESS)
> 
> Dec 02 21:41:41 xx...xx systemd[1]: Starting Cluster
> Controlled rsyslog...
> Dec 02 21:41:41 xx...xx systemd[1]: Started Cluster
> Controlled rsyslog.
> Dec 05 07:41:08 xx...xx systemd[1]: Stopping System
> Logging Service...
> Dec 05 07:41:11 xx...xx systemd[1]: Stopped System
> Logging Service.
> Dec 05 07:41:40 xx...xx systemd[1]: Stopped System
> Logging Service.
> 
> Attempting to view the status through Pacemaker shows:
> [root@xx ~]# crm_resource --force-check -V -r rsyslog
> Error performing operation: Argument list too long
> [root@xx ~]# pcs resource debug-monitor rsyslog --full
> Error performing operation: Argument list too long
> 
> The problem seems to be resolved (temporarily) by restarting corosync
> and then starting the cluster again.
> 
> Has anyone else experienced this?

That is odd behavior. You may want to open a bug report at
bugs.clusterlabs.org and attach your configuration and logs.

On Linux, the system error number for "Argument list too long" is the
same as the OCF monitor status "Not running", so I suspect that it's a
display issue rather than an actual error, but I'm not sure.

Then the question would just be why is rsyslog stopping.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [cluster-lab] reboot standby node

[Ken Gaillot]

On 12/11/2016 04:19 PM, Omar Jaber wrote:
> Hi all ,
> 
> I have cluster contains three  nodes  with different sore  for location 
> constrain and  I have  group resource (it’s a service  exsists  in
> /etc/init.d/  folder)
> 
> Running  on the  node  the  have  the highest score  for   location
> constrain when I  try to  reboot one  of  the standby nodeI  see
> when the standby node become  up  the resource  stopped  in master node
> and restart againafter  I check the  pacemaker  status  I see the
> following error  :
> 
> "error: resource  'resource_name' is active on 2 nodes attempting
> recovery "  
> 
> Then I disables the  pcs  cluster  service in boot t time in standby
> node by run the command  "/_pcs_//cluster disable / " then I reboot the
> node  and I  see the resource  is started in standby node ( because  the
> resource  stored in /etc/init.d folder)
> 
> After that I  run the  pcs cluster  service  in standby node  and  I see
> the same  error is  generated  
> 
> "error: resource  'resource_name' is active on 2 nodes attempting
> recovery "
> 
>  
> 
> The problem  is  without reboot standby node this  problem not  happen
> for  example  
> 
> If  I stop pcs  cluster service  in standby , run the  resource  in
> standby node , then I start  pcs cluster
> 
> The error   "error: resource  'resource_name' is active on 2 nodes
> attempting recovery "   not  generated in this case.

Make sure your resource agent returns exit codes expected by Pacemaker:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-ocf-return-codes

In particular, if a monitor command returns 0 (OCF_SUCCESS), it means
the service is running.

When any node reboots, Pacemaker will "probe" the existing state of all
resources on it, by running a one-time monitor command. If the service
is not running, the command should return 7 (OCF_NOT_RUNNING).

So, I'm guessing that either the resource agent is wrongly returning 0
for monitor when the service is not actually running, or the node is
wrongly starting the service at boot, outside cluster control.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antwort: Re: hawk - pacemaker remote

[Ken Gaillot]

On 12/12/2016 07:05 AM, philipp.achmuel...@arz.at wrote:
> 
>> Von: Ken Gaillot 
>> An: users@clusterlabs.org
>> Datum: 02.12.2016 19:32
>> Betreff: Re: [ClusterLabs] hawk - pacemaker remote
>>
>> On 12/02/2016 07:38 AM, philipp.achmuel...@arz.at wrote:
>> > Hi,
>> >
>> > pacemaker remote nodes do not show up in hawk gui.
>> > regarding documentation, this should work - any hints to activate this?
>> >
>> > thank you!
>> >
>> > env: (SLES12.2)
>> > pacemaker-1.1.15-19.15.x86_64
>> > hawk2-2.0.0+git.1468428505.0135e38-24.32.x86_64
>>
>> I'm not familiar with hawk but based on earlier conversations with
>> someone who uses it, it may only see nodes in the  CIB section,
>> so try setting any permanent node attribute on the remote nodes to make
>> them show up
>>
> 
> tried several things, didn't get this working.
> Any examples how to configure this?
> Also how to configure for VirtualDomain with remote_node enabled

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote/index.html#_integrate_guest_into_cluster

You can set a node attribute on a remote node the same as any other
node, using crm_attribute (or your favorite higher-level CLI or GUI tool)

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antwort: Re: Antwort: Re: hawk - pacemaker remote

[Ken Gaillot]

On 12/13/2016 05:26 AM, philipp.achmuel...@arz.at wrote:
> 
>> Von: Kristoffer Grönlund 
>> An: philipp.achmuel...@arz.at, kgail...@redhat.com, Cluster Labs -
>> All topics related to open-source clustering welcomed
> 
>> Datum: 12.12.2016 16:13
>> Betreff: Re: [ClusterLabs] Antwort: Re:  hawk - pacemaker remote
>>
>> philipp.achmuel...@arz.at writes:
>>
>> >
>> > tried several things, didn't get this working.
>> > Any examples how to configure this?
>> > Also how to configure for VirtualDomain with remote_node enabled
>> >
>> > thank you!
>> >
>>
>> Without any details, it is difficult to help - what things did you try,
>> what does "not working" mean? Hawk can show remote nodes, but it only
>> shows them if they have entries in the nodes section of the
>> configuration (as Ken said).
>>
> 
> hi,
> 
> this is my current testsystem:
> 
> /Online: [ lnx0223a lnx0223b ]/
> /GuestOnline: [ vm-lnx0106a@lnx0223b vm-lnx0107a@lnx0223a ]/
> 
> /Full list of resources:/
> 
> /stonith_sbd (stonith:external/sbd): Started lnx0223a/
> / Clone Set: base-clone [base-group]/
> / Started: [ lnx0223a lnx0223b ]/
> / Stopped: [ vm-lnx0106a vm-lnx0107a ]/
> /FAKE1   (ocf::pacemaker:Dummy): Started lnx0223b/
> / Clone Set: post-clone [postfix-service]/
> / Started: [ vm-lnx0106a vm-lnx0107a ]/
> / Stopped: [ lnx0223a lnx0223b ]/
> /fence-lnx0106a  (stonith:external/libvirt): Started lnx0223b/
> /fence-lnx0107a  (stonith:external/libvirt): Started lnx0223a/
> /lnx0106a(ocf::heartbeat:VirtualDomain): Started lnx0223b/
> /lnx0107a(ocf::heartbeat:VirtualDomain): Started lnx0223a/
> /remote-paceip   (ocf::heartbeat:IPaddr):Started vm-lnx0106a/
> 
> node section shows:
> /.../
> //
> /  /
> //
> /   value="off"/>/
> //
> /  /
> /  /
> //
> /   value="off"/>/
> //
> /  /
> /  /
> //
> /   value="true"/>/
> //
> /  /
> /  /
> //
> /   value="true"/>/
> //
> /  /
> //
> /.../
> 
> 
> Hawk/Status still show remote nodes as "Offline"

Ah, it's good then ;)

Getting them to show up at all is the main goal. I'm going to guess it
pulls the status from the node state section; that has been maintained
for remote nodes only since Pacemaker 1.1.15, and there are still some
corner cases being addressed. Cluster nodes' state is learned
automatically from corosync, but remote nodes have no such mechanism, so
tracking state is much trickier.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] changing default cib.xml directory

[Ken Gaillot]

On 12/13/2016 09:57 AM, Christopher Harvey wrote:
> I was wondering if it is possible to tell pacemaker to store the cib.xml
> file in a specific directory. I looked at the code and searched the web
> a bit and haven't found anything. I just wanted to double check here in
> case I missed anything.
> 
> Thanks,
> Chris

Only when building the source code, with ./configure --localstatedir=DIR
which defaults to /var (pacemaker will always add /lib/pacemaker/cib to it)

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antwort: Re: Antwort: Re: clone resource - pacemaker remote

[Ken Gaillot]

On 12/07/2016 06:26 AM, philipp.achmuel...@arz.at wrote:
>> Von: Ken Gaillot 
>> An: philipp.achmuel...@arz.at, Cluster Labs - All topics related to
>> open-source clustering welcomed 
>> Datum: 05.12.2016 17:38
>> Betreff: Re: Antwort: Re: [ClusterLabs] clone resource - pacemaker remote
>>
>> On 12/05/2016 09:20 AM, philipp.achmuel...@arz.at wrote:
>> > Ken Gaillot  schrieb am 02.12.2016 19:27:09:
>> >
>> >> Von: Ken Gaillot 
>> >> An: users@clusterlabs.org
>> >> Datum: 02.12.2016 19:32
>> >> Betreff: Re: [ClusterLabs] clone resource - pacemaker remote
>> >>
>> >> On 12/02/2016 07:08 AM, philipp.achmuel...@arz.at wrote:
>> >> > hi,
>> >> >
>> >> > what is best way to prevent clone resource trying to run on
> remote/guest
>> >> > nodes?
>> >>
>> >> location constraints with a negative score:
>> >>
>> >> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> >>
> Pacemaker_Explained/index.html#_deciding_which_nodes_a_resource_can_run_on
>> >>
>> >>
>> >> you can even use a single constraint with a rule based on #kind ne
>> >> cluster, so you don't need a separate constraint for each node:
>> >>
>> >> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/
>> >> Pacemaker_Explained/index.html#_node_attribute_expressions
>> >>
>> >>
>> >> alternatively, you can set symmetric-cluster=false and use positive
>> >> constraints for cluster nodes only
>> >>
>> >
>> > set constraint to single primitive, group, or on clone resource?
>> > are there any advantages/disadvantages using one of these methods?
>>
>> When a resource is cloned, you want to refer to the clone name in any
>> constraints, rather than the primitive name.
>>
>> For a group, it doesn't really matter, but it's simplest to use the
>> group name in constraints -- mainly that keeps you from accidentally
>> setting conflicting constraints on different members of the group. And
>> of course group members are automatically ordered/colocated with each
>> other, so you don't need individual constraints for that.
>>
> 
> set location constraint to group didn't work:
> 
> ERROR: error: unpack_location_tags:Constraint
> 'location-base-group': Invalid reference to 'base-group'

Maybe a syntax error in your command, or a bug in the tool you're using?
The CIB XML is fine with something like this:



> for clone it works like expected.
> but crm_mon is showing "disallowed" set as "stopped". is this "works as
> designed" or how to prevent this?

You asked it to :)

-r == show inactive resources

> 
> crm configure show
> ...
> location location-base-clone base-clone resource-discovery=never \
>rule -inf: #kind ne cluster
> ...
> 
> crm_mon -r
>  Clone Set: base-clone [base-group]
>  Started: [ lnx0223a lnx0223b ]
>  Stopped: [ vm-lnx0106a vm-lnx0107a ]
> 
>> >
>> >> >
>> >> > ...
>> >> > node 167873318: lnx0223a \
>> >> > attributes maintenance=off
>> >> > node 167873319: lnx0223b \
>> >> > attributes maintenance=off
>> >> > ...
>> >> > /primitive vm-lnx0107a VirtualDomain \/
>> >> > /params hypervisor="qemu:///system"
>> >> > config="/etc/kvm/lnx0107a.xml" \/
>> >> > /meta remote-node=lnx0107a238 \/
>> >> > /utilization cpu=1 hv_memory=4096/
>> >> > /primitive remote-lnx0106a ocf:pacemaker:remote \/
>> >> > /params server=xx.xx.xx.xx \/
>> >> > /meta target-role=Started/
>> >> > /group base-group dlm clvm vg1/
>> >> > /clone base-clone base-group \/
>> >> > /meta interleave=true target-role=Started/
>> >> > /.../
>> >> >
>> >> > /Dec  1 14:32:57 lnx0223a crmd[9826]:   notice: Initiating start
>> >> > operation dlm_start_0 on lnx0107a238/
>> >> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice:
> executing -
>> >> > rsc:dlm action:start call_id:7/
>> >> > /Dec  1 14:32:58 lnx0107a pacemaker_remoted[1492]:   notice:
> finished -
>> >> > rsc:dlm action:start call_id:7  exit-code:5 exec-time:16ms
>> > 

Re: [ClusterLabs] Warning: handle_startup_fencing: Blind faith: not fencing unseen nodes

[Ken Gaillot]

On 12/14/2016 11:14 AM, Denis Gribkov wrote:
> Hi Everyone,
> 
> Our company have 15-nodes asynchronous cluster without actually
> configured FENCING/STONITH (as I think) features.
> 
> The DC node log getting tons of messages like in subject:
> 
> pengine:  warning: handle_startup_fencing:  Blind faith: not fencing
> unseen nodes

This is logged because you have set "startup-fencing: false".

It's logged as a warning because that setting is potentially dangerous:
a node that hasn't been seen by the cluster is not necessarily down --
it could be up and accessing shared resources, but unable to communicate
with the cluster. The only safe action is for the cluster to fence the node.

As of Pacemaker 1.1.16, the message will be logged only once. Before
Pacemaker 1.1.16, it is logged once per node, every time Pacemaker
checks the cluster state.

Of course, having "stonith-enabled: false" is also dangerous, because
fencing is the only way to recover from certain error conditions.

> the message repeated 15 times before next 15 messages:
> 
> pengine: info: determine_online_status: Node Node1 is online
> 
> ...
> 
> pengine: info: determine_online_status: Node Node15 is online
> 
> 
> The issue looks like similar to:
> 
> http://oss.clusterlabs.org/pipermail/pacemaker/2014-June/021995.html
> 
> but with own features.
> 
> 
> Our variables:
> 
> Oracle Linux Server release 6.8
> 
> Pacemaker 1.1.14-8.el6
> 
> Corosync Cluster Engine, version '1.4.7'
> 
> CMAN 3.0.12.1
> 
> 
> Cluster Properties:
>  cluster-infrastructure: cman
>  cluster-recheck-interval: 1
>  dc-version: 1.1.14-8.el6-70404b0
>  expected-quorum-votes: 3
>  have-watchdog: false
>  last-lrm-refresh: 1481444797
>  maintenance-mode: false
>  no-quorum-policy: ignore
>  startup-fencing: false
>  stonith-enabled: false
>  symmetric-cluster: false
> 
> 
> Example of /etc/cluster/cluster.conf:
> 
> 
>   
> 
>   
> 
>   
> 
>   
> 
>   
>   
> 
> 
> <...>
> 
> 
>   
> 
>   
> 
>   
>   
> 
>   
>   
>   
> 
>   
>   
> 
> 
>   
>   
> 
> 
> Example of /etc/corosync/corosync.conf:
> 
> compatibility: whitetank
> 
> totem {
> version: 2
> secauth: on
> threads: 4
> rrp_mode: active
> 
> interface {
> 
> member {
> memberaddr: PRIVATE_IP_1
> }
> 
> ...
> 
> member {
> memberaddr: PRIVATE_IP_15
> }
> 
> ringnumber: 0
> bindnetaddr: PRIVATE_NET_ADDR
> mcastaddr: 226.0.0.1
> mcastport: 5407
> ttl: 1
> }
> 
>interface {
> 
> member {
> memberaddr: PUBLIC_IP_1
> }
> ...
> 
> member {
> memberaddr: PUBLIC_IP_15
> }
> 
> ringnumber: 1
> bindnetaddr: PUBLIC_NET_ADDR
> mcastaddr: 224.0.0.1
> mcastport: 5405
> ttl: 1
> }
> 
> transport: udpu
> logging {
> fileline: off
> to_stderr: no
> to_logfile: yes
> logfile: /var/log/cluster/corosync.log
> logfile_priority: warning
> to_syslog: no
> debug: off
> timestamp: on
> logger_subsys {
>subsys: AMF
>debug: off
> }
> }
> 
> 
> Please let me know if you will need any other information.
> 
> Thank you.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Random failure with clone of IPaddr2

[Ken Gaillot]

On 12/15/2016 12:37 PM, al...@amisw.com wrote:
> Hi,
> 
> I got some trouble since one week and can't find solution by myself. Any
> help will be really appreciated !
> I use corosync / pacemaker for 3 or 4 years and all works well, for
> failover or load-balancing.
> 
> I have shared ip between 3 servers, and need to remove one for upgrade.
> But after I remove the server from the cluster i got random fail to access
> to my shared ip. I think first that some packet want go to the old server.
> So I put it again in the cluster, can reach it, but random failure is
> still here :-/
> 
> My test is just a curl http://my_ip (or ssh same stuff, random failed to
> connect).
> A ping didn't loose any packet.
> I can reach each of the three servers, but sometime, the request hang, and
> got a timeout.
> I see via tcpdump the packet coming, and resend, but no one respond. How I
> can diagnostic this ?
> I think one request on five fail. But I didn't see any messages in
> firewall or /var/log/message, nothing, just like the switch choose to
> remove random packet. I didn't see any counter on network interface, check
> the iptable setting, recheck the log, recheck all firewall ... Where go
> these packets ??
> 
> I try with another new ip, and same problem append. I try ip on two
> differents subnets (10.xxx and external ip) and same stuff.
> 
> I have no problem with virtual ip in failover mode.
> 
> If someone has any clue ...

Seeing your configuration might help. Did you set globally-unique=true
and clone-node-max=3 on the clone? If not, the other nodes can't pick up
the lost node's share of requests.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Random failure with clone of IPaddr2

[Ken Gaillot]

On 12/15/2016 02:02 PM, al...@amisw.com wrote:
>>
>> Seeing your configuration might help. Did you set globally-unique=true
>> and clone-node-max=3 on the clone? If not, the other nodes can't pick up
>> the lost node's share of requests.
> 
> Yes for both, I have globally-unique=true, and I change clone-node-max=3
> to clone-node-max=2, and now, as I come back to old configuration, I come
> back to clone-node-max=3
> 
> So now I have three node in the cluster.
> Here my config:
> 
> primitive ip_apache_localnet ocf:heartbeat:IPaddr2 params ip="10.0.0.99" 
> cidr_netmask="32" op monitor interval="30s"
> clone cl_ip_apache_localnet ip_apache_localnet \
> meta globally-unique="true" clone-max="3" clone-node-max="1"


^^^ Here you have clone-node-max="1", which will prevent surviving nodes
from picking up any failed node's share of requests. clone-max and
clone-node-max should both stay at 3, regardless of whether you are
intentionally taking down any node.

> target-role="Started" is-managed="true"
> 
>  sudo  /usr/sbin/iptables -L
> CLUSTERIP  all  --  anywhere 10.0.0.99  CLUSTERIP
> hashmode=sourceip-sourceport clustermac=A1:99:D6:EA:43:77 total_nodes=3
> local_node=2 hash_init=0
> 
> and check I have different local_node on each node.
> And just a question. Is the mac adress "normal" ? Doesn't need to begin
> with 01-00-5E ?

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] question about dc-deadtime

[Ken Gaillot]

On 12/15/2016 02:00 PM, Chris Walker wrote:
> Hello,
> 
> I have a quick question about dc-deadtime.  I believe that Digimer and
> others on this list might have already addressed this, but I want to
> make sure I'm not missing something.
> 
> If my understanding is correct, dc-deadtime sets the amount of time that
> must elapse before a cluster is formed (DC is elected, etc), regardless
> of which nodes have joined the cluster.  In other words, even if all
> nodes that are explicitly enumerated in the nodelist section have
> started Pacemaker, they will still wait dc-deadtime before forming a
> cluster.  
> 
> In my case, I have a two-node cluster on which I'd like to allow a
> pretty long time (~5 minutes) for both nodes to join before giving up on
> them.  However, if they both join quickly, I'd like to proceed to form a
> cluster immediately; I don't want to wait for the full five minutes to
> elapse before forming a cluster.  Further, if a node doesn't respond
> within five minutes, I want to fence it and start resources on the node
> that is up.

Pacemaker+corosync behaves as you describe by default.

dc-deadtime is how long to wait for an election to finish, but if the
election finishes sooner than that (i.e. a DC is elected), it stops
waiting. It doesn't even wait for all nodes, just a quorum.

Also, with startup-fencing=true (the default), any unseen nodes will be
fenced, and the remaining nodes will proceed to host resources. Of
course, it needs quorum for this, too.

With two nodes, quorum is handled specially, but that's a different topic.

> With Pacemaker/Heartbeat, the initdead parameter did exactly what I
> want, but I don't see any way to do this with Pacemaker/Corosync.  From
> reading other posts, it looks like people use an external agent to start
> HA daemons once nodes are up ... is this a correct understanding?
> 
> Thanks very much,
> Chris

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Nodes see each other as OFFLINE - fence agent (fence_pcmk) may not be working properly on RHEL 6.5

[Ken Gaillot]

On 12/16/2016 07:46 AM, avinash shankar wrote:
> 
> Hello team,
> 
> I am a newbie in pacemaker and corosync cluster.
> I am facing trouble with fence_agent on RHEL 6.5
> I have installed pcs, pacemaker, corosync, cman on RHEL 6.5 on two
> virtual nodes (libvirt) cluster.
> SELINUX and firewall is completely disabled.
> 
> # yum list installed | egrep 'pacemaker|corosync|cman|fence'
> cman.x86_64  3.0.12.1-78.el6
> @rhel-ha-for-rhel-6-server-rpms
> corosync.x86_64  1.4.7-5.el6
> @rhel-ha-for-rhel-6-server-rpms
> corosynclib.x86_64   1.4.7-5.el6
> @rhel-ha-for-rhel-6-server-rpms
> fence-agents.x86_64  4.0.15-12.el6  
> @rhel-6-server-rpms   
> fence-virt.x86_640.2.3-19.el6   
> @rhel-ha-for-rhel-6-server-eus-rpms
> pacemaker.x86_64 1.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
> pacemaker-cli.x86_64 1.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
> pacemaker-cluster-libs.x86_641.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
> pacemaker-libs.x86_641.1.14-8.el6_8.2   
> @rhel-ha-for-rhel-6-server-rpms
>  
> 
> I bring up cluster using pcs cluster start --all
> also done pcs property set stonith-enabled=false

fence_pcmk simply tells CMAN to use pacemaker's fencing ... it can't
work if pacemaker's fencing is disabled.

> Below is the status
> ---
> # pcs status
> Cluster name: roamclus
> Last updated: Fri Dec 16 18:54:40 2016Last change: Fri Dec 16
> 17:44:50 2016 by root via cibadmin on cnode1
> Stack: cman
> Current DC: NONE
> 2 nodes and 2 resources configured
> 
> Online: [ cnode1 ]
> OFFLINE: [ cnode2 ]
> 
> Full list of resources:
> 
> PCSD Status:
>   cnode1: Online
>   cnode2: Online
> ---
> Same kind of output is observed on other node = cnode2
> So nodes see each other as OFFLINE.
> Expected result is Online: [ cnode1 cnode2 ]
> I did same packages installation on RHEL 6.8 and when I am starting the
> cluster,
> it shows both nodes ONLINE from each other.
> 
> I need to resolve this such that on RHEL 6.5 nodes when we start cluster
> by default
> both nodes should display each others status as online.
> --
> Below is the  /etc/cluster/cluster.conf
> 
> 
>   
>   
> 
>   
> 
>   
> 
>   
> 
> 
>   
> 
>   
> 
>   
> 
>   
>   
>   
> 
>   
>   
> 
> 
>   
> 
> --
> # cat /var/lib/pacemaker/cib/cib.xml
>  num_updates="0" admin_epoch="0" cib-last-written="Fri Dec 16 18:57:10
> 2016" update-origin="cnode1" update-client="cibadmin" update-user="root"
> have-quorum="1" dc-uuid="cnode1">
>   
> 
>   
>  name="have-watchdog" value="false"/>
>  value="1.1.14-8.el6_8.2-70404b0"/>
>  name="cluster-infrastructure" value="cman"/>
>  name="stonith-enabled" value="false"/>
>   
> 
> 
>   
>   
> 
> 
> 
>   
> 
> 
> /var/log/messages have below contents :
> 
> Dec 15 20:29:43 cnode2 kernel: DLM (built Oct 26 2016 10:26:08) installed
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Corosync Cluster
> Engine ('1.4.7'): started and ready to provide service.
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Corosync built-in
> features: nss dbus rdma snmp
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Successfully read
> config from /etc/cluster/cluster.conf
> Dec 15 20:29:46 cnode2 corosync[2464]:   [MAIN  ] Successfully parsed
> cman config
> Dec 15 20:29:46 cnode2 corosync[2464]:   [TOTEM ] Initializing transport
> (UDP/IP Multicast).
> Dec 15 20:29:46 cnode2 corosync[2464]:   [TOTEM ] Initializing
> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Dec 15 20:29:46 cnode2 corosync[2464]:   [TOTEM ] The network interface
> [10.10.18.138] is now up.
> Dec 15 20:29:46 cnode2 corosync[2464]:   [QUORUM] Using quorum provider
> quorum_cman
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync cluster quorum service v0.1
> Dec 15 20:29:46 cnode2 corosync[2464]:   [CMAN  ] CMAN 3.0.12.1 (built
> Feb  1 2016 07:06:19) started
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync CMAN membership service 2.90
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> openais checkpoint service B.01.01
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync extended virtual synchrony service
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync configuration service
> Dec 15 20:29:46 cnode2 corosync[2464]:   [SERV  ] Service engine loaded:
> corosync cluster closed process group service v1.01
>

Re: [ClusterLabs] Cluster failure

[Ken Gaillot]

On 12/20/2016 12:21 AM, Rodrick Brown wrote:
> I'm fairly new to Pacemaker and have a few questions about 
> 
> The following log event and why resources was removed from my cluster 
> Right before the resources being killed SIGTERM I notice the following
> message. 
> Dec 18 19:18:18 clusternode38.mf stonith-ng[10739]:   notice: On loss of
> CCM Quorum: Ignore

The "notice:" part is the severity of the message -- "error:" or
"warning:" is bad, but anything else is purely informational.

In this case, this message indicates that you have
no-quorum-policy=ignore configured. It will be printed every time one of
the pacemaker daemons or tools reads your configuration, and has nothing
to do with your resource problem.

> What exactly does this mean my resources recovered after a few minutes
> and did not fail over any idea what's going on here? 

You'd need to look at a much larger chunk of the logs, typically on more
than one node. The cluster will elect one node to be the "Designated
Controller" (DC), and that node's logs will have essential information
about the decision-making process.

> or documentation I can read that explains what exactly happened? 
> 
> -- 
> 
> 
> *Rodrick Brown */ *Site Reliability Engineer 
> *(917) 445 - 6839 / *rbr...@marketfactory.com
> 
> **425 Broadway #3, New York, NY 10013*

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Can packmaker launch haproxy from new network namespace automatically?

[Ken Gaillot]

On 12/17/2016 07:26 PM, Hao QingFeng wrote:
> Hi Folks,
> 
> I am installing packmaker to manage the cluster of haproxy within
> openstack on ubuntu 16.04.
> 
> I met the problem that haproxy can't start listening for some services
> in vip because the related ports
> 
> were occupied by those native services which listened on 0.0.0.0.
> 
> I opened a bug to openstack team and a buddy told me that I should use
> pacemaker to run haproxy in
> 
> a separate network namespace.  I attached his description here(also in bug):
> 
> <<<
> 
> Fuel runs haproxy via pacemaker (not vis systemd/upstart) and pacemaker
> runs haproxy in a separate network namespace.
> 
> So haproxy does not cause any problems by listedning on 0.0.0.0 since
> it's listening in a separate network namespace.
> 
> You can see it via "ip netns ls" command and then "ip netns exec haproxy
> ip a".
> 
> Did you try to restart haproxy via systemd/upstart? If so then you could
> face this problem. You should use pacemaker to control haproxy service.
> 

> 
> Here is the bug link:
> 
> https://bugs.launchpad.net/openstack-manuals/+bug/1649902
> 
> Actually I did start haproxy with pacemaker but "ip netns ls" show
> nothing and haproxy can't bind some port like 9292 on vip .
> 
> I checked and found that openstack starts including this function from
> fuel 5.0(released in May, 2014).
> 
> But after I downloaded pacemaker's code, did a rough check, I couldn't
> find any related functions(keywords: ip netns, clone, CLONE_NEW...)
> 
> except in the test cases for neutron and ovs etc(if my understanding is
> correct).
> 
> I didn't see any related configuration item in "crm configure show" either.
> 
> 
> So I would like just  to confirm that if pacemaker has such function to
> create a new network namespace
> 
> for haproxy(or other manged service) automatically to avoid such socket
> binding conflict?
> 
> If yes, how to configure it? If no such function, do you have any advice
> on how to solve the problem?

No, pacemaker has no way to do that itself, but maybe you could run
haproxy inside a container, and manage the container as a pacemaker
resource.

> 
> BTW, you can see the detailed configuration information in the bug link,
> if you need more, please let me know.
> 
> Thanks a lot!
> 
> Regards!
> 
> -- 
> 
> QingFeng Hao(Robin)

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] New ClusterLabs logo unveiled :-)

[Ken Gaillot]

Hi all,

ClusterLabs is happy to unveil its new logo! Many thanks to the
designer, Kristoffer Grönlund , who graciously
donated the clever approach.

You can see it on our GitHub page:

  https://github.com/ClusterLabs

It is also now the site icon for www.clusterlabs.org and
wiki.clusterlabgs.org. Your browser might have cached the old version,
so you might not see the new one immediately, but you can see it by
going straight to the links and reloading:

  http://clusterlabs.org/favicon.ico
  http://clusterlabs.org/apple-touch-icon.png

It is also on the wiki banner, though the banner will need some tweaking
to make the best use of it. You might not see it there immediately due
to browser caching and DNS resolver caching (the wiki IP changed
recently as part of an OS upgrade), but it's there. :-)

Wishing everyone a happy holiday season,
-- 
Ken Gaillot 

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Access denied when using Floating IP

[Ken Gaillot]

On 12/26/2016 12:03 AM, Kaushal Shriyan wrote:
> Hi,
> 
> I have set up Highly Available HAProxy Servers with Keepalived and
> Floating IP.  I have the below details 
> 
> *Master Node keepalived.conf*
> 
> global_defs {
> # Keepalived process identifier
> #lvs_id haproxy_DH
> }
> # Script used to check if HAProxy is running
> vrrp_script check_haproxy {
> script "/usr/bin/killall -0 haproxy"
> interval 2
> weight 2
> }
> # Virtual interface
> # The priority specifies the order in which the assigned interface to
> take over in a failover
> vrrp_instance VI_01 {
> state MASTER
> interface eth0
> virtual_router_id 51
> priority 200
> # The virtual ip address shared between the two loadbalancers
> virtual_ipaddress {
> *172.16.0.75/32 *
> }
> track_script {
> check_haproxy
> }
> }
> 
> *Slave Node keepalived.conf*
> 
> global_defs {
> # Keepalived process identifier
> #lvs_id haproxy_DH_passive
> }
> # Script used to check if HAProxy is running
> vrrp_script check_haproxy {
> script "/usr/bin/killall -0 haproxy"
> interval 2
> weight 2
> }
> # Virtual interface
> # The priority specifies the order in which the assigned interface to
> take over in a failover
> vrrp_instance VI_01 {
> state BACKUP
> interface eth0
> virtual_router_id 51
> priority 100
> # The virtual ip address shared between the two loadbalancers
> virtual_ipaddress {
> 172.16.0.75/32 
> }
> track_script {
> check_haproxy
> }
> }
> 
> HAProxy Node 1 has two IP Addresses
> 
> eth0 :- 172.16.0.20 LAN IP of the box Master Node
> eth0 :- 172.16.0.75 Virtual IP
> 
> eth0 :- 172.16.0.21 LAN IP of the box Slave Node
> 
> In MySQL server, i have given access for the Floating IP :- 172.16.0.75
> 
> *GRANT USAGE ON *.* TO 'haproxy_check'@'172.16.0.75';
> *
> *GRANT ALL PRIVILEGES ON *.* TO 'haproxy_root'@'172.16.0.75' IDENTIFIED
> BY PASSWORD '*7A3F28E9F3E3AEFDFF87BCFE119DCF830101DD71' WITH GRANT OPTION;*
> 
> When i try to connect to the MySQL server using floating IP :- 172.16.0.75,
> I get access denied inspite of giving grant access as per the above
> mentioned command. When i try to use the static IP to connect to the
> MySQL server using LAN IP :- 172.16.0.20, it works as expected. is it
> because eth0 has two IPs :- 172.16.0.20 and 172.16.0.75?
> 
> Please do let me know if you need any additional information.
> 
> Regards,
> 
> Kaushal

People on this list tend to be more familiar with pacemaker clusters
than keepalived, but my guess is that mysql's privileges apply to the IP
address that the user is connecting *from*. Try giving the same
privileges to the user at all other local IPs (or @'%' if you don't mind
allowing connections from anywhere, and use a firewall to block unwanted
connections instead).

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] centos 7 drbd fubar

[Ken Gaillot]

On 12/27/2016 03:08 PM, Dimitri Maziuk wrote:
> I ran centos 7.3.1611 update over the holidays and my drbd + nfs + imap
> active-passive pair locked up again. This has now been consistent for at
> least 3 kernel updates. This time I had enough consoles open to run
> fuser & lsof though.
> 
> The procedure:
> 
> 1. pcs cluster standby 
> 2. yum up && reboot 
> 3. pcs cluster unstandby 
> 
> Fine so far.
> 
> 4. pcs cluster standby 
> results in
> 
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:41 INFO: Running 
>> stop for /dev/drbd0 on /raid
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:41 INFO: Trying to 
>> unmount /raid
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:41 ERROR: Couldn't 
>> unmount /raid; trying cleanup with TERM
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:41 INFO: No 
>> processes on /raid were signalled. force_unmount is set to 'yes'
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:42 ERROR: Couldn't 
>> unmount /raid; trying cleanup with TERM
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:42 INFO: No 
>> processes on /raid were signalled. force_unmount is set to 'yes'
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:43 ERROR: Couldn't 
>> unmount /raid; trying cleanup with TERM
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:43 INFO: No 
>> processes on /raid were signalled. force_unmount is set to 'yes'
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:44 ERROR: Couldn't 
>> unmount /raid; trying cleanup with KILL
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:44 INFO: No 
>> processes on /raid were signalled. force_unmount is set to 'yes'
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:45 ERROR: Couldn't 
>> unmount /raid; trying cleanup with KILL
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:46 INFO: No 
>> processes on /raid were signalled. force_unmount is set to 'yes'
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:47 ERROR: Couldn't 
>> unmount /raid; trying cleanup with KILL
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:47 INFO: No 
>> processes on /raid were signalled. force_unmount is set to 'yes'
>> Filesystem(drbd_filesystem)[18277]: 2016/12/23_17:36:48 ERROR: Couldn't 
>> unmount /raid, giving up!
>> Dec 23 17:36:48 [1138] zebrafish.bmrb.wisc.edu   lrmd:   notice: 
>> operation_finished:drbd_filesystem_stop_0:18277:stderr [ umount: 
>> /raid: target i
>> s busy. ]
> 
> ... until the system's powered down. Before power down I ran lsof, it
> hung, and fuser:
> 
>> # fuser -vum /raid
>>  USERPID ACCESS COMMAND
>> /raid:   root kernel mount (root)/raid
> 
> After running yum up on the primary and rebooting it again,
> 
> 5. pcs cluster unstandby 
> causes the same fail to unmount loop on the secondary, that has to be
> powered down until the primary recovers.
> 
> Hopefully I'm doing something wrong, please someone tell me what it is.
> Anyone? Bueller?

That is disconcerting. Since no one here seems to know, have you tried
asking on the drbd list? It sounds like an issue with the drbd kernel
module.

http://lists.linbit.com/listinfo/drbd-user


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Status and help with pgsql RA

[Ken Gaillot]

On 12/28/2016 02:24 PM, Nils Carlson wrote:
> Hi,
> 
> I am looking to set up postgresql in high-availability and have been
> comparing the guide at
> http://wiki.clusterlabs.org/wiki/PgSQL_Replicated_Cluster with the
> contents of the pgsql resource agent on github. It seems that there have
> been substantial improvements in the resource agent regarding the use of
> replication slots.
> 
> Could anybody look at updating the guide, or just sending it out in an
> e-mail to help spread knowledge?
> 
> The replications slots with pacemaker look really cool, if I've
> understood things right there should be no need for manual work after
> node recovery with the replication slots (though there is a risk of a
> full disk)?
> 
> All help, tips and guidance much appreciated.
> 
> Cheers,
> Nils

Hmm, that wiki page could definitely use updating. I'm personally not
familiar with pgsql, so hopefully someone else can chime in.

Another user on this list has made an alternative resource agent that
you might want to check out:

http://lists.clusterlabs.org/pipermail/users/2016-December/004740.html

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] question about dc-deadtime

[Ken Gaillot]

On 01/10/2017 11:59 AM, Chris Walker wrote:
> 
> 
> On Mon, Jan 9, 2017 at 6:55 PM, Andrew Beekhof  <mailto:abeek...@redhat.com>> wrote:
> 
> On Fri, Dec 16, 2016 at 8:52 AM, Chris Walker
> mailto:christopher.wal...@gmail.com>>
> wrote:
> > Thanks for your response Ken.  I'm puzzled ... in my case node remain
> > UNCLEAN (offline) until dc-deadtime expires, even when both nodes are 
> up and
> > corosync is quorate.
> 
> I'm guessing you're starting both nodes at the same time?
> 
> 
> The nodes power on at the same time, but hardware discovery can vary by
> minutes.
> 
>  
> 
> The behaviour you're seeing is arguably a hangover from the multicast
> days (in which case corosync wouldn't have had a node list).
> 
> 
> That makes sense. 
> 
> 
> But since that's not the common case anymore, we could probably
> shortcut the timeout if we know the complete node list and see that
> they are all online.
> 
> 
> That would be ideal.  It's easy enough to work around this in systemd,
> but it seems like the HA stack should be the authority on node status.

I've open a feature request:
http://bugs.clusterlabs.org/show_bug.cgi?id=5310

FYI the priority list is long at this point, so no idea when it might be
addressed.

> Thanks!
> Chris
> 
> 
> >
> > I see the following from crmd when I have dc-deadtime=2min
> >
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: Quorum acquired
> > Dec 15 21:34:33 max04 crmd[13791]:   notice:
> pcmk_quorum_notification: Node
> > max04[2886730248] - state is now member (was (null))
> > Dec 15 21:34:33 max04 crmd[13791]:   notice:
> pcmk_quorum_notification: Node
> > (null)[2886730249] - state is now member (was (null))
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: Notifications disabled
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: The local CRM is
> operational
> > Dec 15 21:34:33 max04 crmd[13791]:   notice: State transition
> S_STARTING ->
> > S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
> > ...
> > Dec 15 21:36:33 max05 crmd[10365]:  warning: FSA: Input
> I_DC_TIMEOUT from
> > crm_timer_popped() received in state S_PENDING
> > Dec 15 21:36:33 max05 crmd[10365]:   notice: State transition
> S_ELECTION ->
> > S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED
> > origin=election_timeout_popped ]
> > Dec 15 21:36:33 max05 crmd[10365]:  warning: FSA: Input
> I_ELECTION_DC from
> > do_election_check() received in state S_INTEGRATION
> > Dec 15 21:36:33 max05 crmd[10365]:   notice: Notifications disabled
> > Dec 15 21:36:33 max04 crmd[13791]:   notice: State transition
> S_PENDING ->
> > S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE
> > origin=do_cl_join_finalize_respond ]
> >
> > only after this do the nodes transition to Online.  This is using the
> > vanilla RHEL7.2 cluster stack and the following options:
> >
> > property cib-bootstrap-options: \
> > no-quorum-policy=ignore \
> > default-action-timeout=120s \
> > pe-warn-series-max=1500 \
> > pe-input-series-max=1500 \
> >     pe-error-series-max=1500 \
> > stonith-action=poweroff \
> > stonith-timeout=900 \
> > dc-deadtime=2min \
> > maintenance-mode=false \
> > have-watchdog=false \
> > dc-version=1.1.13-10.el7-44eb2dd \
> > cluster-infrastructure=corosync
> >
> > Thanks again,
> > Chris
> >
> > On Thu, Dec 15, 2016 at 3:26 PM, Ken Gaillot  <mailto:kgail...@redhat.com>> wrote:
> >>
> >> On 12/15/2016 02:00 PM, Chris Walker wrote:
> >> > Hello,
> >> >
> >> > I have a quick question about dc-deadtime.  I believe that
> Digimer and
> >> > others on this list might have already addressed this, but I
> want to
> >> > make sure I'm not missing something.
> >> >
> >> > If my understanding is correct, dc-deadtime sets the amount of
> time that
> >> > must elapse before a cluster is formed (DC is elected, etc),
> regardless
> >> > of which nodes have joined the cluster.  In other words, even
> if all
> >> > nodes that are explicitly enumerat

Re: [ClusterLabs] No match for shutdown action on

[Ken Gaillot]

On 01/10/2017 11:38 AM, Denis Gribkov wrote:
> Hi Everyone,
> 
> When I run:
> 
> # pcs resource cleanup resource_name
> 
> I'm getting a block of messages in log on current DC node:
> 
> Jan 10 18:12:13 node1 crmd[21635]:  warning: No match for shutdown
> action on node2
> Jan 10 18:12:13 node1 crmd[21635]:  warning: No match for shutdown
> action on node3
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node4
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node5
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node6
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node7
> Jan 10 18:12:14 node1 crmd[21635]:  warning: No match for shutdown
> action on node8
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node5
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node3
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node4
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node8
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node9
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node10
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node11
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node3
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node12
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node4
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node5
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node13
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node8
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node14
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node15
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node6
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node6
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node7
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node7
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node2
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node2
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node16
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node1
> Jan 10 18:12:18 node1 crmd[21635]:  warning: No match for shutdown
> action on node1
> Jan 10 18:12:23 node1 cib[21630]:  warning: A-Sync reply to crmd failed:
> No message of desired type
> 
> At the same time other nodes didn't getting these messages.
> 
> Does anybody know why this issue happening in such cases and how
> possible to fix it?
> 
> Cluster Properties:
>  cluster-infrastructure: cman
>  cluster-recheck-interval: 5min
>  dc-version: 1.1.14-8.el6-70404b0
>  expected-quorum-votes: 3
>  have-watchdog: false
>  last-lrm-refresh: 1484068350
>  maintenance-mode: false
>  no-quorum-policy: ignore
>  pe-error-series-max: 1000
>  pe-input-series-max: 1000
>  pe-warn-series-max: 1000
>  stonith-action: reboot
>  stonith-enabled: false
>  symmetric-cluster: false
> 
> Thank you.

The message is harmless and can be ignored. It only shows up on the node
that is DC.

The "fix" is to upgrade. :-) In later versions, the message was changed
to the more accurate "No reason to expect node  to be down", and it
is now only printed if the node was actually lost.

In the version you have, the message was printed whenever the cluster
checked to see if any known event would have brought down a node,
regardless of whether there was any actual problem. If there is an
actual problem, there will be other messages about that (e.g. node lost
or fenced).

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] New ClusterLabs logo unveiled :-)

[Ken Gaillot]

On 01/11/2017 03:54 AM, Kostiantyn Ponomarenko wrote:
> Nice logo!
> 
> http://wiki.clusterlabgs.org/ doesn't load for me.

Whoops, typo! There should be no "g" in there:

   http://wiki.clusterlabs.org/

> I also have a question which bothers me for a long time. Not a
> significant one, but anyways ... 
> I have seen a lot "Linux-HA" name around. But it seems that the name it
> not used anymore for this particular stack of HA software.
> So I wonder, what it the correct name for this stack? 
> 
> Thank you,
> Kostia

I don't know if there's an "official" name, but I've been calling it the
"ClusterLabs stack".

> On Mon, Jan 2, 2017 at 11:35 AM, Kristoffer Grönlund  <mailto:kgronl...@suse.com>> wrote:
> 
> Ken Gaillot mailto:kgail...@redhat.com>> writes:
> 
> > Hi all,
> >
> > ClusterLabs is happy to unveil its new logo! Many thanks to the
> > designer, Kristoffer Grönlund  <mailto:kgronl...@suse.com>>, who graciously
> > donated the clever approach.
> >
> > You can see it on our GitHub page:
> >
> >   https://github.com/ClusterLabs
> >
> > It is also now the site icon for www.clusterlabs.org 
> <http://www.clusterlabs.org> and
> > wiki.clusterlabgs.org <http://wiki.clusterlabgs.org>. Your browser
> might have cached the old version,
> > so you might not see the new one immediately, but you can see it by
> > going straight to the links and reloading:
> >
> >   http://clusterlabs.org/favicon.ico 
> <http://clusterlabs.org/favicon.ico>
> >   http://clusterlabs.org/apple-touch-icon.png
> <http://clusterlabs.org/apple-touch-icon.png>
> >
> > It is also on the wiki banner, though the banner will need some tweaking
> > to make the best use of it. You might not see it there immediately due
> > to browser caching and DNS resolver caching (the wiki IP changed
> > recently as part of an OS upgrade), but it's there. :-)
> >
> > Wishing everyone a happy holiday season,
> 
> Thanks for using my logo! Nice holiday surprise :)
> 
> Cheers,
> Kristoffer
> 
> > --
> > Ken Gaillot mailto:kgail...@redhat.com>>
> >
> > ___
> > Users mailing list: Users@clusterlabs.org <mailto:Users@clusterlabs.org>
> > http://lists.clusterlabs.org/mailman/listinfo/users
> <http://lists.clusterlabs.org/mailman/listinfo/users>
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
> > Bugs: http://bugs.clusterlabs.org
> 
> --
> // Kristoffer Grönlund
> // kgronl...@suse.com <mailto:kgronl...@suse.com>

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] simple setup and resources on different nodes??

[Ken Gaillot]

On 01/11/2017 10:10 AM, lejeczek wrote:
> hi eveyone,
> I have a simple, test setup, like this:
> 
> $ pcs status
> Cluster name: test_cluster
> WARNING: corosync and pacemaker node names do not match (IPs used in
> setup?)
> Stack: corosync
> Current DC: work2.whale.private (version 1.1.15-11.el7_3.2-e174ec8) -
> partition with quorum
> Last updated: Wed Jan 11 16:02:15 2017Last change: Wed Jan 11
> 15:53:36 2017 by root via crm_attribute on work1.whale.private
> 
> 3 nodes and 3 resources configured
> 
> Online: [ work1.whale.private work2.whale.private work3.whale.private ]
> 
> Full list of resources:
> 
>  Fencing(stonith:fence_xvm):Started work1.whale.private
>  virtualIP(ocf::heartbeat:IPaddr2):Started work2.whale.private
>  Website(ocf::heartbeat:apache):Started work3.whale.private
> 
> Daemon Status:
>   corosync: active/enabled
>   pacemaker: active/enabled
>   pcsd: active/enabled
> 
> 
> but it does not make sense (bare in mind a beginner talks) right?
> How come pcs decided to have VirtualIP and Website are started on
> different nodes?
> And it actually did, virtuall IP pings but httpd is not there, indeed
> running on another node.
> What am I missing?
> 
> many thanks,
> L.

You may want to see "Clusters From Scratch":

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/index.html

especially sections 6.4-6.6

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Log rotation for /var/log/debug-pcmk.log

[Ken Gaillot]

On 01/12/2017 05:33 AM, Jan Pokorný wrote:
> On 11/01/17 18:46 +, Andrew Nagy wrote:
>> I hope this is the right place
>> We have pacemaker running (on at least one) RHEL 7.2 server with a
>> huge (8+ GB) /var/log/debug-pcmk.log file. No one else here knows
>> anything about pacemaker and the person who built/configured the
>> server is unavailable for the next two weeks. There is *no* enough
>> free space on /var/log (bad, I know). How do I fix this with minimal
>> impact to the users or the application (NFS server)? If a service
>> outage is unavoidable, I can work with that too.
> 
> I suppose this, run as root, would work:
> 
>   cp /var/log/debug-pcmk.log   # optional
>   truncate -s 0 /var/log/debug-pcmk.log
> 
> You have to live with a fact that few lines queued to be logged in
> the interim (there may be overlaps runtime-wise) may be dropped.

The location of the log is most likely specified by PCMK_logfile in
/etc/sysconfig/pacemaker.

If it's really set to debug-level verbosity, that's unnecessary during
normal use. It may have been set that way during an issue for
troubleshooting.

Check in the same file that none of the PCMK_debug, PCMK_trace_*, or
PCMK_blackbox variables are set. Any change in that file will take
effect the next time the cluster is restarted on that node.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] just some basics

[Ken Gaillot]

On 01/12/2017 11:53 AM, lejeczek wrote:
> hi everyone,
> 
> I'm going through some introductory stuff, reading about it all and one,
> ok, a few questions come to mind.
> I was hoping, before I find answers somewhere at end of the docs someone
> here could quickly clarify:
> 
> It is one cluster per node, right? Not like, that a cluster is
> denominated by hostname/IP and if a node has multiple netIFs/IPs one
> could have multiple clusters (on that node) as long as these use only
> unique node's netIF/IP ?

Correct

> Would relying on IP type of resources only be safe? I'd imagine there
> are some services/apps that, (when use shared/distributed storage)
> without user interacting with them do not need to be off (because do not
> lock/access anything) and user could not (obviously) interact when that
> IP is being off on a node.

Having only IP resources can sometimes be useful, but it has drawbacks.

First, what's the use of having the IP highly available, if the service
itself goes down? For example, if the IP is for a web server, and the
web service goes down on the node that currently has the IP, the cluster
won't know anything is wrong and so will leave the IP there.

Second, the shared storage access itself typically should be a cluster
resource, to be able to use fencing to recover from split-brain.

Lastly, if you're trying to think of ways to avoid fencing, there's no
safe way to do that. Even for just an IP, if two machines try to start
the same IP, packets will go sometimes to one and sometimes the other,
rendering the service unusable.

> These two questions I form having assumed that it's the nodes/machines
> that are monitored and are declared un/available(those IPs of nodes) and
> resources are simply targets for an action the cluster would perform,
> and have nothing to do with "un/available" decision making process - or
> - I've got it wrong and should keep on reading(regardless probably :)) ?
> 
> many thanks,
> L.

The cluster monitors both the nodes and the resources. By default, a
resource failure doesn't make a node unavailable, but that's configurable.

Also, configuring resources for your services gives you considerable
flexibility in pacemaker -- monitoring/failure handling, complex
dependencies between resources (location/ordering), and high-level
features such as node RAM/CPU utilization, time-based rules, etc.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] get status of each RG

[Ken Gaillot]

On 01/15/2017 10:02 AM, Florin Portase wrote:
> Hello,
> 
> We're about to create some HPOM  (HP Operations Manager ) monitoring
> policies for RHEL7 cluster environment.
> 
> However, it looks like getting status of running defined RG seems way to
> challenging comparing to rgmanager/cman
> 
> SO, if under RHEL6:
> 
> running clustat
> 
> Service Name Owner
> (Last) State
>  ---  -
> -- -
>  service:rg_apache  
> rgcs02_ha   started
>  service:rgcl0111 
> rgcs01_ha   stopped
> 
> Where rgcs02_01/02 are cluster nodes and  rgcl0102 rg_apache are
> resource groups.
> 
> 
> However , under RHEL7 ( pcs ) couldn't fine anything similar.
> 
> I'm not interested in status of EACH refined resources, but only
> resource groups
> 
> something like:
> RG1 running on node2 status: running
> RG2 running on node1 status:stopped

You're right, at the moment there is no easy way to do this with
pacemaker. There is an outstanding feature request to add this ability
to crm_mon.

The closest thing now would be to write a script that checks the
individual resources' statuses with crm_mon (= pcs status) to come up
with a group status.



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker cluster not working after switching from 1.0 to 1.1 (resend as plain text)

[Ken Gaillot]

A preliminary question -- what cluster layer are you running?

Pacemaker 1.0 worked with heartbeat or corosync 1, while Ubuntu 14.04
ships with corosync 2 by default, IIRC. There were major incompatible
changes between corosync 1 and 2, so it's important to get that right
before looking at pacemaker.

A general note, when making such a big jump in the pacemaker version,
I'd recommend running "cibadmin --upgrade" both before exporting the
configuration from 1.0, and again after deploying it on 1.1. This will
apply any transformations needed in the CIB syntax. Pacemaker will do
this on the fly, but doing it manually lets you see any issues early, as
well as being more efficient.

On 01/16/2017 12:24 AM, Rick Kint wrote:
> Sorry about the garbled email. Trying again with plain text.
> 
> 
> 
> A working cluster running on Pacemaker 1.0.12 on RHEL5 has been copied with 
> minimal modifications to Pacemaker 1.1.10 on Ubuntu 14.04. The version string 
> is "1.1.10+git20130802-1ubuntu2.3".
> 
> We run simple active/standby two-node clusters. 
> 
> There are four resources on each node:
> - a stateful resource (Encryptor) representing a process in either active or 
> standby mode.
> -- this process does not maintain persistent data.
> - a clone resource (CredProxy) representing a helper process.
> - two clone resources (Ingress, Egress) representing network interfaces.
> 
> Colocation constraints require that all three clone resources must be in 
> Started role in order for the stateful Encryptor resource to be in Master 
> role.
> 
> The full configuration is at the end of this message.
> 
> The Encryptor resource should fail over on these events:
> - active node (i.e. node containing active Encryptor process) goes down
> - active Encryptor process goes down and cannot be restarted
> - auxiliary CredProxy process on active node goes down and cannot be restarted
> - either interface on active node goes down
> 
> All of these events trigger failover on the old platform (Pacemaker 1.0 on 
> RHEL5).
> 
> However, on the new platform (Pacemaker 1.1 on Ubuntu) neither interface 
> failure nor auxiliary process failure trigger failover. Pacemaker goes into a 
> loop where it starts and stops the active Encryptor resource and never 
> promotes the standby Encryptor resource. Cleaning up the failed resource 
> manually and issuing "crm_resource --cleanup" clears the jam and the standby 
> Encryptor resource is promoted. So does taking the former active node offline 
> completely.
> 
> The pe-input-X.bz2 files show this sequence:
> 
> (EncryptBase:1 is active, EncryptBase:0 is standby)
> 
> T: pacemaker recognizes that Ingress has failed
> transition: recover Ingress on active node
> 
> T+1: transition: recover Ingress on active node
> 
> T+2: transition: recover Ingress on active node
> 
> T+3: transitions: promote EncryptBase:0, demote EncryptBase:1, stop Ingress 
> on active node (no-op)
> 
> T+4: EncryptBase:1 demoted (both clones are now in slave mode), Ingress 
> stopped
> transitions: promote  EncryptBase:0, stop EncryptBase:1
> 
> T+5: EncryptBase:1 stopped, EncryptBase:0 still in slave role
> transitions: promote EncryptBase:0, start EncryptBase:1
> 
> T+6: EncryptBase:1 started (slave role)
> transitions: promote EncryptBase:0, stop EncryptBase:1
> 
> The last two steps repeat. Although pengine has decided that EncryptBase:0 
> should be promoted, Pacemaker keeps stopping and starting EncryptBase:1 (the 
> one on the node with the failed interface) without ever promoting 
> EncryptBase:0.
> 
> More precisely, crmd never issues the command that would cause promotion. For 
> a normal promotion, I see a sequence like this:
> 
> 2017-01-12T20:04:39.887154+00:00 encryptor4 pengine[2201]:   notice: 
> LogActions: Promote EncryptBase:0  (Slave -> Master encryptor4)
> 2017-01-12T20:04:39.888018+00:00 encryptor4 pengine[2201]:   notice: 
> process_pe_message: Calculated Transition 3: 
> /var/lib/pacemaker/pengine/pe-input-3.bz2
> 2017-01-12T20:04:39.888428+00:00 encryptor4 crmd[2202]:   notice: 
> te_rsc_command: Initiating action 9: promote EncryptBase_promote_0 on 
> encryptor4 (local)
> 2017-01-12T20:04:39.903827+00:00 encryptor4 Encryptor_ResourceAgent: INFO: 
> Promoting Encryptor.
> 2017-01-12T20:04:44.959804+00:00 encryptor4 crmd[2202]:   notice: 
> process_lrm_event: LRM operation EncryptBase_promote_0 (call=42, rc=0, 
> cib-update=43, confirmed=true) ok
> 
> in which crmd initiates an action for promotion and the RA logs a message 
> indicating that it was called with the arg "promote".
> 
> In contrast, the looping sections look like this:
> 
> (EncryptBase:1 on encryptor5 is the active/Master instance, EncryptBase:0 on 
> encryptor4 is the standby/Slave instance)
> 
> 2017-01-12T20:12:36.548980+00:00 encryptor4 pengine[2201]:   notice: 
> LogActions: Promote EncryptBase:0(Slave -> Master encryptor4)
> 2017-01-12T20:12:36.549005+00:00 encryptor4 pengine[2201]:   notice: 
> LogActions: StopEncryptBase

Re: [ClusterLabs] Problems with corosync and pacemaker with error scenarios

[Ken Gaillot]

On 01/16/2017 08:56 AM, Gerhard Wiesinger wrote:
> Hello,
> 
> I'm new to corosync and pacemaker and I want to setup a nginx cluster
> with quorum.
> 
> Requirements:
> - 3 Linux maschines
> - On 2 maschines floating IP should be handled and nginx as a load
> balancing proxy
> - 3rd maschine is for quorum only, no services must run there
> 
> Installed on all 3 nodes corosync/pacemaker, firewall ports openend are:
> 5404, 5405, 5406 for udp in both directions

If you're using firewalld, the easiest configuration is:

  firewall-cmd --permanent --add-service=high-availability

If not, depending on what you're running, you may also want to open  TCP
ports 2224 (pcsd), 3121 (Pacemaker Remote), and 21064 (DLM).

> OS: Fedora 25
> 
> Configuration of corosync (only the bindnetaddr is different on every
> maschine) and pacemaker below.

FYI you don't need a different bindnetaddr. You can (and generally
should) use the *network* address, which is the same on all hosts.

> Configuration works so far but error test scenarios don't work like
> expected:
> 1.) I had cases in testing without qourum and quorum again where the
> cluster kept in Stopped state
>   I had to restart the whole stack to get it online again (killall -9
> corosync;systemctl restart corosync;systemctl restart pacemaker)
>   Any ideas?

It will be next to impossible to say without logs. It's definitely not
expected behavior. Stopping is the correct response to losing quorum;
perhaps quorum is not being properly restored for some reason. What is
your test methodology?

> 2.) Restarting pacemaker on inactive node also restarts resources on the
> other active node:
> a.) Everything up & ok
> b.) lb01 handles all resources
> c.) on lb02 which handles no resrouces: systemctl restart pacemaker:
>   All resources will also be restart with a short outage on lb01 (state
> is Stopped, Started[ lb01 lb02 ] and then Started lb02)
>   How can this be avoided?

This is not expected behavior, except with clones, which I don't see you
using.

> 3.) Stopping and starting corosync doesn't awake the node up again:
>   systemctl stop corosync;sleep 10;systemctl restart corosync
>   Online: [ kvm01 lb01 ]
>   OFFLINE: [ lb02 ]
>   Stays in that state until pacemaker is restarted: systemctl restart
> pacemaker
>   Bug?

No, pacemaker should always restart if corosync restarts. That is
specified in the systemd units, so I'm not sure why pacemaker didn't
automatically restart in your case.

> 4.) "systemctl restart corosync" hangs sometimes (waiting 2 min)
>   needs a
>   killall -9 corosync;systemctl restart corosync;systemctl restart
> pacemaker
>   sequence to get it up gain
> 
> 5.) Simulation of split brain: Disabling/reenabling local firewall
> (ports 5404, 5405, 5406) on node lb01 and lb02 for the following ports

FYI for an accurate simulation, be sure to block both incoming and
outgoing traffic on the corosync ports.

> doesn't bring corosync up again after reenabling lb02 firewall
> partition WITHOUT quorum
> Online: [ kvm01 ]
> OFFLINE: [ lb01 lb02 ]
>   NOK: restart on lb02: systemctl restart corosync;systemctl restart
> pacemaker
>   OK:  restart on lb02 and kvm01 (quorum host): systemctl restart
> corosync;systemctl restart pacemaker
>   I also see that non enabled hosts (quorum hosts) are also tried to be
> started on kvm01
>   Started[ kvm01 lb02 ]
>   Started lb02
>   Any ideas?
> 
> I've also written a new ocf:heartbeat:Iprule to modify "ip rule"
> accordingly.
> 
> Versions are:
> corosync: 2.4.2
> pacemaker: 1.1.16
> Kernel: 4.9.3-200.fc25.x86_64
> 
> Thnx.
> 
> Ciao,
> Gerhard
> 
> Corosync config:
> 
> 
> totem {
> version: 2
> cluster_name: lbcluster
> crypto_cipher: aes256
> crypto_hash: sha512
> interface {
> ringnumber: 0
> bindnetaddr: 1.2.3.35
> mcastport: 5405
> }
> transport: udpu
> }
> logging {
> fileline: off
> to_logfile: yes
> to_syslog: yes
> logfile: /var/log/cluster/corosync.log
> debug: off
> timestamp: on
> logger_subsys {
> subsys: QUORUM
> debug: off
> }
> }
> nodelist {
> node {
> ring0_addr: lb01
> nodeid: 1
> }
> node {
> ring0_addr: lb02
> nodeid: 2
> }
> node {
> ring0_addr: kvm01
> nodeid: 3
> }
> }
> quorum {
> # Enable and configure quorum subsystem (default: off)
> # see also corosync.conf.5 and votequorum.5
> #provider: corosync_votequorum
> provider: corosync_votequorum
> # Only for 2 node setup!
> #  two_node: 1
> }
> ==

Re: [ClusterLabs] Antw: Re: VirtualDomain started in two hosts

[Ken Gaillot]

On 01/17/2017 08:52 AM, Ulrich Windl wrote:
 Oscar Segarra  schrieb am 17.01.2017 um 10:15 in
> Nachricht
> :
>> Hi,
>>
>> Yes, I will try to explain myself better.
>>
>> *Initially*
>> On node1 (vdicnode01-priv)
>>> virsh list
>> ==
>> vdicdb01 started
>>
>> On node2 (vdicnode02-priv)
>>> virsh list
>> ==
>> vdicdb02 started
>>
>> --> Now, I execute the migrate command (outside the cluster <-- not using
>> pcs resource move)
>> virsh migrate --live vdicdb01 qemu:/// qemu+ssh://vdicnode02-priv
>> tcp://vdicnode02-priv
> 
> One of the rules of successful clustering is: If resurces are managed by the 
> cluster, they are managed by the cluster only! ;-)
> 
> I guess one node is trying to restart the VM once it vanished, and the other 
> node might try to shut down the VM while it's being migrated.
> Or any other undesired combination...


As Ulrich says here, you can't use virsh to manage VMs once they are
managed by the cluster. Instead, configure your cluster to support live
migration:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-migrating-resources

and then use pcs resource move (which is just location constraints under
the hood) to move VMs.

What's happening in your example is:

* Your VM cluster resource has a monitor operation ensuring that is
running properly on the desired node.

* It is also possible to configure a monitor to ensure that the resource
is not running on nodes where it's not supposed to be (a monitor with
role="Stopped"). You don't have one of these (which is fine, and common).

* When you move the VM, the cluster detects that it is not running on
the node you told it to keep it running on. Because there is no
"Stopped" monitor, the cluster doesn't immediately realize that a new
rogue instance is running on another node. So, the cluster thinks the VM
crashed on the original node, and recovers it by starting it again.

If your goal is to take a VM out of cluster management without stopping
it, you can "unmanage" the resource.


>> *Finally*
>> On node1 (vdicnode01-priv)
>>> virsh list
>> ==
>> *vdicdb01 started*
>>
>> On node2 (vdicnode02-priv)
>>> virsh list
>> ==
>> vdicdb02 started
>> vdicdb01 started
>>
>> If I query cluster pcs status, cluster thinks resource vm-vdicdb01 is only
>> started on node vdicnode01-priv.
>>
>> Thanks a lot.
>>
>>
>>
>> 2017-01-17 10:03 GMT+01:00 emmanuel segura :
>>
>>> sorry,
>>>
>>> But do you mean, when you say, you migrated the vm outside of the
>>> cluster? one server out side of you cluster?
>>>
>>> 2017-01-17 9:27 GMT+01:00 Oscar Segarra :
 Hi,

 I have configured a two node cluster whewe run 4 kvm guests on.

 The hosts are:
 vdicnode01
 vdicnode02

 And I have created a dedicated network card for cluster management. I
>>> have
 created required entries in /etc/hosts:
 vdicnode01-priv
 vdicnode02-priv

 The four guests have collocation rules in order to make them distribute
 proportionally between my two nodes.

 The problem I have is that if I migrate a guest outside the cluster, I
>>> mean
 using the virsh migrate - - live...  Cluster,  instead of moving back the
 guest to its original node (following collocation sets),  Cluster starts
 again the guest and suddenly I have the same guest running on both nodes
 causing xfs corruption in guest.

 Is there any configuration applicable to avoid this unwanted behavior?

 Thanks a lot

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: VirtualDomain started in two hosts

[Ken Gaillot]

On 01/17/2017 10:05 AM, Oscar Segarra wrote:
> Hi, 
> 
> * It is also possible to configure a monitor to ensure that the resource
> is not running on nodes where it's not supposed to be (a monitor with
> role="Stopped"). You don't have one of these (which is fine, and common).
> 
> Can you provide more information/documentation about role="Stopped"

Since you're using pcs, you can either configure monitors when you
create the resource with pcs resource create, or you can add/remove
monitors later with pcs resource op add/remove.

For example:

pcs resource op add my-resource-name op monitor interval=10s role="Stopped"

With a normal monitor op (role="Started" or omitted), the cluster will
run the resource agent's monitor command on any node that's supposed to
be running the resource. With the above example, it will additionally
run a monitor on all other nodes, so that if it finds the resource
running somewhere it's not supposed to be, it can stop it.

Note that each monitor op must have a unique timeout. So if your
existing monitor runs every 10s, you need to pick a different value for
the new monitor.

> And, please, can you explain how VirtualDomain resource agents manages
> the scenario I've presented?
> 
> /What happens If I stop pacemaker and corosync services in all nodes and
> I start them again... ¿will I have all guests running twice?/
> 
> Thanks a lot

If you stop cluster services, by default the cluster will first stop all
resources. You can set maintenance mode, or unmanage one or more
resources, to prevent the stops.

When cluster services first start on a node, the cluster "probes" the
status of all resources on that node, by running a one-time monitor. So
it will detect anything running at that time, and start or stop services
as needed to meet the configured requirements.

> 2017-01-17 16:38 GMT+01:00 Ken Gaillot  <mailto:kgail...@redhat.com>>:
> 
> On 01/17/2017 08:52 AM, Ulrich Windl wrote:
> >>>> Oscar Segarra  <mailto:oscar.sega...@gmail.com>> schrieb am
> 17.01.2017 um 10:15 in
> > Nachricht
> >  
> <mailto:cajq8tag8vhx5j1xqpqmrq-9omfnxkhqs54mbzz491_6df9a...@mail.gmail.com>>:
> >> Hi,
> >>
> >> Yes, I will try to explain myself better.
> >>
> >> *Initially*
> >> On node1 (vdicnode01-priv)
> >>> virsh list
> >> ==
> >> vdicdb01 started
> >>
> >> On node2 (vdicnode02-priv)
> >>> virsh list
> >> ==
> >> vdicdb02 started
> >>
> >> --> Now, I execute the migrate command (outside the cluster <-- not 
> using
> >> pcs resource move)
> >> virsh migrate --live vdicdb01 qemu:/// qemu+ssh://vdicnode02-priv
> >> tcp://vdicnode02-priv
> >
> > One of the rules of successful clustering is: If resurces are managed 
> by the cluster, they are managed by the cluster only! ;-)
> >
> > I guess one node is trying to restart the VM once it vanished, and the 
> other node might try to shut down the VM while it's being migrated.
> > Or any other undesired combination...
> 
> 
> As Ulrich says here, you can't use virsh to manage VMs once they are
> managed by the cluster. Instead, configure your cluster to support live
> migration:
> 
> 
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-migrating-resources
> 
> <http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-migrating-resources>
> 
> and then use pcs resource move (which is just location constraints under
> the hood) to move VMs.
> 
> What's happening in your example is:
> 
> * Your VM cluster resource has a monitor operation ensuring that is
> running properly on the desired node.
> 
> * It is also possible to configure a monitor to ensure that the resource
> is not running on nodes where it's not supposed to be (a monitor with
> role="Stopped"). You don't have one of these (which is fine, and
> common).
> 
> * When you move the VM, the cluster detects that it is not running on
> the node you told it to keep it running on. Because there is no
> "Stopped" monitor, the cluster doesn't immediately realize that a new
> rogue instance is running on another node. So, the cluster thinks the VM
> crashed on the original node, and recovers it by starting it again.
> 
> If your

Re: [ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources

[Ken Gaillot]

On 01/17/2017 10:19 AM, Scott Greenlese wrote:
> Hi..
> 
> I've been testing live guest migration (LGM) with VirtualDomain
> resources, which are guests running on Linux KVM / System Z
> managed by pacemaker.
> 
> I'm looking for documentation that explains how to configure my
> VirtualDomain resources such that they will not timeout
> prematurely when there is a heavy I/O workload running on the guest.
> 
> If I perform the LGM with an unmanaged guest (resource disabled), it
> takes anywhere from 2 - 5 minutes to complete the LGM.
> Example:
> 
> # Migrate guest, specify a timeout value of 600s
> 
> [root@zs95kj VD]# date;virsh --keepalive-interval 10 migrate --live
> --persistent --undefinesource*--timeout 600* --verbose zs95kjg110061
> qemu+ssh://zs90kppcs1/system
> Mon Jan 16 16:35:32 EST 2017
> 
> Migration: [100 %]
> 
> [root@zs95kj VD]# date
> Mon Jan 16 16:40:01 EST 2017
> [root@zs95kj VD]#
> 
> Start: 16:35:32
> End: 16:40:01
> Total: *4 min 29 sec*
> 
> 
> In comparison, when the guest is managed by pacemaker, and enabled for
> LGM ... I get this:
> 
> [root@zs95kj VD]# date;pcs resource show zs95kjg110061_res
> Mon Jan 16 15:13:33 EST 2017
> Resource: zs95kjg110061_res (class=ocf provider=heartbeat
> type=VirtualDomain)
> Attributes: config=/guestxml/nfs1/zs95kjg110061.xml
> hypervisor=qemu:///system migration_transport=ssh
> Meta Attrs: allow-migrate=true remote-node=zs95kjg110061
> remote-addr=10.20.110.61
> Operations: start interval=0s timeout=480
> (zs95kjg110061_res-start-interval-0s)
> stop interval=0s timeout=120 (zs95kjg110061_res-stop-interval-0s)
> monitor interval=30s (zs95kjg110061_res-monitor-interval-30s)
> migrate-from interval=0s timeout=1200
> (zs95kjg110061_res-migrate-from-interval-0s)
> *migrate-to* interval=0s *timeout=1200*
> (zs95kjg110061_res-migrate-to-interval-0s)
> 
> NOTE: I didn't specify any migrate-to value for timeout, so it defaulted
> to 1200. Is this seconds? If so, that's 20 minutes,
> ample time to complete a 5 minute migration.

Not sure where the default of 1200 comes from, but I believe the default
is milliseconds if no unit is specified. Normally you'd specify
something like "timeout=1200s".

> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:27:01 EST 2017
> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): Started zs90kppcs1
> [root@zs95kj VD]#
> 
> 
> [root@zs95kj VD]# date;*pcs resource move zs95kjg110061_res zs95kjpcs1*
> Mon Jan 16 14:45:39 EST 2017
> You have new mail in /var/spool/mail/root
> 
> 
> Jan 16 14:45:37 zs90kp VirtualDomain(zs95kjg110061_res)[21050]: INFO:
> zs95kjg110061: *Starting live migration to zs95kjpcs1 (using: virsh
> --connect=qemu:///system --quiet migrate --live zs95kjg110061
> qemu+ssh://zs95kjpcs1/system ).*
> Jan 16 14:45:57 zs90kp lrmd[12798]: warning:
> zs95kjg110061_res_migrate_to_0 process (PID 21050) timed out
> Jan 16 14:45:57 zs90kp lrmd[12798]: warning:
> zs95kjg110061_res_migrate_to_0:21050 - timed out after 2ms
> Jan 16 14:45:57 zs90kp crmd[12801]: error: Operation
> zs95kjg110061_res_migrate_to_0: Timed Out (node=zs90kppcs1, call=1978,
> timeout=2ms)
> Jan 16 14:45:58 zs90kp journal: operation failed: migration job:
> unexpectedly failed
> [root@zs90KP VD]#
> 
> So, the migration timed out after 2ms. Assuming ms is milliseconds,
> that's only 20 seconds. So, it seems that LGM timeout has
> nothing to do with *migrate-to* on the resource definition.

Yes, ms is milliseconds. Pacemaker internally represents all times in
milliseconds, even though in most actual usage, it has 1-second granularity.

If your specified timeout is 1200ms, I'm not sure why it's using
2ms. There may be a minimum enforced somewhere.

> Also, what is the expected behavior when the migration times out? I
> watched the VirtualDomain resource state during the migration process...
> 
> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:45:57 EST 2017
> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): Started zs90kppcs1
> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:46:02 EST 2017
> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): FAILED zs90kppcs1
> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:46:06 EST 2017
> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): FAILED zs90kppcs1
> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:46:08 EST 2017
> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): FAILED zs90kppcs1
> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:46:10 EST 2017
> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): FAILED zs90kppcs1
> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:46:12 EST 2017
> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): Stopped
> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
> Mon Jan 16 14:46:14 EST 2017
> zs95kjg110061_res (ocf::heartbeat:

Re: [ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources

[Ken Gaillot]

edd0 0xa5f1edd0)
> Jan 17 13:55:14 [27552] zs95kj lrmd: ( lrmd.c:113 ) info: log_finished:
> finished - rsc:zs95kjg110061_res action:migrate_to call_id:941
> pid:135045 exit-code:1 exec-time:20003ms queue-time:0ms
> Jan 17 13:55:14 [27552] zs95kj lrmd: ( lrmd.c:1292 ) trace:
> lrmd_rsc_execute: Nothing further to do for zs95kjg110061_res
> Jan 17 13:55:14 [27555] zs95kj crmd: ( utils.c:1942 ) debug:
> create_operation_update: do_update_resource: Updating resource
> zs95kjg110061_res after migrate_to op Timed Out (interval=0)
> Jan 17 13:55:14 [27555] zs95kj crmd: ( lrm.c:2397 ) error:
> process_lrm_event: Operation zs95kjg110061_res_migrate_to_0: Timed Out
> (node=zs95kjpcs1, call=941, timeout=2ms)
> Jan 17 13:55:14 [27555] zs95kj crmd: ( lrm.c:196 ) debug:
> update_history_cache: Updating history for 'zs95kjg110061_res' with
> migrate_to op
> 
> 
> Any ideas?
> 
> 
> 
> Scott Greenlese ... IBM KVM on System z - Solution Test, Poughkeepsie, N.Y.
> INTERNET: swgre...@us.ibm.com
> 
> 
> Inactive hide details for Ken Gaillot ---01/17/2017 11:41:53 AM---On
> 01/17/2017 10:19 AM, Scott Greenlese wrote: > Hi..Ken Gaillot
> ---01/17/2017 11:41:53 AM---On 01/17/2017 10:19 AM, Scott Greenlese
> wrote: > Hi..
> 
> From: Ken Gaillot 
> To: users@clusterlabs.org
> Date: 01/17/2017 11:41 AM
> Subject: Re: [ClusterLabs] Live Guest Migration timeouts for
> VirtualDomain resources
> 
> 
> 
> 
> 
> On 01/17/2017 10:19 AM, Scott Greenlese wrote:
>> Hi..
>>
>> I've been testing live guest migration (LGM) with VirtualDomain
>> resources, which are guests running on Linux KVM / System Z
>> managed by pacemaker.
>>
>> I'm looking for documentation that explains how to configure my
>> VirtualDomain resources such that they will not timeout
>> prematurely when there is a heavy I/O workload running on the guest.
>>
>> If I perform the LGM with an unmanaged guest (resource disabled), it
>> takes anywhere from 2 - 5 minutes to complete the LGM.
>> Example:
>>
>> # Migrate guest, specify a timeout value of 600s
>>
>> [root@zs95kj VD]# date;virsh --keepalive-interval 10 migrate --live
>> --persistent --undefinesource*--timeout 600* --verbose zs95kjg110061
>> qemu+ssh://zs90kppcs1/system
>> Mon Jan 16 16:35:32 EST 2017
>>
>> Migration: [100 %]
>>
>> [root@zs95kj VD]# date
>> Mon Jan 16 16:40:01 EST 2017
>> [root@zs95kj VD]#
>>
>> Start: 16:35:32
>> End: 16:40:01
>> Total: *4 min 29 sec*
>>
>>
>> In comparison, when the guest is managed by pacemaker, and enabled for
>> LGM ... I get this:
>>
>> [root@zs95kj VD]# date;pcs resource show zs95kjg110061_res
>> Mon Jan 16 15:13:33 EST 2017
>> Resource: zs95kjg110061_res (class=ocf provider=heartbeat
>> type=VirtualDomain)
>> Attributes: config=/guestxml/nfs1/zs95kjg110061.xml
>> hypervisor=qemu:///system migration_transport=ssh
>> Meta Attrs: allow-migrate=true remote-node=zs95kjg110061
>> remote-addr=10.20.110.61
>> Operations: start interval=0s timeout=480
>> (zs95kjg110061_res-start-interval-0s)
>> stop interval=0s timeout=120 (zs95kjg110061_res-stop-interval-0s)
>> monitor interval=30s (zs95kjg110061_res-monitor-interval-30s)
>> migrate-from interval=0s timeout=1200
>> (zs95kjg110061_res-migrate-from-interval-0s)
>> *migrate-to* interval=0s *timeout=1200*
>> (zs95kjg110061_res-migrate-to-interval-0s)
>>
>> NOTE: I didn't specify any migrate-to value for timeout, so it defaulted
>> to 1200. Is this seconds? If so, that's 20 minutes,
>> ample time to complete a 5 minute migration.
> 
> Not sure where the default of 1200 comes from, but I believe the default
> is milliseconds if no unit is specified. Normally you'd specify
> something like "timeout=1200s".
> 
>> [root@zs95kj VD]# date;pcs resource show |grep zs95kjg110061_res
>> Mon Jan 16 14:27:01 EST 2017
>> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): Started zs90kppcs1
>> [root@zs95kj VD]#
>>
>>
>> [root@zs95kj VD]# date;*pcs resource move zs95kjg110061_res zs95kjpcs1*
>> Mon Jan 16 14:45:39 EST 2017
>> You have new mail in /var/spool/mail/root
>>
>>
>> Jan 16 14:45:37 zs90kp VirtualDomain(zs95kjg110061_res)[21050]: INFO:
>> zs95kjg110061: *Starting live migration to zs95kjpcs1 (using: virsh
>> --connect=qemu:///system --quiet migrate --live zs95kjg110061
>> qemu+ssh://zs95kjpcs1/system ).*
>> Jan 16 14:45:57 zs90kp lrmd[12798]: warning:
>> zs95kjg110061_r

Re: [ClusterLabs] Antw: Re: VirtualDomain started in two hosts

[Ken Gaillot]

On 01/18/2017 03:49 AM, Ferenc Wágner wrote:
> Ken Gaillot  writes:
> 
>> * When you move the VM, the cluster detects that it is not running on
>> the node you told it to keep it running on. Because there is no
>> "Stopped" monitor, the cluster doesn't immediately realize that a new
>> rogue instance is running on another node. So, the cluster thinks the VM
>> crashed on the original node, and recovers it by starting it again.
> 
> Ken, do you mean that if a periodic "stopped" monitor is configured, it
> is forced to run immediately (out of schedule) when the regular periodic
> monitor unexpectedly returns with stopped status?  That is, before the
> cluster takes the recovery action?  Conceptually, that would be similar
> to the probe run on node startup.  If not, then maybe it would be a
> useful resource option to have (I mean running cluster-wide probes on an
> unexpected monitor failure, before recovery).  An optional safety check.

No, there is nothing like that currently. The regular and "Stopped"
monitors run independently. Because they must have different intervals,
that does mean that the two sides of the issue may be detected at
different times.

It is an interesting idea to have an option to reprobe on operation
failure. I think it may be overkill; the only failure situation it would
be good for is one like this, where a resource was moved out of cluster
control. The vast majority of failure scenarios wouldn't be helped. If
that sort of thing happens a lot in your cluster, you really need to
figure out how to stop doing that. :)

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Live Guest Migration timeouts for VirtualDomain resources

[Ken Gaillot]

On 01/19/2017 01:36 AM, Ulrich Windl wrote:
>>>> Ken Gaillot  schrieb am 18.01.2017 um 16:32 in 
>>>> Nachricht
> <4b02d3fa-4693-473b-8bed-dc98f9e3f...@redhat.com>:
>> On 01/17/2017 04:45 PM, Scott Greenlese wrote:
>>> Ken and Co,
>>>
>>> Thanks for the useful information.
>>>
> 
> [...]
>>>
>>> Is this internally coded within the class=ocf provider=heartbeat
>>> type=VirtualDomain resource agent?
>>
>> Aha, I just realized what the issue is: the operation name is
>> migrate_to, not migrate-to.
>>
>> For technical reasons, pacemaker can't validate operation names (at the
>> time that the configuration is edited, it does not necessarily have
>> access to the agent metadata).
> 
> BUT the set of operations is finite, right? So if those were in some XML 
> schema, the names could be verified at least (not meaning that the operation 
> is actually supported).
> BTW: Would a "crm configure verify" detect this kijnd of problem?
> 
> [...]
> 
> Ulrich

Yes, it's in the resource agent meta-data. While pacemaker itself uses a
small set of well-defined actions, the agent may define any arbitrarily
named actions it desires, and the user could configure one of these as a
recurring action in pacemaker.

Pacemaker itself has to be liberal about where its configuration comes
from -- the configuration can be edited on a separate machine, which
doesn't have resource agents, and then uploaded to the cluster. So
Pacemaker can't do that validation at configuration time. (It could
theoretically do some checking after the fact when the configuration is
loaded, but this could be a lot of overhead, and there are
implementation issues at the moment.)

Higher-level tools like crmsh and pcs, on the other hand, can make
simplifying assumptions. They can require access to the resource agents
so that they can do extra validation.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how do I disable/negate resource option?

[Ken Gaillot]

On 01/19/2017 06:30 AM, lejeczek wrote:
> hi all
> 
> how can it be done? Is it possible?
> many thanks,
> L.

Check the man page / documentation for whatever tool you're using (crm,
pcs, etc.). Each one has its own syntax.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how do I disable/negate resource option?

[Ken Gaillot]

On 01/19/2017 09:52 AM, lejeczek wrote:
> 
> 
> On 19/01/17 15:30, Ken Gaillot wrote:
>> On 01/19/2017 06:30 AM, lejeczek wrote:
>>> hi all
>>>
>>> how can it be done? Is it possible?
>>> many thanks,
>>> L.
>> Check the man page / documentation for whatever tool you're using (crm,
>> pcs, etc.). Each one has its own syntax.
> 
> I'd think this would be some built-in logic for a given tool, eg pcs
> which I'm using (as am on 7.x) but no. I don't think an RA option(s) can
> be ignored, or even better disabled.
> I'm looking at specific RA(s) descriptions and nothing like "disable"
> can I find there. Which is a shame, I really thought by design it'd be
> possible. Take CTDB, it's a mess, probably because of the version, in
> rhel.7.x must be newer, with which the CTDB has not kept up, it fails.
> I only started checking HA two days ago, and in my case it's a bit
> discouraging experience with that CTDB. Seems like the only road is to
> inside RA definition files. Is it really or I'm missing something
> fundamental, trivial?

Maybe I'm misunderstanding what you mean by "disable".

If you just want to change the value used for the option, you can do:

  pcs resource update RESOURCE_NAME OPTION_NAME=NEW_VALUE

If you mean that the resource agent is buggy, and you want it not to use
a particular option at all, then that can only be addressed within the
agent. If you're comfortable with shell scripting, you can copy it and
make any necessary changes yourself. Also feel free to open a bug at:

  https://github.com/ClusterLabs/resource-agents/issues

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [Question] About log collection of crm_report.

[Ken Gaillot]

On 01/23/2017 04:17 PM, renayama19661...@ybb.ne.jp wrote:
> Hi All,
> 
> When I carry out Pacemaker1.1.15 and Pacemaker1.1.16 in RHEL7.3, log in 
> conjunction with pacemaker is not collected in the file which I collected in 
> sosreport.
>  
> 
> This seems to be caused by the next correction and pacemaker.py script of 
> RHEL7.3.
> 
>  - 
> https://github.com/ClusterLabs/pacemaker/commit/1bcad6a1eced1a3b6c314b05ac1d353adda260f6
>  - 
> https://github.com/ClusterLabs/pacemaker/commit/582e886dd8475f701746999c0093cd9735aca1ed#diff-284d516fab648676f5d93bc5ce8b0fbf
> 
> 
> ---
> (/usr/lib/python2.7/site-packages/sos/plugins/pacemaker.py)
> (snip)
> if not self.get_option("crm_scrub"):
> crm_scrub = ""
> self._log_warn("scrubbing of crm passwords has been disabled:")
> self._log_warn("data collected by crm_report may contain"
>" sensitive values.")
> self.add_cmd_output('crm_report --sos-mode %s -S -d '
> ' --dest %s --from "%s"' %
> (crm_scrub, crm_dest, crm_from),
> chroot=self.tmp_in_sysroot())
> (snip)
> ---
> 
> 
> When a user carries out crm_report in sosreport, what is the reason that set 
> search_logs to 0?
> 
> We think that the one where search_logs works with 1 in sosreport is right.
> 
> 
> Best Regards,
> Hideo Yamauchi.

Hi Hideo,

The --sos-mode option is intended for RHEL integration, so it is only
guaranteed to work with the combination of pacemaker and sosreport
packages delivered with a particular version of RHEL (and its derivatives).

That allows us to make assumptions about what sosreport features are
available. It might be better to detect those features, but we haven't
seen enough usage of sosreport + pacemaker outside RHEL to make that
worth the effort.

In this case, the version of sosreport that will be in RHEL 7.4 will
collect pacemaker.log and corosync.log on its own, so the crm_report in
pacemaker 1.1.16 doesn't need to collect the logs itself.

It might work if you build the latest sosreport:
https://github.com/sosreport/sos

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [Question] About log collection of crm_report.

[Ken Gaillot]

On 01/24/2017 04:41 PM, renayama19661...@ybb.ne.jp wrote:
> Hi Ken,
> 
> Thank you for comment.
> 
> For example, our user does not use pacemaker.log and corosync.log.
> 
> Via a syslog, the user makes setting to output all log to /var/log/ha-log.
> 
> -
> (/etc/corosycn/corosync.conf)
> logging {
> syslog_facility: local1
> debug: off
> }
> 
> (/etc/sysconfig/pacemaker)
> PCMK_logfile=none
> PCMK_logfacility=local1
> PCMK_logpriority=info
> PCMK_fail_fast=yes
> 
> (/etc/rsyslog.conf)
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none;local1.none
> /var/log/messages
> (snip)
> # Save boot messages also to boot.log
> local7.*/var/log/boot.log
> local1.info /var/log/ha-log
> -
> 
> In present crm_report, in the case of the user who output log in a different 
> file, the log is not collected in sosreport.
> 
> Is this not a problem?
> Possibly is all /var/log going to collect it in future in sosreport?
> 
> Of course I know that "/var/log/ha-log" is collected definitely when I carry 
> out crm_report alone.
> I want to know why collection of log of this crm_report was stopped in 
> sosreport.
> 
> For REDHAT, will it be to be enough for collection of sosreport contents?
> If it is such a thing, we can understand.
> 
> - And I test crm_report at the present, but seem to have some problems.
> - I intend to report the problem by Bugzilla again.
> 
> Best Regards,
> Hideo Yamauchi.

Hi Hideo,

You are right, that is a problem. I've opened a Red Hat bug for sosreport:

https://bugzilla.redhat.com/show_bug.cgi?id=1416535

> - Original Message -
>> From: Ken Gaillot 
>> To: users@clusterlabs.org
>> Cc: 
>> Date: 2017/1/24, Tue 08:15
>> Subject: Re: [ClusterLabs] [Question] About log collection of crm_report.
>>
>> On 01/23/2017 04:17 PM, renayama19661...@ybb.ne.jp wrote:
>>>  Hi All,
>>>
>>>  When I carry out Pacemaker1.1.15 and Pacemaker1.1.16 in RHEL7.3, log in 
>> conjunction with pacemaker is not collected in the file which I collected in 
>> sosreport.
>>>   
>>>
>>>  This seems to be caused by the next correction and pacemaker.py script of 
>> RHEL7.3.
>>>
>>>   - 
>> https://github.com/ClusterLabs/pacemaker/commit/1bcad6a1eced1a3b6c314b05ac1d353adda260f6
>>>   - 
>> https://github.com/ClusterLabs/pacemaker/commit/582e886dd8475f701746999c0093cd9735aca1ed#diff-284d516fab648676f5d93bc5ce8b0fbf
>>>
>>>
>>>  ---
>>>  (/usr/lib/python2.7/site-packages/sos/plugins/pacemaker.py)
>>>  (snip)
>>>  if not self.get_option("crm_scrub"):
>>>  crm_scrub = ""
>>>  self._log_warn("scrubbing of crm passwords has been 
>> disabled:")
>>>  self._log_warn("data collected by crm_report may 
>> contain"
>>> " sensitive values.")
>>>  self.add_cmd_output('crm_report --sos-mode %s -S -d '
>>>  ' --dest %s --from "%s"' %
>>>  (crm_scrub, crm_dest, crm_from),
>>>  chroot=self.tmp_in_sysroot())
>>>  (snip)
>>>  ---
>>>
>>>
>>>  When a user carries out crm_report in sosreport, what is the reason that 
>> set search_logs to 0?
>>>
>>>  We think that the one where search_logs works with 1 in sosreport is right.
>>>
>>>
>>>  Best Regards,
>>>  Hideo Yamauchi.
>>
>> Hi Hideo,
>>
>> The --sos-mode option is intended for RHEL integration, so it is only
>> guaranteed to work with the combination of pacemaker and sosreport
>> packages delivered with a particular version of RHEL (and its derivatives).
>>
>> That allows us to make assumptions about what sosreport features are
>> available. It might be better to detect those features, but we haven't
>> seen enough usage of sosreport + pacemaker outside RHEL to make that
>> worth the effort.
>>
>> In this case, the version of sosreport that will be in RHEL 7.4 will
>> collect pacemaker.log and corosync.log on its own, so the crm_report in
>> pacemaker 1.1.16 doesn't need to collect the logs itself.
>>
>> It might work if you build the latest sosreport:
>> https://github.com/sosreport/sos
>>

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker kill does not cause node fault ???

[Ken Gaillot]

On 01/10/2017 04:24 AM, Stefan Schloesser wrote:
> Hi,
> 
> I am currently testing a 2 node cluster under Ubuntu 16.04. The setup seems 
> to be working ok including the STONITH.
> For test purposes I issued a "pkill -f pace" killing all pacemaker processes 
> on one node.
> 
> Result:
> The node is marked as "pending", all resources stay on it. If I manually kill 
> a resource it is not noticed. On the other node a drbd "promote" command 
> fails (drbd is still running as master on the first node).
> 
> Killing the corosync process works as expected -> STONITH.
> 
> Could someone shed some light on this behavior? 
> 
> Thanks,
> 
> Stefan

I suspect that, when you kill pacemakerd, systemd respawns it quickly
enough that fencing is unnecessary. Try "pkill -f pace; systemd stop
pacemaker".

Did you schedule monitor operations on your resources? If not, pacemaker
will not know if they go down.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Resource Priority

[Ken Gaillot]

On 02/01/2017 09:07 AM, Ulrich Windl wrote:
 Chad Cravens  schrieb am 01.02.2017 um 15:52 in
> Nachricht
> :
>> Hello Cluster Fans!
>>
>> I've had a great time working with the clustering software. Implementing a
>> HUGE cluster solution now (100+ resources) and it's working great!
>>
>> I had a question regarding prioritizing resources. Say I have three nodes
>> (A,B,C) and 2 database resources (DB1, DB2). Database resources normally
>> run on A and B, and both failover to C.
>>
>> What I would like to do is prioritize resource DB1 over DB2 if both have to
>> failover to node C. For example, if DB2 has failed over and is running on
>> node C, and at a later time DB1 fails over to node C, that DB2 would stop
>> (no longer running at all on any node) and DB1 would run. Essentially DB1
>> is kicking DB2 off the cluster. I was wondering if there would be a clean
>> way to implement something like this using standard cluster configurations
>> parameters or if I'd have to create a custom RA script that would run
>> cluster commands to do this?
> 
> Hi!
> 
> What about this?: First use utilization constraints to avoid overloading your 
> nodes (change nodes and each resource). The use priorities for the resources. 
> Now when more resources want to run on a node (C) that the node can deal 
> with, the higher priority resources will succeed. Drawback: Stopping a 
> resource can cause some resource shuffling between nodes. We run a similar 
> configuration with SLES11 for a few years now. If resources are becoming 
> tight, test and development resources have to give way for production 
> resources...
> 
> Regards,
> Ulrich

Utilization attributes are a good idea, if overloading the machine is
your concern.

If the DBs simply can't run together, add a colocation constraint with a
negative score.

In either case, you can use the "priority" meta-attribute to say which
one is preferred when both can't run:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_resource_meta_attributes

>>
>> Thanks in advance! Happy clustering :)
>>
>> -- 
>> Kindest Regards,
>> Chad Cravens
>> (843) 291-8340
>>
>> [image: http://www.ossys.com] 
>> [image: http://www.linkedin.com/company/open-source-systems-llc]
>>    [image:
>> https://www.facebook.com/OpenSrcSys] 
>>[image: https://twitter.com/OpenSrcSys] 
>>  [image: http://www.youtube.com/OpenSrcSys]
>>    [image: http://www.ossys.com/feed]
>>    [image: cont...@ossys.com] 
>> Chad Cravens
>> (843) 291-8340
>> chad.crav...@ossys.com 
>> http://www.ossys.com 

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Live Guest Migration timeouts for VirtualDomain resources

[Ken Gaillot]

On 02/01/2017 09:15 AM, Scott Greenlese wrote:
> Hi all...
> 
> Just a quick follow-up.
> 
> Thought I should come clean and share with you that the incorrect
> "migrate-to" operation name defined in my VirtualDomain
> resource was my mistake. It was mis-coded in the virtual guest
> provisioning script. I have since changed it to "migrate_to"
> and of course, the specified live migration timeout value is working
> effectively now. (For some reason, I assumed we were letting that
> operation meta value default).
> 
> I was wondering if someone could refer me to the definitive online link
> for pacemaker resource man pages? I don't see any resource man pages
> installed
> on my system anywhere. I found this one online:
> https://www.mankier.com/7/ocf_heartbeat_VirtualDomain but is there a
> more 'official' page I should refer our
> Linux KVM on System z customers to?

All distributions that I know of include the man pages with the packages
they distribute. Are you building from source? They are named like "man
ocf_heartbeat_IPaddr2".

FYI after following this thread, the pcs developers are making a change
so that pcs refuses to add an unrecognized operation unless the user
uses --force. Thanks for being involved in the community; this is how we
learn to improve!

> Thanks again for your assistance.
> 
> Scott Greenlese ...IBM KVM on System Z Solution Test Poughkeepsie, N.Y.
> INTERNET: swgre...@us.ibm.com
> 
> 
> Inactive hide details for "Ulrich Windl" ---01/27/2017 02:32:43 AM--->>>
> "Scott Greenlese"  schrieb am 27."Ulrich Windl"
> ---01/27/2017 02:32:43 AM--->>> "Scott Greenlese" 
> schrieb am 27.01.2017 um 02:47 in Nachricht
> 
> From: "Ulrich Windl" 
> To: , Scott Greenlese/Poughkeepsie/IBM@IBMUS
> Cc: "Si Bo Niu" , Michael Tebolt/Poughkeepsie/IBM@IBMUS
> Date: 01/27/2017 02:32 AM
> Subject: Antw: Re: [ClusterLabs] Antw: Re: Live Guest Migration timeouts
> for VirtualDomain resources
> 
> 
> 
> 
> 
>>>> "Scott Greenlese"  schrieb am 27.01.2017 um
> 02:47 in
> Nachricht
>  
> m>:
> 
>> Hi guys..
>>
>> Well, today I confirmed that what Ulrich said is correct.  If I update the
>> VirtualDomain resource with the operation name  "migrate_to" instead of
>> "migrate-to",  it effectively overrides and enforces the 1200ms default
>> value to the new value.
>>
>> I am wondering how I would have known that I was using the wrong operation
>> name, when the initial operation name is already incorrect
>> when the resource is created?
> 
> For SLES 11, I made a quick (portable non-portable unstable) try (print
> the operations known to an RA):
> # crm ra info VirtualDomain |sed -n -e "/Operations' defaults/,\$p"
> Operations' defaults (advisory minimum):
> 
>start timeout=90
>stop  timeout=90
>statustimeout=30 interval=10
>monitor   timeout=30 interval=10
>migrate_from  timeout=60
>migrate_totimeout=120
> 
> Regards,
> Ulrich
> 
>>
>> This is what the meta data for my resource looked like after making the
>> update:
>>
>> [root@zs95kj VD]# date;pcs resource update zs95kjg110065_res op migrate_to
>> timeout="360s"
>> Thu Jan 26 16:43:11 EST 2017
>> You have new mail in /var/spool/mail/root
>>
>> [root@zs95kj VD]# date;pcs resource show zs95kjg110065_res
>> Thu Jan 26 16:43:46 EST 2017
>>  Resource: zs95kjg110065_res (class=ocf provider=heartbeat
>> type=VirtualDomain)
>>   Attributes: config=/guestxml/nfs1/zs95kjg110065.xml
>> hypervisor=qemu:///system migration_transport=ssh
>>   Meta Attrs: allow-migrate=true
>>   Operations: start interval=0s timeout=120
>> (zs95kjg110065_res-start-interval-0s)
>>   stop interval=0s timeout=120
>> (zs95kjg110065_res-stop-interval-0s)
>>   monitor interval=30s
> (zs95kjg110065_res-monitor-interval-30s)
>>   migrate-from interval=0s timeout=1200
>> (zs95kjg110065_res-migrate-from-interval-0s)
>>   migrate-to interval=0s timeout=1200
>> (zs95kjg110065_res-migrate-to-interval-0s)   <<< Original op name / value
>>   migrate_to interval=0s timeout=360s
>> (zs95kjg110065_res-migrate_to-interval-0s)  <<< New op name / value
>>
>>
>> Where does that original op name come from in the VirtualDomain resource
>> definition?  How can we get the initial me

Re: [ClusterLabs] Failover When Host is Up, Out of Order Logs

[Ken Gaillot]

On 01/31/2017 11:44 AM, Corey Moullas wrote:
> I have been getting extremely strange behavior from a Corosync/Pacemaker
> install on OVH Public Cloud servers.
> 
>  
> 
> After hours of Googling, I thought I would try posting here to see if
> somebody knows what to do.
> 
>  
> 
> I see this in my logs very frequently:
> 
> Jan 31 10:31:36 [581] fusion01-2 corosync warning [MAIN  ] Corosync main
> process was not scheduled for 24334.5645 ms (threshold is 2400. ms).
> Consider token timeout increase.
> 
> Jan 31 10:31:36 [581] fusion01-2 corosync notice  [TOTEM ] A processor
> failed, forming new configuration.
> 
>  
> 
> I have increased token time to 10s and this still occurs regularly even
> though both hosts are always up.

The "was not scheduled" message means the kernel is not giving corosync
enough CPU time to keep the token alive. The message indicates that it
didn't get scheduled for 24 seconds, which is why 10 seconds wouldn't
help -- but simply raising the timeout isn't a good idea at that point.
You need to figure out why you're starved for CPU. Public clouds don't
generally provide any guarantees, so it may be on their end.

> There are also times when the floating IP script is fired, but corosync
> does not seem aware. When you run crm status it will show the ip being
> bound the fusion01-1 when in fact the script fired and moved it to
> fusion01-2.
> 
>  
> 
> Finally, the timing of the logs seems very odd. This is what the end of
> my corosync log file looks like. Notice the times appear out of order in
> the logs. I’m ripping my hair out with these issues. Anybody have a clue
> what may be going on here?

Both pacemaker and corosync write to the same log (which is probably a
bad idea and will be changed one day, but that's not of concern here).
Each is using its own time source. If pacemaker is getting scheduled
more frequently than corosync, it's certainly possible log messages will
be out of order, since corosync isn't able to write out buffers as often.

> 
>  
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_process_request:   Completed cib_modify operation for
> section status: OK (rc=0, origin=fusion01-1/crmd/245, version=0.81.123)
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:Diff: --- 0.81.123 2
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:Diff: +++ 0.81.124 (null)
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:-- /cib/status/node_state[@id='1']/lrm[@id='1']
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:+  /cib:  @num_updates=124
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_process_request:   Completed cib_delete operation for
> section //node_state[@uname='fusion01-1']/lrm: OK (rc=0,
> origin=fusion01-1/crmd/246, version=0.81.124)
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:Diff: --- 0.81.124 2
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:Diff: +++ 0.81.125 (null)
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:+  /cib:  @num_updates=125
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:+  /cib/status/node_state[@id='1']: 
> @crm-debug-origin=do_lrm_query_internal
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:++ /cib/status/node_state[@id='1']:  
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:++
> 
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:++  
> 
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:++
>  operation_key="FloatIP_start_0" operation="start"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.10"
> transition-key="5:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1"
> transition-magic="0:0;5:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1"
> on_node="fusion01-1" call-id="17" rc-code="0" op-status="0" interval="0"
> last-run="1485859189" last-rc-change="1485859189" e
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:++
>   
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:++  
> 
> 
> Jan 31 10:31:47 [21062] fusion01-2cib: info:
> cib_perform_op:++
>  operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.10"
> transition-key="4:1:7:1fe20aa3-b305-4282-99a3-b1f8190d3c2c"
> transition-magic="0:0;4:1:7:1fe20aa3-b305-4282-99a3-b1f8190d3c2c"
> on_node="fusion01-1" call-id="9" rc-code="0" op-status="0

Re: [ClusterLabs] [Question] About a change of crm_failcount.

[Ken Gaillot]

On 02/02/2017 12:23 PM, renayama19661...@ybb.ne.jp wrote:
> Hi All,
> 
> By the next correction, the user was not able to set a value except zero in 
> crm_failcount.
> 
>  - [Fix: tools: implement crm_failcount command-line options correctly]
>- 
> https://github.com/ClusterLabs/pacemaker/commit/95db10602e8f646eefed335414e40a994498cafd#diff-6e58482648938fd488a920b9902daac4
> 
> However, pgsql RA sets INFINITY in a script.
> 
> ```
> (snip)
> CRM_FAILCOUNT="${HA_SBIN_DIR}/crm_failcount"
> (snip)
> ocf_exit_reason "My data is newer than new master's one. New   master's 
> location : $master_baseline"
> exec_with_retry 0 $CRM_FAILCOUNT -r $OCF_RESOURCE_INSTANCE -U $NODENAME 
> -v INFINITY
> return $OCF_ERR_GENERIC
> (snip)
> ```
> 
> There seems to be the influence only in pgsql somehow or other.
> 
> Can you revise it to set a value except zero in crm_failcount?
> We make modifications to use crm_attribute in pgsql RA if we cannot revise it.
> 
> Best Regards,
> Hideo Yamauchi.

Hmm, I didn't realize that was used. I changed it because it's not a
good idea to set fail-count without also changing last-failure and
having a failed op in the LRM history. I'll have to think about what the
best alternative is.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] A stop job is running for pacemaker high availability cluster manager

[Ken Gaillot]

On 02/02/2017 12:35 PM, Oscar Segarra wrote:
> Hi, 
> 
> I have a two node cluster... when I try to shutdown the physical host I
> get the following message in console: "a stop job is running for
> pacemaker high availability cluster manager" and never stops...

That would be a message from systemd. You'll need to check the pacemaker
status and/or logs to see why pacemaker can't shut down.

Without stonith enabled, pacemaker will be unable to recover if a
resource fails to stop. That could lead to a hang.

> This is my configuration:
> 
> [root@vdicnode01 ~]# pcs config
> Cluster Name: vdic-cluster
> Corosync Nodes:
>  vdicnode01-priv vdicnode02-priv
> Pacemaker Nodes:
>  vdicnode01-priv vdicnode02-priv
> 
> Resources:
>  Resource: nfs-vdic-mgmt-vm-vip (class=ocf provider=heartbeat type=IPaddr)
>   Attributes: ip=192.168.100.200 cidr_netmask=24
>   Operations: start interval=0s timeout=20s
> (nfs-vdic-mgmt-vm-vip-start-interval-0s)
>   stop interval=0s timeout=20s
> (nfs-vdic-mgmt-vm-vip-stop-interval-0s)
>   monitor interval=10s
> (nfs-vdic-mgmt-vm-vip-monitor-interval-10s)
>  Clone: nfs_setup-clone
>   Resource: nfs_setup (class=ocf provider=heartbeat type=ganesha_nfsd)
>Attributes: ha_vol_mnt=/var/run/gluster/shared_storage
>Operations: start interval=0s timeout=5s (nfs_setup-start-interval-0s)
>stop interval=0s timeout=5s (nfs_setup-stop-interval-0s)
>monitor interval=0 timeout=5s (nfs_setup-monitor-interval-0)
>  Clone: nfs-mon-clone
>   Resource: nfs-mon (class=ocf provider=heartbeat type=ganesha_mon)
>Operations: start interval=0s timeout=40s (nfs-mon-start-interval-0s)
>stop interval=0s timeout=40s (nfs-mon-stop-interval-0s)
>monitor interval=10s timeout=10s
> (nfs-mon-monitor-interval-10s)
>  Clone: nfs-grace-clone
>   Meta Attrs: notify=true
>   Resource: nfs-grace (class=ocf provider=heartbeat type=ganesha_grace)
>Meta Attrs: notify=true
>Operations: start interval=0s timeout=40s (nfs-grace-start-interval-0s)
>stop interval=0s timeout=40s (nfs-grace-stop-interval-0s)
>monitor interval=5s timeout=10s
> (nfs-grace-monitor-interval-5s)
>  Resource: vm-vdicone01 (class=ocf provider=heartbeat type=VirtualDomain)
>   Attributes: hypervisor=qemu:///system
> config=/mnt/nfs-vdic-mgmt-vm/vdicone01.xml
> migration_network_suffix=tcp:// migration_transport=ssh
>   Meta Attrs: allow-migrate=true target-role=Stopped
>   Utilization: cpu=1 hv_memory=512
>   Operations: start interval=0s timeout=90 (vm-vdicone01-start-interval-0s)
>   stop interval=0s timeout=90 (vm-vdicone01-stop-interval-0s)
>   monitor interval=20s role=Stopped
> (vm-vdicone01-monitor-interval-20s)
>   monitor interval=30s (vm-vdicone01-monitor-interval-30s)
>  Resource: vm-vdicsunstone01 (class=ocf provider=heartbeat
> type=VirtualDomain)
>   Attributes: hypervisor=qemu:///system
> config=/mnt/nfs-vdic-mgmt-vm/vdicsunstone01.xml
> migration_network_suffix=tcp:// migration_transport=ssh
>   Meta Attrs: allow-migrate=true target-role=Stopped
>   Utilization: cpu=1 hv_memory=1024
>   Operations: start interval=0s timeout=90
> (vm-vdicsunstone01-start-interval-0s)
>   stop interval=0s timeout=90
> (vm-vdicsunstone01-stop-interval-0s)
>   monitor interval=20s role=Stopped
> (vm-vdicsunstone01-monitor-interval-20s)
>   monitor interval=30s (vm-vdicsunstone01-monitor-interval-30s)
>  Resource: vm-vdicdb01 (class=ocf provider=heartbeat type=VirtualDomain)
>   Attributes: hypervisor=qemu:///system
> config=/mnt/nfs-vdic-mgmt-vm/vdicdb01.xml
> migration_network_suffix=tcp:// migration_transport=ssh
>   Meta Attrs: allow-migrate=true target-role=Stopped
>   Utilization: cpu=1 hv_memory=512
>   Operations: start interval=0s timeout=90 (vm-vdicdb01-start-interval-0s)
>   stop interval=0s timeout=90 (vm-vdicdb01-stop-interval-0s)
>   monitor interval=20s role=Stopped
> (vm-vdicdb01-monitor-interval-20s)
>   monitor interval=30s (vm-vdicdb01-monitor-interval-30s)
>  Clone: nfs-vdic-images-vip-clone
>   Resource: nfs-vdic-images-vip (class=ocf provider=heartbeat type=IPaddr)
>Attributes: ip=192.168.100.201 cidr_netmask=24
>Operations: start interval=0s timeout=20s
> (nfs-vdic-images-vip-start-interval-0s)
>stop interval=0s timeout=20s
> (nfs-vdic-images-vip-stop-interval-0s)
>monitor interval=10s
> (nfs-vdic-images-vip-monitor-interval-10s)
>  Resource: vm-vdicudsserver (class=ocf provider=heartbeat
> type=VirtualDomain)
>   Attributes: hypervisor=qemu:///system
> config=/mnt/nfs-vdic-mgmt-vm/vdicudsserver.xml
> migration_network_suffix=tcp:// migration_transport=ssh
>   Meta Attrs: allow-migrate=true target-role=Stopped
>   Utilization: cpu=1 hv_memory=1024
>   Operations: start interval=0s timeout=90
> (vm-vdicudsserver-start-interval-0s)
>   stop interval=0s

Re: [ClusterLabs] Huge amount of files in /var/lib/pacemaker/pengine

[Ken Gaillot]

On 02/02/2017 12:49 PM, Oscar Segarra wrote:
> Hi, 
> 
> A lot of files appear in /var/lib/pacemaker/pengine and fulls my hard disk.
> 
> Is there any way to avoid such amount of files in that directory?
> 
> Thanks in advance!

Pacemaker saves the cluster state at each calculated transition. This
can come in handy when investigating after a problem occurs, to see what
changed and how the cluster responded.

The files are not necessary for cluster operation, so you can clean them
as desired. The cluster can clean them for you based on cluster options;
see pe-error-series-max, pe-warn-series-max, and pe-input-series-max:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/s-cluster-options.html

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action.

[Ken Gaillot]

On 02/02/2017 02:14 PM, Scott Greenlese wrote:
> Hi folks,
> 
> I'm testing iface-bridge resource support on a Linux KVM on System Z
> pacemaker cluster.
> 
> pacemaker-1.1.13-10.el7_2.ibm.1.s390x
> corosync-2.3.4-7.el7_2.ibm.1.s390x
> 
> I created an iface-bridge resource, but specified a non-existent
> bridge_slaves value, vlan1292 (i.e. vlan1292 doesn't exist).
> 
> [root@zs95kj VD]# date;pcs resource create br0_r1
> ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op
> monitor timeout="20s" interval="10s" --disabled
> Wed Feb 1 17:49:16 EST 2017
> [root@zs95kj VD]#
> 
> [root@zs95kj VD]# pcs resource show |grep br0
> br0_r1 (ocf::heartbeat:iface-bridge): FAILED zs93kjpcs1
> [root@zs95kj VD]#
> 
> As you can see, the resource was created, but failed to start on the
> target node zs93kppcs1.
> 
> To my surprise, the target node zs93kppcs1 was unceremoniously fenced.
> 
> pacemaker.log shows a fence (off) action initiated against that target
> node, "because of resource failure(s)" :
> 
> Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2719 ) debug:
> determine_op_status: br0_r1_stop_0 on zs93kjpcs1 returned 'not
> configured' (6) instead of the expected value: 'ok' (0)
> Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2602 ) warning:
> unpack_rsc_op_failure: Processing failed op stop for br0_r1 on
> zs93kjpcs1: not configured (6)
> Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:3244 ) error:
> unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation
> stop failed 'not configured' (6)
> Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2719 ) debug:
> determine_op_status: br0_r1_stop_0 on zs93kjpcs1 returned 'not
> configured' (6) instead of the expected value: 'ok' (0)
> Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:2602 ) warning:
> unpack_rsc_op_failure: Processing failed op stop for br0_r1 on
> zs93kjpcs1: not configured (6)
> Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:3244 ) error:
> unpack_rsc_op: Preventing br0_r1 from re-starting anywhere: operation
> stop failed 'not configured' (6)
> Feb 01 17:55:56 [52941] zs95kj crm_resource: ( unpack.c:96 ) warning:
> pe_fence_node: Node zs93kjpcs1 will be fenced because of resource failure(s)
> 
> 
> Thankfully, I was able to successfully create a iface-bridge resource
> when I changed the bridge_slaves value to an existent vlan interface.
> 
> My main concern is, why would the response to a failed bridge config
> operation warrant a node fence (off) action? Isn't it enough to just
> fail the resource and try another cluster node,
> or at most, give up if it can't be started / configured on any node?
> 
> Is there any way to control this harsh recovery action in the cluster?
> 
> Thanks much..
> 
> 
> Scott Greenlese ... IBM KVM on System Z Solutions Test, Poughkeepsie, N.Y.
> INTERNET: swgre...@us.ibm.com

It's actually the stop operation failure that leads to the fence.

If a resource fails to stop, fencing is the only way pacemaker can
recover the resource elsewhere. Consider a database master -- if it
doesn't stop, starting the master elsewhere could lead to severe data
inconsistency.

You can tell pacemaker to not attempt recovery, by setting on-fail=block
on the stop operation, so it doesn't need to fence. Obviously, that
prevents high availability, as manual intervention is required to do
anything further with the service.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] A stop job is running for pacemaker high availability cluster manager

[Ken Gaillot]

On 02/02/2017 03:06 PM, Oscar Segarra wrote:
> Hi Ken, 
> 
> I have checked the /var/log/cluster/corosync.log and there no
> information about why system hangs stopping... 
> 
> ¿Can you be more specific about what logs to check?
> 
> Thanks a lot.

There, and /var/log/messages sometimes has relevant messages from
non-cluster components.

You'd want to look for messages like "Caught 'Terminated' signal" and
"Shutting down", as well as resources being stopped ("_stop_0"), then
various "Disconnect" and "Stopping" messages as individual daemons exit.

> 2017-02-02 21:10 GMT+01:00 Ken Gaillot  <mailto:kgail...@redhat.com>>:
> 
> On 02/02/2017 12:35 PM, Oscar Segarra wrote:
> > Hi,
> >
> > I have a two node cluster... when I try to shutdown the physical host I
> > get the following message in console: "a stop job is running for
> > pacemaker high availability cluster manager" and never stops...
> 
> That would be a message from systemd. You'll need to check the pacemaker
> status and/or logs to see why pacemaker can't shut down.
> 
> Without stonith enabled, pacemaker will be unable to recover if a
> resource fails to stop. That could lead to a hang.
> 
> > This is my configuration:
> >
> > [root@vdicnode01 ~]# pcs config
> > Cluster Name: vdic-cluster
> > Corosync Nodes:
> >  vdicnode01-priv vdicnode02-priv
> > Pacemaker Nodes:
> >  vdicnode01-priv vdicnode02-priv
> >
> > Resources:
> >  Resource: nfs-vdic-mgmt-vm-vip (class=ocf provider=heartbeat
> type=IPaddr)
> >   Attributes: ip=192.168.100.200 cidr_netmask=24
> >   Operations: start interval=0s timeout=20s
> > (nfs-vdic-mgmt-vm-vip-start-interval-0s)
> >   stop interval=0s timeout=20s
> > (nfs-vdic-mgmt-vm-vip-stop-interval-0s)
> >   monitor interval=10s
> > (nfs-vdic-mgmt-vm-vip-monitor-interval-10s)
> >  Clone: nfs_setup-clone
> >   Resource: nfs_setup (class=ocf provider=heartbeat type=ganesha_nfsd)
> >Attributes: ha_vol_mnt=/var/run/gluster/shared_storage
> >Operations: start interval=0s timeout=5s
> (nfs_setup-start-interval-0s)
> >stop interval=0s timeout=5s
> (nfs_setup-stop-interval-0s)
> >monitor interval=0 timeout=5s
> (nfs_setup-monitor-interval-0)
> >  Clone: nfs-mon-clone
> >   Resource: nfs-mon (class=ocf provider=heartbeat type=ganesha_mon)
> >Operations: start interval=0s timeout=40s
> (nfs-mon-start-interval-0s)
> >stop interval=0s timeout=40s (nfs-mon-stop-interval-0s)
> >monitor interval=10s timeout=10s
> > (nfs-mon-monitor-interval-10s)
> >  Clone: nfs-grace-clone
> >   Meta Attrs: notify=true
> >   Resource: nfs-grace (class=ocf provider=heartbeat
> type=ganesha_grace)
> >Meta Attrs: notify=true
> >Operations: start interval=0s timeout=40s
> (nfs-grace-start-interval-0s)
> >stop interval=0s timeout=40s
> (nfs-grace-stop-interval-0s)
> >monitor interval=5s timeout=10s
> > (nfs-grace-monitor-interval-5s)
> >  Resource: vm-vdicone01 (class=ocf provider=heartbeat
> type=VirtualDomain)
> >   Attributes: hypervisor=qemu:///system
> > config=/mnt/nfs-vdic-mgmt-vm/vdicone01.xml
> > migration_network_suffix=tcp:// migration_transport=ssh
> >   Meta Attrs: allow-migrate=true target-role=Stopped
> >   Utilization: cpu=1 hv_memory=512
> >   Operations: start interval=0s timeout=90
> (vm-vdicone01-start-interval-0s)
> >   stop interval=0s timeout=90
> (vm-vdicone01-stop-interval-0s)
> >   monitor interval=20s role=Stopped
> > (vm-vdicone01-monitor-interval-20s)
> >   monitor interval=30s (vm-vdicone01-monitor-interval-30s)
> >  Resource: vm-vdicsunstone01 (class=ocf provider=heartbeat
> > type=VirtualDomain)
> >   Attributes: hypervisor=qemu:///system
> > config=/mnt/nfs-vdic-mgmt-vm/vdicsunstone01.xml
> > migration_network_suffix=tcp:// migration_transport=ssh
> >   Meta Attrs: allow-migrate=true target-role=Stopped
> >   Utilization: cpu=1 hv_memory=1024
> >   Operations: start interval=0s timeout=90
> > (vm-vdicsunstone01-start-interval-0s)
> >   stop interval=0s timeout=90
> >

Re: [ClusterLabs] Pacemaker kill does not cause node fault ???

[Ken Gaillot]

On 02/03/2017 07:00 AM, RaSca wrote:
> 
> On 03/02/2017 11:06, Ferenc Wágner wrote:
>> Ken Gaillot  writes:
>>
>>> On 01/10/2017 04:24 AM, Stefan Schloesser wrote:
>>>
>>>> I am currently testing a 2 node cluster under Ubuntu 16.04. The setup
>>>> seems to be working ok including the STONITH.
>>>> For test purposes I issued a "pkill -f pace" killing all pacemaker
>>>> processes on one node.
>>>>
>>>> Result:
>>>> The node is marked as "pending", all resources stay on it. If I
>>>> manually kill a resource it is not noticed. On the other node a drbd
>>>> "promote" command fails (drbd is still running as master on the first
>>>> node).
>>>
>>> I suspect that, when you kill pacemakerd, systemd respawns it quickly
>>> enough that fencing is unnecessary. Try "pkill -f pace; systemd stop
>>> pacemaker".
>>
>> What exactly is "quickly enough"?
> 
> What Ken is saying is that Pacemaker, as a service managed by systemd,
> have in its service definition file
> (/usr/lib/systemd/system/pacemaker.service) this option:
> 
> Restart=on-failure
> 
> Looking at [1] it is explained: systemd restarts immediately the process
> if it ends for some unexpected reason (like a forced kill).
> 
> [1] https://www.freedesktop.org/software/systemd/man/systemd.service.html

And the cluster itself is resilient to some daemon restarts. If only
pacemakerd is killed, corosync and pacemaker's crmd can still function
without any issues. When pacemakerd respawns, it reestablishes contact
with any other cluster daemons still running (and its pacemakerd peers
on other cluster nodes).

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Manage Docker service and containers with pacemaker

[Ken Gaillot]

On 02/03/2017 08:15 AM, Stephane Gaucher wrote:
> Hello I am completing a proof of concept.
> 
> Here are the facts:
> An active / passive cluster.Done
> A drbd partition for exchanging files for different servicesDone
> A shared VIP between the two nodesDone
> The docker/containers are functional. I do not want to use docker swarm.
> This is not the purpose of this message.
> 
> The services I develop are in the form of containers. I would like to be
> able to control the docker service with pacemaker. The docker system
> folders are on a drbd partition so available on the second node when it
> is active.
> 
> For the moment it works well, but I have to manually start the docker
> service as well as the containers on the active node for the switch to
> be effective. The server is in high availability in the hardly way.
> 
> So is it possible to create a directive that starts the docker service
> and even the container by using pacemaker.

Yes!

Is it documented? No. :-(

There are three types of support for containers in Pacemaker: the
ocf:heartbeat:docker resource agent, running Pacemaker Remote inside a
container, and Pacemaker's "resource isolation" capability.

The resource agent is pretty easy to figure out from its meta-data and
man page:

  https://www.mankier.com/7/ocf_heartbeat_docker

The Pacemaker Remote documentation gives a few pointers on that approach
under "Containers as Guest Nodes":


http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote/index.html

The resource isolation capability is probably the most interesting. The
best documentation currently is the "Resource Isolation" section of this
document (the "blackbox" use case it mentions is just the
ocf:heartbeat:docker agent mentioned above):


https://github.com/davidvossel/phd/blob/master/doc/presentations/pacemaker_remote_arch.markdown

Testing and feedback on any of these methods are welcome.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [Question] About a change of crm_failcount.

[Ken Gaillot]

On 02/02/2017 12:33 PM, Ken Gaillot wrote:
> On 02/02/2017 12:23 PM, renayama19661...@ybb.ne.jp wrote:
>> Hi All,
>>
>> By the next correction, the user was not able to set a value except zero in 
>> crm_failcount.
>>
>>  - [Fix: tools: implement crm_failcount command-line options correctly]
>>- 
>> https://github.com/ClusterLabs/pacemaker/commit/95db10602e8f646eefed335414e40a994498cafd#diff-6e58482648938fd488a920b9902daac4
>>
>> However, pgsql RA sets INFINITY in a script.
>>
>> ```
>> (snip)
>> CRM_FAILCOUNT="${HA_SBIN_DIR}/crm_failcount"
>> (snip)
>> ocf_exit_reason "My data is newer than new master's one. New   master's 
>> location : $master_baseline"
>> exec_with_retry 0 $CRM_FAILCOUNT -r $OCF_RESOURCE_INSTANCE -U $NODENAME 
>> -v INFINITY
>> return $OCF_ERR_GENERIC
>> (snip)
>> ```
>>
>> There seems to be the influence only in pgsql somehow or other.
>>
>> Can you revise it to set a value except zero in crm_failcount?
>> We make modifications to use crm_attribute in pgsql RA if we cannot revise 
>> it.
>>
>> Best Regards,
>> Hideo Yamauchi.
> 
> Hmm, I didn't realize that was used. I changed it because it's not a
> good idea to set fail-count without also changing last-failure and
> having a failed op in the LRM history. I'll have to think about what the
> best alternative is.

Having a resource agent modify its own fail count is not a good idea,
and could lead to unpredictable behavior. I didn't realize the pgsql
agent did that.

I don't want to re-enable the functionality, because I don't want to
encourage more agents doing this.

There are two alternatives the pgsql agent can choose from:

1. Return a "hard" error such as OCF_ERR_ARGS or OCF_ERR_PERM. When
Pacemaker gets one of these errors from an agent, it will ban the
resource from that node (until the failure is cleared).

2. Use crm_resource --ban instead. This would ban the resource from that
node until the user removes the ban with crm_resource --clear (or by
deleting the ban consraint from the configuration).

I'd recommend #1 since it does not require any pacemaker-specific tools.

We can make sure resource-agents has a fix for this before we release a
new version of Pacemaker. We'll have to publicize as much as possible to
pgsql users that they should upgrade resource-agents before or at the
same time as pacemaker. I see the alternative PAF agent has the same
usage, so it will need to be updated, too.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Pacemaker kill does not cause node fault ???

[Ken Gaillot]

On 02/06/2017 03:28 AM, Ulrich Windl wrote:
>>>> RaSca  schrieb am 03.02.2017 um 14:00 in
> Nachricht
> <0de64981-904f-5bdb-c98f-9c59ee47b...@miamammausalinux.org>:
> 
>> On 03/02/2017 11:06, Ferenc Wágner wrote:
>>> Ken Gaillot  writes:
>>>
>>>> On 01/10/2017 04:24 AM, Stefan Schloesser wrote:
>>>>
>>>>> I am currently testing a 2 node cluster under Ubuntu 16.04. The setup
>>>>> seems to be working ok including the STONITH.
>>>>> For test purposes I issued a "pkill -f pace" killing all pacemaker
>>>>> processes on one node.
>>>>>
>>>>> Result:
>>>>> The node is marked as "pending", all resources stay on it. If I
>>>>> manually kill a resource it is not noticed. On the other node a drbd
>>>>> "promote" command fails (drbd is still running as master on the first
>>>>> node).
>>>>
>>>> I suspect that, when you kill pacemakerd, systemd respawns it quickly
>>>> enough that fencing is unnecessary. Try "pkill -f pace; systemd stop
>>>> pacemaker".
>>>
>>> What exactly is "quickly enough"?
>>
>> What Ken is saying is that Pacemaker, as a service managed by systemd,
>> have in its service definition file
>> (/usr/lib/systemd/system/pacemaker.service) this option:
>>
>> Restart=on-failure
>>
>> Looking at [1] it is explained: systemd restarts immediately the process
>> if it ends for some unexpected reason (like a forced kill).
> 
> Isn't the question: Is crmd a process that is expected to die (and thus need
> restarting)? Or wouldn't one prefer to debug this situation. I fear that
> restarting it might just cover some fatal failure...

If crmd or corosync dies, the node will be fenced (if fencing is enabled
and working). If one of the crmd's persistent connections (such as to
the cib) fails, it will exit, so it ends up the same. But the other
daemons (such as pacemakerd or attrd) can die and respawn without any
risk to services.

The failure will be logged, but it will not be reported in cluster
status, so there is a chance of not noticing it.

> 
>>
>> [1] https://www.freedesktop.org/software/systemd/man/systemd.service.html 
>>
>> -- 
>> RaSca
>> ra...@miamammausalinux.org 

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Failure to configure iface-bridge resource causes cluster node fence action.

[Ken Gaillot]

On 02/06/2017 09:00 AM, Scott Greenlese wrote:
> Further explanation for my concern about --disabled not taking effect
> until after the iface-bridge was configured ...
> 
> The reason I wanted to create the iface-bridge resource "disabled", was
> to allow me the opportunity to impose
> a location constraint / rule on the resource to prevent it from being
> started on certain cluster nodes,
> where the specified slave vlan did not exist.
> 
> In my case, pacemaker assigned the resource to a cluster node where the
> specified slave vlan did not exist, which in turn
> triggered a fenced (off) action against that node (apparently, because
> the device could not be stopped, per Ken's reply earlier).
> 
> Again, my cluster is configured as "symmetric" , so I would have to "opt
> out" my new resource from
> certain cluster nodes via location constraint.
> 
> So, if this really is how --disable is designed to work, is there any
> way to impose a location constraint rule BEFORE
> the iface-bridge resource gets assigned. configured and started on a
> cluster node in a symmetrical cluster?

I would expect --disabled to behave like that already; I'm not sure
what's happening there.

But, you can add a resource and any constraints that apply to it
simultaneously. How to do this depends on whether you want to do it
interactively or scripted, and whether you prefer the low-level tools,
crm shell, or pcs.

If you want to script it via pcs, you can do pcs cluster cib $SOME_FILE,
then pcs -f $SOME_FILE , then pcs cluster
cib-push $SOME_FILE --config.

> 
> Thanks,
> 
> Scott Greenlese ... IBM KVM on System Z - Solutions Test, Poughkeepsie, N.Y.
> INTERNET: swgre...@us.ibm.com
> 
> 
> 
> Inactive hide details for Scott Greenlese---02/03/2017 03:23:40
> PM---Ken, Thanks for the explanation.Scott Greenlese---02/03/2017
> 03:23:40 PM---Ken, Thanks for the explanation.
> 
> From: Scott Greenlese/Poughkeepsie/IBM@IBMUS
> To: kgail...@redhat.com, Cluster Labs - All topics related to
> open-source clustering welcomed 
> Date: 02/03/2017 03:23 PM
> Subject: Re: [ClusterLabs] Failure to configure iface-bridge resource
> causes cluster node fence action.
> 
> 
> 
> 
> 
> Ken,
> 
> Thanks for the explanation.
> 
> One other thing, relating to the iface-bridge resource creation. I
> specified --disabled flag:
> 
>> [root@zs95kj VD]# date;pcs resource create br0_r1
>> ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op
>> monitor timeout="20s" interval="10s" --*disabled*
> 
> Does the bridge device have to be successfully configured by pacemaker
> before disabling the resource? It seems
> that that was the behavior, since it failed the resource and fenced the
> node instead of disabling the resource.
> Just checking with you to be sure.
> 
> Thanks again..
> 
> Scott Greenlese ... IBM KVM on System Z Solutions Test, Poughkeepsie, N.Y.
> INTERNET: swgre...@us.ibm.com
> 
> 
> 
> Inactive hide details for Ken Gaillot ---02/02/2017 03:29:12 PM---On
> 02/02/2017 02:14 PM, Scott Greenlese wrote: > Hi folks,Ken Gaillot
> ---02/02/2017 03:29:12 PM---On 02/02/2017 02:14 PM, Scott Greenlese
> wrote: > Hi folks,
> 
> From: Ken Gaillot 
> To: users@clusterlabs.org
> Date: 02/02/2017 03:29 PM
> Subject: Re: [ClusterLabs] Failure to configure iface-bridge resource
> causes cluster node fence action.
> 
> 
> 
> 
> On 02/02/2017 02:14 PM, Scott Greenlese wrote:
>> Hi folks,
>>
>> I'm testing iface-bridge resource support on a Linux KVM on System Z
>> pacemaker cluster.
>>
>> pacemaker-1.1.13-10.el7_2.ibm.1.s390x
>> corosync-2.3.4-7.el7_2.ibm.1.s390x
>>
>> I created an iface-bridge resource, but specified a non-existent
>> bridge_slaves value, vlan1292 (i.e. vlan1292 doesn't exist).
>>
>> [root@zs95kj VD]# date;pcs resource create br0_r1
>> ocf:heartbeat:iface-bridge bridge_name=br0 bridge_slaves=vlan1292 op
>> monitor timeout="20s" interval="10s" --disabled
>> Wed Feb 1 17:49:16 EST 2017
>> [root@zs95kj VD]#
>>
>> [root@zs95kj VD]# pcs resource show |grep br0
>> br0_r1 (ocf::heartbeat:iface-bridge): FAILED zs93kjpcs1
>> [root@zs95kj VD]#
>>
>> As you can see, the resource was created, but failed to start on the
>> target node zs93kppcs1.
>>
>> To my surprise, the target node zs93kppcs1 was unceremoniously fenced.
>>
>> pacemaker.log shows a fence (off) action initiated again

Re: [ClusterLabs] lrmd segfault

[Ken Gaillot]

On 02/06/2017 05:47 AM, cys wrote:
> Hi All.
> 
> Recently we got a lrmd coredump. It occured only once and  we don't know how 
> to reproduce it.
> The version we use is pacemaker-1.1.15-11. Ths os is centos 7.
> 
> Core was generated by `/usr/libexec/pacemaker/lrmd'.
> Program terminated with signal 11, Segmentation fault.
> #0  __strcasecmp_l_avx () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:164
> 164 movdqu  (%rdi), %xmm1
> (gdb) bt
> #0  __strcasecmp_l_avx () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:164
> #1  0x7fd6d554a53c in crm_str_eq (a=, 
> b=b@entry=0x7fd6d6d42800 "p_vip", use_case=use_case@entry=0) at utils.c:1454
> #2  0x7fd6d5322baa in is_op_blocked (rsc=0x7fd6d6d42800 "p_vip") at 
> services.c:653
> #3  0x7fd6d5322ca5 in services_action_async (op=0x7fd6d6d5f8d0, 
> action_callback=) at services.c:634
> #4  0x7fd6d59af67c in lrmd_rsc_execute_service_lib (cmd=0x7fd6d6d69bd0, 
> rsc=0x7fd6d6d5d6f0) at lrmd.c:1242
> #5  lrmd_rsc_execute (rsc=0x7fd6d6d5d6f0) at lrmd.c:1308
> #6  lrmd_rsc_dispatch (user_data=0x7fd6d6d5d6f0, user_data@entry= reading variable: value has been optimized out>) at lrmd
> #7  0x7fd6d55699f6 in crm_trigger_dispatch (source=0x7fd6d6d59190, 
> callback=, userdata=) at mainloop.c:107
> #8  0x7fd6d29757aa in g_main_context_dispatch () from 
> /lib64/libglib-2.0.so.0
> #9  0x7fd6d2975af8 in g_main_context_iterate.isra.24 () from 
> /lib64/libglib-2.0.so.0
> #10 0x7fd6d2975dca in g_main_loop_run () from /lib64/libglib-2.0.so.0
> #11 0x7fd6d59ad3ad in main (argc=, argv=0x7fff4bd0def8) at 
> main.c:476
> (gdb) p inflight_ops->data
> $4 = (gpointer) 0x7fd6d6d605c0
> (gdb) x/10xg 0x7fd6d6d605c0
> 0x7fd6d6d605c0: 0x 0x00030002
> 0x7fd6d6d605d0: 0x00020004  0x0005
> 0x7fd6d6d605e0: 0x0008  0x
> 0x7fd6d6d605f0: 0x000d  0x000f000e
> 0x7fd6d6d60600: 0x00010001  0x0013
> 
> The memory at inflight_ops->data is not a valid svc_action_t object.
> 
> I saw a similar problem at 
> http://lists.clusterlabs.org/pipermail/users/2017-January/004906.html.
> But it said the problem has gone in 1.1.15.
> 
> Any help would be appreciated.

That's odd -- it does look like the issue fixed by commits 67d68df and
786ebc4 in 1.1.15.

Are you absolutely sure that the node with the issue has 1.1.15, and
that pacemaker has been restarted since 1.1.15 was deployed?

If so, you may want to open an issue at bugs.clusterlabs.org and attach
the output of crm_report covering the time when the problem occurred.


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: Pacemaker kill does not cause node fault ???

[Ken Gaillot]

On 02/07/2017 01:11 AM, Ulrich Windl wrote:
>>>> Ken Gaillot  schrieb am 06.02.2017 um 16:13 in
> Nachricht
> <40eba339-2f46-28b8-4605-c7047e0ee...@redhat.com>:
>> On 02/06/2017 03:28 AM, Ulrich Windl wrote:
>>>>>> RaSca  schrieb am 03.02.2017 um 14:00 in
>>> Nachricht
>>> <0de64981-904f-5bdb-c98f-9c59ee47b...@miamammausalinux.org>:
>>>
>>>> On 03/02/2017 11:06, Ferenc Wágner wrote:
>>>>> Ken Gaillot  writes:
>>>>>
>>>>>> On 01/10/2017 04:24 AM, Stefan Schloesser wrote:
>>>>>>
>>>>>>> I am currently testing a 2 node cluster under Ubuntu 16.04. The setup
>>>>>>> seems to be working ok including the STONITH.
>>>>>>> For test purposes I issued a "pkill -f pace" killing all pacemaker
>>>>>>> processes on one node.
>>>>>>>
>>>>>>> Result:
>>>>>>> The node is marked as "pending", all resources stay on it. If I
>>>>>>> manually kill a resource it is not noticed. On the other node a drbd
>>>>>>> "promote" command fails (drbd is still running as master on the first
>>>>>>> node).
>>>>>>
>>>>>> I suspect that, when you kill pacemakerd, systemd respawns it quickly
>>>>>> enough that fencing is unnecessary. Try "pkill -f pace; systemd stop
>>>>>> pacemaker".
>>>>>
>>>>> What exactly is "quickly enough"?
>>>>
>>>> What Ken is saying is that Pacemaker, as a service managed by systemd,
>>>> have in its service definition file
>>>> (/usr/lib/systemd/system/pacemaker.service) this option:
>>>>
>>>> Restart=on-failure
>>>>
>>>> Looking at [1] it is explained: systemd restarts immediately the process
>>>> if it ends for some unexpected reason (like a forced kill).
>>>
>>> Isn't the question: Is crmd a process that is expected to die (and thus
> need
>>> restarting)? Or wouldn't one prefer to debug this situation. I fear that
>>> restarting it might just cover some fatal failure...
>>
>> If crmd or corosync dies, the node will be fenced (if fencing is enabled
>> and working). If one of the crmd's persistent connections (such as to
>> the cib) fails, it will exit, so it ends up the same. But the other
> 
> But isn't it due to crmd not responding to network packets? So if the timeout
> is long enough, and crmd is started fast enough, will the node really be
> fenced?

If crmd dies, it leaves its corosync process group, and I'm pretty sure
the other nodes will fence it for that reason, regardless of the duration.

>> daemons (such as pacemakerd or attrd) can die and respawn without any
>> risk to services.
>>
>> The failure will be logged, but it will not be reported in cluster
>> status, so there is a chance of not noticing it.
> 
> I don't understand: A node is fenced, but it will not be noted in the cluster
> status???

I meant the case where pacemakerd or attrd respawns quickly. The node is
not fenced in that case, and the only indication will be in the logs.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Using "mandatory" startup order but avoiding depending clones from restart after member of parent clone fails

[Ken Gaillot]

On 02/06/2017 05:25 PM, Alejandro Comisario wrote:
> guys, really happy to post my first doubt.
> 
> i'm kinda having an "conceptual" issue that's bringing me, lots of issues
> i need to ensure that order of starting resources are mandatory but
> that is causing me a huge issue, that is if just one of the members of
> a clone goes down and up (but not all members) all resources depending
> on it are restarted (wich is bad), my workaround is to set order as
> advisory, but that doesnt asure strict order startup.
> 
> eg. clone_b runs on servers_B, and depends on clone_a that runs on servers_A.
> 
> I'll put an example on how i have everything defined between this two clones.
> 
> ### clone_A running on servers A (location rule)
> primitive p_mysql mysql-wss \
> op monitor timeout=55 interval=60 enabled=true on-fail=restart \
> op start timeout=475 interval=0 on-fail=restart \
> op stop timeout=175 interval=0 \
> params socket="/var/run/mysqld/mysqld.sock"
> pid="/var/run/mysqld/mysqld.pid" test_passwd="XXX" test_user=root \
> meta is-managed=true
> 
> clone p_mysql-clone p_mysql \
> meta target-role=Started interleave=false globally-unique=false
> 
> location mysql_location p_mysql-clone resource-discovery=never \
> rule -inf: galera ne 1
> 
> ### clone_B running on servers B (location rule)
> primitive p_keystone apache \
> params configfile="/etc/apache2/apache2.conf" \
> op monitor on-fail=restart interval=60s timeout=60s \
> op start on-fail=restart interval=0 \
> meta target-role=Started migration-threshold=2 failure-timeout=60s
> resource-stickiness=300
> 
> clone p_keystone-clone p_keystone \
> meta target-role=Started interleave=false globally-unique=false
> 
> location keystone_location p_keystone-clone resource-discovery=never \
> rule -inf: keystone ne 1
> 
> order p_clone-mysql-before-p_keystone INF: p_mysql-clone 
> p_keystone-clone:start
> 
> Again just to make my point, if p_mysql-clone looses even one member
> of the clone, ONLY when that member gets back, all members of
> p_keystone-clone gets restarted, and thats NOT what i need, so if i
> change the order from mandatory to advisory, i get what i want
> regarding behaviour of what happens when instances of the clone comes
> and goes, but i loos the strictness of the startup order, which is
> critial for me.
> 
> How can i fix this problem ?
> .. can i ?

I don't think pacemaker can model your desired situation currently.

In OpenStack configs that I'm familiar with, the mysql server (usually
galera) is a master-slave clone, and the constraint used is "promote
mysql then start keystone". That way, if a slave goes away and comes
back, it has no effect.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: [Question] About a change of crm_failcount.

[Ken Gaillot]

On 02/09/2017 05:46 AM, Jehan-Guillaume de Rorthais wrote:
> On Thu, 9 Feb 2017 19:24:22 +0900 (JST)
> renayama19661...@ybb.ne.jp wrote:
> 
>> Hi Ken,
>>
>>
>>> 1. Return a "hard" error such as OCF_ERR_ARGS or OCF_ERR_PERM. When
>>> Pacemaker gets one of these errors from an agent, it will ban the
>>> resource from that node (until the failure is cleared).  
>>
>> The first suggestion does not work well.
>>
>> Even if this returns OCF_ERR_ARGS and OCF_ERR_PERM, it seems to be to be
>> pre_promote(notify) handling of RA. Pacemaker does not record the notify(pre
>> promote) error in CIB.
>>
>>  * https://github.com/ClusterLabs/pacemaker/blob/master/crmd/lrm.c#L2411
>>
>> Because it is not recorded in CIB, there cannot be the thing that pengine
>> works as "hard error".

Ah, I didn't think of that.

> Indeed. That's why PAF use private attribute to give informations between
> actions. We detect the failure during the notify as well, but raise the error
> during the promotion itself. See how I dealt with this in PAF:
> 
> https://github.com/ioguix/PAF/commit/6123025ff7cd9929b56c9af2faaefdf392886e68

That's a nice use of private attributes.

> As private attributes does not work on older stacks, you could rely on local
> temp file as well in $HA_RSCTMP.
> 
>>> 2. Use crm_resource --ban instead. This would ban the resource from that
>>> node until the user removes the ban with crm_resource --clear (or by
>>> deleting the ban consraint from the configuration).  
>>
>> The second suggestion works well.
>> I intend to adopt the second suggestion.
>>
>> As other methods, you think crm_resource -F to be available, but what do you
>> think? I think that last-failure does not have a problem either to let you
>> handle pseudotrouble if it is crm_resource -F.
>>
>> I think whether crm_resource -F is available, but adopt crm_resource -B
>> because RA wants to completely stop pgsql resource.
>>
>> ``` @pgsql RA
>>
>> pgsql_pre_promote() {
>> (snip)
>> if [ "$cmp_location" != "$my_master_baseline" ]; then
>> ocf_exit_reason "My data is newer than new master's one. New
>> master's location : $master_baseline" exec_with_retry 0 $CRM_RESOURCE -B -r
>> $OCF_RESOURCE_INSTANCE -N $NODENAME -Q return $OCF_ERR_GENERIC
>> fi
>> (snip)
>> CRM_FAILCOUNT="${HA_SBIN_DIR}/crm_failcount"
>> CRM_RESOURCE="${HA_SBIN_DIR}/crm_resource"
>> ```
>>
>> I test movement a little more and send a patch.
> 
> I suppose crm_resource -F will just raise the failcount, break the current
> transition and the CRM will recompute another transition paying attention to
> your "failed" resource (will it try to recover it? retry the previous
> transition again?).
> 
> I would bet on crm_resource -B.

Correct, crm_resource -F only simulates OCF_ERR_GENERIC, which is a soft
error. It might be a nice extension to be able to specify the error
code, but in this case, I think crm_resource -B (or the private
attribute approach, if you're OK with limiting it to corosync 2 and
pacemaker 1.1.13+) is better.

>> - Original Message -
>>> From: Ulrich Windl 
>>> To: users@clusterlabs.org; kgail...@redhat.com
>>> Cc: 
>>> Date: 2017/2/6, Mon 17:44
>>> Subject: [ClusterLabs] Antw: Re: [Question] About a change of crm_failcount.
>>>   
>>>>>>  Ken Gaillot  schrieb am 02.02.2017 um   
>>> 19:33 in Nachricht
>>> <91a83571-9930-94fd-e635-962830671...@redhat.com>:  
>>>>  On 02/02/2017 12:23 PM, renayama19661...@ybb.ne.jp wrote:  
>>>>>  Hi All,
>>>>>
>>>>>  By the next correction, the user was not able to set a value except   
>>> zero in   
>>>>  crm_failcount.  
>>>>>
>>>>>   - [Fix: tools: implement crm_failcount command-line options correctly]
>>>>> -   
>>>>   
>>> https://github.com/ClusterLabs/pacemaker/commit/95db10602e8f646eefed335414e40
>>>
>>>>  a994498cafd#diff-6e58482648938fd488a920b9902daac4  
>>>>>
>>>>>  However, pgsql RA sets INFINITY in a script.
>>>>>
>>>>>  ```
>>>>>  (snip)
>>>>>  CRM_FAILCOUNT="${HA_SBIN_DIR}/crm_failcount"
>>>>>  (snip)
>>>>>  ocf_exit_reason "My data is newer than new master's one.   
>>> New   master's