31.08.2017 14:53, Octavian Ciobanu wrote:
I'm back with confirmation that DLM is the one that triggers Mount
resources to stop when the stopped/suspended node rejoins
When the DLM resource is started, on the rejoining node, it tries to get
a free journal but always gets one that is already occupied by another
node and triggers a domino effect, one node jumps to the next node
occupied dlm journal, stopping and restarting the mount resources.
I have plenty of DLM journals (allocated 10 for 3 node configuration) so
for sure there are unused ones available.
There is a way to make DLM keep its journal when the node is stopped and
use it when it starts ? or a way to make it keep it's active allocation
forcing the rejoining node to look for another unoccupied journal
without pushing away the nodes that occupy the journal it allocates for
the node that rejoins?
Did you try my suggestion from the previous e-mail?
'Probably you also do not need 'ordered="true"' for your DLM clone?
Knowing what is DLM, it does not need ordering, its instances may be
safely started in the parallel.'
Best regards
On Mon, Aug 28, 2017 at 6:04 PM, Octavian Ciobanu
<coctavian1...@gmail.com <mailto:coctavian1...@gmail.com>> wrote:
Thank you for info.
Looking over the output of the crm_simulate I've noticed the
"notice" messages and with the help of the debug mode I've found
this sequence in log
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
native_assign_node: Assigning node01 to DLM:2
Aug 28 16:23:19 [13802] node03 crm_simulate: notice:
color_instance: Pre-allocation failed: got node01 instead of node02
Aug 28 16:23:19 [13802] node03 crm_simulate: info:
native_deallocate: Deallocating DLM:2 from core01
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
native_assign_node: Assigning core01 to DLM:3
Aug 28 16:23:19 [13802] node03 crm_simulate: notice:
color_instance: Pre-allocation failed: got node01 instead of node03
Aug 28 16:23:19 [13802] node03 crm_simulate: info:
native_deallocate: Deallocating DLM:3 from core01
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
native_assign_node: Assigning core01 to DLM:0
Aug 28 16:23:19 [13802] node03 crm_simulate: info:
rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone
Aug 28 16:23:19 [13802] node03 crm_simulate: info:
rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
native_assign_node: Assigning node03 to DLM:2
Aug 28 16:23:19 [13802] node03 crm_simulate: info:
rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone
Aug 28 16:23:19 [13802] node03 crm_simulate: info:
rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
native_assign_node: Assigning node02 to DLM:3
This suggests that the restarted node attempts to occupy a dlm
journal that is allocated to another note and by doing so triggers a
chain reaction leading to all resources being restarted on all nodes.
I will try a different approach (based on your suggestions) on
starting DLM, iSCSI and Mount resources and see if this changes
anything.
If based on the log you have any suggestions they are welcome.
Thank you again for the help.
On Mon, Aug 28, 2017 at 3:53 PM, Vladislav Bogdanov
<bub...@hoster-ok.com <mailto:bub...@hoster-ok.com>> wrote:
28.08.2017 14:03, Octavian Ciobanu wrote:
Hey Vladislav,
Thank you for the info. I've tried you suggestions but the
behavior is
still the same. When an offline/standby node rejoins the
cluster all the
resources are first stopped and then started. I've added the
changes
I've made, see below in reply message, next to your suggestions.
Logs on DC (node where you see logs from the pengine process)
should contain references to pe-input-XX.bz2 files. Something
like "notice: Calculated transition XXXX, saving inputs in
/var/lib/pacemaker/pengine/pe-input-XX.bz2"
Locate one for which Stop actions occur.
You can replay them with 'crm_simulate -S -x
/var/lib/pacemaker/pengine/pe-input-XX.bz2' to see if that is
the correct one (look in the middle of output).
After that you may add some debugging:
PCMK_debug=yes PCMK_logfile=./pcmk.log crm_simulate -S -x
/var/lib/pacemaker/pengine/pe-input-XX.bz2
That will produce a big file with all debugging messages enabled.
Try to locate a reason for restarts there.
Best,
Vladislav
Also please look inline (may be info there will be enough so you
won't need to debug).
Once again thank you for info.
Best regards.
Octavian Ciobanu
On Sat, Aug 26, 2017 at 8:17 PM, Vladislav Bogdanov
<bub...@hoster-ok.com <mailto:bub...@hoster-ok.com>
<mailto:bub...@hoster-ok.com <mailto:bub...@hoster-ok.com>>>
wrote:
26.08.2017 19 <tel:26.08.2017%2019>
<tel:26.08.2017%2019>:36, Octavian Ciobanu wrote:
Thank you for your reply.
There is no reason to set location for the
resources, I think,
because
all the resources are set with clone options so they
are started
on all
nodes at the same time.
You still need to colocate "upper" resources with their
dependencies. Otherwise pacemaker will try to start them
even if
their dependencies fail. Order without colocation has
very limited
use (usually when resources may run on different nodes).
For clones
that is even more exotic.
I've added collocation
pcs constraint colocation add iSCSI1-clone with DLM-clone
pcs constraint colocation add iSCSI2-clone with DLM-clone
pcs constraint colocation add iSCSI3-clone with DLM-clone
pcs constraint colocation add Mount1-clone with iSCSI1-clone
pcs constraint colocation add Mount2-clone with iSCSI2-clone
pcs constraint colocation add Mount4-clone with iSCSI3-clone
The result is the same ... all clones are first stopped and
then started
beginning with DLM resource and ending with the Mount ones.
Yep, that was not meant to fix your problem. Just to prevent
future issues.
For you original question: ensure you have
interleave=true set for
all your clones. You seem to miss it for iSCSI ones.
interleave=false (default) is for different uses (when upper
resources require all clone instances to be up).
Modified iSCSI resources and added interleave="true" and
still no change
in behavior.
Weird... Probably you also do not need 'ordered="true"' for your
DLM clone? Knowing what is DLM, it does not need ordering, its
instances may be safely started in the parallel.
Also, just a minor note, iSCSI resources do not actually
depend on
dlm, mounts should depend on it.
I know but the mount resource must know when the iSCSI
resource to whom
is connected is started so the only solution I've seen was
to place DLM
before iSCSI and then Mount. If there is another solution, a
proper way
to do it, please can you give a reference or a place from
where to read
on how to do it ?
You would want to colocate (and order) mount with both DLM and
iSCSI. Multiple colocations/orders for the same resource are
allowed.
For mount you need DLM running and iSCSI disk connected. But you
actually do not need DLM to connect iSCSI disk (so DLM and iSCSI
resources may start in the parallel).
And when it comes to stickiness I forgot to
mention that but it set to 200. and also I have stonith
configured to
use vmware esxi.
Best regards
Octavian Ciobanu
On Sat, Aug 26, 2017 at 6:16 PM, John Keates
<j...@keates.nl <mailto:j...@keates.nl>
<mailto:j...@keates.nl <mailto:j...@keates.nl>>
<mailto:j...@keates.nl <mailto:j...@keates.nl>
<mailto:j...@keates.nl <mailto:j...@keates.nl>>>> wrote:
While I am by no means a CRM/Pacemaker expert, I
only see the
resource primitives and the order constraints.
Wouldn’t you need
location and/or colocation as well as stickiness
settings to
prevent
this from happening? What I think it might be
doing is
seeing the
new node, then trying to move the resources (but
not finding
it a
suitable target) and then moving them back where
they came
from, but
fast enough for you to only see it as a restart.
If you crm_resource -P, it should also restart all
resources, but
put them in the preferred spot. If they end up
in the same
place,
you probably didn’t put and weighing in the
config or have
stickiness set to INF.
Kind regards,
John Keates
On 26 Aug 2017, at 14:23, Octavian Ciobanu
<coctavian1...@gmail.com
<mailto:coctavian1...@gmail.com>
<mailto:coctavian1...@gmail.com
<mailto:coctavian1...@gmail.com>>
<mailto:coctavian1...@gmail.com
<mailto:coctavian1...@gmail.com>
<mailto:coctavian1...@gmail.com
<mailto:coctavian1...@gmail.com>>>> wrote:
Hello all,
While playing with cluster configuration I
noticed a strange
behavior. If I stop/standby cluster services
on one node and
reboot it, when it joins the cluster all the
resources
that were
started and working on active nodes get
stopped and
restarted.
My testing configuration is based on 4
nodes. One node is a
storage node that makes 3 iSCSI targets
available for
the other
nodes to use,it is not configured to join
cluster, and
three nodes
that are configured in a cluster using the
following
commands.
pcs resource create DLM
ocf:pacemaker:controld op monitor
interval="60" on-fail="fence" clone meta
clone-max="3"
clone-node-max="1" interleave="true"
ordered="true"
pcs resource create iSCSI1 ocf:heartbeat:iscsi
portal="10.0.0.1:3260 <http://10.0.0.1:3260>
<http://10.0.0.1:3260>
<http://10.0.0.1:3260/>"
target="iqn.2017-08.example.com
<http://iqn.2017-08.example.com>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>>>:tgt1" op start interval="0"
timeout="20" op stop interval="0"
timeout="20" op monitor
interval="120" timeout="30" clone meta
clone-max="3"
clone-node-max="1"
pcs resource create iSCSI2 ocf:heartbeat:iscsi
portal="10.0.0.1:3260 <http://10.0.0.1:3260>
<http://10.0.0.1:3260>
<http://10.0.0.1:3260/>"
target="iqn.2017-08.example.com
<http://iqn.2017-08.example.com>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>>>:tgt2" op start interval="0"
timeout="20" op stop interval="0"
timeout="20" op monitor
interval="120" timeout="30" clone meta
clone-max="3"
clone-node-max="1"
pcs resource create iSCSI3 ocf:heartbeat:iscsi
portal="10.0.0.1:3260 <http://10.0.0.1:3260>
<http://10.0.0.1:3260>
<http://10.0.0.1:3260/>"
target="iqn.2017-08.example.com
<http://iqn.2017-08.example.com>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>
<http://iqn.2017-08.example.com
<http://iqn.2017-08.example.com>>>:tgt3" op start interval="0"
timeout="20" op stop interval="0"
timeout="20" op monitor
interval="120" timeout="30" clone meta
clone-max="3"
clone-node-max="1"
pcs resource create Mount1
ocf:heartbeat:Filesystem
device="/dev/disk/by-label/MyCluster:Data1"
directory="/mnt/data1"
fstype="gfs2"
options="noatime,nodiratime,rw" op monitor
interval="90" on-fail="fence" clone meta
clone-max="3"
clone-node-max="1" interleave="true"
pcs resource create Mount2
ocf:heartbeat:Filesystem
device="/dev/disk/by-label/MyCluster:Data2"
directory="/mnt/data2"
fstype="gfs2"
options="noatime,nodiratime,rw" op monitor
interval="90" on-fail="fence" clone meta
clone-max="3"
clone-node-max="1" interleave="true"
pcs resource create Mount3
ocf:heartbeat:Filesystem
device="/dev/disk/by-label/MyCluster:Data3"
directory="/mnt/data3"
fstype="gfs2"
options="noatime,nodiratime,rw" op monitor
interval="90" on-fail="fence" clone meta
clone-max="3"
clone-node-max="1" interleave="true"
pcs constraint order DLM-clone then iSCSI1-clone
pcs constraint order DLM-clone then iSCSI2-clone
pcs constraint order DLM-clone then iSCSI3-clone
pcs constraint order iSCSI1-clone then
Mount1-clone
pcs constraint order iSCSI2-clone then
Mount2-clone
pcs constraint order iSCSI3-clone then
Mount3-clone
If I issue the command "pcs cluster standby
node1" or
"pcs cluster
stop" on node 1 and after that I reboot the
node. When
the node
gets back online (unstandby if it was put in
standby
mode) all the
"MountX" resources get stopped on node 3 and
4 and
started again.
Can anyone help me figure out where and what
is the
mistake in my
configuration as I would like to keep the
started
resources on
active nodes (avoid stop and start of
resources)?
Thank you in advance
Octavian Ciobanu
_______________________________________________
Users mailing list: Users@clusterlabs.org
<mailto:Users@clusterlabs.org>
<mailto:Users@clusterlabs.org
<mailto:Users@clusterlabs.org>>
<mailto:Users@clusterlabs.org
<mailto:Users@clusterlabs.org>
<mailto:Users@clusterlabs.org
<mailto:Users@clusterlabs.org>>>
http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>>>
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>>
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
<mailto:Users@clusterlabs.org>
<mailto:Users@clusterlabs.org
<mailto:Users@clusterlabs.org>>
<mailto:Users@clusterlabs.org <mailto:Users@clusterlabs.org>
<mailto:Users@clusterlabs.org
<mailto:Users@clusterlabs.org>>>
http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>>>
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>>
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
<mailto:Users@clusterlabs.org>
<mailto:Users@clusterlabs.org
<mailto:Users@clusterlabs.org>>
http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>>
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
<mailto:Users@clusterlabs.org> <mailto:Users@clusterlabs.org
<mailto:Users@clusterlabs.org>>
http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
<http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>>
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
<mailto:Users@clusterlabs.org>
http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
<mailto:Users@clusterlabs.org>
http://lists.clusterlabs.org/mailman/listinfo/users
<http://lists.clusterlabs.org/mailman/listinfo/users>
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
