Re: [ClusterLabs] Is "Process pause detected" triggered too easily?
Hi Jan, On Tue, 3 Oct 2017, Jan Friesse wrote: > > I hope this makes sense! :) > > I would still have some questions :) but that is really not related to > the problem you have. Questions are welcome! I am new to this stack, so there is certainly room for learning and for improvement. > My personal favorite is consensus timeout. Because you've set (and I > must say according to doc correctly) consensus timeout to 3600 (= 1.2 * > token). Problem is, that result token timeout is not 3000, but with 5 > nodes it is actually 3000 (base token) + (no_nodes - 2) * 650 ms = 4950 > (as you can check by observing runtime.config.totem.token key). So it > may make sense to set consensus timeout to ~6000. Could you clarify the formula for me? I don't see how "- 2" and "650" map to this configuration. And I suppose that on our bigger system (20+5 servers) we need to greatly increase the consensus timeout. Overall, tuning the timeouts seems related to be Black Magic. ;) I liked the idea suggested in an old thread that there would be a spreadsheet (or even just plain formulas) exposing the relation between the various knobs. One thing I wonder is: would it make sense to annotate the state machine diagram in the Totem paper (page 15 of http://www.cs.jhu.edu/~yairamir/tocs.ps.gz) with those tunables? Assuming the paper still reflects the behavior of the current code. > This doesn't change the fact that "bug" is reproducible even with > "correct" consensus, so I will continue working on this issue. Great! Thanks for taking the time to investigate. Cheers, JM -- saff...@gmail.com ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Is "Process pause detected" triggered too easily?
Jean, On Mon, 2 Oct 2017, Jan Friesse wrote: We had one problem on a real deployment of DLM+corosync (5 voters and 20 non-voters, with dlm on those 20, for a specific application that uses What you mean by voters and non-voters? There is 25 nodes in total and each of them is running corosync? Yes, there are 25 servers running corosync: - 5 are configured to have one vote for quorum, on these servers corosync serves no other purpose - 20 have zero vote for quorum, and these servers also run DLM and the application that uses DLM The intent with this configuration is: - to avoid split brain in case of network partition: application servers must be in the same partition as the quorum majority (so, 3 of the 5 "voters") to carry on their operations - to allow independent failure of any number of application servers I hope this makes sense! :) I would still have some questions :) but that is really not related to the problem you have. libdlm). On a reboot of one server running just corosync (which thus did NOT run dlm), a large number of other servers got briefly evicted from the This is kind of weird. AFAIK DLM is joining to CPG group and using CPG membership. So if DLM was not running on the node then other nodes joined to DLM CPG group should not even notice its leave. Indeed, but we saw "Process pause detected" on all servers, and corosync temporarily formed an operational cluster excluding most of the "non-voters" (those with zero quorum vote). Then most servers joined back, but then DLM complained about the "stateful merge". Yep, when this was on all servers it's a huge problem and it explains a lot. What you mean by zero vote? You mean DLM vote or corosync number of votes (related to quorum)? I mean the vote in the corosync quorum, I'm not aware of anything like that with DLM (or maybe you could think of the per-server weight when one manually defines servers that master locks in a lock space, but we don't use that). Got it. I've tried to reproduce the problem and I was not successful with 3 nodes cluster using more or less default config (not changing join/consensus/...). I'll try 5 nodes possibly with totem values and see if problem appears. I've tried again today, and first with just 3 servers (VMs), using the same config I sent earlier (which has 3 nodes with 1 vote, 2 nodes with 0 vote), I could no longer reproduce either. Then I spawned 2 more VMs and had them join the existing 3-node cluster (those I added were the 2 servers with 0 vote), and then I saw the "Process pause ..." log line. And now I have stopped the last 2 servers, and I am back to just 3, and I keep seeing that log line. If you're still curious and if that's useful, I can try to reproduce on a set of VMs where I could give you full ssh access. So good news is that I was able to reproduce it. Even better news is, that I was able to reproduce it even without changing join/consensus/... parameters. What's even better that with parameters changed it's becoming much easier to reproduce the problem. So in theory if I will be able to identify that parameter, it may make sense to increase/decrease that parameter close to infinity/0 and debugging should then become easy. My personal favorite is consensus timeout. Because you've set (and I must say according to doc correctly) consensus timeout to 3600 (= 1.2 * token). Problem is, that result token timeout is not 3000, but with 5 nodes it is actually 3000 (base token) + (no_nodes - 2) * 650 ms = 4950 (as you can check by observing runtime.config.totem.token key). So it may make sense to set consensus timeout to ~6000. This doesn't change the fact that "bug" is reproducible even with "correct" consensus, so I will continue working on this issue. Honza Thanks! Cheers, JM ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] what does cluster do when 'resourceA with resourceB' happens
On Tue, 2017-10-03 at 11:53 +0100, lejeczek wrote: > hi > > I'm reading "An A-Z guide to Pacemaker's Configurations > Options" and in there it read: > "... > So when you are creating colocation constraints, it is > important to consider whether you should > colocate A with B, or B with A. > Another thing to keep in mind is that, assuming A is > colocated with B, the cluster will take into account > A’s preferences when deciding which node to choose for B. > ..." > > I have a healthy cluster, three nodes, a five resources at > given time running on node1. > Then I create a LVM resource which cluster decides to put on > node2. > Then I do: > > $ pcs constraint colocation add resourceA(running@node1) > with aLocalStorage6(newly created, running@node2) > > I cluster moves all the resources to node2. > > And I say whaaat? Something, guide+reality, does not ad up, > right? That colocation constraint says "place aLocalStorage6 somewhere, then place resourceA with it", so it makes sense resourceA moves there. If everything else moves there, too, there must be something else in the configuration telling it to. (Other colocation constraints maybe?) ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Restarting a failed resource on same node
On Mon, 2017-10-02 at 12:32 -0700, Paolo Zarpellon wrote: > Hi, > on a basic 2-node cluster, I have a master-slave resource where > master runs on a node and slave on the other one. If I kill the slave > resource, the resource status goes to "stopped". > Similarly, if I kill the the master resource, the slave one is > promoted to master but the failed one does not restart as slave. > Is there a way to restart failing resources on the same node they > were running? > Thank you in advance. > Regards, > Paolo Restarting on the same node is the default behavior -- something must be blocking it. For example, check your migration-threshold (if restarting fails this many times, it has nowhere to go and will stop). ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] what does cluster do when 'resourceA with resourceB' happens
hi I'm reading "An A-Z guide to Pacemaker's Configurations Options" and in there it read: "... So when you are creating colocation constraints, it is important to consider whether you should colocate A with B, or B with A. Another thing to keep in mind is that, assuming A is colocated with B, the cluster will take into account A’s preferences when deciding which node to choose for B. ..." I have a healthy cluster, three nodes, a five resources at given time running on node1. Then I create a LVM resource which cluster decides to put on node2. Then I do: $ pcs constraint colocation add resourceA(running@node1) with aLocalStorage6(newly created, running@node2) I cluster moves all the resources to node2. And I say whaaat? Something, guide+reality, does not ad up, right? . ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
