Re: [ClusterLabs] 2-Node Cluster Pointless?

[Dimitri Maziuk]

On 04/17/2017 01:50 PM, Digimer wrote:

> That sounds like a use-case where a full HA cluster is overkill already.
> In any case, it would be a tiny fraction of installs and would be
> tangential to the 2v3+ node debate that this thread started with.

A web server with a "master/development" host: postgres 9.x in hot
standby streaming replication, static contents is pushed with zfs
snapshots, the only thing you need to "cluster" is floating ip.

Yes, this works perfectly fine with haresources and a couple of
two-liner mon scripts. And nagios on the "master".

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Dimitri Maziuk]

On 04/17/2017 01:26 PM, Eric Robinson wrote:

> I'm guessing the surviving node broadcasts a gratuitous arp reply. 

You have to fence one node by physically removing power at which point
you don't have a split brain anymore. In a split brain scenario: two
nodes are up, assuming they both send gratuitous arp, one or the other
will win on the router.

If it's the "wrong" one your customer hits the node that can't provide
the service and gets a 404. If it's the right one, the service is still
available despite the split brain.

If you can't ensure your fencing kills the "wrong" node, then the only
practical difference between that and "proper" fencing with split brain
detection and trimmings is the cost of the latter.

Send an SMS to the sysadmin and have them figure it out. Better still,
pay an extra nickel and buy servers that don't go titsup in the first place.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Digimer]

On 17/04/17 02:21 PM, Dimitri Maziuk wrote:
> On 04/17/2017 01:12 PM, Ian wrote:
>>
>> No, I don't understand how it's relevant to the specific topic of avoiding
>> split-brains, either.
> 
>>> Take a simple example of shared-nothing read-only cluster: all you need
>>> to know is that the daemon is bound to '*' and the floating ip is bound
>>> to eth0.
> 
> In shred-nothing cluster "split brain" means whichever MAC address is in
> ARP cache of the border router is the one that gets the traffic. How
> does the existing code figure this one out?

That sounds like a use-case where a full HA cluster is overkill already.
In any case, it would be a tiny fraction of installs and would be
tangential to the 2v3+ node debate that this thread started with.

-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein’s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Eric Robinson]

> In shred-nothing cluster "split brain" means whichever MAC address 
> is in ARP cache of the border router is the one that gets the traffic. 
> How does the existing code figure this one out?

I'm guessing the surviving node broadcasts a gratuitous arp reply. 

--
Eric Robinson

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Eric Robinson]

> This isn't the first time this has come up, so I decided 
> to elaborate on this email by writing an article on the topic.

> It's a first-draft so there are likely spelling/grammar 
> mistakes. However, the body is done.

> https://www.alteeve.com/w/The_2-Node_Myth

It looks like my question was well-timed, as it served as a catalyst for you to 
write the article. Thanks much, I am working through it now and will doubtless 
have some questions and comments. Before I say anything more, I want to do some 
testing in my lab to make sure I have my thoughts collected. 

--
Eric Robinson



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Dimitri Maziuk]

On 04/17/2017 01:12 PM, Ian wrote:
> 
> No, I don't understand how it's relevant to the specific topic of avoiding
> split-brains, either.

>> Take a simple example of shared-nothing read-only cluster: all you need
>> to know is that the daemon is bound to '*' and the floating ip is bound
>> to eth0.

In shred-nothing cluster "split brain" means whichever MAC address is in
ARP cache of the border router is the one that gets the traffic. How
does the existing code figure this one out?

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Digimer]

On 17/04/17 02:12 PM, Ian wrote:
>>  maybe I need another coffee?
> 
> No, I don't understand how it's relevant to the specific topic of
> avoiding split-brains, either.  I suppose it's possible that I also need
> coffee.

I'm trying to make the connection still, honestly, but I am still lost...

> On Mon, Apr 17, 2017 at 1:45 PM, Dimitri Maziuk  > wrote:
> 
> On 04/17/2017 11:58 AM, Digimer wrote:
> 
> > ... Unless I am misunderstanding, your comment is related to
> > serviceability of clusters in general. I'm failing to link the contexts.
> > Similarly, I'm not sure how this relates to "new" vs. "best"...
> 
> You can't know if *a* customer can access the service it provides. You
> can know if the service access point is up and connected to the server
> process.
> 
> Take a simple example of shared-nothing read-only cluster: all you need
> to know is that the daemon is bound to '*' and the floating ip is bound
> to eth0.
> 
> This is the "best" in that it's simple, stupid, does all you you
> need/can do and nothing that doesn't make your cluster run any "better".
>  It's also very unexciting.
> 
> --
> Dimitri Maziuk
> Programmer/sysadmin
> BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org 
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> 
> Project Home: http://www.clusterlabs.org
> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> 
> Bugs: http://bugs.clusterlabs.org
> 
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 


-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein’s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Ian]

>  maybe I need another coffee?

No, I don't understand how it's relevant to the specific topic of avoiding
split-brains, either.  I suppose it's possible that I also need coffee.

On Mon, Apr 17, 2017 at 1:45 PM, Dimitri Maziuk 
wrote:

> On 04/17/2017 11:58 AM, Digimer wrote:
>
> > ... Unless I am misunderstanding, your comment is related to
> > serviceability of clusters in general. I'm failing to link the contexts.
> > Similarly, I'm not sure how this relates to "new" vs. "best"...
>
> You can't know if *a* customer can access the service it provides. You
> can know if the service access point is up and connected to the server
> process.
>
> Take a simple example of shared-nothing read-only cluster: all you need
> to know is that the daemon is bound to '*' and the floating ip is bound
> to eth0.
>
> This is the "best" in that it's simple, stupid, does all you you
> need/can do and nothing that doesn't make your cluster run any "better".
>  It's also very unexciting.
>
> --
> Dimitri Maziuk
> Programmer/sysadmin
> BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
>
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
>
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Dimitri Maziuk]

On 04/17/2017 11:58 AM, Digimer wrote:

> ... Unless I am misunderstanding, your comment is related to
> serviceability of clusters in general. I'm failing to link the contexts.
> Similarly, I'm not sure how this relates to "new" vs. "best"... 

You can't know if *a* customer can access the service it provides. You
can know if the service access point is up and connected to the server
process.

Take a simple example of shared-nothing read-only cluster: all you need
to know is that the daemon is bound to '*' and the floating ip is bound
to eth0.

This is the "best" in that it's simple, stupid, does all you you
need/can do and nothing that doesn't make your cluster run any "better".
 It's also very unexciting.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Dimitri Maziuk]

On 04/17/2017 11:03 AM, Digimer wrote:
> On 17/04/17 11:15 AM, Dmitri Maziuk wrote:

>> What you want to know is whether the customer can access the service.
>> Adding more nodes does not answer that question, but since Andrew is
>> writing cluster software, not providing services, that's not his problem.

> Can you elaborate? I'm not following your point/concern here...
> Availability is all about making sure customers/users can access their
> services.

Which part? The one where you don't know unless your monitor is on the
customer's machine? Or the part where we all like to write something
new, clever, and exciting? Which is usually not the same as the best we
can do for the actual problem at hand?

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Digimer]

On 17/04/17 11:15 AM, Dmitri Maziuk wrote:
> On 2017-04-16 15:04, Eric Robinson wrote:
> 
>>> On 16/04/17 01:53 PM, Eric Robinson wrote:
 I was reading in "Clusters from Scratch" where Beekhof states, "Some
>>> would argue that two-node clusters are always pointless, but that is an
>>> argument for another time."
> 
> What you want to know is whether the customer can access the service.
> Adding more nodes does not answer that question, but since Andrew is
> writing cluster software, not providing services, that's not his problem.
> 
> Dima

Can you elaborate? I'm not following your point/concern here...
Availability is all about making sure customers/users can access their
services.

-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein’s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How to force remove a cluster node?

[Ken Gaillot]

On 04/13/2017 01:11 PM, Scott Greenlese wrote:
> Hi,
> 
> I need to remove some nodes from my existing pacemaker cluster which are
> currently unbootable / unreachable.
> 
> Referenced
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-clusternodemanage-HAAR.html#s2-noderemove-HAAR
> 
> *4.4.4. Removing Cluster Nodes*
> The following command shuts down the specified node and removes it from
> the cluster configuration file, corosync.conf, on all of the other nodes
> in the cluster. For information on removing all information about the
> cluster from the cluster nodes entirely, thereby destroying the cluster
> permanently, refer to _Section 4.6, “Removing the Cluster
> Configuration”_
> .
> 
> pcs cluster node remove /node/
> 
> I ran the command with the cluster active on 3 of the 5 available
> cluster nodes (with quorum). The command fails with:
> 
> [root@zs90KP VD]# date;*pcs cluster node remove zs93kjpcs1*
> Thu Apr 13 13:40:59 EDT 2017
> *Error: pcsd is not running on zs93kjpcs1*
> 
> 
> The node was not removed:
> 
> [root@zs90KP VD]# pcs status |less
> Cluster name: test_cluster_2
> Last updated: Thu Apr 13 14:08:15 2017 Last change: Wed Apr 12 16:40:26
> 2017 by root via cibadmin on zs93KLpcs1
> Stack: corosync
> Current DC: zs90kppcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) -
> partition with quorum
> 45 nodes and 180 resources configured
> 
> Node zs95KLpcs1: UNCLEAN (offline)
> Online: [ zs90kppcs1 zs93KLpcs1 zs95kjpcs1 ]
> *OFFLINE: [ zs93kjpcs1 ]*
> 
> 
> Is there a way to force remove a node that's no longer bootable? If not,
> what's the procedure for removing a rogue cluster node?
> 
> Thank you...
> 
> Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie, N.Y.
> INTERNET: swgre...@us.ibm.com

Yes, the pcs command is just a convenient shorthand for a series of
commands. You want to ensure pacemaker and corosync are stopped on the
node to be removed (in the general case, obviously already done in this
case), remove the node from corosync.conf and restart corosync on all
other nodes, then run "crm_node -R " on any one active node.


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Why shouldn't one store resource configuration in the CIB?

[Ken Gaillot]

On 04/13/2017 11:11 AM, Ferenc Wágner wrote:
> Hi,
> 
> I encountered several (old) statements on various forums along the lines
> of: "the CIB is not a transactional database and shouldn't be used as
> one" or "resource parameters should only uniquely identify a resource,
> not configure it" and "the CIB was not designed to be a configuration
> database but people still use it that way".  Sorry if I misquote these,
> I go by my memories now, I failed to dig up the links by a quick try.
> 
> Well, I've been feeling guilty in the above offenses for years, but it
> worked out pretty well that way which helped to suppress these warnings
> in the back of my head.  Still, I'm curious: what's the reason for these
> warnings, what are the dangers of "abusing" the CIB this way?
> /var/lib/pacemaker/cib/cib.xml is 336 kB with 6 nodes and 155 resources
> configured.  Old Pacemaker versions required tuning PCMK_ipc_buffer to
> handle this, but even the default is big enough nowadays (128 kB after
> compression, I guess).
> 
> Am I walking on thin ice?  What should I look out for?

That's a good question. Certainly, there is some configuration
information in most resource definitions, so it's more a matter of degree.

The main concerns I can think of are:

1. Size: Increasing the CIB size increases the I/O, CPU and networking
overhead of the cluster (and if it crosses the compression threshold,
significantly). It also marginally increases the time it takes the
policy engine to calculate a new state, which slows recovery.

2. Consistency: Clusters can become partitioned. If changes are made on
one or more partitions during the separation, the changes won't be
reflected on all nodes until the partition heals, at which time the
cluster will reconcile them, potentially losing one side's changes. I
suppose this isn't qualitatively different from using a separate
configuration file, but those tend to be more static, and failure to
modify all copies would be more obvious when doing them individually
rather than issuing a single cluster command.

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 2-Node Cluster Pointless?

[Dmitri Maziuk]

On 2017-04-16 15:04, Eric Robinson wrote:


On 16/04/17 01:53 PM, Eric Robinson wrote:

I was reading in "Clusters from Scratch" where Beekhof states, "Some

would argue that two-node clusters are always pointless, but that is an
argument for another time."


What you want to know is whether the customer can access the service. 
Adding more nodes does not answer that question, but since Andrew is 
writing cluster software, not providing services, that's not his problem.


Dima


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] nodes ID assignment issue

[Ken Gaillot]

On 04/13/2017 10:40 AM, Radoslaw Garbacz wrote:
> Hi,
> 
> I have a question regarding building CIB nodes scope and specifically
> assignment to node IDs.
> It seems like the preexisting scope is not honored and nodes can get
> replaced based on check-in order.
> 
> I pre-create the nodes scope because it is faster, then setting
> parameters for all the nodes later (when the number of nodes is large).
> 
> From the listings below, one can see that node with ID=1 was replaced
> with another node (uname), however not the options. This situation
> causes problems when resource assignment is based on rules involving
> node options.
> 
> Is there a way to prevent this rearrangement of 'uname', if not whether
> there is a way to make the options follow 'uname', or maybe the problem
> is somewhere else - corosync configuration perhaps?
> Is the corosync 'nodeid' enforced to be also CIB node 'id'?

Hi,

Yes, for cluster nodes, pacemaker gets the node id from the messaging
layer (corosync). For remote nodes, id and uname are always identical.

> 
> 
> Thanks in advance,
> 
> 
> Below is CIB committed before nodes check-in:
> 
> 
>   
> 
>   
>   
>   
> 
>   
>   
> 
>name="STATE"/>
>   
>name="Primary"/>
> 
>   
>   
> 
>name="STATE"/>
>   
>name="Primary"/>
> 
>   
>   
> 
>name="STATE"/>
>   
>name="Primary"/>
> 
>   
>   
> 
>   
>   
>   
> 
>   
> 
> 
> 
> 
> And automatic changes after nodes check-in:
> 
> 
>   
> 
>   
>   
>   
> 
>   
>   
> 
>name="STATE"/>
>   
>name="Primary"/>
> 
>   
>   
> 
>name="STATE"/>
>   
>name="Primary"/>
> 
>   
>   
> 
>name="STATE"/>
>   
>name="Primary"/>
> 
>   
>   
> 
>   
>   
>   
> 
>   
> 
> 
> 
> 
> -- 
> Best Regards,
> 
> Radoslaw Garbacz
> XtremeData Incorporated

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] KVM virtualdomain - stopped

[Ken Gaillot]

On 04/13/2017 03:01 AM, Jaco van Niekerk wrote:
> 
> Hi
> 
> I am having endless problems with ocf::heartbeat:VirtualDomain when
> failing over to second node. The virtualdomain goes into a stopped state
> 
> virtdom_compact (ocf::heartbeat:VirtualDomain): Stopped
> 
> * virtdom_compact_start_0 on node2.kvm.bitco.co.za 'unknown error' (1):
> call=93, status=complete, exitreason='Failed to start virtual domain
> compact.',
> last-rc-change='Thu Apr 13 09:11:16 2017', queued=0ms, exec=369ms
> 
> I then aren't able to get it started without deleting the resource and
> adding it again:
> 
> pcs resource create virtdom_compact ocf:heartbeat:VirtualDomain
> config=/etc/libvirt/qemu/compact.xml meta allow-migrate=true op monitor
> interval="30"
> 
> Looking at virsh list --all
> virsh list --all
> Id Name State
> 
> 
> It doesn't seam like ocf:heartbeat:VirtualDomain is able to define the
> domain and thus the command can't start the domain:
> 
> virsh start compact
> error: failed to get domain 'compact'
> error: Domain not found: no domain with matching name 'compact'
> 
> Am I missing something in my configuration:
> pcs resource create my_FS ocf:heartbeat:Filesystem params
> device=/dev/sdc1 directory=/images fstype=xfs
> pcs resource create my_VIP ocf:heartbeat:IPaddr2 ip=192.168.99.10
> cidr_netmask=22 op monitor interval=10s
> pcs resource create virtdom_compact ocf:heartbeat:VirtualDomain
> config=/etc/libvirt/qemu/compact.xml meta allow-migrate=true op monitor

^^^ Make sure that config file is available on all nodes. It's a good
idea to try starting the VM outside the cluster (e.g. with virsh) on
each node, before putting it under cluster control.

> interval="30"
> 
> Regards
> 
> 
> * Jaco van Niekerk*
> 
> * Solutions Architect*
> 
>   
>
> 
>  *T:* 087 135  | Ext: 2102
> 
>
> 
>  *E:* j...@bitco.co.za 

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org