Re: [Openais] corosync and qdevice
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com
On 3/20/2013 6:26 PM, eXeC001er wrote:
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com
mailto:fdini...@redhat.com
On 3/20/2013 5:27 PM, eXeC001er wrote:
The first Q:
According to the tests that are part of corosync-sources i
think
that i can:
1. create a daemon that register a QDEVICE and will notify
corosync
about the device (votequorum_qdevice_poll()).
2. implement a master/slave logic and if the qdevice on a
node
wins then
i call votequorum_qdevice_master_wins() on the node and
corosync
notify
another nodes about this, so i can say that the node is
MASTER.
There is no votequorum_qdevice_master_wins() call... where did
you
find it?
I am researching corosync-2.3.0 sources.
whoops.. i wrote it and forgot about it. getting old is bad :)
No it´s a bit more complicated than that.
corosync starts and load the config
later on qdeviced starts and read the config
qdeviced detects that it has to run in master_wins config:
call votequorum_qdevice_master_wins(..., 1);
that calls set a flags for the node and makes sure that the feature
is
enabled internally.
I thought about a different scenario:
master_wins and cast_vote are different flags and they are used in
different cases.
1. uses only cast_vote and the flag can be used to decide that on a
node everything fine and the node is a member of cluster (the cluster
does not have master/slave)
for example in a cluster (3 nodes) i have several qdevices on each node:
storage-qdevice and client-network-qdevice
The API does not support multiple qdevices. This kind of implementation,
where you need to poll multiple targets, can be multiplexed/proxy´d by
the votequorum consumer.
qdevice does not need to know how much to vote. That´s votequorum
problem. How the qdevice implementation handles internal voting/scoring
it´s qdevice implementation problem.
config:
each qdevice has 2 votes
each node has 1 vote
expected votes = 7 votes (1 own vote + 1 vote from anoter node + 2 votes
from each qdevice)
that won´t work. votequorum accepts only one value for qdevice votes.
2. master_wins and cast_vote are used.
in this case cast_vote will work as in case 1 and master_wins will
control master/slave
The need to propagate the value for master_win and it´s status is to allow:
node1 node2 node3 node4
qdevice is master on node3 for example
in case of a 50%/50% split you have:
node1 node2 - not quorate
please correct me if i am wrong
at start point on each node qdevice sets in corosync master_wins=1, each
node is a memeber and node3 is master according to decision of qdevice.
so:
- node1, node2 and node4 have 4 votes
- node3 has 5 votes
at some point we a 50%/50% split:
node1 + node2 AND node3 + node4
so:
- node1, node2 have 2 votes
- node4 has 2 votes
- node3 has 3 votes
the quorum is 3 votes.
I see the following condition
755 if ((qdevice_master_wins)
756 (!quorate)
757 (check_qdevice_master() == 1)) {
758 log_printf(LOGSYS_LEVEL_DEBUG, node is quorate as part of
master_wins partition);
759 quorate = 1;
760 }
and
544 static int check_qdevice_master(void)
545 {
546 struct cluster_node *node = NULL;
547 struct list_head *tmp;
548 int found = 0;
549
550 ENTER();
551
552 list_iterate(tmp, cluster_members_list) {
553 node = list_entry(tmp, struct cluster_node, list);
554 if ((node-state == NODESTATE_MEMBER)
555 (node-flags NODE_FLAGS_QDEVICE_MASTER_WINS)
556 (node-flags NODE_FLAGS_QDEVICE_CAST_VOTE)) {
557 found = 1;
558 }
559 }
560
561 LEAVE();
562 return found;
563 }
and i cannot understand how node4 is a part of master_wins partition.
with the old cman quorum:
node3 quorate
node4 not quorate (that´s also why we force master_win in 2 node only)
with the master_win flag set in votequorum:
node3 quorate
node4 quorate (because part of a partition with a master qdevice)
note that node4 does not have the votes from qdevice at this point.
Fabio
___
Openais mailing list
Openais@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/openais
___
Openais mailing list
Openais@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/openais
Re: [Openais] corosync and qdevice
On 3/21/2013 12:18 PM, eXeC001er wrote:
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com
mailto:fdini...@redhat.com
On 3/20/2013 6:26 PM, eXeC001er wrote:
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com
mailto:fdini...@redhat.com
mailto:fdini...@redhat.com mailto:fdini...@redhat.com
On 3/20/2013 5:27 PM, eXeC001er wrote:
The first Q:
According to the tests that are part of
corosync-sources i think
that i can:
1. create a daemon that register a QDEVICE and will notify
corosync
about the device (votequorum_qdevice_poll()).
2. implement a master/slave logic and if the qdevice
on a node
wins then
i call votequorum_qdevice_master_wins() on the node
and corosync
notify
another nodes about this, so i can say that the node
is MASTER.
There is no votequorum_qdevice_master_wins() call...
where did you
find it?
I am researching corosync-2.3.0 sources.
whoops.. i wrote it and forgot about it. getting old is bad :)
No it´s a bit more complicated than that.
corosync starts and load the config
later on qdeviced starts and read the config
qdeviced detects that it has to run in master_wins config:
call votequorum_qdevice_master_wins(..., 1);
that calls set a flags for the node and makes sure that the
feature is
enabled internally.
I thought about a different scenario:
master_wins and cast_vote are different flags and they are used in
different cases.
1. uses only cast_vote and the flag can be used to decide that on a
node everything fine and the node is a member of cluster (the cluster
does not have master/slave)
for example in a cluster (3 nodes) i have several qdevices on each
node:
storage-qdevice and client-network-qdevice
The API does not support multiple qdevices. This kind of implementation,
where you need to poll multiple targets, can be multiplexed/proxy´d by
the votequorum consumer.
qdevice does not need to know how much to vote. That´s votequorum
problem. How the qdevice implementation handles internal voting/scoring
it´s qdevice implementation problem.
config:
each qdevice has 2 votes
each node has 1 vote
expected votes = 7 votes (1 own vote + 1 vote from anoter node + 2
votes
from each qdevice)
that won´t work. votequorum accepts only one value for qdevice
votes.
2. master_wins and cast_vote are used.
in this case cast_vote will work as in case 1 and master_wins will
control master/slave
The need to propagate the value for master_win and it´s status is to
allow:
node1 node2 node3 node4
qdevice is master on node3 for example
in case of a 50%/50% split you have:
node1 node2 - not quorate
please correct me if i am wrong
at start point on each node qdevice sets in corosync master_wins=1,
each node is a memeber and node3 is master according to decision of qdevice.
so:
- node1, node2 and node4 have 4 votes
- node3 has 5 votes
at some point we a 50%/50% split:
node1 + node2 AND node3 + node4
so:
- node1, node2 have 2 votes
- node4 has 2 votes
- node3 has 3 votes
the quorum is 3 votes.
I see the following condition
755 if ((qdevice_master_wins)
756 (!quorate)
757 (check_qdevice_master() == 1)) {
758 log_printf(LOGSYS_LEVEL_DEBUG, node is quorate as part of
master_wins partition);
759 quorate = 1;
760 }
and
544 static int check_qdevice_master(void)
545 {
546 struct cluster_node *node = NULL;
547 struct list_head *tmp;
548 int found = 0;
549
550 ENTER();
551
552 list_iterate(tmp, cluster_members_list) {
553 node = list_entry(tmp, struct cluster_node, list);
554 if ((node-state == NODESTATE_MEMBER)
555 (node-flags NODE_FLAGS_QDEVICE_MASTER_WINS)
556 (node-flags NODE_FLAGS_QDEVICE_CAST_VOTE)) {
557 found = 1;
558 }
559 }
560
561 LEAVE();
562 return found;
563 }
and i cannot understand how node4 is a part of master_wins partition.
node3 and node4 are in the same partition.
node3 is master and has votes from qdevice.
Re: [Openais] corosync and qdevice
2013/3/21 Fabio M. Di Nitto fdini...@redhat.com
On 3/21/2013 12:18 PM, eXeC001er wrote:
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com
mailto:fdini...@redhat.com
On 3/20/2013 6:26 PM, eXeC001er wrote:
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com
mailto:fdini...@redhat.com
mailto:fdini...@redhat.com mailto:fdini...@redhat.com
On 3/20/2013 5:27 PM, eXeC001er wrote:
The first Q:
According to the tests that are part of
corosync-sources i think
that i can:
1. create a daemon that register a QDEVICE and will
notify
corosync
about the device (votequorum_qdevice_poll()).
2. implement a master/slave logic and if the qdevice
on a node
wins then
i call votequorum_qdevice_master_wins() on the node
and corosync
notify
another nodes about this, so i can say that the node
is MASTER.
There is no votequorum_qdevice_master_wins() call...
where did you
find it?
I am researching corosync-2.3.0 sources.
whoops.. i wrote it and forgot about it. getting old is
bad :)
No it´s a bit more complicated than that.
corosync starts and load the config
later on qdeviced starts and read the config
qdeviced detects that it has to run in master_wins config:
call votequorum_qdevice_master_wins(..., 1);
that calls set a flags for the node and makes sure that the
feature is
enabled internally.
I thought about a different scenario:
master_wins and cast_vote are different flags and they are used in
different cases.
1. uses only cast_vote and the flag can be used to decide that
on a
node everything fine and the node is a member of cluster (the
cluster
does not have master/slave)
for example in a cluster (3 nodes) i have several qdevices on each
node:
storage-qdevice and client-network-qdevice
The API does not support multiple qdevices. This kind of
implementation,
where you need to poll multiple targets, can be multiplexed/proxy´d
by
the votequorum consumer.
qdevice does not need to know how much to vote. That´s votequorum
problem. How the qdevice implementation handles internal
voting/scoring
it´s qdevice implementation problem.
config:
each qdevice has 2 votes
each node has 1 vote
expected votes = 7 votes (1 own vote + 1 vote from anoter node + 2
votes
from each qdevice)
that won´t work. votequorum accepts only one value for qdevice
votes.
2. master_wins and cast_vote are used.
in this case cast_vote will work as in case 1 and master_wins
will
control master/slave
The need to propagate the value for master_win and it´s status is to
allow:
node1 node2 node3 node4
qdevice is master on node3 for example
in case of a 50%/50% split you have:
node1 node2 - not quorate
please correct me if i am wrong
at start point on each node qdevice sets in corosync master_wins=1,
each node is a memeber and node3 is master according to decision of
qdevice.
so:
- node1, node2 and node4 have 4 votes
- node3 has 5 votes
at some point we a 50%/50% split:
node1 + node2 AND node3 + node4
so:
- node1, node2 have 2 votes
- node4 has 2 votes
- node3 has 3 votes
the quorum is 3 votes.
I see the following condition
755 if ((qdevice_master_wins)
756 (!quorate)
757 (check_qdevice_master() == 1)) {
758 log_printf(LOGSYS_LEVEL_DEBUG, node is quorate as part of
master_wins partition);
759 quorate = 1;
760 }
and
544 static int check_qdevice_master(void)
545 {
546 struct cluster_node *node = NULL;
547 struct list_head *tmp;
548 int found = 0;
549
550 ENTER();
551
552 list_iterate(tmp, cluster_members_list) {
553 node = list_entry(tmp, struct cluster_node, list);
554 if ((node-state == NODESTATE_MEMBER)
555 (node-flags NODE_FLAGS_QDEVICE_MASTER_WINS)
556 (node-flags NODE_FLAGS_QDEVICE_CAST_VOTE)) {
557 found = 1;
558 }
559 }
560
561 LEAVE();
562 return found;
563 }
and i cannot understand how node4 is a part of master_wins partition.
node3 and node4 are in the same partition.
node3 is master and has votes from qdevice. Those information are known
to all cluster
Re: [Openais] corosync and qdevice
On 3/20/2013 4:31 PM, eXeC001er wrote: Hello. The corosync-2.x has qdevice logic, but i have not found any details about using the logic. that is because the API is not yet supported. The core is done and ready to be used but there are no real consumers. The first Q: According to the tests that are part of corosync-sources i think that i can: 1. create a daemon that register a QDEVICE and will notify corosync about the device (votequorum_qdevice_poll()). 2. implement a master/slave logic and if the qdevice on a node wins then i call votequorum_qdevice_master_wins() on the node and corosync notify another nodes about this, so i can say that the node is MASTER. There is no votequorum_qdevice_master_wins() call... where did you find it? The second Q: As i understand 'CAST_VOTE=1' flag says to corosync that need to consider the votes of a QDEVICE ? No, cast_vote is not the number of votes. A qdevice can be registered and polling, but it´s not casting a vote. That exactly to implement master_win solution. A master is casting a vote, a slave is not casting a vote. Both needs to poll via qdevice_poll. The old concept that a qdevice needs to know how many votes to cast has been removed. votequorum knows internally how many vote a device should have. ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/openais
Re: [Openais] corosync and qdevice
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com
On 3/20/2013 4:31 PM, eXeC001er wrote:
Hello.
The corosync-2.x has qdevice logic, but i have not found any details
about using the logic.
that is because the API is not yet supported. The core is done and ready
to be used but there are no real consumers.
understood.
The first Q:
According to the tests that are part of corosync-sources i think that i
can:
1. create a daemon that register a QDEVICE and will notify corosync
about the device (votequorum_qdevice_poll()).
2. implement a master/slave logic and if the qdevice on a node wins then
i call votequorum_qdevice_master_wins() on the node and corosync notify
another nodes about this, so i can say that the node is MASTER.
There is no votequorum_qdevice_master_wins() call... where did you find it?
I am researching corosync-2.3.0 sources.
test/testvotequorum2.c (
https://github.com/corosync/corosync/blob/needle-2.3/test/testvotequorum2.c#L104
)
...
if ( (err=votequorum_qdevice_master_wins(handle, QDEVICE,
master_wins)) != CS_OK) {
fprintf(stderr, qdevice_master_wins FAILED: %d\n, err);
ret = -1;
goto out;
}
lib/votequorum.c (
https://github.com/corosync/corosync/blob/needle-2.3/lib/votequorum.c#L652)
cs_error_t votequorum_qdevice_master_wins (
votequorum_handle_t handle,
const char *name,
unsigned int allow)
{
...
}
The second Q:
As i understand 'CAST_VOTE=1' flag says to corosync that need to
consider the votes of a QDEVICE ?
No, cast_vote is not the number of votes. A qdevice can be registered
and polling, but it´s not casting a vote.
i meant that cast_vote is a flag, and when it is TRUE then
corosync consider the votes of a QDEVICE. so you have confirmend
my assumption.
That exactly to implement master_win solution.
A master is casting a vote, a slave is not casting a vote. Both needs to
poll via qdevice_poll.
The old concept that a qdevice needs to know how many votes to cast has
been removed. votequorum knows internally how many vote a device should
have.
___
Openais mailing list
Openais@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/openais
___
Openais mailing list
Openais@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/openais
Re: [Openais] corosync and qdevice
On 3/20/2013 5:27 PM, eXeC001er wrote: The first Q: According to the tests that are part of corosync-sources i think that i can: 1. create a daemon that register a QDEVICE and will notify corosync about the device (votequorum_qdevice_poll()). 2. implement a master/slave logic and if the qdevice on a node wins then i call votequorum_qdevice_master_wins() on the node and corosync notify another nodes about this, so i can say that the node is MASTER. There is no votequorum_qdevice_master_wins() call... where did you find it? I am researching corosync-2.3.0 sources. whoops.. i wrote it and forgot about it. getting old is bad :) No it´s a bit more complicated than that. corosync starts and load the config later on qdeviced starts and read the config qdeviced detects that it has to run in master_wins config: call votequorum_qdevice_master_wins(..., 1); that calls set a flags for the node and makes sure that the feature is enabled internally. then, you decide who is master (and tell corosync) by casting or not casting your vote. master will cast, slaves will not. As for debugging it is useful as it sets some bits around cmap and it makes it easier to detect if nodes have been configured differently. See commit 2f369e7039bc9054033693c56e93db9f4021a73f Fabio ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/openais
Re: [Openais] corosync and qdevice
2013/3/20 Fabio M. Di Nitto fdini...@redhat.com On 3/20/2013 5:27 PM, eXeC001er wrote: The first Q: According to the tests that are part of corosync-sources i think that i can: 1. create a daemon that register a QDEVICE and will notify corosync about the device (votequorum_qdevice_poll()). 2. implement a master/slave logic and if the qdevice on a node wins then i call votequorum_qdevice_master_wins() on the node and corosync notify another nodes about this, so i can say that the node is MASTER. There is no votequorum_qdevice_master_wins() call... where did you find it? I am researching corosync-2.3.0 sources. whoops.. i wrote it and forgot about it. getting old is bad :) No it´s a bit more complicated than that. corosync starts and load the config later on qdeviced starts and read the config qdeviced detects that it has to run in master_wins config: call votequorum_qdevice_master_wins(..., 1); that calls set a flags for the node and makes sure that the feature is enabled internally. I thought about a different scenario: master_wins and cast_vote are different flags and they are used in different cases. 1. uses only cast_vote and the flag can be used to decide that on a node everything fine and the node is a member of cluster (the cluster does not have master/slave) for example in a cluster (3 nodes) i have several qdevices on each node: storage-qdevice and client-network-qdevice config: each qdevice has 2 votes each node has 1 vote expected votes = 7 votes (1 own vote + 1 vote from anoter node + 2 votes from each qdevice) so if one of the qdevices detects some issues it does not cast own votes and the defect node will be excluded by corosync from cluster because the node will have 7 votes. 2. master_wins and cast_vote are used. in this case cast_vote will work as in case 1 and master_wins will control master/slave then, you decide who is master (and tell corosync) by casting or not casting your vote. master will cast, slaves will not. As for debugging it is useful as it sets some bits around cmap and it makes it easier to detect if nodes have been configured differently. See commit 2f369e7039bc9054033693c56e93db9f4021a73f Fabio ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/openais ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/openais
Re: [Openais] corosync and qdevice
On 3/20/2013 6:26 PM, eXeC001er wrote: 2013/3/20 Fabio M. Di Nitto fdini...@redhat.com mailto:fdini...@redhat.com On 3/20/2013 5:27 PM, eXeC001er wrote: The first Q: According to the tests that are part of corosync-sources i think that i can: 1. create a daemon that register a QDEVICE and will notify corosync about the device (votequorum_qdevice_poll()). 2. implement a master/slave logic and if the qdevice on a node wins then i call votequorum_qdevice_master_wins() on the node and corosync notify another nodes about this, so i can say that the node is MASTER. There is no votequorum_qdevice_master_wins() call... where did you find it? I am researching corosync-2.3.0 sources. whoops.. i wrote it and forgot about it. getting old is bad :) No it´s a bit more complicated than that. corosync starts and load the config later on qdeviced starts and read the config qdeviced detects that it has to run in master_wins config: call votequorum_qdevice_master_wins(..., 1); that calls set a flags for the node and makes sure that the feature is enabled internally. I thought about a different scenario: master_wins and cast_vote are different flags and they are used in different cases. 1. uses only cast_vote and the flag can be used to decide that on a node everything fine and the node is a member of cluster (the cluster does not have master/slave) for example in a cluster (3 nodes) i have several qdevices on each node: storage-qdevice and client-network-qdevice The API does not support multiple qdevices. This kind of implementation, where you need to poll multiple targets, can be multiplexed/proxy´d by the votequorum consumer. qdevice does not need to know how much to vote. That´s votequorum problem. How the qdevice implementation handles internal voting/scoring it´s qdevice implementation problem. config: each qdevice has 2 votes each node has 1 vote expected votes = 7 votes (1 own vote + 1 vote from anoter node + 2 votes from each qdevice) that won´t work. votequorum accepts only one value for qdevice votes. 2. master_wins and cast_vote are used. in this case cast_vote will work as in case 1 and master_wins will control master/slave The need to propagate the value for master_win and it´s status is to allow: node1 node2 node3 node4 qdevice is master on node3 for example in case of a 50%/50% split you have: node1 node2 - not quorate with the old cman quorum: node3 quorate node4 not quorate (that´s also why we force master_win in 2 node only) with the master_win flag set in votequorum: node3 quorate node4 quorate (because part of a partition with a master qdevice) note that node4 does not have the votes from qdevice at this point. Fabio ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/openais
