[jira] [Commented] (ZOOKEEPER-2554) reconfig can not add new server as observer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468121#comment-15468121
]
Arshad Mohammad commented on ZOOKEEPER-2554:
It is fine. Thanks [~shralex].
> reconfig can not add new server as observer
> ---
>
> Key: ZOOKEEPER-2554
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2554
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.0, 3.5.1, 3.5.2
>Reporter: Arshad Mohammad
>Assignee: Arshad Mohammad
>Priority: Critical
> Attachments: ReconfigObserverTest.java, ZOOKEEPER-2554-01.patch
>
>
> Try to add new observer server using reconfig API, server gets added as
> participant.
> STEPS:
> # create 3 node cluster.
> {code}
> server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
> server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> {code}
> # Suppose the 2 is the leader in the above cluster. Configure the new server
> as
> {code}
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
> {code}
> # Connect to 1 and execute the reconfig command
> {code}
> zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231",
> null, null, -1, null, null);
> {code}
> # Verify sever 3. It was supposed to run as observer but it is running as
> participant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2554) reconfig can not add new server as observer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467950#comment-15467950
]
Alexander Shraer commented on ZOOKEEPER-2554:
-
Marking this as 'not a problem' for now. Please reopen if you still disagree.
> reconfig can not add new server as observer
> ---
>
> Key: ZOOKEEPER-2554
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2554
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.0, 3.5.1, 3.5.2
>Reporter: Arshad Mohammad
>Assignee: Arshad Mohammad
>Priority: Critical
> Attachments: ReconfigObserverTest.java, ZOOKEEPER-2554-01.patch
>
>
> Try to add new observer server using reconfig API, server gets added as
> participant.
> STEPS:
> # create 3 node cluster.
> {code}
> server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
> server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> {code}
> # Suppose the 2 is the leader in the above cluster. Configure the new server
> as
> {code}
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
> {code}
> # Connect to 1 and execute the reconfig command
> {code}
> zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231",
> null, null, -1, null, null);
> {code}
> # Verify sever 3. It was supposed to run as observer but it is running as
> participant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2554) reconfig can not add new server as observer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466268#comment-15466268
]
Alexander Shraer commented on ZOOKEEPER-2554:
-
Non-voting follower is very different from observer. Observers only get inform
(its like commit) messages, while non-voting followers are just like normal
followers - they get propose and commit, and also reply with ACKs to the leader
but the leader silently ignores their votes. They don't count for a quorum
needed to commit an operation. This mode existed before 3.5 - when you connect
any server to the ZK cluster, it just starts working as a follower. I used it
for reconfiguration as the state for servers that are not part of the
configuration. When a new server joins, it will initially become a non-voting
follower, so it is not necessary to start it as observer, you can just as well
start it as participant, because the conversion OBSERVER -> PARTICIPANT is
automatic as you noticed.
So - just start it as participant and issue a reconfig to turn it into an
observer.
> reconfig can not add new server as observer
> ---
>
> Key: ZOOKEEPER-2554
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2554
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.0, 3.5.1, 3.5.2
>Reporter: Arshad Mohammad
>Assignee: Arshad Mohammad
>Priority: Critical
> Attachments: ReconfigObserverTest.java, ZOOKEEPER-2554-01.patch
>
>
> Try to add new observer server using reconfig API, server gets added as
> participant.
> STEPS:
> # create 3 node cluster.
> {code}
> server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
> server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> {code}
> # Suppose the 2 is the leader in the above cluster. Configure the new server
> as
> {code}
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
> {code}
> # Connect to 1 and execute the reconfig command
> {code}
> zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231",
> null, null, -1, null, null);
> {code}
> # Verify sever 3. It was supposed to run as observer but it is running as
> participant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2554) reconfig can not add new server as observer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466241#comment-15466241
]
Arshad Mohammad commented on ZOOKEEPER-2554:
# bq. any server not currently in the view should be a non-voting participant.
AFAIK, non-voting participant means observer but in the code observer is
changed to participant(voting-participant). So code is doing just opposite of
what is intended, isn't it
# Test case is testing the way new observer should get added. It is not
testing the way it is getting added
For observer, server 3, the type changes multiple times: OBSERVER
-->PARTICIPANT -->OBSERVER
## initialized from configuration
{code}
2016-09-06 07:43:31,783 [myid:3] - INFO [Thread-7:QuorumPeer@415] - learner
type set to OBSERVER
java.lang.Exception: exceptionForStackTrace
at
org.apache.zookeeper.server.quorum.QuorumPeer.setLearnerType(QuorumPeer.java:416)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:179)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
at
org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:245)
{code}
## updated during first leader election
{code}
2016-09-06 07:43:31,808 [myid:3] - INFO
[WorkerReceiver[myid=3]:QuorumPeer@415] - learner type set to PARTICIPANT
java.lang.Exception: exceptionForStackTrace
at
org.apache.zookeeper.server.quorum.QuorumPeer.setLearnerType(QuorumPeer.java:416)
at
org.apache.zookeeper.server.quorum.QuorumPeer.updateLearnerType(QuorumPeer.java:1843)
at
org.apache.zookeeper.server.quorum.QuorumPeer.processReconfig(QuorumPeer.java:1762)
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerReceiver.run(FastLeaderElection.java:299)
at java.lang.Thread.run(Thread.java:745)
{code}
## During reconfig processing
{code}
2016-09-06 07:43:32,155 [myid:3] - INFO
[QuorumPeer[myid=3](plain=/127.0.0.1:11231)(secure=disabled):QuorumPeer@415] -
learner type set to OBSERVER
java.lang.Exception: exceptionForStackTrace
at
org.apache.zookeeper.server.quorum.QuorumPeer.setLearnerType(QuorumPeer.java:416)
at
org.apache.zookeeper.server.quorum.QuorumPeer.updateLearnerType(QuorumPeer.java:1824)
at
org.apache.zookeeper.server.quorum.QuorumPeer.processReconfig(QuorumPeer.java:1762)
at
org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:151)
at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:90)
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1113)
{code}
## In reconfig when type is changed from PARTICIPANT to OBSERVER, it is major
change and exception is thrown and leader election happens again.
# So overall procedure to add the observer becomes very lengthy unnecessarily.
is the expected way to add observers?
> reconfig can not add new server as observer
> ---
>
> Key: ZOOKEEPER-2554
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2554
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.0, 3.5.1, 3.5.2
>Reporter: Arshad Mohammad
>Assignee: Arshad Mohammad
>Priority: Critical
> Attachments: ReconfigObserverTest.java, ZOOKEEPER-2554-01.patch
>
>
> Try to add new observer server using reconfig API, server gets added as
> participant.
> STEPS:
> # create 3 node cluster.
> {code}
> server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
> server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> {code}
> # Suppose the 2 is the leader in the above cluster. Configure the new server
> as
> {code}
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
> {code}
> # Connect to 1 and execute the reconfig command
> {code}
> zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231",
> null, null, -1, null, null);
> {code}
> # Verify sever 3. It was supposed to run as observer but it is running as
> participant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2554) reconfig can not add new server as observer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465940#comment-15465940
]
Alexander Shraer commented on ZOOKEEPER-2554:
-
I haven't looked closely yet, but the code you quote has the following
intention: any server not currently in the view should be a non-voting
participant.
When you connect server 3, it will adopt the leader's config in which it isn't
present, and automatically become such a non-voting participant, before
you invoke the reconfig command. So that's working as intended. In this state,
when you invoke the reconfig it should become observer. I don't know why
this isn't happening, but I think its probably covered by existing tests.
I'll try to find time to look into it, but I'd look into two things: The log of
server 3, and what is the config file on server 3 right before and after the
reconfig is invoked.
> reconfig can not add new server as observer
> ---
>
> Key: ZOOKEEPER-2554
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2554
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.0, 3.5.1, 3.5.2
>Reporter: Arshad Mohammad
>Assignee: Arshad Mohammad
>Priority: Critical
> Attachments: ZOOKEEPER-2554-01.patch
>
>
> Try to add new observer server using reconfig API, server gets added as
> participant.
> STEPS:
> # create 3 node cluster.
> {code}
> server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
> server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> {code}
> # Suppose the 2 is the leader in the above cluster. Configure the new server
> as
> {code}
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
> {code}
> # Connect to 1 and execute the reconfig command
> {code}
> zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231",
> null, null, -1, null, null);
> {code}
> # Verify sever 3. It was supposed to run as observer but it is running as
> participant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2554) reconfig can not add new server as observer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465730#comment-15465730
]
Arshad Mohammad commented on ZOOKEEPER-2554:
As per my understanding, in
{{org.apache.zookeeper.server.quorum.QuorumPeer.updateLearnerType(QuorumVerifier)}}
bellow code causing the problem.
{code}
// I'm not in the view
if (getLearnerType()!=LearnerType.PARTICIPANT){
setLearnerType(LearnerType.PARTICIPANT);
LOG.info("Becoming a non-voting participant");
reconfigFlagSet();
return true;
}
{code}
getLearnerType() is LearnerType.OBSERVER in this case but this code changes
learner type to LearnerType.PARTICIPANT blindly.
> reconfig can not add new server as observer
> ---
>
> Key: ZOOKEEPER-2554
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2554
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.0, 3.5.1, 3.5.2
>Reporter: Arshad Mohammad
>Assignee: Arshad Mohammad
>Priority: Critical
> Attachments: ZOOKEEPER-2554-01.patch
>
>
> Try to add new observer server using reconfig API, server gets added as
> participant.
> STEPS:
> # create 3 node cluster.
> {code}
> server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
> server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> {code}
> # Suppose the 2 is the leader in the above cluster. Configure the new server
> as
> {code}
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
> {code}
> # Connect to 1 and execute the reconfig command
> {code}
> zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231",
> null, null, -1, null, null);
> {code}
> # Verify sever 3. It was supposed to run as observer but it is running as
> participant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2554) reconfig can not add new server as observer
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465723#comment-15465723
]
Arshad Mohammad commented on ZOOKEEPER-2554:
ZOOKEEPER-2554-01.patch is test code patch to reproduce the scenario.
> reconfig can not add new server as observer
> ---
>
> Key: ZOOKEEPER-2554
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2554
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.0, 3.5.1, 3.5.2
>Reporter: Arshad Mohammad
>Assignee: Arshad Mohammad
>Priority: Critical
> Attachments: ZOOKEEPER-2554-01.patch
>
>
> Try to add new observer server using reconfig API, server gets added as
> participant.
> STEPS:
> # create 3 node cluster.
> {code}
> server.0=127.0.0.1:11223:11224:participant;127.0.0.1:11222
> server.1=127.0.0.1:11226:11227:participant;127.0.0.1:11225
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> {code}
> # Suppose the 2 is the leader in the above cluster. Configure the new server
> as
> {code}
> server.2=127.0.0.1:11229:11230:participant;127.0.0.1:11228
> server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231
> {code}
> # Connect to 1 and execute the reconfig command
> {code}
> zkClient.reconfig("server.3=127.0.0.1:11232:11233:observer;127.0.0.1:11231",
> null, null, -1, null, null);
> {code}
> # Verify sever 3. It was supposed to run as observer but it is running as
> participant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
