This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch branch-3.6
in repository https://gitbox.apache.org/repos/asf/zookeeper.git
The following commit(s) were added to refs/heads/branch-3.6 by this push:
new d4681c7 ZOOKEEPER-3720: Fix rolling upgrade failure (invalid protocol
version)
d4681c7 is described below
commit d4681c7bda159fac7aa864b0c71074ce95feaf38
Author: Mate Szalay-Beko <[email protected]>
AuthorDate: Sun Feb 16 10:25:14 2020 +0100
ZOOKEEPER-3720: Fix rolling upgrade failure (invalid protocol version)
The multi-address feature introduced in ZOOKEEPER-3188 required
changes in the Quorum protocol as we had to send all addresses in
the connection initiation message to enable the receiving side to
choose a reachable address in case of network failure.
The new code can handle both the old and the new protocol versions to
avoid 'invalid protocol' error e.g. during rolling restarts. However,
the new protocol version still can not be used during rolling upgrade
if the old servers are not supporting this protocol. In this case the
old and the new servers would form two distinct partitions until all
the servers get upgraded. To support rolling upgrades too, we want to
disable the MultiAddress feature by default and use the old protocol.
If the user would like enable the MultiAddress feature on a 3.6.0
cluster, she/he can do it either by 1) starting the cluster from
scratch (without rolling upgrade), or 2) performing a rolling upgrade
without the MultiAddress feature enabled then doing a rolling restart
with a new configuration where the MultiAddress feature is enabled.
During the rolling restart there will be no partitions, as all the
servers in the cluster now will run ZooKeeper version 3.6.0 which
understands now both protocols.
The changes in this patch:
- introducing new config property: multiAddress.enabled, disabled
by default
- updating QuorumCnxManager to be able to use both protocol versions
and to use the old one if MultiAddress is disabled
- failing with ConfigException if the user provides multiple
addresses in the config while having MultiAddress disabled
- updating the existing MultiAddress related tests to enable the
feature first
- add some new tests
- update the documentation
Testing:
- I ran all the unit tests
- Using https://github.com/symat/zk-rolling-upgrade-test
- I tested rolling upgrade from 3.5.6
- I tested rolling restart to enable the MultiAddress feature
- Using https://github.com/symat/zookeeper-docker-test
- I tested the MultiAddress feature by disabling some virtual
interfaces and waiting for the cluster to recover
Author: Mate Szalay-Beko <[email protected]>
Reviewers: Enrico Olivelli <[email protected]>, Norbert Kalmar
<[email protected]>, Andor Molnar <[email protected]>
Closes #1251 from symat/ZOOKEEPER-3720
(cherry picked from commit 3aa922c5737c9ef0879f290181cb281261c965e0)
Signed-off-by: Enrico Olivelli <[email protected]>
---
.../src/main/resources/markdown/zookeeperAdmin.md | 30 ++++++++++++++++++--
.../zookeeper/server/quorum/QuorumCnxManager.java | 33 ++++++++++++++++------
.../apache/zookeeper/server/quorum/QuorumPeer.java | 24 +++++++++++++++-
.../zookeeper/server/quorum/QuorumPeerConfig.java | 16 +++++++----
.../zookeeper/server/quorum/QuorumPeerMain.java | 1 +
.../zookeeper/server/quorum/CnxManagerTest.java | 10 +++----
.../zookeeper/server/quorum/LearnerTest.java | 8 ++++++
.../quorum/QuorumPeerMainMultiAddressTest.java | 31 ++++++++++++++++++++
.../zookeeper/server/quorum/QuorumSSLTest.java | 3 ++
.../zookeeper/server/quorum/QuorumServerTest.java | 19 +++++++++++++
.../server/quorum/auth/QuorumAuthTestBase.java | 8 ++++++
.../server/quorum/auth/QuorumAuthUpgradeTest.java | 2 ++
.../server/quorum/auth/QuorumDigestAuthTest.java | 2 ++
.../server/quorum/auth/QuorumKerberosAuthTest.java | 2 ++
.../auth/QuorumKerberosHostBasedAuthTest.java | 2 ++
15 files changed, 169 insertions(+), 22 deletions(-)
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
b/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
index cfcc3e5..1c6e130 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
@@ -1132,7 +1132,8 @@ of servers -- that is, when deploying clusters of servers.
<a name="id_multi_address"></a>
Since ZooKeeper 3.6.0 it is possible to specify **multiple addresses** for
each
ZooKeeper server (see
[ZOOKEEPER-3188](https://issues.apache.org/jira/projects/ZOOKEEPER/issues/ZOOKEEPER-3188)).
- This helps to increase availability and adds network level
+ To enable this feature, you must set the *multiAddress.enabled*
configuration property
+ to *true*. This helps to increase availability and adds network level
resiliency to ZooKeeper. When multiple physical network interfaces are
used
for the servers, ZooKeeper is able to bind on all interfaces and runtime
switching
to a working interface in case a network error. The different addresses
can be specified
@@ -1141,7 +1142,18 @@ of servers -- that is, when deploying clusters of
servers.
server.1=zoo1-net1:2888:3888|zoo1-net2:2889:3889
server.2=zoo2-net1:2888:3888|zoo2-net2:2889:3889
server.3=zoo3-net1:2888:3888|zoo3-net2:2889:3889
-
+
+
+ ###### Note
+ >By enabling this feature, the Quorum protocol (ZooKeeper Server-Server
protocol) will change.
+ The users will not notice this and when anyone starts a ZooKeeper cluster
with the new config,
+ everything will work normally. However, it's not possible to enable this
feature and specify
+ multiple addresses during a rolling upgrade if the old ZooKeeper cluster
didn't support the
+ *multiAddress* feature (and the new Quorum protocol). In case if you need
this feature but you
+ also need to perform a rolling upgrade from a ZooKeeper cluster older than
*3.6.0*, then you
+ first need to do the rolling upgrade without enabling the MultiAddress
feature and later make
+ a separate rolling restart with the new configuration where
**multiAddress.enabled** is set
+ to **true** and multiple addresses are provided.
* *syncLimit* :
(No Java system property)
@@ -1489,6 +1501,16 @@ and [SASL authentication for
ZooKeeper](https://cwiki.apache.org/confluence/disp
(e.g. the zk/[email protected] client principal will be authenticated in
ZooKeeper as zk/myhost)
Default: false
+* *multiAddress.enabled* :
+ (Java system property: **zookeeper.multiAddress.enabled**)
+ **New in 3.6.0:**
+ Since ZooKeeper 3.6.0 you can also [specify multiple
addresses](#id_multi_address)
+ for each ZooKeeper server instance (this can increase availability when
multiple physical
+ network interfaces can be used parallel in the cluster). Setting this
parameter to
+ **true** will enable this feature. Please note, that you can not enable
this feature
+ during a rolling upgrade if the version of the old ZooKeeper cluster is
prior to 3.6.0.
+ The default value is **false**.
+
* *multiAddress.reachabilityCheckTimeoutMs* :
(Java system property:
**zookeeper.multiAddress.reachabilityCheckTimeoutMs**)
**New in 3.6.0:**
@@ -1501,7 +1523,8 @@ and [SASL authentication for
ZooKeeper](https://cwiki.apache.org/confluence/disp
in parallel for the different addresses, so the timeout you set here is
the maximum time will be taken
by checking the reachability of all addresses.
The default value is **1000**.
-
+
+ This parameter has no effect, unless you enable the MultiAddress feature
by setting *multiAddress.enabled=true*.
<a name="Experimental+Options%2FFeatures"></a>
@@ -1593,6 +1616,7 @@ the variable does.
Please note, disabling the reachability check will cause the cluster not
to be able to reconfigure
itself properly during network problems, so the disabling is advised only
during testing.
+ This parameter has no effect, unless you enable the MultiAddress feature
by setting *multiAddress.enabled=true*.
<a name="Disabling+data+directory+autocreation"></a>
diff --git
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
index a92fc52..45ffeb8 100644
---
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
+++
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumCnxManager.java
@@ -37,6 +37,8 @@ import java.nio.ByteBuffer;
import java.nio.channels.UnresolvedAddressException;
import java.time.Duration;
import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
import java.util.Collections;
import java.util.Enumeration;
import java.util.HashSet;
@@ -113,9 +115,12 @@ public class QuorumCnxManager {
private AtomicLong observerCounter = new AtomicLong(-1);
/*
- * Protocol identifier used among peers
+ * Protocol identifier used among peers (must be a negative number for
backward compatibility reasons)
*/
- public static final long PROTOCOL_VERSION = -65535L;
+ // the following protocol version was sent in every connection initiation
message since ZOOKEEPER-107 released in 3.5.0
+ public static final long PROTOCOL_VERSION_V1 = -65536L;
+ // ZOOKEEPER-3188 introduced multiple addresses in the connection
initiation message, released in 3.6.0
+ public static final long PROTOCOL_VERSION_V2 = -65535L;
/*
* Max buffer size to be read from the network.
@@ -218,7 +223,7 @@ public class QuorumCnxManager {
public static InitialMessage parse(Long protocolVersion,
DataInputStream din) throws InitialMessageException, IOException {
Long sid;
- if (protocolVersion != PROTOCOL_VERSION) {
+ if (protocolVersion != PROTOCOL_VERSION_V1 && protocolVersion !=
PROTOCOL_VERSION_V2) {
throw new InitialMessageException("Got unrecognized protocol
version %s", protocolVersion);
}
@@ -236,6 +241,8 @@ public class QuorumCnxManager {
throw new InitialMessageException("Read only %s bytes out of
%s sent by server %s", num_read, remaining, sid);
}
+ // in PROTOCOL_VERSION_V1 we expect to get a single address here
represented as a 'host:port' string
+ // in PROTOCOL_VERSION_V2 we expect to get multiple addresses
like: 'host1:port1|host2:port2|...'
String[] addressStrings = new String(b).split("\\|");
List<InetSocketAddress> addresses = new
ArrayList<>(addressStrings.length);
for (String addr : addressStrings) {
@@ -416,10 +423,20 @@ public class QuorumCnxManager {
// Sending id and challenge
- // represents protocol version (in other words - message type)
- dout.writeLong(PROTOCOL_VERSION);
+ // First sending the protocol version (in other words - message
type).
+ // For backward compatibility reasons we stick to the old protocol
version, unless the MultiAddress
+ // feature is enabled. During rolling upgrade, we must make sure
that all the servers can
+ // understand the protocol version we use to avoid multiple
partitions. see ZOOKEEPER-3720
+ long protocolVersion = self.isMultiAddressEnabled() ?
PROTOCOL_VERSION_V2 : PROTOCOL_VERSION_V1;
+ dout.writeLong(protocolVersion);
dout.writeLong(self.getId());
- String addr = self.getElectionAddress().getAllAddresses().stream()
+
+ // now we send our election address. For the new protocol version,
we can send multiple addresses.
+ Collection<InetSocketAddress> addressesToSend = protocolVersion ==
PROTOCOL_VERSION_V2
+ ? self.getElectionAddress().getAllAddresses()
+ : Arrays.asList(self.getElectionAddress().getOne());
+
+ String addr = addressesToSend.stream()
.map(NetUtils::formatInetAddr).collect(Collectors.joining("|"));
byte[] addr_bytes = addr.getBytes();
dout.writeInt(addr_bytes.length);
@@ -639,7 +656,7 @@ public class QuorumCnxManager {
synchronized boolean connectOne(long sid, MultipleAddresses electionAddr) {
if (senderWorkerMap.get(sid) != null) {
LOG.debug("There is a connection already for server {}", sid);
- if (electionAddr.size() > 1 &&
self.isMultiAddressReachabilityCheckEnabled()) {
+ if (self.isMultiAddressEnabled() && electionAddr.size() > 1 &&
self.isMultiAddressReachabilityCheckEnabled()) {
// since ZOOKEEPER-3188 we can use multiple election addresses
to reach a server. It is possible, that the
// one we are using is already dead and we need to clean-up,
so when we will create a new connection
// then we will choose an other one, which is actually
reachable
@@ -710,7 +727,7 @@ public class QuorumCnxManager {
synchronized void connectOne(long sid) {
if (senderWorkerMap.get(sid) != null) {
LOG.debug("There is a connection already for server {}", sid);
- if (self.isMultiAddressReachabilityCheckEnabled()) {
+ if (self.isMultiAddressEnabled() &&
self.isMultiAddressReachabilityCheckEnabled()) {
// since ZOOKEEPER-3188 we can use multiple election addresses
to reach a server. It is possible, that the
// one we are using is already dead and we need to clean-up,
so when we will create a new connection
// then we will choose an other one, which is actually
reachable
diff --git
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeer.java
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeer.java
index 011cc0a..dbd23e9 100644
---
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeer.java
+++
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeer.java
@@ -161,8 +161,22 @@ public class QuorumPeer extends ZooKeeperThread implements
QuorumStats.Provider
this.observerMasterPort = observerMasterPort;
}
- private int multiAddressReachabilityCheckTimeoutMs = (int)
MultipleAddresses.DEFAULT_TIMEOUT.toMillis();
+ public static final String CONFIG_KEY_MULTI_ADDRESS_ENABLED =
"zookeeper.multiAddress.enabled";
+ public static final String CONFIG_DEFAULT_MULTI_ADDRESS_ENABLED = "false";
+
+ private boolean multiAddressEnabled = true;
+ public boolean isMultiAddressEnabled() {
+ return multiAddressEnabled;
+ }
+
+ public void setMultiAddressEnabled(boolean multiAddressEnabled) {
+ this.multiAddressEnabled = multiAddressEnabled;
+ LOG.info("multiAddress.enabled set to {}", multiAddressEnabled);
+ }
+ public static final String
CONFIG_KEY_MULTI_ADDRESS_REACHABILITY_CHECK_TIMEOUT_MS =
"zookeeper.multiAddress.reachabilityCheckTimeoutMs";
+
+ private int multiAddressReachabilityCheckTimeoutMs = (int)
MultipleAddresses.DEFAULT_TIMEOUT.toMillis();
public int getMultiAddressReachabilityCheckTimeoutMs() {
return multiAddressReachabilityCheckTimeoutMs;
}
@@ -172,6 +186,8 @@ public class QuorumPeer extends ZooKeeperThread implements
QuorumStats.Provider
LOG.info("multiAddress.reachabilityCheckTimeoutMs set to {}",
multiAddressReachabilityCheckTimeoutMs);
}
+ public static final String
CONFIG_KEY_MULTI_ADDRESS_REACHABILITY_CHECK_ENABLED =
"zookeeper.multiAddress.reachabilityCheckEnabled";
+
private boolean multiAddressReachabilityCheckEnabled = true;
public boolean isMultiAddressReachabilityCheckEnabled() {
@@ -274,6 +290,12 @@ public class QuorumPeer extends ZooKeeperThread implements
QuorumStats.Provider
}
}
+ boolean multiAddressEnabled = Boolean.parseBoolean(
+
System.getProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
QuorumPeer.CONFIG_DEFAULT_MULTI_ADDRESS_ENABLED));
+ if (!multiAddressEnabled && serverAddresses.length > 1) {
+ throw new ConfigException("Multiple address feature is
disabled, but multiple addresses were specified for sid " + sid);
+ }
+
for (String serverAddress : serverAddresses) {
String serverParts[] =
ConfigUtils.getHostAndPort(serverAddress);
if ((serverClientParts.length > 2) || (serverParts.length < 3)
diff --git
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java
index 102d699..ae73ef3 100644
---
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java
+++
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java
@@ -121,13 +121,13 @@ public class QuorumPeerConfig {
protected int quorumCnxnThreadsSize;
// multi address related configs
+ private boolean multiAddressEnabled = Boolean.parseBoolean(
+ System.getProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
QuorumPeer.CONFIG_DEFAULT_MULTI_ADDRESS_ENABLED));
private boolean multiAddressReachabilityCheckEnabled =
-
Boolean.parseBoolean(System.getProperty("zookeeper.multiAddress.reachabilityCheckEnabled",
- "true"));
+
Boolean.parseBoolean(System.getProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_REACHABILITY_CHECK_ENABLED,
"true"));
private int multiAddressReachabilityCheckTimeoutMs =
-
Integer.parseInt(System.getProperty("zookeeper.multiAddress.reachabilityCheckTimeoutMs",
-
String.valueOf(MultipleAddresses.DEFAULT_TIMEOUT.toMillis())));
-
+
Integer.parseInt(System.getProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_REACHABILITY_CHECK_TIMEOUT_MS,
+
String.valueOf(MultipleAddresses.DEFAULT_TIMEOUT.toMillis())));
/**
* Minimum snapshot retain count.
@@ -398,6 +398,8 @@ public class QuorumPeerConfig {
} else if (key.startsWith("metricsProvider.")) {
String keyForMetricsProvider = key.substring(16);
metricsProviderConfiguration.put(keyForMetricsProvider, value);
+ } else if (key.equals("multiAddress.enabled")) {
+ multiAddressEnabled = Boolean.parseBoolean(value);
} else if (key.equals("multiAddress.reachabilityCheckTimeoutMs")) {
multiAddressReachabilityCheckTimeoutMs =
Integer.parseInt(value);
} else if (key.equals("multiAddress.reachabilityCheckEnabled")) {
@@ -939,6 +941,10 @@ public class QuorumPeerConfig {
return quorumListenOnAllIPs;
}
+ public boolean isMultiAddressEnabled() {
+ return multiAddressEnabled;
+ }
+
public boolean isMultiAddressReachabilityCheckEnabled() {
return multiAddressReachabilityCheckEnabled;
}
diff --git
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerMain.java
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerMain.java
index 04aec05..a6f94ec 100644
---
a/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerMain.java
+++
b/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerMain.java
@@ -204,6 +204,7 @@ public class QuorumPeerMain {
if (config.sslQuorumReloadCertFiles) {
quorumPeer.getX509Util().enableCertFileReloading();
}
+ quorumPeer.setMultiAddressEnabled(config.isMultiAddressEnabled());
quorumPeer.setMultiAddressReachabilityCheckEnabled(config.isMultiAddressReachabilityCheckEnabled());
quorumPeer.setMultiAddressReachabilityCheckTimeoutMs(config.getMultiAddressReachabilityCheckTimeoutMs());
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/CnxManagerTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/CnxManagerTest.java
index 371fc39..269b6a8 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/CnxManagerTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/CnxManagerTest.java
@@ -261,7 +261,7 @@ public class CnxManagerTest extends ZKTestCase {
InetSocketAddress otherAddr =
peers.get(2L).electionAddr.getReachableOrOne();
DataOutputStream dout = new
DataOutputStream(sc.socket().getOutputStream());
- dout.writeLong(QuorumCnxManager.PROTOCOL_VERSION);
+ dout.writeLong(QuorumCnxManager.PROTOCOL_VERSION_V1);
dout.writeLong(2);
String addr = otherAddr.getHostString() + ":" + otherAddr.getPort();
byte[] addr_bytes = addr.getBytes();
@@ -604,7 +604,7 @@ public class CnxManagerTest extends ZKTestCase {
dout.writeBytes(hostport);
din = new DataInputStream(new
ByteArrayInputStream(bos.toByteArray()));
- msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION, din);
+ msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION_V1,
din);
fail("long message accepted");
} catch (InitialMessage.InitialMessageException ex) {
}
@@ -620,7 +620,7 @@ public class CnxManagerTest extends ZKTestCase {
dout.writeBytes(hostport);
din = new DataInputStream(new
ByteArrayInputStream(bos.toByteArray()));
- msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION, din);
+ msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION_V1,
din);
fail("bad hostport accepted");
} catch (InitialMessage.InitialMessageException ex) {
}
@@ -637,7 +637,7 @@ public class CnxManagerTest extends ZKTestCase {
// now parse it
din = new DataInputStream(new
ByteArrayInputStream(bos.toByteArray()));
- msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION, din);
+ msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION_V1,
din);
assertEquals(Long.valueOf(5), msg.sid);
assertEquals(Arrays.asList(new InetSocketAddress("10.0.0.2",
3888)), msg.electionAddr);
} catch (InitialMessage.InitialMessageException ex) {
@@ -656,7 +656,7 @@ public class CnxManagerTest extends ZKTestCase {
// now parse it
din = new DataInputStream(new
ByteArrayInputStream(bos.toByteArray()));
- msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION, din);
+ msg = InitialMessage.parse(QuorumCnxManager.PROTOCOL_VERSION_V2,
din);
assertEquals(Long.valueOf(5), msg.sid);
assertEquals(Arrays.asList(new InetSocketAddress("1.1.1.1", 9999),
new InetSocketAddress("2.2.2.2", 8888),
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/LearnerTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/LearnerTest.java
index 63381fc..97ab5a2 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/LearnerTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/LearnerTest.java
@@ -53,6 +53,7 @@ import org.apache.zookeeper.test.TestUtils;
import org.apache.zookeeper.txn.CreateTxn;
import org.apache.zookeeper.txn.TxnHeader;
import org.apache.zookeeper.util.ServiceUtils;
+import org.junit.After;
import org.junit.Test;
public class LearnerTest extends ZKTestCase {
@@ -134,6 +135,11 @@ public class LearnerTest extends ZKTestCase {
}
}
+ @After
+ public void cleanup() {
+ System.clearProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED);
+ }
+
@Test(expected = IOException.class)
public void connectionRetryTimeoutTest() throws Exception {
Learner learner = new TestLearner();
@@ -178,6 +184,7 @@ public class LearnerTest extends ZKTestCase {
@Test
public void shouldTryMultipleAddresses() throws Exception {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
"true");
TestLearner learner = new TestLearner();
learner.self = new QuorumPeer();
learner.self.setTickTime(2000);
@@ -205,6 +212,7 @@ public class LearnerTest extends ZKTestCase {
@Test
public void multipleAddressesSomeAreFailing() throws Exception {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
"true");
TestLearner learner = new TestLearner();
learner.self = new QuorumPeer();
learner.self.setTickTime(2000);
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumPeerMainMultiAddressTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumPeerMainMultiAddressTest.java
index ae35573..9c2906c 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumPeerMainMultiAddressTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumPeerMainMultiAddressTest.java
@@ -19,6 +19,7 @@
package org.apache.zookeeper.server.quorum;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;
@@ -52,6 +53,7 @@ public class QuorumPeerMainMultiAddressTest extends
QuorumPeerTestBase {
@Before
public void setUp() throws Exception {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED, "true");
ClientBase.setupTestEnv();
System.setProperty("zookeeper.DigestAuthenticationProvider.superDigest",
"super:D/InIHSb7yEEbrWz8b9l71RjZJU="/* password is 'test'*/);
QuorumPeerConfig.setReconfigEnabled(true);
@@ -63,6 +65,7 @@ public class QuorumPeerMainMultiAddressTest extends
QuorumPeerTestBase {
@After
public void tearDown() throws Exception {
super.tearDown();
+ System.clearProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED);
System.clearProperty("zookeeper.jmx.log4j.disable");
}
@@ -230,6 +233,34 @@ public class QuorumPeerMainMultiAddressTest extends
QuorumPeerTestBase {
checkIfZooKeeperQuorumWorks(newQuorumConfig);
}
+ @Test
+ public void shouldFailToReconfigWithMultipleAddressesWhenFeatureIsDisabled()
throws Exception {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED, "false");
+
+ // we have three ZK servers, each server has a single quorumPort and
single electionPort registered
+ QuorumServerConfigBuilder initialQuorumConfig = new
QuorumServerConfigBuilder(hostName, 3, 1);
+
+ // we launch the three servers, each server should use the same initial
config
+ launchServers(Arrays.asList(initialQuorumConfig, initialQuorumConfig,
initialQuorumConfig));
+
+ checkIfZooKeeperQuorumWorks(initialQuorumConfig);
+
+ // we create a new config where we add a new address to one of the servers
with random available ports
+ QuorumServerConfigBuilder newQuorumConfig = new
QuorumServerConfigBuilder(initialQuorumConfig)
+ .addNewServerAddress(FIRST_SERVER);
+
+ ZooKeeperAdmin zkAdmin = newZooKeeperAdmin(initialQuorumConfig);
+
+ // initiating a new incremental reconfig by using the updated ports,
expecting exceptions here
+ try {
+ ReconfigTest.reconfig(zkAdmin, newQuorumConfig.buildAsStringList(),
null, null, -1);
+ fail("Reconfig succeeded with multiple addresses without exception when
the MultiAddress feature is disabled");
+ } catch (KeeperException.BadArgumentsException e) {
+ // do nothing, this is what we expected
+ } catch (Exception e) {
+ fail("Reconfig failed in a wrong way. We expected
KeeperException.BadArgumentsException.");
+ }
+ }
private void launchServers(List<QuorumServerConfigBuilder> builders) throws
IOException, InterruptedException {
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumSSLTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumSSLTest.java
index 09245c5..593edbd 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumSSLTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumSSLTest.java
@@ -430,6 +430,7 @@ public class QuorumSSLTest extends QuorumPeerTestBase {
@After
public void cleanUp() throws Exception {
+ System.clearProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED);
clearSSLSystemProperties();
if (q1 != null) {
q1.shutdown();
@@ -480,6 +481,7 @@ public class QuorumSSLTest extends QuorumPeerTestBase {
@Test
public void testQuorumSSLWithMultipleAddresses() throws Exception {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
"true");
quorumConfiguration = generateMultiAddressQuorumConfiguration();
q1 = new MainThread(1, clientPortQp1, quorumConfiguration,
SSL_QUORUM_ENABLED);
@@ -598,6 +600,7 @@ public class QuorumSSLTest extends QuorumPeerTestBase {
@Test
public void testHostnameVerificationForInvalidMultiAddressServerConfig()
throws Exception {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
"true");
quorumConfiguration = generateMultiAddressQuorumConfiguration();
String badhostnameKeystorePath = tmpDir + "/badhost.jks";
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumServerTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumServerTest.java
index c8cbada..6f5155b 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumServerTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/QuorumServerTest.java
@@ -23,6 +23,7 @@ import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZKTestCase;
import org.apache.zookeeper.server.quorum.QuorumPeer.QuorumServer;
import org.apache.zookeeper.server.quorum.QuorumPeerConfig.ConfigException;
+import org.junit.After;
import org.junit.Test;
public class QuorumServerTest extends ZKTestCase {
@@ -31,6 +32,11 @@ public class QuorumServerTest extends ZKTestCase {
private String ipv6n2 = "[2600:0:0:0:0:0:1:0]";
private String ipv4config = "127.0.0.1:1234:1236";
+ @After
+ public void tearDown() {
+ System.clearProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED);
+ }
+
@Test
public void testToString() throws ConfigException {
String provided = ipv4config + ":participant;0.0.0.0:1237";
@@ -88,6 +94,19 @@ public class QuorumServerTest extends ZKTestCase {
new QuorumServer(0, ipv4config + ":participant;[::1:1237");
}
+ @Test(expected = ConfigException.class)
+ public void
shouldNotAllowMultipleAddressesWhenMultiAddressFeatureIsDisabled() throws
ConfigException {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
"false");
+ new QuorumServer(0, "127.0.0.1:1234:1236|127.0.0.1:2234:2236");
+ }
+
+ @Test
+ public void shouldAllowMultipleAddressesWhenMultiAddressFeatureIsEnabled()
throws ConfigException {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
"true");
+ QuorumServer qs = new QuorumServer(0,
"127.0.0.1:1234:1236|127.0.0.1:2234:2236");
+ assertEquals("MultiAddress parse error",
"127.0.0.1:1234:1236|127.0.0.1:2234:2236:participant", qs.toString());
+ }
+
@Test
public void testWildcard() throws KeeperException.BadArgumentsException {
String[] addrs = new String[]{"127.0.0.1", "[0:0:0:0:0:0:0:1]",
"0.0.0.0", "[::]"};
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthTestBase.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthTestBase.java
index 97650d0..d365c9a 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthTestBase.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthTestBase.java
@@ -29,8 +29,10 @@ import javax.security.auth.login.Configuration;
import org.apache.commons.io.FileUtils;
import org.apache.zookeeper.PortAssignment;
import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.server.quorum.QuorumPeer;
import org.apache.zookeeper.server.quorum.QuorumPeerTestBase.MainThread;
import org.apache.zookeeper.test.ClientBase;
+import org.junit.After;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -69,6 +71,11 @@ public class QuorumAuthTestBase extends ZKTestCase {
}
}
+ @After
+ public void tearDown() throws Exception {
+ System.clearProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED);
+ }
+
protected String startQuorum(final int serverCount, Map<String, String>
authConfigs,
int authServerCount) throws IOException {
return this.startQuorum(serverCount, authConfigs, authServerCount,
false);
@@ -76,6 +83,7 @@ public class QuorumAuthTestBase extends ZKTestCase {
protected String startMultiAddressQuorum(final int serverCount,
Map<String, String> authConfigs,
int authServerCount) throws IOException {
+ System.setProperty(QuorumPeer.CONFIG_KEY_MULTI_ADDRESS_ENABLED,
"true");
return this.startQuorum(serverCount, authConfigs, authServerCount,
true);
}
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthUpgradeTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthUpgradeTest.java
index b944221..ecdfaeb 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthUpgradeTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumAuthUpgradeTest.java
@@ -67,8 +67,10 @@ public class QuorumAuthUpgradeTest extends
QuorumAuthTestBase {
}
@After
+ @Override
public void tearDown() throws Exception {
shutdownAll();
+ super.tearDown();
}
@AfterClass
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
index 7462979..cb28389 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
@@ -60,11 +60,13 @@ public class QuorumDigestAuthTest extends
QuorumAuthTestBase {
}
@After
+ @Override
public void tearDown() throws Exception {
for (MainThread mainThread : mt) {
mainThread.shutdown();
mainThread.deleteBaseDir();
}
+ super.tearDown();
}
@AfterClass
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosAuthTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosAuthTest.java
index e4534df..28b93c7 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosAuthTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosAuthTest.java
@@ -88,11 +88,13 @@ public class QuorumKerberosAuthTest extends
KerberosSecurityTestcase {
}
@After
+ @Override
public void tearDown() throws Exception {
for (MainThread mainThread : mt) {
mainThread.shutdown();
mainThread.deleteBaseDir();
}
+ super.tearDown();
}
@AfterClass
diff --git
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
index fc15049..c9b93c5 100644
---
a/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
+++
b/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
@@ -113,11 +113,13 @@ public class QuorumKerberosHostBasedAuthTest extends
KerberosSecurityTestcase {
}
@After
+ @Override
public void tearDown() throws Exception {
for (MainThread mainThread : mt) {
mainThread.shutdown();
mainThread.deleteBaseDir();
}
+ super.tearDown();
}
@AfterClass
