Flavio, thank you for the suggestion.
I have looked at the documention (relevant snippets pasted in below), and
looked at the presentations
but I still have some questions about WAN configuration:
A <-> B
A <-> C
A <-> D
A is a central processing hub (DC).
B-D are remote colo edge nodes (PODS).
Each POD contains (m) ZK Servers with (q) client connections.
What are the advantages and disadvantages to co-locating ZK Servers across a
WAN? Could you correct my admitedly naïve assumtions here?
1. ZK Servers within a POD would significantly improve read/write performance
within a given POD, v.s. clients within the POD opening connections to the DC.
2. ZK Servers within a POD would provide local file transacted storage of
writes, obviating the need to write that code ourselves.
3. ZK Servers within the POD would be resilient to network connectivity failure
between the POD and the DC. Once connectivity re-established, the ZK Servers in
the POD would sync with the ZK servers in the DC, and, from the perspective of
a client within the POD, everything just worked, and there was no network
4. A WAN topology of co-located ZK servers in both the DC and (n) PODs would
not significantly degrade the performance of the ensemble, provided large blobs
of traffic were not being sent across the network.
Doc references below
(No Java system property)
Enables a hierarchical quorum construction."x" is a group identifier and
the numbers following the "=" sign correspond to server identifiers. The
left-hand side of the assignment is a colon-separated list of server
identifiers. Note that groups must be disjoint and the union of all groups must
be the ZooKeeper ensemble.
(No Java system property)
Used along with "group", it assigns a weight to a server when forming
quorums. Such a value corresponds to the weight of a server when voting. There
are a few parts of ZooKeeper that require voting such as leader election and
the atomic broadcast protocol. By default the weight of server is 1. If the
configuration defines groups, but not weights, then a value of 1 will be
assigned to all servers.
A different construction that uses weights and is useful in wide-area
deployments (co-locations) is a hierarchical one. With this construction, we
split the servers into disjoint groups and assign weights to processes. To form
a quorum, we have to get a hold of enough servers from a majority of groups G,
such that for each group g in G, the sum of votes from g is larger than half of
the sum of weights in g. Interestingly, this construction enables smaller
quorums. If we have, for example, 9 servers, we split them into 3 groups, and
assign a weight of 1 to each server, then we are able to form quorums of size
4. Note that two subsets of processes composed each of a majority of servers
from each of a majority of groups necessarily have a non-empty intersection. It
is reasonable to expect that a majority of co-locations will have a majority of
servers available with high probability.
With ZooKeeper, we provide a user with the ability of configuring servers to
use majority quorums, weights, or a hierarchy of groups.
From: Flavio Junqueira [mailto:f...@yahoo-inc.com]
Sent: Saturday, July 25, 2009 7:55 AM
Subject: Re: Zookeeper WAN Configuration
Todd, you can try using flexible quorums to implementing what your
requesting. You can simulate the behavior I described of observers by
setting the weight of the server to zero. Please check the
Check under "Cluster Options" options like group and weight.
On Jul 24, 2009, at 5:03 PM, Todd Greenwood wrote:
> In the future, once the Observers feature is implemented, then we
> be able to deploy zk servers to both the DC and to the pods...with all
> the goodness that Flavio mentions below.
> -----Original Message-----
> From: Flavio Junqueira [mailto:f...@yahoo-inc.com]
> Sent: Friday, July 24, 2009 4:50 PM
> To: email@example.com
> Subject: Re: Zookeeper WAN Configuration
> Just a few quick observations:
> On Jul 24, 2009, at 4:40 PM, Ted Dunning wrote:
>> On Fri, Jul 24, 2009 at 4:23 PM, Todd Greenwood
>>> Could you explain the idea behind the Observers feature, what this
>>> concept is supposed to address, and how it applies to the WAN
>>> configuration problem in particular?
>> Not really. I am just echoing comments on observers from them that
> Without observers, increasing the number of servers in an ensemble
> enables higher read throughput, but causes write throughput to drop
> because the number of votes to order each write operation increases.
> Essentially, observers are zookeeper servers that don't vote when
> ordering updates to the zookeeper state. Adding observers enables
> higher read throughput affecting minimally write throughput (leader
> still has to send commits to everyone, at least in the version we have
> been working on).
>>> The ideas for federating ZK or allowing observers would likely do
>>> want. I can imagine that an observer would only care that it can
>>> local peers and one of the observers would be elected to get updates
>>> thus would care about the central service).
>>> This certainly sounds like exactly what I want...Was this
>>> introduced in
>>> 3.2 in full, or only partially?
>> I don't think it is even in trunk yet. Look on Jira or at the
>> recent logs
>> of this mailing list.
> It is not on trunk yet.