[
https://issues.apache.org/jira/browse/PHOENIX-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893085#comment-15893085
]
Josh Elser commented on PHOENIX-3654:
-------------------------------------
bq. I believe ephemerals aren't allowed to have children
Yeah, I believe you'd want to create a persistent "parent" znode. This znode
would be what the clients (of the load balancer) would query. Multiple PQS
instances registering themselves would all add themselves as children to that
persistent znode. Clients, when doing a {{getChildren()}} could set (and
maintain) a watcher which would trigger a callback whenever the list of
children change.
bq. The data of the ephemeral node will be connection url in the form of
“jdbc:think:driver:localhost:8765”
Would suggest just using hostname:port instead of the full JDBC url. No need to
include the extra info -- clients just need to know "where to go".
bq. LB will check connection status to PQS server
What does this "check" look like?
bq. If connection status to PQS is not ok, LB will remove this data from cache
and hand over another connection url from cache.
Consider the following case:
* PQS is running, maintaining its znode in ZK
* Client begins check to PQS
* PQS swaps-out/pauses
* Client times out and thinks PQS node is bad
* PQS recovers without losing ZK session
This would result in the client never using this PQS instance again.
bq. We will use apache curator for ZK connection as it has service discovery
module http://curator.apache.org/curator-x-discovery/index.html.
Do we need to pull in curator for this? Is Curator buying enough over the
standard Java ZK API to make it worth it?
bq. It will get the latest data on the # of connection for znodes
bq. The PQS which has least # of connection will be first to be given to thin
client.
How will you know how many connections each PQS instance has from ZooKeeper?
You said the znode contents were just going to be a JDBC url, right?
bq. How will LB takes care of load is being distributed evenly
How are multiple clients of the load balancer aware of each others'
connections? Does this imply that the LB is actually a daemon running outside
of the clients' JVM?
bq. It add watcher so that any change to the ephemeral node will also modify
the cached data.
A concern is that there is an upper limit on the number of watchers that a
ZooKeeper server can support. Considering HBase and whatever else is already
setting/using watchers of its own, we should try to make sure to be "good
citizens". ZOOKEEPER-1177 for context. As you've described, this would result
in each client keeping a zookeeper connection (which, itself, is overhead) open
as well as the watchers set server-side. Architecturally, this is what makes ZK
a bad choice for a load balancer -- once you find a PQS to talk to, you don't
need to maintain the ZK connection anymore. You only need the ZK connection
when the PQS instance you were talking to goes away.
And, on that note, how how PQS failure be handled (or not) by this design?
--
Nice start to a write-up, Rahul. Good work.
> Load Balancer for thin client
> -----------------------------
>
> Key: PHOENIX-3654
> URL: https://issues.apache.org/jira/browse/PHOENIX-3654
> Project: Phoenix
> Issue Type: New Feature
> Affects Versions: 4.8.0
> Environment: Linux 3.13.0-107-generic kernel, v4.9.0-HBase-0.98
> Reporter: Rahul Shrivastava
> Fix For: 4.9.0
>
> Attachments: LoadBalancerDesign.pdf
>
> Original Estimate: 240h
> Remaining Estimate: 240h
>
> We have been having internal discussion on load balancer for thin client for
> PQS. The general consensus we have is to have an embedded load balancer with
> the thin client instead of using external load balancer such as haproxy. The
> idea is to not to have another layer between client and PQS. This reduces
> operational cost for system, which currently leads to delay in executing
> projects.
> But this also comes with challenge of having an embedded load balancer which
> can maintain sticky sessions, do fair load balancing knowing the load
> downstream of PQS server. In addition, load balancer needs to know location
> of multiple PQS server. Now, the thin client needs to keep track of PQS
> servers via zookeeper ( or other means).
> In the new design, the client ( PQS client) , it is proposed to have an
> embedded load balancer.
> Where will the load Balancer sit ?
> The load load balancer will embedded within the app server client.
> How will the load balancer work ?
> Load balancer will contact zookeeper to get location of PQS. In this case,
> PQS needs to register to ZK itself once it comes online. Zookeeper location
> is in hbase-site.xml. It will maintain a small cache of connection to the
> PQS. When a request comes in, it will check for an open connection from the
> cache.
> How will load balancer know load on PQS ?
> To start with, it will pick a random open connection to PQS. This means that
> load balancer does not know PQS load. Later , we can augment the code so that
> thin client can receive load info from PQS and make intelligent decisions.
> How will load balancer maintain sticky sessions ?
> While we still need to investigate how to implement sticky sessions. We can
> look for some open source implementation for the same.
> How will PQS register itself to service locator ?
> PQS will have location of zookeeper in hbase-site.xml and it would register
> itself to the zookeeper. Thin client will find out PQS location using
> zookeeper.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
