[
https://issues.apache.org/jira/browse/HBASE-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236499#comment-13236499
]
nkeywal commented on HBASE-5573:
--------------------------------
bq. System.exit(1);
Actually is was already like that in hbasefsck, I replaced a tracker by a
watcher that does not watch to read the data, that's all.
bq. Why not add a create method to ZooKeeperWatcher that takes a name, conf,
and Abortable? Or is that a ZKW Constructor altogether?
Yes, the question is what to do when you're asked to abort. Here I reused the
approach in hbasefsck, just exit.
bq. N, can you explain more about what is going on here. How is it that we are
not taking a Watcher when we are creating a ZKW? Because we don't call start?
(If so, that'd be 'elegant' solution)
A ZKW is a watcher. When you create a ZKW, you create a RecoverableZooKeeper
with yourself as a parameter. Pseudo code is:
{noformat}
class RecoverableZooKeeper {
ZooKeeper zk;
RecoverableZooKeeper (Watcher w){ zk=new ZooKeeper(w) }
}
class ZooKeeperWatcher implements Watcher {
RecoverableZooKeeper rz;
ZooKeeperWatcher (){ rz = new RecoverableZooKeeper(this); }
}
{noformat}
Using 'this' in a constructor is looking for problems but it works in this case
(remember, that's the existing code, not mine :-) ). Basically all these
classes are very strongly coupled. When I tried to partially decouple them it
exploded in my hands because you anyway need a watcher to manage the session
expiry stuff. I don't have a middle solution here: it's either a full rewriting
with a lot of fun to keep the existing interfaces for backward compatibility or
nothing.
So in the final patch I've just done some cleanup (removed the last usage of
getZooKeeperWatcher) and the usage of any watcher. So there's no proof in the
code, just that actually all the functions we use on the client don't use a
watcher. Anyway, they have a session in the ZK servers so they are expensive.
But thanks to #5399 the session on ZK will be closed after 5 minutes. So if you
have an architecture with clients coming up and down, you will be able to
increase the number of clients.
Three last comments:
- one of the design issue is that there ate two API: you can use directly any
of the ZKW, RZK, RK object or you can go through the static ZKUtils. May be the
intermediate solutions lie around this area.
- even if the existing design should not be shown to innocent scholars it's not
that terrible, because it's small. I didn't really like my first patches
because I was adding more classes and complexity without fixing the design.
- On the long term, I think that it actually make sense to have a watcher in
the client. It's not about the previous code: The previous code was not really
using watchers. The previous code was setting watchers without using them. The
new code (after #5399 and #5573) does not use or set watchers. But when you
have a fat client architecture like we have, it makes sense to share some
global state information, and it scales better when the info is pushed vs.
pulled. Having said that, there are many questions left: possible issues in how
expensive it is with ZooKeeper today, may be ZooKeeper is not really designed
for this (it's not really a global coordination work, as the client would be
readers only) and so on. FWIW, it seems that the current limit is around 10K
sessions in ZK:
{panel}
Patrick Hunt / Nov 18, 2010; 8:57pm
Re: number of clients/watchers
fyi: I haven't heard of anyone running over 10k sessions. I've tried 20k before
and had issues.
[...]
A session is represented by a "ZooKeeper" object. One session per object. So if
you have 10 client hosts each creating it's own ZooKeeper instance you'll have
10 sessions. This is regardless of the number of znodes, watches, etc...
Watches were designed to be lightweight and you can maintain a large number of
them. (25million spread across 500 sessions in my example)
{panel}
There were also a discussion on ZK mailing list about lightweith sessions.
http://markmail.org/message/cyow2xkneh2t3juc
> Replace client ZooKeeper watchers by simple ZooKeeper reads
> -----------------------------------------------------------
>
> Key: HBASE-5573
> URL: https://issues.apache.org/jira/browse/HBASE-5573
> Project: HBase
> Issue Type: Improvement
> Components: client, zookeeper
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5573.v1.patch, 5573.v2.patch, 5573.v4.patch,
> 5573.v6.patch
>
>
> Some code in the package needs to read data in ZK. This could be done by a
> simple read, but is actually implemented with a watcher. This holds ZK
> resources.
> Fixing this could also be an opportunity to remove the need for the client to
> provide the master address and port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
