Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "ZooKeeper/HBaseUseCases" page has been changed by PatrickHunt. http://wiki.apache.org/hadoop/ZooKeeper/HBaseUseCases?action=diff&rev1=8&rev2=9 -------------------------------------------------- Here's an idea, see if I got the idea right, obv would have to flesh this out more but this is the general idea. I've chosen random paths below, obv you'd want some sort of prefix, better names, etc... 1) group membership: - # have a /regionservers znode + 1. have a /regionservers znode - # master watches /regionservers for any child changes + 1. master watches /regionservers for any child changes - # as each region server becomes available to do work (or track state if up but not avail) it creates an ephemeral node + 1. as each region server becomes available to do work (or track state if up but not avail) it creates an ephemeral node * /regionserver/<host:port> = <status> - # master watches /regionserver/<host:port> and cleans up if RS goes away or changes status + 1. master watches /regionserver/<host:port> and cleans up if RS goes away or changes status 2) task assignment (ie dynamic configuration) - # have a /tables znode + 1. have a /tables znode - # /tables/<regionserver by host:port> which gets created when master notices new region server + 1. /tables/<regionserver by host:port> which gets created when master notices new region server * RS host:port watches this node for any child changes - # /tables/<regionserver by host:port>/<regionX> znode for each region assigned to RS host:port + 1. /tables/<regionserver by host:port>/<regionX> znode for each region assigned to RS host:port * RS host:port watches this node in case reassigned by master, or region changes state - # /tables/<regionserver by host:port>/<regionX>/<state>-<seq#> znode created by master + 1. /tables/<regionserver by host:port>/<regionX>/<state>-<seq#> znode created by master * seq ensures order seen by RS * RS deletes old state znodes as it transitions out, oldest entry is the current state, always 1 or more znode here -- the current state @@ -73, +73 @@ See [[http://bit.ly/4ekN8G|this perf doc]] for some ideas, 20 clients doing 50k watches each - 1 million watches on a single core standalone server and still << 5ms avg response time (async ops, keep that in mind re implementation time) YMMV of course but your numbers are well below this. Worst-case scenarios -- say a cascade failure where all RS become disconnected and sessions expire - # master will get notified of 1000 RS gone + 1. master will get notified of 1000 RS gone - # master will delete all nodes in 2) - 1000 RS znodes, 100 regions each RS znode, 1 state (typ) each of the 100 reg + 1. master will delete all nodes in 2) - 1000 RS znodes, 100 regions each RS znode, 1 state (typ) each of the 100 reg * 200k nodes deleted (hint: use async) for each RS Another worst case: - # some set of RS are flapping (this may actually be much worse than just dying) + 1. some set of RS are flapping (this may actually be much worse than just dying) * consider running some sort of health check on RS before assigning work, in case it just dies * or, slowly ramp up the number of regions assigned to the RS, allow it to prove itself vs dumping a number of regions on it and then have it flap... (donno enough about hbase to comment resonably, but thing about something like this) - # for each RS master is deleting 200 znodes + 1. for each RS master is deleting 200 znodes [PDH end]
