Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The following page has been changed by stack: http://wiki.apache.org/hadoop/Hbase/MasterRewrite The comment on the change is: Initial upload of notes from hackathon New page: = Design Notes for [https://issues.apache.org/jira/browse/HBASE-1816 Master Rewrite, HBASE-1816] for hbase 0.21.0 = The notes below grew out of conversation had at the hbase hackathon held at StumbleUpon, August 5-7, 2009 (Jon Gray kept notes. TODO: Link to them when posted). == What does the Master do now? == * Splitting * Should be distributed * Region Assignment * Balancing * Scan root/meta * Make sure Regions are online * Delete parents if no reference * Manage schema alter/online/offline * Admin * Close region * Flush * Compact/Major compact * Watches ZK for its own lease and for regionservers so knows when to run recovery == Design == == Notes == To be organized... For RS Worst case loading calculation * 200 regionservers * 32 logs per regionserver * 200 regions written to * 2GB or 30 hour full log roll * 10MB/sec write speed * 1.2M edits per 2G * 7k writes/second across cluster (?) -- whats this? Wrong. * 1.2M edits per 30 hours? * 100 writes/sec across cluster (?) -- Whats this? Wrong? Assignment / balancing * RS publish load into ZK * /hbase/rsload/startcode({âjsonâ:âloadâ}) * Configure period it is refreshed * Assignment inputs * Load * Requests / sec * regions online * Distribute tables randomly across cluster * Never give table back to who unassigned it * During split, bottom half to the same server, top half reassigned * Assignment Queue * Candidate Queue * Master watches ZK candidate queue /hbase/rsassign/region(last_owned_by_rs) * When new nodes come in, it assigns the out Regionservers put regions into the candidate queue when they unassign/close * To Open Queue * Regionservers watch their own to open queues /hbase/rsopen/region(extra_info, which hlogs to replay or itâs a split, etc) Administrative functions * Hadoop RPC listeners on Master and Regionservers -- master can now push messages * Clients and Master can talk to RS Safe-mode assignment * Collect all regions to assign * Randomize and assign out in bulk, one msg per RS * NO MORE SAFE-MODE * Region assignment is always * Look at all regions to be assigned * Make a single decision for the assignment of all of these META to ZK * New feature in ZK * Sorted tree or list node * Has a getClosestBefore / getRegionForRow(row) * META location stored in ZK * META table now only contains historian information * All other in ZK No more ROOT Worker pool region closing * Parallel flushes No more CHANGE TABLE STATE * Process server shutdown after RS crash * Separate META scan? * To figure out regions on an RS * Separate map of RS -> region * Trade-off between two-writes during assignment Table Schema Information * Online schema edits? * If complex, punt to 0.22 * Rather than storing with each region, stored once in ZK Uptime on UI Admin stuff * Straight from client ->regionserver * No more heartbeat piggyback Scaling documentation * More conf settings for block cache * How to adjust knobs
