Only answer to the above observation is demonstration that ITBLL with meta not on master is as robust as runs that have master carrying meta.
Agree that this is a prerequisite. Another useful measure might be the delay before an assignment under load is visible to clients. > > But making RPC work for assignment here is precisely the problem. There's > no reason master should have to contend with user requests to meta in order > to be able to make updates. And until clients can actually see the change, > it doesn't really matter if the master state has been updated or not. > > In hbase-2.0.0, there'll be a new regime. hbase:meta writing will be single writer by master only. No more contention on writes. Regards contention reading, this is unavoidable. In hbase-2.0.0, only the final publishing step, what we want clients to see, will update hbase:meta. All other transitions will be internal. There is still contention between readers and writers over available handler threads. With a local meta region, you don't have assignment manager having to contend for handler threads in order to perform writes. This is huge for reliability. Without meta on master, it has not been hard to reproduce scenarios where HBase _cannot_ start up from a cold roll with high client traffic. Region assignments just can not complete because master has to compete with all the clients attempting to read new region locations from meta and can't get in the queue. With meta on master, this goes away completely. > > > > > > Above is fine except for the bit where we need to be able to field reads. > > Lets distribute the data to be read over the cluster rather than treat > meta > > reads with kid gloves hosted on a 'special' server; let these 'reads' be > > like any other read the cluster takes (see next point) > > > > > In my opinion, the real "special" part here is the master bit -- which I > think we should be working to make less special and more just a normal bit > of housekeeping spread across nodes -- not the regionserver role. It only > looks special right now because the evolution has stopped in the middle. I > really don't think enshrining master as a separate process is the right way > forward for us. > > I always liked this notion. To be worked out is how Master and hbase:meta hosting would interplay (The RS that is designated Master would also host hbase:meta? Would it be exclusively hosting hbase;meta or hbase:meta would move with Master function.... [Stuff we've talked about before]). I think meta should be tied to the master function for the reasons described above. It's key that the updates to meta be local. I don't think that has to mean that only meta regions are hosted by the regionserver serving as master. There could be other user regions hosted as well, given the server has adequate headroom to handle the master functions. > > > > > Is this just because meta had a dedicated server? > > > > > I'm sure that having dedicated resources for meta helps. But I don't think > that's sufficient. The key is that master writes to meta are local, and do > not have to contend with the user requests to meta. > > It seems premature to be discussing dropping a working implementation which > eliminates painful parts of distributed consensus, until we have a complete > working alternative to evaluate. Until then, why are we looking at > features that are in use and work well? > > > How to move forward here? The Pv2 master is almost done. An ITBLL bakeoff of new Pv2 based assign vs a Master that exclusively hosts hbase:meta? I think that's a necessary test for proving out the new AM implementation. But remember that we are comparing a feature which is actively supporting production workloads with a line of active development. I think there should also be additional testing around situations of high meta load and end-to-end assignment latency.
