Re: [DISCUSS] No regions on Master node in 2.0

(Trying to tie off this thread...) HBASE-18511 changes the configuration hbase.balancer.tablesOnMaster from a list of table names that the master can carry (with 'none' meaning no tables on the master) to instead be a boolean as to whether the Master carries tables/regions or not. When true, the

Re: [DISCUSS] No regions on Master node in 2.0

I still have to review the full AMv2 meta updates path to see whether there may still be "split brain" due to the extra RPC to a remote server. But I really like the notion of keeping the deployment topology of branch-1 by default. The fact is that 2.0 is already lagging, and minimizing the set

Re: [DISCUSS] No regions on Master node in 2.0

> That doesn't solve the same problem.Agreed as mentioned regionserver groups >only provides user-system region isolation.  > That still means that the most important operations are competing for rpc >queue time.Given the previous setup. For meta access contention this should be >addressed by

Re: [DISCUSS] No regions on Master node in 2.0

Pardon my imprecision. Will ensure carrying system tables short circuits RPC. S On Jun 6, 2017 9:17 AM, "Elliott Clark" wrote: That doesn't solve the same problem. Dedicating a remote server for the system tables still means that all the master to system tables mutations

Re: [DISCUSS] No regions on Master node in 2.0

That doesn't solve the same problem. Dedicating a remote server for the system tables still means that all the master to system tables mutations and reads are done over rpc. That still means that the most important operations are competing for rpc queue time. On Fri, Nov 18, 2016 at 11:37 AM,

Re: [DISCUSS] No regions on Master node in 2.0

Back to this hanging thread full of goodness. I'm here now as your RM for hbase2 trying to extract the minimum set of deploy forms we need to support. On a reread of the above, I am thinking the hbase2 default is to have the same shape as hbase1 with Master carrying NO regions (See operators

Re: [DISCUSS] No regions on Master node in 2.0

Just some extra bits of information: Another way to isolate user regions from meta is you can create a regionserver group (HBASE-6721) dedicated to the system tables. This is what we do at Y!. If the load on meta gets too high (and it does), we split meta so the load gets spread across more

Re: [DISCUSS] No regions on Master node in 2.0

On Wed, Nov 16, 2016 at 10:44 PM, Stack wrote: > On Wed, Nov 16, 2016 at 10:57 AM, Gary Helmling > wrote: > >> >> Do you folks run the meta-carrying-master form G? > > Pardon me. I missed a paragraph. I see you folks do deploy this form. St.Ack >

Re: [DISCUSS] No regions on Master node in 2.0

On Wed, Nov 16, 2016 at 10:57 AM, Gary Helmling wrote: > Only answer to the above observation is demonstration that ITBLL with meta > not on master is as robust as runs that have master carrying meta. > > > Agree that this is a prerequisite. Another useful measure might be

Re: [DISCUSS] No regions on Master node in 2.0

On Wed, Nov 16, 2016 at 11:36 AM Ted Yu wrote: > HBASE-16025 is currently marked Critical. > > I think it should be a blocker. > > Done.

Re: [DISCUSS] No regions on Master node in 2.0

HBASE-16025 is currently marked Critical. I think it should be a blocker. Cheers On Nov 16, 2016, at 11:10 AM, Gary Helmling wrote: >> >> Gary has a JIRA HBASE-16025 which would reduce the load on server hosting >> hbase:meta. > Thanks for bringing this issue up, Ted.

Re: [DISCUSS] No regions on Master node in 2.0

On Tue, Nov 15, 2016 at 11:44 AM wrote: > > In the absence of more information, intuition says master carries meta > to avoid a whole class of problems. > Off-hand I think the class of problems we'll eliminate are problems that > are well understood and being constantly

Re: [DISCUSS] No regions on Master node in 2.0

Only answer to the above observation is demonstration that ITBLL with meta not on master is as robust as runs that have master carrying meta. Agree that this is a prerequisite. Another useful measure might be the delay before an assignment under load is visible to clients. > > But making RPC

Re: [DISCUSS] No regions on Master node in 2.0

@Mikhail: Thank you sir for the reference to HBASE-15654 and good to know the efforts on optimizing meta cache. But in our case there might be plenty of new process launched and accessing hbase at the same time, like at the very beginning of some big yarn batch job or during the failover of

Re: [DISCUSS] No regions on Master node in 2.0

I'd like to echo Yu Li. As an operator/user, it is very helpful to be able to run the masters separately. This allows for hot fixes, but also simplifies operations in times of crisis: if there's any real issue, we can restart the masters at will without any fear of impact. If the plan is to

Re: [DISCUSS] No regions on Master node in 2.0

Gary has a JIRA HBASE-16025 which would reduce the load on server hosting hbase:meta. FYI > On Nov 16, 2016, at 2:13 AM, Yu Li wrote: > > Very late to the party +1 (Smile) > > We also offline discussed standalone meta server here in Alibaba since > we've observed crazily

Re: [DISCUSS] No regions on Master node in 2.0

Hi all, I'm not a commiter, but I really love this dicsussion and I want to chime in. I saw 2 very interesting ideas I'm reading between the lines. The first one was to not have any master, but get on RS making master decisions and therefore not hosting any region (or why not just the META). The

Re: [DISCUSS] No regions on Master node in 2.0

Hosting meta on the machine that doesn't serve user regions would help to ensure that updates to meta have higher chance to succeed. But if that machine isn't Master, then we'd introduce yet one more service role to the deployment. And I'd say that different machine roles/service types required

Re: [DISCUSS] No regions on Master node in 2.0

(side note @Yu - FYI there has been a number of fixes/optimizations to the way we cache and invalidate caches of region locations on the client side in 1.3, see for example HBASE-15658, HBASE-15654). On the topic - Hosting meta on the machine that doesn't serve user regions would help to ensure

Re: [DISCUSS] No regions on Master node in 2.0

Very late to the party +1 (Smile) We also offline discussed standalone meta server here in Alibaba since we've observed crazily high QPS on meta caused by online machine learning workload, and in the discussion we also mentioned pros. and cons. of serving meta on HMaster. Since quite some pros.

Re: [DISCUSS] No regions on Master node in 2.0

> In the absence of more information, intuition says master carries meta to  >avoid a whole class of problems. Off-hand I think the class of problems we'll eliminate are problems that are well understood and being constantly dealt with and hardened to this day (ie puts to a region).   > I think

Re: [DISCUSS] No regions on Master node in 2.0

Thanks Stack and Enis. I concur, it's hard to say for those not intimate with the new code. In the absence of more information, intuition says master carries meta to avoid a whole class of problems. On Fri, Nov 11, 2016 at 3:27 PM, Enis Söztutar wrote: > Thanks Stack for

Re: [DISCUSS] No regions on Master node in 2.0

Thanks Stack for reviving this. How to move forward here? The Pv2 master is almost done. An ITBLL bakeoff > of new Pv2 based assign vs a Master that exclusively hosts hbase:meta? > > I think we have to evaluate whether the new pv2 master works with remote meta updates and the fact that those

Re: [DISCUSS] No regions on Master node in 2.0

(Reviving an old thread that needs resolving before 2.0.0. Does Master carry regions in hbase-2.0.0 or not? A strong argument by one of our biggest users is made below that master hosting hbase:meta can be more robust when updates are local and that we can up the throughput of meta operations if

Re: [DISCUSS] No regions on Master node in 2.0

On Mon, Apr 25, 2016 at 11:20 AM Stack wrote: > On Fri, Apr 8, 2016 at 1:42 AM, Elliott Clark wrote: > > > # Without meta on master, we double assign and lose data. > > > > That is currently a fact that I have seen over and over on multiple > loaded > >

Re: [DISCUSS] No regions on Master node in 2.0

On Tue, Apr 12, 2016 at 11:22 AM, Gary Helmling wrote: > ... > > > Currently in the master branch, HMaster hosts 'system tables': e.g. > > hbase:meta. HMaster is doing more than just gardening the cluster, > > bootstrapping and keeping all up and serving healthy as in

Re: [DISCUSS] No regions on Master node in 2.0

On Fri, Apr 8, 2016 at 8:43 AM, Matteo Bertozzi wrote: > ... > if we hard code meta, we need a special logic to load it and from there > start the bootstrap of the other regions. > then there is no way to switch to multiple metas if someone wants that, > unless we keep

Re: [DISCUSS] No regions on Master node in 2.0

On Fri, Apr 8, 2016 at 1:42 AM, Elliott Clark wrote: > # Without meta on master, we double assign and lose data. > > That is currently a fact that I have seen over and over on multiple loaded > clusters. Some abstract clean up of deployment vs losing data is a > no-brainer for

Re: [DISCUSS] No regions on Master node in 2.0

On Tue, Apr 19, 2016 at 1:52 PM, Francis Liu wrote: > Locality is not as useful here writes still end up being remote by virtue > of hdfs. > Removing one hop is still useful. It's the same reason that for hdfs writes the first copy is local.

Re: [DISCUSS] No regions on Master node in 2.0

Very late to the party IMHO having the master doing only gardening and not become a part of the user access path is a good design and something we should stick to. It's good SoC (ie makes gardening tasks more isolated from user workload). > we double assign and lose data. Given that meta

Re: [DISCUSS] No regions on Master node in 2.0

Sorry to be late to the party here. I'll sprinkle my comments over the thread where they make the most sense. > Currently in the master branch, HMaster hosts 'system tables': e.g. > hbase:meta. HMaster is doing more than just gardening the cluster, > bootstrapping and keeping all up and serving

Re: [DISCUSS] No regions on Master node in 2.0

> This simple distinction of HMaster and RegionServer roles is also what our > users know and have gotten used to so needs to be a good reason to change it > (We can still pursue the single binary that can do HMaster or HRegionServer > role determined at runtime). I have always liked the idea

Re: [DISCUSS] No regions on Master node in 2.0

Let me put it this way: Removing meta fixes no issues seen from day to day operations, but makes worse just about everything that has been an issue on loaded clusters. On Fri, Apr 8, 2016 at 9:05 AM, Elliott Clark wrote: > > On Fri, Apr 8, 2016 at 8:59 AM, Matteo Bertozzi

Re: [DISCUSS] No regions on Master node in 2.0

On Fri, Apr 8, 2016 at 8:59 AM, Matteo Bertozzi wrote: > You are still thinking at meta used as a state machine. > to simplify, meta should just be: region:location > > not being able to access meta only means that we can't publish > to the client the new location of the

Re: [DISCUSS] No regions on Master node in 2.0

You are still thinking at meta used as a state machine. to simplify, meta should just be: region:location not being able to access meta only means that we can't publish to the client the new location of the region. when meta will be available, that location will be published. what you are

Re: [DISCUSS] No regions on Master node in 2.0

Proc v2 can't fix that it's harder to get a write into meta when going over rpc. Our try at qos doesn't fix it. As long as critical meta operations are competing with user requests meta will be unstabla I am absolutely confident that meta on master makes hbase lose less data. The itbll tests bear

Re: [DISCUSS] No regions on Master node in 2.0

I think proc-v2 make things easier than having meta hard coded on master. we just read the wal and we get back to the state we were previously. In this case it doesn't make any difference if meta is on master or remote, if we have one or we have hundred. if we hard code meta, we need a special

Re: [DISCUSS] No regions on Master node in 2.0

One thing I'd like to say is that it makes the master startup much more simpler and realiable to put system tables on master. Even if proc-v2 can solve the problem, it makes things complicated, right? I prefer to be sure that meta is always available, in a consistent state. If we really need to

Re: [DISCUSS] No regions on Master node in 2.0

# Without meta on master, we double assign and lose data. I doubt meta on master solve this problem. This has more to do on the fact that balancer, assignment, split, merge are disjoint operations that are not aware of each other. also those operation in general consist of multiple steps and if

Re: [DISCUSS] No regions on Master node in 2.0

Agree on the performance concerns. IMO we should not hurt the performance of small(maybe normal?) clusters when scaling for huge clusters. And I also agree that the current implementation which allows Master to carry system regions is not good(sorry for the chinglish...). At least, it makes the

Re: [DISCUSS] No regions on Master node in 2.0

# Without meta on master, we double assign and lose data. That is currently a fact that I have seen over and over on multiple loaded clusters. Some abstract clean up of deployment vs losing data is a no-brainer for me. Master assignment, region split, region merge are all risky, and all places

[DISCUSS] No regions on Master node in 2.0

I would like to start a discussion on whether Master should be carrying regions or not. No hurry. I see this thread going on a while and what with 2.0 being a ways out yet, there is no need to rush to a decision. First, some background. Currently in the master branch, HMaster hosts 'system