On Thu, Nov 28, 2024 at 7:41 PM Daniel Roudnitsky (BLOOMBERG/ 919 3RD A) <droudnits...@bloomberg.net> wrote: > > +1 to limit quota enforcement to only the enabled quotas. Clusters with only > request quotas don't need NamespaceStateManager checks (and associated issues > with NamespaceStateManager), and those with only namespace quotas don't need > QuotaCache / QuotaRefresherChore running on all region servers.
nod > If your quota region split failed due to NPE in > SPLIT_TABLE_REGION_PRE_OPERATION you may want to check if you were impacted > by https://issues.apache.org/jira/browse/HBASE-28533 Yes, we did observe this at the moment. I was able to confirm via scanning the meta table that the region was still in the OPEN state, so we rolled the active master. Thanks for the pointer and the fix! Thanks, Nick > From: dev@hbase.apache.org At: 11/28/24 09:21:08 UTC-5:00To: > dev@hbase.apache.org > Subject: Re: Quotas on System tables? > > This implies to me that the original intention was to not perform > quota enforcement on system tables. > > https://github.com/apache/hbase/blob/63cdd026f08cdde6ac0fde1342ffd050e8e02441/hb > ase-server/src/main/java/org/apache/hadoop/hbase/namespace/NamespaceStateManager > .java#L214 > > On Thu, Nov 28, 2024 at 2:55 PM Nick Dimiduk <ndimi...@apache.org> wrote: > > > > Heya, > > > > We recently tried to split the hbase:quota table and were surprised to > > hit an NPE. I was able to reproduce this with a simple test, I've > > pasted the stack trace below. > > > > Looking into the nature of this failure, I'm left scratching my head. > > It appears that hbase has some quota enforcement system around the > > number of regions permitted in a namespace. When the split request > > comes in, the NamespaceAuditor is checked and I get this failure, > > apparently because there's no entry for the hbase:quotas table in the > > NamespaceTableAndRegionInfo object. > > > > I can pursue a fix of the bug via some basic null-handling or ensure > > this structure is populated on initialization. However, I'm wondering > > if it is intentional that we enforce any kind of quotas on the hbase > > system namespace at all. This seems like a bug to me. > > > > I'm also curious, is there no way to limit quota enforcement to just > > the type of quotas that are desired? In this case, We only care about > > throttling requests and so I'm surprised to see that namespace limits > > are being checked. > > > > Thanks, > > Nick > > > > 2024-11-28T14:11:38,672 ERROR [PEWorker-2 {}] > > procedure2.ProcedureExecutor(1844): CODE-BUG: Uncaught runtime > > exception: pid=7, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, > > hasLock=true; SplitTableRegionProcedure table=hbase:quota, > > parent=3c1dbd551845ed2e8033b76570c6d8f6, > > daughterA=a8f5e70728d93859d7c3d95aa5a0f63b, > > daughterB=c71f49ad7e689a57742a9fe65b918d08 > > java.lang.NullPointerException: Cannot invoke > > "java.util.concurrent.atomic.AtomicInteger.addAndGet(int)" because the > > return value of "java.util.Map.get(Object)" is null > > at > org.apache.hadoop.hbase.namespace.NamespaceTableAndRegionInfo.incRegionCountForT > able(NamespaceTableAndRegionInfo.java:101) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.namespace.NamespaceStateManager.checkAndUpdateNamespaceR > egionCount(NamespaceStateManager.java:95) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.namespace.NamespaceAuditor.checkQuotaToSplitRegion(Names > paceAuditor.java:116) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.quotas.MasterQuotaManager.onRegionSplit(MasterQuotaManag > er.java:544) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.preSplitRegi > on(SplitTableRegionProcedure.java:607) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromS > tate(SplitTableRegionProcedure.java:298) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromS > tate(SplitTableRegionProcedure.java:98) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachinePro > cedure.java:188) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:941) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExec > utor.java:1825) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureE > xecutor.java:1503) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(P > rocedureExecutor.java:2154) > > ~[classes/:?] > > at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureE > xecutor.java:2181) > > ~[classes/:?] > >
