I think it’s important to exclude meta. That is critical to many aspects of client and server. That probably extends to namespace table too.
That said, it would be nice for quotas to cover the backup tables and actually the quota table itself. We recently had an issue where the regionservers all ddos’d the server holding the quota table because of refreshes. I would rather the quota refresh calls backoff or fail since we have an in memory cache, or even fall back to no quota briefly. We’ve similarly seen the backup table getting hotspot when backups fall behind and are constantly verifying. On Thu, Nov 28, 2024 at 1:41 PM Daniel Roudnitsky (BLOOMBERG/ 919 3RD A) < droudnits...@bloomberg.net> wrote: > > I'm also curious, is there no way to limit quota enforcement to just > > the type of quotas that are desired? In this case, We only care about > > throttling requests and so I'm surprised to see that namespace limits > > are being checked. > +1 to limit quota enforcement to only the enabled quotas. Clusters with > only request quotas don't need NamespaceStateManager checks (and associated > issues with NamespaceStateManager), and those with only namespace quotas > don't need QuotaCache / QuotaRefresherChore running on all region servers. > > If your quota region split failed due to NPE in > SPLIT_TABLE_REGION_PRE_OPERATION you may want to check if you were impacted > by https://issues.apache.org/jira/browse/HBASE-28533 > > From: dev@hbase.apache.org At: 11/28/24 09:21:08 UTC-5:00To: > dev@hbase.apache.org > Subject: Re: Quotas on System tables? > > This implies to me that the original intention was to not perform > quota enforcement on system tables. > > > https://github.com/apache/hbase/blob/63cdd026f08cdde6ac0fde1342ffd050e8e02441/hb > > ase-server/src/main/java/org/apache/hadoop/hbase/namespace/NamespaceStateManager > .java#L214 > <https://github.com/apache/hbase/blob/63cdd026f08cdde6ac0fde1342ffd050e8e02441/hbase-server/src/main/java/org/apache/hadoop/hbase/namespace/NamespaceStateManager.java#L214> > > On Thu, Nov 28, 2024 at 2:55 PM Nick Dimiduk <ndimi...@apache.org> wrote: > > > > Heya, > > > > We recently tried to split the hbase:quota table and were surprised to > > hit an NPE. I was able to reproduce this with a simple test, I've > > pasted the stack trace below. > > > > Looking into the nature of this failure, I'm left scratching my head. > > It appears that hbase has some quota enforcement system around the > > number of regions permitted in a namespace. When the split request > > comes in, the NamespaceAuditor is checked and I get this failure, > > apparently because there's no entry for the hbase:quotas table in the > > NamespaceTableAndRegionInfo object. > > > > I can pursue a fix of the bug via some basic null-handling or ensure > > this structure is populated on initialization. However, I'm wondering > > if it is intentional that we enforce any kind of quotas on the hbase > > system namespace at all. This seems like a bug to me. > > > > I'm also curious, is there no way to limit quota enforcement to just > > the type of quotas that are desired? In this case, We only care about > > throttling requests and so I'm surprised to see that namespace limits > > are being checked. > > > > Thanks, > > Nick > > > > 2024-11-28T14:11:38,672 ERROR [PEWorker-2 {}] > > procedure2.ProcedureExecutor(1844): CODE-BUG: Uncaught runtime > > exception: pid=7, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, > > hasLock=true; SplitTableRegionProcedure table=hbase:quota, > > parent=3c1dbd551845ed2e8033b76570c6d8f6, > > daughterA=a8f5e70728d93859d7c3d95aa5a0f63b, > > daughterB=c71f49ad7e689a57742a9fe65b918d08 > > java.lang.NullPointerException: Cannot invoke > > "java.util.concurrent.atomic.AtomicInteger.addAndGet(int)" because the > > return value of "java.util.Map.get(Object)" is null > > at > > org.apache.hadoop.hbase.namespace.NamespaceTableAndRegionInfo.incRegionCountForT > able(NamespaceTableAndRegionInfo.java:101) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.namespace.NamespaceStateManager.checkAndUpdateNamespaceR > egionCount(NamespaceStateManager.java:95) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.namespace.NamespaceAuditor.checkQuotaToSplitRegion(Names > paceAuditor.java:116) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.quotas.MasterQuotaManager.onRegionSplit(MasterQuotaManag > er.java:544) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.preSplitRegi > on(SplitTableRegionProcedure.java:607) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromS > tate(SplitTableRegionProcedure.java:298) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromS > tate(SplitTableRegionProcedure.java:98) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachinePro > cedure.java:188) > > ~[classes/:?] > > at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:941) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExec > utor.java:1825) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureE > xecutor.java:1503) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(P > rocedureExecutor.java:2154) > > ~[classes/:?] > > at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) > > ~[classes/:?] > > at > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureE > xecutor.java:2181) > > ~[classes/:?] > > >
