On Thu, Nov 28, 2024 at 7:55 PM Bryan Beaudreault <bbeaudrea...@apache.org> wrote: > > I think it’s important to exclude meta. That is critical to many aspects of > client and server. That probably extends to namespace table too. > > That said, it would be nice for quotas to cover the backup tables and > actually the quota table itself. We recently had an issue where the > regionservers all ddos’d the server holding the quota table because of > refreshes. I would rather the quota refresh calls backoff or fail since we > have an in memory cache, or even fall back to no quota briefly. We’ve > similarly seen the backup table getting hotspot when backups fall behind > and are constantly verifying.
All of this consideration is valuable for request quotas. I was surprised to find quota enforcement besides requests, in this case, region count limits. Let's try to sure up some of these systems. Thanks, Nick > On Thu, Nov 28, 2024 at 1:41 PM Daniel Roudnitsky (BLOOMBERG/ 919 3RD A) < > droudnits...@bloomberg.net> wrote: > > > > I'm also curious, is there no way to limit quota enforcement to just > > > the type of quotas that are desired? In this case, We only care about > > > throttling requests and so I'm surprised to see that namespace limits > > > are being checked. > > +1 to limit quota enforcement to only the enabled quotas. Clusters with > > only request quotas don't need NamespaceStateManager checks (and associated > > issues with NamespaceStateManager), and those with only namespace quotas > > don't need QuotaCache / QuotaRefresherChore running on all region servers. > > > > If your quota region split failed due to NPE in > > SPLIT_TABLE_REGION_PRE_OPERATION you may want to check if you were impacted > > by https://issues.apache.org/jira/browse/HBASE-28533 > > > > From: dev@hbase.apache.org At: 11/28/24 09:21:08 UTC-5:00To: > > dev@hbase.apache.org > > Subject: Re: Quotas on System tables? > > > > This implies to me that the original intention was to not perform > > quota enforcement on system tables. > > > > > > https://github.com/apache/hbase/blob/63cdd026f08cdde6ac0fde1342ffd050e8e02441/hb > > > > ase-server/src/main/java/org/apache/hadoop/hbase/namespace/NamespaceStateManager > > .java#L214 > > <https://github.com/apache/hbase/blob/63cdd026f08cdde6ac0fde1342ffd050e8e02441/hbase-server/src/main/java/org/apache/hadoop/hbase/namespace/NamespaceStateManager.java#L214> > > > > On Thu, Nov 28, 2024 at 2:55 PM Nick Dimiduk <ndimi...@apache.org> wrote: > > > > > > Heya, > > > > > > We recently tried to split the hbase:quota table and were surprised to > > > hit an NPE. I was able to reproduce this with a simple test, I've > > > pasted the stack trace below. > > > > > > Looking into the nature of this failure, I'm left scratching my head. > > > It appears that hbase has some quota enforcement system around the > > > number of regions permitted in a namespace. When the split request > > > comes in, the NamespaceAuditor is checked and I get this failure, > > > apparently because there's no entry for the hbase:quotas table in the > > > NamespaceTableAndRegionInfo object. > > > > > > I can pursue a fix of the bug via some basic null-handling or ensure > > > this structure is populated on initialization. However, I'm wondering > > > if it is intentional that we enforce any kind of quotas on the hbase > > > system namespace at all. This seems like a bug to me. > > > > > > I'm also curious, is there no way to limit quota enforcement to just > > > the type of quotas that are desired? In this case, We only care about > > > throttling requests and so I'm surprised to see that namespace limits > > > are being checked. > > > > > > Thanks, > > > Nick > > > > > > 2024-11-28T14:11:38,672 ERROR [PEWorker-2 {}] > > > procedure2.ProcedureExecutor(1844): CODE-BUG: Uncaught runtime > > > exception: pid=7, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, > > > hasLock=true; SplitTableRegionProcedure table=hbase:quota, > > > parent=3c1dbd551845ed2e8033b76570c6d8f6, > > > daughterA=a8f5e70728d93859d7c3d95aa5a0f63b, > > > daughterB=c71f49ad7e689a57742a9fe65b918d08 > > > java.lang.NullPointerException: Cannot invoke > > > "java.util.concurrent.atomic.AtomicInteger.addAndGet(int)" because the > > > return value of "java.util.Map.get(Object)" is null > > > at > > > > org.apache.hadoop.hbase.namespace.NamespaceTableAndRegionInfo.incRegionCountForT > > able(NamespaceTableAndRegionInfo.java:101) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.namespace.NamespaceStateManager.checkAndUpdateNamespaceR > > egionCount(NamespaceStateManager.java:95) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.namespace.NamespaceAuditor.checkQuotaToSplitRegion(Names > > paceAuditor.java:116) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.quotas.MasterQuotaManager.onRegionSplit(MasterQuotaManag > > er.java:544) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.preSplitRegi > > on(SplitTableRegionProcedure.java:607) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromS > > tate(SplitTableRegionProcedure.java:298) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromS > > tate(SplitTableRegionProcedure.java:98) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachinePro > > cedure.java:188) > > > ~[classes/:?] > > > at > > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:941) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExec > > utor.java:1825) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureE > > xecutor.java:1503) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(P > > rocedureExecutor.java:2154) > > > ~[classes/:?] > > > at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) > > > ~[classes/:?] > > > at > > > > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureE > > xecutor.java:2181) > > > ~[classes/:?] > > > > > >
