rmdmattingly opened a new pull request, #8585: URL: https://github.com/apache/hbase/pull/8585
https://issues.apache.org/jira/browse/HBASE-30348 ### Problem Throttles are silently unenforceable on any system table. `RegionServerRpcQuotaManager#getQuota` short-circuits to `NoopOperationQuota` when `TableName#isSystemTable()` is true, and `QuotaCache#getUserLimiter` returns `NoopQuotaLimiter` under the same condition. Because `isSystemTable()` is set for every table in the `hbase` namespace *and* every table in the `backup` namespace, no quota of any kind — user, table, or namespace — can throttle those tables. The quota is accepted, stored, and displayed by `list_quotas`, but the request path never consults a limiter, so nothing is ever enforced and nothing warns the operator. We hit this in production. Reads against `backup:system` were saturating a RegionServer's disk IO. A namespace `READ_SIZE` throttle on `backup`, then a table-level `READ_SIZE` throttle, then a table-level `REQUEST_NUMBER` throttle of 20 req/s were applied in sequence, and all three had exactly zero measurable effect — no change in throttling exceptions, no change in IO. There was no signal anywhere that the quotas could not apply. ### Why the exemption exists, and why it is too broad The exemption guards against a deadlock in the throttling machinery itself: quotas are loaded from `hbase:quota`, and clients resolve region locations through `hbase:meta`. Throttling either table can prevent a throttle from ever being lifted. That reasoning covers those two tables and does not extend to the rest of the `hbase` namespace, and certainly not to the `backup` tables, whose traffic is ordinary read/write load that operators have a legitimate need to bound. ### Change - Add `QuotaUtil#isThrottleExempt(TableName)`, which returns true only for `hbase:meta` and `hbase:quota`. - Use it in place of `TableName#isSystemTable()` in `RegionServerRpcQuotaManager#getQuota` and `QuotaCache#getUserLimiter`. Every other table becomes throttleable, including the remaining `hbase` namespace tables and the backup tables. ### Compatibility This is a behavior change, but only for operators who have already configured a quota that matches a system table. Nothing is throttled by default; a quota must be explicitly set. Two cases are worth calling out: - A namespace throttle on `hbase` or `backup` now takes effect where it previously did not. - A user throttle with no table/namespace filter now also covers that user's requests to system tables (other than meta and quota). Both were arguably the operator's intent when the quota was set, but anyone relying on the old silent no-op should review their existing quotas before upgrading. ### Testing Adds `TestQuotaUtil`, covering that meta and quota are exempt and that the backup tables, the other `hbase` namespace tables, and user tables are not. ### Backports Assuming this looks reasonable, I'd like to bring it to `branch-2` and `branch-2.6` as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
