rmdmattingly opened a new pull request, #7396: URL: https://github.com/apache/hbase/pull/7396
We've been using this at HubSpot quite successfully, and would like to contribute it. Throttling has been great for us at HubSpot — we've written about the success [here](https://product.hubspot.com/blog/hbase-share-resources). But we've had a couple of problems at scale: 1. Users often fail to consistently utilize a meaningful proportion of their throttle. Due to inefficiencies in backoff/retry, and spikes of trying and backing off in tandem around refill intervals, users are often confused by both a steady state utilization of 20-70% of their configured allowance, and a steady state of throttling exceptions. 2. Also, counter-intuitively but depending on the shape of the workload, we have the opposite problem. Access patterns with a ton of machines and a ton of threads all competing for the same contentious throttle allowance may inadvertently DDOS the RegionServer's RPC layer with thousands upon thousands of doomed, redundant retries. This is due to the naive way in which we calculate the RpcThrottlingException's wait interval — we simply assume that the next sufficient refill will be an opportune time to retry, which only really makes sense in a single-threaded environment To fix this, we've implemented a **FeedbackAdaptiveRateLimiter**, inspired by Philipp Janert's _[Feedback Control for Computer Systems](https://www.barnesandnoble.com/w/feedback-control-for-computer-systems-philipp-janert/1116936673?ean=9781449361693)_ The FeedbackAdaptiveRateLimiter works much like the FixedIntervalRateLimiter, but with some additional logic to support dynamic wait interval multiplication and modest oversubscription so as to drive more consistent, more full quota utilization while only serving a fraction of the previous RpcThrottlingException volume. The additional FARL logic can be described with the following categories: **1. Closed-Loop Feedback Control** The limiter implements a classic closed-loop control system where: - Setpoint: The configured resource limit (e.g., 10 requests/second) - Process Variable: Actual resource consumption over time - Control Actions: Adjusting backoff multipliers and oversubscription proportions - Feedback: Monitoring contention and utilization metrics to adapt behavior **2. Proportional Control with Integral Behavior** The implementation uses two separate control mechanisms: Backoff Multiplier Control (`hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FeedbackAdaptiveRateLimiter.java:221-227`): ``` if (hadContentionThisInterval) { currentBackoffMultiplier.set(Math.min( currentBackoffMultiplier.get() + backoffMultiplierIncrement, maxBackoffMultiplier)); } else { currentBackoffMultiplier.set(Math.max( currentBackoffMultiplier.get() - backoffMultiplierDecrement, 1.0)); } ``` This is essentially an integral controller that accumulates error over time—increasing pressure when contention is detected, decreasing it when there's none. Oversubscription Control (`hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FeedbackAdaptiveRateLimiter.java:229-236`): ``` if (avgUtil < minTargetUtilization) { oversubscriptionProportion.set(Math.min( oversubscriptionProportion.get() + oversubscriptionIncrement, maxOversubscription)); } else if (avgUtil >= maxTargetUtilization) { oversubscriptionProportion.set(Math.max( oversubscriptionProportion.get() - oversubscriptionDecrement, 0.0)); } ``` **3. Exponential Moving Average (EMA) for Smoothing** The system uses EMA to track utilization (`hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FeedbackAdaptiveRateLimiter.java:216-218`): double util = (double) consumed / intendedUsage; utilizationEma = emaAlpha * util + (1.0 - emaAlpha) * utilizationEma; This is a standard signal processing technique from control theory to filter out noise and respond smoothly to changes in system behavior. **4. Saturation Limits (Anti-Windup)** The implementation includes caps on both control parameters: - maxBackoffMultiplier (default: 10.0) - maxOversubscription (default: 0.25) This prevents "integral windup"—a common problem in control systems where the controller overshoots dramatically. **5. Error Budget / Deadband** The utilizationErrorBudget parameter (`hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FeedbackAdaptiveRateLimiter.java:95-97`) creates a deadband around the target utilization (1.0): - Target range: [0.975, 1.025] (with default 0.025 error budget) - No control action taken when utilization is within this range - This prevents oscillation around the setpoint—a key stability consideration in control theory **6. Dual-Control Strategy** The system elegantly addresses two different control objectives: 1. Fast Response (Backoff Multiplier): Responds immediately to contention on a per-interval basis 2. Long-term Optimization (Oversubscription): Uses the EMA to gradually tune the system for optimal steady-state utilization This mirrors the proportional + integral (PI) control pattern where you need both fast response to disturbances and elimination of steady-state error. **Why This Design?** The traditional fixed-interval rate limiter suffers from two problems this feedback control approach solves: 1. Thundering herd: Multiple threads hitting the limiter simultaneously cause spikes 2. Under-utilization: Conservative limits lead to wasted capacity By applying control theory: - The backoff multiplier increases backpressure to spread out concurrent requests (damping) - The oversubscription allows slight over-limit to achieve full utilization without violating limits on average - The EMA provides stability by not reacting to transient spikes In practice, we have deployed this with these default settings across all of our hundreds of clusters at HubSpot with great success, powering everything from live user-facing requests to async batch jobs. See our reduced RpcThrottlingException volume below (this is the average of our top _n_ RegionServers by RTE/sec): <img width="606" height="409" alt="Screenshot 2025-10-17 at 10 06 44 AM" src="https://github.com/user-attachments/assets/1c63bea2-c5fe-4522-9249-14891abf22e7"; /> And with the variety of configuration levers available in the FARL, you can truly adapt the rate limiter to appease whatever your priorities might be (strict backoffs, lenient oversubscription). In combination with [HBASE-29663](https://issues.apache.org/jira/browse/HBASE-29663), which made rate limiter configurations dynamically refreshable, this is a powerful combination for improving the usability & scalability of HBase's Quotas system. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
