[
https://issues.apache.org/jira/browse/HBASE-30467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Mattingly updated HBASE-30467:
----------------------------------
Description:
h2. Summary
Scanner close requests can be throttled by HBase Quotas like any other client
RPC. This opens up the door for a worker to accidentally cause server-side
memory leaks of scanners when they are hitting their quotas. I observed this
phenomenon at my day job on a lightweight fork of 2.6.3, where we run hundreds
of HBase clusters and use quotas heavily to ensure quality of service.
h2. Problem
When a client is throttled, any close requests are rejected before they're
processed. In RSRpcServices.scan(), checkQuotaAndGetRegionScannerContext()
calls getRegionScanner() and then RegionServerRpcQuotaManager.checkScanQuota()
for any request with a scannerId, including close-only requests
(closeScanner=true, numberOfRows=0). DefaultOperationQuota.checkScanQuota() has
no exemption, so a throttled user gets RpcThrottlingException and the
closeScanner branch is never reached.
On the client, ScannerCallable.close() sends the close once without retrying
and logs "Ignore, probably already closed", then discards the scanner id. The
RegionScannerHolder stays in scanners until ScannerListener.leaseExpired()
fires after hbase.client.scanner.timeout.period.
The result is that throttling meant to protect the RS increases RS memory
pressure.
h2. Proposed Fix
We can just skip quota checks (use NoopOperationQuota) for close-only scan
requests in checkQuotaAndGetRegionScannerContext(). A close does no reads, so
exempting it can't be used to get around the quota.
was:
## Summary
Scanner close requests can be throttled by HBase Quotas like any other client
RPC. This opens up the door for a worker to accidentally cause server-side
memory leaks of scanners when they are hitting their quotas. I observed this
phenomenon at my day job on a lightweight fork of 2.6.3, where we run hundreds
of HBase clusters and use quotas heavily to ensure quality of service.
## Problem
When a client is throttled, any close requests are rejected before they're
processed. In RSRpcServices.scan(), checkQuotaAndGetRegionScannerContext()
calls getRegionScanner() and then RegionServerRpcQuotaManager.checkScanQuota()
for any request with a scannerId, including close-only requests
(closeScanner=true, numberOfRows=0). DefaultOperationQuota.checkScanQuota() has
no exemption, so a throttled user gets RpcThrottlingException and the
closeScanner branch is never reached.
On the client, ScannerCallable.close() sends the close once without retrying
and logs "Ignore, probably already closed", then discards the scanner id. The
RegionScannerHolder stays in scanners until ScannerListener.leaseExpired()
fires after hbase.client.scanner.timeout.period.
The result is that throttling meant to protect the RS increases RS memory
pressure.
## Proposed Fix
We can just skip quota checks (use NoopOperationQuota) for close-only scan
requests in checkQuotaAndGetRegionScannerContext(). A close does no reads, so
exempting it can't be used to get around the quota.
> Scanner close requests should be exempt from quotas
> ---------------------------------------------------
>
> Key: HBASE-30467
> URL: https://issues.apache.org/jira/browse/HBASE-30467
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 2.6.3
> Reporter: Ray Mattingly
> Priority: Major
>
> h2. Summary
> Scanner close requests can be throttled by HBase Quotas like any other client
> RPC. This opens up the door for a worker to accidentally cause server-side
> memory leaks of scanners when they are hitting their quotas. I observed this
> phenomenon at my day job on a lightweight fork of 2.6.3, where we run
> hundreds of HBase clusters and use quotas heavily to ensure quality of
> service.
> h2. Problem
> When a client is throttled, any close requests are rejected before they're
> processed. In RSRpcServices.scan(), checkQuotaAndGetRegionScannerContext()
> calls getRegionScanner() and then
> RegionServerRpcQuotaManager.checkScanQuota() for any request with a
> scannerId, including close-only requests (closeScanner=true, numberOfRows=0).
> DefaultOperationQuota.checkScanQuota() has no exemption, so a throttled user
> gets RpcThrottlingException and the closeScanner branch is never reached.
> On the client, ScannerCallable.close() sends the close once without retrying
> and logs "Ignore, probably already closed", then discards the scanner id. The
> RegionScannerHolder stays in scanners until ScannerListener.leaseExpired()
> fires after hbase.client.scanner.timeout.period.
> The result is that throttling meant to protect the RS increases RS memory
> pressure.
> h2. Proposed Fix
> We can just skip quota checks (use NoopOperationQuota) for close-only scan
> requests in checkQuotaAndGetRegionScannerContext(). A close does no reads, so
> exempting it can't be used to get around the quota.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
