[
https://issues.apache.org/jira/browse/HBASE-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huginn updated HBASE-30393:
---------------------------
Description:
SimpleRpcServer can retain a stale response timestamp on a connection after a
queued RPC response has been fully written. If a later response on the same
connection cannot make progress, the stale timestamp can make the connection
appear to have been blocked longer than the current response.
When SimpleRpcServerResponder.processResponse writes a response in multiple
attempts, it records lastSentTime while the response is incomplete. The
timestamp is not cleared when the response completes, and a non-blocking write
that makes no progress can refresh the timestamp. The responder purge path can
then close the connection based on stale response state, causing RPC failures
and retries.
The fix is to clear lastSentTime when the current response is fully written,
preserve or update it only while an incomplete response remains, skip purge for
connections with an empty response queue, and make the timestamp safely visible
between the responder and purge paths.
Reproduction and validation:
* Queue a response that is partially written, complete it, then queue a
response whose non-blocking write makes no progress on the same connection.
* Before the fix, TestSimpleRpcServerResponder reproduces stale timestamp
retention, timestamp refresh on a stalled write, and purge of a connection with
an empty response queue.
* After the fix, all three regression tests pass; the related IPC regression
tests and the hbase-server package build also pass.
was:
## What happens
SimpleRpcServer can retain a stale response timestamp on a connection after a
queued RPC response has been fully written. If the connection later has another
response that cannot make progress, the stale timestamp can make the connection
appear to have been blocked longer than the current response.
## When it happens
When SimpleRpcServer sends responses through its direct-write and responder
paths, a response can be partially written and later completed on the same
connection. The responder's purge check uses the connection response timestamp
to identify calls that have not been sent for a long time.
## Impact
A client connection can be closed by the response purge path, causing RPC
failures and retries.
## Root cause
In SimpleRpcServerResponder.processResponse, lastSentTime is set when a
response remains partially written but is not cleared after the response has no
remaining bytes. A later stalled response on the same connection can therefore
inherit a timestamp from an earlier response.
## Proposed fix
Clear the response timestamp when the current response is fully written.
Preserve the timestamp while a response is making no progress, update it when a
stalled response first starts or makes progress, and add regression coverage
for completion and purge behavior.
## Reproduction
Testing evidence will be added by the reporter.
> Clear stale RPC response timestamps after responses complete
> ------------------------------------------------------------
>
> Key: HBASE-30393
> URL: https://issues.apache.org/jira/browse/HBASE-30393
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
> Affects Versions: 2.4.11
> Reporter: huginn
> Assignee: huginn
> Priority: Major
> Labels: pull-request-available
>
> SimpleRpcServer can retain a stale response timestamp on a connection after a
> queued RPC response has been fully written. If a later response on the same
> connection cannot make progress, the stale timestamp can make the connection
> appear to have been blocked longer than the current response.
> When SimpleRpcServerResponder.processResponse writes a response in multiple
> attempts, it records lastSentTime while the response is incomplete. The
> timestamp is not cleared when the response completes, and a non-blocking
> write that makes no progress can refresh the timestamp. The responder purge
> path can then close the connection based on stale response state, causing RPC
> failures and retries.
> The fix is to clear lastSentTime when the current response is fully written,
> preserve or update it only while an incomplete response remains, skip purge
> for connections with an empty response queue, and make the timestamp safely
> visible between the responder and purge paths.
> Reproduction and validation:
> * Queue a response that is partially written, complete it, then queue a
> response whose non-blocking write makes no progress on the same connection.
> * Before the fix, TestSimpleRpcServerResponder reproduces stale timestamp
> retention, timestamp refresh on a stalled write, and purge of a connection
> with an empty response queue.
> * After the fix, all three regression tests pass; the related IPC regression
> tests and the hbase-server package build also pass.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
