[
https://issues.apache.org/jira/browse/HBASE-28620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098097#comment-18098097
]
mazhengxuan commented on HBASE-28620:
-------------------------------------
Hi, I would like to take ownership of this issue and continue the fix.
I reviewed PR #5949 together with the reproduction reported in HBASE-29786.
HBASE-29786 appears to be another occurrence of the same problem on 2.6.2.
The current return-to-break change in #5949 is not safe by itself, because
clearWALEntryBatch may drain the queue while the reader or shipper is still
alive. The reader may enqueue another batch after the drain, and the shipper
may already own an in-flight batch which is no longer present in the queue.
My proposed fix is to make quota cleanup follow batch ownership:
1. The reader releases its current batch on an unsuccessful enqueue, as it
already does, and drains the remaining queue when it exits due to source
termination.
2. The shipper releases a polled batch in a finally block if the batch was not
successfully shipped.
3. Successfully shipped batches continue to release quota through postShipEdits.
This avoids resetting the global totalBufferUsed value and does not depend on
increasing the cleanup timeout. I will also add a deterministic latch-based
regression test covering both an in-flight batch and a queued batch during peer
refresh.
Unless there are objections, I will prepare an updated patch against current
master and then backport it to branch-2.6.
> replication quota leak when peer changes
> ----------------------------------------
>
> Key: HBASE-28620
> URL: https://issues.apache.org/jira/browse/HBASE-28620
> Project: HBase
> Issue Type: Bug
> Components: Replication
> Reporter: WangXin
> Priority: Critical
> Labels: pull-request-available
>
> When peer changes, replication closes the reader and shipper created earlier.
> However, after the specified timeout, the shipper still does not
> automatically close (It was interrupted, but it didn't close properly). The
> existing code simply returns without releasing quota. Not cleaning buffer
> usage.
> In one practice of my company, in this case, the quota was full because it
> was not released in time, so wal reader could not continue read new data and
> replication had a backlog.
>
> The log is as follows:
> 2024-05-20 20:00:00,796 WARN
> [RpcServer.default.FPRWQ.Fifo.read.handler=70,queue=1,port=16020]
> regionserver.ReplicationSourceShipper: Shipper clearWALEntryBatch method
> timed out whilst waiting reader/shipper thread to stop. Not cleaning buffer
> usage. Shipper alive: peer1; Reader alive: false
> 2024-05-20 20:00:01,351 WARN peer=peer1, can't read more edits from WAL as
> buffer usage 268435456B exceeds limit 268435456B
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
