[
https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-18027:
-----------------------------------
Attachment: HBASE-18027-branch-1.patch
The branch-1 patch is simpler. I keep the logging changes and only the clamping
of replication queue capacity limit.
I did look into changing logic in ReplicationSource to check if we go overlimit
_before_ adding an entry to the replication worklist, but the question then is
what to do with that last read entry. Must be robust to failure. In-memory only
'put back' queue will lose data upon failure. Seeking the reader back to before
the most recent read did not test out to be robust with active chaos.
Instead we try to avoid creating an overlarge RPC by setting the replication
queue capacity limit to the lesser of replication.source.size.capacity or 95%
of the RPC size limit. This is trying a lot harder than we did previously to
avoid the problem.
> Replication should respect RPC size limits when batching edits
> --------------------------------------------------------------
>
> Key: HBASE-18027
> URL: https://issues.apache.org/jira/browse/HBASE-18027
> Project: HBase
> Issue Type: Bug
> Components: Replication
> Affects Versions: 2.0.0, 1.4.0, 1.3.1
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18027-branch-1.patch, HBASE-18027.patch,
> HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch
>
>
> In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in
> batches. We create N lists. N is the minimum of configured replicator
> threads, number of 100-waledit batches, or number of current sinks. Every
> pending entry in the replication context is then placed in order by hash of
> encoded region name into one of these N lists. Each of the N lists is then
> sent all at once in one replication RPC. We do not test if the sum of data in
> each N list will exceed RPC size limits. This code presumes each individual
> edit is reasonably small. Not checking for aggregate size while assembling
> the lists into RPCs is an oversight and can lead to replication failure when
> that assumption is violated.
> We can fix this by generating as many replication RPC calls as we need to
> drain a list, keeping each RPC under limit, instead of assuming the whole
> list will fit in one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
