[ 
https://issues.apache.org/jira/browse/HBASE-21461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681685#comment-16681685
 ] 

Wellington Chevreuil commented on HBASE-21461:
----------------------------------------------

Thanks for reviewing this, [~elserj] !
{quote}Is it safe to split up a WalEdit? Wouldn't this introduce the potential 
for us to replicate the wrong state in the destination cluster? I'm not 
positive, but it's giving me pause.
{quote}
It will still replicate in same sequence, however in several batches, instead 
of a single large one. This is currently done synchronously. Also, it preserves 
the OP original timestamp from source, which I think is the most critical here 
to maintain the correct state.

{quote}
If that is safe (or at least, safe enough), couldn't we just split up the 
WalEdit within the source RS and prevent the need for an extra 
CP/user-intervention?
{quote}

That would be the ideal and definitive solution to avoid the problem from 
happening. This CP, however, is thought more as an admin tool (that's why I 
propose it as part of operators tools). For deployments already affected by 
this issue, that would still require the manual intervention for cleanup and 
unblock replications sinks. The CP, once plugged, would just allow the stuck 
replication edits to drain, with no other manual intervention (it's much 
simpler than having to cleanup znodes, oldWALs, run wal player, etc). Also, the 
definitive fixes from 18027 requires an upgrade process. From product 
supporting perspective, many organizations usually have restricted policies for 
upgrades.

{quote}
Could/should we backport 18027 to branch-1.2? Have you looked at that/found it 
infeasible?
{quote}
Yeah, definitely worth try it, I haven't evaluated such backport actually, I 
was trying to integrate it on our own distribution that's based on 1.2 (with 
some divergences), but couldn't manage to get it working properly. I can try a 
"pure" branch-1.2, though.

> Region CoProcessor for splitting large WAL entries in smaller batches, to 
> handle situation when faulty ingestion had created too many mutations for 
> same cell in single batch
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-21461
>                 URL: https://issues.apache.org/jira/browse/HBASE-21461
>             Project: HBase
>          Issue Type: New Feature
>          Components: hbase-operator-tools, Replication
>            Reporter: Wellington Chevreuil
>            Assignee: Wellington Chevreuil
>            Priority: Minor
>         Attachments: 0001-Initial-version-for-WAL-entry-splitter-CP.txt
>
>
> With replication enabled deployments, it's possible that faulty ingestion 
> clients may lead to single WalEntry containing too many edits for same cell. 
> This would cause *ReplicationSink,* in the target cluster, to attempt single 
> batch mutation with too many operations, what in turn can lead to very large 
> RPC requests, which may not fit in the final target RS rpc queue. In this 
> case, the messages below are seen on target RS trying to perform the sink:
> {noformat}
> WARN org.apache.hadoop.hbase.client.AsyncProcess: #690, table=TABLE_NAME, 
> attempt=4/4 failed=2ops, last exception: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException):
>  Call queue is full on /0.0.0.0:60020, is hbase.ipc.server.max.callqueue.size 
> too small? on regionserver01.example.com,60020,1524334173359, tracking 
> started Fri Sep 07 10:35:53 IST 2018; not retrying 2 - final failure
> 2018-09-07 10:40:59,506 ERROR 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Unable to 
> accept edit because:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 
> actions: RemoteWithExtrasException: 2 times, 
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1663)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:982)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:996){noformat}
> When this problem manifests, replication will be stuck and wal files will be 
> piling up on source cluster WALs/oldWALs folder. Typical workaround requires 
> manual cleanup of replication znodes in ZK, and manual WAL replay for the WAL 
> files containing the large entry.
> This CP would handle the issue, by checking for large wal entries and 
> splitting those into smaller batches on the *reReplicateLogEntries* method 
> hook.
> *Additional Note*: HBASE-18027 introduced some safeguards such large RPC 
> requests, which may already help avoid such scenario. That is not available 
> for 1.2 releases, though, and this CP tool may still be relevant for 1.2 
> clusters. It may also be still worth having it to workaround any potential 
> unknown large RPC issue scenarios.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to