[
https://issues.apache.org/jira/browse/HBASE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812628#comment-15812628
]
Vincent Poon commented on HBASE-15995:
--------------------------------------
I added a check against the totalQuota before reading the next entry. So now
TestGlobalThrottler passes without modifications. There's still a race
condition since I don't use a lock: multiple threads could do the check and
all pass the check, then all read an entry which cumulatively goes above the
global quota. But I think this problem even exists in current code - since we
currently check after reading an entry, the global quota can still be exceeded.
That's why in TestGlobalThrottler we check against a threshold of 600, even
though we set the total global buffer to 200, right? And if we added a peer,
we would have to change the threshold to 800? So the potential OOM problem
exists even currently, though checking the global quota reduces that risk
significantly.
> Separate replication WAL reading from shipping
> ----------------------------------------------
>
> Key: HBASE-15995
> URL: https://issues.apache.org/jira/browse/HBASE-15995
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
> Affects Versions: 2.0.0
> Reporter: Vincent Poon
> Assignee: Vincent Poon
> Fix For: 2.0.0
>
> Attachments: HBASE-15995.master.v1.patch,
> HBASE-15995.master.v2.patch, HBASE-15995.master.v3.patch,
> replicationV1_100ms_delay.png, replicationV2_100ms_delay.png
>
>
> Currently ReplicationSource reads edits from the WAL and ships them in the
> same thread.
> By breaking out the reading from the shipping, we can introduce greater
> parallelism and lay the foundation for further refactoring to a pipelined,
> streaming model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
