[
https://issues.apache.org/jira/browse/HBASE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321838#comment-15321838
]
Vincent Poon commented on HBASE-15995:
--------------------------------------
My intention was only to create a single producer thread reading from the WAL,
performing filtering, then putting entries into a buffer/queue. So one
additional thread per SourceWorker thread, which I believe is currently one per
queue. The source thread reads off the queue and continues to ship as before
in a blocking RPC call, so whatever ordering that currently exists will be
preserved.
I also hope to introduce a simplified iterator abstraction on the edit reading,
with an eye towards eventual streaming of edits through a read / filter / ship
pipeline. But thus far no crazy number of threads or nondeterministic behavior.
> Separate replication WAL reading from shipping
> ----------------------------------------------
>
> Key: HBASE-15995
> URL: https://issues.apache.org/jira/browse/HBASE-15995
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
> Affects Versions: 2.0.0
> Reporter: Vincent Poon
> Fix For: 2.0.0
>
>
> Currently ReplicationSource reads edits from the WAL and ships them in the
> same thread.
> By breaking out the reading from the shipping, we can introduce greater
> parallelism and lay the foundation for further refactoring to a pipelined,
> streaming model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
