[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295969#comment-16295969
]
Duo Zhang commented on HBASE-14790:
-----------------------------------
{quote}
any interest in contributing this to HDFS?
{quote}
[~arpitagarwal] This is always our final goal. But the current implementation
is too specific so it is not suitable to move the code to HDFS directly. In
general, the async output in HDFS should be
1. Can write multiple blocks.
2. Can do pipeline recovery, of course you can disable it.
3. Can be both chained and fan-out.
For 1, we need to make the request to NN also asynchronous as when switching
block we need to contact with NN. And also, the current block data visibility
semantic is broken when using fan out, we solve it in HBase but I do not think
it is a good idea to provide a 'broken' implementation to users of HDFS... We
need to redesign the write mechanism, a standard 2PC is enough, or use quorum
read/write(maybe too radical)?
I have already uploaded a POC in HDFS-9924 to provide a asynchronous rpc client
and use it to connect to NN so that we can have a 'AsyncDistributedFileSystem'.
Appreciate that if you can make some progress there.
And for the latency metrics, you can see HBASE-16890, where we test it with
WALPE, PE and also YCSB.
Thanks.
> Implement a new DFSOutputStream for logging WAL only
> ----------------------------------------------------
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Fix For: 2.0.0-beta-1
>
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
