[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295639#comment-16295639
]
Arpit Agarwal edited comment on HBASE-14790 at 12/18/17 8:38 PM:
-----------------------------------------------------------------
This looks interesting.
[~Apache9], any interest in contributing this to HDFS? Also do you have any
measurements for the latency benefit? I skimmed the comments but didn't see it
so I apologize if it's there already.
(Thanks for the heads up [[email protected]]).
was (Author: arpitagarwal):
This looks interesting.
[~Apache9], any interest in contributing this to HDFS? Also do you have any
measurements for the latency benefit? I skimmed the comments but didn't see it
so I apologize if it's there already.
> Implement a new DFSOutputStream for logging WAL only
> ----------------------------------------------------
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Fix For: 2.0.0-beta-1
>
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
