[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041761#comment-15041761
]
Yu Li commented on HBASE-14790:
-------------------------------
bq. hsync periodically, for example, default per 1000ms
AFAIK, DN will do this if {{dfs.datanode.sync.behind.writes}} is set to true,
and the window is 8MB, just correct me if I'm wrong.
Sorry for the interruption but is the discussion here back to the issue
HBASE-14004 reports, say preventing ReplicationSource to replicate any
non-persisted and later-lost data, due to the fact that we depends on hflush
not hsync currently? And it seems we could not resolve the issue by
implementing new DFSOutputStream?
Would it be a better idea to continue the discussion on master-slave
replication consistency issue in HBASE-14004, while focusing on the new
DFSOutputStream here, to make "an optimized DataOutputStream for small writes
like WAL entry"?
> Implement a new DFSOutputStream for logging WAL only
> ----------------------------------------------------
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
> Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
