[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158447#comment-15158447
]
stack commented on HBASE-15264:
-------------------------------
[~Apache9] This patch is four new classes that do not touch any other code.
Could go into master. I think it would be better if the whole working fanout
WAL provider came in as a 'finished' lump so suggest putting these classes on a
'FanOutWAL' branch and then fill in the other necessary pieces around it. When
all running, all current WAL tests with this provider in place are passing, we
can pull into master branch.
But do what is easier on you making progress (master or branch).
I'm game for testing whenever you ready.
On the patch, ok on having the classes in util for reasons above (on commit,
add comment on why core i/o classes are off in a util package...).
That is a fat bunch of params to pass the constructor:
312 FanOutOneBlockAsyncDFSOutput(Configuration conf, FSUtils fsUtils,
DistributedFileSystem dfs,
313 DFSClient client, ClientProtocol namenode, String clientName,
String src, long fileId,
314 LocatedBlock locatedBlock, EventLoop eventLoop, List<Channel>
datanodeList,
315 DataChecksum summer, ByteBufAllocator alloc) {
Thats how DFSOS takes params? Can't do builder because all params are needed?
FanOutOneBlockAsyncDFSOutput is clean, easy enough to follow what is going on
(I think -- smile -- if I can remember my netty-ese). The helper needs a bit of
commentary. You know how it all works at the moment. Would be good to get some
doc in there while fresh in your mind. Like how DFS_CLIENT_ADAPTOR works and
LeaseManager (Sorry you have to do all this reflection stuff but you have it
nicely contained)... even if it was just a note on why you have to do these
gymnastics and the basic technique you have employed.
What did we say about this:
ByteBuf checksumBuf = alloc.directBuffer(checksumLen);
That we are allocating offheap. when writing always.... The offheap
allocations are relatively slow... Can use a pool later?
... and here: ByteBuf newBuf =
alloc.directBuffer().ensureWritable(trailingPartialChunkLen);
Tests look good -- yeah, the create w/o create of parent is needed for the
fencing trick.
This looks great [~Apache9]
> Implement a fan out HDFS OutputStream
> -------------------------------------
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
> Components: util, wal
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch,
> HBASE-15264-v6.patch, HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
