[
https://issues.apache.org/jira/browse/HBASE-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604201#comment-15604201
]
ramkrishna.s.vasudevan commented on HBASE-16891:
------------------------------------------------
The hack that I tried here. Just pasting the snippet. I am already working on a
full fledged patch\
In AsyncProtobufWriter#append
{code}
length.addAndGet(buf.size());
output.write(buf.getBuffer(), 0, buf.size());
buf.reset();
for (Cell cell : entry.getEdit().getCells()) {
length.addAndGet(((KeyValue)cell).getLength());
output.write(((KeyValue)cell).getBuffer(),
((KeyValue)cell).getOffset(), ((KeyValue)cell).getLength());
}
{code}
And inside the Asyncoutput write I tried writing the length and the data
together at one shot.
Something like this
{code}
buf.ensureWritable(len + Bytes.SIZEOF_INT);
if (cryptoCodec == null) {
buf.writeInt(len);
buf.writeBytes(b, off, len);
} else {
ByteBuffer inBuffer = ByteBuffer.wrap(b, off, len);
cryptoCodec.encrypt(inBuffer, buf.nioBuffer(buf.writerIndex(), len));
buf.writerIndex(buf.writerIndex() + len);
}
{code}
> Try copying to the Netty ByteBuf directly from the WALEdit
> ----------------------------------------------------------
>
> Key: HBASE-16891
> URL: https://issues.apache.org/jira/browse/HBASE-16891
> Project: HBase
> Issue Type: Sub-task
> Components: wal
> Affects Versions: 2.0.0
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
> -> The FanOutOneBlockAsyncDFSOutput is a much sophisticated dfs client model
> that works with Netty ByteBuf. Here we hold on connection to the datanodes
> using
> Netty Channels. And the idea is to write data direclty to these channels.
> AsyncHLog gets an append call. The AysncWAL uses the HBase's
> ByteArrayOutputSTream and so the content of the cell is written to this BAOS
> and that is again
> copied to the netty Bytebuf in the FanOutOneBlockAsyncDFSOutput.
> So when the sync call happens this FanoutDFSoutput does the checksum
> calcualtion itself and then writes the content of this buffer direclty to the
> DN channel.
> -> In case of FSHLOg this is different. When an append call comes we direclty
> write the content to the FSDataOutputStream (it is copied to this stream).
> Then here internally there is a checkSum calculation that happens. when a
> sync call happens there is noth ing to do except to notify the NN to flush
> the latest
> data.
> AS we can see from the above that there are two copies in AsyncWAL
> -> From the Cell to the BAOS
> -> From the BAOS to the Netty byte buf
> -> On sync() call, do check sum and finally flush the netty byte buf to the
> DN channel
> In case of FSHLog
> -> From cell to the FSDataoutputstream. data is copied. Check sum happens
> here.
> -> Sync call just tries to notify the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
