[
https://issues.apache.org/jira/browse/HBASE-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638571#comment-13638571
]
stack commented on HBASE-7413:
------------------------------
bq. The usage of cells for both request and response is not very obvious
Why boss? We'd send Cells but return molecules?
bq. It does seem to result in extra copy though, when you build the cellblock,
which is something we want to avoid for WAL path. Why is cell block ByteBuffer?
Because we build the cell block in memory? Yeah. I don't see any way around
it. You have any suggestions? Hopefully w/ a good prefix encoding and maybe a
compression (more copying?) it'll be worth the composition in memory before we
put it on the wire.
Cell block is a byte array. Building it we need to use something that can
grow, the BBOutputStream.
bq. It would require, aside from the task itself, adding a way to build
cellblock directly into output without copy (and getting count from cellscanner
in advance?)
Not sure how to do the above. Certain cellblock types could do this -- i.e. a
write through of Cells -- but cellblocks that do prefix encoding will want to
store up and flush at well defined junctures. The count of Cells you should
have before you start no?
I was hoping we could go from rpc straight into the WAL with the cell blocks
but this is a bit naive since we'll want cellblocks by region?
If not enough Cells, probably no need to make a CellBlock. If values area
large -- > a couple of kilobytes -- probably worth compressing the WAL values.
On the rest of the writeup, thanks for doing it.
Here are some comments.
Can you call it WALKey instead of HLogKey?
bq. Replication scopes have been moved to HLogKey from WALEdit, so the latter
only has KVs.
Excellent! This I see as big improvement.
Can you talk about Cells instead of KVs? If going to all this bother, might as
well go the whole hog and do Cell.
bq. VInt with length, followed by the backing byte array....
Doing the above, I'd argue, is only doing half of the job. If we are going to
convert WAL, lets got off KV and its current serialization. Write the codec
used into the WAL metadata and then write cellblocks (can do an implementation
that writes straight to the WAL -- wrap the WAL outputstream in a
CellOutputStream?).
Writeup is great.
Let me look at the patch now.
> Convert WAL to pb
> -----------------
>
> Key: HBASE-7413
> URL: https://issues.apache.org/jira/browse/HBASE-7413
> Project: HBase
> Issue Type: Sub-task
> Components: wal
> Reporter: stack
> Assignee: Sergey Shelukhin
> Priority: Critical
> Fix For: 0.95.1
>
> Attachments: HBASE-7413-v0.patch, HBASE-7413-v1.patch,
> HBASE-7413-v2.patch
>
>
> From HBASE-7201
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
