[
https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204317#comment-13204317
]
ryan rawson commented on HBASE-5355:
------------------------------------
I had used a number of mechanisms to attempt to reduce the size of the reply.
In no particular order:
- compression (not using the codec stream interface, using a java impl of lzo
or some other fast compression)
- custom-compression, reducing field dup
The bottom line was that the late-compression was pretty expensive to reduce
the size of a reply. Taking 50-100ms to compress more or less negated the
entire benefit at a 5mb response. I ended up dropping this since it wasn't
going to work.
A while back I saw someone who had proposed a compressed representation of KV
that had 'natural prefix' compression. It took advantage of the fact that KVs
are typically stored sorted, so one could have a 'this KV has the same row as
the previous' flag, and ditto for columns, etc.
Aside from that, it might make sense to use the prefix compressed data raw in
the RPC response (as from the blocks), that way there is no re-compression
penalty.
> Compressed RPC's for HBase
> --------------------------
>
> Key: HBASE-5355
> URL: https://issues.apache.org/jira/browse/HBASE-5355
> Project: HBase
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.89.20100924
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> Some application need ability to do large batched writes and reads from a
> remote MR cluster. These eventually get bottlenecked on the network. These
> results are also pretty compressible sometimes.
> The aim here is to add the ability to do compressed calls to the server on
> both the send and receive paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
