[
https://issues.apache.org/jira/browse/HBASE-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648224#comment-15648224
]
Anoop Sam John commented on HBASE-15788:
----------------------------------------
Yes the V4 patch had test fail issue. Latest patch solves that.
CellBlockBuilder is just some refactoring. For decompress op, we used to
create a IS and then pass it and uncompress to another stream. Same now also.
We have APIs which deal with byte[] and ByteBuff. So based on that required
stream is made and passed for uncompress.
Basically we have KeyValue for on heap byte[] backed Cells and OffheapKV for
DBB backed. And ShareableMemory version of both. Am not sure how we can reduce
these. May be the NoTagsXXX versions should be removed? It was done as part
optimization only. So I think we should keep them. ShareableMemory thing is
needed as of now.. Let us see some way we can reduce these versions in another
cleanup?
On ByteBufferSupportOutputStream
ByteBufferOS is already in place in our code base. Basically this is a stream
where we keep adding the written data to a BB. Java ByteArrayOS is adding to a
byte[] and so the name. Same way BBOS. I think that is what BBOS name
suggests. What we need as an interface id diff. We dont worry where the data
is kept within OS. What we are concerned abt is write APIs which take BB as
param. Java OS takes only byte[]. So a new interface we need which is
having write(BB) method in it. And called it BBSOS. Sure we need a better
name. Similar way we need a wrapper OS over any OS which supports BB write
API which implements new interface. BBSOSWrapper is the name given.. We need
better name for sure. Am not good in that :-(
We need this wrapper when we deal with old WAL. (Not AsycnWAL new impl). Here
OS that we use is the OS obtained from DFS client on the WAL file. So we will
end up in making many small byte[]s. New wrapper try to avoid that.
bq.Better to return data and assign to this.data in the caller... doing as
side-effect can confuse....
Will see. We need to return 2 things ByteBuff to read into and CallCleanup
impl which do the BB return to pool..
bq.ByteBuffByteInput is from netty? I see what it is. So, its passing netty
ByteBuff to pb? Needs class comment.
Not Netty. Our ByteBuff.
> Use Offheap ByteBuffers from BufferPool to read RPC requests.
> -------------------------------------------------------------
>
> Key: HBASE-15788
> URL: https://issues.apache.org/jira/browse/HBASE-15788
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: ramkrishna.s.vasudevan
> Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15788.patch, HBASE-15788_V4.patch,
> HBASE-15788_V5.patch
>
>
> Right now, when an RPC request reaches RpcServer, we read the request into an
> on demand created byte[]. When it is write request and including many
> mutations, the request size will be some what larger and we end up creating
> many temp on heap byte[] and causing more GCs.
> We have a ByteBufferPool of fixed sized off heap BBs. This is used at
> RpcServer while sending read response only. We can make use of the same while
> reading reqs also. Instead of reading whole of the request bytes into a
> single BB, we can read into N BBs (based on the req size). When BB is not
> available from pool, we will fall back to old way of on demand on heap byte[]
> creation.
> Remember these are off heap BBs. We read many proto objects from this read
> request bytes (like header, Mutation protos etc). Thanks to PB 3 and our
> shading work as it supports off heap BB now. Also the payload cells are also
> in these DBBs now. The codec decoder can work on these and create off heap
> BBs. Whole of our write path work with Cells now. At the time of addition to
> memstore, these cells are by default copied to MSLAB ( off heap based pooled
> MSLAB issue to follow this one). If MSLAB copy is not possible, we will do a
> copy to on heap byte[].
> One possible down side of this is :
> Before adding to Memstore, we do write to WAL. So the Cells created out of
> the offheap BBs (Codec#Decoder) will be used to write to WAL. The default
> FSHLog works with an OS obtained from DFSClient. This will have only standard
> OS write APIs which is byte[] based. So just to write to WAL, we will end up
> in temp on heap copy for each of the Cell. The other WAL imp (ie. AsynWAL)
> supports writing offheap Cells directly. We have work in progress to make
> AsycnWAL as default. Also we can raise HDFS req to support BB based write
> APIs in their client OS? Until then, will try for a temp workaround solution.
> Patch to say more on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
