[
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074806#comment-14074806
]
Lars Hofhansl commented on HBASE-11544:
---------------------------------------
bq. as the max cell size now is 10mb IIRC, for the robust solution sounds like
we should be able to split the cell and pass the portion of byte array,
representing the cell value?
I think so. We need to decouple the optimal RPC size from the response size.
bq. mvcc readpoint i think yeah, sending partial rows might be much bigger
change
Why? The readpoint is known/fixed by/for the scanner. We already allow sending
partial rows (see Scan.setBatch(...) or the mentioned Scan.setMaxResultSize()).
The "only" new part is hat we'd assemble on the client what is expected by the
API. Of course that means that we can OOM the client when a *single* row gets
really large.
So what I am saying is that we:
# get rid of scanner caching
# get rid of max result size
# define an rpcChunkSize (or something). We'd default that to a useful value
(maybe 128k), the user can then optimize that depending on the network topology
(faster networks would need a larger value)
# the server would gather data until at least one chunk is filled and then
sends the chunk to the client
# the client would gather chunks until it has enough data to send a row up to
the caller
It would be better to even have a full streaming protocol, but that's be a
bigger change and not fit well into RPC/Protobuf.
As usually... Just my $0.02 :)
> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return
> batch even if it means OOME
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Labels: noob
>
> Running some tests, I set hbase.client.scanner.caching=1000. Dataset has
> large cells. I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the
> client whatever we've gathered once we pass out a certain size threshold
> rather than keep accumulating till we OOME.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
