[
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306648#comment-14306648
]
Lars Hofhansl commented on HBASE-11544:
---------------------------------------
Awesome. That's exactly what I had in mind.
Long text and questions follows:
In the meanwhile we have been doing this: Set Scan.caching to a very high
value, and set hbase.client.scanner.max.result.size to 2mb.
(64k is way too small for 1ge or faster networks, takes only 0.5ms to send of
1ge, and is the ballpark of latency which is at least 0.1-1ms typically, 2mb
take 16ms and that's nice to amortize the latency, i.e. for each 16ms we pay
0.1-1ms of latency, which is acceptable)
When we stream we only need to fill the bandwidth x latency product. With above
number that be only 12.5k (125mb/s bandwidth and 0.1ms roundtrip, if the
latency was 1ms it would 125k, and so on). That's for later.
So I do take back my statement about chunk sizes between 1-128k. Since we're
*not* streaming we need to amortize the latency incurred by each RPC request,
and that means at large of chunk size that we can get away with. I found that
2mb is good compromise.
I think hbase.client.scanner.max.result.size should be the default setting!
That will already go a long way. We should do this *now*, in all branches.
With that we always get batches of max 2mb shipped from the server to the
client. In the extreme this will only send a single row back per next() RPC if
that row is > 2mb.
So we only have a problem when a single row is > 2mb. So that's already pretty
good.
Now we need to deal with single large rows. Scanner caching is useless there,
that's why we have batching. That's a bit more tricky, since it's not clear to
me what the client should do. Should it always wait until it has a whole row as
I said above? What if that is too large for *it* handle. It seems we always
have to at least allow for optional batching of partial rows.
Single cells are even worse. I like your proposal because it handles even that,
at least on the server side. The client will still need to materialize the cell.
Now, given that we can solve the chunk size problem for many rows by setting
hbase.client.scanner.max.result.size, and that we probably need to allow the
option to handle partial result at the client anyway, is it worth engaging in
this... Unless we do real streaming?
> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return
> batch even if it means OOME
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Labels: beginner
>
> Running some tests, I set hbase.client.scanner.caching=1000. Dataset has
> large cells. I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the
> client whatever we've gathered once we pass out a certain size threshold
> rather than keep accumulating till we OOME.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
