[
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376940#comment-14376940
]
Andrew Purtell commented on HBASE-11544:
----------------------------------------
Is allocating a ton of NextState objects in hot code either confirmably not a
problem by allocation profiling and/or inspection of generated Hotspot code, or
is the chance of very large rows so severe that pushing this all the way down
into the scanner is absolutely necessary? I don't think we have an answer for
the first part. The second part is opinion, and could be valid, but may need
more discussion.
As for the accounting, I don't find the accounting for result size limits in
RSRpcServices that big of a deal. I also think we are still not accounting for
size correctly (or maybe the latest patches do... not sure I'm totally up to
date): In order to guarantee a uniform view of what "size" is a cell, we should
count the bytes in the key, the bytes in the value, and the bytes in tags, and
that's it. Avoid things like heapSize which factors in magic constants,
internal data structure details, and padding/alignment which can vary by JVM,
platform type, and CLI parameters (on JRE 8+).
> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return
> batch even if it means OOME
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Lawlor
> Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-11544-branch_1_0-v1.patch,
> HBASE-11544-branch_1_0-v2.patch, HBASE-11544-v1.patch, HBASE-11544-v2.patch,
> HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch,
> HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch,
> HBASE-11544-v7.patch, HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch,
> gc.j.png, hits.j.png, mean.png, net.j.png
>
>
> Running some tests, I set hbase.client.scanner.caching=1000. Dataset has
> large cells. I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the
> client whatever we've gathered once we pass out a certain size threshold
> rather than keep accumulating till we OOME.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
