[
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Lawlor updated HBASE-11544:
------------------------------------
Attachment: HBASE-11544-addendum-v1.patch
Work in progress update:
I've been working on an addendum that includes the ScannerContext changes
described above and would like to get some feedback. I am attaching the patch
but I would like to highlight the following point given the discussion above:
* The intention was to modify the RegionScanner/InternalScanner interfaces as
[~stack] and I described above. Specifically, I wanted to have the following
signatures in RegionScanner (and equivalent ones in InternalScanner):
{code}
ScannerContext nextRaw(List<Cell> result) throws IOException;
ScannerContext nextRaw(List<Cell> result, ScannerContext scannerContext)
{code}
** As far as I can tell, the proposed interface change has two problems: In the
event that the first method is called, a ScannerContext object would need to be
created (object creations are what we want to avoid). Also, if the second
method is called, we are simply returning the same object that the caller
passed in, so the return value is redundant
** Instead I made NextState an enum and I return that. A NextState enum was
used instead of the previous boolean return type because it allows the caller
to determine when a partial result has been formed. An argument could be made
that the return type should be boolean and we should just put NextState inside
the context, but I didn't do that because it would make the code messier (would
have to call scannerContext.setState() before every return statement and opens
up the potential to miss setting the state when really we just want to return
it).
* This way ScannerContext simply holds the limits and tracks the progress
towards those limits
So with this patch what we get is:
The good:
* One object creation per session/rpc instead of potentially millions in the
case of large batch scans
* Much more explicit state information is returned from
RegionScanner/InternalScanner
The bad:
* An object is being passed around between scanners whereas we had a primitive
per limit before.
** However, note that the drawback of having a primitive per limit is that it
does not tell us about the progress that has been made towards those limits and
thus any progress must be recalculated by the caller
* The RegionScanner interface is changed from Stable to Evolving due to the
changes necessary in the interface (this change was noted over in HBASE-13306
but given that we are filing an addendum it makes more sense to address it
here).
As this is a work in progress the docs could still use a little love, but at
the very least this patch lets us see the way that Scan RPC's would look server
side in the event that ScannerContext is introduced.
> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return
> batch even if it means OOME
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Lawlor
> Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: Allocation_Hot_Spots.html,
> HBASE-11544-addendum-v1.patch, HBASE-11544-branch_1_0-v1.patch,
> HBASE-11544-branch_1_0-v2.patch, HBASE-11544-v1.patch, HBASE-11544-v2.patch,
> HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch,
> HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch,
> HBASE-11544-v7.patch, HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch,
> gc.j.png, h.png, hits.j.png, m.png, mean.png, net.j.png, q (2).png
>
>
> Running some tests, I set hbase.client.scanner.caching=1000. Dataset has
> large cells. I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the
> client whatever we've gathered once we pass out a certain size threshold
> rather than keep accumulating till we OOME.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
