[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14334019#comment-14334019
]
Jonathan Lawlor commented on HBASE-13071:
-----------------------------------------
Those performance improvements look great [~eshcar]! I agree with [~stack] that
this configuration should be on by default.
I have been thinking about this asynchronous scanner a bit and was wondering
how it deals with the case where the user has specified a maximum result size
via Scan#setMaxResultSize():
* Does the prefetch mechanism attempt to prefetch a certain number of
rows and loop until that number of rows is retrieved (i.e. does the prefetch
only stop making prefetch RPCs once it has accumulated the expected number of
rows)?
* If the prefetch does stop when the maxResultSize limit has been
exceeded, when would the next prefetch be initiated? When next() is called and
the cache is observed to have less than half the specified caching value?
** If that's the case, I was thinking that it may be possible that a
table with very large rows could potentially create memory issues on the client
(and this may be a good reason to make this async behavior configurable rather
that go all-in on async scanners).
** The scenario that I have in mind is the case where the client is not
able to hold X rows in memory where X is half of the specified scanner caching
(i.e. the case where (half_caching_limit * size_of_row) exceeds the avalaible
memory on the client). The synchronous scanners currently guard the client from
out of memory exceptions by considering the remaining result size after each
next RPC -- if the maxResultSize limit is exceeded the scanner stops making
RPCs, regardless of how many rows the scanner has received from RPCs. However,
in the case of asynch scanners (if they work how I am thinking they do) each
call to next() may trigger a prefetch, and each prefetch would receive only Y
rows, where Y << half_of_caching. The prefetches would be triggered on each
call to next() until we have accumulated half_of_caching rows in our scanner
cache and the client may OOM before that limit can be reached
*** In such a case, it may be best to instead use a synch scanner.
Alternatively, we could keep track of the size of results in our cache and
allow the user to specifiy maxSizeOfCache so that prefetches aren't continuosly
fired off on calls to next() when scanning a table with large rows
*** This also raises the question about whether or not it would be
worthwhile to have an asynchronous scanner that used the size of the Results
(in memory) as the deciding factor for prefetching -- i.e. If the size of the
results in the cache is less than half of maxResultSize then perform a prefetch
for another chunk of maxResultSize worth of values... just a thought
My concern above is definitely a corner case issue, but I thought I'd raise it
for discussion.
I like the idea that you pointed out towards the end about ramping the prefetch
size up from a small initial size to the actual prefetch size. Looks like it
would definitely help ease the initial latency jump seen for large prefetches.
Really nice results :)
> Hbase Streaming Scan Feature
> ----------------------------
>
> Key: HBASE-13071
> URL: https://issues.apache.org/jira/browse/HBASE-13071
> Project: HBase
> Issue Type: New Feature
> Reporter: Eshcar Hillel
> Attachments: HBaseStreamingScanDesign.pdf,
> HbaseStreamingScanEvaluation.pdf
>
>
> A scan operation iterates over all rows of a table or a subrange of the
> table. The synchronous nature in which the data is served at the client side
> hinders the speed the application traverses the data: it increases the
> overall processing time, and may cause a great variance in the times the
> application waits for the next piece of data.
> The scanner next() method at the client side invokes an RPC to the
> regionserver and then stores the results in a cache. The application can
> specify how many rows will be transmitted per RPC; by default this is set to
> 100 rows.
> The cache can be considered as a producer-consumer queue, where the hbase
> client pushes the data to the queue and the application consumes it.
> Currently this queue is synchronous, i.e., blocking. More specifically, when
> the application consumed all the data from the cache --- so the cache is
> empty --- the hbase client retrieves additional data from the server and
> re-fills the cache with new data. During this time the application is blocked.
> Under the assumption that the application processing time can be balanced by
> the time it takes to retrieve the data, an asynchronous approach can reduce
> the time the application is waiting for data.
> We attach a design document.
> We also have a patch that is based on a private branch, and some evaluation
> results of this code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
