[
https://issues.apache.org/jira/browse/HBASE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13746728#comment-13746728
]
Lars Hofhansl commented on HBASE-9272:
--------------------------------------
Compared to HBASE-1935 this is *much* simpler since it still uses ClientScanner
under the hood (which I had since factored in its own class).
Using ClientScanner also has the benefit that this is resistant to concurrent
splits.
Simple perf test with 30m rows, 1 col, 100 byte values. Split into 16 regions
on a cluster with 16 region servers.
The performance speedup (scan latency here) is proportional to the number of
threads used when most data is filtered at the server (as expected) - of course
the cluster was not otherwise busy.
Even when *all* rows are returned I see following (scanner caching: 100),
buffer size in ParallelScanner was 1000 and 10000 (made no difference perf
wise):
* Running the ParallelScanner with 50 threads: 40.8s.
* Running the ParallelScanner with 20 threads: 40.3s.
* Running the ParallelScanner with 16 threads: 40.3s.
* Running the ParallelScanner with 10 threads: 59.6s.
* Running the ParallelScanner with 1 thread: 316
* Running the standard scanner: 309
So there is a 1.5% synchronization/context-switching overhead.
It looks like this general approach a viable and by using ClientScanner it is
also ridiculously simple.
> A simple parallel, unordered scanner
> ------------------------------------
>
> Key: HBASE-9272
> URL: https://issues.apache.org/jira/browse/HBASE-9272
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: ParallelClientScanner.java
>
>
> The contract of ClientScanner is to return rows in sort order. That limits
> the order in which region can be scanned.
> I propose a simple ParallelScanner that does not have this requirement and
> queries regions in parallel, return whatever gets returned first.
> This is generally useful for scans that filter a lot of data on the server,
> or in cases where the client can very quickly react to the returned data.
> I have a simple prototype (doesn't do error handling right, and might be a
> bit heavy on the synchronization side - it used a BlockingQueue to hand data
> between the client using the scanner and the threads doing the scanning, it
> also could potentially starve some scanners long enugh to time out at the
> server).
> On the plus side, it's only a 130 lines of code. :)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
