Inline.

> *Region/Meta Cache
> *
> Often times the region list is not "hot" and thrift has to talk to the meta
> table. We have 6k+ regions and growing quickly and expect 1k+/node. Can we
> help our performance by pre-caching all region locations? How many regions
> can thrift keep before over-writing in cache (given default settings)? Does
> it make sense to write a program to cycle through all thrift servers and
> heat up the region cache after a lot of writes?

You may want to keep the size of your regions bigger, to have fewer of
them. Checkout the MAX_FILESIZE config in the shel when
creating/altering a table.

BTW the Thrift server is really just an interface between incoming
thrift calls and the HBase java API. When you say "How many regions
can thrift keep before over-writing in cache", it's actually the hbase
client in HConnectionManager that does it. And the answer to that
question is... well it depends on the memory you give to the thrift
servers because the references are "weak", meaning that the GC will
decide whether to collect some of them if there's memory pressure.

Regarding pre-fetching of region locations, it's already configured to
fetch 10 consecutive rows every time the client has to query the
.META. table, you can change this with hbase.client.prefetch.limit

>
> *Thrift Logs
> *Below is an example of a scan from the thrift logs. I am trying to
> understand what is going on and where things can be sped up. The
> scannerGetList log below took 43ms which is longer than usual. What would
> slow this down, waiting for the region server to respond (there is load on
> the cluster for writes)? It took 12ms to be finished with scanning which
> makes sense, but then it appears to take almost as long to close the scanner
> (11ms). This read took 66ms when in reality it only took 12ms to read the
> data from what I can tell. That does not make sense to me. The disk i/o
> should be the bulk of the time spent on a read and it appears to be little
> of this 66ms. How can we speed up read latency? Faster disks should be the
> answer but I am not sure it makes much of a difference here.
>
> 2011-01-08 16:28:47,921 DEBUG
> org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over
> xxx starting at key '12345'
> 2011-01-08 16:28:47,921 DEBUG
> org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal
> scanner to startKey at '12345'
> 2011-01-08 16:28:47,964 DEBUG
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler: scannerGetList:
> id=145060
> 2011-01-08 16:28:47,976 DEBUG
> org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning
> at REGION => {NAME => ....
> 2011-01-08 16:28:47,987 DEBUG
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler: scannerClose:
> id=145060

What happened here is 3 RPCs: open the scanner, get, and finally
close. Ideally the first two should be done in the same RPC, it
shouldn't even be that hard to implement just for the thrift server,
and then the close isn't required if you provide is stopRow. It seems
in your case a lot of time was spent doing those RPCs, but better
instrumenting would be required to tell whether the 43ms was really
spent on the network or opening the scanner. Have a look at
ThriftServer, it's pretty straight forward.

Regarding the actual reading of the data, if you plan on always
scanning a lot of rows at the same time, I suggest setting
hbase.client.scanner.caching to something bigger than 1. I know it's
dumb to have to restart all the thrift servers in order to change
configuration and that all scans will have the same caching value but
until Scan.setCaching() is exposed to the Thrift client, it has to be
set that way.


J-D

Reply via email to