200 node experiment on bulk loads and scans:
30 KB rows, uncompressed, 1 column family. I generate random data on the fly
4 gig regions, 1 MB Hbase block size,
A) Bulk loads: around 30 MBps per node. Most of the time was taken by my map
reduce job not using the Hbase api.
* I had initially encountered gc issues when I was running MR apps while
using the Hbase API. It is the oft-repeated issue of gc taking up a lot of time
(swap-ins and swap-outs) which results in the RS failing to report to ZK about
its liveness. This results in an exception at the RS.
* The way I fixed the gc problems was by
1) JD's suggestion: To change the GC params in hbase-env.sh
-XX:+DoEscapeAnalysis -XX:+AggressiveOpts
-XX:+UseConcMarkSweepGC -XX:NewSize=64m -XX:MaxNewSize=64m
-XX:CMSInitiatingOccupancyFraction=88 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
2) Increasing the zk timeout and the tick period. This is already
presented in Hbase FAQs
Another suggestion by JD and stack was the vm swappability
parameter, but I didn't't have permissions to reset it.
Update API:
I issued some updates through an MR job that issued batches of puts on existing
and new rows. It added around 17 TB of new data with between 2 and 3 storefiles
per region.
B) Scans: I used TableInputFormat.. I got a rate of 28 MBps per node (in the
average) and 43 MBps per node (in the median).. There were 700 tasks running at
a time, 47000 tasks in total. 6 minute map completion time in the average and 4
minutes in the median. 95th percentile is 20 mins. The max completion time of
some tasks was 10 hours though!
So there is a very small number of tasks that take a really long time to finish
and I havent had enough time to figure out why since I have to rerun the tests
now. I will try to fix this issue and let you guys know..
C) I issued a major_compact on the entire table:
I then issued major_compact on the entire table. The entire set of 117 TB
(after some updates) got read and rewritten: Finished in 28 hours: Roughly 7
MBps per node. With no concurrent compactions in any node, I think this is a
good number.
D) There are more experiments that need to be done but will be only next week.
I will post any updates.
For example, I want to see the effect of number of storefiles per region and
column families on scans and compactions.
Summarizing, for the access patterns that we have been testing for, memory
sensitivity is the one major problem that we had faced. Can you guys let me
know if any of the numbers sound a little off?
Thank you
Vidhya
