Thanks Andrey,
* Setting the autoflush to false and increasing the writeBuffer size to
12MB
improved the writes to 100/s
* custom indexing is good, but our data keeps changing every day. So,
probably
indextable is the best option for us
* Just added one more regionserver and it did not help. Actually it
went back
to 60/s for some strange reason(with one client). The requests in the hbase ui
is not uniform across 2 region servers. One server is doing around 2000 and the
other 500. Probably once the region gets split and when we have lots of data,
writes will improve ? (Now it is just writing to one region for the main table)
* Is there some way to do bulk load the indexedtable? Earlier I have
used the
bulk loader tool (mapreduce job which creates the regions offline) but not sure
whether it works with indexed table.
Thanks,
Murali Krishna
________________________________
From: Andrey Stepachev <[email protected]>
To: [email protected]
Sent: Fri, 3 September, 2010 12:14:29 AM
Subject: Re: HBase secondary index performance
First, check that you connection not in autoflash mode.
Second, you can think about custom indexing instead
of using indexedtable. In my experience custom idexing
(especially if data doesn't modified), is much more performant.
Problem with indexedtable is in fact, that on every insert
hbase performs one (random) get operation (to check, that we doesn't
have previous indexed data, and delete if it exists). Random gets are
lays around 100-400 request per node, so you get 60 looks good
(for indexedtable).
How to build custom indexes you can read
http://brunodumon.wordpress.com/2010/02/17/building-indexes-using-hbase-mapping-strings-numbers-and-dates-onto-bytes/
2010/9/2 Murali Krishna. P <[email protected]>:
> Hi,
> I have an indexedtable with index on around 20 columns. The write
> performance on the original table is around 60 per second. This is just a one
> node setup. Even with mutiple parallel clients, I am getting just 60
> writes/second. That means a total write of 60 * 20 = 1200 writes/second due to
> 20 indextables? This is not good enough for our application. Is this number
>1200
> look right ? I was expecting around 15k.
> I am using 0.20.6 HBase on 0.20.2 Hadoop. hardware config (8g ram, 2core,
> 7.2k rpm disk). Will adding nodes increase the writes linearly?
>
> Thanks,
> Murali Krishna
>
