Thanks Samuru,
I was reading about custom indexing in habse, just wanted to know how are
we
handling the updates incase of custom indexing. Probably if the original data
doesn't change, it might be a good solution. Say, if one of the column value
gets changed in the original table, we need to query the index table for the
orignal column value, delete it and then add an entry for the new value. I
think
this will run into consistency issues since we are doing it in a
non-transactional manner.
Are we always doing full indexing and not worry about increments ? May be
I
am missing something here since I am new to this.
My requirements are such that daily updates are around 10 million records where
most of it are just updates and we want it to be real time (or NRT). Any
suggestions are appreciated.
Thanks,
Murali Krishna
________________________________
From: Samuru Jackson <[email protected]>
To: [email protected]
Sent: Fri, 3 September, 2010 6:24:16 PM
Subject: Re: HBase secondary index performance
Hi,
I wrote my own Indexer and actually I have a pretty good performance.
However, there are still known places where I could gain even more
performance (just not having the time right now).
What is important is to create bulk loads when you are indexing something. I
posted this one before, but maybe you have missed it:
I create a Put List out of those records:
List<Put> pList = new ArrayList<Put>();
where each Put has WriteToWAL set to false;
put.setWriteToWAL(false);
pList.add(p);
Then I set autoflush to false and create a larger writebuffer:
hTable.setAutoFlush(false);
hTable.setWriteBufferSize(
1024*1024*12);
hTable.put(pList);
hTable.setAutoFlush(true);
The following settings have boosted my load performance 5times -
without any bigger performance tunings, no special HW configuration I
achieve 8000-9000 records per second:
p.setWriteToWAL(false);
hTable.setAutoFlush(false);
hTable.setWriteBufferSize(1024*1024*12);
/SJ
http://uncinuscloud.blogspot.com/
On Fri, Sep 3, 2010 at 8:30 AM, Murali Krishna. P <[email protected]>wrote:
> Thanks Andrey,
>
> * Setting the autoflush to false and increasing the writeBuffer size
> to 12MB
> improved the writes to 100/s
> * custom indexing is good, but our data keeps changing every day.
> So, probably
> indextable is the best option for us
> * Just added one more regionserver and it did not help. Actually it
> went back
> to 60/s for some strange reason(with one client). The requests in the hbase
> ui
> is not uniform across 2 region servers. One server is doing around 2000 and
> the
> other 500. Probably once the region gets split and when we have lots of
> data,
> writes will improve ? (Now it is just writing to one region for the main
> table)
> * Is there some way to do bulk load the indexedtable? Earlier I have
> used the
> bulk loader tool (mapreduce job which creates the regions offline) but not
> sure
> whether it works with indexed table.
>
>
> Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: Andrey Stepachev <[email protected]>
> To: [email protected]
> Sent: Fri, 3 September, 2010 12:14:29 AM
> Subject: Re: HBase secondary index performance
>
> First, check that you connection not in autoflash mode.
> Second, you can think about custom indexing instead
> of using indexedtable. In my experience custom idexing
> (especially if data doesn't modified), is much more performant.
> Problem with indexedtable is in fact, that on every insert
> hbase performs one (random) get operation (to check, that we doesn't
> have previous indexed data, and delete if it exists). Random gets are
> lays around 100-400 request per node, so you get 60 looks good
> (for indexedtable).
>
> How to build custom indexes you can read
>
>http://brunodumon.wordpress.com/2010/02/17/building-indexes-using-hbase-mapping-strings-numbers-and-dates-onto-bytes/
>/
>
>
> 2010/9/2 Murali Krishna. P <[email protected]>:
> > Hi,
> > I have an indexedtable with index on around 20 columns. The write
> > performance on the original table is around 60 per second. This is just a
> one
> > node setup. Even with mutiple parallel clients, I am getting just 60
> > writes/second. That means a total write of 60 * 20 = 1200 writes/second
> due to
> > 20 indextables? This is not good enough for our application. Is this
> number
> >1200
> > look right ? I was expecting around 15k.
> > I am using 0.20.6 HBase on 0.20.2 Hadoop. hardware config (8g ram,
> 2core,
> > 7.2k rpm disk). Will adding nodes increase the writes linearly?
> >
> > Thanks,
> > Murali Krishna
> >
>
