Hi, I'm not sure if I understand your problems completely, but relating to your update issue:
HBase keeps versions of your columns. If you have an index on something that needs to be updated you just overwrite the value in the index. There is no need to remove things. I also organize my indexes in separate tables. There is one table for each indexed column of a table and I also keep separate tables for composite indexes. For a fast retrieval I created an indexmanager table which I can use to retrieve the corrsponding indexes for attributes and also keep statistics about them for query planning for instance. Cheers! /SJ ----------- http://uncinuscloud.blogspot.com/ On Sat, Sep 4, 2010 at 9:55 AM, Murali Krishna. P <[email protected]>wrote: > Thanks Samuru, > I was reading about custom indexing in habse, just wanted to know how > are we > handling the updates incase of custom indexing. Probably if the original > data > doesn't change, it might be a good solution. Say, if one of the column > value > gets changed in the original table, we need to query the index table for > the > orignal column value, delete it and then add an entry for the new value. I > think > this will run into consistency issues since we are doing it in a > non-transactional manner. > > Are we always doing full indexing and not worry about increments ? May > be I > am missing something here since I am new to this. > > My requirements are such that daily updates are around 10 million records > where > most of it are just updates and we want it to be real time (or NRT). Any > suggestions are appreciated. > > Thanks, > Murali Krishna > > > > > ________________________________ > From: Samuru Jackson <[email protected]> > To: [email protected] > Sent: Fri, 3 September, 2010 6:24:16 PM > Subject: Re: HBase secondary index performance > > Hi, > > I wrote my own Indexer and actually I have a pretty good performance. > However, there are still known places where I could gain even more > performance (just not having the time right now). > > What is important is to create bulk loads when you are indexing something. > I > posted this one before, but maybe you have missed it: > > I create a Put List out of those records: > > List<Put> pList = new ArrayList<Put>(); > > where each Put has WriteToWAL set to false; > > put.setWriteToWAL(false); > pList.add(p); > > Then I set autoflush to false and create a larger writebuffer: > > hTable.setAutoFlush(false); > hTable.setWriteBufferSize( > 1024*1024*12); > hTable.put(pList); > hTable.setAutoFlush(true); > > The following settings have boosted my load performance 5times - > without any bigger performance tunings, no special HW configuration I > achieve 8000-9000 records per second: > p.setWriteToWAL(false); > hTable.setAutoFlush(false); > hTable.setWriteBufferSize(1024*1024*12); > > > /SJ > http://uncinuscloud.blogspot.com/ > > > > > > > > On Fri, Sep 3, 2010 at 8:30 AM, Murali Krishna. P <[email protected] > >wrote: > > > Thanks Andrey, > > > > * Setting the autoflush to false and increasing the writeBuffer > size > > to 12MB > > improved the writes to 100/s > > * custom indexing is good, but our data keeps changing every day. > > So, probably > > indextable is the best option for us > > * Just added one more regionserver and it did not help. Actually > it > > went back > > to 60/s for some strange reason(with one client). The requests in the > hbase > > ui > > is not uniform across 2 region servers. One server is doing around 2000 > and > > the > > other 500. Probably once the region gets split and when we have lots of > > data, > > writes will improve ? (Now it is just writing to one region for the main > > table) > > * Is there some way to do bulk load the indexedtable? Earlier I > have > > used the > > bulk loader tool (mapreduce job which creates the regions offline) but > not > > sure > > whether it works with indexed table. > > > > > > Thanks, > > Murali Krishna > > > > > > > > > > ________________________________ > > From: Andrey Stepachev <[email protected]> > > To: [email protected] > > Sent: Fri, 3 September, 2010 12:14:29 AM > > Subject: Re: HBase secondary index performance > > > > First, check that you connection not in autoflash mode. > > Second, you can think about custom indexing instead > > of using indexedtable. In my experience custom idexing > > (especially if data doesn't modified), is much more performant. > > Problem with indexedtable is in fact, that on every insert > > hbase performs one (random) get operation (to check, that we doesn't > > have previous indexed data, and delete if it exists). Random gets are > > lays around 100-400 request per node, so you get 60 looks good > > (for indexedtable). > > > > How to build custom indexes you can read > > > > > http://brunodumon.wordpress.com/2010/02/17/building-indexes-using-hbase-mapping-strings-numbers-and-dates-onto-bytes/ > >/ > > > > > > 2010/9/2 Murali Krishna. P <[email protected]>: > > > Hi, > > > I have an indexedtable with index on around 20 columns. The write > > > performance on the original table is around 60 per second. This is just > a > > one > > > node setup. Even with mutiple parallel clients, I am getting just 60 > > > writes/second. That means a total write of 60 * 20 = 1200 writes/second > > due to > > > 20 indextables? This is not good enough for our application. Is this > > number > > >1200 > > > look right ? I was expecting around 15k. > > > I am using 0.20.6 HBase on 0.20.2 Hadoop. hardware config (8g ram, > > 2core, > > > 7.2k rpm disk). Will adding nodes increase the writes linearly? > > > > > > Thanks, > > > Murali Krishna > > > > > > -- /SJ ----------- http://uncinuscloud.blogspot.com/
