What do you have for GC config Schubert? Now its 8ms a random read? St.Ack
On Tue, Aug 18, 2009 at 10:28 AM, Jonathan Gray <[email protected]> wrote: > Schubert, > > I can't think of any reason your random reads would get slower after > inserting more data, besides GC issues. > > Do you have GC logging and JVM metrics logging turned on? I would inspect > those to see if you have any long-running GC pauses, or just lots and lots > of GC going on. > > If I recall, you are running on 4GB nodes, 2GB RS heap, and cohosted > DataNodes and TaskTrackers. We ran for a long time on a similar setup, but > once we moved to 0.20 (and to the CMS garbage collector), we really needed > to add more memory to the nodes and increase RS heap to 4 or 5GB. The CMS > GC is less efficient in memory, but if given sufficient resources, is much > better for overall performance/throughput. > > Also, do you have Ganglia setup? Are you seeing swapping on your RS nodes? > Is there high IO-wait CPU usage? > > JG > > > Schubert Zhang wrote: > >> Addition. >> Only random-reads become very slow, scans and sequential-reads are ok. >> >> >> On Tue, Aug 18, 2009 at 6:02 PM, Schubert Zhang <[email protected]> >> wrote: >> >> stack and J-G, Thank you very much for your helpful comment. >>> >>> But now, we find such a critical issue for random reads. >>> I use sequentical-writes to insert 5GB of data in our HBase table from >>> empty, and ~30 regions are generated. Then the random-reads takes about >>> 30 >>> minutes to complete. And then, I run the sequentical-writes again. Thus, >>> another version of each cell are inserted, thus ~60 regions are >>> generated. >>> But, we I ran the random-reads again to this table, it always take long >>> time >>> (more than 2 hours). >>> >>> I check the heap usage and other metrics, does not find the reason. >>> >>> Bellow is the status of one region server: >>> request=0.0, regions=13, stores=13, storefiles=14, storefileIndexSize=2, >>> memstoreSize=0, usedHeap=1126, maxHeap=1991, blockCacheSize=338001080, >>> blockCacheFree=79686056, blockCacheCount=5014, blockCacheHitRatio=55 >>> >>> Schubert >>> >>> >>> On Tue, Aug 18, 2009 at 5:02 AM, Schubert Zhang <[email protected]> >>> wrote: >>> >>> We have just done a Performance Evaluation on HBase-0.20.0. >>>> Refers to: >>>> >>>> http://docloud.blogspot.com/2009/08/hbase-0200-performance-evaluation.html >>>> >>>> >>> >>
