This is significantly lower than the top write speeds I've seen, like an order of magnitude. And you are running on 4 disks per node so should be way faster. One thing to keep in mind though is HBase does not support concurrent compactions so we don't always fully utilize multi-disk setups. Multiple compactions should be included in the next major release.
What's going on in your logs, especially region servers? Do you see blocking of updates? Anything unusual like flushes that aren't from hitting the flush size? Are you starting from an empty table? Are your insertion keys random? Your 1.5MB/sec/node comes from a steady-state insertion load once the table is evenly distributed across nodes? How many regions at this time and do you see even or uneven load across RS? What I remember was being on the order of 1/2 or 1/4 the raw write throughput of the drives, something in that range though I'm forgetting the details. There's no architectural reason not to be in that range or better. In these calculations, however, all the writes to disk were being used in the calculation (io used for flushes, compactions, etc). Your calculation is based on the actual size of the data, though behind the scenes HBase is writing this multiple times. Did you change the MemStore flush size? You're going to end up doing a ton of compactions if you are flushing small MemStores but have a big max region size. The flush size is one factor. The total heap on each RS and the number of regions per RS will also impact the sizes of flushed files. Each time you do a compaction, you rewrite data, this kills io. There are lots of changes coming up in the next release. Follow along HBASE-2375 and related jiras for the compaction/split/flush improvements being worked on. JG > -----Original Message----- > From: Vidhyashankar Venkataraman [mailto:vidhy...@yahoo-inc.com] > Sent: Tuesday, June 01, 2010 4:21 PM > To: user@hbase.apache.org > Subject: Re: Performance at large number of regions/node > > I have a related question: I tried a simple load experiment too using > Hbase's Java API.. (The nodes do only loading: nothing else.. The > client programs generate random data on the fly to load.. So, no reads > of the input data).. > > 120m rows 15KB each. 2 column families. > 5 region servers, ran around 4 or 5 clients per node on the 5 nodes > that run the region servers.. > > 2MB block size, 2gigs region size, WAL disabled, auto flush disabled.. > 2MB write buffer.. Major compactions disabled.. > > The other configs are quite similar to the configs discussed in this > thread.. > > And I get a throughput of around 1.5 MB per second per node.. > (500 rows per second for the entire cluster).. Do these values seem > reasonable? > > Thanks > Vidhya > > On 5/29/10 6:36 PM, "Jacob Isaac" <ja...@ebrary.com> wrote: > > Hi J-D > > We have 8 drives (~500G per drive - total 4G) per machine > > The metrics from my run indicate that I achieve around > for writes - > around 1 row(5k) in 2ms => 500 rows(5K) in 1 sec => 2.5 Mb/sec > > and from your the observation at StumbleUpon > > 200k rows (presuming 100 bytes per row)/sec => 20Mb/sec > Wow !! that an order of difference > I am sure disabling WAL during the writes is giving you a significant > boost. > > Are you reading the data at the same time as you are writing? > > Thx > Jacob > > On Fri, May 28, 2010 at 9:04 PM, Jean-Daniel Cryans > <jdcry...@apache.org> wrote: > >> What I wanted out of this discussion was to find out whether I am in > the > >> ballpark of what I can juice out of HBase or I am way off the mark. > >> > > > > I understand... but this is a distributed system we're talking about. > > Unless I have the same code, hbase/hadoop version, configuration, > > number of nodes, cpu, RAM, # of HDDs, OS, network equipment, data > set, > > etc... it's really hard to assess right? For starters, I don't think > > you specified the number of drives you have per machine, and HBase is > > mostly IO-bound. > > > > FWIW, here's our experience. At StumbleUpon, we uploaded our main > data > > set consisting of 13B*2 rows on 20 machines (2xi7, 24GB (8 for > HBase), > > 4x 1TB JBOD) with MapReduce (using 8 maps per machine) pulling from a > > MySQL cluster (we were selecting large ranges in batches), inserting > > at an average rate of 150-200k rows per second, peaks at 1M. Our rows > > are a few bytes, mostly integers and some text. We did it in the time > > with HBase 0.20.3 + the parallel-put patch we wrote here (available > in > > trunk) with the configuration I pasted previously. For that upload > the > > WAL was disabled and ALL our tables are LZOed (can't stress enough > the > > importance of compressing your tables!) and 1GB max file size. > > > > My guess is yes you can juice it out more, first by using LZO ;) > > > > Also, are your machines even stressed during the test? Do you > monitor? > > Could you increase the number of clients? > > > > Sorry I can't give you a very clear answer, but without using a > common > > benchmark to compare numbers we're pretty much all in the dark. YCSB > > is one, but IIRC it needs some patches to work efficiently (Todd > > Lipcon from Cloudera has them in his github). > > > > J-D > >
