@Juhani:
How many clients did you test? Maybe the bottleneck was client?

2012/3/19 Ramkrishna.S.Vasudevan <[email protected]>

> Hi Juhani
>
> Can you tell more on how the regions are balanced?
> Are you overloading only specific region server alone?
>
> Regards
> Ram
>
> > -----Original Message-----
> > From: Juhani Connolly [mailto:[email protected]]
> > Sent: Monday, March 19, 2012 4:11 PM
> > To: [email protected]
> > Subject: 0.92 and Read/writes not scaling
> >
> > Hi,
> >
> > We're running into a brick wall where our throughput numbers will not
> > scale as we increase server counts both using custom inhouse tests and
> > ycsb.
> >
> > We're using hbase 0.92 on hadoop 0.20.2(we also experience the same
> > issues using 0.90 before switching our testing to  this version).
> >
> > Our cluster consists of:
> > - Namenode and hmaster on separate servers, 24 core, 64gb
> > - up to 11 datanode/regionservers. 24 core, 64gb, 4 * 1tb disks(hope
> > to get this changed)
> >
> > We have adjusted our gc settings, and mslabs:
> >
> >   <property>
> >     <name>hbase.hregion.memstore.mslab.enabled</name>
> >     <value>true</value>
> >   </property>
> >
> >   <property>
> >     <name>hbase.hregion.memstore.mslab.chunksize</name>
> >     <value>2097152</value>
> >   </property>
> >
> >   <property>
> >     <name>hbase.hregion.memstore.mslab.max.allocation</name>
> >     <value>1024768</value>
> >   </property>
> >
> > hdfs xceivers is set to 8192
> >
> > We've experimented with a variety of handler counts for namenode,
> > datanodes and regionservers with no changes in throughput.
> >
> > For testing with ycsb, we do the following each time(with nothing else
> > using the cluster):
> > - truncate test table
> > - add a small amount of data, then split the table into 32 regions and
> > call balancer from the shell.
> > - load 10m rows
> > - do a 1:2:7 insert:update:read test with 10million rows (64k/sec)
> > - do a 5:5 insert:update test with 10 million rows (23k/sec)
> > - do a pure read test with 10 million rows (75k/sec)
> >
> > We have observed ganglia, iostat -d -x, iptraf, top, dstat and a
> > variety of other diagnostic tools and network/io/cpu/memory as
> > bottlenecks seem highly unlikely as none of them are  ever seriously
> > taxed. This leave me to assume this is some kind of locking issue?
> > Delaying WAL flushes gives a small throughput bump but it doesn't
> > scale.
> >
> > There also doesn't seem to be many figures around to compare ours to.
> > We can get our throughput numbers higher with tricks like not writing
> > the WAL or delaying flushes, batching requests, but nothing seems to
> > scale with additional slaves.
> > Could anyone provide guidance as to what may be preventing throughput
> > figures from scaling as we increase our slave count?
>
>

Reply via email to