Also, the latency on requests is extremely long. If we group them into
sets of 10 puts(128-256 bytes each) before flushing the client table,
latency is over 1 second.
We get entries like this in our logs:
22:17:51,010 WARN org.apache.hadoop.ipc.HBaseServer:
(responseTooSlow):
{"processingtimems":16692,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@65312e3b),
rpc version=1, client version=29,
methodsFingerPrint=54742778","client":"10.172.109.3:42725","starttimems":1332335854317,"queuetimems":6387,"class":"HRegionServer","responsesize":0,"method":"multi"}
Any suggestions as to where we should be digging?
On Fri, Mar 23, 2012 at 4:40 PM, Juhani Connolly <[email protected]> wrote:
> Status update:
>
> - We moved to cdh 4b1, so hbase 0.92 and hdfs 0.23(until now we were
> using 0.20.2 series)
> - Did the tests now with 256/512 regions, the numbers do appear to
> scale which is good.
>
> BUT, our write throughput has gone in the dump. If we disable wal
> writes, we still get nearly 40,000 a second, but with it on, we're
> lucky to get more than 12,000. Before we were getting as high as
> 70,000 grouping puts together. Have set up log collection, and am not
> finding anything unusual in the logs.
>
> Mikael: One of the tests is the ycsb one where we just let it choose
> the size. Our own custom test has a configurable size, but we have
> been testing with entries that are 128-256 bytes per entry, as this is
> what we expect in our application. What exactly should we be looking
> at with the storefiles?
>
> On Wed, Mar 21, 2012 at 2:29 PM, Mikael Sitruk <[email protected]>
> wrote:
>> Juhani,
>> Can you look at the storefiles and tell how they behave during the test?
>> What is the size of the data you insert/update?
>> Mikael
>> On Mar 20, 2012 8:10 PM, "Juhani Connolly" <[email protected]> wrote:
>>
>>> Hi Matt,
>>>
>>> this is something we haven't tested much, we were always running with
>>> about 32 regions which gave enough coverage for an even spread over
>>> all machines.
>>> I will run our tests with enough regions per server to cover all cores
>>> and get back to the ml
>>>
>>> On Tue, Mar 20, 2012 at 1:55 AM, Matt Corgan <[email protected]> wrote:
>>> > I'd be curious to see what happens if you split the table into 1 region
>>> per
>>> > CPU core, so 24 cores * 11 servers = 264 regions. Each region has 1
>>> > memstore which is a ConcurrentSkipListMap, and you're currently hitting
>>> > each CSLM with 8 cores which might be too contentious. Normally in
>>> > production you would want multiple memstores per CPU core.
>>> >
>>> >
>>> > On Mon, Mar 19, 2012 at 5:31 AM, Juhani Connolly <[email protected]>
>>> wrote:
>>> >
>>> >> Actually we did try running off two machines both running our own
>>> >> tests in parallel. Unfortunately the results were a split that results
>>> >> in the same total throughput. We also did the same thing with iperf
>>> >> running from each machine to another machine, indicating 800Mb
>>> >> additional throughput between each pair of machines.
>>> >> However we didn't try these tests very thoroughly so I will revisit
>>> >> them as soon as I get back to the office, thanks.
>>> >>
>>> >> On Mon, Mar 19, 2012 at 9:21 PM, Christian Schäfer <
>>> [email protected]>
>>> >> wrote:
>>> >> > referring to my experiences I expect the client to be the bottleneck,
>>> >> too.
>>> >> >
>>> >> > So try to increase the count of client-machines (not client threads)
>>> >> each with its own unshared network interface.
>>> >> >
>>> >> > In my case I could double write throughput by doubling client machine
>>> >> count with a much smaller system than yours (5 machines, 4gigs RAM
>>> each).
>>> >> >
>>> >> > Good Luck
>>> >> > Chris
>>> >> >
>>> >> >
>>> >> >
>>> >> > ________________________________
>>> >> > Von: Juhani Connolly <[email protected]>
>>> >> > An: [email protected]
>>> >> > Gesendet: 13:02 Montag, 19.März 2012
>>> >> > Betreff: Re: 0.92 and Read/writes not scaling
>>> >> >
>>> >> > I was concerned that may be the case too, which is why we ran the ycsb
>>> >> > tests in addition to our application specific and general performance
>>> >> > tests. checking profiles of the execution just showed the vast
>>> majority
>>> >> of
>>> >> > time spent waiting for responses. these were all run with 400
>>> >> > threads(though we tried more/less just in case)
>>> >> > 2012/03/19 20:57 "Mingjian Deng" <[email protected]>:
>>> >> >
>>> >> >> @Juhani:
>>> >> >> How many clients did you test? Maybe the bottleneck was client?
>>> >> >>
>>> >> >> 2012/3/19 Ramkrishna.S.Vasudevan <[email protected]>
>>> >> >>
>>> >> >> > Hi Juhani
>>> >> >> >
>>> >> >> > Can you tell more on how the regions are balanced?
>>> >> >> > Are you overloading only specific region server alone?
>>> >> >> >
>>> >> >> > Regards
>>> >> >> > Ram
>>> >> >> >
>>> >> >> > > -----Original Message-----
>>> >> >> > > From: Juhani Connolly [mailto:[email protected]]
>>> >> >> > > Sent: Monday, March 19, 2012 4:11 PM
>>> >> >> > > To: [email protected]
>>> >> >> > > Subject: 0.92 and Read/writes not scaling
>>> >> >> > >
>>> >> >> > > Hi,
>>> >> >> > >
>>> >> >> > > We're running into a brick wall where our throughput numbers will
>>> >> not
>>> >> >> > > scale as we increase server counts both using custom inhouse
>>> tests
>>> >> and
>>> >> >> > > ycsb.
>>> >> >> > >
>>> >> >> > > We're using hbase 0.92 on hadoop 0.20.2(we also experience the
>>> same
>>> >> >> > > issues using 0.90 before switching our testing to this version).
>>> >> >> > >
>>> >> >> > > Our cluster consists of:
>>> >> >> > > - Namenode and hmaster on separate servers, 24 core, 64gb
>>> >> >> > > - up to 11 datanode/regionservers. 24 core, 64gb, 4 * 1tb
>>> disks(hope
>>> >> >> > > to get this changed)
>>> >> >> > >
>>> >> >> > > We have adjusted our gc settings, and mslabs:
>>> >> >> > >
>>> >> >> > > <property>
>>> >> >> > > <name>hbase.hregion.memstore.mslab.enabled</name>
>>> >> >> > > <value>true</value>
>>> >> >> > > </property>
>>> >> >> > >
>>> >> >> > > <property>
>>> >> >> > > <name>hbase.hregion.memstore.mslab.chunksize</name>
>>> >> >> > > <value>2097152</value>
>>> >> >> > > </property>
>>> >> >> > >
>>> >> >> > > <property>
>>> >> >> > > <name>hbase.hregion.memstore.mslab.max.allocation</name>
>>> >> >> > > <value>1024768</value>
>>> >> >> > > </property>
>>> >> >> > >
>>> >> >> > > hdfs xceivers is set to 8192
>>> >> >> > >
>>> >> >> > > We've experimented with a variety of handler counts for namenode,
>>> >> >> > > datanodes and regionservers with no changes in throughput.
>>> >> >> > >
>>> >> >> > > For testing with ycsb, we do the following each time(with nothing
>>> >> else
>>> >> >> > > using the cluster):
>>> >> >> > > - truncate test table
>>> >> >> > > - add a small amount of data, then split the table into 32
>>> regions
>>> >> and
>>> >> >> > > call balancer from the shell.
>>> >> >> > > - load 10m rows
>>> >> >> > > - do a 1:2:7 insert:update:read test with 10million rows
>>> (64k/sec)
>>> >> >> > > - do a 5:5 insert:update test with 10 million rows (23k/sec)
>>> >> >> > > - do a pure read test with 10 million rows (75k/sec)
>>> >> >> > >
>>> >> >> > > We have observed ganglia, iostat -d -x, iptraf, top, dstat and a
>>> >> >> > > variety of other diagnostic tools and network/io/cpu/memory as
>>> >> >> > > bottlenecks seem highly unlikely as none of them are ever
>>> seriously
>>> >> >> > > taxed. This leave me to assume this is some kind of locking
>>> issue?
>>> >> >> > > Delaying WAL flushes gives a small throughput bump but it doesn't
>>> >> >> > > scale.
>>> >> >> > >
>>> >> >> > > There also doesn't seem to be many figures around to compare ours
>>> >> to.
>>> >> >> > > We can get our throughput numbers higher with tricks like not
>>> >> writing
>>> >> >> > > the WAL or delaying flushes, batching requests, but nothing
>>> seems to
>>> >> >> > > scale with additional slaves.
>>> >> >> > > Could anyone provide guidance as to what may be preventing
>>> >> throughput
>>> >> >> > > figures from scaling as we increase our slave count?
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >>
>>>