Thanks Ben. See https://issues.apache.org/jira/browse/HBASE-5228
Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) >________________________________ > From: Ben West <[email protected]> >To: "[email protected]" <[email protected]>; Andrew Purtell ><[email protected]> >Sent: Wednesday, January 18, 2012 7:18 AM >Subject: Re: HBase 0.92rc3 rest performance > >Thanks for the quick responses! > >No higher CPU or memory usage as far as I can tell. No WARNs. We're using the >default Jersey (1.4) and Jetty (6.1.26). > >Yes, you only need to start up some number of concurrent GETs. Here is my test >script, if that helps. It is quite simple (in ksh): > >maxId=274894038 >for i in {1..1000} >do > # get a random number > hex=`dd if=/dev/urandom bs=1 count=8 2>/dev/null | > od -tx1 | head -1 | cut -d' ' -f2- | > tr -d ' ' | tr '[a-f]' '[A-F]'` > # convert from hexadecimal to decimal: > dec=`echo "ibase=16; $hex" | bc` > # echo >&2 "DEBUG: hex=<$hex>; dec=<$dec>" > dec=$(( dec % maxId)) > #echo "$dec" > start=$(date +%s%N) > curl -silent http://server.com:8080/table/$dec > /dev/null > elapsed=$(($(date +%s%N) - $start)) > echo $elapsed >done > >I have another script like > >for i in {1..100} >do > ksh myScript >> logfile & >done > > >I will try jstack to see if I can find anything, thanks for the suggestion. > > >----- Original Message ----- >From: Andrew Purtell <[email protected]> >To: "[email protected]" <[email protected]> >Cc: >Sent: Tuesday, January 17, 2012 5:48 PM >Subject: Re: HBase 0.92rc3 rest performance > >jstacks could help point out if the REST server has some internal lock or >monitor contention. > >Internal to the REST server is just the HBase Java client. REST uses a >HTablePool to manage a small pool of HTable instances that interact with the >cluster according to the request at hand. Client changes could show up in REST >but it would be odd (and possibly a REST bug) to see them only there. > >Other things to check: > > - What version of Jetty? > > - What version of Jersey? > > - Any WARNs in logs from the REST server? > >Reproducing this would be as simple as starting up 50 or so concurrent fetches? > >Best regards, > > > - Andy > >Problems worthy of attack prove their worth by hitting back. - Piet Hein (via >Tom White) > > >----- Original Message ----- >> From: Jean-Daniel Cryans <[email protected]> >> To: [email protected] >> Cc: >> Sent: Tuesday, January 17, 2012 1:45 PM >> Subject: Re: HBase 0.92rc3 rest performance >> >>T his seems to single out the REST server since thrift and native >> clients stayed the same. >> >> Can you provide us your test so we can do testing on our side too? >> >> Maybe doing a few jstacks on the REST server could point out the >> obvious bottlenecks. >> >> J-D >> >> On Tue, Jan 17, 2012 at 11:25 AM, Ben West <[email protected]> >> wrote: >>> Hi all >>> >>> We're trying out .92rc3 instead of .90.4, and for the most part >> everything seems fine. But we have a simple test of REST performance which >> is >> basically a large number of cURL jobs getting random rows, and this test is >> running *a lot* slower under .92. >>> >>> When we run just a single client doing REST GETs, the performance is fine. >> But once I have dozens or hundreds of clients, performance is ~20x worse >> than >> under .90.4 (response time is 7-800ms instead of 40-50ms). >>> >>> YCSB has pretty much the same performance under both versions, as do other >> internal tools measuring Thrift and native performance, so I don't feel like >> this is a problem with HBase coresetup (although it could be). I don't see >> anything suspicious in any logs, IO and CPU utilization are both low. Has >> anyone >> run into this or have thoughts on how to troubleshoot? >>> >>> Thanks! >>> Ben >> > > > >
