Thanks for the quick responses!
No higher CPU or memory usage as far as I can tell. No WARNs. We're using the
default Jersey (1.4) and Jetty (6.1.26).
Yes, you only need to start up some number of concurrent GETs. Here is my test
script, if that helps. It is quite simple (in ksh):
maxId=274894038
for i in {1..1000}
do
# get a random number
hex=`dd if=/dev/urandom bs=1 count=8 2>/dev/null |
od -tx1 | head -1 | cut -d' ' -f2- |
tr -d ' ' | tr '[a-f]' '[A-F]'`
# convert from hexadecimal to decimal:
dec=`echo "ibase=16; $hex" | bc`
# echo >&2 "DEBUG: hex=<$hex>; dec=<$dec>"
dec=$(( dec % maxId))
#echo "$dec"
start=$(date +%s%N)
curl -silent http://server.com:8080/table/$dec > /dev/null
elapsed=$(($(date +%s%N) - $start))
echo $elapsed
done
I have another script like
for i in {1..100}
do
ksh myScript >> logfile &
done
I will try jstack to see if I can find anything, thanks for the suggestion.
----- Original Message -----
From: Andrew Purtell <[email protected]>
To: "[email protected]" <[email protected]>
Cc:
Sent: Tuesday, January 17, 2012 5:48 PM
Subject: Re: HBase 0.92rc3 rest performance
jstacks could help point out if the REST server has some internal lock or
monitor contention.
Internal to the REST server is just the HBase Java client. REST uses a
HTablePool to manage a small pool of HTable instances that interact with the
cluster according to the request at hand. Client changes could show up in REST
but it would be odd (and possibly a REST bug) to see them only there.
Other things to check:
- What version of Jetty?
- What version of Jersey?
- Any WARNs in logs from the REST server?
Reproducing this would be as simple as starting up 50 or so concurrent fetches?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via
Tom White)
----- Original Message -----
> From: Jean-Daniel Cryans <[email protected]>
> To: [email protected]
> Cc:
> Sent: Tuesday, January 17, 2012 1:45 PM
> Subject: Re: HBase 0.92rc3 rest performance
>
>T his seems to single out the REST server since thrift and native
> clients stayed the same.
>
> Can you provide us your test so we can do testing on our side too?
>
> Maybe doing a few jstacks on the REST server could point out the
> obvious bottlenecks.
>
> J-D
>
> On Tue, Jan 17, 2012 at 11:25 AM, Ben West <[email protected]>
> wrote:
>> Hi all
>>
>> We're trying out .92rc3 instead of .90.4, and for the most part
> everything seems fine. But we have a simple test of REST performance which is
> basically a large number of cURL jobs getting random rows, and this test is
> running *a lot* slower under .92.
>>
>> When we run just a single client doing REST GETs, the performance is fine.
> But once I have dozens or hundreds of clients, performance is ~20x worse than
> under .90.4 (response time is 7-800ms instead of 40-50ms).
>>
>> YCSB has pretty much the same performance under both versions, as do other
> internal tools measuring Thrift and native performance, so I don't feel like
> this is a problem with HBase coresetup (although it could be). I don't see
> anything suspicious in any logs, IO and CPU utilization are both low. Has
> anyone
> run into this or have thoughts on how to troubleshoot?
>>
>> Thanks!
>> Ben
>
