jstacks could help point out if the REST server has some internal lock or monitor contention.
Internal to the REST server is just the HBase Java client. REST uses a HTablePool to manage a small pool of HTable instances that interact with the cluster according to the request at hand. Client changes could show up in REST but it would be odd (and possibly a REST bug) to see them only there. Other things to check: - What version of Jetty? - What version of Jersey? - Any WARNs in logs from the REST server? Reproducing this would be as simple as starting up 50 or so concurrent fetches? Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) ----- Original Message ----- > From: Jean-Daniel Cryans <[email protected]> > To: [email protected] > Cc: > Sent: Tuesday, January 17, 2012 1:45 PM > Subject: Re: HBase 0.92rc3 rest performance > >T his seems to single out the REST server since thrift and native > clients stayed the same. > > Can you provide us your test so we can do testing on our side too? > > Maybe doing a few jstacks on the REST server could point out the > obvious bottlenecks. > > J-D > > On Tue, Jan 17, 2012 at 11:25 AM, Ben West <[email protected]> > wrote: >> Hi all >> >> We're trying out .92rc3 instead of .90.4, and for the most part > everything seems fine. But we have a simple test of REST performance which is > basically a large number of cURL jobs getting random rows, and this test is > running *a lot* slower under .92. >> >> When we run just a single client doing REST GETs, the performance is fine. > But once I have dozens or hundreds of clients, performance is ~20x worse than > under .90.4 (response time is 7-800ms instead of 40-50ms). >> >> YCSB has pretty much the same performance under both versions, as do other > internal tools measuring Thrift and native performance, so I don't feel like > this is a problem with HBase coresetup (although it could be). I don't see > anything suspicious in any logs, IO and CPU utilization are both low. Has > anyone > run into this or have thoughts on how to troubleshoot? >> >> Thanks! >> Ben >
