On Tue, Oct 19, 2010 at 06:03:01PM -0600, Jeremy Hinegardner wrote: > Hi all, > > I've done a bit of search on this issue, and have yet to find anything > conclusive. As a test case to demonstrate it, I am using HBase 0.20.6 > and stargate. > > I have a test HBase cluster with 1 table and about 60M rows in it, and a > Stargate instance that talks to it. > > I have clients that queue up a random list of rowid's to query stargate via > http://stargate.example.com:3002/table/rowid like requests. > > When I have 3 concurrent clients querying stargate, they all do well and get > a consistent throughput. When I add the 4th client querying stargate, > stargate > comes to a screeching halt and everyone has 0 operations for a long while, > then > a small burst of requests will go through stargate and it will hang for a > while, > and repeat. > > If I then just kill one client, the other 3 start having a good consistent > throughput again. Bring back the 4th client and it comes to a halt. > > If I bypass Stargate completely, and have all clients use HTable instances > directly, then everyone is good. I can go up to as many clients as I need. > > This seems like quite a problem and I was wondering if anyone else is > seeing something similar. >
User error, sorry for the noise, I was not explicitly closing the HTTP connections to the stargate server, and with enough clients, all the server side worker threads used to process requests were locked up with open sockets. After some time, some of the client side connections would time out and close which would free up the server side worker threads, and a new batch would flow through. thank you for your help, -jeremy -- ======================================================================== Jeremy Hinegardner [email protected]
