Thanks for getting back, Ted. I totally understand other priorities and will wait for some feedback. I am adding some more info to this post to allow better diagnosing of performance.
I hit my region servers with a lot of GET requests (~20K per second per regionserver) using asynchbase in my test environment, the storage pattern is very similar to OpenTSDB though with a lot more columns. Each row is around 45-50 bytes long. The regionservers have a lot of RAM available to them (48 out of 60 GB) and they are not sharing resources with anyone else, so memory is not under pressure. The total # of rows in the table is around 100M and growing (there is a put process too) GETs take over 15s for 16K rows, and I don't see any operationTooSlow logs in the regionserver logs either. PUTs take around 1s for 16K rows (deferred log flush is enabled though). I looked at the RPC stats and it seems the RPC threads were always doing something and I assumed my requests were waiting on handlers and so thought of experimenting by increasing number of handlers. But as mentioned in my thread, going above 10K kills my regionserver. Thanks, Viral On Mon, Apr 29, 2013 at 9:43 PM, Ted Yu <[email protected]> wrote: > Viral: > I am currently dealing with some high priority bugs so I didn't have time > to look deeper into your case. > > My feeling is that raising max regionserver handler count shouldn't be the > key to boosting performance. > > Cheers >
