Dmitriy, Did I hear you say that you are instantiating a new Htable for each request? Or was that somebody else?
On Thu, Apr 21, 2011 at 11:04 PM, Stack <[email protected]> wrote: > On Thu, Apr 21, 2011 at 10:49 PM, Dmitriy Lyubimov <[email protected]> > wrote: > > Anyway. For a million requests shot at a region server at various > > speeds between 300 and 500 qps the picture is not pretty. RPC metrics > > are arctually good -- no more than 1ms average per next() and 0 per > > get(). So region server is lightning fast. > > > > This is 3-500 queries per second of 40 rows each? > > > What doesn't seem so fast is RPC. > > OK. > > > As i reported before, i was getting > > 25ms TTLB under the circumstances. In this case all the traffic to the > > node goes thru same client (but in reality of course the node's > > portion per client should be much less). All that traffic is using > > single regionserver node rpc queue as HConnection would not open more > > than one socket to same region. > > You saw "HBASE-2939 Allow Client-Side Connection Pooling"? Would that > help? > > > > And tcp doesn't seem to perform very > > well for some reason in this scenario. > > > > What do you see here D? > > > > The next thing i did was to enable tcp_nodelay on both client and > > server. That got us down even more to 13ms average. > > > > Thats a big difference. > > > > However, it is still about two times slower if i run all processes at > > the same machine (i get around 6-7ms average TTLBs for the same type > > of scan). > > > > Ping time for about same packet size between hosts involved seems to > > revolve around 1ms. Where another 5ms average time are getting lost is > > still a mystery. But oh well i guess it is as good as it gets. > > In real life hbase applications traffic would be much more uniformly > > distributed among regions and this would be much less of an issue > > perhaps. > > > > I also suspect that using udp for short scans and gets might reduce > > latency a bit as well. > > > > Thank you Dmitriy for digging in. Good stuff. > St.Ack >
