On Thu, Apr 21, 2011 at 10:49 PM, Dmitriy Lyubimov <dlie...@gmail.com> wrote: > Anyway. For a million requests shot at a region server at various > speeds between 300 and 500 qps the picture is not pretty. RPC metrics > are arctually good -- no more than 1ms average per next() and 0 per > get(). So region server is lightning fast. >
This is 3-500 queries per second of 40 rows each? > What doesn't seem so fast is RPC. OK. > As i reported before, i was getting > 25ms TTLB under the circumstances. In this case all the traffic to the > node goes thru same client (but in reality of course the node's > portion per client should be much less). All that traffic is using > single regionserver node rpc queue as HConnection would not open more > than one socket to same region. You saw "HBASE-2939 Allow Client-Side Connection Pooling"? Would that help? > And tcp doesn't seem to perform very > well for some reason in this scenario. > What do you see here D? > The next thing i did was to enable tcp_nodelay on both client and > server. That got us down even more to 13ms average. > Thats a big difference. > However, it is still about two times slower if i run all processes at > the same machine (i get around 6-7ms average TTLBs for the same type > of scan). > > Ping time for about same packet size between hosts involved seems to > revolve around 1ms. Where another 5ms average time are getting lost is > still a mystery. But oh well i guess it is as good as it gets. > In real life hbase applications traffic would be much more uniformly > distributed among regions and this would be much less of an issue > perhaps. > > I also suspect that using udp for short scans and gets might reduce > latency a bit as well. > Thank you Dmitriy for digging in. Good stuff. St.Ack
