Thanks, good advice. At a cursory glance at region server logs (didn't find anything interesting in master logs), I already see some GC pauses and complaints about "Slow sync cost". Will ask here with further details if need be after further analysis.
Cheers. ---- Saad On Tue, Apr 19, 2016 at 6:35 PM, Stack <st...@duboce.net> wrote: > On Tue, Apr 19, 2016 at 2:07 PM, Saad Mufti <saad.mu...@gmail.com> wrote: > > > Hi, > > > > I found this blog post from 2014 on sources of HBase client side latency > > which I found useful: > > > > > > > https://hadoop-hbase.blogspot.com/2014/08/hbase-client-response-times.html?showComment=1461099797978#c5266762058464276023 > > > > > It is written by an authority. > > > > > Since this is a bit dated, anyone have any other sources of latency to > add? > > In our experience with HBase 1.x so far we've definitely seen long > > latencies when a region server dies, but we've also seen it during > > auto-splits which this post suggests shouldn't be that long. > > > > > Easy enough to check. Look at the master log and see the steps involved. > > > > > Then we have other unexplained (so far, at least by us) big response time > > spikes and client side timeouts that can last a few minutes or in some > > cases a couple of hours that we'd like to explain. Could these be from > > either automatic major compaction or minor compactions that got upgraded > to > > major? > > > > > These may up latency when running but not for minutes or hours. > > Sounds like something else is going on. > > > > > Any advice on where to start looking to investigate? > > > > > Check master log at the time of slowness and then follow your nose. If you > need more specifics on how to debug, come back here w/ some more detail > around a particular event. > > Yours, > St.Ack > > > > Thanks. > > > > ---- > > Saad > > >
