What would be the best way to check? Region Logs? Sent from my iPhone
On Dec 22, 2012, at 6:33 AM, ramkrishna vasudevan <[email protected]> wrote: > How many threads are doing the puts? > Is there some flushes happening parallely? Are there too many scans going > on in paralle. > > Increasing the Xceivers could solve this problem if they are of shortage. > > Regards > Ram > > On Sat, Dec 22, 2012 at 5:42 PM, Mohammad Tariq <[email protected]> wrote: > >> yeah >> >> Best Regards, >> Tariq >> +91-9741563634 >> https://mtariq.jux.com/ >> >> >> On Sat, Dec 22, 2012 at 6:57 AM, Mohit Anchlia <[email protected] >>> wrote: >> >>> You mean batch multiple put? >>> >>> On Fri, Dec 21, 2012 at 4:16 PM, Mohammad Tariq <[email protected]> >>> wrote: >>> >>>> It might be the RS which could not complete the operation in time. The >>>> appropriate way to find out is to monitor that RS's metrics and see if >>>> anything unusual is happening there. What type of keys are you using? >> It >>> is >>>> time-series data?You might be a victim of RS hotspotting in that case >> or >>>> perhaps some other processes are eating up resources there. Try using >>>> "put(List<Put> puts)" instead of "put(Put put)" and see if it makes any >>>> difference. >>>> >>>> I'm afraid, I can't say anything with 100% confidence as there could be >>> 'n' >>>> reasons which are not traceable from here. Some of the possible reasons >>>> could be : >>>> hotspotting region >>>> too much I/O wait due to Swapping >>>> overloaded disk >>>> slowness due to high cpu consumption >>>> >>>> Best Regards, >>>> Tariq >>>> +91-9741563634 >>>> https://mtariq.jux.com/ >>>> >>>> >>>> On Sat, Dec 22, 2012 at 5:23 AM, Mohit Anchlia <[email protected] >>>>> wrote: >>>> >>>>> I am just doing a put. This operation generally takes 10ms but in >> this >>>> case >>>>> it took more than 10sec. Nothing out of ordinary in the logs >>>>> >>>>> On Fri, Dec 21, 2012 at 3:26 PM, Mohammad Tariq <[email protected]> >>>>> wrote: >>>>> >>>>>> what exactly is the operation your trying to do?how is your >> network's >>>>>> health?is swapping too high at RS side?anything odd in your RS >> logs? >>>>>> >>>>>> Best Regards, >>>>>> Tariq >>>>>> +91-9741563634 >>>>>> https://mtariq.jux.com/ >>>>>> >>>>>> >>>>>> On Sat, Dec 22, 2012 at 4:36 AM, Mohit Anchlia < >>> [email protected] >>>>>>> wrote: >>>>>> >>>>>>> I looked at that link, but couldn't find anything useful. How do >> I >>>>> check >>>>>> if >>>>>>> it was client who didn't write data within that time or if it was >>>>> region >>>>>>> server that didn't finish operation in time. >>>>>>> >>>>>>> On Fri, Dec 21, 2012 at 2:54 PM, Mohammad Tariq < >>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> The socket through which your client is communicating is >> getting >>>>> closed >>>>>>>> before the operation could get finished. May be it is taking >>> longer >>>>>> than >>>>>>>> usual or something. >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Tariq >>>>>>>> +91-9741563634 >>>>>>>> https://mtariq.jux.com/ >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Dec 22, 2012 at 4:08 AM, Mohammad Tariq < >>>> [email protected] >>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello Mohit, >>>>>>>>> >>>>>>>>> You might this link< >>>>>>>> http://hbase.apache.org/book/ops.monitoring.html>useful. >>>>>>>>> >>>>>>>>> Best Regards, >>>>>>>>> Tariq >>>>>>>>> +91-9741563634 >>>>>>>>> https://mtariq.jux.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Dec 22, 2012 at 2:09 AM, Mohit Anchlia < >>>>>> [email protected] >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Could someone help me understand what this really means. Is >>> this >>>>> the >>>>>>>>>> network transfer taking long from client -> server or region >>>>> server >>>>>>>> taking >>>>>>>>>> long time writing to the memory? >>>>>>>>>> >>>>>>>>>> 2012-12-21 10:54:21,980 WARN >>> org.apache.hadoop.ipc.HBaseServer: >>>>>>>>>> (responseTooSlow): {"processingtimems":135652,"call":"multi( >>>>>>>>>> org.apache.hadoop.hbase.client.MultiAction@28338472), rpc >>>>>> version=1, >>>>>>>>>> client >>>>>>>>>> version=29, methodsFingerPrint=54742778","client":" >>>>> 10.18.3.80:48218 >> ","starttimems":1356115926326,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} >>>>>>>>>> 2012-12-21 10:54:21,985 WARN >>> org.apache.hadoop.ipc.HBaseServer: >>>>> IPC >>>>>>>> Server >>>>>>>>>> handler 26 on 60020 caught: >>>>> java.nio.channels.ClosedChannelException >>>>>>>>>> at >>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) >>>>>>>>>> at >>>>>>>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) >>>>>>>>>> at >> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653) >>>>>>>>>> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924) >>>>>>>>>> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003) >>>>>>>>>> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409) >>>>>>>>>> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346) >>
