[jira] Commented: (HBASE-1394) Uploads sometimes fall to 0 requests/second (Binding up on HLog#append?)

[stack (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HBASE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707615#action_12707615
 ] 

stack commented on HBASE-1394:
------------------------------

This is our main bottleneck inputting to hdfs.  To see for yourself, add 
logging like below:

{code}
Index: src/java/org/apache/hadoop/hbase/regionserver/HLog.java
===================================================================
--- src/java/org/apache/hadoop/hbase/regionserver/HLog.java     (revision 
773167)
+++ src/java/org/apache/hadoop/hbase/regionserver/HLog.java     (working copy)
@@ -556,11 +556,19 @@
   
   private void doWrite(HLogKey logKey, HLogEdit logEdit, boolean sync)
   throws IOException {
+    long now = System.currentTimeMillis();
     try {
       this.writer.append(logKey, logEdit);
       if (sync || this.unflushedEntries.incrementAndGet() >= flushlogentries) {
         sync();
       }
+      /* Enable this if you want to see how sometimes we're filling the hdfs
+         queue of packets so we stall.  See HBASE-1394.
+      long took = System.currentTimeMillis() - now;
+      if (LOG.isDebugEnabled() && took > 1000) {
+        LOG.debug(Thread.currentThread().getName() + " took " + took + "ms to 
append.");
+      }
+      */
     } catch (IOException e) {
       LOG.fatal("Could not append. Requesting close of log", e);
       requestLogRoll();
{code}

We should work on the writing of files direct to HDFS for hbase to pick up 
afterward.

> Uploads sometimes fall to 0 requests/second (Binding up on HLog#append?)
> ------------------------------------------------------------------------
>
>                 Key: HBASE-1394
>                 URL: https://issues.apache.org/jira/browse/HBASE-1394
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>
> Trying to figure why rate sometimes goes to zero.
> Studying the reginoserver, HLog#append looks like a possible culprit.
> {code}
> "IPC Server handler 7 on 60021" daemon prio=10 tid=0x000000004057dc00 
> nid=0x1bc4 in Object.wait() [0x0000000043393000..0x0000000043393b80]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at 
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:2964)
>         - locked <0x00007f9e3e449ff0> (a java.util.LinkedList)
>         - locked <0x00007f9e3e449e18> (a 
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
>         at 
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
>         at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
>         at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
>         - locked <0x00007f9e3e449e18> (a 
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
>         at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
>         at java.io.DataOutputStream.write(DataOutputStream.java:90)
>         - locked <0x00007f9e434e5588> (a 
> org.apache.hadoop.fs.FSDataOutputStream)
>         at 
> org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1020)
>         - locked <0x00007f9e434e55c0> (a 
> org.apache.hadoop.io.SequenceFile$Writer)
>         at 
> org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:984)
>         - locked <0x00007f9e434e55c0> (a 
> org.apache.hadoop.io.SequenceFile$Writer)
>         at org.apache.hadoop.hbase.regionserver.HLog.doWrite(HLog.java:565)
>         at org.apache.hadoop.hbase.regionserver.HLog.append(HLog.java:521)
>         - locked <0x00007f9dfa376f70> (a java.lang.Object)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1777)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1348)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1289)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1727)
>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
>         at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.