[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593155#comment-13593155 ] Hudson commented on HBASE-6748: --- Integrated in HBase-0.94-security-on-Hadoop-23 #12 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/12/]) HBASE-7914 Port the fix of HBASE-6748 into 0.94 branch (Jeffrey Zhong) (Revision 145) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.95.0, 0.94.6 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.95.0, 0.94.6 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586686#comment-13586686 ] Hudson commented on HBASE-6748: --- Integrated in HBase-0.94 #859 (See [https://builds.apache.org/job/HBase-0.94/859/]) HBASE-7914 Port the fix of HBASE-6748 into 0.94 branch (Jeffrey Zhong) (Revision 145) Result = SUCCESS tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586899#comment-13586899 ] Hudson commented on HBASE-6748: --- Integrated in HBase-0.94-security #112 (See [https://builds.apache.org/job/HBase-0.94-security/112/]) HBASE-7914 Port the fix of HBASE-6748 into 0.94 branch (Jeffrey Zhong) (Revision 145) Result = SUCCESS tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.95.0 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565991#comment-13565991 ] Jeffrey Zhong commented on HBASE-6748: -- @Lars, I can port it to 0.94.6. Thanks, -Jeffrey Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.6 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554324#comment-13554324 ] Hadoop QA commented on HBASE-6748: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564990/hbase-6748_1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4029//console This message is automatically generated. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554408#comment-13554408 ] Sergey Shelukhin commented on HBASE-6748: - Patch needs to be rebased. {code} if (failedDeletions.size() 0) { ListString tmpPaths = new ArrayListString(failedDeletions); ... failedDeletions.removeAll(tmpPaths); } {code} This is not thread safe, entries can be added during the deletes. In {code} if (needAbandonRetries(rc, {code} cases, the code doesn't call deleteNodeFailure/etc., which is called in case of failure below in each respective method. That was the existing behavior in call cases; however, createNodeFailure call was added to if (needAbandonRetries(rc, ...)) in create. Why is it inconsistent? Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554410#comment-13554410 ] Sergey Shelukhin commented on HBASE-6748: - *all cases Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803 ... [2012-09-01
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554417#comment-13554417 ] Sergey Shelukhin commented on HBASE-6748: - About the first one, sure, looked at it wrong, should be safe because Set will not allow multiple entries with the same path so remove will not remove multiple entries. However, what will happen if there are multiple deletes for the same path and they fail in parallel? Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554422#comment-13554422 ] Sergey Shelukhin commented on HBASE-6748: - Ah, nm, we discussed and the set will just drop the 2nd update. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554426#comment-13554426 ] Sergey Shelukhin commented on HBASE-6748: - +1; the latter concern also seems to preserve old behavior. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554503#comment-13554503 ] Ted Yu commented on HBASE-6748: --- Integrated to trunk. Thanks for the patch, Jeff. Thanks for the review, Sergey. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554632#comment-13554632 ] Hudson commented on HBASE-6748: --- Integrated in HBase-TRUNK #3752 (See [https://builds.apache.org/job/HBase-TRUNK/3752/]) HBASE-6748 Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback (Jeffrey Zhong) (Revision 1433733) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554650#comment-13554650 ] Hudson commented on HBASE-6748: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #349 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/349/]) HBASE-6748 Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback (Jeffrey Zhong) (Revision 1433733) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748_1.patch, hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553468#comment-13553468 ] Ted Yu commented on HBASE-6748: --- {code} + * @param ZooKeeper exception code integer value + * @return true when need to abandon retries otherwise false + */ + private boolean needAbandonRetries(int statusCode) { {code} @param doesn't match actual parameter. {code} - LOG.error(ZK session expired. Master is expected to shut down. Abandoning retries.); +if (needAbandonRetries(rc)) { + LOG.warn(GetData from znode + path + is abandoned.); {code} We would see two consecutive logs about abandoning retries. It is better to pass call site information as String to needAbandonRetries(). Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553478#comment-13553478 ] Hadoop QA commented on HBASE-6748: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564841/hbase-6748.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4019//console This message is automatically generated. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Assignee: Jeffrey Zhong Fix For: 0.96.0, 0.94.5 Attachments: hbase-6748.patch You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479628#comment-13479628 ] Lars Hofhansl commented on HBASE-6748: -- [~jeason], do you have a patch? Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1, 0.96.0 Reporter: Jieshan Bean Priority: Critical Fix For: 0.94.3, 0.96.0 You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803 ... [2012-09-01 11:41:03,307] [ERROR]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453587#comment-13453587 ] Jieshan Bean commented on HBASE-6748: - Yes. Long.MAX_VALUE is the problem. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0, 0.94.1 Reporter: Jieshan Bean Priority: Critical Fix For: 0.96.0, 0.94.3 You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803 ... [2012-09-01 11:41:03,307] [ERROR]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453665#comment-13453665 ] Jimmy Xiang commented on HBASE-6748: Did this happen when the master starts up, or during a region server failure handling? If the ZK session times out, the master will abort, right? Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0, 0.94.1 Reporter: Jieshan Bean Priority: Critical Fix For: 0.96.0, 0.94.3 You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453666#comment-13453666 ] Jieshan Bean commented on HBASE-6748: - Either. Both Master starts up and region server failure handling may trigger HLog splitting. Yes, I think HMaster should abort when sessionTimeout happens. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0, 0.94.1 Reporter: Jieshan Bean Priority: Critical Fix For: 0.96.0, 0.94.3 You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13452053#comment-13452053 ] Ted Yu commented on HBASE-6748: --- @Jieshan: Can you tell us under what circumstance the above recursion happened ? Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0, 0.94.1 Reporter: Jieshan Bean Priority: Critical You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803 ... [2012-09-01 11:41:03,307] [ERROR]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13452618#comment-13452618 ] Jieshan Bean commented on HBASE-6748: - From the logs, we can see SessionTimeoutException happened. So the request of deleteNode was rejected each time. DeleteAsyncCallback does not handle the exceptions correctly. Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0, 0.94.1 Reporter: Jieshan Bean Priority: Critical You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13452735#comment-13452735 ] Lars Hofhansl commented on HBASE-6748: -- Looks like the retry counter underflowed? Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0, 0.94.1 Reporter: Jieshan Bean Priority: Critical You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775803 ... [2012-09-01 11:41:03,307] [ERROR] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
[jira] [Commented] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[ https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13452739#comment-13452739 ] Lars Hofhansl commented on HBASE-6748: -- Looks like it coming from setDone: {code} // delete the task node in zk. Keep trying indefinitely - its an async // call and no one is blocked waiting for this node to be deleted. All // task names are unique (log.timestamp) there is no risk of deleting // a future task. deleteNode(path, Long.MAX_VALUE); {code} Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback --- Key: HBASE-6748 URL: https://issues.apache.org/jira/browse/HBASE-6748 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0, 0.94.1 Reporter: Jieshan Bean Priority: Critical You can ealily understand the problem from the below logs: {code} [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=3 [2012-09-01 11:41:02,062] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=2 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=1 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978] create rc =SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=0 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 [2012-09-01 11:41:02,063] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775807 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775806 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775805 [2012-09-01 11:41:02,064] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846 remaining retries=9223372036854775804 [2012-09-01 11:41:02,065] [WARN ] [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1] [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052] delete rc=SESSIONEXPIRED for