[
https://issues.apache.org/jira/browse/HBASE-19554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300816#comment-16300816
]
Duo Zhang edited comment on HBASE-19554 at 12/22/17 1:43 AM:
-------------------------------------------------------------
Checked recent pre commit building, seems much better. And this a failure case
https://builds.apache.org/job/PreCommit-HBASE-Build/10609/artifact/patchprocess/patch-unit-hbase-server.txt
The error message
{noformat}
[ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 157.505
s <<< FAILURE! - in org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL
[ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL)
Time elapsed: 49.462 s <<< ERROR!
java.lang.RuntimeException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=11, exceptions:
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:59 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:00 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:02 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:06 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:16 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:26 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:36 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=11, exceptions:
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:59 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:00 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:02 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:06 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:16 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:26 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:36 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745
failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: java.net.ConnectException: Connection refused
{noformat}
The test output xml can not be generated which makes it really hard to find out
the real problem...
was (Author: apache9):
Checked recent pre commit building, seems much better. And this a failure case
https://builds.apache.org/job/PreCommit-HBASE-Build/10609/artifact/patchprocess/patch-unit-hbase-server.txt
The error message
{quote}
[ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 157.505
s <<< FAILURE! - in org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL
[ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL)
Time elapsed: 49.462 s <<< ERROR!
java.lang.RuntimeException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=11, exceptions:
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:59 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:00 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:02 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:06 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:16 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:26 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:36 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=11, exceptions:
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:59 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.io.IOException: Call to
604d085d7ec5/172.17.0.2:57745 failed on local exception:
org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:00 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:02 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:06 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:16 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:26 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:36 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131,
pause=100, maxAttempts=11}, java.net.ConnectException: Call to
604d085d7ec5/172.17.0.2:57745 failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745
failed on connection exception:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by:
org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: java.net.ConnectException: Connection refused
{quote}
The test output xml can not be generated which makes it really hard to find out
the real problem...
> AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit
> --------------------------------------------------------------
>
> Key: HBASE-19554
> URL: https://issues.apache.org/jira/browse/HBASE-19554
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Attachments: HBASE-19554.patch
>
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/10554/artifact/patchprocess/patch-unit-hbase-server.txt
> The error message is a bit strange:
> {quote}
> [ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL)
> Time elapsed: 20.627 s <<< ERROR!
> org.apache.hadoop.hbase.TableNotFoundException: Region of
> 'hbase:namespace,,1513320505933.451650152885a3b41d0b1110deca513c.' is
> expected in the table of 'testThreeRSAbort', but hbase:meta says it is in the
> table of 'hbase:namespace'. hbase:meta might be damaged.
> {quote}
> It fails for both FSHLog and AsyncFSWAL. Need to dig more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
