[
https://issues.apache.org/jira/browse/HDDS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740953#comment-17740953
]
Hongbing Wang commented on HDDS-8985:
-------------------------------------
Hi, [~adoroszlai] May I ask a similar question? When I run the test in
*integration-test* locally (e.g. org.apache.hadoop.ozone.om.TestListStatus), I
encounter the same error, as follows:
{noformat}
...
2023-07-07 16:06:49,649 [main] INFO node.SCMNodeManager
(SCMNodeManager.java:<init>(157)) - Entering startup safe mode.
...
2023-07-07 16:08:47,527
[67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64] INFO
impl.LeaderElection (LogUtils.java:infoOrTrace(137)) - Exception 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNKNOWN: Channel
Pipeline: [ProtocolNegotiators$ProxyProtocolNegotiationHandler#0,
WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
2023-07-07 16:08:47,527
[67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64] INFO
impl.LeaderElection (LogUtils.java:infoOrTrace(137)) - Exception 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNKNOWN: Channel
Pipeline: [ProtocolNegotiators$ProxyProtocolNegotiationHandler#0,
WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
2023-07-07 16:08:47,527
[67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64] INFO
impl.LeaderElection (LeaderElection.java:askForVotes(323)) -
67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64
PRE_VOTE round 0: result REJECTED
2023-07-07 16:08:47,527
[67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64] INFO
server.RaftServer$Division (RaftServerImpl.java:setRole(329)) -
67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3: changes role from
CANDIDATE to FOLLOWER at term 0 for REJECTED
2023-07-07 16:08:47,527
[67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64] INFO
impl.RoleInfo (RoleInfo.java:shutdownLeaderElection(130)) -
67b10fc6-e776-4b37-8d09-463dd175232b: shutdown
67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64
2023-07-07 16:08:47,527
[67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-LeaderElection64] INFO
impl.RoleInfo (RoleInfo.java:updateAndGet(139)) -
67b10fc6-e776-4b37-8d09-463dd175232b: start
67b10fc6-e776-4b37-8d09-463dd175232b@group-F0F3BF71F1D3-FollowerState
2023-07-07 16:08:47,603 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(224)) - Nodes are
ready. Got 3 of 3 DN Heartbeats.
2023-07-07 16:08:47,604 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(227)) - Waiting for
cluster to exit safe mode
2023-07-07 16:08:47,604 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(229)) - SCM became
leader
2023-07-07 16:08:48,609 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(224)) - Nodes are
ready. Got 3 of 3 DN Heartbeats.
2023-07-07 16:08:48,609 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(227)) - Waiting for
cluster to exit safe mode
2023-07-07 16:08:48,609 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(229)) - SCM became
leader
2023-07-07 16:08:49,614 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(224)) - Nodes are
ready. Got 3 of 3 DN Heartbeats.
2023-07-07 16:08:49,614 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(227)) - Waiting for
cluster to exit safe mode
2023-07-07 16:08:49,614 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(229)) - SCM became
leader
2023-07-07 16:08:50,617 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(224)) - Nodes are
ready. Got 3 of 3 DN Heartbeats.
2023-07-07 16:08:50,617 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(227)) - Waiting for
cluster to exit safe mode
2023-07-07 16:08:50,617 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(229)) - SCM became
leader
2023-07-07 16:08:51,622 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(224)) - Nodes are
ready. Got 3 of 3 DN Heartbeats.
2023-07-07 16:08:51,622 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(227)) - Waiting for
cluster to exit safe mode
2023-07-07 16:08:51,622 [main] INFO ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(229)) - SCM became
leader
...
{noformat}
Do you have the experience to solve this problem ?
> Intermittent timeout exiting safe mode in HA secure tests
> ---------------------------------------------------------
>
> Key: HDDS-8985
> URL: https://issues.apache.org/jira/browse/HDDS-8985
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Attila Doroszlai
> Assignee: Attila Doroszlai
> Priority: Major
>
> Secure HA acceptance tests frequently time out waiting for SCM exit from safe
> mode. It seems to have gotten worse recently.
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/01/06/19373/acceptance-HA/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/01/12/19500/acceptance-HA/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/02/20/20232/acceptance-HA/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/03/15/20775/acceptance-HA/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/03/16/20834/acceptance-HA/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/03/20/20900/acceptance-HA/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/04/24/21817/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/04/27/21907/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/03/22001/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/08/22169/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/16/22432/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/19/22543/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/30/22797/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/31/22858/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/02/22920/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/06/23030/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/19/23503/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/21/23569/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/22/23656/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/26/23761/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/27/23799/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/27/23807/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/29/23917/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/30/23936/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/30/23938/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/30/23956/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/07/06/24043/acceptance-HA-secure/output.log
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/07/06/24047/acceptance-HA-secure/output.log
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
