Daniel created HDDS-6443:
----------------------------
Summary: Datanode on K8s service is abnormal: UNAVAILABLE: io
exception
Key: HDDS-6443
URL: https://issues.apache.org/jira/browse/HDDS-6443
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 1.2.1
Environment: The environment information is as follows:
1. Kubernetes: RK2 deployment K8s environment
Rke2 version v1.21.9 + rke2r1
2. Ozone version: 1.2.1
Reporter: Daniel
h2. 1. Test demo
Kubernetes Demo is provided in the following path:
Ozone - 1.2.1 / kubernetes/examples/ozone
h2. 2. Process
Use the following command to deploy Ozone:
kubectl apply-f .
Error messages are displayed on three DataNode nodes:
{code:java}
2022-03-14 07:07:46 INFO RaftServerConfigKeys:44 -
raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:46 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29
2022-03-14 07:07:46 INFO LeaderElection:306 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29
ELECTION round 0: submit vote requests at term 23 for 0:
[e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0,
8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:1,
f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:0],
old=null
2022-03-14 07:07:46 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:47 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:47 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:47 INFO LeaderElection:89 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29:
ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:47 INFO LeaderElection:89 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28:
ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:47 INFO LeaderElection:137 - Exception 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:47 INFO LeaderElection:308 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29
ELECTION round 0: result REJECTED
2022-03-14 07:07:47 INFO LeaderElection:308 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28
ELECTION round 0: result REJECTED
2022-03-14 07:07:47 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from
CANDIDATE to FOLLOWER at term 23 for REJECTED
2022-03-14 07:07:47 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from
CANDIDATE to FOLLOWER at term 20 for REJECTED
2022-03-14 07:07:47 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection29
2022-03-14 07:07:47 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection28
2022-03-14 07:07:47 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:47 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:52 INFO FollowerState:134 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState: change
to CANDIDATE, lastRpcElapsedTime:5083479945ns, electionTimeout:5083ms
2022-03-14 07:07:52 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:52 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from
FOLLOWER to CANDIDATE at term 20 for changeToCandidate
2022-03-14 07:07:52 INFO RaftServerConfigKeys:44 -
raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30
2022-03-14 07:07:52 INFO LeaderElection:306 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30
ELECTION round 0: submit vote requests at term 21 for 0:
[e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0,
8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:0,
f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:1],
old=null
2022-03-14 07:07:52 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:89 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30:
ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:308 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30
ELECTION round 0: result REJECTED
2022-03-14 07:07:52 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from
CANDIDATE to FOLLOWER at term 21 for REJECTED
2022-03-14 07:07:52 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection30
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:52 INFO FollowerState:134 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState: change
to CANDIDATE, lastRpcElapsedTime:5155386314ns, electionTimeout:5155ms
2022-03-14 07:07:52 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:52 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from
FOLLOWER to CANDIDATE at term 23 for changeToCandidate
2022-03-14 07:07:52 INFO RaftServerConfigKeys:44 -
raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31
2022-03-14 07:07:52 INFO LeaderElection:306 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31
ELECTION round 0: submit vote requests at term 24 for 0:
[e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0,
8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:1,
f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:0],
old=null
2022-03-14 07:07:52 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:89 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31:
ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:137 - Exception 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:52 INFO LeaderElection:308 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31
ELECTION round 0: result REJECTED
2022-03-14 07:07:52 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from
CANDIDATE to FOLLOWER at term 24 for REJECTED
2022-03-14 07:07:52 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection31
2022-03-14 07:07:52 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:57 INFO FollowerState:134 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState: change
to CANDIDATE, lastRpcElapsedTime:5070286545ns, electionTimeout:5070ms
2022-03-14 07:07:57 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:57 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from
FOLLOWER to CANDIDATE at term 21 for changeToCandidate
2022-03-14 07:07:57 INFO RaftServerConfigKeys:44 -
raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32
2022-03-14 07:07:57 INFO LeaderElection:306 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32
ELECTION round 0: submit vote requests at term 22 for 0:
[e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0,
8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:0,
f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:1],
old=null
2022-03-14 07:07:57 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:89 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32:
ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:308 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32
ELECTION round 0: result REJECTED
2022-03-14 07:07:57 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC: changes role from
CANDIDATE to FOLLOWER at term 22 for REJECTED
2022-03-14 07:07:57 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-LeaderElection32
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-5C6EE74DD0AC-FollowerState
2022-03-14 07:07:57 INFO FollowerState:134 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState: change
to CANDIDATE, lastRpcElapsedTime:5088311189ns, electionTimeout:5088ms
2022-03-14 07:07:57 INFO RoleInfo:108 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
2022-03-14 07:07:57 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from
FOLLOWER to CANDIDATE at term 24 for changeToCandidate
2022-03-14 07:07:57 INFO RaftServerConfigKeys:44 -
raft.server.leaderelection.pre-vote = false (custom)
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33
2022-03-14 07:07:57 INFO LeaderElection:306 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33
ELECTION round 0: submit vote requests at term 25 for 0:
[e234f526-1428-46aa-b0d8-38c691c3be9b|rpc:10.42.10.15:9856|admin:10.42.10.15:9857|client:10.42.10.15:9858|dataStream:|priority:0,
8b4da263-b3d6-413e-8365-50f05ed24042|rpc:10.42.9.21:9856|admin:10.42.9.21:9857|client:10.42.9.21:9858|dataStream:|priority:1,
f99fd814-46cc-4192-b81d-1c98b5861cb7|rpc:10.42.7.23:9856|admin:10.42.7.23:9857|client:10.42.7.23:9858|dataStream:|priority:0],
old=null
2022-03-14 07:07:57 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:137 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33 got
exception when requesting votes: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:89 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33:
ELECTION REJECTED received 0 response(s) and 2 exception(s):
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 0:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:137 - Exception 1:
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
2022-03-14 07:07:57 INFO LeaderElection:308 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33
ELECTION round 0: result REJECTED
2022-03-14 07:07:57 INFO RaftServer$Division:290 -
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3: changes role from
CANDIDATE to FOLLOWER at term 25 for REJECTED
2022-03-14 07:07:57 INFO RoleInfo:121 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
shutdown
f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-LeaderElection33
2022-03-14 07:07:57 INFO RoleInfo:130 - f99fd814-46cc-4192-b81d-1c98b5861cb7:
start f99fd814-46cc-4192-b81d-1c98b5861cb7@group-AAC5FB8455D3-FollowerState
{code}
The logs for all three DataNode Pods are similar, and the logs are constantly
refreshed.
h2. Validation functions
kubectl exec --stdin -n ozone --tty datanode-0 -- /bin/bash
bash-4.2$ ozone sh volume create volume1
bash-4.2$ ozone sh bucket create /volume1/bucket1
INTERNAL_ERROR Allocated 0 blocks. Requested 1 blocks
According to the error log information，we can see that the DataNode is abnormal.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
