Nilotpal Nandi created HDDS-1125:
------------------------------------
Summary: java.lang.InterruptedException seen in datanode logs
Key: HDDS-1125
URL: https://issues.apache.org/jira/browse/HDDS-1125
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Nilotpal Nandi
steps taken :
--------------------
# created 12 datanodes cluster and running workload on all the nodes
exception seen :
-------------------------
{noformat}
2019-02-15 10:16:48,713 ERROR org.apache.ratis.server.impl.LogAppender:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
2019-02-15 10:16:48,714 ERROR org.apache.ratis.server.impl.LogAppender:
GrpcLogAppender(943007c8-4fdd-4926-89e2-2c8c52c05073 ->
8c77b16b-8054-49e3-b669-1ff759cfd271) hit IOException while loading raft log
org.apache.ratis.server.storage.RaftLogIOException:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:440)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
... 6 more
2019-02-15 10:16:48,715 ERROR org.apache.ratis.server.impl.LogAppender:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
2019-02-15 10:16:48,715 ERROR org.apache.ratis.server.impl.LogAppender:
GrpcLogAppender(943007c8-4fdd-4926-89e2-2c8c52c05073 ->
a40a7b01-a30b-469c-b373-9fcb20a126ed) hit IOException while loading raft log
org.apache.ratis.server.storage.RaftLogIOException:
943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3,
i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:440)
at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
at
org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at
org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
... 6 more
2019-02-15 10:16:48,723 WARN
org.apache.ratis.grpc.client.GrpcClientProtocolService:
943007c8-4fdd-4926-89e2-2c8c52c05073-5: onError:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED:
cancelled before receiving half close
2019-02-15 10:16:48,724 INFO org.apache.ratis.server.impl.RoleInfo:
943007c8-4fdd-4926-89e2-2c8c52c05073: start FollowerState
2019-02-15 10:16:48,726 INFO org.apache.ratis.server.impl.RaftServerImpl:
943007c8-4fdd-4926-89e2-2c8c52c05073 changes role from FOLLOWER to FOLLOWER at
term 4 for recognizeCandidate:8c77b16b-8054-49e3-b669-1ff759cfd271
2019-02-15 10:16:48,726 INFO org.apache.ratis.server.impl.RoleInfo:
943007c8-4fdd-4926-89e2-2c8c52c05073: shutdown FollowerState
2019-02-15 10:16:48,726 INFO org.apache.ratis.server.impl.RoleInfo:
943007c8-4fdd-4926-89e2-2c8c52c05073: start FollowerState
2019-02-15 10:16:48,726 INFO org.apache.ratis.server.impl.FollowerState:
943007c8-4fdd-4926-89e2-2c8c52c05073: FollowerState was interrupted:
java.lang.InterruptedException: sleep interrupted
2019-02-15 10:16:48,727 INFO org.apache.ratis.server.impl.RaftServerImpl:
943007c8-4fdd-4926-89e2-2c8c52c05073: change Leader from null to
8c77b16b-8054-49e3-b669-1ff759cfd271 at term 4 for appendEntries, leader
elected after 17ms
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
