Nilotpal Nandi created HDDS-722:
-----------------------------------
Summary: ozone datanodes failed to start on few nodes
Key: HDDS-722
URL: https://issues.apache.org/jira/browse/HDDS-722
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 0.3.0
Reporter: Nilotpal Nandi
steps taken :
------------------
# put few keys using ozonefs.
# stopped all services of the cluster.
# started om and scm.
# After sometime , started datanodes.
All datanodes failed to start . Out of 12 datanodes, 4 datanodes failed to
start.
Here is the datanode log snippet :
------------------------------------------------
{noformat}
2018-10-24 04:49:30,594 ERROR org.apache.ratis.server.impl.StateMachineUpdater:
Terminating with exit status 2:
StateMachineUpdater-9524f4e2-9031-4852-ab7c-11c2da3460db: the
StateMachineUpdater hits Throwable
org.apache.ratis.server.storage.RaftLogIOException: java.io.IOException:
Premature EOF from inputStream
at org.apache.ratis.server.storage.LogSegment.loadCache(LogSegment.java:299)
at
org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:192)
at
org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:142)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Premature EOF from inputStream
at org.apache.ratis.util.IOUtils.readFully(IOUtils.java:100)
at org.apache.ratis.server.storage.LogReader.decodeEntry(LogReader.java:250)
at org.apache.ratis.server.storage.LogReader.readEntry(LogReader.java:155)
at
org.apache.ratis.server.storage.LogInputStream.nextEntry(LogInputStream.java:128)
at
org.apache.ratis.server.storage.LogSegment.readSegmentFile(LogSegment.java:110)
at org.apache.ratis.server.storage.LogSegment.access$400(LogSegment.java:43)
at
org.apache.ratis.server.storage.LogSegment$LogEntryLoader.load(LogSegment.java:167)
at
org.apache.ratis.server.storage.LogSegment$LogEntryLoader.load(LogSegment.java:161)
at org.apache.ratis.server.storage.LogSegment.loadCache(LogSegment.java:295)
... 3 more
2018-10-24 04:49:30,598 INFO org.apache.hadoop.ozone.HddsDatanodeService:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HddsDatanodeService at
ctr-e138-1518143905142-541661-01-000003.hwx.site/172.27.57.0
************************************************************/
2018-10-24 04:49:30,598 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Thread
Interrupted waiting to refresh disk information: sleep interrupted
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
