[ https://issues.apache.org/jira/browse/HDFS-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480017#comment-16480017 ]
KaiXu commented on HDFS-8298: ----------------------------- The same issue on HDP2.6, any comments on how to workaround? 2018-05-18 09:31:39,653 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: flush failed for required journal (JournalAndStream(mgr=QJM to [172.31.48.34:8485, 172.31.48.54:8485, 172.31.48.64:8485], stream=QuorumOutputStream starting at txid 9751808)) java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond. at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137) at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107) at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113) at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107) at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:533) at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393) at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:57) at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:529) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:707) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:641) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2691) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2556) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:408) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345) 2018-05-18 09:31:39,654 WARN client.QuorumJournalManager (QuorumOutputStream.java:abort(72)) - Aborting QuorumOutputStream starting at txid 9751808 2018-05-18 09:31:39,654 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1647)) - BLOCK* neededReplications = 0, pendingReplications = 0. 2018-05-18 09:31:39,663 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2018-05-18 09:31:39,691 INFO namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: > HA: NameNode should not shut down completely without quorum, doesn't recover > from temporary network outages > ----------------------------------------------------------------------------------------------------------- > > Key: HDFS-8298 > URL: https://issues.apache.org/jira/browse/HDFS-8298 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode, qjm > Affects Versions: 2.6.0, 2.7.3 > Environment: multiple clients, HDP 2.2, HDP 2.5, CDH etc > Reporter: Hari Sekhon > Priority: Major > > In an HDFS HA setup if there is a temporary problem with contacting journal > nodes (eg. network interruption), the NameNode shuts down entirely, when it > should instead go in to a standby mode so that it can stay online and retry > to achieve quorum later. > If both NameNodes shut themselves off like this then even after the temporary > network outage is resolved, the entire cluster remains offline indefinitely > until operator intervention, whereas it could have self-repaired after > re-contacting the journalnodes and re-achieving quorum. > {code}2015-04-15 15:59:26,900 FATAL namenode.FSEditLog > (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: flush failed for > required journal (JournalAndStre > am(mgr=QJM to [<ip>:8485, <ip>:8485, <ip>:8485], stream=QuorumOutputStream > starting at txid 54270281)) > java.io.IOException: Interrupted waiting 20000ms for a quorum of nodes to > respond. > at > org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:134) > at > org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107) > at > org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113) > at > org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107) > at > org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:533) > at > org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393) > at > org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:57) > at > org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:529) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:639) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:388) > at java.lang.Thread.run(Thread.java:745) > 2015-04-15 15:59:26,901 WARN client.QuorumJournalManager > (QuorumOutputStream.java:abort(72)) - Aborting QuorumOutputStream starting at > txid 54270281 > 2015-04-15 15:59:26,904 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - > Exiting with status 1 > 2015-04-15 15:59:27,001 INFO namenode.NameNode (StringUtils.java:run(659)) - > SHUTDOWN_MSG: > /************************************************************ > SHUTDOWN_MSG: Shutting down NameNode at <custom_scrubbed>/<ip> > ************************************************************/{code} > Hari Sekhon > http://www.linkedin.com/in/harisekhon -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org