[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file

2018-02-28 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380902#comment-16380902
 ] 

Ted Yu commented on HDFS-13206:
---

{code}
devtmpfs   65906460 0  65906460   0% /dev
tmpfs  65914268 0  65914268   0% /dev/shm
{code}

> IllegalStateException: Unable to finalize edits file
> 
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402): 
> Error: finalize log segment 1, 658 failed for (journal 
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
>  stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file 
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
>   at 
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
>   at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
>   at 
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file

2018-02-28 Thread Rushabh S Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380888#comment-16380888
 ] 

Rushabh S Shah commented on HDFS-13206:
---

{noformat}
2018-02-28 18:39:59,969 WARN  
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@1a7d7a7e]
 namenode.NameNodeResourceChecker$CheckedVolume(89): Space available on volume 
'/dev/sdc1' is 0, which is below the configured reserved amount 104857600
2018-02-28 18:39:59,970 WARN  
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@1a7d7a7e]
 namenode.FSNamesystem$NameNodeResourceMonitor(3847): NameNode low on available 
disk space. Entering safe mode.
{noformat}
Check {{/dev/sdc1}} volume.

> IllegalStateException: Unable to finalize edits file
> 
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402): 
> Error: finalize log segment 1, 658 failed for (journal 
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
>  stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file 
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
>   at 
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
>   at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
>   at 
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file

2018-02-28 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380829#comment-16380829
 ] 

Ted Yu commented on HDFS-13206:
---

I don't think that was the case.

I used {{df}} command on the Linux where the test was run.
No partition was nearly half used.

> IllegalStateException: Unable to finalize edits file
> 
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402): 
> Error: finalize log segment 1, 658 failed for (journal 
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
>  stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file 
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
>   at 
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
>   at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
>   at 
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file

2018-02-28 Thread Kihwal Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380822#comment-16380822
 ] 

Kihwal Lee commented on HDFS-13206:
---

It looks like the NN went into safe mode because the disk ran out of space.

> IllegalStateException: Unable to finalize edits file
> 
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402): 
> Error: finalize log segment 1, 658 failed for (journal 
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
>  stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file 
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
>   at 
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
>   at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
>   at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
>   at 
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org