[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096254#comment-13096254 ] Suresh Srinivas commented on HDFS-988: -- +1 for the 20-security patch. saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.20.205.0, 0.22.0 Attachments: 988-fixups.txt, HDFS-988.20-security.patch, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096261#comment-13096261 ] Suresh Srinivas commented on HDFS-988: -- I committed the patch to 0.20-security branch. saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.20.205.0, 0.22.0 Attachments: 988-fixups.txt, HDFS-988.20-security.patch, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096262#comment-13096262 ] Suresh Srinivas commented on HDFS-988: -- +1 for the patch. saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.20.205.0, 0.22.0 Attachments: 988-fixups.txt, HDFS-988.20-security.patch, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080642#comment-13080642 ] Tsz Wo (Nicholas), SZE commented on HDFS-988: - Hi Eli and Todd, it seems that this issue has introduced a deadlock; see HDFS-2229. saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.22.0 Attachments: 988-fixups.txt, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050133#comment-13050133 ] Hudson commented on HDFS-988: - Integrated in Hadoop-Hdfs-trunk-Commit #746 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/]) saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.22.0 Attachments: 988-fixups.txt, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13048278#comment-13048278 ] Hadoop QA commented on HDFS-988: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482272/hdfs-988-7.patch against trunk revision 1134869. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/775//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/775//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/775//console This message is automatically generated. saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.22.0 Attachments: 988-fixups.txt, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13048301#comment-13048301 ] Hudson commented on HDFS-988: - Integrated in Hadoop-Hdfs-22-branch #66 (See [https://builds.apache.org/job/Hadoop-Hdfs-22-branch/66/]) HDFS-988. saveNamespace race can corrupt the edits log. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1134860 Files : * /hadoop/hdfs/branches/branch-0.22/src/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/hdfs/branches/branch-0.22/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestSafeMode.java * /hadoop/hdfs/branches/branch-0.22/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/hdfs/branches/branch-0.22/src/test/hdfs/org/apache/hadoop/hdfs/TestSafeMode.java * /hadoop/hdfs/branches/branch-0.22/CHANGES.txt * /hadoop/hdfs/branches/branch-0.22/src/test/hdfs/org/apache/hadoop/hdfs/DFSTestUtil.java saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.22.0 Attachments: 988-fixups.txt, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-988) saveNamespace race can corrupt the edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13048406#comment-13048406 ] Eli Collins commented on HDFS-988: -- Marking issue as fixed, we now have patches checked into branch-20-append, branch-22 and trunk. saveNamespace race can corrupt the edits log Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Eli Collins Priority: Blocker Fix For: 0.20-append, 0.22.0 Attachments: 988-fixups.txt, HDFS-988_fix_synchs.patch, hdfs-988-2.patch, hdfs-988-3.patch, hdfs-988-4.patch, hdfs-988-5.patch, hdfs-988-6.patch, hdfs-988-7.patch, hdfs-988-b22-1.patch, hdfs-988.txt, saveNamespace.txt, saveNamespace_20-append.patch The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
