[jira] Updated: (HDFS-1105) Balancer improvement
[ https://issues.apache.org/jira/browse/HDFS-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1105: Component/s: balancer Hadoop Flags: [Reviewed] > Balancer improvement > > > Key: HDFS-1105 > URL: https://issues.apache.org/jira/browse/HDFS-1105 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov > Fix For: 0.23.0 > > Attachments: HDFS-1105.2.patch, HDFS-1105.3.patch, HDFS-1105.4.patch, > HDFS-1105.patch > > > We were seeing some weird issues with the balancer in our cluster: > 1) it can get stuck during an iteration and only restarting it helps > 2) the iterations are highly inefficient. With 20 minutes iteration it moves > 7K blocks a minute for the first 6 minutes and hundreds of blocks in the next > 14 minutes > 3) it can hit namenode and the network pretty hard > A few improvements we came up with as a result: > Making balancer more deterministic in terms of running time of iteration, > improving the efficiency and making the load configurable: > Make many of the constants configurable command line parameters: Iteration > length, number of blocks to move in parallel to a given node and in cluster > overall. > Terminate transfers that are still in progress after iteration is over. > Previously iteration time was the time window in which the balancer was > scheduling the moves and then it would wait for the moves to finish > indefinitely. Each scheduling task can run up to iteration time or even > longer. This means if you have too many of them and they are long your actual > iterations are longer than 20 minutes. Now each scheduling task has a time of > the start of iteration and it should schedule the moves only if it did not > run out of time. So the tasks that have started after the iteration is over > will not schedule any moves. > The number of move threads and dispatch threads is configurable so that > depending on the load of the cluster you can run it slower. > I will attach a patch, please let me know what you think and what can be done > better. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1105) Balancer improvement
[ https://issues.apache.org/jira/browse/HDFS-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1105: Status: Patch Available (was: Open) > Balancer improvement > > > Key: HDFS-1105 > URL: https://issues.apache.org/jira/browse/HDFS-1105 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov > Fix For: 0.23.0 > > Attachments: HDFS-1105.2.patch, HDFS-1105.3.patch, HDFS-1105.4.patch, > HDFS-1105.patch > > > We were seeing some weird issues with the balancer in our cluster: > 1) it can get stuck during an iteration and only restarting it helps > 2) the iterations are highly inefficient. With 20 minutes iteration it moves > 7K blocks a minute for the first 6 minutes and hundreds of blocks in the next > 14 minutes > 3) it can hit namenode and the network pretty hard > A few improvements we came up with as a result: > Making balancer more deterministic in terms of running time of iteration, > improving the efficiency and making the load configurable: > Make many of the constants configurable command line parameters: Iteration > length, number of blocks to move in parallel to a given node and in cluster > overall. > Terminate transfers that are still in progress after iteration is over. > Previously iteration time was the time window in which the balancer was > scheduling the moves and then it would wait for the moves to finish > indefinitely. Each scheduling task can run up to iteration time or even > longer. This means if you have too many of them and they are long your actual > iterations are longer than 20 minutes. Now each scheduling task has a time of > the start of iteration and it should schedule the moves only if it did not > run out of time. So the tasks that have started after the iteration is over > will not schedule any moves. > The number of move threads and dispatch threads is configurable so that > depending on the load of the cluster you can run it slower. > I will attach a patch, please let me know what you think and what can be done > better. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1105) Balancer improvement
[ https://issues.apache.org/jira/browse/HDFS-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1105: Fix Version/s: 0.23.0 Status: Open (was: Patch Available) > Balancer improvement > > > Key: HDFS-1105 > URL: https://issues.apache.org/jira/browse/HDFS-1105 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov > Fix For: 0.23.0 > > Attachments: HDFS-1105.2.patch, HDFS-1105.3.patch, HDFS-1105.4.patch, > HDFS-1105.patch > > > We were seeing some weird issues with the balancer in our cluster: > 1) it can get stuck during an iteration and only restarting it helps > 2) the iterations are highly inefficient. With 20 minutes iteration it moves > 7K blocks a minute for the first 6 minutes and hundreds of blocks in the next > 14 minutes > 3) it can hit namenode and the network pretty hard > A few improvements we came up with as a result: > Making balancer more deterministic in terms of running time of iteration, > improving the efficiency and making the load configurable: > Make many of the constants configurable command line parameters: Iteration > length, number of blocks to move in parallel to a given node and in cluster > overall. > Terminate transfers that are still in progress after iteration is over. > Previously iteration time was the time window in which the balancer was > scheduling the moves and then it would wait for the moves to finish > indefinitely. Each scheduling task can run up to iteration time or even > longer. This means if you have too many of them and they are long your actual > iterations are longer than 20 minutes. Now each scheduling task has a time of > the start of iteration and it should schedule the moves only if it did not > run out of time. So the tasks that have started after the iteration is over > will not schedule any moves. > The number of move threads and dispatch threads is configurable so that > depending on the load of the cluster you can run it slower. > I will attach a patch, please let me know what you think and what can be done > better. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: trunkLocalNameImage1.patch > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.22.0 > > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: (was: trunkLocalNameImage1.patch) > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.22.0 > > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: trunkLocalNameImage1.patch trunkLocalnameImage1.patch fixes a few bugs in the previous patch. > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.22.0 > > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1516) mvn-install is broken after 0.22 branch creation
[ https://issues.apache.org/jira/browse/HDFS-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik reassigned HDFS-1516: Assignee: Konstantin Boudnik > mvn-install is broken after 0.22 branch creation > > > Key: HDFS-1516 > URL: https://issues.apache.org/jira/browse/HDFS-1516 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.23.0 > > Attachments: HDFS-1516.patch, HDFS-1516.patch > > > Version HAS to be bumped for system testing framework artifacts (as mentioned > in the build.xml file) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1486) Generalize CLITest structure and interfaces to faciliate upstream adoption (e.g. for web testing)
[ https://issues.apache.org/jira/browse/HDFS-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1486: - Affects Version/s: (was: 0.22.0) 0.23.0 > Generalize CLITest structure and interfaces to faciliate upstream adoption > (e.g. for web testing) > - > > Key: HDFS-1486 > URL: https://issues.apache.org/jira/browse/HDFS-1486 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-1486.patch, HDFS-1486.patch, HDFS-1486.patch, > HDFS-1486.patch > > > HDFS part of HADOOP-7014. HDFS side of TestCLI doesn't require any special > changes but needs to be aligned with Common -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1486) Generalize CLITest structure and interfaces to faciliate upstream adoption (e.g. for web testing)
[ https://issues.apache.org/jira/browse/HDFS-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1486: - Attachment: HDFS-1486.patch Changes corresponding to the latest HADOOP-7014. TestStorageRestore is failing because of HDFS-1496 > Generalize CLITest structure and interfaces to faciliate upstream adoption > (e.g. for web testing) > - > > Key: HDFS-1486 > URL: https://issues.apache.org/jira/browse/HDFS-1486 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-1486.patch, HDFS-1486.patch, HDFS-1486.patch, > HDFS-1486.patch > > > HDFS part of HADOOP-7014. HDFS side of TestCLI doesn't require any special > changes but needs to be aligned with Common -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1103) Replica recovery doesn't distinguish between flushed-but-corrupted last chunk and unflushed last chunk
[ https://issues.apache.org/jira/browse/HDFS-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935564#action_12935564 ] Hairong Kuang commented on HDFS-1103: - RBW B and RBW C's GS gets bumped up by pipeline recovery since the client detects that A in pipeline is down. Even if the client fails before pipeline recovery starts, the lease recovery does not takes RWR into consideration when there is RBW or finalized replica available. > Replica recovery doesn't distinguish between flushed-but-corrupted last chunk > and unflushed last chunk > -- > > Key: HDFS-1103 > URL: https://issues.apache.org/jira/browse/HDFS-1103 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Priority: Blocker > Attachments: hdfs-1103-test.txt > > > When the DN creates a replica under recovery, it calls validateIntegrity, > which truncates the last checksum chunk off of a replica if it is found to be > invalid. Then when the block recovery process happens, this shortened block > wins over a longer replica from another node where there was no corruption. > Thus, if just one of the DNs has an invalid last checksum chunk, data that > has been sync()ed to other datanodes can be lost. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1103) Replica recovery doesn't distinguish between flushed-but-corrupted last chunk and unflushed last chunk
[ https://issues.apache.org/jira/browse/HDFS-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935560#action_12935560 ] Thanh Do commented on HDFS-1103: "I do not think that this is the case in 0.21 & the trunk. In our lease recovery algorithm in 0.21, If there are 2 RBWs and 1 RWR, 1 RWR is excluded from the lease recovery. In the scenario that you described, RBW B and RBW C's GS is bumped and the length of recovered two replicas is truncated to MIN( len(B), len(C)). " Hairong, can you explain to me that why RBW B and RBW C's GS are bumped up. Is that because of the lease recovery protocol? But from my understanding, from Todd description, NN lease recovery is trigger after Machine A report... > Replica recovery doesn't distinguish between flushed-but-corrupted last chunk > and unflushed last chunk > -- > > Key: HDFS-1103 > URL: https://issues.apache.org/jira/browse/HDFS-1103 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Priority: Blocker > Attachments: hdfs-1103-test.txt > > > When the DN creates a replica under recovery, it calls validateIntegrity, > which truncates the last checksum chunk off of a replica if it is found to be > invalid. Then when the block recovery process happens, this shortened block > wins over a longer replica from another node where there was no corruption. > Thus, if just one of the DNs has an invalid last checksum chunk, data that > has been sync()ed to other datanodes can be lost. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1458) Improve checkpoint performance by avoiding unnecessary image downloads
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1458: Resolution: Fixed Status: Resolved (was: Patch Available) I've just committed this. > Improve checkpoint performance by avoiding unnecessary image downloads > -- > > Key: HDFS-1458 > URL: https://issues.apache.org/jira/browse/HDFS-1458 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: checkpoint-checkfsimageissame.patch, > trunkNoDownloadImage.patch, trunkNoDownloadImage1.patch, > trunkNoDownloadImage2.patch, trunkNoDownloadImage3.patch > > > If secondary namenode could verify that the image it has on its disk is the > same as the one in the primary NameNode, it could skip downloading the image > from the primary NN, thus completely eliminating the image download overhead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by avoiding unnecessary image downloads
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935556#action_12935556 ] Hairong Kuang commented on HDFS-1458: - I ran the tests on my local machine. Those are the failed test TestBlockRecovery TestBlockTokenwithDFS TestDFSClientRetires TestStorageRestore TestSaveNamespace None of them are related to my patch. > Improve checkpoint performance by avoiding unnecessary image downloads > -- > > Key: HDFS-1458 > URL: https://issues.apache.org/jira/browse/HDFS-1458 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: checkpoint-checkfsimageissame.patch, > trunkNoDownloadImage.patch, trunkNoDownloadImage1.patch, > trunkNoDownloadImage2.patch, trunkNoDownloadImage3.patch > > > If secondary namenode could verify that the image it has on its disk is the > same as the one in the primary NameNode, it could skip downloading the image > from the primary NN, thus completely eliminating the image download overhead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1458) Improve checkpoint performance by avoiding unnecessary image downloads
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1458: Fix Version/s: (was: 0.22.0) 0.23.0 Hadoop Flags: [Reviewed] Status: Patch Available (was: Open) > Improve checkpoint performance by avoiding unnecessary image downloads > -- > > Key: HDFS-1458 > URL: https://issues.apache.org/jira/browse/HDFS-1458 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: checkpoint-checkfsimageissame.patch, > trunkNoDownloadImage.patch, trunkNoDownloadImage1.patch, > trunkNoDownloadImage2.patch, trunkNoDownloadImage3.patch > > > If secondary namenode could verify that the image it has on its disk is the > same as the one in the primary NameNode, it could skip downloading the image > from the primary NN, thus completely eliminating the image download overhead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1503) TestSaveNamespace fails
[
https://issues.apache.org/jira/browse/HDFS-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935546#action_12935546
]
Konstantin Shvachko commented on HDFS-1503:
---
+1. Tested on my box: testCrashWhileSavingSecondImage() fails without the
patch, but passes with it.
> TestSaveNamespace fails
> ---
>
> Key: HDFS-1503
> URL: https://issues.apache.org/jira/browse/HDFS-1503
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Eli Collins
>Assignee: Todd Lipcon
>Priority: Minor
> Attachments: hdfs-1503.txt,
> TEST-org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace.txt
>
>
> Will attach the full log. Here's the relevant snippet:
> {noformat}
> Exception in thread "FSImageSaver for
> /home/eli/src/hdfs4/build/test/data/dfs/na
> me1 of type IMAGE_AND_EDITS" java.lang.RuntimeException: Injected fault:
> saveFSI
> mage second time
>
> at java.lang.Thread.run(Thread.java:619)
> Exception in thread "FSImageSaver for
> /home/eli/src/hdfs4/build/test/data/dfs/na
> me2 of type IMAGE_AND_EDITS" java.lang.StackOverflowError
> at java.util.Arrays.copyOf(Arrays.java:2882)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1503) TestSaveNamespace fails
[
https://issues.apache.org/jira/browse/HDFS-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935516#action_12935516
]
Konstantin Shvachko commented on HDFS-1503:
---
Thanks for clarifications, Todd. I was not sure if the new code broke the test
itself or the feature it is testing. Now I understand it the test problem.
> TestSaveNamespace fails
> ---
>
> Key: HDFS-1503
> URL: https://issues.apache.org/jira/browse/HDFS-1503
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Eli Collins
>Assignee: Todd Lipcon
>Priority: Minor
> Attachments: hdfs-1503.txt,
> TEST-org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace.txt
>
>
> Will attach the full log. Here's the relevant snippet:
> {noformat}
> Exception in thread "FSImageSaver for
> /home/eli/src/hdfs4/build/test/data/dfs/na
> me1 of type IMAGE_AND_EDITS" java.lang.RuntimeException: Injected fault:
> saveFSI
> mage second time
>
> at java.lang.Thread.run(Thread.java:619)
> Exception in thread "FSImageSaver for
> /home/eli/src/hdfs4/build/test/data/dfs/na
> me2 of type IMAGE_AND_EDITS" java.lang.StackOverflowError
> at java.util.Arrays.copyOf(Arrays.java:2882)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1391) Exiting safemode takes a long time when there are lots of blocks in the HDFS
[ https://issues.apache.org/jira/browse/HDFS-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935508#action_12935508 ] Hadoop QA commented on HDFS-1391: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12454967/excessReplicas.1_trunk.txt against trunk revision 1038370. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 101 release audit warnings (more than the trunk's current 98 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestLargeBlock -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/5//testReport/ Release audit warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/5//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/5//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/5//console This message is automatically generated. > Exiting safemode takes a long time when there are lots of blocks in the HDFS > > > Key: HDFS-1391 > URL: https://issues.apache.org/jira/browse/HDFS-1391 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: excessReplicas.1_trunk.txt > > > When the namenode decides to exit safemode, it acquires the FSNamesystem > lock and then iterates over all blocks in the blocksmap to determine if any > block has any excess replicas. This call takes upwards of 5 minutes on a > cluster that has 100 million blocks. This delays namenode restart to a good > extent. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-900) Corrupt replicas are not tracked correctly through block report from DN
[ https://issues.apache.org/jira/browse/HDFS-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935471#action_12935471 ] Todd Lipcon commented on HDFS-900: -- Yes, I believe this reproduces in 0.21 and 0.22 (the to-reproduce.patch was against trunk at the time though likely is out of date now) > Corrupt replicas are not tracked correctly through block report from DN > --- > > Key: HDFS-900 > URL: https://issues.apache.org/jira/browse/HDFS-900 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Priority: Critical > Attachments: log-commented, to-reproduce.patch > > > This one is tough to describe, but essentially the following order of events > is seen to occur: > # A client marks one replica of a block to be corrupt by telling the NN about > it > # Replication is then scheduled to make a new replica of this node > # The replication completes, such that there are now 3 good replicas and 1 > corrupt replica > # The DN holding the corrupt replica sends a block report. Rather than > telling this DN to delete the node, the NN instead marks this as a new *good* > replica of the block, and schedules deletion on one of the good replicas. > I don't know if this is a dataloss bug in the case of 1 corrupt replica with > dfs.replication=2, but it seems feasible. I will attach a debug log with some > commentary marked by '>', plus a unit test patch which I can get > to reproduce this behavior reliably. (it's not a proper unit test, just some > edits to an existing one to show it) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-900) Corrupt replicas are not tracked correctly through block report from DN
[ https://issues.apache.org/jira/browse/HDFS-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935390#action_12935390 ] Thanh Do commented on HDFS-900: --- Todd, do you see this on 0.21.0? Is this a bug of handling corrupt replicas at NN? > Corrupt replicas are not tracked correctly through block report from DN > --- > > Key: HDFS-900 > URL: https://issues.apache.org/jira/browse/HDFS-900 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Priority: Critical > Attachments: log-commented, to-reproduce.patch > > > This one is tough to describe, but essentially the following order of events > is seen to occur: > # A client marks one replica of a block to be corrupt by telling the NN about > it > # Replication is then scheduled to make a new replica of this node > # The replication completes, such that there are now 3 good replicas and 1 > corrupt replica > # The DN holding the corrupt replica sends a block report. Rather than > telling this DN to delete the node, the NN instead marks this as a new *good* > replica of the block, and schedules deletion on one of the good replicas. > I don't know if this is a dataloss bug in the case of 1 corrupt replica with > dfs.replication=2, but it seems feasible. I will attach a debug log with some > commentary marked by '>', plus a unit test patch which I can get > to reproduce this behavior reliably. (it's not a proper unit test, just some > edits to an existing one to show it) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by avoiding unnecessary image downloads
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935242#action_12935242 ] Hairong Kuang commented on HDFS-1458: - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to i [exec] nclude 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] > Improve checkpoint performance by avoiding unnecessary image downloads > -- > > Key: HDFS-1458 > URL: https://issues.apache.org/jira/browse/HDFS-1458 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.22.0 > > Attachments: checkpoint-checkfsimageissame.patch, > trunkNoDownloadImage.patch, trunkNoDownloadImage1.patch, > trunkNoDownloadImage2.patch, trunkNoDownloadImage3.patch > > > If secondary namenode could verify that the image it has on its disk is the > same as the one in the primary NameNode, it could skip downloading the image > from the primary NN, thus completely eliminating the image download overhead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
