[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386029#comment-14386029 ] Joe Pallas commented on HDFS-7722: -- Sure, [~eddyxu]. HDFS-5194 is about improving support for alternative storage implementations. Assuming that volumes always correspond to directories in the local file system limits the ability to implement other storage architectures, such as a directly attached object store or perhaps a block-level device with a lightweight user-level "file system" layer optimized for storing block replicas. The {{FsDatasetSpi}} interface tries to abstract out the essentials of storing replicas, and {{FsVolumeSpi}} is an abstract unit of storage used by the dataset to represent some subset of all the available storage (typically a single drive in the default implementation). Advertising that volumes are directories doesn't just limit alternative implementations, it also makes it harder to evolve the default implementation, because the scope of changes is harder to determine once implementation details leak through the abstraction. That's my perspective. Maintaining these abstractions takes some some work, but it has benefits for readability/maintainability of the default implementation as well as for alternative implementations. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384627#comment-14384627 ] Lei (Eddy) Xu commented on HDFS-7722: - Hi, [~jpallas] Could you elaborate more ? > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384312#comment-14384312 ] Joe Pallas commented on HDFS-7722: -- Eliminating that "pervasive assumption" probably falls within the scope of HDFS-5194. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369529#comment-14369529 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2087 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2087/]) Fix CHANGES.txt for HDFS-7722. (arp: rev 02a67aad65e790cddba6f49658664f459e1de788) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369491#comment-14369491 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #137 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/137/]) Fix CHANGES.txt for HDFS-7722. (arp: rev 02a67aad65e790cddba6f49658664f459e1de788) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369424#comment-14369424 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #128 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/128/]) Fix CHANGES.txt for HDFS-7722. (arp: rev 02a67aad65e790cddba6f49658664f459e1de788) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369403#comment-14369403 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2069 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2069/]) Fix CHANGES.txt for HDFS-7722. (arp: rev 02a67aad65e790cddba6f49658664f459e1de788) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368934#comment-14368934 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Yarn-trunk #871 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/871/]) Fix CHANGES.txt for HDFS-7722. (arp: rev 02a67aad65e790cddba6f49658664f459e1de788) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368912#comment-14368912 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #137 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/137/]) Fix CHANGES.txt for HDFS-7722. (arp: rev 02a67aad65e790cddba6f49658664f459e1de788) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367781#comment-14367781 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-trunk-Commit #7362 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7362/]) Fix CHANGES.txt for HDFS-7722. (arp: rev 02a67aad65e790cddba6f49658664f459e1de788) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360493#comment-14360493 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #131 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/131/]) HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) (cmccabe: rev b49c3a1813aa8c5b05fe6c02a653286c573137ca) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360425#comment-14360425 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2063 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2063/]) HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) (cmccabe: rev b49c3a1813aa8c5b05fe6c02a653286c573137ca) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360432#comment-14360432 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/122/]) HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) (cmccabe: rev b49c3a1813aa8c5b05fe6c02a653286c573137ca) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360358#comment-14360358 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2081 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2081/]) HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) (cmccabe: rev b49c3a1813aa8c5b05fe6c02a653286c573137ca) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360245#comment-14360245 ] Hudson commented on HDFS-7722: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #865 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/865/]) HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) (cmccabe: rev b49c3a1813aa8c5b05fe6c02a653286c573137ca) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360232#comment-14360232 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #131 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/131/]) HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) (cmccabe: rev b49c3a1813aa8c5b05fe6c02a653286c573137ca) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359258#comment-14359258 ] Hudson commented on HDFS-7722: -- FAILURE: Integrated in Hadoop-trunk-Commit #7311 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7311/]) HDFS-7722. DataNode#checkDiskError should also remove Storage when error is found. (Lei Xu via Colin P. McCabe) (cmccabe: rev b49c3a1813aa8c5b05fe6c02a653286c573137ca) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Fix For: 2.7.0 > > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359180#comment-14359180 ] Colin Patrick McCabe commented on HDFS-7722: +1. Thanks, Eddy. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358032#comment-14358032 ] Hadoop QA commented on HDFS-7722: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704044/HDFS-7722.004.patch against trunk revision 7a346bc. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9846//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9846//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1435#comment-1435 ] Hadoop QA commented on HDFS-7722: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704044/HDFS-7722.004.patch against trunk revision 7a346bc. {color:red}-1 patch{color}. Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9845//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357758#comment-14357758 ] Hadoop QA commented on HDFS-7722: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch against trunk revision 7a346bc. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9844//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355401#comment-14355401 ] Hadoop QA commented on HDFS-7722: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch against trunk revision a380643. {color:red}-1 patch{color}. Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9821//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353698#comment-14353698 ] Hadoop QA commented on HDFS-7722: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch against trunk revision d6e05c5. {color:red}-1 patch{color}. Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9801//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353649#comment-14353649 ] Hadoop QA commented on HDFS-7722: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch against trunk revision d6e05c5. {color:red}-1 patch{color}. Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9800//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch, HDFS-7722.003.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353605#comment-14353605 ] Chris Nauroth commented on HDFS-7722: - Eddy, it looks good. I have just one minor nit. In {{TestDataNodeVolumeFailureReporting}}, please remove the commented out lines of test code for the final version of the patch. Also, we can no longer remove the import of {{org.apache.hadoop.hdfs.protocol.Block}}, because another patch started using it recently. bq. I suggest to have a following JIRA... Please feel free to do that if you wish, but I actually don't think it's necessary. In general, I don't expect permanent removal of a volume to be the typical recovery procedure. Instead, I expect a more typical recovery procedure to be like you described: replace the faulty disk. Since that works fine, I think it would be overkill at this point to put in dedicated functionality to cover something that is probably a very rare edge case in practical deployments. Thanks for working on this! > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351218#comment-14351218 ] Lei (Eddy) Xu commented on HDFS-7722: - [~cmccabe] Thanks for the review. I will make a patch to address your comments. [~cnauroth] Yes, you are right on this one. Sure, I believe we can hold committing this. A review from you early next week would be much appreciated! To add some background, the rationale of this patch is providing user a convenient way to fix bad disks without touching configuration files, in the meantime, also preserving disk failure information for reporting purpose. bq. I would like us to have some means to take corrective action and clear the volume failure information "online". For this concern, I suggest to have a following JIRA to let {{DataNode#parseChangedVolume}} to detect volumes that * is not in {{FsVolumeList}} * is not in {{DFS_DATANODE_DATA_DIR_KEYS}} * and is in {{volumeFailureInfos}} as {{DataNode#ChangedVolumes#deactiveLocations}}. So that the following logic can clear this failure info if the user _intents_ to do so. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351167#comment-14351167 ] Chris Nauroth commented on HDFS-7722: - [~eddyxu], sorry I haven't had a chance to dig into this patch yet. If I understand correctly, you're saying that removing a path from configuration and running reconfig will not clear volume failure information, but keeping the path in configuration, fixing the disk at that mount point and running reconfig will clear it. Do I have it right? I would like us to have some means to take corrective action and clear the volume failure information "online". As long as that's still possible in some way, then it's probably sticking to the spirit of the code I wrote earlier. Would you mind holding off the commit until early next week so I can take a closer look? Thanks! > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351152#comment-14351152 ] Colin Patrick McCabe commented on HDFS-7722: Eddy and I had an offline discussion about the use of {{Set}} here. It seems that there is a pervasive assumption elsewhere in the code that FsVolumeSpi instances are directories. For example, in these interface methods: {code} /** @return the base path to the volume */ public String getBasePath(); /** @return the path to the volume */ public String getPath(String bpid) throws IOException; /** @return the directory for the finalized blocks in the block pool. */ public File getFinalizedDir(String bpid) throws IOException; {code} So I think using {{Set}} is OK here for now, since it fits in with the rest of the code. We will probably have to revisit this later, but it seems outside the scope of this jira. One thing I really like about this patch is the fact we no longer hold the {{FsDatasetImpl}} mutex while scanning every volume. This alone is a very important improvement. I think it makes sense to leave the failure information around when removing volumes due to the disk checker. {code} 685 LOG.info("Deactivating volumes: " + 686 Joiner.on(",").join(absoluteVolumePaths)); {code} We should print out the value of {{clearFailure}} here. +1 once that's addressed. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346356#comment-14346356 ] Hadoop QA commented on HDFS-7722: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702322/HDFS-7722.002.patch against trunk revision 5af693f. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9716//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9716//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, > HDFS-7722.002.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335752#comment-14335752 ] Lei (Eddy) Xu commented on HDFS-7722: - {{TestDataNodeVolumeFailureReporting}} is relevant. I will work on fixing it. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335745#comment-14335745 ] Hadoop QA commented on HDFS-7722: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12700593/HDFS-7722.001.patch against trunk revision 9a37247. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9660//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9660//console This message is automatically generated. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315064#comment-14315064 ] Colin Patrick McCabe commented on HDFS-7722: FsDatasetSpi#checkDataDir is really checking volumes, and returning a list of failed volumes. But it returns a {{Set}} instead, and then we look up the {{FsVolume}} objects again based on that. Why? Why not simply pass back a list of {{FsVolumeRef}} objects that we could use directly? Similarly, {{FsVolumeList#checkDirs}} should be returning a list of {{FsVolumeRef}}, not {{FsVolumeImpl}}. We should be rate-limiting {{FsVolumeList#checkDirs}} as well. Please remember that this scans *all* files on a volume, which is an expensive operation. If we hit a bunch of I/O operations, we could end up calling checkDirs over and over within the same minute or two. We should rate-limit this so that we ignore calls to checkDirs on a volume that happen more than once every 10 minutes or something. {code} +// Disable the volume from the service + asyncDiskService.removeVolume(fv.getCurrentDir()); {code} Why are we duplicating all the logic of {{FsDatasetImpl#removeVolumes}} here? If we want {{checkDirs}} to remove a volume, it should just call that function. That would also avoid the need to have {{DataNode#checkDiskError}} do anything but simply call {{FsDatasetSpi#checkDirs}}. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7722.000.patch > > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306893#comment-14306893 ] Srikanth Upputuri commented on HDFS-7722: - Reassigned to you, [~eddyxu]. > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305930#comment-14305930 ] Lei (Eddy) Xu commented on HDFS-7722: - [~usrikanth] Would you mind to let me take on this one? I already have a patch available for it. Thanks a lot! > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu >Assignee: Srikanth Upputuri > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301785#comment-14301785 ] Colin Patrick McCabe commented on HDFS-7722: good idea > DataNode#checkDiskError should also remove Storage when error is found. > --- > > Key: HDFS-7722 > URL: https://issues.apache.org/jira/browse/HDFS-7722 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Lei (Eddy) Xu > > When {{DataNode#checkDiskError}} found disk errors, it removes all block > metadatas from {{FsDatasetImpl}}. However, it does not removed the > corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}. > The result is that, we could not directly run {{reconfig}} to hot swap the > failure disks without changing the configure file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)