[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386029#comment-14386029
]
Joe Pallas commented on HDFS-7722:
--
Sure, [~eddyxu]. HDFS-5194 is about improving support for alternative storage
implementations. Assuming that volumes always correspond to directories in the
local file system limits the ability to implement other storage architectures,
such as a directly attached object store or perhaps a block-level device with a
lightweight user-level file system layer optimized for storing block replicas.
The {{FsDatasetSpi}} interface tries to abstract out the essentials of storing
replicas, and {{FsVolumeSpi}} is an abstract unit of storage used by the
dataset to represent some subset of all the available storage (typically a
single drive in the default implementation). Advertising that volumes are
directories doesn't just limit alternative implementations, it also makes it
harder to evolve the default implementation, because the scope of changes is
harder to determine once implementation details leak through the abstraction.
That's my perspective. Maintaining these abstractions takes some some work,
but it has benefits for readability/maintainability of the default
implementation as well as for alternative implementations.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384312#comment-14384312
]
Joe Pallas commented on HDFS-7722:
--
Eliminating that pervasive assumption probably falls within the scope of
HDFS-5194.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384627#comment-14384627
]
Lei (Eddy) Xu commented on HDFS-7722:
-
Hi, [~jpallas] Could you elaborate more ?
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368912#comment-14368912
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #137 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/137/])
Fix CHANGES.txt for HDFS-7722. (arp: rev
02a67aad65e790cddba6f49658664f459e1de788)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368934#comment-14368934
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #871 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/871/])
Fix CHANGES.txt for HDFS-7722. (arp: rev
02a67aad65e790cddba6f49658664f459e1de788)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369424#comment-14369424
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #128 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/128/])
Fix CHANGES.txt for HDFS-7722. (arp: rev
02a67aad65e790cddba6f49658664f459e1de788)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369403#comment-14369403
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2069 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2069/])
Fix CHANGES.txt for HDFS-7722. (arp: rev
02a67aad65e790cddba6f49658664f459e1de788)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369491#comment-14369491
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #137 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/137/])
Fix CHANGES.txt for HDFS-7722. (arp: rev
02a67aad65e790cddba6f49658664f459e1de788)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369529#comment-14369529
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2087 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2087/])
Fix CHANGES.txt for HDFS-7722. (arp: rev
02a67aad65e790cddba6f49658664f459e1de788)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367781#comment-14367781
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7362 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7362/])
Fix CHANGES.txt for HDFS-7722. (arp: rev
02a67aad65e790cddba6f49658664f459e1de788)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360245#comment-14360245
]
Hudson commented on HDFS-7722:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #865 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/865/])
HDFS-7722. DataNode#checkDiskError should also remove Storage when error is
found. (Lei Xu via Colin P. McCabe) (cmccabe: rev
b49c3a1813aa8c5b05fe6c02a653286c573137ca)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360432#comment-14360432
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #122 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/122/])
HDFS-7722. DataNode#checkDiskError should also remove Storage when error is
found. (Lei Xu via Colin P. McCabe) (cmccabe: rev
b49c3a1813aa8c5b05fe6c02a653286c573137ca)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360425#comment-14360425
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2063 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2063/])
HDFS-7722. DataNode#checkDiskError should also remove Storage when error is
found. (Lei Xu via Colin P. McCabe) (cmccabe: rev
b49c3a1813aa8c5b05fe6c02a653286c573137ca)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360232#comment-14360232
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #131 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/131/])
HDFS-7722. DataNode#checkDiskError should also remove Storage when error is
found. (Lei Xu via Colin P. McCabe) (cmccabe: rev
b49c3a1813aa8c5b05fe6c02a653286c573137ca)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360493#comment-14360493
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #131 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/131/])
HDFS-7722. DataNode#checkDiskError should also remove Storage when error is
found. (Lei Xu via Colin P. McCabe) (cmccabe: rev
b49c3a1813aa8c5b05fe6c02a653286c573137ca)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360358#comment-14360358
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2081 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2081/])
HDFS-7722. DataNode#checkDiskError should also remove Storage when error is
found. (Lei Xu via Colin P. McCabe) (cmccabe: rev
b49c3a1813aa8c5b05fe6c02a653286c573137ca)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359258#comment-14359258
]
Hudson commented on HDFS-7722:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7311 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7311/])
HDFS-7722. DataNode#checkDiskError should also remove Storage when error is
found. (Lei Xu via Colin P. McCabe) (cmccabe: rev
b49c3a1813aa8c5b05fe6c02a653286c573137ca)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Fix For: 2.7.0
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359180#comment-14359180
]
Colin Patrick McCabe commented on HDFS-7722:
+1. Thanks, Eddy.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357758#comment-14357758
]
Hadoop QA commented on HDFS-7722:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch
against trunk revision 7a346bc.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9844//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1435#comment-1435
]
Hadoop QA commented on HDFS-7722:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704044/HDFS-7722.004.patch
against trunk revision 7a346bc.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9845//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358032#comment-14358032
]
Hadoop QA commented on HDFS-7722:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704044/HDFS-7722.004.patch
against trunk revision 7a346bc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9846//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9846//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch, HDFS-7722.004.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355401#comment-14355401
]
Hadoop QA commented on HDFS-7722:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch
against trunk revision a380643.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9821//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353649#comment-14353649
]
Hadoop QA commented on HDFS-7722:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch
against trunk revision d6e05c5.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9800//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353698#comment-14353698
]
Hadoop QA commented on HDFS-7722:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12703499/HDFS-7722.003.patch
against trunk revision d6e05c5.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9801//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch, HDFS-7722.003.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353605#comment-14353605
]
Chris Nauroth commented on HDFS-7722:
-
Eddy, it looks good. I have just one minor nit. In
{{TestDataNodeVolumeFailureReporting}}, please remove the commented out lines
of test code for the final version of the patch. Also, we can no longer remove
the import of {{org.apache.hadoop.hdfs.protocol.Block}}, because another patch
started using it recently.
bq. I suggest to have a following JIRA...
Please feel free to do that if you wish, but I actually don't think it's
necessary. In general, I don't expect permanent removal of a volume to be the
typical recovery procedure. Instead, I expect a more typical recovery
procedure to be like you described: replace the faulty disk. Since that works
fine, I think it would be overkill at this point to put in dedicated
functionality to cover something that is probably a very rare edge case in
practical deployments.
Thanks for working on this!
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351152#comment-14351152
]
Colin Patrick McCabe commented on HDFS-7722:
Eddy and I had an offline discussion about the use of {{SetFile}} here. It
seems that there is a pervasive assumption elsewhere in the code that
FsVolumeSpi instances are directories. For example, in these interface methods:
{code}
/** @return the base path to the volume */
public String getBasePath();
/** @return the path to the volume */
public String getPath(String bpid) throws IOException;
/** @return the directory for the finalized blocks in the block pool. */
public File getFinalizedDir(String bpid) throws IOException;
{code}
So I think using {{SetFile}} is OK here for now, since it fits in with the
rest of the code. We will probably have to revisit this later, but it seems
outside the scope of this jira.
One thing I really like about this patch is the fact we no longer hold the
{{FsDatasetImpl}} mutex while scanning every volume. This alone is a very
important improvement.
I think it makes sense to leave the failure information around when removing
volumes due to the disk checker.
{code}
685 LOG.info(Deactivating volumes: +
686 Joiner.on(,).join(absoluteVolumePaths));
{code}
We should print out the value of {{clearFailure}} here.
+1 once that's addressed.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351167#comment-14351167
]
Chris Nauroth commented on HDFS-7722:
-
[~eddyxu], sorry I haven't had a chance to dig into this patch yet. If I
understand correctly, you're saying that removing a path from configuration and
running reconfig will not clear volume failure information, but keeping the
path in configuration, fixing the disk at that mount point and running reconfig
will clear it. Do I have it right? I would like us to have some means to take
corrective action and clear the volume failure information online. As long
as that's still possible in some way, then it's probably sticking to the spirit
of the code I wrote earlier.
Would you mind holding off the commit until early next week so I can take a
closer look? Thanks!
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351218#comment-14351218
]
Lei (Eddy) Xu commented on HDFS-7722:
-
[~cmccabe] Thanks for the review. I will make a patch to address your comments.
[~cnauroth] Yes, you are right on this one. Sure, I believe we can hold
committing this. A review from you early next week would be much appreciated!
To add some background, the rationale of this patch is providing user a
convenient way to fix bad disks without touching configuration files, in the
meantime, also preserving disk failure information for reporting purpose.
bq. I would like us to have some means to take corrective action and clear the
volume failure information online.
For this concern, I suggest to have a following JIRA to let
{{DataNode#parseChangedVolume}} to detect volumes that
* is not in {{FsVolumeList}}
* is not in {{DFS_DATANODE_DATA_DIR_KEYS}}
* and is in {{volumeFailureInfos}}
as {{DataNode#ChangedVolumes#deactiveLocations}}. So that the following logic
can clear this failure info if the user _intents_ to do so.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346356#comment-14346356
]
Hadoop QA commented on HDFS-7722:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702322/HDFS-7722.002.patch
against trunk revision 5af693f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9716//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9716//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch,
HDFS-7722.002.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335752#comment-14335752
]
Lei (Eddy) Xu commented on HDFS-7722:
-
{{TestDataNodeVolumeFailureReporting}} is relevant. I will work on fixing it.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335745#comment-14335745
]
Hadoop QA commented on HDFS-7722:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12700593/HDFS-7722.001.patch
against trunk revision 9a37247.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9660//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9660//console
This message is automatically generated.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315064#comment-14315064
]
Colin Patrick McCabe commented on HDFS-7722:
FsDatasetSpi#checkDataDir is really checking volumes, and returning a list of
failed volumes. But it returns a {{SetFile}} instead, and then we look up
the {{FsVolume}} objects again based on that. Why? Why not simply pass back a
list of {{FsVolumeRef}} objects that we could use directly?
Similarly, {{FsVolumeList#checkDirs}} should be returning a list of
{{FsVolumeRef}}, not {{FsVolumeImpl}}.
We should be rate-limiting {{FsVolumeList#checkDirs}} as well. Please remember
that this scans *all* files on a volume, which is an expensive operation. If
we hit a bunch of I/O operations, we could end up calling checkDirs over and
over within the same minute or two. We should rate-limit this so that we
ignore calls to checkDirs on a volume that happen more than once every 10
minutes or something.
{code}
+// Disable the volume from the service
+ asyncDiskService.removeVolume(fv.getCurrentDir());
{code}
Why are we duplicating all the logic of {{FsDatasetImpl#removeVolumes}} here?
If we want {{checkDirs}} to remove a volume, it should just call that function.
That would also avoid the need to have {{DataNode#checkDiskError}} do anything
but simply call {{FsDatasetSpi#checkDirs}}.
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-7722.000.patch
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306893#comment-14306893
]
Srikanth Upputuri commented on HDFS-7722:
-
Reassigned to you, [~eddyxu].
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305930#comment-14305930
]
Lei (Eddy) Xu commented on HDFS-7722:
-
[~usrikanth] Would you mind to let me take on this one? I already have a patch
available for it.
Thanks a lot!
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Srikanth Upputuri
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
[
https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301785#comment-14301785
]
Colin Patrick McCabe commented on HDFS-7722:
good idea
DataNode#checkDiskError should also remove Storage when error is found.
---
Key: HDFS-7722
URL: https://issues.apache.org/jira/browse/HDFS-7722
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
When {{DataNode#checkDiskError}} found disk errors, it removes all block
metadatas from {{FsDatasetImpl}}. However, it does not removed the
corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.
The result is that, we could not directly run {{reconfig}} to hot swap the
failure disks without changing the configure file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
