[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[ https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140377#comment-14140377 ] Hudson commented on HDFS-6686: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #685 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/685/]) HDFS-6686. Change BlockPlacementPolicy to use fallback when some storage types are unavailable. (szetszwo: rev ac5e8aed7ca1e9493f96f8795d0caafd5282b9a7) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Archival Storage: Use fallback storage types Key: HDFS-6686 URL: https://issues.apache.org/jira/browse/HDFS-6686 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Fix For: Archival Storage (HDFS-6584) Attachments: h6686_20140721.patch, h6686_20140721c.patch, h6686_20140723.patch HDFS-6671 changes replication monitor to use block storage policy for replication. It should also use the fallback storage types when a particular type of storage is full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[ https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140528#comment-14140528 ] Hudson commented on HDFS-6686: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1901 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1901/]) HDFS-6686. Change BlockPlacementPolicy to use fallback when some storage types are unavailable. (szetszwo: rev ac5e8aed7ca1e9493f96f8795d0caafd5282b9a7) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Archival Storage: Use fallback storage types Key: HDFS-6686 URL: https://issues.apache.org/jira/browse/HDFS-6686 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Fix For: Archival Storage (HDFS-6584) Attachments: h6686_20140721.patch, h6686_20140721c.patch, h6686_20140723.patch HDFS-6671 changes replication monitor to use block storage policy for replication. It should also use the fallback storage types when a particular type of storage is full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[ https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140583#comment-14140583 ] Hudson commented on HDFS-6686: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1876 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/]) HDFS-6686. Change BlockPlacementPolicy to use fallback when some storage types are unavailable. (szetszwo: rev ac5e8aed7ca1e9493f96f8795d0caafd5282b9a7) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java Archival Storage: Use fallback storage types Key: HDFS-6686 URL: https://issues.apache.org/jira/browse/HDFS-6686 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Fix For: Archival Storage (HDFS-6584) Attachments: h6686_20140721.patch, h6686_20140721c.patch, h6686_20140723.patch HDFS-6671 changes replication monitor to use block storage policy for replication. It should also use the fallback storage types when a particular type of storage is full. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[ https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071430#comment-14071430 ] Tsz Wo Nicholas Sze commented on HDFS-6686: --- Vinay, # Good catch. We should pass excludedNodes. # For getAdditionalDatanode(), since the write is already started, there are already some data in the block. So using replication fallback is correct. Creation fallback usually is a subset of replication fallback since if there are not enough storages available, we may fails creation. However, if we fail replication, it may result in data loss. Arpit, # Good catch. We need to pass newBlock as a parameter. # numOfResults indeed is the number of existing replicas (numOfResults is used to determine whether local host/local rack/remote rack should be chosen from). numOfReplicas (call it n) is the number of replicas to be chosen. In our case, we try to select n storage types but we may only able to get m n, where m = storageTypes.size(). So we should update numOfReplicas. Thanks both of you for the careful reviews! Archival Storage: Use fallback storage types Key: HDFS-6686 URL: https://issues.apache.org/jira/browse/HDFS-6686 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: h6686_20140721.patch, h6686_20140721c.patch HDFS-6671 changes replication monitor to use block storage policy for replication. It should also use the fallback storage types when a particular type of storage is full. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[ https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071502#comment-14071502 ] Vinayakumar B commented on HDFS-6686: - +1 for Latest patch, lgtm. Thanks [~szetszwo]. Archival Storage: Use fallback storage types Key: HDFS-6686 URL: https://issues.apache.org/jira/browse/HDFS-6686 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: h6686_20140721.patch, h6686_20140721c.patch, h6686_20140723.patch HDFS-6671 changes replication monitor to use block storage policy for replication. It should also use the fallback storage types when a particular type of storage is full. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[
https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071377#comment-14071377
]
Vinayakumar B commented on HDFS-6686:
-
I have 2 doubts
1. In the below code, for the retry, whether {{oldExcludedNodes}} should be
passed or {{excludedNodes}} should be passed..? since {{avoidStaleNodes}} is
passed as false.?
By default, {{avoidStaleNodes}} is false, hence {{oldExcludedNodes}} will be
null, so I feel, original {{excludedNodes}} will be missed and it can choose
already excluded node again.. no?
{code} if (storageTypes.size() 0) {
// Retry chooseTarget with fallback storage types
unavailableStorages.add(curStorageType);
return chooseTarget(numOfReplicas, writer, oldExcludedNodes, blocksize,
maxNodesPerRack, results, false, storagePolicy,
unavailableStorages);
}{code}
2. For the PipelineRecovery (getAdditionalDatanode()) calls also, it uses
replicationFallback storageType, whether this is Ok?
Archival Storage: Use fallback storage types
Key: HDFS-6686
URL: https://issues.apache.org/jira/browse/HDFS-6686
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Attachments: h6686_20140721.patch, h6686_20140721c.patch
HDFS-6671 changes replication monitor to use block storage policy for
replication. It should also use the fallback storage types when a particular
type of storage is full.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[ https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071394#comment-14071394 ] Arpit Agarwal commented on HDFS-6686: - Hi Nicholas, took a look at the patch. Also have two questions: # The recursive call on line 393 passes the same results list so the callee might choose the replication fallback instead of creation fallback. # Perhaps I misunderstood - on line 318, should we update numOfResults instead of numOfReplicas? Later we check numOfResults before dequeuing entries from storageTypes. Archival Storage: Use fallback storage types Key: HDFS-6686 URL: https://issues.apache.org/jira/browse/HDFS-6686 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: h6686_20140721.patch, h6686_20140721c.patch HDFS-6671 changes replication monitor to use block storage policy for replication. It should also use the fallback storage types when a particular type of storage is full. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6686) Archival Storage: Use fallback storage types
[ https://issues.apache.org/jira/browse/HDFS-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068239#comment-14068239 ] Vinayakumar B commented on HDFS-6686: - This patch doesn't apply properly. I think this depends on HDFS-6710. Archival Storage: Use fallback storage types Key: HDFS-6686 URL: https://issues.apache.org/jira/browse/HDFS-6686 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: h6686_20140721.patch HDFS-6671 changes replication monitor to use block storage policy for replication. It should also use the fallback storage types when a particular type of storage is full. -- This message was sent by Atlassian JIRA (v6.2#6252)
