[
https://issues.apache.org/jira/browse/HDFS-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16577529#comment-16577529
]
Hudson commented on HDFS-11248:
-------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14752 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14752/])
HDFS-11248: [SPS]: Handle partial block location movements. Contributed
(umamahesh: rev b7bed9f00a25bcad6f9c3543f5a1fb0a1f23b0e9)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockStorageMovementAttemptedItems.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySatisfier.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySatisfier.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockStorageMovementTracker.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockStorageMovementAttemptedItems.java
> [SPS]: Handle partial block location movements
> ----------------------------------------------
>
> Key: HDFS-11248
> URL: https://issues.apache.org/jira/browse/HDFS-11248
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Affects Versions: HDFS-10285
> Reporter: Rakesh R
> Assignee: Rakesh R
> Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-11248-HDFS-10285-00.patch,
> HDFS-11248-HDFS-10285-01.patch, HDFS-11248-HDFS-10285-02.patch,
> HDFS-11248-HDFS-10285-03.patch, HDFS-11248-HDFS-10285-04.patch,
> HDFS-11248-HDFS-10285-05.patch
>
>
> This jira is to handle partial block location movements due to unavailability
> of target nodes for the matching storage type.
> For example, We have only A(disk,archive), B(disk) and C(disk,archive) are
> live nodes with A & C have archive storage type. Say, we have a block with
> locations {{A(disk), B(disk), C(disk)}}. Again assume, user changed the
> storage policy to COLD. Now, SPS internally starts preparing the src-target
> pairing like, {{src=> (A, B, C) and target=> (A, C)}} and sends
> BLOCK_STORAGE_MOVEMENT to the coordinator. SPS is skipping B as it doesn't
> have archive media to indicate that it should do retries to satisfy all block
> locations after some time. On receiving the movement command, coordinator
> will pair the src-target node to schedule actual physical movements like,
> {{movetask=> (A, A), (B, C)}}. Here ideally it should do {{(C, C)}} instead
> of {{(B, C)}} but mistakenly choosing the source C and creates problem.
> IMHO, the implicit assumptions of retry needed is creating confusions and
> leads to coding mistakes. One idea to fix this problem is to create a new
> flag {{retryNeeded}} flag to make it more readable. With this, SPS will
> prepare only the matching pair and dummy source slots will be avoided like,
> {{src=> (A, C) and target=> (A, C)}} and mark {{retryNeeded=true}} to convey
> the message that this {{trackId}} has only partial blocks movements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
