[
https://issues.apache.org/jira/browse/HDFS-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11248:
----------------------------
Fix Version/s: 3.2.0
> [SPS]: Handle partial block location movements
> ----------------------------------------------
>
> Key: HDFS-11248
> URL: https://issues.apache.org/jira/browse/HDFS-11248
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Affects Versions: HDFS-10285
> Reporter: Rakesh R
> Assignee: Rakesh R
> Priority: Major
> Fix For: HDFS-10285, 3.2.0
>
> Attachments: HDFS-11248-HDFS-10285-00.patch,
> HDFS-11248-HDFS-10285-01.patch, HDFS-11248-HDFS-10285-02.patch,
> HDFS-11248-HDFS-10285-03.patch, HDFS-11248-HDFS-10285-04.patch,
> HDFS-11248-HDFS-10285-05.patch
>
>
> This jira is to handle partial block location movements due to unavailability
> of target nodes for the matching storage type.
> For example, We have only A(disk,archive), B(disk) and C(disk,archive) are
> live nodes with A & C have archive storage type. Say, we have a block with
> locations {{A(disk), B(disk), C(disk)}}. Again assume, user changed the
> storage policy to COLD. Now, SPS internally starts preparing the src-target
> pairing like, {{src=> (A, B, C) and target=> (A, C)}} and sends
> BLOCK_STORAGE_MOVEMENT to the coordinator. SPS is skipping B as it doesn't
> have archive media to indicate that it should do retries to satisfy all block
> locations after some time. On receiving the movement command, coordinator
> will pair the src-target node to schedule actual physical movements like,
> {{movetask=> (A, A), (B, C)}}. Here ideally it should do {{(C, C)}} instead
> of {{(B, C)}} but mistakenly choosing the source C and creates problem.
> IMHO, the implicit assumptions of retry needed is creating confusions and
> leads to coding mistakes. One idea to fix this problem is to create a new
> flag {{retryNeeded}} flag to make it more readable. With this, SPS will
> prepare only the matching pair and dummy source slots will be avoided like,
> {{src=> (A, C) and target=> (A, C)}} and mark {{retryNeeded=true}} to convey
> the message that this {{trackId}} has only partial blocks movements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
