[
https://issues.apache.org/jira/browse/HDFS-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749800#comment-15749800
]
Uma Maheswara Rao G commented on HDFS-11193:
--------------------------------------------
{quote}
While testing I found an issue in when there is no target node with the
required storage type logic. For example, I have a block with locations
A(disk), B(disk), C(disk) and assume only A, B and C are live nodes with A & C
have archive storage type. Again assume, user changed the storage policy to
COLD. Now, SPS internally starts preparing the src-target pairing like, src=>
(A, B, C) and target=> (A, C). Its skipping B as it doesn't have archive media
and this is an indication that SPS should do retries for satisfying all of its
block locations. On the other side, coordinator will pair the src-target node
for actual physical movement like, movetask=> (A, A), (B, C). Here ideally it
should do (C, C) instead of (B, C) but mistakenly choosing the source C. I
think, the implicit assumptions of retry needed will create confusions and
coding mistakes like this. In this patch, I've created a new flag retryNeeded
flag to make it more readable. Now, SPS will prepare only the matching pair and
dummy source slots will be avoided like, src=> (A, C) and target=> (A, C) and
set retryNeeded=true to convey the message that this trackId has only partial
blocks movements.
{quote}
This is a good test case. Since this is a bug in existing part, can we make
separate JIRA for this issue? Otherwise the current JIRA will mark that issue.
I would like to review that fix separately than combining with this patch.
{quote}
Thanks for this idea. Following is my analysis on this approach. As we know,
presently NN is passing simple Block objects to the coordinator datanode for
movement. Inorder to do the internal block constrcution at the DN side, it
requires the BlockInfoStriped complex object and the blockIndices array. I
think passing list of simple object is better compare to the complex object,
this will keep all the computation complexities at the SPS side and makes the
coordinator logic more readable. I'd prefer to keep the internal block
constrcution logic at the NN side. Does this make sense to you?
{quote}
I think this is ok. My suggestion was based on EC. When we send ECRecovery
command, we will just send Block and liveBlocks indices, actual Blk id will be
constructed back at DN. It should not be a big deal. Since this command
combines both EC and non EC, lets just construct full details at NN only.
> [SPS]: Erasure coded files should be considered for satisfying storage policy
> -----------------------------------------------------------------------------
>
> Key: HDFS-11193
> URL: https://issues.apache.org/jira/browse/HDFS-11193
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Rakesh R
> Assignee: Rakesh R
> Attachments: HDFS-11193-HDFS-10285-00.patch,
> HDFS-11193-HDFS-10285-01.patch, HDFS-11193-HDFS-10285-02.patch
>
>
> Erasure coded striped files supports storage policies {{HOT, COLD, ALLSSD}}.
> {{HdfsAdmin#satisfyStoragePolicy}} API call on a directory should consider
> all immediate files under that directory and need to check that, the files
> really matching with namespace storage policy. All the mismatched striped
> blocks should be chosen for block movement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
