[
https://issues.apache.org/jira/browse/HDFS-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738932#comment-15738932
]
Uma Maheswara Rao G commented on HDFS-11193:
--------------------------------------------
[~rakeshr], Thanks for working on this task. Please check my feedback on this
patch. BTW, looks like it needs rebase on latest code.
# -
{code}
+ if (ErasureCodingPolicyManager
+ .checkStoragePolicySuitableForECStripedMode(
+ existingStoragePolicyID)) {
+ expectedStorageTypes = existingStoragePolicy
+ .chooseStorageTypes((short) blockInfo.getCapacity());
+ } else {
+ // Currently we support only limited policies (HOT, COLD, ALLSSD)
+ // for EC striped mode files.
+ // Mover tool will ignore to move the blocks if the storage policy
+ // is not in EC Striped mode supported policies
+ LOG.warn("The storage policy " + existingStoragePolicy.getName()
+ + " is not suitable for Striped EC files. "
+ + "So, Ignoring to move the blocks");
+ return;
}
{code}
Since we are simply returning here, this item may be added to attempts lists
and later it will be retried? but retries are not needed right?
# -
{code}
// Currently we support only limited policies (HOT, COLD, ALLSSD)
+ // for EC striped mode files.
+ // Mover tool will ignore to move the blocks if the storage policy
+ // is not in EC Striped mode supported policies
{code}
Can you update this comment. It should be “SPS”
# -
{code}
if (blockInfo.isStriped()) {
+ // For a striped block, it needs to construct internal block at the given
+ // index of a block group. Here it is iterating over all the block
indices
+ // and construct internal blocks which can be then considered for block
+ // movement.
+ BlockInfoStriped sBlockInfo = (BlockInfoStriped) blockInfo;
+ for (StorageAndBlockIndex si : sBlockInfo.getStorageAndIndexInfos()) {
+ if (si.getBlockIndex() >= 0) {
+ DatanodeDescriptor dn = si.getStorage().getDatanodeDescriptor();
+ DatanodeInfo[] srcNode = new DatanodeInfo[1];
+ StorageType[] srcStorageType = new StorageType[1];
+ DatanodeInfo[] targetNode = new DatanodeInfo[1];
+ StorageType[] targetStorageType = new StorageType[1];
+ for (int i = 0; i < sourceNodes.size(); i++) {
+ DatanodeInfo node = sourceNodes.get(i);
+ if (node.equals(dn)) {
+ srcNode[0] = node;
+ srcStorageType[0] = sourceStorageTypes.get(i);
+ if (targetNodes.size() > i) {
+ targetNode[0] = targetNodes.get(i);
+ targetStorageType[0] = targetStorageTypes.get(i);
+ } else {
+ // empty target
+ targetNode = new DatanodeInfo[0];
+ targetStorageType = new StorageType[0];
+ }
+ break; // found matching source-target nodes
+ }
+ }
+ // construct internal block
+ long blockId = blockInfo.getBlockId() + si.getBlockIndex();
+ long numBytes = StripedBlockUtil.getInternalBlockLength(
+ sBlockInfo.getNumBytes(), sBlockInfo.getCellSize(),
+ sBlockInfo.getDataBlockNum(), si.getBlockIndex());
+ Block blk = new Block(blockId, numBytes,
+ blockInfo.getGenerationStamp());
+ BlockMovingInfo blkMovingInfo = new BlockMovingInfo(blk, srcNode,
+ targetNode, srcStorageType, targetStorageType);
+ blkMovingInfos.add(blkMovingInfo);
{code}
One another idea in my mind is that, how about just including blockIndexes in
the case of Striped? Like adding new parameter blockIndexes, which will be
empty array in the case of non striped. So, that DN can interpret block ids in
striped case. So, single blockInfo can be included instead of 9 items in the
case of 6,3 EC. But this is also fine but it needs more items to be included in
list. Just a thought.
# -No comment
Thanks for adding more test cases.
> [SPS]: Erasure coded files should be considered for satisfying storage policy
> -----------------------------------------------------------------------------
>
> Key: HDFS-11193
> URL: https://issues.apache.org/jira/browse/HDFS-11193
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Rakesh R
> Assignee: Rakesh R
> Attachments: HDFS-11193-HDFS-10285-00.patch,
> HDFS-11193-HDFS-10285-01.patch
>
>
> Erasure coded striped files supports storage policies {{HOT, COLD, ALLSSD}}.
> {{HdfsAdmin#satisfyStoragePolicy}} API call on a directory should consider
> all immediate files under that directory and need to check that, the files
> really matching with namespace storage policy. All the mismatched striped
> blocks should be chosen for block movement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
