[jira] [Commented] (HDFS-8120) Erasure coding: created util class to analyze striped block groups

[Jing Zhao (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490640#comment-14490640
 ] 

Jing Zhao commented on HDFS-8120:
---------------------------------

Thanks for working on this, Zhe!
{code}
+    int numStripes = (int) (blkGroupSize / (cellSize * dataBlkNum));
+    // Parity block
+    if (idxInBlockGroup >= dataBlkNum) {
+      return numStripes * cellSize;
+    }
{code}

For parity blocks, we need to consider the last partial stripe. But here we 
only get the number of full stripes.

Actually I started working on this {{checkReplicaCorrupt}} bug early this week 
but got distracted by some other issues. Please see if the following code makes 
sense to you:
{code}
public long getInternalBlockLength(Block block) {
  Preconditions.checkArgument(this.getBlockId() ==
      BlockIdManager.convertToStripedID(block.getBlockId()));
  final int index = BlockIdManager.getBlockIndex(block);
  Preconditions.checkArgument(index < getTotalBlockNum());

  if (this.getNumBytes() == 0) {
    return 0;
  }
  final long stripeLength = BLOCK_STRIPED_CELL_SIZE * this.getDataBlockNum();
  // for parity blocks
  if (index >= this.getDataBlockNum()) {
    return ((this.getNumBytes() - 1) / stripeLength + 1) * 
BLOCK_STRIPED_CELL_SIZE;
  }
  // for data blocks
  long length = this.getNumBytes() / stripeLength * BLOCK_STRIPED_CELL_SIZE;
  long lastStripe = this.getNumBytes() % stripeLength;
  return length + Math.min(BLOCK_STRIPED_CELL_SIZE,
      Math.max(0, lastStripe - BLOCK_GROUP_INDEX_MASK * index));
}
{code}

More importantly, because this computation is critical, we need to have unit 
tests covering *all* the scenarios.

> Erasure coding: created util class to analyze striped block groups
> ------------------------------------------------------------------
>
>                 Key: HDFS-8120
>                 URL: https://issues.apache.org/jira/browse/HDFS-8120
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch
>
>
> The patch adds logic of calculating size of individual blocks in a striped 
> block group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)