[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HDFS-14768:
-------------------------
Description:
We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
index[3,4], increase the index 6 datanode's
pendingReplicationWithoutTargets that make it large than
replicationStreamsHardLimit(we set 14). Then, After the method
chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
[0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
In method scheduleReconstruction of BlockManager, the additionalReplRequired is
9 - 7 = 2. After Namenode choose two target Datanode, will assign a erasureCode
task to target datanode.
When datanode get the task will build targetIndices from liveBlockIndices and
target length.
{code:java}
// code placeholder
targetIndices = new short[targets.length]; private void initTargetIndices() {
BitSet bitset = reconstructor.getLiveBitSet(); int m = 0; hasValidTargets =
false; for (int i = 0; i < dataBlkNum + parityBlkNum; i++) { if (!bitset.get)
{ if (reconstructor.getBlockLen > 0) { if (m < targets.length) {
targetIndices[m++] = (short)i; hasValidTargets = true; }
} } }
{code}
was:
We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
index[3,4], increase the index 6 datanode's
pendingReplicationWithoutTargets that make it large than
replicationStreamsHardLimit(we set 14). Then, After the method
chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
[0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
In method scheduleReconstruction of BlockManager, the additionalReplRequired is
9 - 7 = 2. After Namenode choose two target Datanode, will assign a erasureCode
task to target datanode.
When datanode get the task will build targetIndices from liveBlockIndices and
target length.
{code:java}
// code placeholder
{code}
targetIndices = new short[targets.length];
private void initTargetIndices() {
BitSet bitset = reconstructor.getLiveBitSet();
int m = 0;
hasValidTargets = false;
for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
if (!bitset.get(i)) {
if (reconstructor.getBlockLen(i) > 0) {
if (m < targets.length) {
targetIndices[m++] = (short)i;
hasValidTargets = true;
}
}
}
}
> In some cases, erasure blocks are corruption when they are rebuilt.
> --------------------------------------------------------------------
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
> Affects Versions: 3.0.2
> Reporter: guojh
> Priority: Major
>
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length]; private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet(); int m = 0; hasValidTargets =
> false; for (int i = 0; i < dataBlkNum + parityBlkNum; i++) { if
> (!bitset.get) { if (reconstructor.getBlockLen > 0) { if (m <
> targets.length) { targetIndices[m++] = (short)i;
> hasValidTargets = true; } } } }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
