[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HDFS-14768:
-------------------------
Description:
We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
index[3,4], increase the index 6 datanode's
pendingReplicationWithoutTargets that make it large than
replicationStreamsHardLimit(we set 14). Then, After the method
chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
[0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
In method scheduleReconstruction of BlockManager, the additionalReplRequired is
9 - 7 = 2. After Namenode choose two target Datanode, will assign a erasureCode
task to target datanode.
When datanode get the task will build targetIndices from liveBlockIndices and
target length. the code is blow.
{code:java}
// code placeholder
targetIndices = new short[targets.length];
private void initTargetIndices() {
BitSet bitset = reconstructor.getLiveBitSet();
int m = 0; hasValidTargets = false;
for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
if (!bitset.get) {
if (reconstructor.getBlockLen > 0) {
if (m < targets.length) {
targetIndices[m++] = (short)i;
hasValidTargets = true;
}
}
}
}
{code}
targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
The StripedReader is aways create reader from first 6 index block, and is
[0,1,2,3,4,5]
Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
bug. the block index6's data is c
was:
We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
index[3,4], increase the index 6 datanode's
pendingReplicationWithoutTargets that make it large than
replicationStreamsHardLimit(we set 14). Then, After the method
chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
[0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
In method scheduleReconstruction of BlockManager, the additionalReplRequired is
9 - 7 = 2. After Namenode choose two target Datanode, will assign a erasureCode
task to target datanode.
When datanode get the task will build targetIndices from liveBlockIndices and
target length.
{code:java}
// code placeholder
targetIndices = new short[targets.length]; private void initTargetIndices() {
BitSet bitset = reconstructor.getLiveBitSet(); int m = 0; hasValidTargets =
false; for (int i = 0; i < dataBlkNum + parityBlkNum; i++) { if (!bitset.get)
{ if (reconstructor.getBlockLen > 0) { if (m < targets.length) {
targetIndices[m++] = (short)i; hasValidTargets = true; }
} } }
{code}
> In some cases, erasure blocks are corruption when they are rebuilt.
> --------------------------------------------------------------------
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
> Affects Versions: 3.0.2
> Reporter: guojh
> Priority: Major
>
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is c
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
