[
https://issues.apache.org/jira/browse/HDFS-16538?focusedWorklogId=756267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756267
]
ASF GitHub Bot logged work on HDFS-16538:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Apr/22 09:13
Start Date: 13/Apr/22 09:13
Worklog Time Spent: 10m
Work Description: liubingxing opened a new pull request, #4167:
URL: https://github.com/apache/hadoop/pull/4167
We found this error if the #StripeReader.readStripe() have more than one
block read failed.
```
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.<init>(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at
org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
```
```java
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
}
```
If there are two blocks read failed, the #readDataForDecoding() will be
called twice;
The **decodeInputs array** will be initialized twice, and the **parity
data** in decodeInputs array which filled by #readParityChunks will be set to
null.
Issue Time Tracking
-------------------
Worklog Id: (was: 756267)
Remaining Estimate: 0h
Time Spent: 10m
> EC decoding failed due to not enough valid inputs
> --------------------------------------------------
>
> Key: HDFS-16538
> URL: https://issues.apache.org/jira/browse/HDFS-16538
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: qinyuren
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We found this error if the #StripeReader.readStripe() have more than one
> block read failed.
>
> {code:java}
> Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
> inputs are provided, not recoverable
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.<init>(ByteBufferDecodingState.java:47)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at
> org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> {code}
>
>
> {code:java}
> while (!futures.isEmpty()) {
> try {
> StripingChunkReadResult r = StripedBlockUtil
> .getNextCompletedStripedRead(service, futures, 0);
> dfsStripedInputStream.updateReadStats(r.getReadStats());
> DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
> r, alignedStripe);
> StripingChunk returnedChunk = alignedStripe.chunks[r.index];
> Preconditions.checkNotNull(returnedChunk);
> Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
> if (r.state == StripingChunkReadResult.SUCCESSFUL) {
> returnedChunk.state = StripingChunk.FETCHED;
> alignedStripe.fetchedChunksNum++;
> updateState4SuccessRead(r);
> if (alignedStripe.fetchedChunksNum == dataBlkNum) {
> clearFutures();
> break;
> }
> } else {
> returnedChunk.state = StripingChunk.MISSING;
> // close the corresponding reader
> dfsStripedInputStream.closeReader(readerInfos[r.index]);
> final int missing = alignedStripe.missingChunksNum;
> alignedStripe.missingChunksNum++;
> checkMissingBlocks();
> readDataForDecoding();
> readParityChunks(alignedStripe.missingChunksNum - missing);
> } {code}
> If there are two blocks read failed, the #readDataForDecoding() will be
> called twice;
> The *decodeInputs array* will be initialized twice, and the *parity* *data*
> in decodeInputs array which filled by #readParityChunks will be set to null.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
