[
https://issues.apache.org/jira/browse/HDFS-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153386#comment-16153386
]
Kai Zheng commented on HDFS-12392:
----------------------------------
Thanks [~Sammi] for catching this!! You figured out the root cause.
1. The fix broke the assumption laid out in the coder framework that all the
input buffers should have the same readable bytes length. So please fix the
issue in HDFS side, trying to keep the assumption.
In ByteBufferEncodingState.java:
{code}
void checkBuffers(ByteBuffer[] buffers) {
"Invalid buffer found, not allowing null");
}
- if (buffer.remaining() != encodeLength) {
+ if (buffer.remaining() < encodeLength) {
throw new HadoopIllegalArgumentException(
- "Invalid buffer, not of length " + encodeLength);
+ "Invalid buffer, length smaller than " + encodeLength);
}
{code}
2. The added test case {{testDifferentErasureCodingPolicyCellSize}} looks
great. Some minors like:
{code}
new ErasureCodingPolicy[]{newPolicy1}
{code}
=>
{code}
new ErasureCodingPolicy[] { newPolicy1 };
{code}
> Write striped file failure
> --------------------------
>
> Key: HDFS-12392
> URL: https://issues.apache.org/jira/browse/HDFS-12392
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
> Affects Versions: 3.0.0-alpha3
> Reporter: SammiChen
> Assignee: SammiChen
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12392.001.patch
>
>
> Root cause: The buffer size returned by ElasticByteBufferPool.getBuffer() is
> more than caller expected.
> Exception stack:
> org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of
> length 4096
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferEncodingState.checkBuffers(ByteBufferEncodingState.java:99)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferEncodingState.<init>(ByteBufferEncodingState.java:46)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureEncoder.encode(RawErasureEncoder.java:67)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.encode(DFSStripedOutputStream.java:368)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.writeParityCells(DFSStripedOutputStream.java:942)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:547)
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:217)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:125)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:111)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94)
> at org.apache.hadoop.hdfs.DFSTestUtil.writeFile(DFSTestUtil.java:834)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
