[
https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200769#comment-15200769
]
Kai Zheng commented on HADOOP-11540:
------------------------------------
Hi Colin, sorry I have to re-clarify some bit about the point I quoted below.
bq. Why not have the Java coders reset the buffers at the beginning as well,
then?
When I looked closely to the codes (quite some time ago I wrote the codes so I
almost forgot them all.), I found all the coders will call the following
function at the very beginning of {{encode/decode}} call, so Java coders do
{{resetBuffer}} already because they want to init output buffers using the
default method by default. The calls are made in {{AbstractRawErasureEncoder}}
and {{AbstractRawErasureDecoder}}, which both Java coders and Native coders
will extend from. If I move the logic to Java coders out of the common base,
then there are quite a few coders and classes to change, also involving an
extra loop to reset the output buffers in addition to the checkParameterBuffers
call.
{code}
/**
* Check and ensure the buffers are of the length specified by dataLen, also
* ensure the buffers are direct buffers or not according to isDirectBuffer.
* If the buffers are output buffers, ensure they will be ZEROed.
* @param buffers the buffers to check
* @param allowNull whether to allow any element to be null or not
* @param dataLen the length of data available in the buffer to ensure with
* @param isDirectBuffer is direct buffer or not to ensure with
* @param isOutputs is output buffer or not
*/
protected void checkParameterBuffers(ByteBuffer[] buffers, boolean
allowNull, int dataLen, boolean isDirectBuffer, boolean isOutputs) {
for (ByteBuffer buffer : buffers) {
if (buffer == null && !allowNull) {
throw new HadoopIllegalArgumentException(
"Invalid buffer found, not allowing null");
} else if (buffer != null) {
if (buffer.remaining() != dataLen) {
throw new HadoopIllegalArgumentException(
"Invalid buffer, not of length " + dataLen);
}
if (buffer.isDirect() != isDirectBuffer) {
throw new HadoopIllegalArgumentException(
"Invalid buffer, isDirect should be " + isDirectBuffer);
}
if (isOutputs && wantInitOutputs()) {
resetBuffer(buffer, dataLen);
}
}
}
}
{code}
I'm going to refine the relevant methods some bit, add more Javadocs and
comments to make they're more readable. Hope it works for you.
> Raw Reed-Solomon coder using Intel ISA-L library
> ------------------------------------------------
>
> Key: HADOOP-11540
> URL: https://issues.apache.org/jira/browse/HADOOP-11540
> Project: Hadoop Common
> Issue Type: Sub-task
> Affects Versions: HDFS-7285
> Reporter: Zhe Zhang
> Assignee: Kai Zheng
> Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch,
> HADOOP-11540-v2.patch, HADOOP-11540-v4.patch, HADOOP-11540-v5.patch,
> HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel
> ISAL-v1.pdf
>
>
> This is to provide RS codec implementation using Intel ISA-L library for
> encoding and decoding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
