[
https://issues.apache.org/jira/browse/HADOOP-10047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798260#comment-13798260
]
Colin Patrick McCabe commented on HADOOP-10047:
-----------------------------------------------
bq. I did that because I wanted to extend the compressor with as much
compatibility as possible. There is no JNI code for indirect buffers here, it
copies into a direct buffer - as the codecs have always done. But it does add
complexity which is unnecessary.
Copies are a performance killer, and I'm sure that nobody is in favor of
"unnecessary complexity" (to use your words) :) Let's just make it an
interface for direct byte buffers.
bq. I posted the API docs hoping for some commentary on people who use the
CompressionCodecs. From my side, I modeled it around the API needs of ORC when
I designed it and extended it to add the ability to decompress zero-copy reads
which cross block boundaries (in zlib+bzip2 at least - unlikely for SNAPPY/LZ4
native APIs).
I think this interface should support short reads so that the
compressor/decompressor can stop on a codec block boundary if possible. That
way, we can be as efficient as possible.
bq. In that angle, I only need the Decompressor APIs and I do not think I
intend to use the Compressor APIs at all. That was done to round it off cleanly.
Why not just do the decompressor for now? If we try to do both, it may take
twice as long :) We can always do the compressor in a follow-up JIRA.
bq. If you have an idea on adding it without adding more derivatives of
CodecPool, CompressionCodec and CompressionCodecFactory, I will be interested
in hearing that. The down-casting seemed like the easiest way to check for the
API's presence. But that was just the path of least resistance for me.
{code}
public interface DirectDecompressionCodec extends CompressionCodec {
/**
* Create a new {@link DirectDecompressor} for use by this {@link
DirectDecompressionCodec}.
*
* @return a new direct decompressor for use by this codec
*/
DirectDecompressor createDirectDecompressor();
}
{code}
{{CompressionCodec}} instances that have direct support can implement this;
ones that don't won't implement this. {{DirectDecompressor}} can be a
standalone interface, not related to {{Decompressor}}, and the classes that
implement it can be separate too. As you said, it's a lot simpler with
separate classes.
> Allow Compressor/Decompressor APIs to expose a Direct ByteBuffer API
> --------------------------------------------------------------------
>
> Key: HADOOP-10047
> URL: https://issues.apache.org/jira/browse/HADOOP-10047
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io
> Reporter: Gopal V
> Assignee: Gopal V
> Labels: compression
> Attachments: DirectCompressor.html, DirectDecompressor.html,
> HADOOP-10047-WIP.patch, HADOOP-10047-with-tests.patch
>
>
> With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform
> all I/O operations without copying data into byte[] buffers or other buffers
> which wrap over them.
> This is a proposal for adding new DirectCompressor and DirectDecompressor
> interfaces to the io.compress, to indicate codecs which want to surface the
> direct buffer layer upwards.
> The implementation may or may not copy the buffers passed in, but should work
> with direct heap/mmap buffers and cannot assume .array() availability.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
