[
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410219#comment-16410219
]
ASF GitHub Bot commented on AVRO-2162:
--------------------------------------
scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176558252
##########
File path: lang/java/avro/src/main/java/org/apache/avro/file/XZCodec.java
##########
@@ -57,62 +52,22 @@ public String getName() {
}
@Override
- public ByteBuffer compress(ByteBuffer data) throws IOException {
- ByteArrayOutputStream baos = getOutputBuffer(data.remaining());
- OutputStream ios = new XZCompressorOutputStream(baos, compressionLevel);
- writeAndClose(data, ios);
- return ByteBuffer.wrap(baos.toByteArray());
+ protected OutputStream compressedStream(OutputStream output)
+ throws IOException {
+ return new XZCompressorOutputStream(output, compressionLevel);
}
@Override
- public ByteBuffer decompress(ByteBuffer data) throws IOException {
- ByteArrayOutputStream baos = getOutputBuffer(data.remaining());
- InputStream bytesIn = new ByteArrayInputStream(
- data.array(),
- data.arrayOffset() + data.position(),
- data.remaining());
- InputStream ios = new XZCompressorInputStream(bytesIn);
- try {
- IOUtils.copy(ios, baos);
- } finally {
- ios.close();
- }
- return ByteBuffer.wrap(baos.toByteArray());
+ protected InputStream uncompressedStream(InputStream input)
+ throws IOException {
+ return new XZCompressorInputStream(input);
}
- private void writeAndClose(ByteBuffer data, OutputStream to) throws
IOException {
- byte[] input = data.array();
- int offset = data.arrayOffset() + data.position();
- int length = data.remaining();
- try {
- to.write(input, offset, length);
- } finally {
- to.close();
- }
- }
-
- // get and initialize the output buffer for use.
- private ByteArrayOutputStream getOutputBuffer(int suggestedLength) {
- if (null == outputBuffer) {
- outputBuffer = new ByteArrayOutputStream(suggestedLength);
- }
- outputBuffer.reset();
- return outputBuffer;
- }
-
- @Override
- public int hashCode() {
- return compressionLevel;
- }
+ @Override public int hashCode() { return getName().hashCode(); }
@Override
- public boolean equals(Object obj) {
- if (this == obj)
- return true;
- if (getClass() != obj.getClass())
- return false;
- XZCodec other = (XZCodec)obj;
Review comment:
The equals method for the XZ codec was wrong -- according to the
specification of Codec, it should equal if it is mutually decompressible. The
compression level is used for the compressor but does not affect
decompressibility.
I made the implementation of hashCode and equals consistent across the
codecs when appropriate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add Zstandard compression to avro file format
> ---------------------------------------------
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Scott Carey
> Priority: Major
>
> I'd like to add Zstandard compression for Avro.
> At compression level 1 It is almost as fast as Snappy at compression, with
> compression ratios more like gzip. At higher levels of compression, it is
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x
> faster decompression.
>
> Adding it to Java is fairly easy. We'll need to say something about it in
> the spec however, as an 'optinal' codec.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
