[
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410223#comment-16410223
]
ASF GitHub Bot commented on AVRO-2162:
--------------------------------------
scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176558658
##########
File path: lang/java/avro/src/test/java/org/apache/avro/file/TestCodecs.java
##########
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.avro.file;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Random;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+
+@RunWith(Parameterized.class)
+public class TestCodecs {
+
+ @Parameters(name = "Codec: {0}")
+ public static Collection<Object[]> data() {
+ return Arrays.asList(new Object[][] {
+ { new NullCodec() },
+ { new DeflateCodec(3) },
+ { new BZip2Codec() },
+ { new SnappyCodec() },
+ { new XZCodec(3) },
+ { new ZstandardCodec(3) }
+ });
+ }
+
+ private final Codec codec;
+ private final byte[] zeroes = new byte[1024*1024];
Review comment:
all zeroes tends to compress massively -- which can uncover bugs in buffer
sizing when decompressing.
pure random tends to be _larger_ when compressed than uncompressed and may
find bugs in buffer sizing when compressing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add Zstandard compression to avro file format
> ---------------------------------------------
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Scott Carey
> Priority: Major
>
> I'd like to add Zstandard compression for Avro.
> At compression level 1 It is almost as fast as Snappy at compression, with
> compression ratios more like gzip. At higher levels of compression, it is
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x
> faster decompression.
>
> Adding it to Java is fairly easy. We'll need to say something about it in
> the spec however, as an 'optinal' codec.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
