[
https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527317#comment-16527317
]
Benson Qiu commented on AVRO-2195:
----------------------------------
Initial Patch (AVRO-2195.patch):
* Code is based on AVRO-1373
* Existing tests pass. However, I did not write new unit tests yet.
* I did a manual test by creating a quick Java application that writes Avro
files using zstandard compression:
{code:java}
// ...
dataFileWriter.setCodec(CodecFactory.zstandardCodec());
// ...
while (dataFileReader.hasNext()) {
record = dataFileReader.next();
dataFileWriter.append(record);
}
// ...
{code}
In my manual test, I observed that my Java application needs a dependency on
`ztd-jni`. Without that dependency, we get a `ClassNotFoundException` for
`com.github.luben.zstd.ZstdOutputStream`.
{code:java}
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>1.3.4-10</version>
</dependency>{code}
> Add Zstandard Codec
> -------------------
>
> Key: AVRO-2195
> URL: https://issues.apache.org/jira/browse/AVRO-2195
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.9.0
> Reporter: Benson Qiu
> Priority: Major
> Labels: patch
> Attachments: AVRO-2195.patch
>
>
> Inspired by AVRO-1373. The Zstandard algorithm is available in the
> commons-library, which Avro projects already depend on.
> In a quick test that I did, Zstandard had a better compression ratio than
> deflate (compression level 9), with significantly faster compression times.
> [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
