[
https://issues.apache.org/jira/browse/HBASE-26259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421866#comment-17421866
]
Andrew Kyle Purtell edited comment on HBASE-26259 at 9/29/21, 2:00 AM:
-----------------------------------------------------------------------
I have completed scale load testing with the codecs I am most interested in --
lz4/lz4-java, lz4/aircompressor, and snappy/aircompressor -- and the system is
stable with them in operation replacing the Hadoop native codecs, and in fact
in some cases the Hadoop native codecs (e.g. the native LZ4 codec) can cause
data corruption where in the same repeatable scenarios
(IntegrationTestLoadCommonCrawl) our new codecs do not. The lzma and
snappy/xerial codecs were fine when tested at smaller scale too.
Both HFile block and WAL value compression were tested.
was (Author: apurtell):
I have completed scale load testing with the codecs I am most interested in --
lz4/lz4-java, lz4/aircompressor, and snappy/aircompressor -- and the system is
stable with them in operation replacing the Hadoop native codecs, and in fact
in some cases the Hadoop native codecs (e.g. the native LZ4 codec) can cause
data corruption where in the same repeatable scenarios
(IntegrationTestLoadCommonCrawl) our new codecs do not. The lzma and
snappy/xerial codecs were fine when tested at smaller scale too.
> Fallback support to pure Java compression
> -----------------------------------------
>
> Key: HBASE-26259
> URL: https://issues.apache.org/jira/browse/HBASE-26259
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
> Attachments: BenchmarkCodec.java, BenchmarksMain.java,
> RandomDistribution.java, ac_lz4_results.pdf, ac_snappy_results.pdf,
> ac_zstd_results.pdf, lz4_lz4-java_result.pdf, xerial_snappy_results.pdf
>
>
> Airlift’s aircompressor
> (https://search.maven.org/artifact/io.airlift/aircompressor) is an Apache 2
> licensed library, for Java 8 and up, available in Maven central, which
> provides pure Java implementations of gzip, lz4, lzo, snappy, and zstd and
> Hadoop compression codecs for same, claiming “_they are typically 300% faster
> than the JNI wrappers_.” (https://github.com/airlift/aircompressor). This
> library is under active development and up to date releases because it is
> used by Trino.
> Proposed changes:
> * Modify Compression.java such that compression codec implementation classes
> can be specified by configuration. Currently they are hardcoded as strings.
> * Pull in aircompressor as a ‘compile’ time dependency so it will be bundled
> into our build and made available on the server classpath.
> * Modify Compression.java to fall back to an aircompressor pure Java
> implementation if schema specifies a compression algorithm, a Hadoop native
> codec was specified as desired implementation, but the requisite native
> support is somehow not available.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
