[
https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415363#comment-17415363
]
Praveen Nishchal edited comment on LUCENE-8739 at 9/15/21, 7:51 AM:
--------------------------------------------------------------------
I have developed new custom codec which integrates Zstd compression and
decompression in StoredFieldFormat only. It uses Zstd-JNI
([https://github.com/luben/zstd-jni]). With reuters21578 (plain text Document
derived from reuters21578) corpus benchmark run for index and search, following
high level observations were made:
# Zstd provides a better compression ratio compared to lz4. Benchmark
run(index) shows 30% smaller size .fdt(Stored Field data) file compared to LZ4.
# Index run with Zstd has almost same throughput as that of index run with LZ4.
# Search run with Zstd has 6% faster QPS than search run with LZ4
Above implementation is written in Java without dictionary
compression/decompression at default compression level of 3 with 600 KB chunk
size (10 * 60 * 1024 , same as LZ4).
With all these observations, Zstd option alongside LZ4 and deflate looks
promising!! Kindly share thoughts!
was (Author: pru30):
_I have developed new custom codec which integrates Zstd compression and
decompression in StoredFieldFormat_ _only. It uses Zstd-JNI
([https://github.com/luben/zstd-jni]). With reuters21578 (plain text Document
derived from reuters21578) corpus benchmark run for index and search, following
high level observations were made:_
__
# _Zstd provides a better compression ratio compared to lz4. Benchmark
run(index) shows 30% smaller size .fdt(Stored Field data) file compared to LZ4._
# _Index run with Zstd has almost same throughput as that of index run with
LZ4._
# _Search run with Zstd has 6% faster QPS than search run with LZ4_
__
_Above implementation is written in Java without dictionary
compression/decompression at default compression level of 3 with 600 KB chunk
size (10 * 60 * 1024 , same as LZ4)._
__
_With all these observations, Zstd option alongside LZ4 and deflate looks
promising!! Kindly share thoughts!_
> ZSTD Compressor support in Lucene
> ---------------------------------
>
> Key: LUCENE-8739
> URL: https://issues.apache.org/jira/browse/LUCENE-8739
> Project: Lucene - Core
> Issue Type: New Feature
> Components: core/codecs
> Reporter: Sean Torres
> Priority: Minor
> Labels: features
> Time Spent: 1h
> Remaining Estimate: 0h
>
> ZStandard has a great speed and compression ratio tradeoff.
> ZStandard is open source compression from Facebook.
> More about ZSTD
> [https://github.com/facebook/zstd]
> [https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
