[
https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17468417#comment-17468417
]
Adrien Grand commented on LUCENE-8739:
--------------------------------------
I updated block sizes so that ZSTD uses the same block sizes as
BEST_COMPRESSION and it looks much better now.
||Codec ||Indexing time (ms) ||Disk usage (MB) || Retrieval time per 10k docs
(ms) ||
| BEST_SPEED (LZ4 with small blocks) | 35383 | 90.175 | 190.17524 |
| BEST_COMPRESSION (vanilla zlib, DEFLATE level=6) | 76671 | 58.682 |
1910.42106 |
| BEST_COMPRESSION (Cloudflare zlib, DEFLATE level=6) | 54791 | 58.601 |
1395.53593 |
| ZSTD dict (level=1) | 24687 | 63.324 | 928.73997 |
| ZSTD dict (level=2) | 24934 | 63.722 | 977.29911 |
| ZSTD dict (level=3) | 28285 | 62.072 | 938.10886 |
| ZSTD dict (level=4) | 37863 | 60.427 | 969.18655 |
| ZSTD dict (level=5) | 45479 | 59.317 | 941.20922 |
| ZSTD dict (level=6) | 57842 | 58.481 | 881.69049 |
| ZSTD dict (level=7) | 65796 | 58.107 | 886.42249 |
On this dataset, the main benefit seems to be the retrieval speed. Regarding
indexing times and space efficiency, either you go with level 5 and you are
faster to index data but less space-efficient than DEFLATE (with the Cloudflare
zlib), or you go with level 6 and you are more space-efficient but slower to
index.
> ZSTD Compressor support in Lucene
> ---------------------------------
>
> Key: LUCENE-8739
> URL: https://issues.apache.org/jira/browse/LUCENE-8739
> Project: Lucene - Core
> Issue Type: New Feature
> Components: core/codecs
> Reporter: Sean Torres
> Priority: Minor
> Labels: features
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> ZStandard has a great speed and compression ratio tradeoff.
> ZStandard is open source compression from Facebook.
> More about ZSTD
> [https://github.com/facebook/zstd]
> [https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
