nickva opened a new pull request, #6028:
URL: https://github.com/apache/couchdb/pull/6028
This is using the built-in zstd API from Erlang/OTP 28+. For 26 and 27 it
falls back to `deflate`. Otherwise it's not that different than deflate and
snappy which we already support.
An interesting feature of zstd is to use dictionaries. This PR doesn't do
it, but it sets up a shim `couch_zstd` module where that could plug-in
eventually. The idea is to train a dictionary it based on document bodies for
each database before compaction and then saving the dictionary. zstd as is in
Erlang/OTP doesn't expose a trainign API but we can still use a
concatenative/raw dictionary:
```erlang
> Dict = ~"{\"type\":\"foo\",\"bar\":1,\"baz\":\"a\"}".
<<"{\"type\":\"foo\",\"bar\":1,\"baz\":\"a\"}">>
> {ok, C} = zstd:dict(compress, Dict).
{ok,{cdict,#Ref<0.4075923125.4131520536.31325>}}
> Doc = ~"{\"type\":\"foo\", \"bar\":2, \"baz\":\"b\"}".
<<"{\"type\":\"foo\", \"bar\":2, \"baz\":\"b\"}">>
> byte_size(iolist_to_binary(zstd:compress(Doc, #{dictionary=>C}))).
29
> byte_size(iolist_to_binary(zstd:compress(Doc))).
43
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
