[
https://issues.apache.org/jira/browse/CASSANDRA-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687784#comment-17687784
]
Stefan Miklosovic edited comment on CASSANDRA-17021 at 2/13/23 8:02 AM:
------------------------------------------------------------------------
[~yifanc]
I am curious how you solved the training for each SSTable. My solution (I do
not have a working prototype yet) is to basically iterate over all partitions /
mutations to be written before they are flushed in Flushing class, keep
reference to that built dictionary and then use it again upon actual
compressing which happens later down the road on the write path.
I am also curious how you modified ZstdCompressor to achieve this. I extracted
the common logic of ZstdCompressor to an abstract class and I created
ZstdDictionaryCompressor where I want to keep track to these dictionaries to
use them upon eventual compression / decompression.
The next question is related to performance. Reading through Zstd
documentation, they are saying that dictionaries are good to use for small data
only. But if there is an SSTable having hundreds to megabytes, how would
dictionary look like? What benefit there is to have a dictionary of, lets say,
1Mb, on data having 150Mb+?
Another question - does this mean that we need to load a dictionary _every
single time we do a read_? Because data are not possible to decompress without
dictionary, we need to load dictionary every single time if we do not cache it
and I think that caching does not help either if reading is random enough to
not hit that particular dictionary again.
Last but not least, I think it would be interesting to have a dictionary _per
partition_. That both solves the problem of not needing to load dictionary for
whole SSTable as well as it is smaller so reading it is way more effective.
was (Author: smiklosovic):
I am curious how you solved the training for each SSTable. My solution (I do
not have a working prototype yet) is to basically iterate over all partitions /
mutations to be written before they are flushed in Flushing class, keep
reference to that built dictionary and then use it again upon actual
compressing which happens later down the road on the write path.
I am also curious how you modified ZstdCompressor to achieve this. I extracted
the common logic of ZstdCompressor to an abstract class and I created
ZstdDictionaryCompressor where I want to keep track to these dictionaries to
use them upon eventual compression / decompression.
The next question is related to performance. Reading through Zstd
documentation, they are saying that dictionaries are good to use for small data
only. But if there is an SSTable having hundreds to megabytes, how would
dictionary look like? What benefit there is to have a dictionary of, lets say,
1Mb, on data having 150Mb+?
Another question - does this mean that we need to load a dictionary _every
single time we do a read_? Because data are not possible to decompress without
dictionary, we need to load dictionary every single time if we do not cache it
and I think that caching does not help either if reading is random enough to
not hit that particular dictionary again.
Last but not least, I think it would be interesting to have a dictionary _per
partition_. That both solves the problem of not needing to load dictionary for
whole SSTable as well as it is smaller so reading it is way more effective.
> Enhance Zstd support in Cassandra with dictionaries
> ---------------------------------------------------
>
> Key: CASSANDRA-17021
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17021
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Compression
> Reporter: Dinesh Joshi
> Assignee: Yifan Cai
> Priority: Normal
>
> Currently Cassandra supports zstd compression. However, Zstd also supports
> dictionaries to enhance not only the compression ratio but also the speed.
> Dictionaries can show 3-4x savings. We should add support to train
> dictionaries, ideally per SSTable this will yield the maximum gains.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
