Re: CASSANDRA-14482
I plan on doing some more thorough testing, it’s just a matter of finding time. On Fri, Mar 1, 2019 at 7:17 PM Dinesh Joshi wrote: > Thanks for testing this out Jon. Apart from the compression ratio could > you also test the insertion rate and GC? 14482 allows you to use full range > of Zstd compression levels. > > Dinesh > > > On Mar 1, 2019, at 6:41 PM, Jonathan Haddad wrote: > > > > Hey all, > > > > I finally got around to doing some testing. Nothing too crazy, I had it > > run on my laptop while I did other things around the house. > > > > Test 1: Inserting Random Data in a K/V table, 10 million inserts > > > > LZ4 compression rate: 0.909857609644112 > > ZStd: 0.6136099401596449 > > > > Test 2: Inserting fairly compressable text data into K/V table: 20 > million > > inserts > > LZ4: 0.6950432247957942 > > ZStd: 0.4797311577845362 > > > > Definitely worth it at first glance. More testing to come. > > > > Jon > > > > > > > > > > On Sun, Feb 17, 2019 at 8:46 PM dinesh.jo...@yahoo.com.INVALID > > wrote: > > > >> Thanks all for your input. The consensus is to go forward with this > ticket. > >> Dinesh > >> > >>On Friday, February 15, 2019, 12:54:20 PM PST, Sumanth Pasupuleti < > >> spasupul...@netflix.com.INVALID> wrote: > >> > >> +1 > >> > >>> On Fri, Feb 15, 2019 at 12:14 PM Dikang Gu wrote: > >>> > >>> +1 > >>> > >>> On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella > > >>> wrote: > >>> > We have been using Zstd compressor across different products/services > >>> here > and have seen significant improvements, getting this in 4.0 would be a > >>> big > win. > > +1 > > Thanks, > Vinay Chella > > > > On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa > wrote: > > > > +1 > > > > -- > > Jeff Jirsa > > > > > >> On Feb 15, 2019, at 9:35 AM, Jonathan Ellis > >>> wrote: > >> > >> IMO "add a new compression class that has demonstrable benefits to > Sushma > >> and Joseph" is sufficiently noninvasive that we should allow it > >> into > 4.0. > >> > >> On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi > >> wrote: > >> > >>> Hey folks, > >>> > >>> Just wanted to get a pulse on whether we can proceed with ZStd > support. > >>> The consensus on the ticket was that it’s a very valuable addition > > without > >>> any risk of destabilizing 4.0. It’s ready to go if there aren’t > >> any > >>> objections. > >>> > >>> Dinesh > >>> > >>> > >>> - > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org > >>> > >>> > >> > >> -- > >> Jonathan Ellis > >> co-founder, http://www.datastax.com > >> @spyced > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > >>> > >>> > >>> -- > >>> Dikang > >>> > > > > > > > > -- > > Jon Haddad > > http://www.rustyrazorblade.com > > twitter: rustyrazorblade > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade
Re: CASSANDRA-14482
Thanks for testing this out Jon. Apart from the compression ratio could you also test the insertion rate and GC? 14482 allows you to use full range of Zstd compression levels. Dinesh > On Mar 1, 2019, at 6:41 PM, Jonathan Haddad wrote: > > Hey all, > > I finally got around to doing some testing. Nothing too crazy, I had it > run on my laptop while I did other things around the house. > > Test 1: Inserting Random Data in a K/V table, 10 million inserts > > LZ4 compression rate: 0.909857609644112 > ZStd: 0.6136099401596449 > > Test 2: Inserting fairly compressable text data into K/V table: 20 million > inserts > LZ4: 0.6950432247957942 > ZStd: 0.4797311577845362 > > Definitely worth it at first glance. More testing to come. > > Jon > > > > > On Sun, Feb 17, 2019 at 8:46 PM dinesh.jo...@yahoo.com.INVALID > wrote: > >> Thanks all for your input. The consensus is to go forward with this ticket. >> Dinesh >> >>On Friday, February 15, 2019, 12:54:20 PM PST, Sumanth Pasupuleti < >> spasupul...@netflix.com.INVALID> wrote: >> >> +1 >> >>> On Fri, Feb 15, 2019 at 12:14 PM Dikang Gu wrote: >>> >>> +1 >>> >>> On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella >>> wrote: >>> We have been using Zstd compressor across different products/services >>> here and have seen significant improvements, getting this in 4.0 would be a >>> big win. +1 Thanks, Vinay Chella > On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa wrote: > > +1 > > -- > Jeff Jirsa > > >> On Feb 15, 2019, at 9:35 AM, Jonathan Ellis >>> wrote: >> >> IMO "add a new compression class that has demonstrable benefits to Sushma >> and Joseph" is sufficiently noninvasive that we should allow it >> into 4.0. >> >> On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi >> wrote: >> >>> Hey folks, >>> >>> Just wanted to get a pulse on whether we can proceed with ZStd support. >>> The consensus on the ticket was that it’s a very valuable addition > without >>> any risk of destabilizing 4.0. It’s ready to go if there aren’t >> any >>> objections. >>> >>> Dinesh >>> >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: dev-h...@cassandra.apache.org >>> >>> >> >> -- >> Jonathan Ellis >> co-founder, http://www.datastax.com >> @spyced > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > >>> >>> >>> -- >>> Dikang >>> > > > > -- > Jon Haddad > http://www.rustyrazorblade.com > twitter: rustyrazorblade - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: CASSANDRA-14482
Hey all, I finally got around to doing some testing. Nothing too crazy, I had it run on my laptop while I did other things around the house. Test 1: Inserting Random Data in a K/V table, 10 million inserts LZ4 compression rate: 0.909857609644112 ZStd: 0.6136099401596449 Test 2: Inserting fairly compressable text data into K/V table: 20 million inserts LZ4: 0.6950432247957942 ZStd: 0.4797311577845362 Definitely worth it at first glance. More testing to come. Jon On Sun, Feb 17, 2019 at 8:46 PM dinesh.jo...@yahoo.com.INVALID wrote: > Thanks all for your input. The consensus is to go forward with this ticket. > Dinesh > > On Friday, February 15, 2019, 12:54:20 PM PST, Sumanth Pasupuleti < > spasupul...@netflix.com.INVALID> wrote: > > +1 > > On Fri, Feb 15, 2019 at 12:14 PM Dikang Gu wrote: > > > +1 > > > > On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella > > wrote: > > > > > We have been using Zstd compressor across different products/services > > here > > > and have seen significant improvements, getting this in 4.0 would be a > > big > > > win. > > > > > > +1 > > > > > > Thanks, > > > Vinay Chella > > > > > > > > > On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa wrote: > > > > > > > +1 > > > > > > > > -- > > > > Jeff Jirsa > > > > > > > > > > > > > On Feb 15, 2019, at 9:35 AM, Jonathan Ellis > > wrote: > > > > > > > > > > IMO "add a new compression class that has demonstrable benefits to > > > Sushma > > > > > and Joseph" is sufficiently noninvasive that we should allow it > into > > > 4.0. > > > > > > > > > > On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi > > > > > wrote: > > > > > > > > > >> Hey folks, > > > > >> > > > > >> Just wanted to get a pulse on whether we can proceed with ZStd > > > support. > > > > >> The consensus on the ticket was that it’s a very valuable addition > > > > without > > > > >> any risk of destabilizing 4.0. It’s ready to go if there aren’t > any > > > > >> objections. > > > > >> > > > > >> Dinesh > > > > >> > > > > >> > > - > > > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > >> > > > > >> > > > > > > > > > > -- > > > > > Jonathan Ellis > > > > > co-founder, http://www.datastax.com > > > > > @spyced > > > > > > > > - > > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > > > > > > > > > > -- > > Dikang > > -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade
Re: CASSANDRA-14482
Thanks all for your input. The consensus is to go forward with this ticket. Dinesh On Friday, February 15, 2019, 12:54:20 PM PST, Sumanth Pasupuleti wrote: +1 On Fri, Feb 15, 2019 at 12:14 PM Dikang Gu wrote: > +1 > > On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella > wrote: > > > We have been using Zstd compressor across different products/services > here > > and have seen significant improvements, getting this in 4.0 would be a > big > > win. > > > > +1 > > > > Thanks, > > Vinay Chella > > > > > > On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa wrote: > > > > > +1 > > > > > > -- > > > Jeff Jirsa > > > > > > > > > > On Feb 15, 2019, at 9:35 AM, Jonathan Ellis > wrote: > > > > > > > > IMO "add a new compression class that has demonstrable benefits to > > Sushma > > > > and Joseph" is sufficiently noninvasive that we should allow it into > > 4.0. > > > > > > > > On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi > > > > wrote: > > > > > > > >> Hey folks, > > > >> > > > >> Just wanted to get a pulse on whether we can proceed with ZStd > > support. > > > >> The consensus on the ticket was that it’s a very valuable addition > > > without > > > >> any risk of destabilizing 4.0. It’s ready to go if there aren’t any > > > >> objections. > > > >> > > > >> Dinesh > > > >> > > > >> > - > > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > > > >> > > > >> > > > > > > > > -- > > > > Jonathan Ellis > > > > co-founder, http://www.datastax.com > > > > @spyced > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > > > > -- > Dikang >
Re: CASSANDRA-14482
+1 On Fri, Feb 15, 2019 at 12:14 PM Dikang Gu wrote: > +1 > > On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella > wrote: > > > We have been using Zstd compressor across different products/services > here > > and have seen significant improvements, getting this in 4.0 would be a > big > > win. > > > > +1 > > > > Thanks, > > Vinay Chella > > > > > > On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa wrote: > > > > > +1 > > > > > > -- > > > Jeff Jirsa > > > > > > > > > > On Feb 15, 2019, at 9:35 AM, Jonathan Ellis > wrote: > > > > > > > > IMO "add a new compression class that has demonstrable benefits to > > Sushma > > > > and Joseph" is sufficiently noninvasive that we should allow it into > > 4.0. > > > > > > > > On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi > > > > wrote: > > > > > > > >> Hey folks, > > > >> > > > >> Just wanted to get a pulse on whether we can proceed with ZStd > > support. > > > >> The consensus on the ticket was that it’s a very valuable addition > > > without > > > >> any risk of destabilizing 4.0. It’s ready to go if there aren’t any > > > >> objections. > > > >> > > > >> Dinesh > > > >> > > > >> > - > > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > > > >> > > > >> > > > > > > > > -- > > > > Jonathan Ellis > > > > co-founder, http://www.datastax.com > > > > @spyced > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > > > > -- > Dikang >
Re: CASSANDRA-14482
+1 On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella wrote: > We have been using Zstd compressor across different products/services here > and have seen significant improvements, getting this in 4.0 would be a big > win. > > +1 > > Thanks, > Vinay Chella > > > On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa wrote: > > > +1 > > > > -- > > Jeff Jirsa > > > > > > > On Feb 15, 2019, at 9:35 AM, Jonathan Ellis wrote: > > > > > > IMO "add a new compression class that has demonstrable benefits to > Sushma > > > and Joseph" is sufficiently noninvasive that we should allow it into > 4.0. > > > > > > On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi > > > wrote: > > > > > >> Hey folks, > > >> > > >> Just wanted to get a pulse on whether we can proceed with ZStd > support. > > >> The consensus on the ticket was that it’s a very valuable addition > > without > > >> any risk of destabilizing 4.0. It’s ready to go if there aren’t any > > >> objections. > > >> > > >> Dinesh > > >> > > >> - > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > > >> > > >> > > > > > > -- > > > Jonathan Ellis > > > co-founder, http://www.datastax.com > > > @spyced > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > -- Dikang
Re: CASSANDRA-14482
We have been using Zstd compressor across different products/services here and have seen significant improvements, getting this in 4.0 would be a big win. +1 Thanks, Vinay Chella On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa wrote: > +1 > > -- > Jeff Jirsa > > > > On Feb 15, 2019, at 9:35 AM, Jonathan Ellis wrote: > > > > IMO "add a new compression class that has demonstrable benefits to Sushma > > and Joseph" is sufficiently noninvasive that we should allow it into 4.0. > > > > On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi > > wrote: > > > >> Hey folks, > >> > >> Just wanted to get a pulse on whether we can proceed with ZStd support. > >> The consensus on the ticket was that it’s a very valuable addition > without > >> any risk of destabilizing 4.0. It’s ready to go if there aren’t any > >> objections. > >> > >> Dinesh > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> > > > > -- > > Jonathan Ellis > > co-founder, http://www.datastax.com > > @spyced > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: CASSANDRA-14482
Thanks Ariel & Jonathan. Michael, I have addressed the comments in the issue so we're using the latest jar now. I've posted the updated patch in the latest comment of the ticket. In addition there is a JMH performance benchmark in the C* repo (CompressorPerformance) that tests various Compressors. Depending on the `compression_level` setting we can see a boost in compression speed. However, the major win with Zstd is not only the speed but the superior compression ratio. While lz4 can achieve about 2.101, Zstd can achieve a ratio of 2.877 which is a huge difference. The comparisons are here - https://github.com/facebook/zstd Dinesh On Friday, February 15, 2019, 9:56:49 AM PST, Michael Shuler wrote: +0.5 I skimmed the jira and github diff and a few things came to mind: - There are multiple comments about using an older jar than the latest version. - I did not see any performance test results to form an opinion on any gains/caveats as a user. This was the first thing I looked for. - I did not see any conf/cassandra.yaml comment in the diff for the valid class_name configuration option to use. Seems interesting, and there are comments about 4.0 being in a freeze and all, but OK on it being non-default. -- Michael On 2/15/19 11:43 AM, Ariel Weisberg wrote: > Hi, > > I am +1 since it's an additional compressor and not the default. > > Ariel > > On Fri, Feb 15, 2019, at 11:41 AM, Dinesh Joshi wrote: >> Hey folks, >> >> Just wanted to get a pulse on whether we can proceed with ZStd support. >> The consensus on the ticket was that it’s a very valuable addition >> without any risk of destabilizing 4.0. It’s ready to go if there aren’t >> any objections. >> >> Dinesh >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: CASSANDRA-14482
Seems low risk, potentially high reward. I can run some tests next week to get a rough idea of how compression ratios differ as well as if there's a difference in performance. I won't be testing correctness, just looking at the performance profile. Jon On Fri, Feb 15, 2019 at 9:56 AM Michael Shuler wrote: > +0.5 > > I skimmed the jira and github diff and a few things came to mind: > - There are multiple comments about using an older jar than the latest > version. > - I did not see any performance test results to form an opinion on any > gains/caveats as a user. This was the first thing I looked for. > - I did not see any conf/cassandra.yaml comment in the diff for the > valid class_name configuration option to use. > > Seems interesting, and there are comments about 4.0 being in a freeze > and all, but OK on it being non-default. > > -- > Michael > > On 2/15/19 11:43 AM, Ariel Weisberg wrote: > > Hi, > > > > I am +1 since it's an additional compressor and not the default. > > > > Ariel > > > > On Fri, Feb 15, 2019, at 11:41 AM, Dinesh Joshi wrote: > >> Hey folks, > >> > >> Just wanted to get a pulse on whether we can proceed with ZStd support. > >> The consensus on the ticket was that it’s a very valuable addition > >> without any risk of destabilizing 4.0. It’s ready to go if there aren’t > >> any objections. > >> > >> Dinesh > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade
Re: CASSANDRA-14482
+0.5 I skimmed the jira and github diff and a few things came to mind: - There are multiple comments about using an older jar than the latest version. - I did not see any performance test results to form an opinion on any gains/caveats as a user. This was the first thing I looked for. - I did not see any conf/cassandra.yaml comment in the diff for the valid class_name configuration option to use. Seems interesting, and there are comments about 4.0 being in a freeze and all, but OK on it being non-default. -- Michael On 2/15/19 11:43 AM, Ariel Weisberg wrote: > Hi, > > I am +1 since it's an additional compressor and not the default. > > Ariel > > On Fri, Feb 15, 2019, at 11:41 AM, Dinesh Joshi wrote: >> Hey folks, >> >> Just wanted to get a pulse on whether we can proceed with ZStd support. >> The consensus on the ticket was that it’s a very valuable addition >> without any risk of destabilizing 4.0. It’s ready to go if there aren’t >> any objections. >> >> Dinesh >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: CASSANDRA-14482
Hi, I am +1 since it's an additional compressor and not the default. Ariel On Fri, Feb 15, 2019, at 11:41 AM, Dinesh Joshi wrote: > Hey folks, > > Just wanted to get a pulse on whether we can proceed with ZStd support. > The consensus on the ticket was that it’s a very valuable addition > without any risk of destabilizing 4.0. It’s ready to go if there aren’t > any objections. > > Dinesh > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: CASSANDRA-14482
IMO "add a new compression class that has demonstrable benefits to Sushma and Joseph" is sufficiently noninvasive that we should allow it into 4.0. On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi wrote: > Hey folks, > > Just wanted to get a pulse on whether we can proceed with ZStd support. > The consensus on the ticket was that it’s a very valuable addition without > any risk of destabilizing 4.0. It’s ready to go if there aren’t any > objections. > > Dinesh > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > -- Jonathan Ellis co-founder, http://www.datastax.com @spyced
