Re: [VOTE] (Take 2) Release Apache Cassandra 3.0.16
Tests are green on our side. +1 Regards, Vinay Chell a On Thu, Feb 15, 2018 at 4:30 PM, Nate McCallwrote: > Thanks, Jason!! > > On Fri, Feb 16, 2018 at 1:18 PM, Jason Brown wrote: > > Nate, I can do this. > > > > On Thu, Feb 15, 2018 at 2:51 PM, Nate McCall wrote: > > > >> > > >> > My circleci test runs fail on lack of resources and I understand now > >> > that Jason used the circleci configuration from the trunk branch to > run > >> > the 3.0 and 3.11 branches. These branches should get commits for a > >> > working CI configuration in-tree. > >> > > >> > >> Quick follow up - does someone have an issue/action item on updating > these: > >> > >> https://github.com/apache/cassandra/blob/cassandra-3.0/circle.yml > >> https://github.com/apache/cassandra/blob/cassandra-3.11/circle.yml > >> > >> Still showing as out of date(?). > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
RE: Release votes
I'd like to help as well. For me the issue is I have only my time to contribute. The resources to test Cassandra extensively are beyond that of most individuals including me, aren't they? If resources are made available and I can help, count me in. Also, perhaps having a standard reference like Slender Cassandra (18 nodes total, two regions, three AZ's total, six nodes per AZ) would help. I'll have it done very soon. Kenneth Brotman -Original Message- From: kurt greaves [mailto:k...@instaclustr.com] Sent: Thursday, February 15, 2018 3:48 PM To: dev@cassandra.apache.org Subject: Re: Release votes It seems there has been a bit of a slip in testing as of recently, mostly due to the fact that there's no canonical testing environment that isn't flaky. We probably need to come up with some ideas and a plan on how we're going to do testing in the future, and how we're going to make testing accessible for all contributors. I think this is the only way we're really going to change behaviour. Having an incredibly tedious process and then being aggressive about it only leads to resentment and workarounds. I'm completely unsure of where dtests are at since the conversion to pytest, and there's a lot of failing dtests on the ASF jenkins jobs (which appear to be running pytest). As there's currently not a lot of visibility into what people are doing with CircleCI for this it's hard to say if things are better over there. I'd like to help here if anyone wants to fill me in. On 15 February 2018 at 21:14, Josh McKenziewrote: > > > > We’ve said in the past that we don’t release without green tests. > > The PMC gets to vote and enforce it. If you don’t vote yes without > > seeing the > test > > results, that enforces it. > > I think this is noble and ideal in theory. In practice, the tests take > long enough, hardware infra has proven flaky enough, and the tests > *themselves* flaky enough, that there's been a consistent low-level of > test failure noise that makes separating signal from noise in this > context very time consuming. Reference 3.11-test-all for example re:noise: > https://builds.apache.org/view/A-D/view/Cassandra/job/ > Cassandra-3.11-test-all/test/?width=1024=768 > > Having spearheaded burning test failures to 0 multiple times and have > them regress over time, my gut intuition is we should have one person > as our Source of Truth with a less-flaky source for release-vetting CI > (dedicated hardware, circle account, etc) we can use as a reference to > vote on release SHA's. > > We’ve declared this a requirement multiple times > > Declaring things != changed behavior, and thus != changed culture. The > culture on this project is one of having a constant low level of test > failure noise in our CI as a product of our working processes. Unless > we change those (actually block release w/out green board, actually > aggressively block merge w/any failing tests, aggressively > retroactively track down test failures on a daily basis and RCA), the > situation won't improve. Given that this is a volunteer organization / > project, that kind of daily time investment is a big ask. > > On Thu, Feb 15, 2018 at 1:10 PM, Jeff Jirsa wrote: > > > Moving this to it’s own thread: > > > > We’ve declared this a requirement multiple times and then we > > occasionally get a critical issue and have to decide whether it’s > > worth the delay. I assume Jason’s earlier -1 on attempt 1 was an > > enforcement of that earlier stated goal. > > > > It’s up to the PMC. We’ve said in the past that we don’t release > > without green tests. The PMC gets to vote and enforce it. If you > > don’t vote yes without seeing the test results, that enforces it. > > > > -- > > Jeff Jirsa > > > > > > > On Feb 15, 2018, at 9:49 AM, Josh McKenzie > wrote: > > > > > > What would it take for us to get green utest/dtests as a blocking > > > part > of > > > the release process? i.e. "for any given SHA, here's a link to the > tests > > > that passed" in the release vote email? > > > > > > That being said, +1. > > > > > >> On Wed, Feb 14, 2018 at 4:33 PM, Nate McCall > > wrote: > > >> > > >> +1 > > >> > > >> On Thu, Feb 15, 2018 at 9:40 AM, Michael Shuler < > mich...@pbandjelly.org > > > > > >> wrote: > > >>> I propose the following artifacts for release as 3.0.16. > > >>> > > >>> sha1: 890f319142ddd3cf2692ff45ff28e71001365e96 > > >>> Git: > > >>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= > > >> shortlog;h=refs/tags/3.0.16-tentative > > >>> Artifacts: > > >>> https://repository.apache.org/content/repositories/ > > >> orgapachecassandra-1157/org/apache/cassandra/apache-cassandra/3.0 > > >> .16/ > > >>> Staging repository: > > >>> https://repository.apache.org/content/repositories/ > > >> orgapachecassandra-1157/ > > >>> > > >>> Debian and RPM packages are available here: > > >>>
Re: [VOTE] (Take 2) Release Apache Cassandra 3.0.16
Thanks, Jason!! On Fri, Feb 16, 2018 at 1:18 PM, Jason Brownwrote: > Nate, I can do this. > > On Thu, Feb 15, 2018 at 2:51 PM, Nate McCall wrote: > >> > >> > My circleci test runs fail on lack of resources and I understand now >> > that Jason used the circleci configuration from the trunk branch to run >> > the 3.0 and 3.11 branches. These branches should get commits for a >> > working CI configuration in-tree. >> > >> >> Quick follow up - does someone have an issue/action item on updating these: >> >> https://github.com/apache/cassandra/blob/cassandra-3.0/circle.yml >> https://github.com/apache/cassandra/blob/cassandra-3.11/circle.yml >> >> Still showing as out of date(?). >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [VOTE] (Take 2) Release Apache Cassandra 3.0.16
Nate, I can do this. On Thu, Feb 15, 2018 at 2:51 PM, Nate McCallwrote: > > > > My circleci test runs fail on lack of resources and I understand now > > that Jason used the circleci configuration from the trunk branch to run > > the 3.0 and 3.11 branches. These branches should get commits for a > > working CI configuration in-tree. > > > > Quick follow up - does someone have an issue/action item on updating these: > > https://github.com/apache/cassandra/blob/cassandra-3.0/circle.yml > https://github.com/apache/cassandra/blob/cassandra-3.11/circle.yml > > Still showing as out of date(?). > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: Release votes
It seems there has been a bit of a slip in testing as of recently, mostly due to the fact that there's no canonical testing environment that isn't flaky. We probably need to come up with some ideas and a plan on how we're going to do testing in the future, and how we're going to make testing accessible for all contributors. I think this is the only way we're really going to change behaviour. Having an incredibly tedious process and then being aggressive about it only leads to resentment and workarounds. I'm completely unsure of where dtests are at since the conversion to pytest, and there's a lot of failing dtests on the ASF jenkins jobs (which appear to be running pytest). As there's currently not a lot of visibility into what people are doing with CircleCI for this it's hard to say if things are better over there. I'd like to help here if anyone wants to fill me in. On 15 February 2018 at 21:14, Josh McKenziewrote: > > > > We’ve said in the past that we don’t release without green tests. The PMC > > gets to vote and enforce it. If you don’t vote yes without seeing the > test > > results, that enforces it. > > I think this is noble and ideal in theory. In practice, the tests take long > enough, hardware infra has proven flaky enough, and the tests *themselves* > flaky enough, that there's been a consistent low-level of test failure > noise that makes separating signal from noise in this context very time > consuming. Reference 3.11-test-all for example re:noise: > https://builds.apache.org/view/A-D/view/Cassandra/job/ > Cassandra-3.11-test-all/test/?width=1024=768 > > Having spearheaded burning test failures to 0 multiple times and have them > regress over time, my gut intuition is we should have one person as our > Source of Truth with a less-flaky source for release-vetting CI (dedicated > hardware, circle account, etc) we can use as a reference to vote on release > SHA's. > > We’ve declared this a requirement multiple times > > Declaring things != changed behavior, and thus != changed culture. The > culture on this project is one of having a constant low level of test > failure noise in our CI as a product of our working processes. Unless we > change those (actually block release w/out green board, actually > aggressively block merge w/any failing tests, aggressively retroactively > track down test failures on a daily basis and RCA), the situation won't > improve. Given that this is a volunteer organization / project, that kind > of daily time investment is a big ask. > > On Thu, Feb 15, 2018 at 1:10 PM, Jeff Jirsa wrote: > > > Moving this to it’s own thread: > > > > We’ve declared this a requirement multiple times and then we occasionally > > get a critical issue and have to decide whether it’s worth the delay. I > > assume Jason’s earlier -1 on attempt 1 was an enforcement of that earlier > > stated goal. > > > > It’s up to the PMC. We’ve said in the past that we don’t release without > > green tests. The PMC gets to vote and enforce it. If you don’t vote yes > > without seeing the test results, that enforces it. > > > > -- > > Jeff Jirsa > > > > > > > On Feb 15, 2018, at 9:49 AM, Josh McKenzie > wrote: > > > > > > What would it take for us to get green utest/dtests as a blocking part > of > > > the release process? i.e. "for any given SHA, here's a link to the > tests > > > that passed" in the release vote email? > > > > > > That being said, +1. > > > > > >> On Wed, Feb 14, 2018 at 4:33 PM, Nate McCall > > wrote: > > >> > > >> +1 > > >> > > >> On Thu, Feb 15, 2018 at 9:40 AM, Michael Shuler < > mich...@pbandjelly.org > > > > > >> wrote: > > >>> I propose the following artifacts for release as 3.0.16. > > >>> > > >>> sha1: 890f319142ddd3cf2692ff45ff28e71001365e96 > > >>> Git: > > >>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= > > >> shortlog;h=refs/tags/3.0.16-tentative > > >>> Artifacts: > > >>> https://repository.apache.org/content/repositories/ > > >> orgapachecassandra-1157/org/apache/cassandra/apache-cassandra/3.0.16/ > > >>> Staging repository: > > >>> https://repository.apache.org/content/repositories/ > > >> orgapachecassandra-1157/ > > >>> > > >>> Debian and RPM packages are available here: > > >>> http://people.apache.org/~mshuler > > >>> > > >>> *** This release addresses an important fix for CASSANDRA-14092 *** > > >>>"Max ttl of 20 years will overflow localDeletionTime" > > >>>https://issues.apache.org/jira/browse/CASSANDRA-14092 > > >>> > > >>> The vote will be open for 72 hours (longer if needed). > > >>> > > >>> [1]: (CHANGES.txt) https://goo.gl/rLj59Z > > >>> [2]: (NEWS.txt) https://goo.gl/EkrT4G > > >>> > > >>> > - > > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org > > >>> > > >> > > >>
Re: [VOTE] (Take 2) Release Apache Cassandra 3.0.16
> > My circleci test runs fail on lack of resources and I understand now > that Jason used the circleci configuration from the trunk branch to run > the 3.0 and 3.11 branches. These branches should get commits for a > working CI configuration in-tree. > Quick follow up - does someone have an issue/action item on updating these: https://github.com/apache/cassandra/blob/cassandra-3.0/circle.yml https://github.com/apache/cassandra/blob/cassandra-3.11/circle.yml Still showing as out of date(?). - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Release votes
> > We’ve said in the past that we don’t release without green tests. The PMC > gets to vote and enforce it. If you don’t vote yes without seeing the test > results, that enforces it. I think this is noble and ideal in theory. In practice, the tests take long enough, hardware infra has proven flaky enough, and the tests *themselves* flaky enough, that there's been a consistent low-level of test failure noise that makes separating signal from noise in this context very time consuming. Reference 3.11-test-all for example re:noise: https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-3.11-test-all/test/?width=1024=768 Having spearheaded burning test failures to 0 multiple times and have them regress over time, my gut intuition is we should have one person as our Source of Truth with a less-flaky source for release-vetting CI (dedicated hardware, circle account, etc) we can use as a reference to vote on release SHA's. We’ve declared this a requirement multiple times Declaring things != changed behavior, and thus != changed culture. The culture on this project is one of having a constant low level of test failure noise in our CI as a product of our working processes. Unless we change those (actually block release w/out green board, actually aggressively block merge w/any failing tests, aggressively retroactively track down test failures on a daily basis and RCA), the situation won't improve. Given that this is a volunteer organization / project, that kind of daily time investment is a big ask. On Thu, Feb 15, 2018 at 1:10 PM, Jeff Jirsawrote: > Moving this to it’s own thread: > > We’ve declared this a requirement multiple times and then we occasionally > get a critical issue and have to decide whether it’s worth the delay. I > assume Jason’s earlier -1 on attempt 1 was an enforcement of that earlier > stated goal. > > It’s up to the PMC. We’ve said in the past that we don’t release without > green tests. The PMC gets to vote and enforce it. If you don’t vote yes > without seeing the test results, that enforces it. > > -- > Jeff Jirsa > > > > On Feb 15, 2018, at 9:49 AM, Josh McKenzie wrote: > > > > What would it take for us to get green utest/dtests as a blocking part of > > the release process? i.e. "for any given SHA, here's a link to the tests > > that passed" in the release vote email? > > > > That being said, +1. > > > >> On Wed, Feb 14, 2018 at 4:33 PM, Nate McCall > wrote: > >> > >> +1 > >> > >> On Thu, Feb 15, 2018 at 9:40 AM, Michael Shuler > > >> wrote: > >>> I propose the following artifacts for release as 3.0.16. > >>> > >>> sha1: 890f319142ddd3cf2692ff45ff28e71001365e96 > >>> Git: > >>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= > >> shortlog;h=refs/tags/3.0.16-tentative > >>> Artifacts: > >>> https://repository.apache.org/content/repositories/ > >> orgapachecassandra-1157/org/apache/cassandra/apache-cassandra/3.0.16/ > >>> Staging repository: > >>> https://repository.apache.org/content/repositories/ > >> orgapachecassandra-1157/ > >>> > >>> Debian and RPM packages are available here: > >>> http://people.apache.org/~mshuler > >>> > >>> *** This release addresses an important fix for CASSANDRA-14092 *** > >>>"Max ttl of 20 years will overflow localDeletionTime" > >>>https://issues.apache.org/jira/browse/CASSANDRA-14092 > >>> > >>> The vote will be open for 72 hours (longer if needed). > >>> > >>> [1]: (CHANGES.txt) https://goo.gl/rLj59Z > >>> [2]: (NEWS.txt) https://goo.gl/EkrT4G > >>> > >>> - > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org > >>> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Release votes
Moving this to it’s own thread: We’ve declared this a requirement multiple times and then we occasionally get a critical issue and have to decide whether it’s worth the delay. I assume Jason’s earlier -1 on attempt 1 was an enforcement of that earlier stated goal. It’s up to the PMC. We’ve said in the past that we don’t release without green tests. The PMC gets to vote and enforce it. If you don’t vote yes without seeing the test results, that enforces it. -- Jeff Jirsa > On Feb 15, 2018, at 9:49 AM, Josh McKenziewrote: > > What would it take for us to get green utest/dtests as a blocking part of > the release process? i.e. "for any given SHA, here's a link to the tests > that passed" in the release vote email? > > That being said, +1. > >> On Wed, Feb 14, 2018 at 4:33 PM, Nate McCall wrote: >> >> +1 >> >> On Thu, Feb 15, 2018 at 9:40 AM, Michael Shuler >> wrote: >>> I propose the following artifacts for release as 3.0.16. >>> >>> sha1: 890f319142ddd3cf2692ff45ff28e71001365e96 >>> Git: >>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= >> shortlog;h=refs/tags/3.0.16-tentative >>> Artifacts: >>> https://repository.apache.org/content/repositories/ >> orgapachecassandra-1157/org/apache/cassandra/apache-cassandra/3.0.16/ >>> Staging repository: >>> https://repository.apache.org/content/repositories/ >> orgapachecassandra-1157/ >>> >>> Debian and RPM packages are available here: >>> http://people.apache.org/~mshuler >>> >>> *** This release addresses an important fix for CASSANDRA-14092 *** >>>"Max ttl of 20 years will overflow localDeletionTime" >>>https://issues.apache.org/jira/browse/CASSANDRA-14092 >>> >>> The vote will be open for 72 hours (longer if needed). >>> >>> [1]: (CHANGES.txt) https://goo.gl/rLj59Z >>> [2]: (NEWS.txt) https://goo.gl/EkrT4G >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: dev-h...@cassandra.apache.org >>> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [VOTE] (Take 2) Release Apache Cassandra 3.0.16
On 02/15/2018 11:49 AM, Josh McKenzie wrote: > What would it take for us to get green utest/dtests as a blocking part of > the release process? i.e. "for any given SHA, here's a link to the tests > that passed" in the release vote email? Jason graciously linked to his passing test-all runs for this sha and for the 3.11.2 take 3 sha. Super appreciated. My circleci test runs fail on lack of resources and I understand now that Jason used the circleci configuration from the trunk branch to run the 3.0 and 3.11 branches. These branches should get commits for a working CI configuration in-tree. Basically, as it stands, I have no public link to post for reliable test results, although I can list a summary of private results. The feedback from the votes this week actually shows to me that a group effort "I ran here, this failed, let's fix that, etc." is super valuable. I could do it all myself for releases, but then I'm doing it all.. I'm all for voting with your test results and we rally around fixing the last few things together. This was rather enjoyable, even though it took a few tries, but there are multiple people running tests and evaluating the results. Win. -- Michael - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [VOTE] (Take 3) Release Apache Cassandra 3.11.2
+1 On Wed, Feb 14, 2018 at 4:45 PM, Jason Brownwrote: > I ran the unit tests and they are green: > https://circleci.com/gh/jasobrown/workflows/cassandra/ > tree/3.11.2-candidate-2 > > +1. Thanks, Michael. > > On Wed, Feb 14, 2018 at 1:34 PM, Nate McCall wrote: > > > +1 > > > > On Thu, Feb 15, 2018 at 10:09 AM, Michael Shuler > > > wrote: > > > I propose the following artifacts for release as 3.11.2. > > > > > > sha1: 1d506f9d09c880ff2b2693e3e27fa58c02ecf398 > > > Git: > > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= > > shortlog;h=refs/tags/3.11.2-tentative > > > Artifacts: > > > https://repository.apache.org/content/repositories/ > > orgapachecassandra-1158/org/apache/cassandra/apache-cassandra/3.11.2/ > > > Staging repository: > > > https://repository.apache.org/content/repositories/ > > orgapachecassandra-1158/ > > > > > > Debian and RPM packages are available here: > > > http://people.apache.org/~mshuler > > > > > > *** This release addresses an important fix for CASSANDRA-14092 *** > > > "Max ttl of 20 years will overflow localDeletionTime" > > > https://issues.apache.org/jira/browse/CASSANDRA-14092 > > > > > > The vote will be open for 72 hours (longer if needed). > > > > > > [1]: (CHANGES.txt) https://goo.gl/RLZLrR > > > [2]: (NEWS.txt) https://goo.gl/kpnVHp > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > >
Re: [VOTE] (Take 2) Release Apache Cassandra 3.0.16
What would it take for us to get green utest/dtests as a blocking part of the release process? i.e. "for any given SHA, here's a link to the tests that passed" in the release vote email? That being said, +1. On Wed, Feb 14, 2018 at 4:33 PM, Nate McCallwrote: > +1 > > On Thu, Feb 15, 2018 at 9:40 AM, Michael Shuler > wrote: > > I propose the following artifacts for release as 3.0.16. > > > > sha1: 890f319142ddd3cf2692ff45ff28e71001365e96 > > Git: > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= > shortlog;h=refs/tags/3.0.16-tentative > > Artifacts: > > https://repository.apache.org/content/repositories/ > orgapachecassandra-1157/org/apache/cassandra/apache-cassandra/3.0.16/ > > Staging repository: > > https://repository.apache.org/content/repositories/ > orgapachecassandra-1157/ > > > > Debian and RPM packages are available here: > > http://people.apache.org/~mshuler > > > > *** This release addresses an important fix for CASSANDRA-14092 *** > > "Max ttl of 20 years will overflow localDeletionTime" > > https://issues.apache.org/jira/browse/CASSANDRA-14092 > > > > The vote will be open for 72 hours (longer if needed). > > > > [1]: (CHANGES.txt) https://goo.gl/rLj59Z > > [2]: (NEWS.txt) https://goo.gl/EkrT4G > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: row tombstones as a separate sstable citizen
Worth a JIRA, yes On Wed, Feb 14, 2018 at 9:45 AM, Carl Muellerwrote: > So is this at least a decent candidate for a feature request ticket? > > > On Tue, Feb 13, 2018 at 8:09 PM, Carl Mueller < > carl.muel...@smartthings.com> > wrote: > > > I'm particularly interested in getting the tombstones to "promote" up the > > levels of LCS more quickly. Currently they get attached at the low level > > and don't propagate up to higher levels until enough activity at a lower > > level promotes the data. Meanwhile, LCS means compactions can occur in > > parallel at each level. So row tombstones in their own sstable could be > up > > promoted the LCS levels preferentially before normal processes would move > > them up. > > > > So if the delete-only sstables could move up more quickly, the compaction > > at the levels would happen more quickly. > > > > The threshold stuff is nice if I read 7019 correctly, but what is the % > > there? % of rows? % of columns? or % of the size of the sstable? Row > > tombstones are pretty compact being just the rowkey and the tombstone > > marker. So if 7019 is triggered at 10% of the sstable size, even a > crapton > > of tombstones deleting practially the entire database would only be a > small > > % size of the sstable. > > > > Since the row tombstones are so compact, that's why I think they are good > > candidates for special handling. > > > > On Tue, Feb 13, 2018 at 5:22 PM, J. D. Jordan > > > wrote: > > > >> Have you taken a look at the new stuff introduced by > >> https://issues.apache.org/jira/browse/CASSANDRA-7019 ? I think it may > >> go a ways to reducing the need for something complicated like this. > >> Though it is an interesting idea as special handling for bulk deletes. > >> If they were truly just sstables that only contained deletes the logic > from > >> 7109 would probably go a long ways. Though if you are bulk inserting > >> deletes that is what you would end up with, so maybe it already works. > >> > >> -Jeremiah > >> > >> > On Feb 13, 2018, at 6:04 PM, Jeff Jirsa wrote: > >> > > >> > On Tue, Feb 13, 2018 at 2:38 PM, Carl Mueller < > >> carl.muel...@smartthings.com> > >> > wrote: > >> > > >> >> In process of doing my second major data purge from a cassandra > system. > >> >> > >> >> Almost all of my purging is done via row tombstones. While performing > >> this > >> >> the second time while trying to cajole compaction to occur (in 2.1.x, > >> >> LevelledCompaction) to goddamn actually compact the data, I've been > >> >> thinking as to why there isn't a separate set of sstable > infrastructure > >> >> setup for row deletion tombstones. > >> >> > >> >> I'm imagining that row tombstones are written to separate sstables > than > >> >> mainline data updates/appends and range/column tombstones. > >> >> > >> >> By writing them to separate sstables, the compaction systems can > >> >> preferentially merge / process them when compacting sstables. > >> >> > >> >> This would create an additional sstable for lookup in the bloom > >> filters, > >> >> granted. I had visions of short circuiting the lookups to other > >> sstables if > >> >> a row tombstone was present in one of the special row tombstone > >> sstables. > >> >> > >> >> > >> > All of the above sounds really interesting to me, but I suspect it's a > >> LOT > >> > of work to make it happen correctly. > >> > > >> > You'd almost end up with 2 sets of logs for the LSM - a tombstone > >> > log/generation, and a data log/generation, and the tombstone logs > would > >> be > >> > read-only inputs to data compactions. > >> > > >> > > >> >> But that would only be possible if there was the notion of a "super > row > >> >> tombstone" that permanently deleted a rowkey and all future writes > >> would be > >> >> invalidated. Kind of like how a tombstone with a mistakenly huge > >> timestamp > >> >> becomes a sneaky permanent tombstone, but intended. There could be a > >> >> special operation / statement to undo this permanent tombstone, and > >> since > >> >> the row tombstones would be in their own dedicated sstables, they > could > >> >> process and compact more quickly, with prioritization by the > compactor. > >> >> > >> >> > >> > This part sounds way less interesting to me (other than the fact you > can > >> > already do this with a timestamp in the future, but it'll gc away at > >> gcgs). > >> > > >> > > >> >> I'm thinking there must be something I am forgetting in the > >> >> read/write/compaction paths that invalidate this. > >> >> > >> > > >> > There are a lot of places where we do "smart" things to make sure we > >> don't > >> > accidentally resurrect data. Read path includes old sstables for > >> tombstones > >> > for example. Those all need to be concretely identified and handled > (and > >> > tested),. > >> > > > > >
Re: [VOTE] Release Apache Cassandra 2.2.12
+1 On 2018-02-13 19:54, Josh McKenzie wrote: +1 On Feb 13, 2018 12:19 PM, "Jason Brown"wrote: +1 On Tue, Feb 13, 2018 at 7:55 AM, Jon Haddad wrote: +1 On Feb 13, 2018, at 7:21 AM, Marcus Eriksson wrote: +1 On Tue, Feb 13, 2018 at 4:18 PM, Gary Dusbabek wrote: +1 On Mon, Feb 12, 2018 at 2:30 PM, Michael Shuler < mich...@pbandjelly.org wrote: I propose the following artifacts for release as 2.2.12. sha1: 1602e606348959aead18531cb8027afb15f276e7 Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= shortlog;h=refs/tags/2.2.12-tentative Artifacts: https://repository.apache.org/content/repositories/ orgapachecassandra-1153/org/apache/cassandra/apache- cassandra/2.2.12/ Staging repository: https://repository.apache.org/content/repositories/ orgapachecassandra-1153/ Debian and RPM packages are available here: http://people.apache.org/~mshuler *** This release addresses an important fix for CASSANDRA-14092 *** "Max ttl of 20 years will overflow localDeletionTime" https://issues.apache.org/jira/browse/CASSANDRA-14092 The vote will be open for 72 hours (longer if needed). [1]: (CHANGES.txt) https://goo.gl/QkJeXH [2]: (NEWS.txt) https://goo.gl/A4iKFb - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org