Cassandra doesn't flush any commit log files into cdc_raw directory
Hi all, We're working on a Kafka connector to capture data changes in Cassandra by processing commit log files in the cdc_raw directory. After we enabled CDC on a few tables, we didn't observe any commit log files getting flushed into cdc_raw directory as expected, but got WriteTimeoutException in Cassandra DB. Here's how we reproduce the issue: 1. Our Cassandra Settings: - Cassandra Version: 3.11.9 - Related configs in Cassandra.yaml: - cdc_enabled: true - cdc_total_space_in_mb: 4096 - commitlog_segment_size_in_mb: 32mb - commitlog_total_space_in_mb: 8192 - commitlog_sync: periodic - commitlog_sync_period_in_ms: 1 2. Enable CDC on a few tables by CQL: ALTER TABLE foo WITH cdc=true; 3. After a few days, we get *WriteTimeoutException* in Cassandra DB. However at the same time, cdc_raw directory is still empty with no commit log flushed/copied into it at all. I want to understand why there's no commit log file flushed into cdc_raw directory at all even when the threshold cdc_total_space_in_mb has been reached and write suspension has been triggered in Cassandra DB. This sounds like a bug and currently makes the CDC feature useless. Thanks so much, Bingqin Zhou
Re: [DISCUSSION] Attracting new contributors
> > Is it possible if a new comer works together with a mentor on an issue? > This way mentor can gradually introduce the newcomer into the codebase, and > newcomer would get timely feedback too. It is complicated unfortunately because most of us have limited bandwidth and we are spread across different time zones. Mentors can provide some plans on how to address a specific issue and answer questions in an asynchronous way but more than that will be difficult in my opinion. Le mer. 28 avr. 2021 à 13:28, Manish G a écrit : > Is it possible if a new comer works together with a mentor on an issue? > This way mentor can gradually introduce the newcomer into the codebase, and > newcomer would get timely feedback too. > > > On Wed, Apr 28, 2021 at 2:51 PM Benedict Elliott Smith < > bened...@apache.org> > wrote: > > > > I believe that it can be a virtuous circle where we produce new > > committers that help mentoring newcomers. > > > > That's the dream, and kudos for keeping it alive! I have become jaded > > about this possibility, after years of trying. > > > > > > On 28/04/2021, 10:18, "Benjamin Lerer" wrote: > > > > > > > > I think there are two main hurdles, one is restoring contributor > > interest > > > in mentoring, and the other is finding newcomers that actually want > > to > > > stick around. > > > > > > I am interested in mentoring new committers to help the project grow > > and > > some of the new committers expressed the same interest to me. I > believe > > that it can be a virtuous circle where we produce new committers that > > help > > mentoring newcomers. > > > > What we need is to be well organized and make sure that we have a > > reasonable response time to newcomers. > > > > Berenguer created this board to help to track newcomers > contributions: > > > > > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=463=2088 > > Apparently Brandon is cheating to appear as a newcomer but we will > > solve > > that. He should be at the Nightmare level ;-) > > > > Le mer. 28 avr. 2021 à 10:54, Benedict Elliott Smith < > > bened...@apache.org> > > a écrit : > > > > > I think there are two main hurdles, one is restoring contributor > > interest > > > in mentoring, and the other is finding newcomers that actually want > > to > > > stick around. These are perhaps two sides of the same coin, though. > > An ugly > > > truth is that it isn't very enjoyable or rewarding to help > newcomers > > when > > > they mostly don't stick around - often even to complete their first > > patch! > > > The patches are mostly uninteresting, the work often done to a low > > > standard, and it is easy to underestimate the amount of time > > involved in > > > every such failed interaction. > > > > > > I think making it easier to contribute and demonstrate a lasting > > interest > > > in the project without the hand holding of long term contributors > may > > > benefit both sides of the equation, as it is more rewarding to help > > > somebody who's demonstrated a persistent interest in the community. > > > > > > > > > On 28/04/2021, 03:24, "Paulo Motta" > > wrote: > > > > > > > There is no great hurdle in finding something to work on, > it's > > > solely finding > > > someone with the knowledge that can help you work on something > > and > > > progress > > > it to commit. > > > > > > I agree the primary challenge is to engage existing > contributors > > to > > > mentor > > > newcomers, but this doesn’t preclude having good documentation > > and a > > > well > > > maintained task pool to allow newcomers to self-serve as much > as > > > possible > > > and reduce the mentoring burden, so these efforts are > > complimentary. > > > > > > For instance, a few students were interested in picking random > > tasks to > > > work on in preparation for Google Summer of Code, but it was > not > > > straightforward for them to find a task to work on because we > > don’t > > > consistently label tickets as “Low Hanging Fruit” and the ones > > that are > > > tagged sometimes don’t have meaningful descriptions making it > > hard for > > > these students to get started on tasks without unnecessarily > > taking > > > some > > > time from the mentor (which could have been saved if the tasks > > were > > > properly described and labeled in the first place). > > > > > > On Tue, 27 Apr 2021 at 22:24 Kane Wilson wrote: > > > > > > > The main problem, as has always been, is that the big players > > have a > > > > stranglehold on all the committer resources, and bringing in > > new > > > > contributors is not high on their priorities. All that's > really > > > required > > > > here is that existing committers
Re: [DISCUSSION] Attracting new contributors
Is it possible if a new comer works together with a mentor on an issue? This way mentor can gradually introduce the newcomer into the codebase, and newcomer would get timely feedback too. On Wed, Apr 28, 2021 at 2:51 PM Benedict Elliott Smith wrote: > > I believe that it can be a virtuous circle where we produce new > committers that help mentoring newcomers. > > That's the dream, and kudos for keeping it alive! I have become jaded > about this possibility, after years of trying. > > > On 28/04/2021, 10:18, "Benjamin Lerer" wrote: > > > > > I think there are two main hurdles, one is restoring contributor > interest > > in mentoring, and the other is finding newcomers that actually want > to > > stick around. > > > I am interested in mentoring new committers to help the project grow > and > some of the new committers expressed the same interest to me. I believe > that it can be a virtuous circle where we produce new committers that > help > mentoring newcomers. > > What we need is to be well organized and make sure that we have a > reasonable response time to newcomers. > > Berenguer created this board to help to track newcomers contributions: > > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=463=2088 > Apparently Brandon is cheating to appear as a newcomer but we will > solve > that. He should be at the Nightmare level ;-) > > Le mer. 28 avr. 2021 à 10:54, Benedict Elliott Smith < > bened...@apache.org> > a écrit : > > > I think there are two main hurdles, one is restoring contributor > interest > > in mentoring, and the other is finding newcomers that actually want > to > > stick around. These are perhaps two sides of the same coin, though. > An ugly > > truth is that it isn't very enjoyable or rewarding to help newcomers > when > > they mostly don't stick around - often even to complete their first > patch! > > The patches are mostly uninteresting, the work often done to a low > > standard, and it is easy to underestimate the amount of time > involved in > > every such failed interaction. > > > > I think making it easier to contribute and demonstrate a lasting > interest > > in the project without the hand holding of long term contributors may > > benefit both sides of the equation, as it is more rewarding to help > > somebody who's demonstrated a persistent interest in the community. > > > > > > On 28/04/2021, 03:24, "Paulo Motta" > wrote: > > > > > There is no great hurdle in finding something to work on, it's > > solely finding > > someone with the knowledge that can help you work on something > and > > progress > > it to commit. > > > > I agree the primary challenge is to engage existing contributors > to > > mentor > > newcomers, but this doesn’t preclude having good documentation > and a > > well > > maintained task pool to allow newcomers to self-serve as much as > > possible > > and reduce the mentoring burden, so these efforts are > complimentary. > > > > For instance, a few students were interested in picking random > tasks to > > work on in preparation for Google Summer of Code, but it was not > > straightforward for them to find a task to work on because we > don’t > > consistently label tickets as “Low Hanging Fruit” and the ones > that are > > tagged sometimes don’t have meaningful descriptions making it > hard for > > these students to get started on tasks without unnecessarily > taking > > some > > time from the mentor (which could have been saved if the tasks > were > > properly described and labeled in the first place). > > > > On Tue, 27 Apr 2021 at 22:24 Kane Wilson wrote: > > > > > The main problem, as has always been, is that the big players > have a > > > stranglehold on all the committer resources, and bringing in > new > > > contributors is not high on their priorities. All that's really > > required > > > here is that existing committers are directed to spend some > > non-negligible > > > portion of their time assisting non-committers (especially > those not > > > already employed in their own organisation). That should > really be a > > > starting point, as any other measures you take will not help > until > > the time > > > is allocated so people can actually receive feedback and help > from > > the > > > small pool of knowledge available. > > > > > > There is no great hurdle in finding something to work on, it's > solely > > > finding someone with the knowledge that can help you work on > > something and > > > progress it to commit. > > > > > > > > > > Run a committer incubator program: Take applications for a >
Re: [DISCUSSION] Attracting new contributors
> I believe that it can be a virtuous circle where we produce new committers > that help mentoring newcomers. That's the dream, and kudos for keeping it alive! I have become jaded about this possibility, after years of trying. On 28/04/2021, 10:18, "Benjamin Lerer" wrote: > > I think there are two main hurdles, one is restoring contributor interest > in mentoring, and the other is finding newcomers that actually want to > stick around. I am interested in mentoring new committers to help the project grow and some of the new committers expressed the same interest to me. I believe that it can be a virtuous circle where we produce new committers that help mentoring newcomers. What we need is to be well organized and make sure that we have a reasonable response time to newcomers. Berenguer created this board to help to track newcomers contributions: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=463=2088 Apparently Brandon is cheating to appear as a newcomer but we will solve that. He should be at the Nightmare level ;-) Le mer. 28 avr. 2021 à 10:54, Benedict Elliott Smith a écrit : > I think there are two main hurdles, one is restoring contributor interest > in mentoring, and the other is finding newcomers that actually want to > stick around. These are perhaps two sides of the same coin, though. An ugly > truth is that it isn't very enjoyable or rewarding to help newcomers when > they mostly don't stick around - often even to complete their first patch! > The patches are mostly uninteresting, the work often done to a low > standard, and it is easy to underestimate the amount of time involved in > every such failed interaction. > > I think making it easier to contribute and demonstrate a lasting interest > in the project without the hand holding of long term contributors may > benefit both sides of the equation, as it is more rewarding to help > somebody who's demonstrated a persistent interest in the community. > > > On 28/04/2021, 03:24, "Paulo Motta" wrote: > > > There is no great hurdle in finding something to work on, it's > solely finding > someone with the knowledge that can help you work on something and > progress > it to commit. > > I agree the primary challenge is to engage existing contributors to > mentor > newcomers, but this doesn’t preclude having good documentation and a > well > maintained task pool to allow newcomers to self-serve as much as > possible > and reduce the mentoring burden, so these efforts are complimentary. > > For instance, a few students were interested in picking random tasks to > work on in preparation for Google Summer of Code, but it was not > straightforward for them to find a task to work on because we don’t > consistently label tickets as “Low Hanging Fruit” and the ones that are > tagged sometimes don’t have meaningful descriptions making it hard for > these students to get started on tasks without unnecessarily taking > some > time from the mentor (which could have been saved if the tasks were > properly described and labeled in the first place). > > On Tue, 27 Apr 2021 at 22:24 Kane Wilson wrote: > > > The main problem, as has always been, is that the big players have a > > stranglehold on all the committer resources, and bringing in new > > contributors is not high on their priorities. All that's really > required > > here is that existing committers are directed to spend some > non-negligible > > portion of their time assisting non-committers (especially those not > > already employed in their own organisation). That should really be a > > starting point, as any other measures you take will not help until > the time > > is allocated so people can actually receive feedback and help from > the > > small pool of knowledge available. > > > > There is no great hurdle in finding something to work on, it's solely > > finding someone with the knowledge that can help you work on > something and > > progress it to commit. > > > > > > > Run a committer incubator program: Take applications for a small > number > > > of spots(5-10) and mentor these new engineers through learning the > code > > > base, understanding the contribution process, and eventually making > > > substantive code contributions to the project. The eventual goal > is that > > > those who finish will be added as a committer to the project. This > could > > be > > > as big or small as we want but I can see all sorts of great things > that > > > could
Re: [DISCUSSION] Attracting new contributors
> > I think there are two main hurdles, one is restoring contributor interest > in mentoring, and the other is finding newcomers that actually want to > stick around. I am interested in mentoring new committers to help the project grow and some of the new committers expressed the same interest to me. I believe that it can be a virtuous circle where we produce new committers that help mentoring newcomers. What we need is to be well organized and make sure that we have a reasonable response time to newcomers. Berenguer created this board to help to track newcomers contributions: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=463=2088 Apparently Brandon is cheating to appear as a newcomer but we will solve that. He should be at the Nightmare level ;-) Le mer. 28 avr. 2021 à 10:54, Benedict Elliott Smith a écrit : > I think there are two main hurdles, one is restoring contributor interest > in mentoring, and the other is finding newcomers that actually want to > stick around. These are perhaps two sides of the same coin, though. An ugly > truth is that it isn't very enjoyable or rewarding to help newcomers when > they mostly don't stick around - often even to complete their first patch! > The patches are mostly uninteresting, the work often done to a low > standard, and it is easy to underestimate the amount of time involved in > every such failed interaction. > > I think making it easier to contribute and demonstrate a lasting interest > in the project without the hand holding of long term contributors may > benefit both sides of the equation, as it is more rewarding to help > somebody who's demonstrated a persistent interest in the community. > > > On 28/04/2021, 03:24, "Paulo Motta" wrote: > > > There is no great hurdle in finding something to work on, it's > solely finding > someone with the knowledge that can help you work on something and > progress > it to commit. > > I agree the primary challenge is to engage existing contributors to > mentor > newcomers, but this doesn’t preclude having good documentation and a > well > maintained task pool to allow newcomers to self-serve as much as > possible > and reduce the mentoring burden, so these efforts are complimentary. > > For instance, a few students were interested in picking random tasks to > work on in preparation for Google Summer of Code, but it was not > straightforward for them to find a task to work on because we don’t > consistently label tickets as “Low Hanging Fruit” and the ones that are > tagged sometimes don’t have meaningful descriptions making it hard for > these students to get started on tasks without unnecessarily taking > some > time from the mentor (which could have been saved if the tasks were > properly described and labeled in the first place). > > On Tue, 27 Apr 2021 at 22:24 Kane Wilson wrote: > > > The main problem, as has always been, is that the big players have a > > stranglehold on all the committer resources, and bringing in new > > contributors is not high on their priorities. All that's really > required > > here is that existing committers are directed to spend some > non-negligible > > portion of their time assisting non-committers (especially those not > > already employed in their own organisation). That should really be a > > starting point, as any other measures you take will not help until > the time > > is allocated so people can actually receive feedback and help from > the > > small pool of knowledge available. > > > > There is no great hurdle in finding something to work on, it's solely > > finding someone with the knowledge that can help you work on > something and > > progress it to commit. > > > > > > > Run a committer incubator program: Take applications for a small > number > > > of spots(5-10) and mentor these new engineers through learning the > code > > > base, understanding the contribution process, and eventually making > > > substantive code contributions to the project. The eventual goal > is that > > > those who finish will be added as a committer to the project. This > could > > be > > > as big or small as we want but I can see all sorts of great things > that > > > could come of this. > > > > > > This is a great idea as a follow up (i.e, after there is evidence > that > > contributions are being progressed), as it would give a more concrete > > process and confidence for existing contributors that they can > eventually > > become committers, and insight into what work is required. > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: [DISCUSSION] Attracting new contributors
I think there are two main hurdles, one is restoring contributor interest in mentoring, and the other is finding newcomers that actually want to stick around. These are perhaps two sides of the same coin, though. An ugly truth is that it isn't very enjoyable or rewarding to help newcomers when they mostly don't stick around - often even to complete their first patch! The patches are mostly uninteresting, the work often done to a low standard, and it is easy to underestimate the amount of time involved in every such failed interaction. I think making it easier to contribute and demonstrate a lasting interest in the project without the hand holding of long term contributors may benefit both sides of the equation, as it is more rewarding to help somebody who's demonstrated a persistent interest in the community. On 28/04/2021, 03:24, "Paulo Motta" wrote: > There is no great hurdle in finding something to work on, it's solely finding someone with the knowledge that can help you work on something and progress it to commit. I agree the primary challenge is to engage existing contributors to mentor newcomers, but this doesn’t preclude having good documentation and a well maintained task pool to allow newcomers to self-serve as much as possible and reduce the mentoring burden, so these efforts are complimentary. For instance, a few students were interested in picking random tasks to work on in preparation for Google Summer of Code, but it was not straightforward for them to find a task to work on because we don’t consistently label tickets as “Low Hanging Fruit” and the ones that are tagged sometimes don’t have meaningful descriptions making it hard for these students to get started on tasks without unnecessarily taking some time from the mentor (which could have been saved if the tasks were properly described and labeled in the first place). On Tue, 27 Apr 2021 at 22:24 Kane Wilson wrote: > The main problem, as has always been, is that the big players have a > stranglehold on all the committer resources, and bringing in new > contributors is not high on their priorities. All that's really required > here is that existing committers are directed to spend some non-negligible > portion of their time assisting non-committers (especially those not > already employed in their own organisation). That should really be a > starting point, as any other measures you take will not help until the time > is allocated so people can actually receive feedback and help from the > small pool of knowledge available. > > There is no great hurdle in finding something to work on, it's solely > finding someone with the knowledge that can help you work on something and > progress it to commit. > > > > Run a committer incubator program: Take applications for a small number > > of spots(5-10) and mentor these new engineers through learning the code > > base, understanding the contribution process, and eventually making > > substantive code contributions to the project. The eventual goal is that > > those who finish will be added as a committer to the project. This could > be > > as big or small as we want but I can see all sorts of great things that > > could come of this. > > > This is a great idea as a follow up (i.e, after there is evidence that > contributions are being progressed), as it would give a more concrete > process and confidence for existing contributors that they can eventually > become committers, and insight into what work is required. > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
