One way to flush commitlog is nodetool flush On Cassandra nodes . On Tue, May 4, 2021 at 3:58 PM Bingqin Zhou <bingq...@wepay.com> wrote:
> Hi Kane, > > Thank you for the insights! > > Reducing the total space on its own will help, however definitely test >> this as such a large drop could result in a massive increase in SSTables >> and thus compaction overhead. You'll in general want to look into any >> property that makes memtables flush more frequently (which is based on heap >> size and some tuning properties in cassandra.yaml). > > > If we decrease *memtable_heap_sapce_in_mb* and *memtable_off_space_in_mb*, > is it going to cause more compaction activities potentially as well? > > I'm typically not a fan of using a database as a streaming/workflow >> service, so I have to ask have you considered managing this from your >> clients rather than using CDC in C*? > > > Actually, the design and initiation of our service is based on the fact > that the CDC feature in Cassandra is used for streaming data changes in > Cassandra with low latency. If this is not the case, may I understand > what's the purpose and the intended use case for the CDC feature in > Cassandra please? > > Thank you so much! > Bingqin Zhou > > On Mon, May 3, 2021 at 5:00 PM Kane Wilson <k...@raft.so> wrote: > >> (removing dev) >> >> commitlog_segment_size_in_mb isn't going to help, in fact you probably >> don't want to modify this as it'll reduce the maximum size of your >> mutations. >> Reducing the total space on its own will help, however definitely test >> this as such a large drop could result in a massive increase in SSTables >> and thus compaction overhead. You'll in general want to look into any >> property that makes memtables flush more frequently (which is based on heap >> size and some tuning properties in cassandra.yaml). >> >> I'm typically not a fan of using a database as a streaming/workflow >> service, so I have to ask have you considered managing this from your >> clients rather than using CDC in C*? >> >> raft.so - Cassandra consulting, support, and managed services >> >> >> On Tue, May 4, 2021 at 4:16 AM Bingqin Zhou <bingq...@wepay.com> wrote: >> >>> Hi, >>> >>> We're working with the CDC feature to develop an agent to stream changes >>> in Cassandra DB into Kafka. However, the CDC feature doesn't work well for >>> us so far because CommitLog files are rarely flushed into cdc_raw >>> directory, and the frequency can be as low as a few months. >>> >>> Is there any suggested and feasible way to increase the frequency for >>> Cassandra to flush CommitLog files please? >>> >>> We're thinking about decreasing *commitlog_segment_size_in_mb* from 32 >>> to 16, and decreasing *commitlog_total_space_in_mb* from 8192 to 160. >>> Does this sound like a reasonable approach? Is there any concern or >>> anything we need to be warned about trying this please? >>> >>> Thank you! >>> >>> Bingqin Zhou >>> >>
