Re: Cassandra doesn't flush any commit log files into cdc_raw directory
Hi Ahmed, Thank you for your insights! I don't think the write rates are slow for the tables I enable CDC on, otherwise, the commit log sizes won't go over the cdc_total_space_in_mb (4096) quickly. I'll try to dig more into what affects the speed of memtable flushes. Bingqin Zhou On Thu, Apr 29, 2021 at 1:11 AM Ahmed Eljami wrote: > Hi Bingqin, > > When cdc_raw directory is full, Cassandra rejects new writes on this node > with the following message in the log: > > - Rejecting Mutation containing CDC-enabled table. > > https://github.com/apache/cassandra/blob/cassandra-3.11.9/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManagerCDC.java#L125 > > So, I'm not sorry if WriteTimeoutException is related to cdc feature in > your case. > > For your issue with no commitLog flushed/copied into cdc_raw directory, > can you try with an explicit nodetool flush and see if commitLog will be > transferred ? For tables with a "very low" write rates, memtable can take a > lot of time before be flushed on disk. > > Cheers, > Ahmed > > > Le jeu. 29 avr. 2021 à 00:23, Bingqin Zhou a > écrit : > >> Hi all, >> >> We're working on a Kafka connector to capture data changes in Cassandra by >> processing commit log files in the cdc_raw directory. After we enabled CDC >> on a few tables, we didn't observe any commit log files getting flushed >> into cdc_raw directory as expected, but got WriteTimeoutException in >> Cassandra DB. >> >> Here's how we reproduce the issue: >> >> 1. Our Cassandra Settings: >> >> - Cassandra Version: 3.11.9 >> - Related configs in Cassandra.yaml: >>- cdc_enabled: true >>- cdc_total_space_in_mb: 4096 >>- commitlog_segment_size_in_mb: 32mb >>- commitlog_total_space_in_mb: 8192 >>- commitlog_sync: periodic >>- commitlog_sync_period_in_ms: 1 >> >> 2. Enable CDC on a few tables by CQL: >> ALTER TABLE foo WITH cdc=true; >> >> 3. After a few days, we get *WriteTimeoutException* in Cassandra DB. >> However at the same time, cdc_raw directory is still empty with no commit >> log flushed/copied into it at all. >> >> I want to understand why there's no commit log file flushed into >> cdc_raw directory at all even when the threshold cdc_total_space_in_mb has >> been reached and write suspension has been triggered in Cassandra DB. This >> sounds like a bug and currently makes the CDC feature useless. >> >> Thanks so much, >> Bingqin Zhou >> > > > -- > Cordialement; > > Ahmed ELJAMI >
Re: Cassandra doesn't flush any commit log files into cdc_raw directory
Hi Bingqin, When cdc_raw directory is full, Cassandra rejects new writes on this node with the following message in the log: - Rejecting Mutation containing CDC-enabled table. https://github.com/apache/cassandra/blob/cassandra-3.11.9/src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentManagerCDC.java#L125 So, I'm not sorry if WriteTimeoutException is related to cdc feature in your case. For your issue with no commitLog flushed/copied into cdc_raw directory, can you try with an explicit nodetool flush and see if commitLog will be transferred ? For tables with a "very low" write rates, memtable can take a lot of time before be flushed on disk. Cheers, Ahmed Le jeu. 29 avr. 2021 à 00:23, Bingqin Zhou a écrit : > Hi all, > > We're working on a Kafka connector to capture data changes in Cassandra by > processing commit log files in the cdc_raw directory. After we enabled CDC > on a few tables, we didn't observe any commit log files getting flushed > into cdc_raw directory as expected, but got WriteTimeoutException in > Cassandra DB. > > Here's how we reproduce the issue: > > 1. Our Cassandra Settings: > > - Cassandra Version: 3.11.9 > - Related configs in Cassandra.yaml: >- cdc_enabled: true >- cdc_total_space_in_mb: 4096 >- commitlog_segment_size_in_mb: 32mb >- commitlog_total_space_in_mb: 8192 >- commitlog_sync: periodic >- commitlog_sync_period_in_ms: 1 > > 2. Enable CDC on a few tables by CQL: > ALTER TABLE foo WITH cdc=true; > > 3. After a few days, we get *WriteTimeoutException* in Cassandra DB. > However at the same time, cdc_raw directory is still empty with no commit > log flushed/copied into it at all. > > I want to understand why there's no commit log file flushed into > cdc_raw directory at all even when the threshold cdc_total_space_in_mb has > been reached and write suspension has been triggered in Cassandra DB. This > sounds like a bug and currently makes the CDC feature useless. > > Thanks so much, > Bingqin Zhou > -- Cordialement; Ahmed ELJAMI
Cassandra doesn't flush any commit log files into cdc_raw directory
Hi all, We're working on a Kafka connector to capture data changes in Cassandra by processing commit log files in the cdc_raw directory. After we enabled CDC on a few tables, we didn't observe any commit log files getting flushed into cdc_raw directory as expected, but got WriteTimeoutException in Cassandra DB. Here's how we reproduce the issue: 1. Our Cassandra Settings: - Cassandra Version: 3.11.9 - Related configs in Cassandra.yaml: - cdc_enabled: true - cdc_total_space_in_mb: 4096 - commitlog_segment_size_in_mb: 32mb - commitlog_total_space_in_mb: 8192 - commitlog_sync: periodic - commitlog_sync_period_in_ms: 1 2. Enable CDC on a few tables by CQL: ALTER TABLE foo WITH cdc=true; 3. After a few days, we get *WriteTimeoutException* in Cassandra DB. However at the same time, cdc_raw directory is still empty with no commit log flushed/copied into it at all. I want to understand why there's no commit log file flushed into cdc_raw directory at all even when the threshold cdc_total_space_in_mb has been reached and write suspension has been triggered in Cassandra DB. This sounds like a bug and currently makes the CDC feature useless. Thanks so much, Bingqin Zhou