Re: Commitlogs are filling the Full Disk space and nodes are down

[Jeff Jirsa]

There's an open bug for users that have offheap memtables and secondary
index - there's at least a few people reporting an error flushing that
blocks future flushes.

If you're seeing that, and use that combo, you may want to switch to
on-heap memtables (or contribute a patch to fix the offheap+2i interaction)

On Tue, Jan 30, 2018 at 9:06 AM, Chris Lohfink  wrote:

> The commitlog growing is often a symptom of a problem. If the memtable
> flush or post flush fails in anyway, the commitlogs will not be
> recycled/deleted and will continue to pool up.
>
> Might want to go back in logs earlier to make sure theres nothing like the
> postmemtable flusher getting a permission error (some tooling creates
> commitlogs so if run by wrong user can create this prooblem), or a memtable
> flush error.  You can also check tpstats to see if tasks are queued up in
> postmemtable flusher and jstack to see where the active ones are stuck if
> they are.
>
> Chris
>
> On Jan 30, 2018, at 4:20 AM, Amit Singh  wrote:
>
> Hi,
>
> When you actually say nodetool flush, data from memTable goes to disk
> based structure as SStables and side by side , commit logs segments for
> that particular data get written off and its continuous process . May be in
> your case , you can decrease the value of  below uncommented property in
> Cassandra.yaml
>
> commitlog_total_space_in_mb
>
> Also this is what is it used for
>
> # Total space to use for commit logs on disk.
> #
> # If space gets above this value, Cassandra will flush every dirty CF
> # in the oldest segment and remove it.  So a small total commitlog space
> # will tend to cause more flush activity on less-active columnfamilies.
> #
> # The default value is the smaller of 8192, and 1/4 of the total space
> # of the commitlog volume.
>
>
> *From:* Mokkapati, Bhargav (Nokia - IN/Chennai) [mailto:bhargav.mokkapati@
> nokia.com ]
> *Sent:* Tuesday, January 30, 2018 4:00 PM
> *To:* user@cassandra.apache.org
> *Subject:* Commitlogs are filling the Full Disk space and nodes are down
>
> Hi Team,
>
> My Cassandra version : Apache Cassandra 3.0.13
>
> Cassandra nodes are down due to Commitlogs are getting filled up until
> full disk size.
>
> 
>
> With “Nodetool flush” I didn’t see any commitlogs deleted.
>
> Can anyone tell me how to flush the commitlogs without losing data.
>
> Thanks,
> Bhargav M
>
>
>

Re: Commitlogs are filling the Full Disk space and nodes are down

[Chris Lohfink]

The commitlog growing is often a symptom of a problem. If the memtable flush or 
post flush fails in anyway, the commitlogs will not be recycled/deleted and 
will continue to pool up.

Might want to go back in logs earlier to make sure theres nothing like the 
postmemtable flusher getting a permission error (some tooling creates 
commitlogs so if run by wrong user can create this prooblem), or a memtable 
flush error.  You can also check tpstats to see if tasks are queued up in 
postmemtable flusher and jstack to see where the active ones are stuck if they 
are.

Chris

> On Jan 30, 2018, at 4:20 AM, Amit Singh  wrote:
> 
> Hi,
>  
> When you actually say nodetool flush, data from memTable goes to disk based 
> structure as SStables and side by side , commit logs segments for that 
> particular data get written off and its continuous process . May be in your 
> case , you can decrease the value of  below uncommented property in 
> Cassandra.yaml 
>  
> commitlog_total_space_in_mb
>  
> Also this is what is it used for 
>  
> # Total space to use for commit logs on disk.
> #
> # If space gets above this value, Cassandra will flush every dirty CF
> # in the oldest segment and remove it.  So a small total commitlog space
> # will tend to cause more flush activity on less-active columnfamilies.
> #
> # The default value is the smaller of 8192, and 1/4 of the total space
> # of the commitlog volume.
>  
>  
> From: Mokkapati, Bhargav (Nokia - IN/Chennai) 
> [mailto:bhargav.mokkap...@nokia.com] 
> Sent: Tuesday, January 30, 2018 4:00 PM
> To: user@cassandra.apache.org
> Subject: Commitlogs are filling the Full Disk space and nodes are down
>  
> Hi Team,
>  
> My Cassandra version : Apache Cassandra 3.0.13
>  
> Cassandra nodes are down due to Commitlogs are getting filled up until full 
> disk size.
>  
> 
>  
> With “Nodetool flush” I didn’t see any commitlogs deleted.
>  
> Can anyone tell me how to flush the commitlogs without losing data.
>  
> Thanks,
> Bhargav M

RE: Commitlogs are filling the Full Disk space and nodes are down

[Amit Singh]

Hi,

 

When you actually say nodetool flush, data from memTable goes to disk based
structure as SStables and side by side , commit logs segments for that
particular data get written off and its continuous process . May be in your
case , you can decrease the value of  below uncommented property in
Cassandra.yaml 

 

commitlog_total_space_in_mb

 

Also this is what is it used for 

 

# Total space to use for commit logs on disk.

#

# If space gets above this value, Cassandra will flush every dirty CF

# in the oldest segment and remove it.  So a small total commitlog space

# will tend to cause more flush activity on less-active columnfamilies.

#

# The default value is the smaller of 8192, and 1/4 of the total space

# of the commitlog volume.

 

 

From: Mokkapati, Bhargav (Nokia - IN/Chennai)
[mailto:bhargav.mokkap...@nokia.com] 
Sent: Tuesday, January 30, 2018 4:00 PM
To: user@cassandra.apache.org
Subject: Commitlogs are filling the Full Disk space and nodes are down

 

Hi Team,

 

My Cassandra version : Apache Cassandra 3.0.13

 

Cassandra nodes are down due to Commitlogs are getting filled up until full
disk size.

 



 

With "Nodetool flush" I didn't see any commitlogs deleted.

 

Can anyone tell me how to flush the commitlogs without losing data.

 

Thanks,

Bhargav M

Commitlogs are filling the Full Disk space and nodes are down

[Mokkapati, Bhargav (Nokia - IN/Chennai)]

Hi Team,

My Cassandra version : Apache Cassandra 3.0.13

Cassandra nodes are down due to Commitlogs are getting filled up until full 
disk size.

[cid:image001.jpg@01D399E3.666CF940]

With "Nodetool flush" I didn't see any commitlogs deleted.

Can anyone tell me how to flush the commitlogs without losing data.

Thanks,
Bhargav M