Re: High disk usage casaandra 3.11.7

[Abdul Patel]

Twcs is best for TTL not for excipilitly delete correct?


On Friday, September 17, 2021, Abdul Patel  wrote:

> 48hrs deletion is deleting older data more than 48hrs .
> LCS was used as its more of an write once and read many application.
>
> On Friday, September 17, 2021, Bowen Song  wrote:
>
>> Congratulation! You've just found out the cause of it. Does all data get
>> deletes 48 hours after they are inserted? If so, are you sure LCS is the
>> right compaction strategy for this table? TWCS sounds like a much better
>> fit for this purpose.
>> On 17/09/2021 19:16, Abdul Patel wrote:
>>
>> Thanks.
>> Application deletes data every 48hrs of older data.
>> Auto compaction works but as space is full ..errorlog only says not
>> enough space to run compaction.
>>
>>
>> On Friday, September 17, 2021, Bowen Song  wrote:
>>
>>> If major compaction is failing due to disk space constraint, you could
>>> copy the files to another server and run a major compaction there instead
>>> (i.e.: start cassandra on new server but not joining the existing cluster).
>>> If you must replace the node, at least use the
>>> '-Dcassandra.replace_address=...' parameter instead of 'nodetool
>>> decommission' and then re-add, because the later changes the token ranges
>>> on the node, and that makes troubleshooting harder.
>>>
>>> 22GB of data amplifies to nearly 300GB sounds very impossible to me,
>>> there must be something else going on. Have you turned off auto compaction?
>>> Did you change the default parameters (namely, the 'fanout_size') for LCS?
>>> If this doesn't give you a clue, have a look at the SSTable data files, do
>>> you notice anything unusual? For example, too many small files, or some
>>> files are extraordinarily large. Also have a look at the logs, is there
>>> anything unusual? Also, do you know the application logic? Does it do a
>>> lots of delete or update (including 'upsert')? Writes with TTL? Does the
>>> table has a default TTL?
>>> On 17/09/2021 13:45, Abdul Patel wrote:
>>>
>>> Close 300 gb data. Nodetool decommission/removenode and added back one
>>> node ans it came back to 22Gb.
>>> Cant run major compaction as no space much left.
>>>
>>> On Friday, September 17, 2021, Bowen Song  wrote:
>>>
 Okay, so how big exactly is the data on disk? You said removing and
 adding a new node gives you 20GB on disk, was that done via the
 '-Dcassandra.replace_address=...' parameter? If not, the new node will
 almost certainly have a different token range and not directly comparable
 to the existing node if you have uneven partitions or small number of
 partitions in the table. Also, try major compaction, it's a lot easier than
 replacing a node.


 On 17/09/2021 12:28, Abdul Patel wrote:

 Yes i checked and cleared all snapshots and also i had incremental
 backups in backup folder ..i removed the same .. its purely data..


 On Friday, September 17, 2021, Bowen Song  wrote:

> Assuming your total disk space is a lot bigger than 50GB in size
> (accounting for disk space amplification, commit log, logs, OS data, 
> etc.),
> I would suspect the disk space is being used by something else. Have you
> checked that the disk space is actually being used by the cassandra data
> directory? If so, have a look at 'nodetool listsnapshots' command output 
> as
> well.
>
>
> On 17/09/2021 05:48, Abdul Patel wrote:
>
>> Hello
>>
>> We have cassandra with leveledcompaction strategy, recently found
>> filesystem almost 90% full but the data was only 10m records.
>> Manual compaction will work? As not sure its recommended and space is
>> also constraint ..tried removing and adding one node and now data is at
>> 20GB which looks appropropiate.
>> So is only solution to reclaim space is remove/add node?
>>
>

Re: High disk usage casaandra 3.11.7

[Abdul Patel]

48hrs deletion is deleting older data more than 48hrs .
LCS was used as its more of an write once and read many application.

On Friday, September 17, 2021, Bowen Song  wrote:

> Congratulation! You've just found out the cause of it. Does all data get
> deletes 48 hours after they are inserted? If so, are you sure LCS is the
> right compaction strategy for this table? TWCS sounds like a much better
> fit for this purpose.
> On 17/09/2021 19:16, Abdul Patel wrote:
>
> Thanks.
> Application deletes data every 48hrs of older data.
> Auto compaction works but as space is full ..errorlog only says not enough
> space to run compaction.
>
>
> On Friday, September 17, 2021, Bowen Song  wrote:
>
>> If major compaction is failing due to disk space constraint, you could
>> copy the files to another server and run a major compaction there instead
>> (i.e.: start cassandra on new server but not joining the existing cluster).
>> If you must replace the node, at least use the
>> '-Dcassandra.replace_address=...' parameter instead of 'nodetool
>> decommission' and then re-add, because the later changes the token ranges
>> on the node, and that makes troubleshooting harder.
>>
>> 22GB of data amplifies to nearly 300GB sounds very impossible to me,
>> there must be something else going on. Have you turned off auto compaction?
>> Did you change the default parameters (namely, the 'fanout_size') for LCS?
>> If this doesn't give you a clue, have a look at the SSTable data files, do
>> you notice anything unusual? For example, too many small files, or some
>> files are extraordinarily large. Also have a look at the logs, is there
>> anything unusual? Also, do you know the application logic? Does it do a
>> lots of delete or update (including 'upsert')? Writes with TTL? Does the
>> table has a default TTL?
>> On 17/09/2021 13:45, Abdul Patel wrote:
>>
>> Close 300 gb data. Nodetool decommission/removenode and added back one
>> node ans it came back to 22Gb.
>> Cant run major compaction as no space much left.
>>
>> On Friday, September 17, 2021, Bowen Song  wrote:
>>
>>> Okay, so how big exactly is the data on disk? You said removing and
>>> adding a new node gives you 20GB on disk, was that done via the
>>> '-Dcassandra.replace_address=...' parameter? If not, the new node will
>>> almost certainly have a different token range and not directly comparable
>>> to the existing node if you have uneven partitions or small number of
>>> partitions in the table. Also, try major compaction, it's a lot easier than
>>> replacing a node.
>>>
>>>
>>> On 17/09/2021 12:28, Abdul Patel wrote:
>>>
>>> Yes i checked and cleared all snapshots and also i had incremental
>>> backups in backup folder ..i removed the same .. its purely data..
>>>
>>>
>>> On Friday, September 17, 2021, Bowen Song  wrote:
>>>
 Assuming your total disk space is a lot bigger than 50GB in size
 (accounting for disk space amplification, commit log, logs, OS data, etc.),
 I would suspect the disk space is being used by something else. Have you
 checked that the disk space is actually being used by the cassandra data
 directory? If so, have a look at 'nodetool listsnapshots' command output as
 well.


 On 17/09/2021 05:48, Abdul Patel wrote:

> Hello
>
> We have cassandra with leveledcompaction strategy, recently found
> filesystem almost 90% full but the data was only 10m records.
> Manual compaction will work? As not sure its recommended and space is
> also constraint ..tried removing and adding one node and now data is at
> 20GB which looks appropropiate.
> So is only solution to reclaim space is remove/add node?
>

Re: ShortReadPartitionsProtection

[Jeff Jirsa]

Short read protection is a feature added in 3.0 to work around a possible 
situation in 2.1 where we could fail to return all rows in a result

The basic premise is that when you read, we ask for the same number of rows 
from all of the replicas involved in the query. It’s possible, with the right 
chance / luck / conditions, that each replica will return 5000 rows but the 
results won’t be the same 5000, and when you merge them together you can end up 
with less than 5000 live rows (basically if a couple instances each miss some 
tombstones).

Cassandra 3 and newer try to detect and fix this by doing extra reads when they 
detect the result is shorter than the actual requested page - they protect 
against short results by reading again.

In 3.0-3.11 this was silent and invisible
In 4.0 it logs 
There’s nothing you can or should do about this, but it’s letting you know it’s 
doing an extra read to ensure correctness. 


> On Sep 17, 2021, at 12:15 PM, James Brown  wrote:
> 
> 
> I upgraded a test cluster from 3.11.11 to 4.0.1 today and the following 
> message started appearing the logs fairly regularly:
> 
> INFO  [Native-Transport-Requests-1] ShortReadPartitionsProtection.java:153 - 
> Requesting 4951 extra rows from 
> Full(/10.132.39.33:25469,(8972614703113750908,-9163822348452166994]) for 
> short read protection
> 
> This message never appeared under 3.x and the string "short read protection" 
> doesn't seem to occur in any documentation. There aren't any interesting 
> comments or docstrings in that java file.
> 
> Is this just an artifact of the very small amount of data in this testing 
> cluster, or is something broken?
> -- 
> James Brown
> Engineer

ShortReadPartitionsProtection

[James Brown]

I upgraded a test cluster from 3.11.11 to 4.0.1 today and the following
message started appearing the logs fairly regularly:

INFO  [Native-Transport-Requests-1] ShortReadPartitionsProtection.java:153
- Requesting 4951 extra rows from
Full(/10.132.39.33:25469,(8972614703113750908,-9163822348452166994])
for short read protection

This message never appeared under 3.x and the string "short read
protection" doesn't seem to occur in any documentation. There aren't any
interesting comments or docstrings in that java file.

Is this just an artifact of the very small amount of data in this testing
cluster, or is something broken?
-- 
James Brown
Engineer

Re: High disk usage casaandra 3.11.7

[Bowen Song]

Congratulation! You've just found out the cause of it. Does all data get 
deletes 48 hours after they are inserted? If so, are you sure LCS is the 
right compaction strategy for this table? TWCS sounds like a much better 
fit for this purpose.


On 17/09/2021 19:16, Abdul Patel wrote:

Thanks.
Application deletes data every 48hrs of older data.
Auto compaction works but as space is full ..errorlog only says not 
enough space to run compaction.



On Friday, September 17, 2021, Bowen Song > wrote:


If major compaction is failing due to disk space constraint, you
could copy the files to another server and run a major compaction
there instead (i.e.: start cassandra on new server but not joining
the existing cluster). If you must replace the node, at least use
the '-Dcassandra.replace_address=...' parameter instead of
'nodetool decommission' and then re-add, because the later changes
the token ranges on the node, and that makes troubleshooting harder.

22GB of data amplifies to nearly 300GB sounds very impossible to
me, there must be something else going on. Have you turned off
auto compaction? Did you change the default parameters (namely,
the 'fanout_size') for LCS? If this doesn't give you a clue, have
a look at the SSTable data files, do you notice anything unusual?
For example, too many small files, or some files are
extraordinarily large. Also have a look at the logs, is there
anything unusual? Also, do you know the application logic? Does it
do a lots of delete or update (including 'upsert')? Writes with
TTL? Does the table has a default TTL?

On 17/09/2021 13:45, Abdul Patel wrote:

Close 300 gb data. Nodetool decommission/removenode and added
back one node ans it came back to 22Gb.
Cant run major compaction as no space much left.

On Friday, September 17, 2021, Bowen Song mailto:bo...@bso.ng>> wrote:

Okay, so how big exactly is the data on disk? You said
removing and adding a new node gives you 20GB on disk, was
that done via the '-Dcassandra.replace_address=...'
parameter? If not, the new node will almost certainly have a
different token range and not directly comparable to the
existing node if you have uneven partitions or small number
of partitions in the table. Also, try major compaction, it's
a lot easier than replacing a node.


On 17/09/2021 12:28, Abdul Patel wrote:

Yes i checked and cleared all snapshots and also i had
incremental backups in backup folder ..i removed the same ..
its purely data..


On Friday, September 17, 2021, Bowen Song mailto:bo...@bso.ng>> wrote:

Assuming your total disk space is a lot bigger than 50GB
in size (accounting for disk space amplification, commit
log, logs, OS data, etc.), I would suspect the disk
space is being used by something else. Have you checked
that the disk space is actually being used by the
cassandra data directory? If so, have a look at
'nodetool listsnapshots' command output as well.


On 17/09/2021 05:48, Abdul Patel wrote:

Hello

We have cassandra with leveledcompaction strategy,
recently found filesystem almost 90% full but the
data was only 10m records.
Manual compaction will work? As not sure its
recommended and space is also constraint ..tried
removing and adding one node and now data is at 20GB
which looks appropropiate.
So is only solution to reclaim space is remove/add node?

Re: High disk usage casaandra 3.11.7

[Abdul Patel]

Thanks.
Application deletes data every 48hrs of older data.
Auto compaction works but as space is full ..errorlog only says not enough
space to run compaction.


On Friday, September 17, 2021, Bowen Song  wrote:

> If major compaction is failing due to disk space constraint, you could
> copy the files to another server and run a major compaction there instead
> (i.e.: start cassandra on new server but not joining the existing cluster).
> If you must replace the node, at least use the
> '-Dcassandra.replace_address=...' parameter instead of 'nodetool
> decommission' and then re-add, because the later changes the token ranges
> on the node, and that makes troubleshooting harder.
>
> 22GB of data amplifies to nearly 300GB sounds very impossible to me, there
> must be something else going on. Have you turned off auto compaction? Did
> you change the default parameters (namely, the 'fanout_size') for LCS? If
> this doesn't give you a clue, have a look at the SSTable data files, do you
> notice anything unusual? For example, too many small files, or some files
> are extraordinarily large. Also have a look at the logs, is there anything
> unusual? Also, do you know the application logic? Does it do a lots of
> delete or update (including 'upsert')? Writes with TTL? Does the table has
> a default TTL?
>
> On 17/09/2021 13:45, Abdul Patel wrote:
>
> Close 300 gb data. Nodetool decommission/removenode and added back one
> node ans it came back to 22Gb.
> Cant run major compaction as no space much left.
>
> On Friday, September 17, 2021, Bowen Song  wrote:
>
>> Okay, so how big exactly is the data on disk? You said removing and
>> adding a new node gives you 20GB on disk, was that done via the
>> '-Dcassandra.replace_address=...' parameter? If not, the new node will
>> almost certainly have a different token range and not directly comparable
>> to the existing node if you have uneven partitions or small number of
>> partitions in the table. Also, try major compaction, it's a lot easier than
>> replacing a node.
>>
>>
>> On 17/09/2021 12:28, Abdul Patel wrote:
>>
>> Yes i checked and cleared all snapshots and also i had incremental
>> backups in backup folder ..i removed the same .. its purely data..
>>
>>
>> On Friday, September 17, 2021, Bowen Song  wrote:
>>
>>> Assuming your total disk space is a lot bigger than 50GB in size
>>> (accounting for disk space amplification, commit log, logs, OS data, etc.),
>>> I would suspect the disk space is being used by something else. Have you
>>> checked that the disk space is actually being used by the cassandra data
>>> directory? If so, have a look at 'nodetool listsnapshots' command output as
>>> well.
>>>
>>>
>>> On 17/09/2021 05:48, Abdul Patel wrote:
>>>
 Hello

 We have cassandra with leveledcompaction strategy, recently found
 filesystem almost 90% full but the data was only 10m records.
 Manual compaction will work? As not sure its recommended and space is
 also constraint ..tried removing and adding one node and now data is at
 20GB which looks appropropiate.
 So is only solution to reclaim space is remove/add node?

>>>

Re: TWCS on Non TTL Data

[Bowen Song]

If you use TWCS with TTL, the old SSTables won't be compacted, the 
entire SSTable file will get dropped after it expires. I don't think you 
will need to manage the compaction or cleanup at all, as they are 
automatic. There's no space limit on the table holding the near-term 
data other than the overall free disk space. There's only a time limit 
on that table.


On 17/09/2021 16:51, Isaeed Mohanna wrote:


Thank for the help,

How does the compaction run? Does it clean old compaction files while 
running or only at the end, I want to manage the free space so not run 
out while its running?


*From:* Jim Shaw 
*Sent:* Wednesday, September 15, 2021 3:49 PM
*To:* user@cassandra.apache.org
*Subject:* Re: TWCS on Non TTL Data

You may try roll up the data, i.e.  a table only 1 month data, old 
data roll up to a table keep a year data.


Thanks,

Jim

On Wed, Sep 15, 2021 at 1:26 AM Isaeed Mohanna > wrote:


My cluster column is the time series timestamp, so basically
sourceId, metric type for partition key and timestamp for the
clustering key the rest of the fields are just values outside of
the primary key. Our reads request are simply give me values for a
time range of a specific sourceId,Metric combination. So I am
guess that during read the sstables that contain the partition key
will be found and out of those the ones that are out of the range
will be excluded, correct?

In practice our queries are up to a month by default, only rarely
we fetch more when someone is exporting the data or so.

In reality also we get old data, that is a source will send its
information late instead of sending it in realtime it will send
all last month\week\day data at once, in that case I guess the
data will end up in current bucket, will that affect performance?

Assuming I start with a  1 week bucket, I could later change the
time window right?

Thanks

*From:* Jeff Jirsa mailto:jji...@gmail.com>>
*Sent:* Tuesday, September 14, 2021 10:35 PM
*To:* cassandra mailto:user@cassandra.apache.org>>
*Subject:* Re: TWCS on Non TTL Data

Inline

On Tue, Sep 14, 2021 at 11:47 AM Isaeed Mohanna mailto:isa...@xsense.co>> wrote:

Hi Jeff

My data is partitioned by a sourceId and metric, a source is
usually active up to a year after which there is no additional
writes for the partition, and reads become scarce, so although
this is not an explicit time component, its time based, will
that suffice?

I guess it means that a single read may touch a year of sstables.
Not great, but perhaps not fatal. Hopefully your reads avoid that
in practice. We'd need the full schema to be very sure (does
clustering column include month/day? if so, there are cases where
that can help exclude sstables)

If I use a  week bucket we will be able to serve last few days
reads from one file and last month from ~5 which is the most
common queries, do u think doing a months bucket a good idea?
That will allow reading from one file most of the time but the
size of each SSTable will be ~5 times bigger

It'll be 1-4 for most common (up to 4 for same bucket reads
because STCS in the first bucket is triggered at min_threshold=4),
and 5 max, seems reasonable. Way better than the 200 or so you're
doing now.

When changing the compaction strategy via JMX, do I need to
issue the alter table command at the end so it will be
reflected in the schema or is it taking care of automatically?
(I am using cassandra 3.11.11)

At the end, yes.

Thanks a lot for your help.

*From:* Jeff Jirsa mailto:jji...@gmail.com>>
*Sent:* Tuesday, September 14, 2021 4:51 PM
*To:* cassandra mailto:user@cassandra.apache.org>>
*Subject:* Re: TWCS on Non TTL Data

On Tue, Sep 14, 2021 at 5:42 AM Isaeed Mohanna
mailto:isa...@xsense.co>> wrote:

Hi

I have a table that stores time series data, the data is
not TTLed since we want to retain the data for the
foreseeable future, and there are no updates or deletes.
(deletes could happens rarely in case some scrambled data
reached the table, but its extremely rare).

Usually we do constant write of incoming data to the table
~ 5 milion a day, mostly newly generated data in the past
week, but we also get old data that got stuck somewhere
but not that often. Usually our reads are for the most
recent data last month – three. But we do fetch old data
as well in a specific time period in the past.

Lately we have been facing performance trouble with this
table see histogram below, When compaction is working on
the table the performance even drops to 10-20 

RE: TWCS on Non TTL Data

[Isaeed Mohanna]

Thank for the help,
How does the compaction run? Does it clean old compaction files while running 
or only at the end, I want to manage the free space so not run out while its 
running?


From: Jim Shaw 
Sent: Wednesday, September 15, 2021 3:49 PM
To: user@cassandra.apache.org
Subject: Re: TWCS on Non TTL Data

You may try roll up the data, i.e.  a table only 1 month data, old data roll up 
to a table keep a year data.

Thanks,
Jim

On Wed, Sep 15, 2021 at 1:26 AM Isaeed Mohanna 
mailto:isa...@xsense.co>> wrote:
My cluster column is the time series timestamp, so basically sourceId, metric 
type for partition key and timestamp for the clustering key the rest of the 
fields are just values outside of the primary key. Our reads request are simply 
give me values for a time range of a specific sourceId,Metric combination. So I 
am guess that during read the sstables that contain the partition key will be 
found and out of those the ones that are out of the range will be excluded, 
correct?
In practice our queries are up to a month by default, only rarely we fetch more 
when someone is exporting the data or so.

In reality also we get old data, that is a source will send its information 
late instead of sending it in realtime it will send all last month\week\day 
data at once, in that case I guess the data will end up in current bucket, will 
that affect performance?

Assuming I start with a  1 week bucket, I could later change the time window 
right?

Thanks


From: Jeff Jirsa mailto:jji...@gmail.com>>
Sent: Tuesday, September 14, 2021 10:35 PM
To: cassandra mailto:user@cassandra.apache.org>>
Subject: Re: TWCS on Non TTL Data

Inline

On Tue, Sep 14, 2021 at 11:47 AM Isaeed Mohanna 
mailto:isa...@xsense.co>> wrote:
Hi Jeff
My data is partitioned by a sourceId and metric, a source is usually active up 
to a year after which there is no additional writes for the partition, and 
reads become scarce, so although this is not an explicit time component, its 
time based, will that suffice?

I guess it means that a single read may touch a year of sstables. Not great, 
but perhaps not fatal. Hopefully your reads avoid that in practice. We'd need 
the full schema to be very sure (does clustering column include month/day? if 
so, there are cases where that can help exclude sstables)


If I use a  week bucket we will be able to serve last few days reads from one 
file and last month from ~5 which is the most common queries, do u think doing 
a months bucket a good idea? That will allow reading from one file most of the 
time but the size of each SSTable will be ~5 times bigger

It'll be 1-4 for most common (up to 4 for same bucket reads because STCS in the 
first bucket is triggered at min_threshold=4), and 5 max, seems reasonable. Way 
better than the 200 or so you're doing now.


When changing the compaction strategy via JMX, do I need to issue the alter 
table command at the end so it will be reflected in the schema or is it taking 
care of automatically? (I am using cassandra 3.11.11)


At the end, yes.

Thanks a lot for your help.








From: Jeff Jirsa mailto:jji...@gmail.com>>
Sent: Tuesday, September 14, 2021 4:51 PM
To: cassandra mailto:user@cassandra.apache.org>>
Subject: Re: TWCS on Non TTL Data



On Tue, Sep 14, 2021 at 5:42 AM Isaeed Mohanna 
mailto:isa...@xsense.co>> wrote:
Hi
I have a table that stores time series data, the data is not TTLed since we 
want to retain the data for the foreseeable future, and there are no updates or 
deletes. (deletes could happens rarely in case some scrambled data reached the 
table, but its extremely rare).
Usually we do constant write of incoming data to the table ~ 5 milion a day, 
mostly newly generated data in the past week, but we also get old data that got 
stuck somewhere but not that often. Usually our reads are for the most recent 
data last month – three. But we do fetch old data as well in a specific time 
period in the past.
Lately we have been facing performance trouble with this table see histogram 
below, When compaction is working on the table the performance even drops to 
10-20 seconds!!
Percentile  SSTables Write Latency  Read LatencyPartition Size  
  Cell Count
  (micros)  (micros)   (bytes)
50%   215.00 17.08  89970.66  1916  
 149
75%   446.00 24.60 223875.79  2759  
 215
95%   535.00 35.43 464228.84  8239  
 642
98%   642.00 51.01 668489.53 24601  
1916
99%   642.00 73.46 962624.93 42510  
3311
Min 0.00  2.30  10090.8143  
   0
Max   770.00   1358.102395318.86   5839588  
  454826

As u can see we are scaning hundreds of sstables, 

Re: High disk usage casaandra 3.11.7

[Bowen Song]

If major compaction is failing due to disk space constraint, you could 
copy the files to another server and run a major compaction there 
instead (i.e.: start cassandra on new server but not joining the 
existing cluster). If you must replace the node, at least use the 
'-Dcassandra.replace_address=...' parameter instead of 'nodetool 
decommission' and then re-add, because the later changes the token 
ranges on the node, and that makes troubleshooting harder.


22GB of data amplifies to nearly 300GB sounds very impossible to me, 
there must be something else going on. Have you turned off auto 
compaction? Did you change the default parameters (namely, the 
'fanout_size') for LCS? If this doesn't give you a clue, have a look at 
the SSTable data files, do you notice anything unusual? For example, too 
many small files, or some files are extraordinarily large. Also have a 
look at the logs, is there anything unusual? Also, do you know the 
application logic? Does it do a lots of delete or update (including 
'upsert')? Writes with TTL? Does the table has a default TTL?


On 17/09/2021 13:45, Abdul Patel wrote:
Close 300 gb data. Nodetool decommission/removenode and added back one 
node ans it came back to 22Gb.

Cant run major compaction as no space much left.

On Friday, September 17, 2021, Bowen Song > wrote:


Okay, so how big exactly is the data on disk? You said removing
and adding a new node gives you 20GB on disk, was that done via
the '-Dcassandra.replace_address=...' parameter? If not, the new
node will almost certainly have a different token range and not
directly comparable to the existing node if you have uneven
partitions or small number of partitions in the table. Also, try
major compaction, it's a lot easier than replacing a node.


On 17/09/2021 12:28, Abdul Patel wrote:

Yes i checked and cleared all snapshots and also i had
incremental backups in backup folder ..i removed the same .. its
purely data..


On Friday, September 17, 2021, Bowen Song mailto:bo...@bso.ng>> wrote:

Assuming your total disk space is a lot bigger than 50GB in
size (accounting for disk space amplification, commit log,
logs, OS data, etc.), I would suspect the disk space is being
used by something else. Have you checked that the disk space
is actually being used by the cassandra data directory? If
so, have a look at 'nodetool listsnapshots' command output as
well.


On 17/09/2021 05:48, Abdul Patel wrote:

Hello

We have cassandra with leveledcompaction strategy,
recently found filesystem almost 90% full but the data
was only 10m records.
Manual compaction will work? As not sure its recommended
and space is also constraint ..tried removing and adding
one node and now data is at 20GB which looks appropropiate.
So is only solution to reclaim space is remove/add node?

Re: High disk usage casaandra 3.11.7

[Abdul Patel]

Close 300 gb data. Nodetool decommission/removenode and added back one node
ans it came back to 22Gb.
Cant run major compaction as no space much left.

On Friday, September 17, 2021, Bowen Song  wrote:

> Okay, so how big exactly is the data on disk? You said removing and adding
> a new node gives you 20GB on disk, was that done via the
> '-Dcassandra.replace_address=...' parameter? If not, the new node will
> almost certainly have a different token range and not directly comparable
> to the existing node if you have uneven partitions or small number of
> partitions in the table. Also, try major compaction, it's a lot easier than
> replacing a node.
>
>
> On 17/09/2021 12:28, Abdul Patel wrote:
>
> Yes i checked and cleared all snapshots and also i had incremental backups
> in backup folder ..i removed the same .. its purely data..
>
>
> On Friday, September 17, 2021, Bowen Song  wrote:
>
>> Assuming your total disk space is a lot bigger than 50GB in size
>> (accounting for disk space amplification, commit log, logs, OS data, etc.),
>> I would suspect the disk space is being used by something else. Have you
>> checked that the disk space is actually being used by the cassandra data
>> directory? If so, have a look at 'nodetool listsnapshots' command output as
>> well.
>>
>>
>> On 17/09/2021 05:48, Abdul Patel wrote:
>>
>>> Hello
>>>
>>> We have cassandra with leveledcompaction strategy, recently found
>>> filesystem almost 90% full but the data was only 10m records.
>>> Manual compaction will work? As not sure its recommended and space is
>>> also constraint ..tried removing and adding one node and now data is at
>>> 20GB which looks appropropiate.
>>> So is only solution to reclaim space is remove/add node?
>>>
>>

Re: High disk usage casaandra 3.11.7

[Dipan Shah]

Hello Abdul,

Adding to what Bowen already shared for snapshots.

Assuming that you're not just amplifying disk space by updating\deleting 
existing data many times, these are the following things that you should 
consider:

  *   Manual snapshots
 *   Check (nodetool listsnapshots) and remove (nodetool clearsnapshot) 
unwanted snapshots
  *   Automatic snapshots
 *   You can have unwanted snapshots if auto snapshot is enabled and you're 
frequently dropping, trucating or scrubbing tables. Check if that is the case
  *   Incremental backups
 *   Check if you have enabled incremental backups. Those files do not get 
deleted on their own and need to be cleaned out regularly
  *   Not running cleanup after adding new nodes to the cluster
 *   Check if you have recently added nodes to the cluster and missed 
running cleanups after that
  *   Compaction failing due to low disk space
 *   Cassandra will not be able to compact data (and free up space) if it 
does not have the required disk space to rewrite files. Check system.log for 
compaction error

Thanks,

Dipan Shah


From: Bowen Song 
Sent: Friday, September 17, 2021 4:53 PM
To: user@cassandra.apache.org 
Subject: Re: High disk usage casaandra 3.11.7

Assuming your total disk space is a lot bigger than 50GB in size
(accounting for disk space amplification, commit log, logs, OS data,
etc.), I would suspect the disk space is being used by something else.
Have you checked that the disk space is actually being used by the
cassandra data directory? If so, have a look at 'nodetool listsnapshots'
command output as well.


On 17/09/2021 05:48, Abdul Patel wrote:
> Hello
>
> We have cassandra with leveledcompaction strategy, recently found
> filesystem almost 90% full but the data was only 10m records.
> Manual compaction will work? As not sure its recommended and space is
> also constraint ..tried removing and adding one node and now data is
> at 20GB which looks appropropiate.
> So is only solution to reclaim space is remove/add node?

Re: High disk usage casaandra 3.11.7

[Bowen Song]

Okay, so how big exactly is the data on disk? You said removing and 
adding a new node gives you 20GB on disk, was that done via the 
'-Dcassandra.replace_address=...' parameter? If not, the new node will 
almost certainly have a different token range and not directly 
comparable to the existing node if you have uneven partitions or small 
number of partitions in the table. Also, try major compaction, it's a 
lot easier than replacing a node.



On 17/09/2021 12:28, Abdul Patel wrote:
Yes i checked and cleared all snapshots and also i had incremental 
backups in backup folder ..i removed the same .. its purely data..



On Friday, September 17, 2021, Bowen Song > wrote:


Assuming your total disk space is a lot bigger than 50GB in size
(accounting for disk space amplification, commit log, logs, OS
data, etc.), I would suspect the disk space is being used by
something else. Have you checked that the disk space is actually
being used by the cassandra data directory? If so, have a look at
'nodetool listsnapshots' command output as well.


On 17/09/2021 05:48, Abdul Patel wrote:

Hello

We have cassandra with leveledcompaction strategy, recently
found filesystem almost 90% full but the data was only 10m
records.
Manual compaction will work? As not sure its recommended and
space is also constraint ..tried removing and adding one node
and now data is at 20GB which looks appropropiate.
So is only solution to reclaim space is remove/add node?

Re: High disk usage casaandra 3.11.7

[Abdul Patel]

Yes i checked and cleared all snapshots and also i had incremental backups
in backup folder ..i removed the same .. its purely data..


On Friday, September 17, 2021, Bowen Song  wrote:

> Assuming your total disk space is a lot bigger than 50GB in size
> (accounting for disk space amplification, commit log, logs, OS data, etc.),
> I would suspect the disk space is being used by something else. Have you
> checked that the disk space is actually being used by the cassandra data
> directory? If so, have a look at 'nodetool listsnapshots' command output as
> well.
>
>
> On 17/09/2021 05:48, Abdul Patel wrote:
>
>> Hello
>>
>> We have cassandra with leveledcompaction strategy, recently found
>> filesystem almost 90% full but the data was only 10m records.
>> Manual compaction will work? As not sure its recommended and space is
>> also constraint ..tried removing and adding one node and now data is at
>> 20GB which looks appropropiate.
>> So is only solution to reclaim space is remove/add node?
>>
>

Re: High disk usage casaandra 3.11.7

[Bowen Song]

Assuming your total disk space is a lot bigger than 50GB in size 
(accounting for disk space amplification, commit log, logs, OS data, 
etc.), I would suspect the disk space is being used by something else. 
Have you checked that the disk space is actually being used by the 
cassandra data directory? If so, have a look at 'nodetool listsnapshots' 
command output as well.



On 17/09/2021 05:48, Abdul Patel wrote:

Hello

We have cassandra with leveledcompaction strategy, recently found 
filesystem almost 90% full but the data was only 10m records.
Manual compaction will work? As not sure its recommended and space is 
also constraint ..tried removing and adding one node and now data is 
at 20GB which looks appropropiate.

So is only solution to reclaim space is remove/add node?

Re: state of incremental repairs in cassandra 3.x

[Alexander DEJANOVSKI]

Hi James,

I'd recommend to upgrade to 4.0.1 if you intend to use incremental repair.
The changes from CASSANDRA-9143
 are massive and
couldn't be backported to the 3.11 branch.

When moving to incremental, and in order to limit anticompaction on the
first run, I'd recommend to:
- mark all sstables as repaired
- run a full repair
- schedule very regular (daily) incremental repairs

Bye,

Alex


Le jeu. 16 sept. 2021 à 23:03, C. Scott Andreas  a
écrit :

> Hi James, thanks for reaching out.
>
> A large number of fixes have landed for Incremental Repair in the 3.x
> series, though it's possible some may have been committed to 4.0 without a
> backport. Incremental repair works well on Cassandra 4.0.1. I'd start here
> to ensure you're picking up all fixes that went in, though I do think it's
> likely to work well on a recent 3.0.x build as well (I'm less familiar with
> the 3.11.x series).
>
> – Scott
>
> On Sep 16, 2021, at 1:02 PM, James Brown  wrote:
>
>
> There's been a lot of back and forth on the wider Internet and in this
> mailing list about whether incremental repairs are fatally flawed in
> Cassandra 3.x or whether they're still a good default. What's the current
> best thinking? The most recent 3.x documentation
>  still
> advocates in favor of using incremental repairs...
>
> CASSANDRA-9143  is
> marked as fixed in 4.0; did any improvements make it into any of the 3.11.x
> releases?
>
> If I need the performance of incremental repairs, should I just be
> plotting a 4.0.x upgrade?
>
> --
> James Brown
> Engineer
>
>
>