Re: Backups eating up disk space

[Vladimir Yudovin]

Yes, you can. It's just hardlinks to tables files, so if some file is still 
active it will remain intact.



Best regards, Vladimir Yudovin, 

Winguzone - Cloud Cassandra Hosting






 On Mon, 27 Feb 2017 09:27:50 -0500 Kunal Gangakhedkar 
kgangakhed...@gmail.com wrote 




Hi all,



Is it safe to delete the backup folders from various CFs from 'system' keyspace 
too?

I seem to have missed them in the last cleanup - and now, the size_estimates 
and compactions_in_progress seem to have grown large ( 200G and ~6G 
respectively).



Can I remove them too?



Thanks,

Kunal




On 13 January 2017 at 18:30, Kunal Gangakhedkar kgangakhed...@gmail.com 
wrote:








Great, thanks a lot to all for the help :)



I finally took the dive and went with Razi's suggestions.

In summary, this is what I did:

turn off incremental backups on each of the nodes in rolling fashion

remove the 'backups' directory from each keyspace on each node.

This ended up freeing up almost 350GB on each node - yay :)




Again, thanks a lot for the help, guys.



Kunal




On 12 January 2017 at 21:15, Khaja, Raziuddin (NIH/NLM/NCBI) [C] 
raziuddin.kh...@nih.gov wrote:

snapshots are slightly different than backups.

 

In my explanation of the hardlinks created in the backups folder, notice that 
compacted sstables, never end up in the backups folder.

 

On the other hand, a snapshot is meant to represent the data at a particular 
moment in time. Thus, the snapshots directory contains hardlinks to all active 
sstables at the time the snapshot was taken, which would include: compacted 
sstables; and any sstables from memtable flush or streamed from other nodes 
that both exist in the table directory and the backups directory.

 

So, that would be the difference between snapshots and backups.

 

Best regards,

-Razi

 

 

From:  Alain RODRIGUEZ arodr...@gmail.com
 Reply-To: "user@cassandra.apache.org" user@cassandra.apache.org
 Date: Thursday, January 12, 2017 at 9:16 AM


To: "user@cassandra.apache.org" user@cassandra.apache.org

 Subject: Re: Backups eating up disk space






 


My 2 cents, 

 


As I mentioned earlier, we're not currently using snapshots - it's only the 
backups that are bothering me right now.

 


I believe backups folder is just the new name for the previously called 
snapshots folder. But I can be completely wrong, I haven't played that much 
with snapshots in new versions yet.


 


Anyway, some operations in Apache Cassandra can trigger a snapshot:


 


- Repair (when not using parallel option but sequential repairs instead)


- Truncating a table (by default)


- Dropping a table (by default)


- Maybe other I can't think of... ?


 


If you want to clean space but still keep a backup you can run:


 


"nodetool clearsnapshots"


"nodetool snapshot whatever"


 


This way and for a while, data won't be taking space as old files will be 
cleaned and new files will be only hardlinks as detailed above. Then you might 
want to work at a proper backup policy, probably implying getting data out of 
production server (a lot of people uses S3 or similar services). Or just do 
that from time to time, meaning you only keep a backup and disk space behaviour 
will be hard to predict.


 


C*heers,


---


Alain Rodriguez - @arodream -  al...@thelastpickle.com


France


 


The Last Pickle - Apache Cassandra Consulting


http://www.thelastpickle.com




 

2017-01-12 6:42 GMT+01:00 Prasenjit Sarkar prasenjit.sar...@datos.io:

Hi Kunal, 

 


Razi's post does give a very lucid description of how cassandra manages the 
hard links inside the backup directory.


 


Where it needs clarification is the following:


-- incremental backups is a system wide setting and so its an all or 
nothing approach


 


-- as multiple people have stated, incremental backups do not create hard 
links to compacted sstables. however, this can bloat the size of your backups


 


-- again as stated, it is a general industry practice to place backups in a 
different secondary storage location than the main production site. So best to 
move it to the secondary storage before applying rm on the backups folder


 


In my experience with production clusters, managing the backups folder across 
multiple nodes can be painful if the objective is to ever recover data. With 
the usual disclaimers, better to rely on third party vendors to accomplish the 
needful rather than scripts/tablesnap.


 


Regards


Prasenjit 

 

 

On Wed, Jan 11, 2017 at 7:49 AM, Khaja, Raziuddin (NIH/NLM/NCBI) [C] 
raziuddin.kh...@nih.gov wrote:

Hello Kunal,

 

Caveat: I am not a super-expert on Cassandra, but it helps to explain to 
others, in order to eventually become an expert, so if my explanation is wrong, 
I would hope others would correct me. J

 

The active sstables/data files are are all the files located in the directory 
for the table.

You can safely remove all files under the backups/ directory and the directory 

Re: Is periodic manual repair necessary?

[Edward Capriolo]

There are 4 anti entropy systems in cassandra.

Hinted handoff
Read repair
Commit logs
Repair commamd

All are basically best effort.

Commit logs get corrupt and only flush periodically.

Bits rot on disk and while crossing networks network

Read repair is async and only happens randomly

Hinted handoff stops after some time and is not guarenteed.
On Monday, February 27, 2017, Thakrar, Jayesh 
wrote:

> Thanks Roth and Oskar for your quick responses.
>
>
>
> This is a single datacenter, multi-rack setup.
>
>
>
> > A TTL is technically similar to a delete - in the end both create
> tombstones.
>
> >If you want to eliminate the possibility of resurrected deleted data, you
> should run repairs.
>
> So why do I need to worry about data resurrection?
>
> Because, the TTL for the data is specified at the row level (atleast in
> this case) i.e. across ALL columns across ALL replicas.
>
> So they all will have the same data or wont have the data at all (i.e. it
> would have been tombstoned).
>
>
>
>
>
> > If you can guarantuee a 100% that data is read-repaired before
> gc_grace_seconds after the data has been TTL'ed, you won't need an extra
> repair.
>
> Why read-repaired before "gc_grace_period"?
>
> Isn't gc_grace_period the grace period for compaction to occur?
>
> So if the data was not consistent and read-repair happens before that,
> then well and good.
>
> Does read-repair not happen after gc/compaction?
>
> If this table has data being constantly/periodically inserted, then
> compaction will also happen accordingly, right?
>
>
>
> Thanks,
>
> Jayesh
>
>
>
>
>
> *From: *Benjamin Roth  >
> *Date: *Monday, February 27, 2017 at 11:53 AM
> *To: * >
> *Subject: *Re: Is periodic manual repair necessary?
>
>
>
> A TTL is technically similar to a delete - in the end both create
> tombstones.
>
> If you want to eliminate the possibility of resurrected deleted data, you
> should run repairs.
>
>
>
> If you can guarantuee a 100% that data is read-repaired before
> gc_grace_seconds after the data has been TTL'ed, you won't need an extra
> repair.
>
>
>
> 2017-02-27 18:29 GMT+01:00 Oskar Kjellin  >:
>
> Are you running multi dc?
>
> Skickat från min iPad
>
>
> 27 feb. 2017 kl. 16:08 skrev Thakrar, Jayesh  >:
>
> Suppose I have an application, where there are no deletes, only 5-10% of
> rows being occasionally updated (and that too only once) and a lot of reads.
>
>
>
> Furthermore, I have replication = 3 and both read and write are configured
> for local_quorum.
>
>
>
> Occasionally, servers do go into maintenance.
>
>
>
> I understand when the maintenance is longer than the period for
> hinted_handoffs to be preserved, they are lost and servers may have stale
> data.
>
> But I do expect it to be rectified on reads. If the stale data is not read
> again, I don’t care for it to be corrected as then the data will be
> automatically purged because of TTL.
>
>
>
> In such a situation, do I need to have a periodic (weekly?) manual/batch
> read_repair process?
>
>
>
> Thanks,
>
> Jayesh Thakrar
>
>
>
>
>
> --
>
> Benjamin Roth
> Prokurist
>
> Jaumo GmbH · www.jaumo.com
> Wehrstraße 46 · 73035 Göppingen · Germany
> Phone +49 7161 304880-6 · Fax +49 7161 304880-1
> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
>


-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Re: Is periodic manual repair necessary?

[Thakrar, Jayesh]

Thanks Roth and Oskar for your quick responses.

This is a single datacenter, multi-rack setup.

> A TTL is technically similar to a delete - in the end both create tombstones.
>If you want to eliminate the possibility of resurrected deleted data, you 
>should run repairs.
So why do I need to worry about data resurrection?
Because, the TTL for the data is specified at the row level (atleast in this 
case) i.e. across ALL columns across ALL replicas.
So they all will have the same data or wont have the data at all (i.e. it would 
have been tombstoned).


> If you can guarantuee a 100% that data is read-repaired before 
> gc_grace_seconds after the data has been TTL'ed, you won't need an extra 
> repair.
Why read-repaired before "gc_grace_period"?
Isn't gc_grace_period the grace period for compaction to occur?
So if the data was not consistent and read-repair happens before that, then 
well and good.
Does read-repair not happen after gc/compaction?
If this table has data being constantly/periodically inserted, then compaction 
will also happen accordingly, right?

Thanks,
Jayesh


From: Benjamin Roth 
Date: Monday, February 27, 2017 at 11:53 AM
To: 
Subject: Re: Is periodic manual repair necessary?

A TTL is technically similar to a delete - in the end both create tombstones.
If you want to eliminate the possibility of resurrected deleted data, you 
should run repairs.

If you can guarantuee a 100% that data is read-repaired before gc_grace_seconds 
after the data has been TTL'ed, you won't need an extra repair.

2017-02-27 18:29 GMT+01:00 Oskar Kjellin 
>:
Are you running multi dc?

Skickat från min iPad

27 feb. 2017 kl. 16:08 skrev Thakrar, Jayesh 
>:
Suppose I have an application, where there are no deletes, only 5-10% of rows 
being occasionally updated (and that too only once) and a lot of reads.

Furthermore, I have replication = 3 and both read and write are configured for 
local_quorum.

Occasionally, servers do go into maintenance.

I understand when the maintenance is longer than the period for hinted_handoffs 
to be preserved, they are lost and servers may have stale data.
But I do expect it to be rectified on reads. If the stale data is not read 
again, I don’t care for it to be corrected as then the data will be 
automatically purged because of TTL.

In such a situation, do I need to have a periodic (weekly?) manual/batch 
read_repair process?

Thanks,
Jayesh Thakrar



--
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Attached profiled data but need help understanding it

[Kant Kodali]

Hi! My answers are inline.

On Mon, Feb 27, 2017 at 11:48 AM, Kant Kodali  wrote:

>
>
> On Mon, Feb 27, 2017 at 10:30 AM, Romain Hardouin 
> wrote:
>
>> Hi,
>>
>> Regarding shared pool workers see CASSANDRA-11966. You may have to
>> backport it depending on your Cassandra version.
>>
>
> *we are currently using 3.0.9.  should we use 3.8 or 3.10?*
>
>>
>> Did you try to lower compaction throughput to see if it helps? Be sure to
>> keep an eye on pending compactions, SSTables count and SSTable per read of
>> course.
>>
>
>*I haven't done any tuning yet. Can I change this on a running
> instance? If so, how? or does it require a downtime?*
>
>>
>> "alloc" is the memory allocation rate. You can see that compactions are
>> GC intensive.
>>
>
>> You won't be able to achieve impressive writes/s with LWT. But maybe
>> there is room for improvement. Try GC tuning, different pool sizes for NTR,
>> concurrent reads and writes, compaction executors, etc. Also check if you
>> can improve network latency (e.g. VF or ENA on AWS).
>>
>
>* GC seems to be fine because when I checked GC is about 0.25%. Total
> GC time is about 6minutes since the node is up and running for about 50
> hours*.
>
>>
>> What LWT rate would you want to achieve?  *5000 is the target.*
>>
>> Best,
>>
>> Romain
>>
>>
>>
>> Le Lundi 27 février 2017 12h48, Kant Kodali  a écrit :
>>
>>
>> Also Attached is a flamed graph generated from a thread dump.
>>
>> On Mon, Feb 27, 2017 at 2:32 AM, Kant Kodali  wrote:
>>
>> Hi,
>>
>> Attached are the stats of my Cassandra node running on a 4-core CPU. I am
>> using sjk-plus tool for the first time so what are the things I should
>> watched out for in my attached screenshot? I can see the CPU is almost
>> maxed out but should I say that is because of compaction or
>> shared-worker-pool threads (which btw, I dont know what they are doing
>> perhaps I need to take threadump)? Also what is alloc for each thread?
>>
>> I have a insert heavy workload (almost like an ingest running against
>> cassandra cluster) and in my case all writes are LWT.
>>
>> The current throughput is 1500 writes/sec where each write is about 1KB.
>> How can I tune something for a higher throughput? Any pointers or
>> suggestions would help.
>>
>> Thanks much,
>> kant
>>
>>
>>
>>
>>
>

Re: Attached profiled data but need help understanding it

[Kant Kodali]

On Mon, Feb 27, 2017 at 10:30 AM, Romain Hardouin 
wrote:

> Hi,
>
> Regarding shared pool workers see CASSANDRA-11966. You may have to
> backport it depending on your Cassandra version.
>

*we are currently using 3.0.9.  should we use 3.8 or 3.10?*

>
> Did you try to lower compaction throughput to see if it helps? Be sure to
> keep an eye on pending compactions, SSTables count and SSTable per read of
> course.
>

   *I haven't done any tuning yet. Can I change this on a running instance?
If so, how? or does it require a downtime?*

>
> "alloc" is the memory allocation rate. You can see that compactions are GC
> intensive.
>

> You won't be able to achieve impressive writes/s with LWT. But maybe there
> is room for improvement. Try GC tuning, different pool sizes for NTR,
> concurrent reads and writes, compaction executors, etc. Also check if you
> can improve network latency (e.g. VF or ENA on AWS).
>

   * GC seems to be fine because when I checked GC is about 0.25%. Total GC
time is about 6minutes since the node is up and running for about 50 hours*.

>
> What LWT rate would you want to achieve?  *5000 is the target.*
>
> Best,
>
> Romain
>
>
>
> Le Lundi 27 février 2017 12h48, Kant Kodali  a écrit :
>
>
> Also Attached is a flamed graph generated from a thread dump.
>
> On Mon, Feb 27, 2017 at 2:32 AM, Kant Kodali  wrote:
>
> Hi,
>
> Attached are the stats of my Cassandra node running on a 4-core CPU. I am
> using sjk-plus tool for the first time so what are the things I should
> watched out for in my attached screenshot? I can see the CPU is almost
> maxed out but should I say that is because of compaction or
> shared-worker-pool threads (which btw, I dont know what they are doing
> perhaps I need to take threadump)? Also what is alloc for each thread?
>
> I have a insert heavy workload (almost like an ingest running against
> cassandra cluster) and in my case all writes are LWT.
>
> The current throughput is 1500 writes/sec where each write is about 1KB.
> How can I tune something for a higher throughput? Any pointers or
> suggestions would help.
>
> Thanks much,
> kant
>
>
>
>
>

Re: Attached profiled data but need help understanding it

[Romain Hardouin]

Hi,
Regarding shared pool workers see CASSANDRA-11966. You may have to backport it 
depending on your Cassandra version. 
Did you try to lower compaction throughput to see if it helps? Be sure to keep 
an eye on pending compactions, SSTables count and SSTable per read of course.
"alloc" is the memory allocation rate. You can see that compactions are GC 
intensive.
You won't be able to achieve impressive writes/s with LWT. But maybe there is 
room for improvement. Try GC tuning, different pool sizes for NTR, concurrent 
reads and writes, compaction executors, etc. Also check if you can improve 
network latency (e.g. VF or ENA on AWS).
What LWT rate would you want to achieve?
Best,
Romain
 

Le Lundi 27 février 2017 12h48, Kant Kodali  a écrit :
 

 Also Attached is a flamed graph generated from a thread dump.
On Mon, Feb 27, 2017 at 2:32 AM, Kant Kodali  wrote:

Hi,
Attached are the stats of my Cassandra node running on a 4-core CPU. I am using 
sjk-plus tool for the first time so what are the things I should watched out 
for in my attached screenshot? I can see the CPU is almost maxed out but should 
I say that is because of compaction or shared-worker-pool threads (which btw, I 
dont know what they are doing perhaps I need to take threadump)? Also what is 
alloc for each thread? 
I have a insert heavy workload (almost like an ingest running against cassandra 
cluster) and in my case all writes are LWT.
The current throughput is 1500 writes/sec where each write is about 1KB. How 
can I tune something for a higher throughput? Any pointers or suggestions would 
help.

Thanks much,kant



   

Re: secondary index on static column

[Romain Hardouin]

Hi,
Sorry for the delay, I created a ticket with steps to reproduce the issue: 
https://issues.apache.org/jira/browse/CASSANDRA-13277
Best,
Romain
 

Le Jeudi 2 février 2017 16h53, Micha  a écrit :
 

 Hi,

it's a 3.9, installed on a jessie system.

For me it's like this:
I have a three node cluster.
When creating the keyspace with replication factor 3 it works.
When creating the keyspace with replication factor 2 it doesn't work and
shows the weird behavior.

This is a fresh install, I also have tried it multiple times and the
result is the same. As SASI indices work, I use those.
But I would like to solve this.

Cheers,
 Michael





On 02.02.2017 15:06, Romain Hardouin wrote:
> Hi,
> 
> What's your C* 3.X version?
> I've just tested it on 3.9 and it works:
> 
> cqlsh> SELECT * FROM test.idx_static where id2=22;
> 
>  id  | added                          | id2 | source | dest
> -+-+-++--
>  id1 | 2017-01-27 23:00:00.00+ |  22 |  src1 | dst1
> 
> (1 rows)
> 
> Maybe your dataset is incorrect (try on a new table) or you hit a bug.
> 
> Best,
> 
> Romain
> 
> 
> 
> Le Vendredi 27 janvier 2017 9h44, Micha  a écrit :
> 
> 
> Hi,
> 
> I'm quite new to cassandra and allthough there is much info on the net,
> sometimes I cannot find the solution to a problem.
> 
> In this case, I have a second index on a static column and I don't
> understand the answer I get from my select.
> 
> A cut down version of the table is:
> 
> create table demo (id text, id2 bigint static, added timestamp, source
> text static, dest text, primary key (id, added));
> 
> create index on demo (id2);
> 
> id and id2 match one to one.
> 
> I make one insert:
> insert into demo (id, id2, added, source, dest) values ('id1', 22,
> '2017-01-28', 'src1', 'dst1');
> 
> 
> The "select from demo;" gives the expected answer of the one inserted row.
> 
> But "select from demo where id2=22" gives 70 rows as result (all the same).
> 
> Why? I have read
> https://www.datastax.com/dev/blog/cassandra-native-secondary-index-deep-dive
> 
> but I don't get it...
> 
> thanks for answering,
> Michael
> 
> 
> 


   

Re: Is periodic manual repair necessary?

[Benjamin Roth]

A TTL is technically similar to a delete - in the end both create
tombstones.
If you want to eliminate the possibility of resurrected deleted data, you
should run repairs.

If you can guarantuee a 100% that data is read-repaired before
gc_grace_seconds after the data has been TTL'ed, you won't need an extra
repair.

2017-02-27 18:29 GMT+01:00 Oskar Kjellin :

> Are you running multi dc?
>
> Skickat från min iPad
>
> 27 feb. 2017 kl. 16:08 skrev Thakrar, Jayesh  >:
>
> Suppose I have an application, where there are no deletes, only 5-10% of
> rows being occasionally updated (and that too only once) and a lot of reads.
>
>
>
> Furthermore, I have replication = 3 and both read and write are configured
> for local_quorum.
>
>
>
> Occasionally, servers do go into maintenance.
>
>
>
> I understand when the maintenance is longer than the period for
> hinted_handoffs to be preserved, they are lost and servers may have stale
> data.
>
> But I do expect it to be rectified on reads. If the stale data is not read
> again, I don’t care for it to be corrected as then the data will be
> automatically purged because of TTL.
>
>
>
> In such a situation, do I need to have a periodic (weekly?) manual/batch
> read_repair process?
>
>
>
> Thanks,
>
> Jayesh Thakrar
>
>


-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Is periodic manual repair necessary?

[Oskar Kjellin]

Are you running multi dc?

Skickat från min iPad

> 27 feb. 2017 kl. 16:08 skrev Thakrar, Jayesh :
> 
> Suppose I have an application, where there are no deletes, only 5-10% of rows 
> being occasionally updated (and that too only once) and a lot of reads.
>  
> Furthermore, I have replication = 3 and both read and write are configured 
> for local_quorum.
>  
> Occasionally, servers do go into maintenance.
>  
> I understand when the maintenance is longer than the period for 
> hinted_handoffs to be preserved, they are lost and servers may have stale 
> data.
> But I do expect it to be rectified on reads. If the stale data is not read 
> again, I don’t care for it to be corrected as then the data will be 
> automatically purged because of TTL.
>  
> In such a situation, do I need to have a periodic (weekly?) manual/batch 
> read_repair process?
>  
> Thanks,
> Jayesh Thakrar

Re: Which compaction strategy when modeling a dumb set

[Benjamin Roth]

This is not a queue pattern and I'd recommend LCS for better read
performance.

2017-02-27 16:06 GMT+01:00 Rakesh Kumar :

> Do you update this table when an event is processed?  If yes, it is
> considered a good practice for Cassandra.  I read somewhere that using
> Cassandra as a queuing table is anti pattern.
> 
> From: Vincent Rischmann 
> Sent: Friday, February 24, 2017 06:24
> To: user@cassandra.apache.org
> Subject: Which compaction strategy when modeling a dumb set
>
> Hello,
>
> I'm using a table like this:
>
>CREATE TABLE myset (id uuid PRIMARY KEY)
>
> which is basically a set I use for deduplication, id is a unique id for an
> event, when I process the event I insert the id, and before processing I
> check if it has already been processed for deduplication.
>
> It works well enough, but I'm wondering which compaction strategy I should
> use. I expect maybe 1% or less of events will end up duplicated (thus not
> generating an insert), so the workload will probably be 50% writes 50% read.
>
> Is LCS a good strategy here or should I stick with STCS ?
>



-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Which compaction strategy when modeling a dumb set

[Rakesh Kumar]

typo: " If yes, it is considered a good practice for Cassandra"
should read as
" If yes, is it considered a good practice for Cassandra ?" 

From: Rakesh Kumar 
Sent: Monday, February 27, 2017 10:06
To: user@cassandra.apache.org
Subject: Re: Which compaction strategy when modeling a dumb set

Do you update this table when an event is processed?  If yes, it is considered 
a good practice for Cassandra.  I read somewhere that using Cassandra as a 
queuing table is anti pattern.

From: Vincent Rischmann 
Sent: Friday, February 24, 2017 06:24
To: user@cassandra.apache.org
Subject: Which compaction strategy when modeling a dumb set

Hello,

I'm using a table like this:

   CREATE TABLE myset (id uuid PRIMARY KEY)

which is basically a set I use for deduplication, id is a unique id for an 
event, when I process the event I insert the id, and before processing I check 
if it has already been processed for deduplication.

It works well enough, but I'm wondering which compaction strategy I should use. 
I expect maybe 1% or less of events will end up duplicated (thus not generating 
an insert), so the workload will probably be 50% writes 50% read.

Is LCS a good strategy here or should I stick with STCS ?

Is periodic manual repair necessary?

[Thakrar, Jayesh]

Suppose I have an application, where there are no deletes, only 5-10% of rows 
being occasionally updated (and that too only once) and a lot of reads.

Furthermore, I have replication = 3 and both read and write are configured for 
local_quorum.

Occasionally, servers do go into maintenance.

I understand when the maintenance is longer than the period for hinted_handoffs 
to be preserved, they are lost and servers may have stale data.
But I do expect it to be rectified on reads. If the stale data is not read 
again, I don’t care for it to be corrected as then the data will be 
automatically purged because of TTL.

In such a situation, do I need to have a periodic (weekly?) manual/batch 
read_repair process?

Thanks,
Jayesh Thakrar

Re: Which compaction strategy when modeling a dumb set

[Rakesh Kumar]

Do you update this table when an event is processed?  If yes, it is considered 
a good practice for Cassandra.  I read somewhere that using Cassandra as a 
queuing table is anti pattern.

From: Vincent Rischmann 
Sent: Friday, February 24, 2017 06:24
To: user@cassandra.apache.org
Subject: Which compaction strategy when modeling a dumb set

Hello,

I'm using a table like this:

   CREATE TABLE myset (id uuid PRIMARY KEY)

which is basically a set I use for deduplication, id is a unique id for an 
event, when I process the event I insert the id, and before processing I check 
if it has already been processed for deduplication.

It works well enough, but I'm wondering which compaction strategy I should use. 
I expect maybe 1% or less of events will end up duplicated (thus not generating 
an insert), so the workload will probably be 50% writes 50% read.

Is LCS a good strategy here or should I stick with STCS ?

Re: Backups eating up disk space

[Kunal Gangakhedkar]

Hi all,

Is it safe to delete the backup folders from various CFs from 'system'
keyspace too?
I seem to have missed them in the last cleanup - and now, the
size_estimates and compactions_in_progress seem to have grown large ( >200G
and ~6G respectively).

Can I remove them too?

Thanks,
Kunal

On 13 January 2017 at 18:30, Kunal Gangakhedkar 
wrote:

> Great, thanks a lot to all for the help :)
>
> I finally took the dive and went with Razi's suggestions.
> In summary, this is what I did:
>
>- turn off incremental backups on each of the nodes in rolling fashion
>- remove the 'backups' directory from each keyspace on each node.
>
> This ended up freeing up almost 350GB on each node - yay :)
>
> Again, thanks a lot for the help, guys.
>
> Kunal
>
> On 12 January 2017 at 21:15, Khaja, Raziuddin (NIH/NLM/NCBI) [C] <
> raziuddin.kh...@nih.gov> wrote:
>
>> snapshots are slightly different than backups.
>>
>>
>>
>> In my explanation of the hardlinks created in the backups folder, notice
>> that compacted sstables, never end up in the backups folder.
>>
>>
>>
>> On the other hand, a snapshot is meant to represent the data at a
>> particular moment in time. Thus, the snapshots directory contains hardlinks
>> to all active sstables at the time the snapshot was taken, which would
>> include: compacted sstables; and any sstables from memtable flush or
>> streamed from other nodes that both exist in the table directory and the
>> backups directory.
>>
>>
>>
>> So, that would be the difference between snapshots and backups.
>>
>>
>>
>> Best regards,
>>
>> -Razi
>>
>>
>>
>>
>>
>> *From: *Alain RODRIGUEZ 
>> *Reply-To: *"user@cassandra.apache.org" 
>> *Date: *Thursday, January 12, 2017 at 9:16 AM
>>
>> *To: *"user@cassandra.apache.org" 
>> *Subject: *Re: Backups eating up disk space
>>
>>
>>
>> My 2 cents,
>>
>>
>>
>> As I mentioned earlier, we're not currently using snapshots - it's only
>> the backups that are bothering me right now.
>>
>>
>>
>> I believe backups folder is just the new name for the previously called
>> snapshots folder. But I can be completely wrong, I haven't played that much
>> with snapshots in new versions yet.
>>
>>
>>
>> Anyway, some operations in Apache Cassandra can trigger a snapshot:
>>
>>
>>
>> - Repair (when not using parallel option but sequential repairs instead)
>>
>> - Truncating a table (by default)
>>
>> - Dropping a table (by default)
>>
>> - Maybe other I can't think of... ?
>>
>>
>>
>> If you want to clean space but still keep a backup you can run:
>>
>>
>>
>> "nodetool clearsnapshots"
>>
>> "nodetool snapshot "
>>
>>
>>
>> This way and for a while, data won't be taking space as old files will be
>> cleaned and new files will be only hardlinks as detailed above. Then you
>> might want to work at a proper backup policy, probably implying getting
>> data out of production server (a lot of people uses S3 or similar
>> services). Or just do that from time to time, meaning you only keep a
>> backup and disk space behaviour will be hard to predict.
>>
>>
>>
>> C*heers,
>>
>> ---
>>
>> Alain Rodriguez - @arodream - al...@thelastpickle.com
>>
>> France
>>
>>
>>
>> The Last Pickle - Apache Cassandra Consulting
>>
>> http://www.thelastpickle.com
>>
>>
>>
>> 2017-01-12 6:42 GMT+01:00 Prasenjit Sarkar :
>>
>> Hi Kunal,
>>
>>
>>
>> Razi's post does give a very lucid description of how cassandra manages
>> the hard links inside the backup directory.
>>
>>
>>
>> Where it needs clarification is the following:
>>
>> --> incremental backups is a system wide setting and so its an all or
>> nothing approach
>>
>>
>>
>> --> as multiple people have stated, incremental backups do not create
>> hard links to compacted sstables. however, this can bloat the size of your
>> backups
>>
>>
>>
>> --> again as stated, it is a general industry practice to place backups
>> in a different secondary storage location than the main production site. So
>> best to move it to the secondary storage before applying rm on the backups
>> folder
>>
>>
>>
>> In my experience with production clusters, managing the backups folder
>> across multiple nodes can be painful if the objective is to ever recover
>> data. With the usual disclaimers, better to rely on third party vendors to
>> accomplish the needful rather than scripts/tablesnap.
>>
>>
>>
>> Regards
>>
>> Prasenjit
>>
>>
>>
>>
>>
>> On Wed, Jan 11, 2017 at 7:49 AM, Khaja, Raziuddin (NIH/NLM/NCBI) [C] <
>> raziuddin.kh...@nih.gov> wrote:
>>
>> Hello Kunal,
>>
>>
>>
>> Caveat: I am not a super-expert on Cassandra, but it helps to explain to
>> others, in order to eventually become an expert, so if my explanation is
>> wrong, I would hope others would correct me. J
>>
>>
>>
>> The active sstables/data files are are all the files located in the
>> directory for the table.
>>
>> You can safely remove all files under the 

Re: Unsubscribe

[Ashish Disawal]

Please send mail to.
user-unsubscr...@cassandra.apache.org

--
Ashish Disawal

On Mon, Feb 27, 2017 at 5:57 PM, Sujeet Kumar 
wrote:

>

Re: Unsubscribe

[Ashish Disawal]

Please send mail to.
user-unsubscr...@cassandra.apache.org

--
Ashish Disawal

2017-02-27 17:58 GMT+05:30 ROUVREAU Eric - externe <
eric-externe.rouvr...@enedis.fr>:

>
>
>
>
> Cordialement,
>
> *Eric **ROUVREAU*
>
> Prestataire Expert technique SGDB - CDC Middleware
>
> *Enedis* - Direction des Systèmes d'Information
> Pôle AUDES - Opérateur Informatique - Département Ingénierie et
> Infrastructures - (DIGIT) - Groupe Infrastructures applicatives
> 117 bd Marius Vivier Merle - 69329 Lyon Cedex 03
> +33(0) 4 37 23 35 00 <+33%204%2037%2023%2035%2000>
> *eric-externe.rouvr...@enedis.fr *
>
> *Merci de penser à l'environnement avant d'imprimer ce message*
>
> 
> 
> --
> Ce message est destiné exclusivement aux personnes ou entités auxquelles
> il est adressé et peut contenir des informations privilégiées ou
> confidentielles. Si vous avez reçu ce document par erreur, merci de nous
> l'indiquer par retour et procéder à sa destruction.
> This message is intended for the use of the individual or entity to whom
> it is addressed and may contain information, that is privileged or
> confidential. If you have received this communication by mistake, please
> notify us immediately by electronic mail, and delete the original message.
>
>
>
>
>

Re: Which compaction strategy when modeling a dumb set

[Vincent Rischmann]

No I don't store events in Cassandra.



The real thing I'm doing is couting stuff: each event has a type, a user
associated with it, some other metadata. When I process an event I need
to increment those counters only if the event hasn't already been
processed. Our input event stream is Kafka and it's not uncommon that we
get the same event twice, due to our clients app not being reliable.


Right now I haven't found a good solution to this that doesn't involve a
read before write, but I'd love to hear your suggestions




On Mon, Feb 27, 2017, at 12:01 PM, Vladimir Yudovin wrote:

> Do you also store events in Cassandra? If yes, why not to add
> "processed" flag to existing table(s), and fetch non-processed events
> with single SELECT?
> 

> Best regards, Vladimir Yudovin, 

> *Winguzone[1] - Cloud Cassandra Hosting*

> 

> 

>  On Fri, 24 Feb 2017 06:24:09 -0500 *Vincent Rischmann
> * wrote 
> 

>> Hello,

>> 

>> I'm using a table like this:

>> 

>>CREATE TABLE myset (id uuid PRIMARY KEY)

>> 

>> which is basically a set I use for deduplication, id is a unique id
>> for an event, when I process the event I insert the id, and before
>> processing I check if it has already been processed for
>> deduplication.
>> 

>> It works well enough, but I'm wondering which compaction strategy I
>> should use. I expect maybe 1% or less of events will end up
>> duplicated (thus not generating an insert), so the workload will
>> probably be 50% writes 50% read.
>> 

>> Is LCS a good strategy here or should I stick with STCS ?

> 




Links:

  1. https://winguzone.com?from=list

Unsubscribe

[ROUVREAU Eric - externe]

Cordialement,
Eric  ROUVREAU
Prestataire Expert technique SGDB - CDC Middleware
Enedis - Direction des Systèmes d'Information
Pôle AUDES - Opérateur Informatique - Département Ingénierie et Infrastructures 
- (DIGIT) - Groupe Infrastructures applicatives
117 bd Marius Vivier Merle - 69329 Lyon Cedex 03
+33(0) 4 37 23 35 00
eric-externe.rouvr...@enedis.fr
Merci de penser à l'environnement avant d'imprimer ce message
--
Ce message est destiné exclusivement aux personnes ou entités auxquelles il est 
adressé et peut contenir des informations privilégiées ou confidentielles. Si 
vous avez reçu ce document par erreur, merci de nous l'indiquer par retour et 
procéder à sa destruction.
This message is intended for the use of the individual or entity to whom it is 
addressed and may contain information, that is privileged or confidential. If 
you have received this communication by mistake, please notify us immediately 
by electronic mail, and delete the original message.

Unsubscribe

[Sujeet Kumar]

Re: Attached profiled data but need help understanding it

[Kant Kodali]

Also Attached is a flamed graph generated from a thread dump.

On Mon, Feb 27, 2017 at 2:32 AM, Kant Kodali  wrote:

> Hi,
>
> Attached are the stats of my Cassandra node running on a 4-core CPU. I am
> using sjk-plus tool for the first time so what are the things I should
> watched out for in my attached screenshot? I can see the CPU is almost
> maxed out but should I say that is because of compaction or
> shared-worker-pool threads (which btw, I dont know what they are doing
> perhaps I need to take threadump)? Also what is alloc for each thread?
>
> I have a insert heavy workload (almost like an ingest running against
> cassandra cluster) and in my case all writes are LWT.
>
> The current throughput is 1500 writes/sec where each write is about 1KB.
> How can I tune something for a higher throughput? Any pointers or
> suggestions would help.
>
> Thanks much,
> kant
>

Re: Which compaction strategy when modeling a dumb set

[Vladimir Yudovin]

Do you also store events in Cassandra? If yes, why not to add "processed" flag 
to existing table(s), and fetch non-processed events with single SELECT?



Best regards, Vladimir Yudovin, 

Winguzone - Cloud Cassandra Hosting






 On Fri, 24 Feb 2017 06:24:09 -0500 Vincent Rischmann 
m...@vrischmann.me wrote 




Hello,



I'm using a table like this:



   CREATE TABLE myset (id uuid PRIMARY KEY)



which is basically a set I use for deduplication, id is a unique id for an 
event, when I process the event I insert the id, and before processing I check 
if it has already been processed for deduplication.



It works well enough, but I'm wondering which compaction strategy I should use. 
I expect maybe 1% or less of events will end up duplicated (thus not generating 
an insert), so the workload will probably be 50% writes 50% read.



Is LCS a good strategy here or should I stick with STCS ?