Re: need to reclaim space with TWCS

[brian . spindler]

Got it.  Thanks again. 

> On Jan 20, 2018, at 11:17 AM, Alexander Dejanovski  
> wrote:
> 
> I would turn background read repair off on the table to improve the overlap 
> issue, but you'll still have foreground read repair if you use quorum reads 
> anyway.
> 
> So put dclocal_... to 0.0.
> 
> The commit you're referring to has been merged in 3.11.1 as 2.1 doesn't 
> patched anymore.
> 
> 
>> Le sam. 20 janv. 2018 à 16:55, Brian Spindler  a 
>> écrit :
>> Hi Alexander, after re-reading this 
>> https://issues.apache.org/jira/browse/CASSANDRA-13418 it seems you would 
>> recommend leaving dclocal_read_repair at maybe 10%  is that true?  
>> 
>> Also, has this been patched to 2.1?  
>> https://github.com/thelastpickle/cassandra/commit/58440e707cd6490847a37dc8d76c150d3eb27aab#diff-e8e282423dcbf34d30a3578c8dec15cdR176
>>  
>> 
>> Cheers, 
>> 
>> -B
>> 
>> 
>>> On Sat, Jan 20, 2018 at 10:49 AM Brian Spindler  
>>> wrote:
>>> Hi Alexander,  Thanks for your response!  I'll give it a shot.
>>> 
 On Sat, Jan 20, 2018 at 10:22 AM Alexander Dejanovski 
  wrote:
 Hi Brian,
 
 You should definitely set unchecked_tombstone_compaction to true and set 
 the interval to the default of 1 day. Use a tombstone_threshold of 0.6 for 
 example and see how that works.
 Tombstones will get purged depending on your partitioning as their 
 partition needs to be fully contained within a single sstable.
 
 Deleting the sstables by hand is theoretically possible but should be kept 
 as a last resort option if you're running out of space.
 
 Cheers,
 
 
> Le sam. 20 janv. 2018 à 15:41, Brian Spindler  
> a écrit :
> I probably should have mentioned our setup: we’re on Cassandra version 
> 2.1.15.
> 
> 
>> On Sat, Jan 20, 2018 at 9:33 AM Brian Spindler 
>>  wrote:
>> Hi, I have several column families using TWCS and it’s great.  
>> Unfortunately we seem to have missed the great advice in Alex’s article 
>> here: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html about 
>> setting the appropriate aggressive tombstone settings and now we have 
>> lots of timestamp overlaps and disk space to reclaim. 
>>  
>> I am trying to figure the best way out of this. Lots of the SSTables 
>> with overlapping timestamps in newer SSTables have droppable tombstones 
>> at like 0.895143957 or something similar, very close to 0.90 where the 
>> full sstable will drop afaik.  
>>  
>> I’m thinking to do the following immediately:
>>  
>> Set unchecked_tombstone_compaction = true
>> Set tombstone_compaction_interval == TTL + gc_grace_seconds
>> Set dclocal_read_repair_chance = 0.0 (currently 0.1)
>>  
>> If I do this, can I expect TWCS/C* to reclaim the space from those 
>> SSTables with 0.89* droppable tombstones?   Or do I (can I?) manually 
>> delete these files and will c* just ignore the overlapping data and 
>> treat as tombstoned?  
>>  
>> What else should/could be done? 
>>  
>> Thank you in advance for your advice,
>>  
>> __
>> Brian Spindler 
>>  
>>  
 
 -- 
 -
 Alexander Dejanovski
 France
 @alexanderdeja
 
 Consultant
 Apache Cassandra Consulting
 http://www.thelastpickle.com
> 
> -- 
> -
> Alexander Dejanovski
> France
> @alexanderdeja
> 
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com

Re: need to reclaim space with TWCS

[Alexander Dejanovski]

I would turn background read repair off on the table to improve the overlap
issue, but you'll still have foreground read repair if you use quorum reads
anyway.

So put dclocal_... to 0.0.

The commit you're referring to has been merged in 3.11.1 as 2.1 doesn't
patched anymore.

Le sam. 20 janv. 2018 à 16:55, Brian Spindler  a
écrit :

> Hi Alexander, after re-reading this
> https://issues.apache.org/jira/browse/CASSANDRA-13418 it seems you would
> recommend leaving dclocal_read_repair at maybe 10%  is that true?
>
> Also, has this been patched to 2.1?
> https://github.com/thelastpickle/cassandra/commit/58440e707cd6490847a37dc8d76c150d3eb27aab#diff-e8e282423dcbf34d30a3578c8dec15cdR176
>
>
> Cheers,
>
> -B
>
>
> On Sat, Jan 20, 2018 at 10:49 AM Brian Spindler 
> wrote:
>
>> Hi Alexander,  Thanks for your response!  I'll give it a shot.
>>
>> On Sat, Jan 20, 2018 at 10:22 AM Alexander Dejanovski <
>> a...@thelastpickle.com> wrote:
>>
>>> Hi Brian,
>>>
>>> You should definitely set unchecked_tombstone_compaction to true and set
>>> the interval to the default of 1 day. Use a tombstone_threshold of 0.6 for
>>> example and see how that works.
>>> Tombstones will get purged depending on your partitioning as their
>>> partition needs to be fully contained within a single sstable.
>>>
>>> Deleting the sstables by hand is theoretically possible but should be
>>> kept as a last resort option if you're running out of space.
>>>
>>> Cheers,
>>>
>>> Le sam. 20 janv. 2018 à 15:41, Brian Spindler 
>>> a écrit :
>>>
 I probably should have mentioned our setup: we’re on Cassandra version
 2.1.15.


 On Sat, Jan 20, 2018 at 9:33 AM Brian Spindler <
 brian.spind...@gmail.com> wrote:

> Hi, I have several column families using TWCS and it’s great.
> Unfortunately we seem to have missed the great advice in Alex’s article
> here: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html about
> setting the appropriate aggressive tombstone settings and now we have lots
> of timestamp overlaps and disk space to reclaim.
>
>
>
> I am trying to figure the best way out of this. Lots of the SSTables
> with overlapping timestamps in newer SSTables have droppable tombstones at
> like 0.895143957 or something similar, very close to 0.90 where the full
> sstable will drop afaik.
>
>
>
> I’m thinking to do the following immediately:
>
>
>
> Set *unchecked_tombstone_compaction = true*
>
> Set* tombstone_compaction_interval == TTL + gc_grace_seconds*
>
> Set* dclocal_read_repair_chance = 0.0 (currently 0.1)*
>
>
>
> If I do this, can I expect TWCS/C* to reclaim the space from those
> SSTables with 0.89* droppable tombstones?   Or do I (can I?) manually
> delete these files and will c* just ignore the overlapping data and treat
> as tombstoned?
>
>
>
> What else should/could be done?
>
>
>
> Thank you in advance for your advice,
>
>
>
> *__*
>
> *Brian Spindler *
>
>
>
>
>
 --
>>> -
>>> Alexander Dejanovski
>>> France
>>> @alexanderdeja
>>>
>>> Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: need to reclaim space with TWCS

[Brian Spindler]

Hi Alexander, after re-reading this
https://issues.apache.org/jira/browse/CASSANDRA-13418 it seems you would
recommend leaving dclocal_read_repair at maybe 10%  is that true?

Also, has this been patched to 2.1?
https://github.com/thelastpickle/cassandra/commit/58440e707cd6490847a37dc8d76c150d3eb27aab#diff-e8e282423dcbf34d30a3578c8dec15cdR176


Cheers,

-B


On Sat, Jan 20, 2018 at 10:49 AM Brian Spindler 
wrote:

> Hi Alexander,  Thanks for your response!  I'll give it a shot.
>
> On Sat, Jan 20, 2018 at 10:22 AM Alexander Dejanovski <
> a...@thelastpickle.com> wrote:
>
>> Hi Brian,
>>
>> You should definitely set unchecked_tombstone_compaction to true and set
>> the interval to the default of 1 day. Use a tombstone_threshold of 0.6 for
>> example and see how that works.
>> Tombstones will get purged depending on your partitioning as their
>> partition needs to be fully contained within a single sstable.
>>
>> Deleting the sstables by hand is theoretically possible but should be
>> kept as a last resort option if you're running out of space.
>>
>> Cheers,
>>
>> Le sam. 20 janv. 2018 à 15:41, Brian Spindler 
>> a écrit :
>>
>>> I probably should have mentioned our setup: we’re on Cassandra version
>>> 2.1.15.
>>>
>>>
>>> On Sat, Jan 20, 2018 at 9:33 AM Brian Spindler 
>>> wrote:
>>>
 Hi, I have several column families using TWCS and it’s great.
 Unfortunately we seem to have missed the great advice in Alex’s article
 here: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html about
 setting the appropriate aggressive tombstone settings and now we have lots
 of timestamp overlaps and disk space to reclaim.



 I am trying to figure the best way out of this. Lots of the SSTables
 with overlapping timestamps in newer SSTables have droppable tombstones at
 like 0.895143957 or something similar, very close to 0.90 where the full
 sstable will drop afaik.



 I’m thinking to do the following immediately:



 Set *unchecked_tombstone_compaction = true*

 Set* tombstone_compaction_interval == TTL + gc_grace_seconds*

 Set* dclocal_read_repair_chance = 0.0 (currently 0.1)*



 If I do this, can I expect TWCS/C* to reclaim the space from those
 SSTables with 0.89* droppable tombstones?   Or do I (can I?) manually
 delete these files and will c* just ignore the overlapping data and treat
 as tombstoned?



 What else should/could be done?



 Thank you in advance for your advice,



 *__*

 *Brian Spindler *





>>> --
>> -
>> Alexander Dejanovski
>> France
>> @alexanderdeja
>>
>> Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>

Re: need to reclaim space with TWCS

[Brian Spindler]

Hi Alexander,  Thanks for your response!  I'll give it a shot.

On Sat, Jan 20, 2018 at 10:22 AM Alexander Dejanovski <
a...@thelastpickle.com> wrote:

> Hi Brian,
>
> You should definitely set unchecked_tombstone_compaction to true and set
> the interval to the default of 1 day. Use a tombstone_threshold of 0.6 for
> example and see how that works.
> Tombstones will get purged depending on your partitioning as their
> partition needs to be fully contained within a single sstable.
>
> Deleting the sstables by hand is theoretically possible but should be kept
> as a last resort option if you're running out of space.
>
> Cheers,
>
> Le sam. 20 janv. 2018 à 15:41, Brian Spindler 
> a écrit :
>
>> I probably should have mentioned our setup: we’re on Cassandra version
>> 2.1.15.
>>
>>
>> On Sat, Jan 20, 2018 at 9:33 AM Brian Spindler 
>> wrote:
>>
>>> Hi, I have several column families using TWCS and it’s great.
>>> Unfortunately we seem to have missed the great advice in Alex’s article
>>> here: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html about
>>> setting the appropriate aggressive tombstone settings and now we have lots
>>> of timestamp overlaps and disk space to reclaim.
>>>
>>>
>>>
>>> I am trying to figure the best way out of this. Lots of the SSTables
>>> with overlapping timestamps in newer SSTables have droppable tombstones at
>>> like 0.895143957 or something similar, very close to 0.90 where the full
>>> sstable will drop afaik.
>>>
>>>
>>>
>>> I’m thinking to do the following immediately:
>>>
>>>
>>>
>>> Set *unchecked_tombstone_compaction = true*
>>>
>>> Set* tombstone_compaction_interval == TTL + gc_grace_seconds*
>>>
>>> Set* dclocal_read_repair_chance = 0.0 (currently 0.1)*
>>>
>>>
>>>
>>> If I do this, can I expect TWCS/C* to reclaim the space from those
>>> SSTables with 0.89* droppable tombstones?   Or do I (can I?) manually
>>> delete these files and will c* just ignore the overlapping data and treat
>>> as tombstoned?
>>>
>>>
>>>
>>> What else should/could be done?
>>>
>>>
>>>
>>> Thank you in advance for your advice,
>>>
>>>
>>>
>>> *__*
>>>
>>> *Brian Spindler *
>>>
>>>
>>>
>>>
>>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: need to reclaim space with TWCS

[Alexander Dejanovski]

Hi Brian,

You should definitely set unchecked_tombstone_compaction to true and set
the interval to the default of 1 day. Use a tombstone_threshold of 0.6 for
example and see how that works.
Tombstones will get purged depending on your partitioning as their
partition needs to be fully contained within a single sstable.

Deleting the sstables by hand is theoretically possible but should be kept
as a last resort option if you're running out of space.

Cheers,

Le sam. 20 janv. 2018 à 15:41, Brian Spindler  a
écrit :

> I probably should have mentioned our setup: we’re on Cassandra version
> 2.1.15.
>
>
> On Sat, Jan 20, 2018 at 9:33 AM Brian Spindler 
> wrote:
>
>> Hi, I have several column families using TWCS and it’s great.
>> Unfortunately we seem to have missed the great advice in Alex’s article
>> here: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html about
>> setting the appropriate aggressive tombstone settings and now we have lots
>> of timestamp overlaps and disk space to reclaim.
>>
>>
>>
>> I am trying to figure the best way out of this. Lots of the SSTables with
>> overlapping timestamps in newer SSTables have droppable tombstones at like
>> 0.895143957 or something similar, very close to 0.90 where the full sstable
>> will drop afaik.
>>
>>
>>
>> I’m thinking to do the following immediately:
>>
>>
>>
>> Set *unchecked_tombstone_compaction = true*
>>
>> Set* tombstone_compaction_interval == TTL + gc_grace_seconds*
>>
>> Set* dclocal_read_repair_chance = 0.0 (currently 0.1)*
>>
>>
>>
>> If I do this, can I expect TWCS/C* to reclaim the space from those
>> SSTables with 0.89* droppable tombstones?   Or do I (can I?) manually
>> delete these files and will c* just ignore the overlapping data and treat
>> as tombstoned?
>>
>>
>>
>> What else should/could be done?
>>
>>
>>
>> Thank you in advance for your advice,
>>
>>
>>
>> *__*
>>
>> *Brian Spindler *
>>
>>
>>
>>
>>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: need to reclaim space with TWCS

[Brian Spindler]

I probably should have mentioned our setup: we’re on Cassandra version
2.1.15.


On Sat, Jan 20, 2018 at 9:33 AM Brian Spindler 
wrote:

> Hi, I have several column families using TWCS and it’s great.
> Unfortunately we seem to have missed the great advice in Alex’s article
> here: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html about
> setting the appropriate aggressive tombstone settings and now we have lots
> of timestamp overlaps and disk space to reclaim.
>
>
>
> I am trying to figure the best way out of this. Lots of the SSTables with
> overlapping timestamps in newer SSTables have droppable tombstones at like
> 0.895143957 or something similar, very close to 0.90 where the full sstable
> will drop afaik.
>
>
>
> I’m thinking to do the following immediately:
>
>
>
> Set *unchecked_tombstone_compaction = true*
>
> Set* tombstone_compaction_interval == TTL + gc_grace_seconds*
>
> Set* dclocal_read_repair_chance = 0.0 (currently 0.1)*
>
>
>
> If I do this, can I expect TWCS/C* to reclaim the space from those
> SSTables with 0.89* droppable tombstones?   Or do I (can I?) manually
> delete these files and will c* just ignore the overlapping data and treat
> as tombstoned?
>
>
>
> What else should/could be done?
>
>
>
> Thank you in advance for your advice,
>
>
>
> *__*
>
> *Brian Spindler *
>
>
>
>
>

need to reclaim space with TWCS

[Brian Spindler]

Hi, I have several column families using TWCS and it’s great.
Unfortunately we seem to have missed the great advice in Alex’s article
here: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html about
setting the appropriate aggressive tombstone settings and now we have lots
of timestamp overlaps and disk space to reclaim.



I am trying to figure the best way out of this. Lots of the SSTables with
overlapping timestamps in newer SSTables have droppable tombstones at like
0.895143957 or something similar, very close to 0.90 where the full sstable
will drop afaik.



I’m thinking to do the following immediately:



Set *unchecked_tombstone_compaction = true*

Set* tombstone_compaction_interval == TTL + gc_grace_seconds*

Set* dclocal_read_repair_chance = 0.0 (currently 0.1)*



If I do this, can I expect TWCS/C* to reclaim the space from those SSTables
with 0.89* droppable tombstones?   Or do I (can I?) manually delete these
files and will c* just ignore the overlapping data and treat as tombstoned?




What else should/could be done?



Thank you in advance for your advice,



*__*

*Brian Spindler *

Re: Unable to change IP address from private to public

[Kyrylo Lebediev]

Haven't tried this by myself, but I suspect that in order to add public 
addresses for existing nodes for inter-dc communication, following method might 
work (just my guess):

 - take a node, shutdown C* there, change it's private IP, add  public IP, 
update cassandra.yaml accordingly (the same way as you described +set new 
private IP) and replace 'old node' as described here: 
http://docs.datastax.com/en/archived/cassandra/2.2/cassandra/operations/opsReplaceNode.html
 - do the same for the rest of nodes, one by one.


But it's better to test this approach somewhere in sandbox.

Also, AFAIK, only *listen* parameters have influence on how C* builds its ring. 
*rpc* settings are for communications b/w clients and Cassandra.


Regards,

Kyrill

Replacing a dead node or dead seed 
node
docs.datastax.com
Steps to replace a node that has died for some reason, such as hardware failure.




From: Thomas Goossens 
Sent: Friday, January 19, 2018 2:18:14 PM
To: user@cassandra.apache.org
Subject: Unable to change IP address from private to public

Hello,

For an 8-node Cassandra 2.2.11 cluster which is spread over two datacenters, I 
am looking to change from using private IP addresses to the combination of 
private and public IP addresses and interfaces. The cluster uses the 
GossipingPropertyFileSnitch.

The aim is to end up with each peer being known by their public IP address, but 
communicating over the private network where possible. We are looking to make 
this change in rolling fashion, so without taking down the entire cluster.

To this end, I have made the following configuration changes:

  1.  cassandra-rackdc.properties: add the "prefer_local=true" option
  2.  cassandra.yaml:
- change the broadcast_address from the private address to the public address
- add "listen_on_broadcast_address: true"
- leave the "listen_address" to be the private IP address
- change the broadcast_rpc_address from the private address to the public 
address
  3.  infrastructure:
- amend firewall settings to allow TCP traffic between peers on all interfaces 
for the relevant ports

Upon trying to make the described changes on a non-seed node, the following 
happens:
- The node appears to start up normally
- Upon running nodetool status on the changed node, all peers appear to be 
down, except the local node:

Datacenter: DC-A
=
DN  10.x.x.x
DN  10.x.x.x
DN  10.x.x.x
DN  10.x.x.x
DN  10.x.x.x

Datacenter: DC-B
=
DN  10.x.x.x
DN  10.x.x.x
UN  123.x.x.x

- Upon running nodetool status on any other node, all peers appear to be up, 
except the changed node which is still known under its old IP address:

Datacenter: DC-A
=
UN  10.x.x.x
UN  10.x.x.x
UN  10.x.x.x
UN  10.x.x.x
UN  10.x.x.x

Datacenter: DC-B
=
UN  10.x.x.x
UN  10.x.x.x
DN  10.x.x.x

Reverting the changes in cassandra.yaml and restarting that node causes the 
cluster to go back to normal. I have tried various combinations of 
private/public IP address settings, all to no avail.

I have successfully set up a similar configuration in a test cluster, but I 
have had to bring down the entire cluster in order to get it to work.

My question is: is it possible to make such a change in a phased way without 
bringing down the entire cluster? If yes, what is the best approach?

Many thanks in advance.

Thomas



--



Thomas Goossens
CTO


[Drillster]

Rijnzathe 16
3454 PV De Meern
Netherlands
+31 88 375 0500
www.drillster.com