Re: Nodes show different number of tokens than initially

[Oleksandr Shulgin]

On Fri, Feb 2, 2018 at 2:37 AM, kurt greaves  wrote:

> So one time I tried to understand why only a single node could have a
> token, and it appeared that it came over the fence from facebook and has
> been kept ever since. Personally I don't think it's necessary, and agree
> that it is kind of problematic (but there's probably lot's of stuff that
> relies on this now). Multiple DC's is one example but the same could apply
> to racks. There's no real reason (with NTS) that two nodes in separate
> racks can't have the same token. In fact being able to do this would make
> token allocation much simpler, and smart allocation algorithms could work
> much better with vnodes.
>

I understand that it might be way too late to change this.

My biggest gripe though is that all these subtle (but essential for real
understanding) details are ever so poorly documented.  I hope with the move
away from DataStax to Community website this might gradually improve.

Regards,
--
Alex

RE: Old tombstones not being cleaned up

[Steinmaurer, Thomas]

Right. In this case, cleanup should have done the necessary work here.

Thomas

From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
Sent: Freitag, 02. Februar 2018 06:59
To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

We did start with a 3 node cluster and a RF of 3, then added another 3 nodes 
and again another 3 nodes. So it is a good guess :)
But I have run both repair and cleanup against the table on all nodes, would 
that not have removed any stray partitions?
tor. 1. feb. 2018 kl. 22.31 skrev Steinmaurer, Thomas 
>:
Did you started with a 9 node cluster from the beginning or did you extend / 
scale out your cluster (with vnodes) beyond the replication factor?

If later applies and if you are deleting by explicit deletes and not via TTL, 
then nodes might not see the deletes anymore, as a node might not own the 
partition anymore after a topology change (e.g. scale out beyond the keyspace 
RF).

Just a very wild guess.

Thomas

From: Bo Finnerup Madsen 
[mailto:bo.gunder...@gmail.com]
Sent: Donnerstag, 01. Februar 2018 22:14

To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

We do not use TTL anywhere...records are inserted and deleted "manually" by our 
software.
tor. 1. feb. 2018 kl. 18.29 skrev Jonathan Haddad 
>:
Changing the defaul TTL doesn’t change the TTL on the existing data, only new 
data. It’s only set if you don’t supply one yourself.

On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen 
> wrote:
Hi,

We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally 
runs fine, but we have one table that are causing OOMs because an enormous 
amount of tombstones.
Looking at the data in the table (sstable2json), the first of the tombstones 
are almost a year old. The table was initially created with a gc_grace_period 
of 10 days, but I have now lowered it to 1 hour.
I have run a full repair of the table across all nodes. I have forced several 
major compactions of the table by using "nodetool compact", and also tried to 
switch from LeveledCompaction to SizeTierCompaction and back.

What could cause cassandra to keep these tombstones?

sstable2json:
{"key": "foo",
 "cells": 
[["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
   
["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
   
["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
   
["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
   
["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
   
["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
   
["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
   
["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
   
["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
...
{"key": "bar",
 "metadata": {"deletionInfo": 
{"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
 "cells": 
[["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
   
["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
   
["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],


sstablemetadata:
stablemetadata 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
SSTable: 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.10
Minimum timestamp: 1488976211688000
Maximum timestamp: 1517468644066000
SSTable max local deletion time: 2147483647
Compression ratio: 0.5121956624389545
Estimated droppable tombstones: 18.00161766553587
SSTable Level: 0
Repaired at: 0
ReplayPosition(segmentId=1517168739626, 
position=22690189)
Estimated tombstone drop times:%n
1488976211: 1
1489906506:  4706
1490174752:  6111
1490449759:  6554
1490735410:  6559
1491016789:  6369
1491347982: 10216
1491680214: 13502
...

desc:
CREATE TABLE xxx.yyy (
ti text,
uuid 

Re: Old tombstones not being cleaned up

[Bo Finnerup Madsen]

We did start with a 3 node cluster and a RF of 3, then added another 3
nodes and again another 3 nodes. So it is a good guess :)
But I have run both repair and cleanup against the table on all nodes,
would that not have removed any stray partitions?

tor. 1. feb. 2018 kl. 22.31 skrev Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com>:

> Did you started with a 9 node cluster from the beginning or did you extend
> / scale out your cluster (with vnodes) beyond the replication factor?
>
>
>
> If later applies and if you are deleting by explicit deletes and not via
> TTL, then nodes might not see the deletes anymore, as a node might not own
> the partition anymore after a topology change (e.g. scale out beyond the
> keyspace RF).
>
>
>
> Just a very wild guess.
>
>
>
> Thomas
>
>
>
> *From:* Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
> *Sent:* Donnerstag, 01. Februar 2018 22:14
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Old tombstones not being cleaned up
>
>
>
> We do not use TTL anywhere...records are inserted and deleted "manually"
> by our software.
>
> tor. 1. feb. 2018 kl. 18.29 skrev Jonathan Haddad :
>
> Changing the defaul TTL doesn’t change the TTL on the existing data, only
> new data. It’s only set if you don’t supply one yourself.
>
>
>
> On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen <
> bo.gunder...@gmail.com> wrote:
>
> Hi,
>
>
>
> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
> generally runs fine, but we have one table that are causing OOMs because an
> enormous amount of tombstones.
>
> Looking at the data in the table (sstable2json), the first of the
> tombstones are almost a year old. The table was initially created with a
> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
>
> I have run a full repair of the table across all nodes. I have forced
> several major compactions of the table by using "nodetool compact", and
> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>
>
>
> What could cause cassandra to keep these tombstones?
>
>
>
> sstable2json:
>
> {"key": "foo",
>
>  "cells":
> [["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
>
>
>  
> ["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
>
>
>  
> ["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
>
>
>  
> ["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
>
>
>  
> ["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
>
>
>  
> ["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
>
>
>  
> ["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
>
>
>  
> ["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
>
>
>  
> ["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
>
> ...
>
> {"key": "bar",
>
>  "metadata": {"deletionInfo":
> {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
>
>  "cells":
> [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
>
>
>  
> ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
>
>
>  
> ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],
>
> 
>
>
>
> sstablemetadata:
>
> stablemetadata
> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
>
> SSTable:
> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
>
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>
> Bloom Filter FP chance: 0.10
>
> Minimum timestamp: 1488976211688000
>
> Maximum timestamp: 1517468644066000
>
> SSTable max local deletion time: 2147483647
>
> Compression ratio: 0.5121956624389545
>
> Estimated droppable tombstones: 18.00161766553587
>
> SSTable Level: 0
>
> Repaired at: 0
>
> ReplayPosition(segmentId=1517168739626, position=22690189
> <22%2069%2001%2089>)
>
> Estimated tombstone drop times:%n
>
> 1488976211: 1
>
> 1489906506:  4706
>
> 1490174752:  6111
>
> 1490449759:  6554
>
> 1490735410:  6559
>
> 1491016789:  6369
>
> 1491347982: 10216
>
> 1491680214: 13502
>
> ...
>
>
>
> desc:
>
> CREATE TABLE xxx.yyy (
>
> ti text,
>
> uuid text,
>
> json_data text,
>
> PRIMARY KEY (ti, uuid)
>
> ) WITH CLUSTERING ORDER BY (uuid ASC)
>
> AND bloom_filter_fp_chance = 0.1
>
> AND caching = '{"keys":"ALL", 

Re: Nodes show different number of tokens than initially

[kurt greaves]

So one time I tried to understand why only a single node could have a
token, and it appeared that it came over the fence from facebook and has
been kept ever since. Personally I don't think it's necessary, and agree
that it is kind of problematic (but there's probably lot's of stuff that
relies on this now). Multiple DC's is one example but the same could apply
to racks. There's no real reason (with NTS) that two nodes in separate
racks can't have the same token. In fact being able to do this would make
token allocation much simpler, and smart allocation algorithms could work
much better with vnodes.

On 1 February 2018 at 17:35, Oleksandr Shulgin  wrote:

> On Thu, Feb 1, 2018 at 5:19 AM, Jeff Jirsa  wrote:
>
>>
>>> The reason I find it surprising, is that it makes very little *sense* to
>>> put a token belonging to a mode from one DC between tokens of nodes from
>>> another one.
>>>
>>
>> I don't want to really turn this into an argument over what should and
>> shouldn't make sense, but I do agree, it doesn't make sense to put a token
>> on one node in one DC onto another node in another DC.
>>
>
> This is not what I was trying to say.  I should have used an example to
> express myself clearer.  Here goes (disclaimer: it might sound like a rant,
> take it with a grain of salt):
>
> $ ccm create -v 3.0.15 -n 3:3 -s 2dcs
>
> For a more meaningful multi-DC setup than the default SimpleStrategy, use
> NTS:
>
> $ ccm node1 cqlsh -e "ALTER KEYSPACE system_auth WITH replication =
> {'class': 'NetworkTopologyStrategy', 'dc1': 2, 'dc2': 2};"
>
> $ ccm node1 nodetool ring
>
> Datacenter: dc1
> ==
> AddressRackStatus State   LoadOwns
> Token
>
> 3074457345618258602
> 127.0.0.1  r1  Up Normal  117.9 KB66.67%
> -9223372036854775808
> 127.0.0.2  r1  Up Normal  131.56 KB   66.67%
> -3074457345618258603
> 127.0.0.3  r1  Up Normal  117.88 KB   66.67%
> 3074457345618258602
>
> Datacenter: dc2
> ==
> AddressRackStatus State   LoadOwns
> Token
>
> 3074457345618258702
> 127.0.0.4  r1  Up Normal  121.54 KB   66.67%
> -9223372036854775708
> 127.0.0.5  r1  Up Normal  118.59 KB   66.67%
> -3074457345618258503
> 127.0.0.6  r1  Up Normal  114.12 KB   66.67%
> 3074457345618258702
>
> Note that CCM is aware of the cross-DC clashes and selects the tokens for
> the dc2 shifted by a 100.
>
> Then look at the token ring (output abbreviated and aligned by me):
>
> $ ccm node1 nodetool describering system_auth
>
> Schema Version:4f7d0ad0-350d-3ea0-ae8b-53d5bc34fc7e
> TokenRange:
> TokenRange(start_token:-9223372036854775808,
> end_token:-9223372036854775708, endpoints:[127.0.0.4, 127.0.0.2,
> 127.0.0.5, 127.0.0.3], ... TokenRange(start_token:-9223372036854775708,
> end_token:-3074457345618258603, endpoints:[127.0.0.2, 127.0.0.5,
> 127.0.0.3, 127.0.0.6], ... TokenRange(start_token:-3074457345618258603,
> end_token:-3074457345618258503, endpoints:[127.0.0.5, 127.0.0.3,
> 127.0.0.6, 127.0.0.1], ...
> TokenRange(start_token:-3074457345618258503, end_token:
> 3074457345618258602, endpoints:[127.0.0.3, 127.0.0.6, 127.0.0.1,
> 127.0.0.4], ... TokenRange(start_token: 3074457345618258602, end_token:
> 3074457345618258702, endpoints:[127.0.0.6, 127.0.0.1, 127.0.0.4,
> 127.0.0.2], ...
> TokenRange(start_token: 3074457345618258702, end_token:-9223372036854775808,
> endpoints:[127.0.0.1, 127.0.0.4, 127.0.0.2, 127.0.0.5], ...
>
> So in this setup, every token range has one end contributed by a node from
> dc1 and the other end -- from dc2.  That doesn't model anything in the real
> topology of the cluster.
>
> I see that it's easy to lump together tokens from all nodes and sort them,
> to produce a single token ring (and this is obviously the reason why tokens
> have to be unique throughout the cluster as a whole).  That doesn't mean
> it's a meaningful thing to do.
>
> This introduces complexity which not present in the problem domain
> initially.  This was a deliberate choice of developers, dare I say, to
> complect the separate DCs together in a single token ring.  This has
> profound consequences from the operations side.  If anything, it prevents
> bootstrapping multiple nodes at the same time even if they are in different
> DCs.  Or would you suggest to set consistent_range_movement=false and
> hope it will work out?
>
> If the whole reason for having separate DCs is to provide isolation, I
> fail to see how the single token ring design does anything towards
> achieving that.
>
> But also being very clear (I want to make sure I understand what you're
>> saying): that's a manual thing you did, Cassandra didn't do it for you,
>> right? The fact that Cassandra didn't STOP you from doing it could be
>> considered a bug, but YOU made that config choice?
>>
>
> Yes, we have chosen exactly the same token for two nodes in different DCs
> 

Re: Setting min_index_interval to 1?

[Nate McCall]

>
>
> Another was the crazy idea I started with of setting min_index_interval to
> 1. My guess was that this would cause it to read all index entries, and
> effectively have them all cached permanently. And it would read them
> straight out of the SSTables on every restart. Would this work? Other than
> probably causing a really long startup time, are there issues with this?
>
>
I've never tried that. It sounds like you understand the potential impact
on memory and startup time. If you have the data in such a way that you can
easily experiment, I would like to see a breakdown of the impact on
response time vs. memory usage as well as where the point of diminishing
returns is on turning this down towards 1 (I think there will be a sweet
spot somewhere).

Re: CDC usability and future development

[Jay Zhuang]

We did a POC to improve CDC feature as an interface (
https://github.com/ngcc/ngcc2017/blob/master/CassandraDataIngestion.pdf),
so the user doesn't have to read the commit log directly. We deployed the
change to a test cluster and doing more tests for production traffics, will
send out the design proposal, POC and the test result soon.

We have the same problem to get the full row value for CDC downstream
pipeline. We used to do a readback, right now our CDC downstream stores all
the data (in Hive), so no need to read back. For Cassandra CDC feature, I
don't think it should provide the full row value, as it supposed to be
Change Data Capture. But it's still a problem for the range delete, as it
cannot read back deleted data. So we're purposing an option to expand the
range delete in CDC if the user really wants it.


On Wed, Jan 31, 2018 at 7:32 AM Josh McKenzie  wrote:

> CDC provides only the mutation as opposed to the full column value, which
>> tends to be of limited use for us. Applications might want to know the full
>> column value, without having to issue a read back. We also see value in
>> being able to publish the full column value both before and after the
>> update. This is especially true when deleting a column since this stream
>> may be joined with others, or consumers may require other fields to
>> properly process the delete.
>
>
> Philosophically, my first pass at the feature prioritized minimizing
> impact to node performance first and usability second, punting a lot of the
> de-duplication and RbW implications of having full column values, or
> materializing stuff off-heap for consumption from a user and flagging as
> persisted to disk etc, for future work on the feature. I don't personally
> have any time to devote to moving the feature forward now but as Jeff
> indicates, Jay and Simon are both active in the space and taking up the
> torch.
>
>
> On Tue, Jan 30, 2018 at 8:35 PM, Jeff Jirsa  wrote:
>
>> Here's a deck of some proposed additions, discussed at one of the NGCC
>> sessions last fall:
>>
>> https://github.com/ngcc/ngcc2017/blob/master/CassandraDataIngestion.pdf
>>
>>
>>
>> On Tue, Jan 30, 2018 at 5:10 PM, Andrew Prudhomme  wrote:
>>
>> > Hi all,
>> >
>> > We are currently designing a system that allows our Cassandra clusters
>> to
>> > produce a stream of data updates. Naturally, we have been evaluating if
>> CDC
>> > can aid in this endeavor. We have found several challenges in using CDC
>> for
>> > this purpose.
>> >
>> > CDC provides only the mutation as opposed to the full column value,
>> which
>> > tends to be of limited use for us. Applications might want to know the
>> full
>> > column value, without having to issue a read back. We also see value in
>> > being able to publish the full column value both before and after the
>> > update. This is especially true when deleting a column since this stream
>> > may be joined with others, or consumers may require other fields to
>> > properly process the delete.
>> >
>> > Additionally, there is some difficulty with processing CDC itself such
>> as:
>> > - Updates not being immediately available (addressed by CASSANDRA-12148)
>> > - Each node providing an independent streams of updates that must be
>> > unified and deduplicated
>> >
>> > Our question is, what is the vision for CDC development? The current
>> > implementation could work for some use cases, but is a ways from a
>> general
>> > streaming solution. I understand that the nature of Cassandra makes this
>> > quite complicated, but are there any thoughts or desires on the future
>> > direction of CDC?
>> >
>> > Thanks
>> >
>> >
>>
>
>

Setting min_index_interval to 1?

[Dan Kinder]

Hi, I have an unusual case here: I'm wondering what will happen if I
set min_index_interval to 1.

Here's the logic. Suppose I have a table where I really want to squeeze as
many reads/sec out of it as possible, and where the row data size is much
larger than the keys. E.g. the keys are a few bytes, the row data is ~500KB.

This table would be a great candidate for key caching. Let's suppose I have
enough memory to have every key cached. However, it's a lot of data, and
the reads are very random. So it would take a very long time for that cache
to warm up.

One solution is that I write a little app to go through every key to warm
it up manually, and ensure that Cassandra has key_cache_keys_to_save set to
save the whole thing on restart. (Anyone know of a better way of doing
this?)

Another was the crazy idea I started with of setting min_index_interval to
1. My guess was that this would cause it to read all index entries, and
effectively have them all cached permanently. And it would read them
straight out of the SSTables on every restart. Would this work? Other than
probably causing a really long startup time, are there issues with this?

Thanks,
-dan

Re: Converting from Apache Cassandra 2.2.6 to 3.x

[Christopher Lord]

If upgrading to 3.11.x be aware of the dropped support for the legacy auth
tables. The relevant notes from NEWS.txt:

The authentication & authorization subsystems have been redesigned to
 support role based access control (RBAC), resulting in a change to the
 schema of the system_auth keyspace. See below for more detail.
 For systems already using the internal auth implementations, the
process
 for converting existing data during a rolling upgrade is
straightforward.
 As each node is restarted, it will attempt to convert any data in the
 legacy tables into the new schema. Until enough nodes to satisfy the
 replication strategy for the system_auth keyspace are upgraded and so
have
 the new schema, this conversion will fail with the failure being
reported
 in the system log.
 During the upgrade, Cassandra's internal auth classes will continue to
use
 the legacy tables, so clients experience no disruption. Issuing DCL
 statements during an upgrade is not supported.
 Once all nodes are upgraded, an operator with superuser privileges
should
 drop the legacy tables, system_auth.users, system_auth.credentials and
 system_auth.permissions. Doing so will prompt Cassandra to switch over
to
 the new tables without requiring any further intervention.
 While the legacy tables are present a restarted node will re-run the
data
 conversion and report the outcome so that operators can verify that it
is
safe to drop them.


If you don't drop these legacy tables before the upgrade you may find
yourself unable to access the cluster.

On 2 February 2018 at 09:49, William Boutin 
wrote:

> We are converting our product from Apache Cassandra 2.2.6 to 3.x. What
> issues may we run into when we convert?
>
>
>
>
>
> [image: Ericsson] 
>
> *WILLIAM L. BOUTIN *
> Engineer IV - Sftwr
> BMDA PADB DSE DU CC NGEE
>
>
> *Ericsson
> *
> 1 Ericsson Drive, US PI06 1.S747
> Piscataway, NJ, 08854, USA
> Phone (913) 241-5574
> Mobile (732) 213-1368
> Emergency (732) 354-1263
> william.bou...@ericsson.com
> www.ericsson.com
>
> [image: http://www.ericsson.com/current_campaign]
> 
>
> Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242.
> This Communication is Confidential. We only send and receive email on the
> basis of the terms set out at www.ericsson.com/email_disclaimer
>
>
>



-- 


*Chris Lord*
*Devops Engineer*+61458181093

[image: https://www.instaclustr.com] 

   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

RE: Old tombstones not being cleaned up

[ZAIDI, ASAD A]

No it doesn’t. unchecked_tombstone_compaction sub property is common in all 
STCS, DTCS & LCS. Though you can also use jmxterm tool and invoke different 
compaction on single node if you desire.


From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
Sent: Thursday, February 01, 2018 3:17 PM
To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

I, almost, tried that today :) I ran a repair, changed the compaction algorithm 
from leveled to sizetierd and back. This definitely forced a compaction, but 
the tombstones are still there.
Will setting the unchecked_tombstone_compaction force another type of 
compaction?
tor. 1. feb. 2018 kl. 19.37 skrev ZAIDI, ASAD A 
>:
Make data consistent (run repair), reduce gc_grace_seconds (try set it to 0 
temporarily though careful as this can affect hinted handoff!)  and set table’s 
compaction sub-property i.e. unchecked_tombstone_compaction to true. Compaction 
 will  take care of tombstones!


From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Thursday, February 01, 2018 11:29 AM
To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

Changing the defaul TTL doesn’t change the TTL on the existing data, only new 
data. It’s only set if you don’t supply one yourself.

On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen 
> wrote:
Hi,

We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally 
runs fine, but we have one table that are causing OOMs because an enormous 
amount of tombstones.
Looking at the data in the table (sstable2json), the first of the tombstones 
are almost a year old. The table was initially created with a gc_grace_period 
of 10 days, but I have now lowered it to 1 hour.
I have run a full repair of the table across all nodes. I have forced several 
major compactions of the table by using "nodetool compact", and also tried to 
switch from LeveledCompaction to SizeTierCompaction and back.

What could cause cassandra to keep these tombstones?

sstable2json:
{"key": "foo",
 "cells": 
[["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
   
["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
   
["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
   
["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
   
["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
   
["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
   
["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
   
["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
   
["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
...
{"key": "bar",
 "metadata": {"deletionInfo": 
{"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
 "cells": 
[["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
   
["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
   
["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],


sstablemetadata:
stablemetadata 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
SSTable: 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.10
Minimum timestamp: 1488976211688000
Maximum timestamp: 1517468644066000
SSTable max local deletion time: 2147483647
Compression ratio: 0.5121956624389545
Estimated droppable tombstones: 18.00161766553587
SSTable Level: 0
Repaired at: 0
ReplayPosition(segmentId=1517168739626, 
position=22690189)
Estimated tombstone drop times:%n
1488976211: 1
1489906506:  4706
1490174752:  6111
1490449759:  6554
1490735410:  6559
1491016789:  6369
1491347982: 10216
1491680214: 13502
...

desc:
CREATE TABLE xxx.yyy (
ti text,
uuid text,
json_data text,
PRIMARY KEY (ti, uuid)
) WITH CLUSTERING ORDER BY (uuid ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 

RE: Converting from Apache Cassandra 2.2.6 to 3.x

[ZAIDI, ASAD A]

You may want to upgrade python and java/JDK version with Cassandra upgrade. 
please refer to CHANGES.txt for all updates & improvement made in your selected 
3.x version.



From: William Boutin [mailto:william.bou...@ericsson.com]
Sent: Thursday, February 01, 2018 4:49 PM
To: user@cassandra.apache.org
Cc: William Boutin 
Subject: Converting from Apache Cassandra 2.2.6 to 3.x

We are converting our product from Apache Cassandra 2.2.6 to 3.x. What issues 
may we run into when we convert?


[Image removed by sender. 
Ericsson]
WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[Image removed by sender. 
http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

Converting from Apache Cassandra 2.2.6 to 3.x

[William Boutin]

We are converting our product from Apache Cassandra 2.2.6 to 3.x. What issues 
may we run into when we convert?


[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

Re: unable to start cassandra 3.11.1

[Kant Kodali]

Ok I saw the ticket looks like this java version "1.8.0_162" wont work!

On Thu, Feb 1, 2018 at 2:43 PM, Kant Kodali  wrote:

> Hi Justin,
>
> I am using
>
> java version "1.8.0_162"
>
> Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
>
>
> Thanks!
>
> On Thu, Feb 1, 2018 at 2:40 PM, Justin Cameron 
> wrote:
>
>> Unfortunately C* 3.11.1 is incompatible with the latest version of Java.
>> You'll need to either downgrade to Java 1.8.0.151-5 or wait for C* 3.11.2
>> (see https://issues.apache.org/jira/browse/CASSANDRA-14173 for details)
>>
>> On Fri, 2 Feb 2018 at 09:35 Kant Kodali  wrote:
>>
>>> Hi All,
>>>
>>> I am unable to start cassandra 3.11.1. Below is the stack trace.
>>>
>>> Exception (java.lang.AbstractMethodError) encountered during startup: 
>>> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
>>> java.lang.AbstractMethodError: 
>>> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
>>> at 
>>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
>>> at 
>>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
>>> at 
>>> javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
>>> at 
>>> org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
>>> ERROR 22:33:49 Exception encountered during startup
>>> java.lang.AbstractMethodError: 
>>> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
>>> at 
>>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
>>>  ~[na:1.8.0_162]
>>> at 
>>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
>>>  ~[na:1.8.0_162]
>>> at 
>>> javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
>>>  ~[na:1.8.0_162]
>>> at 
>>> org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
>>>  ~[apache-cassandra-3.11.1.jar:3.11.1]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
>>>  [apache-cassandra-3.11.1.jar:3.11.1]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
>>>  [apache-cassandra-3.11.1.jar:3.11.1]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
>>>  [apache-cassandra-3.11.1.jar:3.11.1]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) 
>>> [apache-cassandra-3.11.1.jar:3.11.1]
>>>
>>> --
>>
>>
>> *Justin Cameron*Senior Software Engineer
>>
>>
>> 
>>
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information.  If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>
>

Re: unable to start cassandra 3.11.1

[Kant Kodali]

Hi Justin,

I am using

java version "1.8.0_162"

Java(TM) SE Runtime Environment (build 1.8.0_162-b12)


Thanks!

On Thu, Feb 1, 2018 at 2:40 PM, Justin Cameron 
wrote:

> Unfortunately C* 3.11.1 is incompatible with the latest version of Java.
> You'll need to either downgrade to Java 1.8.0.151-5 or wait for C* 3.11.2
> (see https://issues.apache.org/jira/browse/CASSANDRA-14173 for details)
>
> On Fri, 2 Feb 2018 at 09:35 Kant Kodali  wrote:
>
>> Hi All,
>>
>> I am unable to start cassandra 3.11.1. Below is the stack trace.
>>
>> Exception (java.lang.AbstractMethodError) encountered during startup: 
>> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
>> java.lang.AbstractMethodError: 
>> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
>> at 
>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
>> at 
>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
>> at 
>> javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
>> at 
>> org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
>> at 
>> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
>> at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
>> at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
>> at 
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
>> ERROR 22:33:49 Exception encountered during startup
>> java.lang.AbstractMethodError: 
>> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
>> at 
>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
>>  ~[na:1.8.0_162]
>> at 
>> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
>>  ~[na:1.8.0_162]
>> at 
>> javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
>>  ~[na:1.8.0_162]
>> at 
>> org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
>>  ~[apache-cassandra-3.11.1.jar:3.11.1]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
>>  [apache-cassandra-3.11.1.jar:3.11.1]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) 
>> [apache-cassandra-3.11.1.jar:3.11.1]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
>>  [apache-cassandra-3.11.1.jar:3.11.1]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) 
>> [apache-cassandra-3.11.1.jar:3.11.1]
>>
>> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> 
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>

Re: unable to start cassandra 3.11.1

[Justin Cameron]

Unfortunately C* 3.11.1 is incompatible with the latest version of Java.
You'll need to either downgrade to Java 1.8.0.151-5 or wait for C* 3.11.2
(see https://issues.apache.org/jira/browse/CASSANDRA-14173 for details)

On Fri, 2 Feb 2018 at 09:35 Kant Kodali  wrote:

> Hi All,
>
> I am unable to start cassandra 3.11.1. Below is the stack trace.
>
> Exception (java.lang.AbstractMethodError) encountered during startup: 
> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
> java.lang.AbstractMethodError: 
> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
> at 
> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
> at 
> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
> at 
> javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
> at 
> org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
> at 
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
> ERROR 22:33:49 Exception encountered during startup
> java.lang.AbstractMethodError: 
> org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
> at 
> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
>  ~[na:1.8.0_162]
> at 
> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
>  ~[na:1.8.0_162]
> at 
> javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
>  ~[na:1.8.0_162]
> at 
> org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
>  ~[apache-cassandra-3.11.1.jar:3.11.1]
> at 
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
>  [apache-cassandra-3.11.1.jar:3.11.1]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) 
> [apache-cassandra-3.11.1.jar:3.11.1]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
>  [apache-cassandra-3.11.1.jar:3.11.1]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) 
> [apache-cassandra-3.11.1.jar:3.11.1]
>
> --


*Justin Cameron*Senior Software Engineer





This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

unable to start cassandra 3.11.1

[Kant Kodali]

Hi All,

I am unable to start cassandra 3.11.1. Below is the stack trace.

Exception (java.lang.AbstractMethodError) encountered during startup:
org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
java.lang.AbstractMethodError:
org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
at 
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
at 
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
at 
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
at 
org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
at 
org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
ERROR 22:33:49 Exception encountered during startup
java.lang.AbstractMethodError:
org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
at 
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
~[na:1.8.0_162]
at 
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
~[na:1.8.0_162]
at 
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
~[na:1.8.0_162]
at 
org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
~[apache-cassandra-3.11.1.jar:3.11.1]
at 
org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
[apache-cassandra-3.11.1.jar:3.11.1]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
[apache-cassandra-3.11.1.jar:3.11.1]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
[apache-cassandra-3.11.1.jar:3.11.1]
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
[apache-cassandra-3.11.1.jar:3.11.1]

Re: Not what I‘ve expected Performance

[Jürgen Albersdorfer]

I changed it a little to spark.sql and extracted such a partitioning key table 
as You did with the userid and joined this to my table to copy and safe this to 
cassandra seemed in a First Test to utilize every given Bit of Performance the 
Cluster can provide. Dont yet know why the first code did not perform as 
expected.

Von meinem iPhone gesendet

> Am 01.02.2018 um 22:09 schrieb kurt greaves :
> 
> That extra code is not necessary, it's just to only retrieve a sampling of 
> let's. You don't want it if you're copying the whole table. It sounds like 
> you're taking the right approach, probably just need some more tuning. Might 
> be on the Cassandra side as well (concurrent_reads/writes).
> 
> 
> On 1 Feb. 2018 19:06, "Jürgen Albersdorfer"  wrote:
> Hi Kurt, thanks for your response.
> I indeed utilized Spark - what I've forgot to mention - and I did it nearly 
> the same as in the example you gave me.
> Just without that .select(PK).sample(false, 0.1) Instruction which I don't 
> actually get what it's useful for - and maybe that's the key to the castle.
> 
> I already found out that I require some more Spark Executors - really lots of 
> them.
> And it was a bad Idea in the first place to ./spark-submit without any 
> parameters about executor-memory, total-executor-cores and especially 
> executor-cores.
> I now submitted it with --executor-cores 1 --total-executor-cores 100 -- 
> executor-memory 8G to get more Executors out of my Cluster.
> Without that limits, a Spark Executor will utilize all of the available 
> cores. With the limitations, The Spark Worker will be able to start more 
> Workers in parallel which boosts in my example,
> but is still way to slow and far away from requiring to throttle it. And that 
> is what I actually expected when 100 Processes start beating with the 
> Database Cluster.
> 
> Definitelly I'll give your Code a try.
> 
> 2018-02-01 6:36 GMT+01:00 kurt greaves :
>> How are you copying? With CQLSH COPY or your own script? If you've got spark 
>> already it's quite simple to copy between tables and it should be pretty 
>> much as fast as you can get it. (you may even need to throttle). There's 
>> some sample code here (albeit it's copying between clusters but easily 
>> tailored to copy between tables). 
>> https://www.instaclustr.com/support/documentation/apache-spark/using-spark-to-sample-data-from-one-cassandra-cluster-and-write-to-another/
>> 
>>> On 30 January 2018 at 21:05, Jürgen Albersdorfer  
>>> wrote:
>>> Hi, We are using C* 3.11.1 with a 9 Node Cluster built on CentOS Servers 
>>> eac=
>>> h having 2x Quad Core Xeon, 128GB of RAM and two separate 2TB spinning 
>>> Disks=
>>> , one for Log one for Data with Spark on Top.
>>> 
>>> Due to bad Schema (Partitions of about 4 to 8 GB) I need to copy a whole 
>>> Tab=
>>> le into another having same fields but different partitioning.=20
>>> 
>>> I expected glowing Iron when I started the copy Job, but instead cannot 
>>> even=
>>> See some Impact on CPU, mem or disks. - but the Job does copy the Data over=
>>> veeerry slowly at about a MB or two per Minute.
>>> 
>>> Any suggestion where to start investigation?
>>> 
>>> Thanks already
>>> 
>>> Von meinem iPhone gesendet
>>> 
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
>> 
> 
> 

RE: Old tombstones not being cleaned up

[Steinmaurer, Thomas]

Did you started with a 9 node cluster from the beginning or did you extend / 
scale out your cluster (with vnodes) beyond the replication factor?

If later applies and if you are deleting by explicit deletes and not via TTL, 
then nodes might not see the deletes anymore, as a node might not own the 
partition anymore after a topology change (e.g. scale out beyond the keyspace 
RF).

Just a very wild guess.

Thomas

From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
Sent: Donnerstag, 01. Februar 2018 22:14
To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

We do not use TTL anywhere...records are inserted and deleted "manually" by our 
software.
tor. 1. feb. 2018 kl. 18.29 skrev Jonathan Haddad 
>:
Changing the defaul TTL doesn’t change the TTL on the existing data, only new 
data. It’s only set if you don’t supply one yourself.

On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen 
> wrote:
Hi,

We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally 
runs fine, but we have one table that are causing OOMs because an enormous 
amount of tombstones.
Looking at the data in the table (sstable2json), the first of the tombstones 
are almost a year old. The table was initially created with a gc_grace_period 
of 10 days, but I have now lowered it to 1 hour.
I have run a full repair of the table across all nodes. I have forced several 
major compactions of the table by using "nodetool compact", and also tried to 
switch from LeveledCompaction to SizeTierCompaction and back.

What could cause cassandra to keep these tombstones?

sstable2json:
{"key": "foo",
 "cells": 
[["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
   
["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
   
["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
   
["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
   
["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
   
["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
   
["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
   
["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
   
["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
...
{"key": "bar",
 "metadata": {"deletionInfo": 
{"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
 "cells": 
[["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
   
["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
   
["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],


sstablemetadata:
stablemetadata 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
SSTable: 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.10
Minimum timestamp: 1488976211688000
Maximum timestamp: 1517468644066000
SSTable max local deletion time: 2147483647
Compression ratio: 0.5121956624389545
Estimated droppable tombstones: 18.00161766553587
SSTable Level: 0
Repaired at: 0
ReplayPosition(segmentId=1517168739626, 
position=22690189)
Estimated tombstone drop times:%n
1488976211: 1
1489906506:  4706
1490174752:  6111
1490449759:  6554
1490735410:  6559
1491016789:  6369
1491347982: 10216
1491680214: 13502
...

desc:
CREATE TABLE xxx.yyy (
ti text,
uuid text,
json_data text,
PRIMARY KEY (ti, uuid)
) WITH CLUSTERING ORDER BY (uuid ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';

jmx 

Re: Old tombstones not being cleaned up

[Bo Finnerup Madsen]

I, almost, tried that today :) I ran a repair, changed the compaction
algorithm from leveled to sizetierd and back. This definitely forced a
compaction, but the tombstones are still there.
Will setting the unchecked_tombstone_compaction force another type of
compaction?

tor. 1. feb. 2018 kl. 19.37 skrev ZAIDI, ASAD A :

> Make data consistent (run repair), reduce gc_grace_seconds (try set it to
> 0 temporarily though careful as this can affect hinted handoff!)  and set
> table’s compaction sub-property i.e. unchecked_tombstone_compaction to
> true. Compaction  will  take care of tombstones!
>
>
>
>
>
> *From:* Jonathan Haddad [mailto:j...@jonhaddad.com]
> *Sent:* Thursday, February 01, 2018 11:29 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Old tombstones not being cleaned up
>
>
>
> Changing the defaul TTL doesn’t change the TTL on the existing data, only
> new data. It’s only set if you don’t supply one yourself.
>
>
>
> On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen <
> bo.gunder...@gmail.com> wrote:
>
> Hi,
>
>
>
> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
> generally runs fine, but we have one table that are causing OOMs because an
> enormous amount of tombstones.
>
> Looking at the data in the table (sstable2json), the first of the
> tombstones are almost a year old. The table was initially created with a
> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
>
> I have run a full repair of the table across all nodes. I have forced
> several major compactions of the table by using "nodetool compact", and
> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>
>
>
> What could cause cassandra to keep these tombstones?
>
>
>
> sstable2json:
>
> {"key": "foo",
>
>  "cells":
> [["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
>
>
>  
> ["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
>
>
>  
> ["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
>
>
>  
> ["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
>
>
>  
> ["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
>
>
>  
> ["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
>
>
>  
> ["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
>
>
>  
> ["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
>
>
>  
> ["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
>
> ...
>
> {"key": "bar",
>
>  "metadata": {"deletionInfo":
> {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
>
>  "cells":
> [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
>
>
>  
> ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
>
>
>  
> ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],
>
> 
>
>
>
> sstablemetadata:
>
> stablemetadata
> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
>
> SSTable:
> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
>
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>
> Bloom Filter FP chance: 0.10
>
> Minimum timestamp: 1488976211688000
>
> Maximum timestamp: 1517468644066000
>
> SSTable max local deletion time: 2147483647
>
> Compression ratio: 0.5121956624389545
>
> Estimated droppable tombstones: 18.00161766553587
>
> SSTable Level: 0
>
> Repaired at: 0
>
> ReplayPosition(segmentId=1517168739626, position=22690189
> <22%2069%2001%2089>)
>
> Estimated tombstone drop times:%n
>
> 1488976211: 1
>
> 1489906506:  4706
>
> 1490174752:  6111
>
> 1490449759:  6554
>
> 1490735410:  6559
>
> 1491016789:  6369
>
> 1491347982: 10216
>
> 1491680214: 13502
>
> ...
>
>
>
> desc:
>
> CREATE TABLE xxx.yyy (
>
> ti text,
>
> uuid text,
>
> json_data text,
>
> PRIMARY KEY (ti, uuid)
>
> ) WITH CLUSTERING ORDER BY (uuid ASC)
>
> AND bloom_filter_fp_chance = 0.1
>
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
>
> AND comment = ''
>
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>
> AND dclocal_read_repair_chance = 0.1
>
> AND default_time_to_live = 0
>
> AND 

Re: Old tombstones not being cleaned up

[Bo Finnerup Madsen]

We do not use TTL anywhere...records are inserted and deleted "manually" by
our software.

tor. 1. feb. 2018 kl. 18.29 skrev Jonathan Haddad :

> Changing the defaul TTL doesn’t change the TTL on the existing data, only
> new data. It’s only set if you don’t supply one yourself.
>
> On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen <
> bo.gunder...@gmail.com> wrote:
>
>> Hi,
>>
>> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
>> generally runs fine, but we have one table that are causing OOMs because an
>> enormous amount of tombstones.
>> Looking at the data in the table (sstable2json), the first of the
>> tombstones are almost a year old. The table was initially created with a
>> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
>> I have run a full repair of the table across all nodes. I have forced
>> several major compactions of the table by using "nodetool compact", and
>> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>>
>> What could cause cassandra to keep these tombstones?
>>
>> sstable2json:
>> {"key": "foo",
>>  "cells":
>> [["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
>>
>>  
>> ["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
>>
>>  
>> ["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
>>
>>  
>> ["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
>>
>>  
>> ["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
>>
>>  
>> ["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
>>
>>  
>> ["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
>>
>>  
>> ["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
>>
>>  
>> ["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
>> ...
>> {"key": "bar",
>>  "metadata": {"deletionInfo":
>> {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
>>  "cells":
>> [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
>>
>>  
>> ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
>>
>>  
>> ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],
>> 
>>
>> sstablemetadata:
>> stablemetadata
>> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
>> SSTable:
>> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.10
>> Minimum timestamp: 1488976211688000
>> Maximum timestamp: 1517468644066000
>> SSTable max local deletion time: 2147483647
>> Compression ratio: 0.5121956624389545
>> Estimated droppable tombstones: 18.00161766553587
>> SSTable Level: 0
>> Repaired at: 0
>> ReplayPosition(segmentId=1517168739626, position=22690189
>> <22%2069%2001%2089>)
>> Estimated tombstone drop times:%n
>> 1488976211: 1
>> 1489906506:  4706
>> 1490174752:  6111
>> 1490449759:  6554
>> 1490735410:  6559
>> 1491016789:  6369
>> 1491347982: 10216
>> 1491680214: 13502
>> ...
>>
>> desc:
>> CREATE TABLE xxx.yyy (
>> ti text,
>> uuid text,
>> json_data text,
>> PRIMARY KEY (ti, uuid)
>> ) WITH CLUSTERING ORDER BY (uuid ASC)
>> AND bloom_filter_fp_chance = 0.1
>> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
>> AND comment = ''
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 0
>> AND gc_grace_seconds = 3600
>> AND max_index_interval = 2048
>> AND memtable_flush_period_in_ms = 0
>> AND min_index_interval = 128
>> AND read_repair_chance = 0.0
>> AND speculative_retry = '99.0PERCENTILE';
>>
>> jmx props(picture):
>> [image: image.png]
>>
>

Re: Old tombstones not being cleaned up

[Bo Finnerup Madsen]

I have forced several compactions without the tombstones being cleaned.
Compactions was forced both using "nodetool compact" and by changeing
compaction algorithem from leved to sizedtiered and back...

tor. 1. feb. 2018 kl. 15.54 skrev James Shaw :

> i see leveled compaction used, if it's last, it will have to stay until
> next level compaction happens, then will be gone, right ?
>
> On Thu, Feb 1, 2018 at 2:33 AM, Bo Finnerup Madsen  > wrote:
>
>> Hi,
>>
>> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
>> generally runs fine, but we have one table that are causing OOMs because an
>> enormous amount of tombstones.
>> Looking at the data in the table (sstable2json), the first of the
>> tombstones are almost a year old. The table was initially created with a
>> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
>> I have run a full repair of the table across all nodes. I have forced
>> several major compactions of the table by using "nodetool compact", and
>> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>>
>> What could cause cassandra to keep these tombstones?
>>
>> sstable2json:
>> {"key": "foo",
>>  "cells":
>> [["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
>>
>>  
>> ["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
>>
>>  
>> ["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
>>
>>  
>> ["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
>>
>>  
>> ["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
>>
>>  
>> ["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
>>
>>  
>> ["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
>>
>>  
>> ["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
>>
>>  
>> ["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
>> ...
>> {"key": "bar",
>>  "metadata": {"deletionInfo":
>> {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
>>  "cells":
>> [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
>>
>>  
>> ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
>>
>>  
>> ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],
>> 
>>
>> sstablemetadata:
>> stablemetadata
>> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
>> SSTable:
>> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.10
>> Minimum timestamp: 1488976211688000
>> Maximum timestamp: 1517468644066000
>> SSTable max local deletion time: 2147483647 <(214)%20748-3647>
>> Compression ratio: 0.5121956624389545
>> Estimated droppable tombstones: 18.00161766553587
>> SSTable Level: 0
>> Repaired at: 0
>> ReplayPosition(segmentId=1517168739626, position=22690189
>> <22%2069%2001%2089>)
>> Estimated tombstone drop times:%n
>> 1488976211: 1
>> 1489906506:  4706
>> 1490174752:  6111
>> 1490449759:  6554
>> 1490735410:  6559
>> 1491016789:  6369
>> 1491347982: 10216
>> 1491680214: 13502
>> ...
>>
>> desc:
>> CREATE TABLE xxx.yyy (
>> ti text,
>> uuid text,
>> json_data text,
>> PRIMARY KEY (ti, uuid)
>> ) WITH CLUSTERING ORDER BY (uuid ASC)
>> AND bloom_filter_fp_chance = 0.1
>> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
>> AND comment = ''
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> AND compression = {'sstable_compression': 'org.apache.cassandra.io
>> .compress.LZ4Compressor'}
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 0
>> AND gc_grace_seconds = 3600
>> AND max_index_interval = 2048
>> AND memtable_flush_period_in_ms = 0
>> AND min_index_interval = 128
>> AND read_repair_chance = 0.0
>> AND speculative_retry = '99.0PERCENTILE';
>>
>> jmx props(picture):
>> [image: image.png]
>>
>
>

Re: Not what I‘ve expected Performance

[kurt greaves]

That extra code is not necessary, it's just to only retrieve a sampling of
let's. You don't want it if you're copying the whole table. It sounds like
you're taking the right approach, probably just need some more tuning.
Might be on the Cassandra side as well (concurrent_reads/writes).


On 1 Feb. 2018 19:06, "Jürgen Albersdorfer"  wrote:

Hi Kurt, thanks for your response.
I indeed utilized Spark - what I've forgot to mention - and I did it nearly
the same as in the example you gave me.
Just without that .select(PK).sample(false, 0.1) Instruction which I don't
actually get what it's useful for - and maybe that's the key to the castle.

I already found out that I require some more Spark Executors - really lots
of them.
And it was a bad Idea in the first place to ./spark-submit without any
parameters about executor-memory, total-executor-cores and especially
executor-cores.
I now submitted it with --executor-cores 1 --total-executor-cores 100 --
executor-memory 8G to get more Executors out of my Cluster.
Without that limits, a Spark Executor will utilize all of the available
cores. With the limitations, The Spark Worker will be able to start more
Workers in parallel which boosts in my example,
but is still way to slow and far away from requiring to throttle it. And
that is what I actually expected when 100 Processes start beating with the
Database Cluster.

Definitelly I'll give your Code a try.

2018-02-01 6:36 GMT+01:00 kurt greaves :

> How are you copying? With CQLSH COPY or your own script? If you've got
> spark already it's quite simple to copy between tables and it should be
> pretty much as fast as you can get it. (you may even need to throttle).
> There's some sample code here (albeit it's copying between clusters but
> easily tailored to copy between tables). https://www.instaclus
> tr.com/support/documentation/apache-spark/using-spark-to-
> sample-data-from-one-cassandra-cluster-and-write-to-another/
>
> On 30 January 2018 at 21:05, Jürgen Albersdorfer 
> wrote:
>
>> Hi, We are using C* 3.11.1 with a 9 Node Cluster built on CentOS Servers
>> eac=
>> h having 2x Quad Core Xeon, 128GB of RAM and two separate 2TB spinning
>> Disks=
>> , one for Log one for Data with Spark on Top.
>>
>> Due to bad Schema (Partitions of about 4 to 8 GB) I need to copy a whole
>> Tab=
>> le into another having same fields but different partitioning.=20
>>
>> I expected glowing Iron when I started the copy Job, but instead cannot
>> even=
>> See some Impact on CPU, mem or disks. - but the Job does copy the Data
>> over=
>> veeerry slowly at about a MB or two per Minute.
>>
>> Any suggestion where to start investigation?
>>
>> Thanks already
>>
>> Von meinem iPhone gesendet
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: Documentation question

[Alain RODRIGUEZ]

Hello Mikael,

I believe that this licence apply to the documentation:
https://github.com/apache/cassandra/blob/trunk/LICENSE.txt#L90-L129. The
documentation is in the same repository than the code now.

I believe you can, but with some rules to respect.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-02-01 13:58 GMT+00:00 Mikael :

> Hi!
>
> I have a commercial application that use Cassandra and the manual for it
> includes a link to the Cassandra documentation, but I will include the most
> basic stuff in the manual like how to setup Cassandra, configuration and so
> on, so the question is if I need to write this myself from scratch or can I
> include parts of the Apache Cassandra documentation in my manual ?
>
> Mikael
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: multiple tables vs. partitions and TTL

[Alain RODRIGUEZ]

Hello Marcus,

We want to store bigger amounts of data (> 30mio rows containing blobs)
>

This should be perfectly fine for a partition. We use to recommend up to
100 MB per partition, which is a soft limit, as some use cases work very
well with bigger partition.

will be deleted depending on the type of data on a monthly base
> Some data would survive for two month only, other data for 3-5 years.
>

Do you need to store all these distinct type of data in the same table? If
not, you could could make a very nice usage of TWCS with fixed TTLs for
each table and expiring SSTable on old buckets for example. A colleague
wrote about this topic and I believe that it could be a good fit in this
case: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html.

which will allow a single delete command, followed by a compaction
>

Be careful with deletes and tombstones in Cassandra, it is sadly not that
straightforward.
- Tombstones should have been replicated to all replicas to prevent
resurrecting data (I heard people calling it 'Zombies' or 'ghosts').
- Cassandra waits for gc_grace_seconds after the data expiration before a
compaction can actually remove the data
- All the data shadowed by the tombstone and the tombstone itself should
all be part of the same compaction to actually be possibly evicted.
- ...

It's a tricky topic. I wrote about it last year as there were a lot of
questions about tombstones:
thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html. I hope
it will be of some use to you.

or alternatively to have multiple tables (one per month when the deletion
> process would just drop the table).
> The logic to retrieve that data is per record, so we know both the
> retention period and the id (uuid) of the addressed record,
> so multiple tables can be handled.
>

I would try to break things using the data workflow (or data type in your
case) rather than using tables for time buckets. TWCS might then be a very
good alternative, probably way easier to handle and to reason about.

Ideally, you want to make sure that no read request can ever touch a
tombstone now, in the design phase, as much as possible. You can for
exemple make the day part of the partition key (not only this, it should be
mixed with the natural identifier to form a composite key and to prevent
all the writes from one entire day to write only to RF nodes typically...).

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com




2018-02-01 8:16 GMT+00:00 Marcus Haarmann :

> Hi experts,
>
> I have a design issue here:
> We want to store bigger amounts of data (> 30mio rows containing blobs)
> which will be deleted depending on the type
> of data on a monthly base (not in the same order as the data entered the
> system).
> Some data would survive for two month only, other data for 3-5 years.
>
> The choice now is to have one table only with TTL per partition and
> partitions per deletion month (when the data should be deleted)
> which will allow a single delete command, followed by a compaction
> or alternatively to have multiple tables (one per month when the deletion
> process would just drop the table).
> The logic to retrieve that data is per record, so we know both the
> retention period and the id (uuid) of the addressed record,
> so multiple tables can be handled.
>
> Since it would be one table per deletion month, I do not expect more than
> 1000-2000 tables, depending on the
> retention period of the data.
>
> The benefit creating multiple tables would be that there are no tombstones
> while more tables take more memory in the nodes.
> The one table approach would make the compaction process take longer and
> produce more I/O activity because
> the compaction would regenerate multiple tables internally.
>
> Any thoughts on this ?
> We want to use 9 nodes, cassandra 3.11 on Linux, total data amount
> expected ~15-20 TB.
>
> Thank you very much,
>
> Marcus Haarmann
>

Re: multiple tables vs. partitions and TTL

[James Shaw]

if me, I will go 1 table, just think too much labor to manage many tables
and also how reliable while switching tables.
Regarding tombstones,  may try some ways to fight:
reasonable partition size  ( big partition with large tombstones will be a
problem);
don't query tombstones as possible,  in application coding, put timestamp >
expired time in where condition, so will not touch tombstones, in table
side,  timestamp column in cluster key, with desc order, so better
performance.

James

On Thu, Feb 1, 2018 at 3:16 AM, Marcus Haarmann 
wrote:

> Hi experts,
>
> I have a design issue here:
> We want to store bigger amounts of data (> 30mio rows containing blobs)
> which will be deleted depending on the type
> of data on a monthly base (not in the same order as the data entered the
> system).
> Some data would survive for two month only, other data for 3-5 years.
>
> The choice now is to have one table only with TTL per partition and
> partitions per deletion month (when the data should be deleted)
> which will allow a single delete command, followed by a compaction
> or alternatively to have multiple tables (one per month when the deletion
> process would just drop the table).
> The logic to retrieve that data is per record, so we know both the
> retention period and the id (uuid) of the addressed record,
> so multiple tables can be handled.
>
> Since it would be one table per deletion month, I do not expect more than
> 1000-2000 tables, depending on the
> retention period of the data.
>
> The benefit creating multiple tables would be that there are no tombstones
> while more tables take more memory in the nodes.
> The one table approach would make the compaction process take longer and
> produce more I/O activity because
> the compaction would regenerate multiple tables internally.
>
> Any thoughts on this ?
> We want to use 9 nodes, cassandra 3.11 on Linux, total data amount
> expected ~15-20 TB.
>
> Thank you very much,
>
> Marcus Haarmann
>

RE: Old tombstones not being cleaned up

[ZAIDI, ASAD A]

Make data consistent (run repair), reduce gc_grace_seconds (try set it to 0 
temporarily though careful as this can affect hinted handoff!)  and set table’s 
compaction sub-property i.e. unchecked_tombstone_compaction to true. Compaction 
 will  take care of tombstones!


From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Thursday, February 01, 2018 11:29 AM
To: user@cassandra.apache.org
Subject: Re: Old tombstones not being cleaned up

Changing the defaul TTL doesn’t change the TTL on the existing data, only new 
data. It’s only set if you don’t supply one yourself.

On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen 
> wrote:
Hi,

We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally 
runs fine, but we have one table that are causing OOMs because an enormous 
amount of tombstones.
Looking at the data in the table (sstable2json), the first of the tombstones 
are almost a year old. The table was initially created with a gc_grace_period 
of 10 days, but I have now lowered it to 1 hour.
I have run a full repair of the table across all nodes. I have forced several 
major compactions of the table by using "nodetool compact", and also tried to 
switch from LeveledCompaction to SizeTierCompaction and back.

What could cause cassandra to keep these tombstones?

sstable2json:
{"key": "foo",
 "cells": 
[["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
   
["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
   
["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
   
["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
   
["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
   
["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
   
["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
   
["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
   
["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
...
{"key": "bar",
 "metadata": {"deletionInfo": 
{"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
 "cells": 
[["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
   
["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
   
["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],


sstablemetadata:
stablemetadata 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
SSTable: 
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.10
Minimum timestamp: 1488976211688000
Maximum timestamp: 1517468644066000
SSTable max local deletion time: 2147483647
Compression ratio: 0.5121956624389545
Estimated droppable tombstones: 18.00161766553587
SSTable Level: 0
Repaired at: 0
ReplayPosition(segmentId=1517168739626, position=22690189)
Estimated tombstone drop times:%n
1488976211: 1
1489906506:  4706
1490174752:  6111
1490449759:  6554
1490735410:  6559
1491016789:  6369
1491347982: 10216
1491680214: 13502
...

desc:
CREATE TABLE xxx.yyy (
ti text,
uuid text,
json_data text,
PRIMARY KEY (ti, uuid)
) WITH CLUSTERING ORDER BY (uuid ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';

jmx props(picture):
[image.png]

Re: Nodes show different number of tokens than initially

[Oleksandr Shulgin]

On Thu, Feb 1, 2018 at 5:19 AM, Jeff Jirsa  wrote:

>
>> The reason I find it surprising, is that it makes very little *sense* to
>> put a token belonging to a mode from one DC between tokens of nodes from
>> another one.
>>
>
> I don't want to really turn this into an argument over what should and
> shouldn't make sense, but I do agree, it doesn't make sense to put a token
> on one node in one DC onto another node in another DC.
>

This is not what I was trying to say.  I should have used an example to
express myself clearer.  Here goes (disclaimer: it might sound like a rant,
take it with a grain of salt):

$ ccm create -v 3.0.15 -n 3:3 -s 2dcs

For a more meaningful multi-DC setup than the default SimpleStrategy, use
NTS:

$ ccm node1 cqlsh -e "ALTER KEYSPACE system_auth WITH replication =
{'class': 'NetworkTopologyStrategy', 'dc1': 2, 'dc2': 2};"

$ ccm node1 nodetool ring

Datacenter: dc1
==
AddressRackStatus State   LoadOwns
Token

3074457345618258602
127.0.0.1  r1  Up Normal  117.9 KB66.67%
-9223372036854775808
127.0.0.2  r1  Up Normal  131.56 KB   66.67%
-3074457345618258603
127.0.0.3  r1  Up Normal  117.88 KB   66.67%
3074457345618258602

Datacenter: dc2
==
AddressRackStatus State   LoadOwns
Token

3074457345618258702
127.0.0.4  r1  Up Normal  121.54 KB   66.67%
-9223372036854775708
127.0.0.5  r1  Up Normal  118.59 KB   66.67%
-3074457345618258503
127.0.0.6  r1  Up Normal  114.12 KB   66.67%
3074457345618258702

Note that CCM is aware of the cross-DC clashes and selects the tokens for
the dc2 shifted by a 100.

Then look at the token ring (output abbreviated and aligned by me):

$ ccm node1 nodetool describering system_auth

Schema Version:4f7d0ad0-350d-3ea0-ae8b-53d5bc34fc7e
TokenRange:
TokenRange(start_token:-9223372036854775808, end_token:-9223372036854775708,
endpoints:[127.0.0.4, 127.0.0.2, 127.0.0.5, 127.0.0.3], ... TokenRange(
start_token:-9223372036854775708, end_token:-3074457345618258603,
endpoints:[127.0.0.2, 127.0.0.5, 127.0.0.3, 127.0.0.6], ...
TokenRange(start_token:-3074457345618258603, end_token:-3074457345618258503,
endpoints:[127.0.0.5, 127.0.0.3, 127.0.0.6, 127.0.0.1], ...
TokenRange(start_token:-3074457345618258503, end_token:
3074457345618258602, endpoints:[127.0.0.3, 127.0.0.6, 127.0.0.1,
127.0.0.4], ... TokenRange(start_token: 3074457345618258602, end_token:
3074457345618258702, endpoints:[127.0.0.6, 127.0.0.1, 127.0.0.4,
127.0.0.2], ...
TokenRange(start_token: 3074457345618258702, end_token:-9223372036854775808,
endpoints:[127.0.0.1, 127.0.0.4, 127.0.0.2, 127.0.0.5], ...

So in this setup, every token range has one end contributed by a node from
dc1 and the other end -- from dc2.  That doesn't model anything in the real
topology of the cluster.

I see that it's easy to lump together tokens from all nodes and sort them,
to produce a single token ring (and this is obviously the reason why tokens
have to be unique throughout the cluster as a whole).  That doesn't mean
it's a meaningful thing to do.

This introduces complexity which not present in the problem domain
initially.  This was a deliberate choice of developers, dare I say, to
complect the separate DCs together in a single token ring.  This has
profound consequences from the operations side.  If anything, it prevents
bootstrapping multiple nodes at the same time even if they are in different
DCs.  Or would you suggest to set consistent_range_movement=false and hope
it will work out?

If the whole reason for having separate DCs is to provide isolation, I fail
to see how the single token ring design does anything towards achieving
that.

But also being very clear (I want to make sure I understand what you're
> saying): that's a manual thing you did, Cassandra didn't do it for you,
> right? The fact that Cassandra didn't STOP you from doing it could be
> considered a bug, but YOU made that config choice?
>

Yes, we have chosen exactly the same token for two nodes in different DCs
because we were unaware of this globally uniqueness requirement.  Yes, we
believe it's a bug that Cassandra didn't stop us from doing that.

You can trivially predict what would happen with SimpleStrategy in
> multi-DC: run nodetool ring, and the first RF nodes listed after a given
> token own that data, regardless of which DC they're in. Because it's all
> one big ring.


In any case I don't think SimpleStrategy is a valid argument to consider in
multi-DC setup.  It is true that you can start a cluster spanning multiple
DCs from scratich while using SimpleStrategy, but there is no way to add a
new DC to the cluster unless you go NTS, so why pulling this example?

Cheers,
--
Alex

Re: Old tombstones not being cleaned up

[Jonathan Haddad]

Changing the defaul TTL doesn’t change the TTL on the existing data, only
new data. It’s only set if you don’t supply one yourself.

On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen 
wrote:

> Hi,
>
> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
> generally runs fine, but we have one table that are causing OOMs because an
> enormous amount of tombstones.
> Looking at the data in the table (sstable2json), the first of the
> tombstones are almost a year old. The table was initially created with a
> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
> I have run a full repair of the table across all nodes. I have forced
> several major compactions of the table by using "nodetool compact", and
> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>
> What could cause cassandra to keep these tombstones?
>
> sstable2json:
> {"key": "foo",
>  "cells":
> [["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
>
>  
> ["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
>
>  
> ["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
>
>  
> ["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
>
>  
> ["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
>
>  
> ["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
>
>  
> ["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
>
>  
> ["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
>
>  
> ["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
> ...
> {"key": "bar",
>  "metadata": {"deletionInfo":
> {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
>  "cells":
> [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
>
>  
> ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
>
>  
> ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],
> 
>
> sstablemetadata:
> stablemetadata
> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
> SSTable:
> /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.10
> Minimum timestamp: 1488976211688000
> Maximum timestamp: 1517468644066000
> SSTable max local deletion time: 2147483647
> Compression ratio: 0.5121956624389545
> Estimated droppable tombstones: 18.00161766553587
> SSTable Level: 0
> Repaired at: 0
> ReplayPosition(segmentId=1517168739626, position=22690189)
> Estimated tombstone drop times:%n
> 1488976211: 1
> 1489906506:  4706
> 1490174752:  6111
> 1490449759:  6554
> 1490735410:  6559
> 1491016789:  6369
> 1491347982: 10216
> 1491680214: 13502
> ...
>
> desc:
> CREATE TABLE xxx.yyy (
> ti text,
> uuid text,
> json_data text,
> PRIMARY KEY (ti, uuid)
> ) WITH CLUSTERING ORDER BY (uuid ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 3600
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
>
> jmx props(picture):
> [image: image.png]
>

Re: TWCS not deleting expired sstables

[Thakrar, Jayesh]

Thank you so much Kurt - that helped!!

So here's the thing, I was logged into the server as my own id and had 
Cassandra binaries under my home directory with the default configuration.
Cassandra was running on the server with a service account and with its own 
user-id and configuration.

I assumed that like nodetool, this command would also use the Cassandra service 
JMX or server port.
However your suggestion made me realize that that's not the case.

So I copied the service account's Cassandra/conf to my local install and things 
worked.

I also looked up the source code and it did not indicate any "connection" via 
service or jmx port :)

So now need to see why the data is not getting purged.

Thanks again!!

From: kurt greaves 
Date: Wednesday, January 31, 2018 at 11:27 PM
To: User 
Subject: Re: TWCS not deleting expired sstables

Well, that shouldn't happen. Seems like it's possibly not looking in the 
correct location for data directories. Try setting CASSANDRA_INCLUDE=http://cassandra.in.sh>> prior to running the script?
e.g: 
CASSANDRA_INCLUDE=/cassandra.in.sh
 sstableexpiredblockers ae raw_logs_by_user

On 30 January 2018 at 15:34, Thakrar, Jayesh 
> wrote:
Thanks Kurt and Kenneth.

Now only if they would work as expected.

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
 -lt | tail
-rw-r--r--. 1 vchadoop vchadoop286889260 Sep 18 14:14 mc-1070-big-Index.db
-rw-r--r--. 1 vchadoop vchadoop12236 Sep 13 20:53 
mc-178-big-Statistics.db
-rw-r--r--. 1 vchadoop vchadoop   92 Sep 13 20:53 mc-178-big-TOC.txt
-rw-r--r--. 1 vchadoop vchadoop  9371211 Sep 13 20:53 
mc-178-big-CompressionInfo.db
-rw-r--r--. 1 vchadoop vchadoop   10 Sep 13 20:53 
mc-178-big-Digest.crc32
-rw-r--r--. 1 vchadoop vchadoop  13609890747 Sep 13 20:53 
mc-178-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  1394968 Sep 13 20:53 mc-178-big-Summary.db
-rw-r--r--. 1 vchadoop vchadoop 11172592 Sep 13 20:53 mc-178-big-Filter.db
-rw-r--r--. 1 vchadoop vchadoop190508739 Sep 13 20:53 mc-178-big-Index.db
drwxr-xr-x. 2 vchadoop vchadoop   10 Sep 12 21:47 backups

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
 ae raw_logs_by_user
Exception in thread "main" java.lang.IllegalArgumentException: Unknown 
keyspace/table ae.raw_logs_by_user
at 
org.apache.cassandra.tools.SSTableExpiredBlockers.main(SSTableExpiredBlockers.java:66)

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
 system peers
No sstables for system.peers

node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
 -l ../../system/peers-37f71aca7dc2383ba70672528af04d4f/
total 308
drwxr-xr-x. 2 vchadoop vchadoop 10 Sep 11 22:59 backups
-rw-rw-r--. 1 vchadoop vchadoop 83 Jan 25 02:11 
mc-137-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop 180369 Jan 25 02:11 mc-137-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 25 02:11 mc-137-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 64 Jan 25 02:11 mc-137-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop386 Jan 25 02:11 mc-137-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5171 Jan 25 02:11 mc-137-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 25 02:11 mc-137-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 25 02:11 mc-137-big-TOC.txt
-rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:11 
mc-138-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop   9723 Jan 29 21:11 mc-138-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:11 mc-138-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:11 mc-138-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop 17 Jan 29 21:11 mc-138-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5015 Jan 29 21:11 mc-138-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:11 mc-138-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:11 mc-138-big-TOC.txt
-rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:53 
mc-139-big-CompressionInfo.db
-rw-rw-r--. 1 vchadoop vchadoop  18908 Jan 29 21:53 mc-139-big-Data.db
-rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:53 mc-139-big-Digest.crc32
-rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:53 mc-139-big-Filter.db
-rw-rw-r--. 1 vchadoop vchadoop 36 Jan 29 21:53 mc-139-big-Index.db
-rw-rw-r--. 1 vchadoop vchadoop   5055 Jan 29 21:53 mc-139-big-Statistics.db
-rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:53 mc-139-big-Summary.db
-rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:53 mc-139-big-TOC.txt



From: Kenneth Brotman 
Date: Tuesday, January 30, 2018 at 7:37 AM
To: 

Re: Old tombstones not being cleaned up

[James Shaw]

i see leveled compaction used, if it's last, it will have to stay until
next level compaction happens, then will be gone, right ?

On Thu, Feb 1, 2018 at 2:33 AM, Bo Finnerup Madsen 
wrote:

> Hi,
>
> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
> generally runs fine, but we have one table that are causing OOMs because an
> enormous amount of tombstones.
> Looking at the data in the table (sstable2json), the first of the
> tombstones are almost a year old. The table was initially created with a
> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
> I have run a full repair of the table across all nodes. I have forced
> several major compactions of the table by using "nodetool compact", and
> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>
> What could cause cassandra to keep these tombstones?
>
> sstable2json:
> {"key": "foo",
>  "cells": [["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-
> 25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
>["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-
> c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
>["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-
> ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
>["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-
> ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
>["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-
> d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],
>["2777-4b1a-4267-8efc-c43054e63170:_","2777-
> 4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],
>["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-
> f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],
>["63da-f165-449b-b65d-2b7869368734:_","63da-
> f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],
>["656f-f8b5-472b-93ed-1a893002f027:_","656f-
> f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
> ...
> {"key": "bar",
>  "metadata": {"deletionInfo": {"markedForDeleteAt":1517402198585982,"
> localDeletionTime":1517402198}},
>  "cells": [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-
> ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],
>["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-
> 7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],
>["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-
> aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],
> 
>
> sstablemetadata:
> stablemetadata /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1
> c6/ddp-yyy-ka-2741-Data.db
> SSTable: /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1
> c6/ddp-yyy-ka-2741
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.10
> Minimum timestamp: 1488976211688000
> Maximum timestamp: 1517468644066000
> SSTable max local deletion time: 2147483647 <(214)%20748-3647>
> Compression ratio: 0.5121956624389545
> Estimated droppable tombstones: 18.00161766553587
> SSTable Level: 0
> Repaired at: 0
> ReplayPosition(segmentId=1517168739626, position=22690189)
> Estimated tombstone drop times:%n
> 1488976211: 1
> 1489906506:  4706
> 1490174752:  6111
> 1490449759:  6554
> 1490735410:  6559
> 1491016789:  6369
> 1491347982: 10216
> 1491680214: 13502
> ...
>
> desc:
> CREATE TABLE xxx.yyy (
> ti text,
> uuid text,
> json_data text,
> PRIMARY KEY (ti, uuid)
> ) WITH CLUSTERING ORDER BY (uuid ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> LeveledCompactionStrategy'}
> AND compression = {'sstable_compression': 'org.apache.cassandra.io.
> compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 3600
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
>
> jmx props(picture):
> [image: image.png]
>

Re: Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

On Thu, Feb 1, 2018 at 9:23 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On 1 Feb 2018 06:51, "kurt greaves"  wrote:
>
> Would you be able to create a JIRA ticket for this? Not sure if this is
> still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
> really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
> it's an issue there as well.​
>
>
> Our next planned step is to upgrade to 3.0.15, so we will see pretty soon
> if that's still an issue in the newer version.
>

Still the same issue with 3.0.  I've opened a ticket:
https://issues.apache.org/jira/browse/CASSANDRA-14210

--
Alex

RE: group by select queries

[Modha, Digant]

Jira created: CASSANDRA-14209.


From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Thursday, February 01, 2018 12:38 AM
To: User
Subject: Re: group by select queries

Seems problematic. Would you be able to create a JIRA ticket with the above 
information/examples?

On 30 January 2018 at 22:41, Modha, Digant 
> wrote:
It was local quorum.  There’s no difference with CONSISTENCY ALL.

Consistency level set to LOCAL_QUORUM.
cassandra@cqlsh> select * from wp.position  where account_id = 'user_1';

account_id | security_id | counter | avg_exec_price | pending_quantity | 
quantity | transaction_id | update_time
+-+-++--+--++-
 user_1 |AMZN |   2 | 1239.2 |0 | 
1011 |   null | 2018-01-25 17:18:07.158000+
 user_1 |AMZN |   1 | 1239.2 |0 | 
1010 |   null | 2018-01-25 17:18:07.158000+

(2 rows)
cassandra@cqlsh> select * from wp.position  where account_id = 'user_1' group 
by security_id;

account_id | security_id | counter | avg_exec_price | pending_quantity | 
quantity | transaction_id | update_time
+-+-++--+--++-
 user_1 |AMZN |   1 | 1239.2 |0 | 
1010 |   null | 2018-01-25 17:18:07.158000+

(1 rows)
cassandra@cqlsh> select account_id,security_id, counter, 
avg_exec_price,quantity, update_time from wp.position  where account_id = 
'user_1' group by security_id ;

account_id | security_id | counter | avg_exec_price | quantity | update_time
+-+-++--+-
 user_1 |AMZN |   2 | 1239.2 | 1011 | 2018-01-25 
17:18:07.158000+

(1 rows)
cassandra@cqlsh>  consistency all;
Consistency level set to ALL.
cassandra@cqlsh> select * from wp.position  where account_id = 'user_1' group 
by security_id;

account_id | security_id | counter | avg_exec_price | pending_quantity | 
quantity | transaction_id | update_time
+-+-++--+--++-
 user_1 |AMZN |   1 | 1239.2 |0 | 
1010 |   null | 2018-01-25 17:18:07.158000+

(1 rows)
cassandra@cqlsh> select account_id,security_id, counter, 
avg_exec_price,quantity, update_time from wp.position  where account_id = 
'user_1' group by security_id ;

account_id | security_id | counter | avg_exec_price | quantity | update_time
+-+-++--+-
 user_1 |AMZN |   2 | 1239.2 | 1011 | 2018-01-25 
17:18:07.158000+


From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Monday, January 29, 2018 11:03 PM
To: User
Subject: Re: group by select queries

What consistency were you querying at? Can you retry with CONSISTENCY ALL?

​


TD Securities disclaims any liability or losses either direct or consequential 
caused by the use of this information. This communication is for informational 
purposes only and is not intended as an offer or solicitation for the purchase 
or sale of any financial instrument or as an official confirmation of any 
transaction. TD Securities is neither making any investment recommendation nor 
providing any professional or advisory services relating to the activities 
described herein. All market prices, data and other information are not 
warranted as to completeness or accuracy and are subject to change without 
notice Any products described herein are (i) not insured by the FDIC, (ii) not 
a deposit or other obligation of, or guaranteed by, an insured depository 
institution and (iii) subject to investment risks, including possible loss of 
the principal amount invested. The information shall not be further distributed 
or duplicated in whole or in part by any means without the prior written 
consent of TD Securities. TD Securities is a trademark of The Toronto-Dominion 
Bank and represents TD Securities (USA) LLC and certain investment banking 
activities of The Toronto-Dominion Bank and its subsidiaries.


TD Securities disclaims any liability or losses either direct or consequential 
caused by the use of this information. This communication is for informational 
purposes only and is not intended as an offer or solicitation for the purchase 
or sale of any financial instrument or as an official confirmation of any 
transaction. TD Securities is neither making any investment recommendation nor 
providing any professional or advisory services relating to the activities 
described 

Documentation question

[Mikael]

Hi!

I have a commercial application that use Cassandra and the manual for it 
includes a link to the Cassandra documentation, but I will include the 
most basic stuff in the manual like how to setup Cassandra, 
configuration and so on, so the question is if I need to write this 
myself from scratch or can I include parts of the Apache Cassandra 
documentation in my manual ?


Mikael



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: group by select queries

[DuyHai Doan]

Worth digging into the source code of GROUP BY but as far as I remember,
using GROUP BY without any aggregation function will lead to C* picking
just the first row (or maybe last, not sure on this point) row at hand.

About ordering, since the grouping is on a component of partition key, do
not expect any sensible order since only token order matters

On Thu, Feb 1, 2018 at 6:38 AM, kurt greaves  wrote:

> Seems problematic. Would you be able to create a JIRA ticket with the
> above information/examples?
>
> On 30 January 2018 at 22:41, Modha, Digant 
> wrote:
>
>> It was local quorum.  There’s no difference with CONSISTENCY ALL.
>>
>>
>>
>> Consistency level set to LOCAL_QUORUM.
>>
>> cassandra@cqlsh> select * from wp.position  where account_id = 'user_1';
>>
>>
>>
>> account_id | security_id | counter | avg_exec_price | pending_quantity |
>> quantity | transaction_id | update_time
>>
>> +-+-++--
>> +--++---
>> --
>>
>>  user_1 |AMZN |   2 | 1239.2 |0
>> | 1011 |   null | 2018-01-25 17:18:07.158000+
>>
>>  user_1 |AMZN |   1 | 1239.2 |0
>> | 1010 |   null | 2018-01-25 17:18:07.158000+
>>
>>
>>
>> (2 rows)
>>
>> cassandra@cqlsh> select * from wp.position  where account_id = 'user_1'
>> group by security_id;
>>
>>
>>
>> account_id | security_id | counter | avg_exec_price | pending_quantity |
>> quantity | transaction_id | update_time
>>
>> +-+-++--
>> +--++---
>> --
>>
>>  user_1 |AMZN |   1 | 1239.2 |0
>> | 1010 |   null | 2018-01-25 17:18:07.158000+
>>
>>
>>
>> (1 rows)
>>
>> cassandra@cqlsh> select account_id,security_id, counter,
>> avg_exec_price,quantity, update_time from wp.position  where account_id =
>> 'user_1' group by security_id ;
>>
>>
>>
>> account_id | security_id | counter | avg_exec_price | quantity |
>> update_time
>>
>> +-+-++--
>> +-
>>
>>  user_1 |AMZN |   2 | 1239.2 | 1011 |
>> 2018-01-25 17:18:07.158000+
>>
>>
>>
>> (1 rows)
>>
>> cassandra@cqlsh>  consistency all;
>>
>> Consistency level set to ALL.
>>
>> cassandra@cqlsh> select * from wp.position  where account_id = 'user_1'
>> group by security_id;
>>
>>
>>
>> account_id | security_id | counter | avg_exec_price | pending_quantity |
>> quantity | transaction_id | update_time
>>
>> +-+-++--
>> +--++---
>> --
>>
>>  user_1 |AMZN |   1 | 1239.2 |0
>> | 1010 |   null | 2018-01-25 17:18:07.158000+
>>
>>
>>
>> (1 rows)
>>
>> cassandra@cqlsh> select account_id,security_id, counter,
>> avg_exec_price,quantity, update_time from wp.position  where account_id =
>> 'user_1' group by security_id ;
>>
>>
>>
>> account_id | security_id | counter | avg_exec_price | quantity |
>> update_time
>>
>> +-+-++--
>> +-
>>
>>  user_1 |AMZN |   2 | 1239.2 | 1011 |
>> 2018-01-25 17:18:07.158000+
>>
>>
>>
>>
>>
>> *From:* kurt greaves [mailto:k...@instaclustr.com]
>> *Sent:* Monday, January 29, 2018 11:03 PM
>> *To:* User
>> *Subject:* Re: group by select queries
>>
>>
>>
>> What consistency were you querying at? Can you retry with CONSISTENCY ALL?
>>
>>
>>
>> ​
>>
>>
>> TD Securities disclaims any liability or losses either direct or
>> consequential caused by the use of this information. This communication is
>> for informational purposes only and is not intended as an offer or
>> solicitation for the purchase or sale of any financial instrument or as an
>> official confirmation of any transaction. TD Securities is neither making
>> any investment recommendation nor providing any professional or advisory
>> services relating to the activities described herein. All market prices,
>> data and other information are not warranted as to completeness or accuracy
>> and are subject to change without notice Any products described herein are
>> (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or
>> guaranteed by, an insured depository institution and (iii) subject to
>> investment risks, including possible loss of the principal amount invested.
>> The information shall not be further distributed or duplicated in whole or
>> in part by any means without the prior written consent of TD Securities. TD
>> Securities is a trademark of The Toronto-Dominion Bank and represents TD
>> Securities (USA) 

Re: Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

On 1 Feb 2018 06:51, "kurt greaves"  wrote:

Would you be able to create a JIRA ticket for this? Not sure if this is
still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
it's an issue there as well.​


Our next planned step is to upgrade to 3.0.15, so we will see pretty soon
if that's still an issue in the newer version.

--
Alex

multiple tables vs. partitions and TTL

[Marcus Haarmann]

Hi experts, 

I have a design issue here: 
We want to store bigger amounts of data (> 30mio rows containing blobs) which 
will be deleted depending on the type 
of data on a monthly base (not in the same order as the data entered the 
system). 
Some data would survive for two month only, other data for 3-5 years. 

The choice now is to have one table only with TTL per partition and partitions 
per deletion month (when the data should be deleted) 
which will allow a single delete command, followed by a compaction 
or alternatively to have multiple tables (one per month when the deletion 
process would just drop the table). 
The logic to retrieve that data is per record, so we know both the retention 
period and the id (uuid) of the addressed record, 
so multiple tables can be handled. 

Since it would be one table per deletion month, I do not expect more than 
1000-2000 tables, depending on the 
retention period of the data. 

The benefit creating multiple tables would be that there are no tombstones 
while more tables take more memory in the nodes. 
The one table approach would make the compaction process take longer and 
produce more I/O activity because 
the compaction would regenerate multiple tables internally. 

Any thoughts on this ? 
We want to use 9 nodes, cassandra 3.11 on Linux, total data amount expected 
~15-20 TB. 

Thank you very much, 

Marcus Haarmann 

Re: Not what I‘ve expected Performance

[Jürgen Albersdorfer]

Hi Kurt, thanks for your response.
I indeed utilized Spark - what I've forgot to mention - and I did it nearly
the same as in the example you gave me.
Just without that .select(PK).sample(false, 0.1) Instruction which I don't
actually get what it's useful for - and maybe that's the key to the castle.

I already found out that I require some more Spark Executors - really lots
of them.
And it was a bad Idea in the first place to ./spark-submit without any
parameters about executor-memory, total-executor-cores and especially
executor-cores.
I now submitted it with --executor-cores 1 --total-executor-cores 100 --
executor-memory 8G to get more Executors out of my Cluster.
Without that limits, a Spark Executor will utilize all of the available
cores. With the limitations, The Spark Worker will be able to start more
Workers in parallel which boosts in my example,
but is still way to slow and far away from requiring to throttle it. And
that is what I actually expected when 100 Processes start beating with the
Database Cluster.

Definitelly I'll give your Code a try.

2018-02-01 6:36 GMT+01:00 kurt greaves :

> How are you copying? With CQLSH COPY or your own script? If you've got
> spark already it's quite simple to copy between tables and it should be
> pretty much as fast as you can get it. (you may even need to throttle).
> There's some sample code here (albeit it's copying between clusters but
> easily tailored to copy between tables). https://www.
> instaclustr.com/support/documentation/apache-spark/
> using-spark-to-sample-data-from-one-cassandra-cluster-
> and-write-to-another/
>
> On 30 January 2018 at 21:05, Jürgen Albersdorfer 
> wrote:
>
>> Hi, We are using C* 3.11.1 with a 9 Node Cluster built on CentOS Servers
>> eac=
>> h having 2x Quad Core Xeon, 128GB of RAM and two separate 2TB spinning
>> Disks=
>> , one for Log one for Data with Spark on Top.
>>
>> Due to bad Schema (Partitions of about 4 to 8 GB) I need to copy a whole
>> Tab=
>> le into another having same fields but different partitioning.=20
>>
>> I expected glowing Iron when I started the copy Job, but instead cannot
>> even=
>> See some Impact on CPU, mem or disks. - but the Job does copy the Data
>> over=
>> veeerry slowly at about a MB or two per Minute.
>>
>> Any suggestion where to start investigation?
>>
>> Thanks already
>>
>> Von meinem iPhone gesendet
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>