Re: TWCS not deleting expired sstables

[Thakrar, Jayesh]

Hi Brian,

Thanks for your response.
Yes, I did look at that post.
In fact after reading that post, I set the "unchecked_tombstone_compaction" to 
true.

For the sstable in the example below (and its neighbors), all the data for 
those time windows have been compacted into a single sstable, so there is no 
dependency or delay caused from other sstables.

Thanks,
Jayesh


From: 
Date: Sunday, January 28, 2018 at 1:02 PM
To: 
Subject: Re: TWCS not deleting expired sstables

I would start here:  http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html

Specifically the “Hints and repairs” and “Timestamp overlap” sections might be 
of use.
-B

On Jan 25, 2018, at 11:05 AM, Thakrar, Jayesh 
> wrote:
Wondering if I can get some pointers to what's happening here and why sstables 
that I think should be expired are not being dropped.

Here's the table's compaction property - note also set 
"unchecked_tombstone_compaction" to true.

compaction = {'class': 
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
'compaction_window_size': '7', 'compaction_window_unit': 'DAYS', 
'max_threshold': '4', 'min_threshold': '4', 'unchecked_tombstone_compaction': 
'true'}

We insert data with timestamp and TTL programmatically.

Here's one set of sstable that I expect to be removed:

$ ls -lt *Data.db | tail -5
-rw-r--r--. 1 vchadoop vchadoop  31245097312 Sep 20 17:16 mc-1308-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  31524316252 Sep 19 14:27 mc-1187-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  21405216502 Sep 18 14:14 mc-1070-big-Data.db
-rw-r--r--. 1 vchadoop vchadoop  13609890747 Sep 13 20:53 mc-178-big-Data.db

$ date +%s
1516895877

$ date
Thu Jan 25 15:58:00 UTC 2018

$ sstablemetadata $PWD/mc-130-big-Data.db | head -20
SSTable: 
/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1/mc-130-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.01
Minimum timestamp: 14966028
Maximum timestamp: 14980788
SSTable min local deletion time: 1507924954
SSTable max local deletion time: 1509400832
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 0.17430158132352797
TTL min: 2630598
TTL max: 4086188
First token: -9177441867697829836 (key=823134638755651936)
Last token: 9155171035305804798 (key=395118640769012487)
minClustringValues: [-1, da, 3, 1498082382078, -9223371818124305448, 
-9223371652504795402, -1]
maxClustringValues: [61818, tpt, 325, 149660280, -4611686088173246790, 
9223372014135560885, 1]
Estimated droppable tombstones: 1.1983492967652476
SSTable Level: 0
Repaired at: 0
Replay positions covered: {CommitLogPosition(segmentId=1505171071629, 
position=7157684)=CommitLogPosition(segmentId=1505171075152, position=6263269)}
totalColumnsSet: 111047277

Re: What happens if multiple processes send create table if not exist statement to cassandra?

[Nate McCall]

> Thanks a lot for that explanation Jeff!! I am trying to see if there is
> any JIRA ticket that talks about incorporating LWT in scenarios you
> mentioned?
>

https://issues.apache.org/jira/browse/CASSANDRA-10699

Re: Cassandra Repair Duration.

[Brian Spindler]

It's all here:
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesWhen.html


-B


On Thu, Jan 25, 2018 at 6:08 AM Karthick V  wrote:

> *You can (and probably should) run repairs as apart of routine
>> maintenance.*
>
>
>  Can u explain any use case for why do we need this?
>
>
>
>
>
> On Wed, Jan 24, 2018 at 5:35 PM,  wrote:
>
>> Hi Karthick, repairs can be tricky.
>>
>> You can (and probably should) run repairs as apart of routine
>> maintenance.  And of course absolutely if you lose a node in a bad way.  If
>> you decommission a node for example, no “extra” repair needed.
>>
>> If you are using TWCS you should probably not run repairs on those cf.
>>
>> We have a combination of scripts and locks to run repairs across an 18
>> node cluster 1 node at a time, typically takes around 2-3days and so we run
>> it once a week.
>>
>> The great folks at tlp have put together http://cassandra-reaper.io/ which
>> makes managing repairs even easier and probably more performant since as I
>> understand, it used range repairs.
>>
>> Good luck,
>> -B
>>
>>
>> On Jan 24, 2018, at 4:57 AM, Karthick V  wrote:
>>
>> Periodically I have been running Full repair process befor GC Grace
>> period as mentioned in the best practices.Initially, all went well but as
>> the data size increases Repair duration has increased drastically and we
>> are also facing Query timeouts during that time and we have tried
>> incremental repair facing some OOM issues.
>>
>> After running a repair process for more than 80 Hours we have ended up
>> with the question
>>
>> why can't we run a repair process if and only if a Cassandra node got a
>> downtime?
>>
>> Say if there is no downtime during a GC grace period Do we still face
>> Inconsistency among nodes? if yes, then doesn't Hinted Handoff handle
>> those?
>>
>> Cluster Info: Having two DataCenter with 8 machines each with a disk size
>> of 1TB, C* v_2.1.13  and having around 420GB data each.
>>
>> On Wed, Jan 24, 2018 at 2:46 PM, Karthick V 
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Re: TWCS not deleting expired sstables

[brian . spindler]

I would start here:  http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html

Specifically the “Hints and repairs” and “Timestamp overlap” sections might be 
of use.  

-B

> On Jan 25, 2018, at 11:05 AM, Thakrar, Jayesh  
> wrote:
> 
> Wondering if I can get some pointers to what's happening here and why 
> sstables that I think should be expired are not being dropped.
>  
> Here's the table's compaction property - note also set 
> "unchecked_tombstone_compaction" to true.
>  
> compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '7', 'compaction_window_unit': 'DAYS', 
> 'max_threshold': '4', 'min_threshold': '4', 'unchecked_tombstone_compaction': 
> 'true'}
>  
> We insert data with timestamp and TTL programmatically.
>  
> Here's one set of sstable that I expect to be removed:
>  
> $ ls -lt *Data.db | tail -5
> -rw-r--r--. 1 vchadoop vchadoop  31245097312 Sep 20 17:16 mc-1308-big-Data.db
> -rw-r--r--. 1 vchadoop vchadoop  31524316252 Sep 19 14:27 mc-1187-big-Data.db
> -rw-r--r--. 1 vchadoop vchadoop  21405216502 Sep 18 14:14 mc-1070-big-Data.db
> -rw-r--r--. 1 vchadoop vchadoop  13609890747 Sep 13 20:53 mc-178-big-Data.db
>  
> $ date +%s
> 1516895877
>  
> $ date 
> Thu Jan 25 15:58:00 UTC 2018
>  
> $ sstablemetadata $PWD/mc-130-big-Data.db | head -20
> SSTable: 
> /ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1/mc-130-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.01
> Minimum timestamp: 14966028
> Maximum timestamp: 14980788
> SSTable min local deletion time: 1507924954
> SSTable max local deletion time: 1509400832
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.17430158132352797
> TTL min: 2630598
> TTL max: 4086188
> First token: -9177441867697829836 (key=823134638755651936)
> Last token: 9155171035305804798 (key=395118640769012487)
> minClustringValues: [-1, da, 3, 1498082382078, -9223371818124305448, 
> -9223371652504795402, -1]
> maxClustringValues: [61818, tpt, 325, 149660280, -4611686088173246790, 
> 9223372014135560885, 1]
> Estimated droppable tombstones: 1.1983492967652476
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1505171071629, 
> position=7157684)=CommitLogPosition(segmentId=1505171075152, 
> position=6263269)}
> totalColumnsSet: 111047277
>  

Re: What happens if multiple processes send create table if not exist statement to cassandra?

[Kant Kodali]

Thanks a lot for that explanation Jeff!! I am trying to see if there is any 
JIRA ticket that talks about incorporating LWT in scenarios you mentioned?

Sent from my iPhone

> On Jan 27, 2018, at 1:41 PM, Jeff Jirsa  wrote:
> 
> Originally we would make tables based on keyspace name / table name pairs, 
> which was fine unless you dropped a table and recreated it, which could 
> happen while one node was offline / split network / gc pause. The recreation 
> scenario could allow data to be resurrected after a drop. 
> 
> So we augmented that (years and years ago) to have a uuid identifier for the 
> table, so now we can differentiate between table creations - if you drop a 
> table and recreate it, the new table has a different id.
> 
> However, if you issue a create table on two instances at the same time, 
> neither thinks the table exists, each generates their own cfid, two ids get 
> created. Schema eventually gets store inside Cassandra, so last write wins, 
> and the first ID seen gets stomped by the second. The race typically 
> manifests as one instance throwing errors about cfid not found, or a data 
> directory that doesn’t match the cfid in the schema (so a restart creates an 
> empty data directory), or similar situations like that.
> 
> The actual plumbing to use strong consistency (actually do paxos or some 
> other election to make sure exactly one id wins) is planned, likely for 4.0, 
> but doesn’t exist in any released version now
> 
> So again, don’t programmatically create tables if there’s a race possible, it 
> may work fine most of the time, but there’s a risk of ugly failure.
> 
> -- 
> Jeff Jirsa
> 
> 
>> On Jan 27, 2018, at 1:23 PM, Kant Kodali  wrote:
>> 
>> May I know why? 
>> 
>> Sent from my iPhone
>> 
>>> On Jan 27, 2018, at 12:36 PM, Jeff Jirsa  wrote:
>>> 
>>> Yes it causes issues
>>> 
>>> 
>>> -- 
>>> Jeff Jirsa
>>> 
>>> 
 On Jan 27, 2018, at 12:17 PM, Kant Kodali  wrote:
 
 Schema changes I assume you guys are talking about different create table 
 or alter table statements. What if multiple threads issue same exact 
 create table if not exists statement? Will that cause issues?
 
 Sent from my iPhone
 
> On Jan 27, 2018, at 11:41 AM, Carlos Rolo  wrote:
> 
> Don't do that. Worst case you might get different schemas in flight and 
> no agreement on your cluster.  If you are already doing that, check 
> "nodetool describecluster" after you do that.
> 
> Like Jeff said, it is likely to cause problems.
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: 
> linkedin.com/in/carlosjuzarterolo 
> Mobile: +351 918 918 100 
> www.pythian.com
> 
>> On Sat, Jan 27, 2018 at 7:25 PM, Jeff Jirsa  wrote:
>> It’s not LWT. Don’t do programmatic schema changes that can race, it’s 
>> likely to cause problems
>> 
>> 
>> --
>> Jeff Jirsa
>> 
>> 
>> > On Jan 27, 2018, at 10:19 AM, Kant Kodali  wrote:
>> >
>> > Hi All,
>> >
>> > What happens if multiple processes send create table if not exist 
>> > statement to cassandra? will there be any data corruption or any other 
>> > issues if I send "create table if not exist" request often?
>> >
>> > I dont see any entry in system.paxos table so is it fair to say "IF 
>> > NOT EXISTS" doesn't automatically imply LWT?
>> >
>> > Thanks!
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> 
> --
> 
> 
> 
> 

Re: AbstractMethodError from JMXServerUtils after update from Java 1.8.0_112 to 1.8.0_162

[Edward Ribeiro]

FYI, there's a patch coming soon to address this:
https://issues.apache.org/jira/browse/CASSANDRA-14173

Cheers,
Ed

On Thu, Jan 18, 2018 at 7:46 PM, Michael Shuler 
wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-14173
>
> On 01/18/2018 03:29 PM, Stephen Rosenthal wrote:
> > Hi,
> >
> >
> >
> > I got the following error after updating my Cassandra system from Java
> > 1.8.0_112 to 1.8.0_162:
> >
> >
> >
> > java.lang.AbstractMethodError:
> > org.apache.cassandra.utils.JMXServerUtils$Exporter.
> exportObject(Ljava/rmi/Remote;ILjava/rmi/server/
> RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/
> misc/ObjectInputFilter;)Ljava/rmi/Remote;
> >
> >
> >
> > I did the usual Googling and here’s what I’ve concluded:
> >
> >   * Between Java 1.8.0_152 and 1.8.0_162, the interface
> > com.sun.jmx.remote.internal.RMIExporter was changed to add a 5^th
> > argument to the exportObject method. The class in a Sun “internal”
> > package so it probably wasn’t intended for use outside of the JDK.
> >   * Cassandra references RMIExporter in
> > org.apache.cassandra.utils.JMXServerUtils but uses only 4 arguments
> > in the exportObject method:
> > https://github.com/apache/cassandra/blob/cassandra-3.11/
> src/java/org/apache/cassandra/utils/JMXServerUtils.java
> >   * I was running on Cassandra 3.11 but I also tried 3.11.1 and the
> > problem remained.
> >
> >
> >
> > I couldn’t find anyone else reporting this bug so I must be doing
> > something different. Have others seen this bug? Or is it something
> > obvious, i.e. does Cassandra not support running on Java 1.8.0_162 yet?
> >
> >
> >
> > Thanks!
> >
> > Stephen
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

high load on node

[Jerome Basa]

hi,

so there’s a compaction that’s stuck on 100% and it’s not progressing at
all. also, i’ve noticed that the load is high on the node. restarting C*
fixed the issue. any idea what’s the root cause? thanks

$ nodetool compactionstats
pending tasks: 2
 id   compaction type   keyspace
tablecompletedtotalunit   progress
   a5963a50-03b5-11e8-94d9-953cd75aaad2Compaction system
paxos   4882104332   488276   bytes100.00%
Active compaction remaining time :   0h00m00s

cassandra version: 3.0.14

regards,

-jerome