RE: Java 11 support in Cassandra 4.0 + Early Testing and Feedback

[Kyrylo Lebediev]

As many people use Oracle JDK, I think it worth mentioning that according 
Oracle Support Roadmap there are some changes in their policies for Java 11 and 
above (http://www.oracle.com/technetwork/java/eol-135779.html).

In particular:
“Starting with Java SE 9, in addition to providing Oracle JDK for free under 
the BCL, Oracle 
also started providing builds of OpenJDK under an 
open source license (similar to 
that of Linux). Oracle is working to make the Oracle JDK and OpenJDK builds 
from Oracle 
interchangeable
 - targeting developers and organizations that do not want commercial support 
or enterprise management tools. Beginning with Oracle Java SE 11 (18.9 LTS), 
the Oracle JDK will continue to be available royalty-free for development, 
testing, prototyping or demonstrating purposes. As announced in September 
2017,
 with the OracleJDK and builds of Oracle OpenJDK being interchangeable for 
releases of Java SE 11 and later, the Oracle JDK will primarily be for 
commercial and support customers and OpenJDK builds from Oracle are for those 
who do not want commercial support or enterprise management tools.”

What these statements mean for Cassandra users in terms of which JDK (there are 
several OSS alternatives available) is the best to use in case of absence of 
active Oracle/Java Subscription?

Regards,
Kyrill

From: Jonathan Haddad 
Sent: Thursday, August 16, 2018 9:02 PM
To: user 
Subject: Java 11 support in Cassandra 4.0 + Early Testing and Feedback

Hey folks,

As we start to get ready to feature freeze trunk for 4.0, it's going to be 
important to get a lot of community feedback.  This is going to be a big 
release for a number of reasons.

* Virtual tables.  Finally a nice way of querying for system metrics & status
* Streaming optimizations 
(https://cassandra.apache.org/blog/2018/08/07/faster_streaming_in_cassandra.html)
* Groundwork for strongly consistent schema changes
* Optimizations to internode communcation
* Experimental support for Java 11

I (and many others) would like Cassandra to be rock solid on day one if its 
release.  The best way to ensure that happens is if people provide feedback.  
One of the areas we're going to need a lot of feedback on is on how things work 
with Java 11, especially if you have a way of simulating a real world workload 
on a staging cluster.  I've written up instructions here on how to start 
testing: http://thelastpickle.com/blog/2018/08/16/java11.html

Java 11 hasn't been released yet, but that doesn't mean it's not a good time to 
test.  Any bugs we can identify now will help us get to a stable release 
faster.  If you rely on Cassandra for your business, please take some time to 
participate in the spirit of OSS by helping test & provide feedback to the team.

Thanks everyone!
---
Jon Haddad
Principal Consultant, The Last Pickle

RE: [EXTERNAL] Re: adding multiple node to a cluster, cleanup and num_tokens

[onmstester onmstester]

Why not setting default vnodes count to that recommendation in Cassandra 
installation files?  Sent using Zoho Mail  On Tue, 04 Sep 2018 17:35:54 
+0430 Durity, Sean R  wrote    Longer term, I 
agree with Oleksandr, the recommendation for number of vnodes is now much 
smaller than 256. I am using 8 or 16.     Sean Durity  

Failure reason for batch conditional update

[Long Quanzheng]

Hi
I did a logged batch of conditional update, and sometimes(not always) got
failure, and return of conflicting columns show that there is no
conflicting value. Is there any other reason that batch can fail?

My batch is like:
```
update tblA set col1=1 where pk=1 and col1=1; //to make sure no other
update on this row
update tblA set col2=2,col3=3 where pk=2 and col2=1;
update tblA set col4=4,col5=5 where pk=2 and col2=1;//because there are too
many columns on this row to update, we use multiple queries to make it
looks better
update tblA set col7=7,col6=6 where pk=3 and col7=7;// also make sure no
other update on this row
``

And sometimes it would return applied=false, with returning that
the row of pk=1 matches the condition
the row of pk=2 matches the condition, but applied=false
the row of pk=3 matches the condition, but applied=false

Is there any explanation?

I am using latest gocql, cassandra is Cassandra 3.0.14.10
-- 

Thanks,
Quanzheng

Fwd: Re: Cluster CPU usage limit

[onmstester onmstester]

Actually its 256 Native Transport threads, the number of concurrent threads on 
each node is 32. My main concern is, What amount of CPU capacity should i keep 
free for tasks other than write that includes compaction and read? Sent using 
Zoho Mail  Forwarded message  From : Elliott Sims 
 To :  Date : Fri, 07 Sep 
2018 08:05:27 +0430 Subject : Re: Cluster CPU usage limit  
Forwarded message  It's interesting and a bit surprising that 256 
write threads isn't enough.  Even with a lot of cores, I'd expect you to be 
able to saturate CPU with that many threads.  I'd make sure you don't have 
other bottlenecks, like GC, IOPs, network, or "microbursts" where your load is 
actually fluctuating between 20-100% CPU. Admittedly, I actually did get best 
results with 256 threads (and haven't tested higher, but lower is definitely 
not enough), but every advice I've seen is for a lower write thread count being 
optimal for most cases. On Thu, Sep 6, 2018 at 5:51 AM, onmstester onmstester 
 wrote: IMHO, Cassandra write is more of a CPU bound task, 
so while determining cluster write throughput, what CPU usage percent (avg 
among all cluster nodes) should be determined as limit?   Rephrase: what's the 
normal CPU usage in Cassandra cluster (while no compaction, streaming or 
heavy-read running) ? For a cluster with 10 nodes, i got 700K write per seconds 
for my data model, average cpu load is about 40%, i'm going to increase number 
of native threads (now is 256) and native queue (1024) to increase throughput 
(and CPU usage subsequently). Sent using Zoho Mail

Re: Too many tombstones using TTL

[Charulata Sharma (charshar)]

Hi,
I have struggled a lot with tombstones and finally learnt the following:


-  Deletes are not the only operation that cause tombstones. Check if 
you are inserting any nulls in any of the table columns.

If yes then if you use Prepared statements, then you can unset the null value.

-  You can forcibly force garbage collection on the specific table and 
this makes a huge difference.

(You can read my blog on this. I have mentioned all the steps that we carried 
out. )
https://medium.com/cassandra-tombstones-clearing-use-case/the-curios-case-of-tombstones-d897f681a378




Thanks,
Charu



From: Python_Max 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, January 16, 2018 at 7:26 AM
To: "user@cassandra.apache.org" 
Subject: Re: Too many tombstones using TTL

Thanks for a very helpful reply.
Will try to refactor the code accordingly.

On Tue, Jan 16, 2018 at 4:36 PM, Alexander Dejanovski 
mailto:a...@thelastpickle.com>> wrote:
I would not plan on deleting data at the row level as you'll end up with a lot 
of tombstones eventually (and you won't even notice them).
It's not healthy to allow that many tombstones to be read, and while your 
latency may fit your SLA now, it may not in the future.
Tombstones are going to create a lot of heap pressure and eventually trigger 
long GC pauses, which then tend to affect the whole cluster (a slow node is 
worse than a down node).

You should definitely separate data that is TTLed and data that is not in 
different tables so that you can adjust compaction strategies, gc_grace_seconds 
and read patterns accordingly. I understand that it will complexify your code, 
but it will prevent severe performance issues in Cassandra.

Tombstones won't be a problem for repair, they will get repaired as classic 
cells. They negatively affect the read path mostly, and use space on disk.

On Tue, Jan 16, 2018 at 2:12 PM Python_Max 
mailto:python@gmail.com>> wrote:
Hello.

I was planning to remove a row (not partition).

Most of the tombstones are seen in the use case of geographic grid with X:Y as 
partition key and object id (timeuuid) as clustering key where objects could be 
temporary with TTL about 10 hours or fully persistent.
When I select all objects in specific X:Y I can even hit 100k (default) limit 
for some X:Y. I have changed this limit to 500k since 99.9p read latency is < 
75ms so I should not (?) care how many tombstones while read latency is fine.

Splitting entities to temporary and permanent and using different compaction 
strategies is an option but it will lead to code duplication and 2x read 
queries.

Is my assumption correct about tombstones are not so big problem as soon as 
read latency and disk usage are okey? Are tombstones affect repair time (using 
reaper)?

Thanks.


On Tue, Jan 16, 2018 at 11:32 AM, Alexander Dejanovski 
mailto:a...@thelastpickle.com>> wrote:
Hi,

could you be more specific about the deletes you're planning to perform ?
This will end up moving your problem somewhere else as you'll be generating new 
tombstones (and if you're planning on deleting rows, be aware that row level 
tombstones aren't reported anywhere in the metrics, logs and query traces).
Currently you can delete your data at the partition level, which will create a 
single tombstone that will shadow all your expired (and non expired) data and 
is very efficient. The read path is optimized for such tombstones and the data 
won't be fully read from disk nor exchanged between replicas. But that's of 
course if your use case allows to delete full partitions.

We usually model so that we can restrict our reads to live data.
If you're creating time series, your clustering key should include a timestamp, 
which you can use to avoid reading expired data. If your TTL is set to 60 days, 
you can read only data that is strictly younger than that.
Then you can partition by time ranges, and access exclusively partitions that 
have no chance to be expired yet.
Those techniques usually work better with TWCS, but the former could make you 
hit a lot of SSTables if your partitions can spread over all time buckets, so 
only use TWCS if you can restrict individual reads to up to 4 time windows.

Cheers,


On Tue, Jan 16, 2018 at 10:01 AM Python_Max 
mailto:python@gmail.com>> wrote:
Hi.

Thank you very much for detailed explanation.
Seems that there is nothing I can do about it except delete records by key 
instead of expiring.


On Fri, Jan 12, 2018 at 7:30 PM, Alexander Dejanovski 
mailto:a...@thelastpickle.com>> wrote:
Hi,

As DuyHai said, different TTLs could theoretically be set for different cells 
of the same row. And one TTLed cell could be shadowing another cell that has no 
TTL (say you forgot to set a TTL and set one afterwards by performing an 
update), or vice versa.
One cell could also be missing from a node without Cassandra knowing. So 
turning an incomplete row that only has expired cells into a tombstone row 
could lead to wrong results being 

RE: nodetool cleanup - compaction remaining time

[Steinmaurer, Thomas]

I have created https://issues.apache.org/jira/browse/CASSANDRA-14701

Please adapt as needed. Thanks!

Thomas

From: Jeff Jirsa 
Sent: Donnerstag, 06. September 2018 07:52
To: cassandra 
Subject: Re: nodetool cleanup - compaction remaining time

Probably worth a JIRA (especially if you can repro in 3.0 or higher, since 2.1 
is critical fixes only)

On Wed, Sep 5, 2018 at 10:46 PM Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

is it a known issue / limitation that cleanup compactions aren’t counted in the 
compaction remaining time?

nodetool compactionstats -H

pending tasks: 1
   compaction type   keyspace   table   completed totalunit   
progress
   CleanupXXX YYY   908.16 GB   1.13 TB   bytes 
78.63%
Active compaction remaining time :   0h00m00s


This is with 2.1.18.


Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313