Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

[Ben Slater]

We’re seeing evidence across our fleet that AWS has rolled something out in
the last 24 hours that has significantly reduce the performance impacts -
back pretty close to pre-patch levels. Yet to see if the impacts come back
with o/s patching on top of the improved hypervisor.

Cheers
Ben



On Thu, 11 Jan 2018 at 05:32 Jon Haddad  wrote:

> For what it’s worth, we (TLP) just posted some results comparing pre and
> post meltdown statistics:
> http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html
>
>
> On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> m4.xlarge do have PCID to my knowledge, but possibly we need a rather new
> kernel 4.14. But I fail to see how this could help anyway, cause this looks
> highly Amazon Hypervisor patch related and we do not have the production
> instances patched at OS/VM level (yet).
>
> Thomas
>
> *From:* Dor Laor [mailto:d...@scylladb.com ]
> *Sent:* Dienstag, 09. Jänner 2018 19:30
> *To:* user@cassandra.apache.org
> *Subject:* Re: Meltdown/Spectre Linux patch - Performance impact on
> Cassandra?
>
> Make sure you pick instances with PCID cpu capability, their TLB overhead
> flush
> overhead is much smaller
>
> On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Quick follow up.
>
> Others in AWS reporting/seeing something similar, e.g.:
> https://twitter.com/BenBromhead/status/950245250504601600
>
> So, while we have seen an relative CPU increase of ~ 50% since Jan 4,
> 2018, we now also have applied a kernel update at OS/VM level on a single
> node (loadtest and not production though), thus more or less double patched
> now. Additional CPU impact by OS/VM level kernel patching is more or less
> negligible, so looks highly Hypervisor related.
>
> Regards,
> Thomas
>
> *From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
> *Sent:* Freitag, 05. Jänner 2018 12:09
> *To:* user@cassandra.apache.org
> *Subject:* Meltdown/Spectre Linux patch - Performance impact on Cassandra?
>
> Hello,
>
> has anybody already some experience/results if a patched Linux kernel
> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>
> In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
> most likely correlating with Amazon finished patching the underlying
> Hypervisor infrastructure …
>
> Anybody else seeing a similar CPU increase?
>
> Thanks,
> Thomas
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>
>
>

-- 


*Ben Slater*

*Chief Product Officer *

   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and 

Re: Too many tombstones using TTL

[Alexander Dejanovski]

Hi,

As DuyHai said, different TTLs could theoretically be set for different
cells of the same row. And one TTLed cell could be shadowing another cell
that has no TTL (say you forgot to set a TTL and set one afterwards by
performing an update), or vice versa.
One cell could also be missing from a node without Cassandra knowing. So
turning an incomplete row that only has expired cells into a tombstone row
could lead to wrong results being returned at read time : the tombstone row
could potentially shadow a valid live cell from another replica.

Cassandra needs to retain each TTLed cell and send it to replicas during
reads to cover all possible cases.


On Fri, Jan 12, 2018 at 5:28 PM Python_Max  wrote:

> Thank you for response.
>
> I know about the option of setting TTL per column or even per item in
> collection. However in my example entire row has expired, shouldn't
> Cassandra be able to detect this situation and spawn a single tombstone for
> entire row instead of many?
> Is there any reason not doing this except that no one needs it? Is this
> suitable for feature request or improvement?
>
> Thanks.
>
> On Wed, Jan 10, 2018 at 4:52 PM, DuyHai Doan  wrote:
>
>> "The question is why Cassandra creates a tombstone for every column
>> instead of single tombstone per row?"
>>
>> --> Simply because technically it is possible to set different TTL value
>> on each column of a CQL row
>>
>> On Wed, Jan 10, 2018 at 2:59 PM, Python_Max  wrote:
>>
>>> Hello, C* users and experts.
>>>
>>> I have (one more) question about tombstones.
>>>
>>> Consider the following example:
>>> cqlsh> create keyspace test_ttl with replication = {'class':
>>> 'SimpleStrategy', 'replication_factor': '1'}; use test_ttl;
>>> cqlsh> create table items(a text, b text, c1 text, c2 text, c3 text,
>>> primary key (a, b));
>>> cqlsh> insert into items(a,b,c1,c2,c3) values('AAA', 'BBB', 'C111',
>>> 'C222', 'C333') using ttl 60;
>>> bash$ nodetool flush
>>> bash$ sleep 60
>>> bash$ nodetool compact test_ttl items
>>> bash$ sstabledump mc-2-big-Data.db
>>>
>>> [
>>>   {
>>> "partition" : {
>>>   "key" : [ "AAA" ],
>>>   "position" : 0
>>> },
>>> "rows" : [
>>>   {
>>> "type" : "row",
>>> "position" : 58,
>>> "clustering" : [ "BBB" ],
>>> "liveness_info" : { "tstamp" : "2018-01-10T13:29:25.777Z", "ttl"
>>> : 60, "expires_at" : "2018-01-10T13:30:25Z", "expired" : true },
>>> "cells" : [
>>>   { "name" : "c1", "deletion_info" : { "local_delete_time" :
>>> "2018-01-10T13:29:25Z" }
>>>   },
>>>   { "name" : "c2", "deletion_info" : { "local_delete_time" :
>>> "2018-01-10T13:29:25Z" }
>>>   },
>>>   { "name" : "c3", "deletion_info" : { "local_delete_time" :
>>> "2018-01-10T13:29:25Z" }
>>>   }
>>> ]
>>>   }
>>> ]
>>>   }
>>> ]
>>>
>>> The question is why Cassandra creates a tombstone for every column
>>> instead of single tombstone per row?
>>>
>>> In production environment I have a table with ~30 columns and It gives
>>> me a warning for 30k tombstones and 300 live rows. It is 30 times more then
>>> it could be.
>>> Can this behavior be tuned in some way?
>>>
>>> Thanks.
>>>
>>> --
>>> Best regards,
>>> Python_Max.
>>>
>>
>>
>
>
> --
> Best regards,
> Python_Max.
>


-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Too many tombstones using TTL

[Python_Max]

Thank you for response.

I know about the option of setting TTL per column or even per item in
collection. However in my example entire row has expired, shouldn't
Cassandra be able to detect this situation and spawn a single tombstone for
entire row instead of many?
Is there any reason not doing this except that no one needs it? Is this
suitable for feature request or improvement?

Thanks.

On Wed, Jan 10, 2018 at 4:52 PM, DuyHai Doan  wrote:

> "The question is why Cassandra creates a tombstone for every column
> instead of single tombstone per row?"
>
> --> Simply because technically it is possible to set different TTL value
> on each column of a CQL row
>
> On Wed, Jan 10, 2018 at 2:59 PM, Python_Max  wrote:
>
>> Hello, C* users and experts.
>>
>> I have (one more) question about tombstones.
>>
>> Consider the following example:
>> cqlsh> create keyspace test_ttl with replication = {'class':
>> 'SimpleStrategy', 'replication_factor': '1'}; use test_ttl;
>> cqlsh> create table items(a text, b text, c1 text, c2 text, c3 text,
>> primary key (a, b));
>> cqlsh> insert into items(a,b,c1,c2,c3) values('AAA', 'BBB', 'C111',
>> 'C222', 'C333') using ttl 60;
>> bash$ nodetool flush
>> bash$ sleep 60
>> bash$ nodetool compact test_ttl items
>> bash$ sstabledump mc-2-big-Data.db
>>
>> [
>>   {
>> "partition" : {
>>   "key" : [ "AAA" ],
>>   "position" : 0
>> },
>> "rows" : [
>>   {
>> "type" : "row",
>> "position" : 58,
>> "clustering" : [ "BBB" ],
>> "liveness_info" : { "tstamp" : "2018-01-10T13:29:25.777Z", "ttl"
>> : 60, "expires_at" : "2018-01-10T13:30:25Z", "expired" : true },
>> "cells" : [
>>   { "name" : "c1", "deletion_info" : { "local_delete_time" :
>> "2018-01-10T13:29:25Z" }
>>   },
>>   { "name" : "c2", "deletion_info" : { "local_delete_time" :
>> "2018-01-10T13:29:25Z" }
>>   },
>>   { "name" : "c3", "deletion_info" : { "local_delete_time" :
>> "2018-01-10T13:29:25Z" }
>>   }
>> ]
>>   }
>> ]
>>   }
>> ]
>>
>> The question is why Cassandra creates a tombstone for every column
>> instead of single tombstone per row?
>>
>> In production environment I have a table with ~30 columns and It gives me
>> a warning for 30k tombstones and 300 live rows. It is 30 times more then it
>> could be.
>> Can this behavior be tuned in some way?
>>
>> Thanks.
>>
>> --
>> Best regards,
>> Python_Max.
>>
>
>


-- 
Best regards,
Python_Max.

Alter composite column

[Joel Samuelsson]

Hi,

I have an older system (C* 2.1) using Thrift tables on which I want to
alter a column composite. Right now it looks like (int, int, string) but I
want it to be (int, int, string, string). Is it possible to do this on a
live cluster without deleting the old data? Can you point me to some
documentation about this? I can't seem to find it any more.

Best regards,
Joel

Re: sstabledump tries to delete a file

[Python_Max]

Done https://issues.apache.org/jira/browse/CASSANDRA-14166

On Wed, Jan 10, 2018 at 8:49 PM, Chris Lohfink  wrote:

> Yes it should be read only, open a jira please. It does look like if the
> fp changed it would rebuild or if your missing. When it builds the table
> metadata from the sstable it can just set the properties to match that of
> the sstable to prevent this.
>
> Chris
>
> On Wed, Jan 10, 2018 at 4:16 AM, Python_Max  wrote:
>
>> Hello all.
>>
>> I have an error when trying to dump SSTable (Cassandra 3.11.1):
>>
>> $ sstabledump mc-56801-big-Data.db
>> Exception in thread "main" FSWriteError in /var/lib/cassandra/data/> ace>//mc-56801-big-Summary.db
>> at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(Fil
>> eUtils.java:142)
>> at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(Fil
>> eUtils.java:159)
>> at org.apache.cassandra.io.sstable.format.SSTableReader.saveSum
>> mary(SSTableReader.java:935)
>> at org.apache.cassandra.io.sstable.format.SSTableReader.saveSum
>> mary(SSTableReader.java:920)
>> at org.apache.cassandra.io.sstable.format.SSTableReader.load(
>> SSTableReader.java:788)
>> at org.apache.cassandra.io.sstable.format.SSTableReader.load(
>> SSTableReader.java:731)
>> at org.apache.cassandra.io.sstable.format.SSTableReader.open(
>> SSTableReader.java:516)
>> at org.apache.cassandra.io.sstable.format.SSTableReader.openNoV
>> alidation(SSTableReader.java:396)
>> at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.
>> java:191)
>> Caused by: java.nio.file.AccessDeniedException:
>> /var/lib/cassandra/data///mc-56801-big-Summary.db
>> at sun.nio.fs.UnixException.translateToIOException(UnixExceptio
>> n.java:84)
>> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.
>> java:102)
>> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.
>> java:107)
>> at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemP
>> rovider.java:244)
>> at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSys
>> temProvider.java:103)
>> at java.nio.file.Files.delete(Files.java:1126)
>> at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(Fil
>> eUtils.java:136)
>> ... 8 more
>>
>> Seems that sstabledump tries to delete and recreate summary file which I
>> think is risky because external modification to files that should be
>> modified only by Cassandra itself can lead to unpredictable behavior.
>> When I copy all related files and change it's owner to myself and run
>> sstabledump in that directory then Summary.db file is recreated but it's
>> md5 is exactly the same as original Summary.db's file.
>>
>> I indeed have changed bloom_filter_fp_chance couple months ago, so I
>> believe that's the reason why SSTableReader wants to recreate summary file.
>>
>> After nodetool scrub an error still happens.
>>
>> I have not found any issues like this in bug tracker.
>> Shouldn't sstabledump be read only?
>>
>> --
>> Best regards,
>> Python_Max.
>>
>
>


-- 
Best regards,
Python_Max.

Re: Even after the drop table, the data actually was not erased.

[Alain RODRIGUEZ]

Hello,

However, the actual size of the data directory did not decrease at all.
> Disk Load monitored by JMX has been decreased.


This sounds like 'auto_snapshot' is enabled. This option will trigger a
snapshot before any table drop / truncate to prevent user mistakes mostly.
Then the data is removed but as it is still referenced by the snapshot
(hard link), space cannot be freed.

Running 'nodetool clearsnapshot' should help reducing the dataset size in
this situation.


The client fails to establish a connection and I see the following
> exceptions in the Cassandra logs.

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
> for cfId…


This does not look like a failed connection to me but rather a try to query
some inexistent data. If that's the data you just deleted (keyspace /
table), this is expected. If not there is an other issue, I hope not
related to the delete in this case...

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2018-01-12 7:14 GMT+00:00 Eunsu Kim :

> hi everyone
>
> On the development server, I dropped all the tables and even keyspace
> dropped to change the table schema.
> Then I created the keyspace and the table.
>
> However, the actual size of the data directory did not decrease at all.
> Disk Load monitored by JMX has been decreased.
>
>
>
> After that, Cassandra does not work normally.
>
> The client fails to establish a connection and I see the following
> exceptions in the Cassandra logs.
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
> for cfId…….org.apache.cassandra.io.FSReadError: java.io.IOException:
> Digest mismatch exception……
>
>
> After the data is forcibly deleted, Cassandra is restarted in a clean
> state and works well.
>
> Can anyone guess why this is happening?
>
> Thank you in advance.
>

Re: Calling StorageService.loadNewSSTables function results in deadlock with compaction background task.

[Desimpel, Ignace]

issue created https://issues.apache.org/jira/browse/CASSANDRA-14164