RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-14 Thread Steinmaurer, Thomas 
Ben,

regarding OS/VM level patching impact. We see almost zero additional impact 
with 4.9.51-10.52.amzn1.x86_64 vs. 4.9.75-25.55.amzn1.x86_64 
(https://alas.aws.amazon.com/ALAS-2018-939.html) on a m5.2xlarge. m5 instance 
type family is rather new and AWS told us to give them a try compared to m4, as 
they are running a different hypervisor technology etc.

Before the additional hypervisor patch on Jan 12, we saw a relative CPU 
improvement of ~ 29% at an advertised ECU improvement of 19%. With the 
additional Jan 12 AWS patching at hypervisor level (?), we see m5 vs. m4 almost 
at the same CPU level. So, m4 on XEN looks pretty good again. Same as before 
again (as you have already mentioned) + with the 2 mentioned kernels above in 
comparison, the situation looks, even with the additional OS/VM level patching. 
Thus, currently we do not see any further action items needed now.

Thomas

From: Ben Slater [mailto:ben.sla...@instaclustr.com]
Sent: Freitag, 12. Jänner 2018 23:37
To: user@cassandra.apache.org
Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

We’re seeing evidence across our fleet that AWS has rolled something out in the 
last 24 hours that has significantly reduce the performance impacts - back 
pretty close to pre-patch levels. Yet to see if the impacts come back with o/s 
patching on top of the improved hypervisor.

Cheers
Ben



On Thu, 11 Jan 2018 at 05:32 Jon Haddad 
<j...@jonhaddad.com<mailto:j...@jonhaddad.com>> wrote:
For what it’s worth, we (TLP) just posted some results comparing pre and post 
meltdown statistics: 
http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html



On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:

m4.xlarge do have PCID to my knowledge, but possibly we need a rather new 
kernel 4.14. But I fail to see how this could help anyway, cause this looks 
highly Amazon Hypervisor patch related and we do not have the production 
instances patched at OS/VM level (yet).

Thomas

From: Dor Laor [mailto:d...@scylladb.com]
Sent: Dienstag, 09. Jänner 2018 19:30
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Make sure you pick instances with PCID cpu capability, their TLB overhead flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Quick follow up.

Others in AWS reporting/seeing something similar, 
e.g.:https://twitter.com/BenBromhead/status/950245250504601600

So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we 
now also have applied a kernel update at OS/VM level on a single node (loadtest 
and not production though), thus more or less double patched now. Additional 
CPU impact by OS/VM level kernel patching is more or less negligible, so looks 
highly Hypervisor related.

Regards,
Thomas

From: Steinmaurer, Thomas 
[mailto:thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>]
Sent: Freitag, 05. Jänner 2018 12:09
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Hello,

has anybody already some experience/results if a patched Linux kernel regarding 
Meltdown/Spectre is affecting performance of Cassandra negatively?

In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
likely correlating with Amazon finished patching the underlying Hypervisor 
infrastructure …

Anybody else seeing a similar CPU increase?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+31

RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-13 Thread Carlos Rolo 
We are seeing almost no impact on Azure (+1 or 2%). Non-patched OS.

On AWS we don't have any solid data yet.

On 13 Jan 2018 09:46, "Steinmaurer, Thomas" <
thomas.steinmau...@dynatrace.com> wrote:

Hello Ben,



thanks for the notice. Similar here + others reporting as well:
https://blog.appoptics.com/visualizing-meltdown-aws/





Regards,

Thomas



*From:* Ben Slater [mailto:ben.sla...@instaclustr.com]
*Sent:* Freitag, 12. Jänner 2018 23:37

*To:* user@cassandra.apache.org
*Subject:* Re: Meltdown/Spectre Linux patch - Performance impact on
Cassandra?



We’re seeing evidence across our fleet that AWS has rolled something out in
the last 24 hours that has significantly reduce the performance impacts -
back pretty close to pre-patch levels. Yet to see if the impacts come back
with o/s patching on top of the improved hypervisor.



Cheers

Ben







On Thu, 11 Jan 2018 at 05:32 Jon Haddad <j...@jonhaddad.com> wrote:

For what it’s worth, we (TLP) just posted some results comparing pre and
post meltdown statistics: http://thelastpickle.com/blog/2018/
01/10/meltdown-impact-on-latency.html





On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:



m4.xlarge do have PCID to my knowledge, but possibly we need a rather new
kernel 4.14. But I fail to see how this could help anyway, cause this looks
highly Amazon Hypervisor patch related and we do not have the production
instances patched at OS/VM level (yet).



Thomas



*From:* Dor Laor [mailto:d...@scylladb.com <d...@scylladb.com>]
*Sent:* Dienstag, 09. Jänner 2018 19:30
*To:* user@cassandra.apache.org
*Subject:* Re: Meltdown/Spectre Linux patch - Performance impact on
Cassandra?



Make sure you pick instances with PCID cpu capability, their TLB overhead
flush

overhead is much smaller



On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

Quick follow up.



Others in AWS reporting/seeing something similar, e.g.:https://twitter.com/
BenBromhead/status/950245250504601600



So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018,
we now also have applied a kernel update at OS/VM level on a single node
(loadtest and not production though), thus more or less double patched now.
Additional CPU impact by OS/VM level kernel patching is more or less
negligible, so looks highly Hypervisor related.



Regards,

Thomas



*From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
*Sent:* Freitag, 05. Jänner 2018 12:09
*To:* user@cassandra.apache.org
*Subject:* Meltdown/Spectre Linux patch - Performance impact on Cassandra?



Hello,



has anybody already some experience/results if a patched Linux kernel
regarding Meltdown/Spectre is affecting performance of Cassandra negatively?



In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
most likely correlating with Amazon finished patching the underlying
Hypervisor infrastructure …



Anybody else seeing a similar CPU increase?



Thanks,

Thomas



The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or
disclose it to anyone else. If you received it in error please notify us
immediately and then destroy it. Dynatrace Austria GmbH (registration
number FN 91482h) is a company registered in Linz whose registered office
is at 4040 Linz, Austria, Freistädterstraße 313
<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>

The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or
disclose it to anyone else. If you received it in error please notify us
immediately and then destroy it. Dynatrace Austria GmbH (registration
number FN 91482h) is a company registered in Linz whose registered office
is at 4040 Linz, Austria, Freistädterstraße 313
<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>



The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or
disclose it to anyone else. If you received it in error please notify us
immediately and then destroy it. Dynatrace Austria GmbH (registration
number FN 91482h) is a company registered in Linz whose registered office
is at 4040 Linz, Austria, Freistädterstraße 313
<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>





-- 

*Ben Slater*
*Chief Product Officer*
* <https://www.instaclustr.com/>*

<https://www.facebook.com/instaclustr&

RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-13 Thread Steinmaurer, Thomas 
Hello Ben,

thanks for the notice. Similar here + others reporting as well: 
https://blog.appoptics.com/visualizing-meltdown-aws/


Regards,
Thomas

From: Ben Slater [mailto:ben.sla...@instaclustr.com]
Sent: Freitag, 12. Jänner 2018 23:37
To: user@cassandra.apache.org
Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

We’re seeing evidence across our fleet that AWS has rolled something out in the 
last 24 hours that has significantly reduce the performance impacts - back 
pretty close to pre-patch levels. Yet to see if the impacts come back with o/s 
patching on top of the improved hypervisor.

Cheers
Ben



On Thu, 11 Jan 2018 at 05:32 Jon Haddad 
<j...@jonhaddad.com<mailto:j...@jonhaddad.com>> wrote:
For what it’s worth, we (TLP) just posted some results comparing pre and post 
meltdown statistics: 
http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html



On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:

m4.xlarge do have PCID to my knowledge, but possibly we need a rather new 
kernel 4.14. But I fail to see how this could help anyway, cause this looks 
highly Amazon Hypervisor patch related and we do not have the production 
instances patched at OS/VM level (yet).

Thomas

From: Dor Laor [mailto:d...@scylladb.com]
Sent: Dienstag, 09. Jänner 2018 19:30
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Make sure you pick instances with PCID cpu capability, their TLB overhead flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Quick follow up.

Others in AWS reporting/seeing something similar, 
e.g.:https://twitter.com/BenBromhead/status/950245250504601600

So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we 
now also have applied a kernel update at OS/VM level on a single node (loadtest 
and not production though), thus more or less double patched now. Additional 
CPU impact by OS/VM level kernel patching is more or less negligible, so looks 
highly Hypervisor related.

Regards,
Thomas

From: Steinmaurer, Thomas 
[mailto:thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>]
Sent: Freitag, 05. Jänner 2018 12:09
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Hello,

has anybody already some experience/results if a patched Linux kernel regarding 
Meltdown/Spectre is affecting performance of Cassandra negatively?

In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
likely correlating with Amazon finished patching the underlying Hypervisor 
infrastructure …

Anybody else seeing a similar CPU increase?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>



--

Ben Slater
Chief Product Officer
[https://cdn2.hubspot.net/hubfs/2549680/Instaclustr-Navy-logo-new.png]<https://www.instaclustr.com/>

[http

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-12 Thread Ben Slater 
We’re seeing evidence across our fleet that AWS has rolled something out in
the last 24 hours that has significantly reduce the performance impacts -
back pretty close to pre-patch levels. Yet to see if the impacts come back
with o/s patching on top of the improved hypervisor.

Cheers
Ben



On Thu, 11 Jan 2018 at 05:32 Jon Haddad <j...@jonhaddad.com> wrote:

> For what it’s worth, we (TLP) just posted some results comparing pre and
> post meltdown statistics:
> http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html
>
>
> On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> m4.xlarge do have PCID to my knowledge, but possibly we need a rather new
> kernel 4.14. But I fail to see how this could help anyway, cause this looks
> highly Amazon Hypervisor patch related and we do not have the production
> instances patched at OS/VM level (yet).
>
> Thomas
>
> *From:* Dor Laor [mailto:d...@scylladb.com <d...@scylladb.com>]
> *Sent:* Dienstag, 09. Jänner 2018 19:30
> *To:* user@cassandra.apache.org
> *Subject:* Re: Meltdown/Spectre Linux patch - Performance impact on
> Cassandra?
>
> Make sure you pick instances with PCID cpu capability, their TLB overhead
> flush
> overhead is much smaller
>
> On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Quick follow up.
>
> Others in AWS reporting/seeing something similar, e.g.:
> https://twitter.com/BenBromhead/status/950245250504601600
>
> So, while we have seen an relative CPU increase of ~ 50% since Jan 4,
> 2018, we now also have applied a kernel update at OS/VM level on a single
> node (loadtest and not production though), thus more or less double patched
> now. Additional CPU impact by OS/VM level kernel patching is more or less
> negligible, so looks highly Hypervisor related.
>
> Regards,
> Thomas
>
> *From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
> *Sent:* Freitag, 05. Jänner 2018 12:09
> *To:* user@cassandra.apache.org
> *Subject:* Meltdown/Spectre Linux patch - Performance impact on Cassandra?
>
> Hello,
>
> has anybody already some experience/results if a patched Linux kernel
> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>
> In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
> most likely correlating with Amazon finished patching the underlying
> Hypervisor infrastructure …
>
> Anybody else seeing a similar CPU increase?
>
> Thanks,
> Thomas
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
>
>
>

-- 


*Ben Slater*

*Chief Product Officer <https://www.instaclustr.com/>*

<https://www.facebook.com/instaclustr>   <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>

Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>.

This

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-10 Thread Jon Haddad 
For what it’s worth, we (TLP) just posted some results comparing pre and post 
meltdown statistics: 
http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html 
<http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html>

> On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas 
> <thomas.steinmau...@dynatrace.com> wrote:
> 
> m4.xlarge do have PCID to my knowledge, but possibly we need a rather new 
> kernel 4.14. But I fail to see how this could help anyway, cause this looks 
> highly Amazon Hypervisor patch related and we do not have the production 
> instances patched at OS/VM level (yet). <>
>  
> Thomas
>  
> From: Dor Laor [mailto:d...@scylladb.com] 
> Sent: Dienstag, 09. Jänner 2018 19:30
> To: user@cassandra.apache.org
> Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?
>  
> Make sure you pick instances with PCID cpu capability, their TLB overhead 
> flush
> overhead is much smaller
>  
> On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas 
> <thomas.steinmau...@dynatrace.com <mailto:thomas.steinmau...@dynatrace.com>> 
> wrote:
> Quick follow up. <>
>  
> Others in AWS reporting/seeing something similar, 
> e.g.:https://twitter.com/BenBromhead/status/950245250504601600 
> <https://twitter.com/BenBromhead/status/950245250504601600>
>  
> So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, 
> we now also have applied a kernel update at OS/VM level on a single node 
> (loadtest and not production though), thus more or less double patched now. 
> Additional CPU impact by OS/VM level kernel patching is more or less 
> negligible, so looks highly Hypervisor related.
>  
> Regards,
> Thomas
>  
> From: Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com 
> <mailto:thomas.steinmau...@dynatrace.com>] 
> Sent: Freitag, 05. Jänner 2018 12:09
> To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra?
>  
> Hello,
>  
> has anybody already some experience/results if a patched Linux kernel 
> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>  
> In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
> likely correlating with Amazon finished patching the underlying Hypervisor 
> infrastructure …
>  
> Anybody else seeing a similar CPU increase?
>  
> Thanks,
> Thomas
>  
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313 
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313 
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
>  
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313

RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-10 Thread Steinmaurer, Thomas 
m4.xlarge do have PCID to my knowledge, but possibly we need a rather new 
kernel 4.14. But I fail to see how this could help anyway, cause this looks 
highly Amazon Hypervisor patch related and we do not have the production 
instances patched at OS/VM level (yet).

Thomas

From: Dor Laor [mailto:d...@scylladb.com]
Sent: Dienstag, 09. Jänner 2018 19:30
To: user@cassandra.apache.org
Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Make sure you pick instances with PCID cpu capability, their TLB overhead flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Quick follow up.

Others in AWS reporting/seeing something similar, e.g.: 
https://twitter.com/BenBromhead/status/950245250504601600

So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we 
now also have applied a kernel update at OS/VM level on a single node (loadtest 
and not production though), thus more or less double patched now. Additional 
CPU impact by OS/VM level kernel patching is more or less negligible, so looks 
highly Hypervisor related.

Regards,
Thomas

From: Steinmaurer, Thomas 
[mailto:thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>]
Sent: Freitag, 05. Jänner 2018 12:09
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Hello,

has anybody already some experience/results if a patched Linux kernel regarding 
Meltdown/Spectre is affecting performance of Cassandra negatively?

In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
likely correlating with Amazon finished patching the underlying Hypervisor 
infrastructure …

Anybody else seeing a similar CPU increase?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313=gmail=g>

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-09 Thread Jeff Jirsa 
Longer than that. Years. Check /proc/cpuinfo


-- 
Jeff Jirsa


> On Jan 9, 2018, at 11:19 PM, daemeon reiydelle  wrote:
> 
> Good luck with that. Pcid out since mid 2017 as I recall? 
> 
> 
> Daemeon (Dæmœn) Reiydelle
> USA 1.415.501.0198
> 
> On Jan 9, 2018 10:31 AM, "Dor Laor"  wrote:
> Make sure you pick instances with PCID cpu capability, their TLB overhead 
> flush
> overhead is much smaller
> 
>> On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas 
>>  wrote:
>> Quick follow up.
>> 
>>  
>> 
>> Others in AWS reporting/seeing something similar, e.g.: 
>> https://twitter.com/BenBromhead/status/950245250504601600
>> 
>>  
>> 
>> So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, 
>> we now also have applied a kernel update at OS/VM level on a single node 
>> (loadtest and not production though), thus more or less double patched now. 
>> Additional CPU impact by OS/VM level kernel patching is more or less 
>> negligible, so looks highly Hypervisor related.
>> 
>>  
>> 
>> Regards,
>> 
>> Thomas
>> 
>>  
>> 
>> From: Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com] 
>> Sent: Freitag, 05. Jänner 2018 12:09
>> To: user@cassandra.apache.org
>> Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra?
>> 
>>  
>> 
>> Hello,
>> 
>>  
>> 
>> has anybody already some experience/results if a patched Linux kernel 
>> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>> 
>>  
>> 
>> In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
>> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
>> likely correlating with Amazon finished patching the underlying Hypervisor 
>> infrastructure …
>> 
>>  
>> 
>> Anybody else seeing a similar CPU increase?
>> 
>>  
>> 
>> Thanks,
>> 
>> Thomas
>> 
>>  
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313
> 
>

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-09 Thread Dor Laor 
Hard to tell from the first 10 google search results which Intel CPUs
has it so I went to ask my /proc/cpuinfo, turns out my >1 year Dell XPS
laptop has it. AWS's i3 has it too.

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb
rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor
ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic
movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm
3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi
flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida
arat pln pts hwp hwp_notify hwp_act_window hwp_epp


On Tue, Jan 9, 2018 at 11:19 PM, daemeon reiydelle 
wrote:

> Good luck with that. Pcid out since mid 2017 as I recall?
>
>
> Daemeon (Dæmœn) Reiydelle
> USA 1.415.501.0198 <(415)%20501-0198>
>
> On Jan 9, 2018 10:31 AM, "Dor Laor"  wrote:
>
> Make sure you pick instances with PCID cpu capability, their TLB overhead
> flush
> overhead is much smaller
>
> On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
>> Quick follow up.
>>
>>
>>
>> Others in AWS reporting/seeing something similar, e.g.:
>> https://twitter.com/BenBromhead/status/950245250504601600
>>
>>
>>
>> So, while we have seen an relative CPU increase of ~ 50% since Jan 4,
>> 2018, we now also have applied a kernel update at OS/VM level on a single
>> node (loadtest and not production though), thus more or less double patched
>> now. Additional CPU impact by OS/VM level kernel patching is more or less 
>> negligible,
>> so looks highly Hypervisor related.
>>
>>
>>
>> Regards,
>>
>> Thomas
>>
>>
>>
>> *From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
>> *Sent:* Freitag, 05. Jänner 2018 12:09
>> *To:* user@cassandra.apache.org
>> *Subject:* Meltdown/Spectre Linux patch - Performance impact on
>> Cassandra?
>>
>>
>>
>> Hello,
>>
>>
>>
>> has anybody already some experience/results if a patched Linux kernel
>> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>>
>>
>>
>> In production, all nodes running in AWS with m4.xlarge, we see up to a
>> 50% relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
>> most likely correlating with Amazon finished patching the underlying
>> Hypervisor infrastructure …
>>
>>
>>
>> Anybody else seeing a similar CPU increase?
>>
>>
>>
>> Thanks,
>>
>> Thomas
>>
>>
>>
>> The contents of this e-mail are intended for the named addressee only. It
>> contains information that may be confidential. Unless you are the named
>> addressee or an authorized designee, you may not copy or use it, or
>> disclose it to anyone else. If you received it in error please notify us
>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>> number FN 91482h) is a company registered in Linz whose registered office
>> is at 4040 Linz, Austria, Freistädterstraße 313
>> 
>> The contents of this e-mail are intended for the named addressee only. It
>> contains information that may be confidential. Unless you are the named
>> addressee or an authorized designee, you may not copy or use it, or
>> disclose it to anyone else. If you received it in error please notify us
>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>> number FN 91482h) is a company registered in Linz whose registered office
>> is at 4040 Linz, Austria, Freistädterstraße 313
>> 
>>
>
>
>

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-09 Thread daemeon reiydelle 
Good luck with that. Pcid out since mid 2017 as I recall?


Daemeon (Dæmœn) Reiydelle
USA 1.415.501.0198

On Jan 9, 2018 10:31 AM, "Dor Laor"  wrote:

Make sure you pick instances with PCID cpu capability, their TLB overhead
flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Quick follow up.
>
>
>
> Others in AWS reporting/seeing something similar, e.g.:
> https://twitter.com/BenBromhead/status/950245250504601600
>
>
>
> So, while we have seen an relative CPU increase of ~ 50% since Jan 4,
> 2018, we now also have applied a kernel update at OS/VM level on a single
> node (loadtest and not production though), thus more or less double patched
> now. Additional CPU impact by OS/VM level kernel patching is more or less 
> negligible,
> so looks highly Hypervisor related.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
> *Sent:* Freitag, 05. Jänner 2018 12:09
> *To:* user@cassandra.apache.org
> *Subject:* Meltdown/Spectre Linux patch - Performance impact on Cassandra?
>
>
>
> Hello,
>
>
>
> has anybody already some experience/results if a patched Linux kernel
> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>
>
>
> In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
> most likely correlating with Amazon finished patching the underlying
> Hypervisor infrastructure …
>
>
>
> Anybody else seeing a similar CPU increase?
>
>
>
> Thanks,
>
> Thomas
>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-09 Thread Tony Anecito 
Hi All,Has anyone seen any test results for SQL Server? Although I am a 
Cassandra user I do use SQL Server for other companies.
Thanks,-Tony

  From: Dor Laor <d...@scylladb.com>
 To: user@cassandra.apache.org 
 Sent: Tuesday, January 9, 2018 10:31 AM
 Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?
   
Make sure you pick instances with PCID cpu capability, their TLB overhead 
flushoverhead is much smaller
On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas 
<thomas.steinmau...@dynatrace.com> wrote:

Quick follow up. Others in AWS reporting/seeing something similar, 
e.g.:https://twitter.com/ BenBromhead/status/ 950245250504601600 So, while we 
have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we now also have 
applied a kernel update at OS/VM level on a single node (loadtest and not 
production though), thus more or less double patched now. Additional CPU impact 
by OS/VM level kernel patching is more or lessnegligible, so looks highly 
Hypervisor related. Regards,Thomas From: Steinmaurer, Thomas 
[mailto:thomas.steinmaurer@ dynatrace.com]
Sent: Freitag, 05. Jänner 2018 12:09
To: user@cassandra.apache.org
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra? Hello, 
has anybody already some experience/results if a patched Linux kernel regarding 
Meltdown/Spectre is affecting performance of Cassandra negatively? In 
production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
likely correlating with Amazon finished patching the underlying Hypervisor 
infrastructure … Anybody else seeing a similar CPU increase? Thanks,Thomas The 
contents of this e-mail are intended for the named addressee only. It contains 
information that may be confidential. Unless you are the named addressee or an 
authorized designee, you may not copy or use it, or disclose it to anyone else. 
If you received it in error please notify us immediately and then destroy it. 
Dynatrace Austria GmbH (registration number FN 91482h) is a company registered 
in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 
313The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-09 Thread Dor Laor 
Make sure you pick instances with PCID cpu capability, their TLB overhead
flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Quick follow up.
>
>
>
> Others in AWS reporting/seeing something similar, e.g.:
> https://twitter.com/BenBromhead/status/950245250504601600
>
>
>
> So, while we have seen an relative CPU increase of ~ 50% since Jan 4,
> 2018, we now also have applied a kernel update at OS/VM level on a single
> node (loadtest and not production though), thus more or less double patched
> now. Additional CPU impact by OS/VM level kernel patching is more or less 
> negligible,
> so looks highly Hypervisor related.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
> *Sent:* Freitag, 05. Jänner 2018 12:09
> *To:* user@cassandra.apache.org
> *Subject:* Meltdown/Spectre Linux patch - Performance impact on Cassandra?
>
>
>
> Hello,
>
>
>
> has anybody already some experience/results if a patched Linux kernel
> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>
>
>
> In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
> most likely correlating with Amazon finished patching the underlying
> Hypervisor infrastructure …
>
>
>
> Anybody else seeing a similar CPU increase?
>
>
>
> Thanks,
>
> Thomas
>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>

RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-09 Thread Steinmaurer, Thomas 
Quick follow up.

Others in AWS reporting/seeing something similar, e.g.: 
https://twitter.com/BenBromhead/status/950245250504601600

So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we 
now also have applied a kernel update at OS/VM level on a single node (loadtest 
and not production though), thus more or less double patched now. Additional 
CPU impact by OS/VM level kernel patching is more or less negligible, so looks 
highly Hypervisor related.

Regards,
Thomas

From: Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
Sent: Freitag, 05. Jänner 2018 12:09
To: user@cassandra.apache.org
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Hello,

has anybody already some experience/results if a patched Linux kernel regarding 
Meltdown/Spectre is affecting performance of Cassandra negatively?

In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
likely correlating with Amazon finished patching the underlying Hypervisor 
infrastructure ...

Anybody else seeing a similar CPU increase?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-06 Thread Erick Ramirez 
Thanks for the insight, Romain, and providing those numbers. Looking
forward to others posting their stats here.

We are running up some tests and will share when available. Cheers!

On Sat, Jan 6, 2018 at 3:44 AM, Romain Hardouin  wrote:

> Hi,
>
> We also noticed an increase of CPU - both system and user - on our
> c3.4xlarge fleet.
> So far it's really visible with max(%user) and especially max(%system), it
> has doubled!
> I graphed a ratio "write/s / %system", it's interesting to see how the
> value dropped yesterday, you can see it here: https://ibb.co/dnVcHG
>
> For reference: https://aws.amazon.com/fr/security/security-bulletins/
> AWS-2018-013/
>
> Best,
>
> Romain
>
>
> Le vendredi 5 janvier 2018 à 13:09:35 UTC+1, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> a écrit :
>
>
> Hello,
>
>
>
> has anybody already some experience/results if a patched Linux kernel
> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>
>
>
> In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
> most likely correlating with Amazon finished patching the underlying
> Hypervisor infrastructure …
>
>
>
> Anybody else seeing a similar CPU increase?
>
>
>
> Thanks,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-05 Thread Romain Hardouin 
  Hi,
We also noticed an increase of CPU - both system and user - on our c3.4xlarge 
fleet. So far it's really visible with max(%user) and especially max(%system), 
it has doubled!I graphed a ratio "write/s / %system", it's interesting to see 
how the value dropped yesterday, you can see it here: https://ibb.co/dnVcHG
For reference: 
https://aws.amazon.com/fr/security/security-bulletins/AWS-2018-013/
Best,
Romain

Le vendredi 5 janvier 2018 à 13:09:35 UTC+1, Steinmaurer, Thomas 
 a écrit :  
 
  
Hello,
 
  
 
has anybody already some experience/results if a patched Linux kernel regarding 
Meltdown/Spectre is affecting performance of Cassandra negatively?
 
  
 
In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
likely correlating with Amazon finished patching the underlying Hypervisor 
infrastructure …
 
  
 
Anybody else seeing a similar CPU increase?
 
  
 
Thanks,
 
Thomas
 
  
 The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-05 Thread Tom van der Woerdt 
Hi Thomas,

No clue about AWS, and it is of course highly dependent on hardware, but on
CentOS 7 on bare metal, the patched kernel
(kernel-3.10.0-693.11.6.el7.x86_64) seems to have a roughly 50% CPU
increase compared to an unpatched kernel
(kernel-3.10.0-693.11.1.el7.x86_64). On a happier note, the latest mainline
kernel from elrepo (kernel-ml-4.14.11-1.el7.elrepo.x86_64) seems to recover
the entire performance loss, likely due to recent PCID patches (or the
other 3+ years of kernel development).

That's on lab servers though, the numbers here will likely vary a lot based
on test setup, and may not be reproducible for production workloads.

If you have the infrastructure to test a variety of kernels, I'd be very
interested to see your numbers.

Thanks,

Tom van der Woerdt
Site Reliability Engineer

Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
Direct +31207153426
[image: Booking.com] 
The world's #1 accommodation site
43 languages, 198+ offices worldwide, 120,000+ global destinations,
1,550,000+ room nights booked every day
No booking fees, best price always guaranteed
Subsidiary of the Priceline Group (NASDAQ: PCLN)

On Fri, Jan 5, 2018 at 12:09 PM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> has anybody already some experience/results if a patched Linux kernel
> regarding Meltdown/Spectre is affecting performance of Cassandra negatively?
>
>
>
> In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
> relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
> most likely correlating with Amazon finished patching the underlying
> Hypervisor infrastructure …
>
>
>
> Anybody else seeing a similar CPU increase?
>
>
>
> Thanks,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> 
> ädterstra
> 
> ße 313
> 
>