Ben, regarding OS/VM level patching impact. We see almost zero additional impact with 4.9.51-10.52.amzn1.x86_64 vs. 4.9.75-25.55.amzn1.x86_64 (https://alas.aws.amazon.com/ALAS-2018-939.html) on a m5.2xlarge. m5 instance type family is rather new and AWS told us to give them a try compared to m4, as they are running a different hypervisor technology etc.
Before the additional hypervisor patch on Jan 12, we saw a relative CPU improvement of ~ 29% at an advertised ECU improvement of 19%. With the additional Jan 12 AWS patching at hypervisor level (?), we see m5 vs. m4 almost at the same CPU level. So, m4 on XEN looks pretty good again. Same as before again (as you have already mentioned) + with the 2 mentioned kernels above in comparison, the situation looks, even with the additional OS/VM level patching. Thus, currently we do not see any further action items needed now. Thomas From: Ben Slater [mailto:ben.sla...@instaclustr.com] Sent: Freitag, 12. Jänner 2018 23:37 To: user@cassandra.apache.org Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra? We’re seeing evidence across our fleet that AWS has rolled something out in the last 24 hours that has significantly reduce the performance impacts - back pretty close to pre-patch levels. Yet to see if the impacts come back with o/s patching on top of the improved hypervisor. Cheers Ben On Thu, 11 Jan 2018 at 05:32 Jon Haddad <j...@jonhaddad.com<mailto:j...@jonhaddad.com>> wrote: For what it’s worth, we (TLP) just posted some results comparing pre and post meltdown statistics: http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas <thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> wrote: m4.xlarge do have PCID to my knowledge, but possibly we need a rather new kernel 4.14. But I fail to see how this could help anyway, cause this looks highly Amazon Hypervisor patch related and we do not have the production instances patched at OS/VM level (yet). Thomas From: Dor Laor [mailto:d...@scylladb.com] Sent: Dienstag, 09. Jänner 2018 19:30 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra? Make sure you pick instances with PCID cpu capability, their TLB overhead flush overhead is much smaller On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> wrote: Quick follow up. Others in AWS reporting/seeing something similar, e.g.:https://twitter.com/BenBromhead/status/950245250504601600 So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we now also have applied a kernel update at OS/VM level on a single node (loadtest and not production though), thus more or less double patched now. Additional CPU impact by OS/VM level kernel patching is more or less negligible, so looks highly Hypervisor related. Regards, Thomas From: Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>] Sent: Freitag, 05. Jänner 2018 12:09 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra? Hello, has anybody already some experience/results if a patched Linux kernel regarding Meltdown/Spectre is affecting performance of Cassandra negatively? In production, all nodes running in AWS with m4.xlarge, we see up to a 50% relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most likely correlating with Amazon finished patching the underlying Hypervisor infrastructure … Anybody else seeing a similar CPU increase? Thanks, Thomas The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> -- Ben Slater Chief Product Officer [https://cdn2.hubspot.net/hubfs/2549680/Instaclustr-Navy-logo-new.png]<https://www.instaclustr.com/> [http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/facebook_sig.png]<https://www.facebook.com/instaclustr> [http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/twitter_sig.png] <https://twitter.com/instaclustr> [http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/linkedin_sig.png] <https://www.linkedin.com/company/instaclustr> Read our latest technical blog posts here<https://www.instaclustr.com/blog/>. This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message. The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
