Repair giving error

2018-01-13 Thread Akshit Jain 
​I have a 10 node C* cluster with 4-5 keyspaces​.
I tried to perform nodetool repair one by one for each keyspace.
For some keyspaces the repair passed but for some it gave this error:
​
I am not able to figure out what is causing this issue.The replica nodes
are up and I am able to ping them from this node.​
​Any suggestions?​

*Error I am getting on incremental repair:*

*[2018-01-10 12:50:14,047] Did not get positive replies from all endpoints.
List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]*









*-- StackTrace --java.lang.RuntimeException: Repair job has failed with the
error message: [2018-01-10 12:50:14,047] Did not get positive replies from
all endpoints. List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]at
org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)*

RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-13 Thread Carlos Rolo 
We are seeing almost no impact on Azure (+1 or 2%). Non-patched OS.

On AWS we don't have any solid data yet.

On 13 Jan 2018 09:46, "Steinmaurer, Thomas" <
thomas.steinmau...@dynatrace.com> wrote:

Hello Ben,



thanks for the notice. Similar here + others reporting as well:
https://blog.appoptics.com/visualizing-meltdown-aws/





Regards,

Thomas



*From:* Ben Slater [mailto:ben.sla...@instaclustr.com]
*Sent:* Freitag, 12. Jänner 2018 23:37

*To:* user@cassandra.apache.org
*Subject:* Re: Meltdown/Spectre Linux patch - Performance impact on
Cassandra?



We’re seeing evidence across our fleet that AWS has rolled something out in
the last 24 hours that has significantly reduce the performance impacts -
back pretty close to pre-patch levels. Yet to see if the impacts come back
with o/s patching on top of the improved hypervisor.



Cheers

Ben







On Thu, 11 Jan 2018 at 05:32 Jon Haddad  wrote:

For what it’s worth, we (TLP) just posted some results comparing pre and
post meltdown statistics: http://thelastpickle.com/blog/2018/
01/10/meltdown-impact-on-latency.html





On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:



m4.xlarge do have PCID to my knowledge, but possibly we need a rather new
kernel 4.14. But I fail to see how this could help anyway, cause this looks
highly Amazon Hypervisor patch related and we do not have the production
instances patched at OS/VM level (yet).



Thomas



*From:* Dor Laor [mailto:d...@scylladb.com ]
*Sent:* Dienstag, 09. Jänner 2018 19:30
*To:* user@cassandra.apache.org
*Subject:* Re: Meltdown/Spectre Linux patch - Performance impact on
Cassandra?



Make sure you pick instances with PCID cpu capability, their TLB overhead
flush

overhead is much smaller



On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

Quick follow up.



Others in AWS reporting/seeing something similar, e.g.:https://twitter.com/
BenBromhead/status/950245250504601600



So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018,
we now also have applied a kernel update at OS/VM level on a single node
(loadtest and not production though), thus more or less double patched now.
Additional CPU impact by OS/VM level kernel patching is more or less
negligible, so looks highly Hypervisor related.



Regards,

Thomas



*From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
*Sent:* Freitag, 05. Jänner 2018 12:09
*To:* user@cassandra.apache.org
*Subject:* Meltdown/Spectre Linux patch - Performance impact on Cassandra?



Hello,



has anybody already some experience/results if a patched Linux kernel
regarding Meltdown/Spectre is affecting performance of Cassandra negatively?



In production, all nodes running in AWS with m4.xlarge, we see up to a 50%
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018,
most likely correlating with Amazon finished patching the underlying
Hypervisor infrastructure …



Anybody else seeing a similar CPU increase?



Thanks,

Thomas



The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or
disclose it to anyone else. If you received it in error please notify us
immediately and then destroy it. Dynatrace Austria GmbH (registration
number FN 91482h) is a company registered in Linz whose registered office
is at 4040 Linz, Austria, Freistädterstraße 313


The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or
disclose it to anyone else. If you received it in error please notify us
immediately and then destroy it. Dynatrace Austria GmbH (registration
number FN 91482h) is a company registered in Linz whose registered office
is at 4040 Linz, Austria, Freistädterstraße 313




The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or
disclose it to anyone else. If you received it in error please notify us
immediately and then destroy it. Dynatrace Austria GmbH (registration
number FN 91482h) is a company registered in Linz whose registered office
is at 4040 Linz, Austria, Freistädterstraße 313






-- 

*Ben Slater*
*Chief Product Officer*
* *

RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-13 Thread Steinmaurer, Thomas 
Hello Ben,

thanks for the notice. Similar here + others reporting as well: 
https://blog.appoptics.com/visualizing-meltdown-aws/


Regards,
Thomas

From: Ben Slater [mailto:ben.sla...@instaclustr.com]
Sent: Freitag, 12. Jänner 2018 23:37
To: user@cassandra.apache.org
Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

We’re seeing evidence across our fleet that AWS has rolled something out in the 
last 24 hours that has significantly reduce the performance impacts - back 
pretty close to pre-patch levels. Yet to see if the impacts come back with o/s 
patching on top of the improved hypervisor.

Cheers
Ben



On Thu, 11 Jan 2018 at 05:32 Jon Haddad 
> wrote:
For what it’s worth, we (TLP) just posted some results comparing pre and post 
meltdown statistics: 
http://thelastpickle.com/blog/2018/01/10/meltdown-impact-on-latency.html



On Jan 10, 2018, at 1:57 AM, Steinmaurer, Thomas 
> 
wrote:

m4.xlarge do have PCID to my knowledge, but possibly we need a rather new 
kernel 4.14. But I fail to see how this could help anyway, cause this looks 
highly Amazon Hypervisor patch related and we do not have the production 
instances patched at OS/VM level (yet).

Thomas

From: Dor Laor [mailto:d...@scylladb.com]
Sent: Dienstag, 09. Jänner 2018 19:30
To: user@cassandra.apache.org
Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Make sure you pick instances with PCID cpu capability, their TLB overhead flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas 
> 
wrote:
Quick follow up.

Others in AWS reporting/seeing something similar, 
e.g.:https://twitter.com/BenBromhead/status/950245250504601600

So, while we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we 
now also have applied a kernel update at OS/VM level on a single node (loadtest 
and not production though), thus more or less double patched now. Additional 
CPU impact by OS/VM level kernel patching is more or less negligible, so looks 
highly Hypervisor related.

Regards,
Thomas

From: Steinmaurer, Thomas 
[mailto:thomas.steinmau...@dynatrace.com]
Sent: Freitag, 05. Jänner 2018 12:09
To: user@cassandra.apache.org
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

Hello,

has anybody already some experience/results if a patched Linux kernel regarding 
Meltdown/Spectre is affecting performance of Cassandra negatively?

In production, all nodes running in AWS with m4.xlarge, we see up to a 50% 
relative (e.g. AVG CPU from 40% => 60%) CPU increase since Jan 4, 2018, most 
likely correlating with Amazon finished patching the underlying Hypervisor 
infrastructure …

Anybody else seeing a similar CPU increase?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 
313



--

Ben Slater
Chief Product Officer
[https://cdn2.hubspot.net/hubfs/2549680/Instaclustr-Navy-logo-new.png]