Re: replace dead node vs remove node

[Anthony Grasso]

Hi Peng,

Correct, you would want to repair in either case.

Regards,
Anthony


On Fri, 23 Mar 2018 at 14:09, Peng Xiao <2535...@qq.com> wrote:

> Hi Anthony,
>
> there is a problem with replacing dead node as per the blog,if the
> replacement process takes longer than max_hint_window_in_ms,we must run
> repair to make the replaced node consistent again, since it missed ongoing
> writes during bootstrapping.but for a great cluster,repair is a painful
> process.
>
> Thanks,
> Peng Xiao
>
>
>
> -- 原始邮件 --
> *发件人:* "Anthony Grasso";
> *发送时间:* 2018年3月22日(星期四) 晚上7:13
> *收件人:* "user";
> *主题:* Re: replace dead node vs remove node
>
> Hi Peng,
>
> Depending on the hardware failure you can do one of two things:
>
> 1. If the disks are intact and uncorrupted you could just use the disks
> with the current data on them in the new node. Even if the IP address
> changes for the new node that is fine. In that case all you need to do is
> run repair on the new node. The repair will fix any writes the node missed
> while it was down. This process is similar to the scenario in this blog
> post:
> http://thelastpickle.com/blog/2018/02/21/replace-node-without-bootstrapping.html
>
> 2. If the disks are inaccessible or corrupted, then use the method as
> described in the blogpost you linked to. The operation is similar to
> bootstrapping a new node. There is no need to perform any other remove or
> join operation on the failed or new nodes. As per the blog post, you
> definitely want to run repair on the new node as soon as it joins the
> cluster. In this case here, the data on the failed node is effectively lost
> and replaced with data from other nodes in the cluster.
>
> Hope this helps.
>
> Regards,
> Anthony
>
>
> On Thu, 22 Mar 2018 at 20:52, Peng Xiao <2535...@qq.com> wrote:
>
>> Dear All,
>>
>> when one node failure with hardware errors,it will be in DN status in the
>> cluster.Then if we are not able to handle this error in three hours(max
>> hints window),we will loss data,right?we have to run repair to keep the
>> consistency.
>> And as per
>> https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html,we
>> can replace this dead node,is it the same as bootstrap new node?that means
>> we don't need to remove node and rejoin?
>> Could anyone please advise?
>>
>> Thanks,
>> Peng Xiao
>>
>>
>>
>>
>>

Re: Cassandra 2.1 on Xenial

[Michael Shuler]

Looked at your error again. The cassandra-stress example appears as if
thrift is disabled, which is the default stress profile. Try
`cassandra-stress write -mode native cql3 ...` for native cql stress runs.

Michael

On 03/22/2018 11:36 PM, Michael Shuler wrote:
> dh_python is a build dependency. The runtime dependency is python.
> 
> https://github.com/apache/cassandra/blob/cassandra-2.1/debian/control#L6
> https://github.com/apache/cassandra/blob/cassandra-2.1/debian/control#L14
> 
> Just upgrading to the latest 2.1.x should fix all these issues your
> having. :)
> 
> Michael
> 
> On 03/22/2018 09:59 PM, Cyril Scetbon wrote:
>> Yeah, I also had to grab a new version of the cassandra-driver which was
>> fixed in 2.1.16 https://issues.apache.org/jira/browse/CASSANDRA-11850
>> otherwise cqlsh would not work with python 2.7.12.
>>
>> I’m surprised dh-python is not a requirement on the Cassandra package in
>> your debian/control 
>>
>> I also noticed that I get issues with cassandra-stress on
>> Xenial https://pastebin.com/raw/r2umZXSV, Any idea ? 
>>
>> Bests
>> —
>> Cyril Scetbon
>>
>>> On Mar 19, 2018, at 12:16 PM, Michael Shuler >> > wrote:
>>>
>>> Yep. This was fixed for 2.1.15, 2.2.7, 3.0.6, 3.6 in
>>> https://issues.apache.org/jira/browse/CASSANDRA-10853
>>>
>>> -- 
>>> Kind regards,
>>> Michael
>>>
>>> On 03/18/2018 10:55 PM, Cyril Scetbon wrote:
 Okay I found that when building the package, the new helper dh_python2
 must be used to fix that issue.

 —
 Cyril Scetbon

> On Mar 18, 2018, at 11:08 AM, Cyril Scetbon  
> > wrote:
>
> Hey guys,
>
> Having to still use Cassandra 2.1, I have installed it on Ubuntu
> Xenial and I have an issue with cqlsh. I was able to fix it by
> installing python-support and a fix from 2.1.16. However I’d like to
> know if there is a way to do it without installing an old package
> (python-support) on Xenial. dh-python is supposed to have replaced
> python-support, however cqlsh complains when it’s not installed : 
>
> Traceback (most recent call last):
>   File "/usr/bin/cqlsh", line 121, in 
>     from cqlshlib import cql3handling, cqlhandling, pylexotron,
> sslhandling
> ImportError: No module named cqlshlib
>
> Is there a better way than installing that old package ?
>
> Thanks 
> —
> Cyril Scetbon
>

>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> 
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
>>
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra 2.1 on Xenial

[Michael Shuler]

dh_python is a build dependency. The runtime dependency is python.

https://github.com/apache/cassandra/blob/cassandra-2.1/debian/control#L6
https://github.com/apache/cassandra/blob/cassandra-2.1/debian/control#L14

Just upgrading to the latest 2.1.x should fix all these issues your
having. :)

Michael

On 03/22/2018 09:59 PM, Cyril Scetbon wrote:
> Yeah, I also had to grab a new version of the cassandra-driver which was
> fixed in 2.1.16 https://issues.apache.org/jira/browse/CASSANDRA-11850
> otherwise cqlsh would not work with python 2.7.12.
> 
> I’m surprised dh-python is not a requirement on the Cassandra package in
> your debian/control 
> 
> I also noticed that I get issues with cassandra-stress on
> Xenial https://pastebin.com/raw/r2umZXSV, Any idea ? 
> 
> Bests
> —
> Cyril Scetbon
> 
>> On Mar 19, 2018, at 12:16 PM, Michael Shuler > > wrote:
>>
>> Yep. This was fixed for 2.1.15, 2.2.7, 3.0.6, 3.6 in
>> https://issues.apache.org/jira/browse/CASSANDRA-10853
>>
>> -- 
>> Kind regards,
>> Michael
>>
>> On 03/18/2018 10:55 PM, Cyril Scetbon wrote:
>>> Okay I found that when building the package, the new helper dh_python2
>>> must be used to fix that issue.
>>>
>>> —
>>> Cyril Scetbon
>>>
 On Mar 18, 2018, at 11:08 AM, Cyril Scetbon 
 > wrote:

 Hey guys,

 Having to still use Cassandra 2.1, I have installed it on Ubuntu
 Xenial and I have an issue with cqlsh. I was able to fix it by
 installing python-support and a fix from 2.1.16. However I’d like to
 know if there is a way to do it without installing an old package
 (python-support) on Xenial. dh-python is supposed to have replaced
 python-support, however cqlsh complains when it’s not installed : 

 Traceback (most recent call last):
   File "/usr/bin/cqlsh", line 121, in 
     from cqlshlib import cql3handling, cqlhandling, pylexotron,
 sslhandling
 ImportError: No module named cqlshlib

 Is there a better way than installing that old package ?

 Thanks 
 —
 Cyril Scetbon

>>>
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> 
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: How to Protect Tracing Requests From Client Side

[Mick Semb Wever]

>
> Is there a way to protect C* on the server side from tracing commands that
> are executed from clients?
>


If you really needed a way to completely disable all and any possibility of
tracing you could start each C* node with tracing switched to a noop
implementation.

eg, add to the jvm.options file the line

-Dcassandra.custom_tracing_class=somepackage.NoOpTracing


while also putting into each $CASSANDRA_HOME/lib/ a jar file containing
this NoOpTracing class…

```
package somepackage;

import java.net.InetAddress;
import java.nio.ByteBuffer;
import java.util.Map;
import java.util.UUID;
import org.apache.cassandra.tracing.*;
import org.apache.cassandra.utils.FBUtilities;

/** Starting Cassandra with
'-Dcassandra.custom_tracing_class=org.apache.cassandra.tracing.NoOpTracing'
 * will forcibly disable all tracing.
 *
 * This can be useful in defensive environments.
 */
public final class NoOpTracing extends Tracing {

@Override
protected void stopSessionImpl() {}

@Override
public TraceState begin(String request, InetAddress client, Map parameters) {
return NoOpTraceState.INSTANCE;
}

@Override
protected TraceState newTraceState(InetAddress coordinator, UUID
sessionId, TraceType traceType) {
return NoOpTraceState.INSTANCE;
}

@Override
public void trace(ByteBuffer sessionId, String message, int ttl) {}

private static class NoOpTraceState extends TraceState {
private static final NoOpTraceState INSTANCE = new NoOpTraceState();
private NoOpTraceState() {
super(FBUtilities.getBroadcastAddress(), UUID.randomUUID(),
TraceType.NONE);
}
@Override
protected void traceImpl(String message) {}
}
}
```

regards,
Mick


-- 
Mick Semb Wever
Australia

The Last Pickle
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: replace dead node vs remove node

[Jonathan Haddad]

Ah sorry - I misread the original post - for some reason I had it in my
head the question was about bootstrap.

Carry on.

On Thu, Mar 22, 2018 at 8:35 PM Jonathan Haddad  wrote:

> Under normal circumstances this is not true.
>
> Take a look at org.apache.cassandra.service.StorageProxy#performWrite, it
> grabs both the natural endpoints and the pending endpoints (new nodes).
> They're eventually passed through
> to 
> org.apache.cassandra.locator.AbstractReplicationStrategy#getWriteResponseHandler,
> which keeps track of both the current endpoints and the pending ones.
> Later, it gets to the actual work:
>
> performer.apply(mutation, Iterables.concat(naturalEndpoints, 
> pendingEndpoints), responseHandler, localDataCenter, consistency_level);
>
> The signature of this method is:
>
> public interface WritePerformer
> {
> public void apply(IMutation mutation,
>   Iterable targets,
>   AbstractWriteResponseHandler responseHandler,
>   String localDataCenter,
>   ConsistencyLevel consistencyLevel) throws 
> OverloadedException;
> }
>
> Notice the targets?  That's the list of all current owners and pending
> owners.  The list is a concatenation of the natural endpoints and the
> pending ones.
>
> Pending owners are listed in org.apache.cassandra.locator.TokenMetadata
>
> // this is a cache of the calculation from {tokenToEndpointMap, 
> bootstrapTokens, leavingEndpoints}
> private final ConcurrentMap pendingRanges = new 
> ConcurrentHashMap();
>
>
> TL;DR: mutations are sent to nodes being bootstrapped.
>
> Jon
>
>
> On Thu, Mar 22, 2018 at 8:09 PM Peng Xiao <2535...@qq.com> wrote:
>
>> Hi Anthony,
>>
>> there is a problem with replacing dead node as per the blog,if the
>> replacement process takes longer than max_hint_window_in_ms,we must run
>> repair to make the replaced node consistent again, since it missed ongoing
>> writes during bootstrapping.but for a great cluster,repair is a painful
>> process.
>>
>> Thanks,
>> Peng Xiao
>>
>>
>>
>> -- 原始邮件 --
>> *发件人:* "Anthony Grasso";
>> *发送时间:* 2018年3月22日(星期四) 晚上7:13
>> *收件人:* "user";
>> *主题:* Re: replace dead node vs remove node
>>
>> Hi Peng,
>>
>> Depending on the hardware failure you can do one of two things:
>>
>> 1. If the disks are intact and uncorrupted you could just use the disks
>> with the current data on them in the new node. Even if the IP address
>> changes for the new node that is fine. In that case all you need to do is
>> run repair on the new node. The repair will fix any writes the node missed
>> while it was down. This process is similar to the scenario in this blog
>> post:
>> http://thelastpickle.com/blog/2018/02/21/replace-node-without-bootstrapping.html
>>
>> 2. If the disks are inaccessible or corrupted, then use the method as
>> described in the blogpost you linked to. The operation is similar to
>> bootstrapping a new node. There is no need to perform any other remove or
>> join operation on the failed or new nodes. As per the blog post, you
>> definitely want to run repair on the new node as soon as it joins the
>> cluster. In this case here, the data on the failed node is effectively lost
>> and replaced with data from other nodes in the cluster.
>>
>> Hope this helps.
>>
>> Regards,
>> Anthony
>>
>>
>> On Thu, 22 Mar 2018 at 20:52, Peng Xiao <2535...@qq.com> wrote:
>>
>>> Dear All,
>>>
>>> when one node failure with hardware errors,it will be in DN status in
>>> the cluster.Then if we are not able to handle this error in three hours(max
>>> hints window),we will loss data,right?we have to run repair to keep the
>>> consistency.
>>> And as per
>>> https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html,we
>>> can replace this dead node,is it the same as bootstrap new node?that means
>>> we don't need to remove node and rejoin?
>>> Could anyone please advise?
>>>
>>> Thanks,
>>> Peng Xiao
>>>
>>>
>>>
>>>
>>>

Understanding Blocked and All Time Blocked columns in tpstats

[John Sanda]

I have been doing some work on a cluster that is impacted by
https://issues.apache.org/jira/browse/CASSANDRA-11363. Reading through the
ticket prompted me to take a closer look at
org.apache.cassandra.concurrent.SEPExecutor. I am looking at the 3.0.14
code. I am a little confused about the Blocked and All Time Blocked columns
reported in nodetool tpstats and reported by StatusLogger. I understand
that there is a queue for tasks. In the case of RequestThreadPoolExecutor,
the size of that queue can be controlled via the
cassandra.max_queued_native_transport_requests system property.

I have been looking at SEPExecutor.addTask(FutureTask task), and here is
my question. If the queue is full, as defined by SEPExector.maxTasksQueued,
are tasks rejected? I do not fully grok the code, but it looks like it is
possible for tasks to be rejected here (some code and comments omitted for
brevity):

public void addTask(FutureTask task)
{
tasks.add(task);
...
else if (taskPermits >= maxTasksQueued)
{
WaitQueue.Signal s = hasRoom.register();

if (taskPermits(permits.get()) > maxTasksQueued)
{
if (takeWorkPermit(true))
pool.schedule(new Work(this))

metrics.totalBlocked.inc();
metrics.currentBlocked.inc();
s.awaitUninterruptibly();
metrics.currentBlocked.dec();
}
else
s.cancel();
}
}

The first thing that happens is that the task is added to the tasks queue.
pool.schedule() only gets called if takeWorkPermit() returns true. I am
still studying the code, but can someone explain what exactly happens when
the queue is full?


- John

Re: replace dead node vs remove node

[Jonathan Haddad]

Under normal circumstances this is not true.

Take a look at org.apache.cassandra.service.StorageProxy#performWrite, it
grabs both the natural endpoints and the pending endpoints (new nodes).
They're eventually passed through
to 
org.apache.cassandra.locator.AbstractReplicationStrategy#getWriteResponseHandler,
which keeps track of both the current endpoints and the pending ones.
Later, it gets to the actual work:

performer.apply(mutation, Iterables.concat(naturalEndpoints,
pendingEndpoints), responseHandler, localDataCenter,
consistency_level);

The signature of this method is:

public interface WritePerformer
{
public void apply(IMutation mutation,
  Iterable targets,
  AbstractWriteResponseHandler responseHandler,
  String localDataCenter,
  ConsistencyLevel consistencyLevel) throws
OverloadedException;
}

Notice the targets?  That's the list of all current owners and pending
owners.  The list is a concatenation of the natural endpoints and the
pending ones.

Pending owners are listed in org.apache.cassandra.locator.TokenMetadata

// this is a cache of the calculation from {tokenToEndpointMap,
bootstrapTokens, leavingEndpoints}
private final ConcurrentMap pendingRanges =
new ConcurrentHashMap();


TL;DR: mutations are sent to nodes being bootstrapped.

Jon


On Thu, Mar 22, 2018 at 8:09 PM Peng Xiao <2535...@qq.com> wrote:

> Hi Anthony,
>
> there is a problem with replacing dead node as per the blog,if the
> replacement process takes longer than max_hint_window_in_ms,we must run
> repair to make the replaced node consistent again, since it missed ongoing
> writes during bootstrapping.but for a great cluster,repair is a painful
> process.
>
> Thanks,
> Peng Xiao
>
>
>
> -- 原始邮件 --
> *发件人:* "Anthony Grasso";
> *发送时间:* 2018年3月22日(星期四) 晚上7:13
> *收件人:* "user";
> *主题:* Re: replace dead node vs remove node
>
> Hi Peng,
>
> Depending on the hardware failure you can do one of two things:
>
> 1. If the disks are intact and uncorrupted you could just use the disks
> with the current data on them in the new node. Even if the IP address
> changes for the new node that is fine. In that case all you need to do is
> run repair on the new node. The repair will fix any writes the node missed
> while it was down. This process is similar to the scenario in this blog
> post:
> http://thelastpickle.com/blog/2018/02/21/replace-node-without-bootstrapping.html
>
> 2. If the disks are inaccessible or corrupted, then use the method as
> described in the blogpost you linked to. The operation is similar to
> bootstrapping a new node. There is no need to perform any other remove or
> join operation on the failed or new nodes. As per the blog post, you
> definitely want to run repair on the new node as soon as it joins the
> cluster. In this case here, the data on the failed node is effectively lost
> and replaced with data from other nodes in the cluster.
>
> Hope this helps.
>
> Regards,
> Anthony
>
>
> On Thu, 22 Mar 2018 at 20:52, Peng Xiao <2535...@qq.com> wrote:
>
>> Dear All,
>>
>> when one node failure with hardware errors,it will be in DN status in the
>> cluster.Then if we are not able to handle this error in three hours(max
>> hints window),we will loss data,right?we have to run repair to keep the
>> consistency.
>> And as per
>> https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html,we
>> can replace this dead node,is it the same as bootstrap new node?that means
>> we don't need to remove node and rejoin?
>> Could anyone please advise?
>>
>> Thanks,
>> Peng Xiao
>>
>>
>>
>>
>>

Re: How to Protect Tracing Requests From Client Side

[Christopher Bradford]

`nodetool settraceeprobability` controls the *automated* tracing within a
single node based on the value set. It may be some or none, but it doesn't
effect queries which are explicitly marked for tracing by the driver within
your application. You can test this by running CQLSH and enabling TRACING
with "TRACING ON;". Now run a query and trace results will be returned.

To resolve your problem it may be worth checking to see if you could limit
the ability to modify tables within the system_traces keyspace via
permissions.

~Chris

On Thu, Mar 22, 2018 at 6:07 PM shalom sagges 
wrote:

> Thanks a lot Rahul! :-)
>
> On Thu, Mar 22, 2018 at 8:03 PM, Rahul Singh  > wrote:
>
>> Execute ‘nodetool settraceprobability 0’ on all nodes. It does zero
>> percentage of he tracing.
>>
>> --
>> Rahul Singh
>> rahul.si...@anant.us
>>
>> Anant Corporation
>>
>> On Mar 22, 2018, 11:10 AM -0500, shalom sagges ,
>> wrote:
>>
>> Hi All,
>>
>> Is there a way to protect C* on the server side from tracing commands
>> that are executed from clients?
>>
>> Thanks!
>>
>>
>

Re: 回复： replace dead node vs remove node

[Jeff Jirsa]

Subrange repair of only the neighbors is sufficient

Break the range covering the dead node into ~100 splits and repair those splits 
individually in sequence. You don’t have to repair the whole range all at once



-- 
Jeff Jirsa


> On Mar 22, 2018, at 8:08 PM, Peng Xiao <2535...@qq.com> wrote:
> 
> Hi Anthony,
> 
> there is a problem with replacing dead node as per the blog,if the 
> replacement process takes longer than max_hint_window_in_ms,we must run 
> repair to make the replaced node consistent again, since it missed ongoing 
> writes during bootstrapping.but for a great cluster,repair is a painful 
> process.
>  
> Thanks,
> Peng Xiao
> 
> 
> 
> -- 原始邮件 --
> 发件人: "Anthony Grasso";
> 发送时间: 2018年3月22日(星期四) 晚上7:13
> 收件人: "user";
> 主题: Re: replace dead node vs remove node
> 
> Hi Peng,
> 
> Depending on the hardware failure you can do one of two things:
> 
> 1. If the disks are intact and uncorrupted you could just use the disks with 
> the current data on them in the new node. Even if the IP address changes for 
> the new node that is fine. In that case all you need to do is run repair on 
> the new node. The repair will fix any writes the node missed while it was 
> down. This process is similar to the scenario in this blog post: 
> http://thelastpickle.com/blog/2018/02/21/replace-node-without-bootstrapping.html
> 
> 2. If the disks are inaccessible or corrupted, then use the method as 
> described in the blogpost you linked to. The operation is similar to 
> bootstrapping a new node. There is no need to perform any other remove or 
> join operation on the failed or new nodes. As per the blog post, you 
> definitely want to run repair on the new node as soon as it joins the 
> cluster. In this case here, the data on the failed node is effectively lost 
> and replaced with data from other nodes in the cluster.
> 
> Hope this helps.
> 
> Regards,
> Anthony
> 
> 
>> On Thu, 22 Mar 2018 at 20:52, Peng Xiao <2535...@qq.com> wrote:
>> Dear All,
>> 
>> when one node failure with hardware errors,it will be in DN status in the 
>> cluster.Then if we are not able to handle this error in three hours(max 
>> hints window),we will loss data,right?we have to run repair to keep the 
>> consistency.
>> And as per 
>> https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html,we
>>  can replace this dead node,is it the same as bootstrap new node?that means 
>> we don't need to remove node and rejoin?
>> Could anyone please advise?
>> 
>> Thanks,
>> Peng Xiao
>> 
>>  
>> 
>> 

?????? replace dead node vs remove node

[Peng Xiao]

Hi Anthony,


there is a problem with replacing dead node as per the blog,if the replacement 
process takes longer than max_hint_window_in_ms,we must run repair to make the 
replaced node consistent again, since it missed ongoing writes during 
bootstrapping.but for a great cluster,repair is a painful process.
 
Thanks,
Peng Xiao






--  --
??: "Anthony Grasso";
: 2018??3??22??(??) 7:13
??: "user";

: Re: replace dead node vs remove node



Hi Peng,

Depending on the hardware failure you can do one of two things:



1. If the disks are intact and uncorrupted you could just use the disks with 
the current data on them in the new node. Even if the IP address changes for 
the new node that is fine. In that case all you need to do is run repair on the 
new node. The repair will fix any writes the node missed while it was down. 
This process is similar to the scenario in this blog post: 
http://thelastpickle.com/blog/2018/02/21/replace-node-without-bootstrapping.html


2. If the disks are inaccessible or corrupted, then use the method as described 
in the blogpost you linked to. The operation is similar to bootstrapping a new 
node. There is no need to perform any other remove or join operation on the 
failed or new nodes. As per the blog post, you definitely want to run repair on 
the new node as soon as it joins the cluster. In this case here, the data on 
the failed node is effectively lost and replaced with data from other nodes in 
the cluster.


Hope this helps.


Regards,
Anthony


On Thu, 22 Mar 2018 at 20:52, Peng Xiao <2535...@qq.com> wrote:

Dear All,


when one node failure with hardware errors,it will be in DN status in the 
cluster.Then if we are not able to handle this error in three hours(max hints 
window),we will loss data,right?we have to run repair to keep the consistency.
And as per 
https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html,we
 can replace this dead node,is it the same as bootstrap new node?that means we 
don't need to remove node and rejoin?
Could anyone please advise?


Thanks,
Peng Xiao

Re: Cassandra 2.1 on Xenial

[Cyril Scetbon]

Yeah, I also had to grab a new version of the cassandra-driver which was fixed 
in 2.1.16 https://issues.apache.org/jira/browse/CASSANDRA-11850 otherwise cqlsh 
would not work with python 2.7.12.

I’m surprised dh-python is not a requirement on the Cassandra package in your 
debian/control 

I also noticed that I get issues with cassandra-stress on Xenial 
https://pastebin.com/raw/r2umZXSV, Any idea ? 

Bests
—
Cyril Scetbon

> On Mar 19, 2018, at 12:16 PM, Michael Shuler  wrote:
> 
> Yep. This was fixed for 2.1.15, 2.2.7, 3.0.6, 3.6 in
> https://issues.apache.org/jira/browse/CASSANDRA-10853 
> 
> 
> -- 
> Kind regards,
> Michael
> 
> On 03/18/2018 10:55 PM, Cyril Scetbon wrote:
>> Okay I found that when building the package, the new helper dh_python2
>> must be used to fix that issue.
>> 
>> —
>> Cyril Scetbon
>> 
>>> On Mar 18, 2018, at 11:08 AM, Cyril Scetbon >> 
>>> >> wrote:
>>> 
>>> Hey guys,
>>> 
>>> Having to still use Cassandra 2.1, I have installed it on Ubuntu
>>> Xenial and I have an issue with cqlsh. I was able to fix it by
>>> installing python-support and a fix from 2.1.16. However I’d like to
>>> know if there is a way to do it without installing an old package
>>> (python-support) on Xenial. dh-python is supposed to have replaced
>>> python-support, however cqlsh complains when it’s not installed : 
>>> 
>>> Traceback (most recent call last):
>>>   File "/usr/bin/cqlsh", line 121, in 
>>> from cqlshlib import cql3handling, cqlhandling, pylexotron,
>>> sslhandling
>>> ImportError: No module named cqlshlib
>>> 
>>> Is there a better way than installing that old package ?
>>> 
>>> Thanks 
>>> —
>>> Cyril Scetbon
>>> 
>> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 

?????? disable compaction in bootstrap process

[Peng Xiao]

Sorry Alain,maybe some misunderstanding here,I mean to disable Compaction in 
the bootstrapping process,then enable it after the bootstrapping.




--  --
??: ""<2535...@qq.com>;
: 2018??3??23??(??) 10:54
??: "user";

: ?? disable compaction in bootstrap process



Thanks Alain.We are using C* 2.1.18,7core/30G/1.5T ssd,as the cluster is 
growing too fast,we are painful in bootstrap/rebuild/remove node.


Thanks,
Peng Xiao


--  --
??: "Alain RODRIGUEZ";
: 2018??3??22??(??) 7:31
??: "user cassandra.apache.org";

: Re: disable compaction in bootstrap process



Hello,
 
Is it reasonable to disable compaction on all the source node?
I would say no, as a short answer.

You can, I did it for some operations in the past. Technically no problem you 
can do that. It will most likely improve the response time of the queries 
immediately as it seems that in your cluster compactions are impacting the 
transactions.

That being said, the impact in the middle/long term will be substantially 
worst. Compactions allow fragments of rows to be merged so the reads can be 
more efficient, hitting the disk just once ideally (at least to reach a 
reasonably low number of hits on the disk). Also, when enabling compactions 
back you might have troubles again as compaction will have to catch up.

Imho, disabling compaction should be an action to take unless your 
understanding about compaction is good enough and you are in a very specific 
case that requires it.
In any case, I would recommend you to stay away from using this solution as a 
quick workaround. It could lead to really wrong situations. Without mentioning 
tombstones that would stack there. Plus, doing this on all the nodes at once is 
really calling for troubles as all the nodes performances might degrade at the 
same pace, roughly.

I would suggest a troubleshooting on why compactions are actually impacting the 
read/write performances.


We probably can help with this here as I believe all the Cassandra users had to 
deal with this at some point (at least people running with 'limited' hardware 
compared to the needs).

Here are some questions that I believe might be useful for us to help you or 
even for you to troubleshoot.

- Is Cassandra limiting thing or resources reaching a limit?
- Is the cluster CPU or Disk bounded?
- What are the number of concurrent compactors and compaction speed in use?
- What hardware are you relying on?
- What version are you using?
- Is compaction keeping up? What compactions strategy are you using?
- 'nodetool tpstats' might also give information on pending and dropped tasks. 
It might be useful.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain


The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2018-03-22 9:09 GMT+00:00 Peng Xiao <2535...@qq.com>:
Dear All,


We noticed that when bootstrap new node,the source node is also quite busy 
doing compactions which impact the rt severely.Is it reasonable to disable 
compaction on all the source node?


Thanks,
Peng Xiao

?????? disable compaction in bootstrap process

[Peng Xiao]

Thanks Alain.We are using C* 2.1.18,7core/30G/1.5T ssd,as the cluster is 
growing too fast,we are painful in bootstrap/rebuild/remove node.


Thanks,
Peng Xiao


--  --
??: "Alain RODRIGUEZ";
: 2018??3??22??(??) 7:31
??: "user cassandra.apache.org";

: Re: disable compaction in bootstrap process



Hello,
 
Is it reasonable to disable compaction on all the source node?
I would say no, as a short answer.

You can, I did it for some operations in the past. Technically no problem you 
can do that. It will most likely improve the response time of the queries 
immediately as it seems that in your cluster compactions are impacting the 
transactions.

That being said, the impact in the middle/long term will be substantially 
worst. Compactions allow fragments of rows to be merged so the reads can be 
more efficient, hitting the disk just once ideally (at least to reach a 
reasonably low number of hits on the disk). Also, when enabling compactions 
back you might have troubles again as compaction will have to catch up.

Imho, disabling compaction should be an action to take unless your 
understanding about compaction is good enough and you are in a very specific 
case that requires it.
In any case, I would recommend you to stay away from using this solution as a 
quick workaround. It could lead to really wrong situations. Without mentioning 
tombstones that would stack there. Plus, doing this on all the nodes at once is 
really calling for troubles as all the nodes performances might degrade at the 
same pace, roughly.

I would suggest a troubleshooting on why compactions are actually impacting the 
read/write performances.


We probably can help with this here as I believe all the Cassandra users had to 
deal with this at some point (at least people running with 'limited' hardware 
compared to the needs).

Here are some questions that I believe might be useful for us to help you or 
even for you to troubleshoot.

- Is Cassandra limiting thing or resources reaching a limit?
- Is the cluster CPU or Disk bounded?
- What are the number of concurrent compactors and compaction speed in use?
- What hardware are you relying on?
- What version are you using?
- Is compaction keeping up? What compactions strategy are you using?
- 'nodetool tpstats' might also give information on pending and dropped tasks. 
It might be useful.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain


The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2018-03-22 9:09 GMT+00:00 Peng Xiao <2535...@qq.com>:
Dear All,


We noticed that when bootstrap new node,the source node is also quite busy 
doing compactions which impact the rt severely.Is it reasonable to disable 
compaction on all the source node?


Thanks,
Peng Xiao

Re: Using Spark to delete from Transactional Cluster

[Rahul Singh]

Short answer : it works. You can even run “delete” statements from within Spark 
once you know which keys to delete. Not elegant but it works.

It will create a bunch of tombstones and you may need to spread your deletes 
over days. Another thing to consider is instead of deleting setting a TTL which 
will eventually get cleansed.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 22, 2018, 2:19 PM -0500, Charulata Sharma (charshar) 
, wrote:
> Hi,
>    Wanted to know the community’s experiences and feedback on using Apache 
> Spark to delete data from C* transactional cluster.
> We have spark installed in our analytical C* cluster and so far we have been 
> using Spark only for analytics purposes.
>
> However, now with advanced features of Spark 2.0, I am considering using 
> spark-cassandra connector for deletes instead of a series of Delete Prepared 
> Statements
> So essentially the deletes will happen on the analytical cluster and they 
> will be replicated over to transactional cluster by means of our keyspace 
> replication strategies.
>
> Are there any risks involved in this ??
>
> Thanks,
> Charu
>

Re: cassl 2.1.x seed node update via JMX

[Jeff Jirsa]

What's the "one-year gossip bug" in this context?


On Thu, Mar 22, 2018 at 3:26 PM, Carl Mueller 
wrote:

> Thanks. The rolling restart triggers the gossip bug so that's a no go.
> We'lre going to migrate off the clsuter. Thanks!
>
>
>
> On Thu, Mar 22, 2018 at 5:04 PM, Nate McCall 
> wrote:
>
>> This capability was *just* added in CASSANDRA-14190 and only in trunk.
>>
>> Previously (as described in the ticket above), the seed node list is only
>> updated when doing a shadow round, removing an endpoint or restarting (look
>> for callers of o.a.c.gms.Gossiper#buildSeedsList() if you're curious).
>>
>> A rolling restart is the usual SOP for that.
>>
>> On Fri, Mar 23, 2018 at 9:54 AM, Carl Mueller <
>> carl.muel...@smartthings.com> wrote:
>>
>>> We have a cluster that is subject to the one-year gossip bug.
>>>
>>> We'd like to update the seed node list via JMX without restart, since
>>> our foolishly single-seed-node in this forsaken cluster is being autoculled
>>> in AWS.
>>>
>>> Is this possible? It is not marked volatile in the Config of the source
>>> code, so I doubt it.
>>>
>>
>>
>>
>> --
>> -
>> Nate McCall
>> Wellington, NZ
>> @zznate
>>
>> CTO
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
>

Re: cassl 2.1.x seed node update via JMX

[Carl Mueller]

Thanks. The rolling restart triggers the gossip bug so that's a no go.
We'lre going to migrate off the clsuter. Thanks!



On Thu, Mar 22, 2018 at 5:04 PM, Nate McCall  wrote:

> This capability was *just* added in CASSANDRA-14190 and only in trunk.
>
> Previously (as described in the ticket above), the seed node list is only
> updated when doing a shadow round, removing an endpoint or restarting (look
> for callers of o.a.c.gms.Gossiper#buildSeedsList() if you're curious).
>
> A rolling restart is the usual SOP for that.
>
> On Fri, Mar 23, 2018 at 9:54 AM, Carl Mueller <
> carl.muel...@smartthings.com> wrote:
>
>> We have a cluster that is subject to the one-year gossip bug.
>>
>> We'd like to update the seed node list via JMX without restart, since our
>> foolishly single-seed-node in this forsaken cluster is being autoculled in
>> AWS.
>>
>> Is this possible? It is not marked volatile in the Config of the source
>> code, so I doubt it.
>>
>
>
>
> --
> -
> Nate McCall
> Wellington, NZ
> @zznate
>
> CTO
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: How to Protect Tracing Requests From Client Side

[shalom sagges]

Thanks a lot Rahul! :-)

On Thu, Mar 22, 2018 at 8:03 PM, Rahul Singh 
wrote:

> Execute ‘nodetool settraceprobability 0’ on all nodes. It does zero
> percentage of he tracing.
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Mar 22, 2018, 11:10 AM -0500, shalom sagges ,
> wrote:
>
> Hi All,
>
> Is there a way to protect C* on the server side from tracing commands that
> are executed from clients?
>
> Thanks!
>
>

Re: cassl 2.1.x seed node update via JMX

[Nate McCall]

This capability was *just* added in CASSANDRA-14190 and only in trunk.

Previously (as described in the ticket above), the seed node list is only
updated when doing a shadow round, removing an endpoint or restarting (look
for callers of o.a.c.gms.Gossiper#buildSeedsList() if you're curious).

A rolling restart is the usual SOP for that.

On Fri, Mar 23, 2018 at 9:54 AM, Carl Mueller 
wrote:

> We have a cluster that is subject to the one-year gossip bug.
>
> We'd like to update the seed node list via JMX without restart, since our
> foolishly single-seed-node in this forsaken cluster is being autoculled in
> AWS.
>
> Is this possible? It is not marked volatile in the Config of the source
> code, so I doubt it.
>



-- 
-
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

Listening to Cassandra on IPv4 and IPv6 at the same time

[sudheer k]

Hi,

Can we listen to Cassandra on IPV4 and IPV6 at the same time? When I refer
to some documents on the internet, it says I can only bind to one address
at a point in time.

In our application, we are talking to Cassandra on FQDN and application
gets either of IPv4 or IPv6 connecting to Cassandra. So, Cassandra has to
bind to both IPv4 and IPv6 at a point in time.

If there is any configuration related to this change, can anyone please let
me know?

--
Regards
Sudheer

cassl 2.1.x seed node update via JMX

[Carl Mueller]

We have a cluster that is subject to the one-year gossip bug.

We'd like to update the seed node list via JMX without restart, since our
foolishly single-seed-node in this forsaken cluster is being autoculled in
AWS.

Is this possible? It is not marked volatile in the Config of the source
code, so I doubt it.

Using Spark to delete from Transactional Cluster

[Charulata Sharma (charshar)]

Hi,
   Wanted to know the community’s experiences and feedback on using Apache 
Spark to delete data from C* transactional cluster.
We have spark installed in our analytical C* cluster and so far we have been 
using Spark only for analytics purposes.

However, now with advanced features of Spark 2.0, I am considering using 
spark-cassandra connector for deletes instead of a series of Delete Prepared 
Statements
So essentially the deletes will happen on the analytical cluster and they will 
be replicated over to transactional cluster by means of our keyspace 
replication strategies.

Are there any risks involved in this ??

Thanks,
Charu

Re: How to Protect Tracing Requests From Client Side

[Rahul Singh]

Execute ‘nodetool settraceprobability 0’ on all nodes. It does zero percentage 
of he tracing.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 22, 2018, 11:10 AM -0500, shalom sagges , wrote:
> Hi All,
>
> Is there a way to protect C* on the server side from tracing commands that 
> are executed from clients?
>
> Thanks!

How to Protect Tracing Requests From Client Side

[shalom sagges]

Hi All,

Is there a way to protect C* on the server side from tracing commands that
are executed from clients?

Thanks!

Re: disable compaction in bootstrap process

[Alain RODRIGUEZ]

Hello,


> Is it reasonable to disable compaction on all the source node?


I would say no, as a short answer.

You can, I did it for some operations in the past. Technically no problem
you can do that. It will most likely improve the response time of the
queries immediately as it seems that in your cluster compactions are
impacting the transactions.

That being said, the impact in the middle/long term will be substantially
worst. Compactions allow fragments of rows to be merged so the reads can be
more efficient, hitting the disk just once ideally (at least to reach a
reasonably low number of hits on the disk). Also, when enabling compactions
back you might have troubles again as compaction will have to catch up.

Imho, disabling compaction should be an action to take unless your
understanding about compaction is good enough and you are in a very
specific case that requires it.
In any case, I would recommend you to stay away from using this solution as
a quick workaround. It could lead to really wrong situations. Without
mentioning tombstones that would stack there. Plus, doing this on all the
nodes at once is really calling for troubles as all the nodes performances
might degrade at the same pace, roughly.

I would suggest a troubleshooting on why compactions are actually impacting
the read/write performances.

We probably can help with this here as I believe all the Cassandra users
had to deal with this at some point (at least people running with 'limited'
hardware compared to the needs).

Here are some questions that I believe might be useful for us to help you
or even for you to troubleshoot.

- Is Cassandra limiting thing or resources reaching a limit?
- Is the cluster CPU or Disk bounded?
- What are the number of concurrent compactors and compaction speed in use?
- What hardware are you relying on?
- What version are you using?
- Is compaction keeping up? What compactions strategy are you using?
- 'nodetool tpstats' might also give information on pending and dropped
tasks. It might be useful.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-03-22 9:09 GMT+00:00 Peng Xiao <2535...@qq.com>:

> Dear All,
>
> We noticed that when bootstrap new node,the source node is also quite
> busy doing compactions which impact the rt severely.Is it reasonable to
> disable compaction on all the source node?
>
> Thanks,
> Peng Xiao
>

Re: replace dead node vs remove node

[Anthony Grasso]

Hi Peng,

Depending on the hardware failure you can do one of two things:

1. If the disks are intact and uncorrupted you could just use the disks
with the current data on them in the new node. Even if the IP address
changes for the new node that is fine. In that case all you need to do is
run repair on the new node. The repair will fix any writes the node missed
while it was down. This process is similar to the scenario in this blog
post:
http://thelastpickle.com/blog/2018/02/21/replace-node-without-bootstrapping.html

2. If the disks are inaccessible or corrupted, then use the method as
described in the blogpost you linked to. The operation is similar to
bootstrapping a new node. There is no need to perform any other remove or
join operation on the failed or new nodes. As per the blog post, you
definitely want to run repair on the new node as soon as it joins the
cluster. In this case here, the data on the failed node is effectively lost
and replaced with data from other nodes in the cluster.

Hope this helps.

Regards,
Anthony


On Thu, 22 Mar 2018 at 20:52, Peng Xiao <2535...@qq.com> wrote:

> Dear All,
>
> when one node failure with hardware errors,it will be in DN status in the
> cluster.Then if we are not able to handle this error in three hours(max
> hints window),we will loss data,right?we have to run repair to keep the
> consistency.
> And as per
> https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html,we
> can replace this dead node,is it the same as bootstrap new node?that means
> we don't need to remove node and rejoin?
> Could anyone please advise?
>
> Thanks,
> Peng Xiao
>
>
>
>
>

replace dead node vs remove node

[Peng Xiao]

Dear All,


when one node failure with hardware errors,it will be in DN status in the 
cluster.Then if we are not able to handle this error in three hours(max hints 
window),we will loss data,right?we have to run repair to keep the consistency.
And as per 
https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html,we
 can replace this dead node,is it the same as bootstrap new node?that means we 
don't need to remove node and rejoin?
Could anyone please advise?


Thanks,
Peng Xiao

disable compaction in bootstrap process

[Peng Xiao]

Dear All,


We noticed that when bootstrap new node,the source node is also quite busy 
doing compactions which impact the rt severely.Is it reasonable to disable 
compaction on all the source node?


Thanks,
Peng Xiao