Re: auto_bootstrap for seed node

[Peng Xiao]

We followed this 
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_dc_to_cluster_t.html,
but it does not mention that change bootstrap for seed nodes after the rebuild.


Thanks,
Peng Xiao 




-- Original --
From:  "Ali Hubail";
Date:  Wed, Mar 28, 2018 10:48 AM
To:  "user";

Subject:  Re: auto_bootstrap for seed node



You might want to follow DataStax docs on this one: 
 
For adding a DC to an existing cluster: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAddDCToCluster.html
 
For adding a new node to an existing cluster: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAddNodeToCluster.html
 
 
briefly speaking, 
adding one node to an existing cluster --> use auto_bootstrap 
adding a DC to an existing cluster --> rebuild 
 
You need to check the version of c* that you're running, and make sure you pick 
the right doc version for that. 
 
Most of my colleagues miss very important steps while adding/removing 
nodes/cluster, but if they stick to the docs, they always get it done right. 
 
Hope this helps 

 Ali Hubail
 
 Confidentiality warning: This message and any attachments are intended only 
for the persons to whom this message is addressed, are confidential, and may be 
privileged. If you are not the intended recipient, you are hereby notified that 
any review, retransmission, conversion to hard copy, copying, modification, 
circulation or other use of this message and any attachments is strictly 
prohibited. If you receive this message in error, please notify the sender 
immediately by return email, and delete this message and any attachments from 
your system. Petrolink International Limited its subsidiaries, holding 
companies and affiliates disclaims all responsibility from and accepts no 
liability whatsoever for the consequences of any unauthorized person acting, or 
refraining from acting, on any information contained in this message. For 
security purposes, staff training, to assist in resolving complaints and to 
improve our customer service, email communications may be monitored and 
telephone calls may be recorded. 
 
 
   "Peng Xiao" <2535...@qq.com>  
03/27/2018 09:39 PM 
   Please respond to
 user@cassandra.apache.org

 
 To
 "user" ,  
  cc
  
  Subject
 auto_bootstrap for seed node
 

 

 
 
 
Dear All, 
 
For adding a new DC ,we need to set auto_bootstrap: false and then run the 
rebuild,finally we need to change auto_bootstrap: true,but for seed nodes,it 
seems that we still need to keep bootstrap false? 
Could anyone please confirm? 
 
Thanks, 
Peng Xiao

Re: auto_bootstrap for seed node

[Ali Hubail]

You might want to follow DataStax docs on this one:

For adding a DC to an existing cluster: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAddDCToCluster.html
For adding a new node to an existing cluster: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAddNodeToCluster.html

briefly speaking,
adding one node to an existing cluster --> use auto_bootstrap
adding a DC to an existing cluster --> rebuild

You need to check the version of c* that you're running, and make sure you 
pick the right doc version for that.

Most of my colleagues miss very important steps while adding/removing 
nodes/cluster, but if they stick to the docs, they always get it done 
right.

Hope this helps

Ali Hubail

Confidentiality warning: This message and any attachments are intended 
only for the persons to whom this message is addressed, are confidential, 
and may be privileged. If you are not the intended recipient, you are 
hereby notified that any review, retransmission, conversion to hard copy, 
copying, modification, circulation or other use of this message and any 
attachments is strictly prohibited. If you receive this message in error, 
please notify the sender immediately by return email, and delete this 
message and any attachments from your system. Petrolink International 
Limited its subsidiaries, holding companies and affiliates disclaims all 
responsibility from and accepts no liability whatsoever for the 
consequences of any unauthorized person acting, or refraining from acting, 
on any information contained in this message. For security purposes, staff 
training, to assist in resolving complaints and to improve our customer 
service, email communications may be monitored and telephone calls may be 
recorded.



"Peng Xiao" <2535...@qq.com> 
03/27/2018 09:39 PM
Please respond to
user@cassandra.apache.org


To
"user" , 
cc

Subject
auto_bootstrap for seed node






Dear All,

For adding a new DC ,we need to set auto_bootstrap: false and then run the 
rebuild,finally we need to change auto_bootstrap: true,but for seed 
nodes,it seems that we still need to keep bootstrap false?
Could anyone please confirm?

Thanks,
Peng Xiao

auto_bootstrap for seed node

[Peng Xiao]

Dear All,

For adding a new DC ,we need to set auto_bootstrap: false and then run the 
rebuild,finally we need to change auto_bootstrap: true,but for seed nodes,it 
seems that we still need to keep bootstrap false?
Could anyone please confirm?


Thanks,
Peng Xiao

Re: 答复: 答复: A node down every day in a 6 nodes cluster

[Jeff Jirsa]

java.lang.OutOfMemoryError: Java heap space


You’re oom’ ing 

-- 
Jeff Jirsa


> On Mar 27, 2018, at 6:45 PM, Xiangfei Ni  wrote:
> 
> Hi Jeff,
> Today another node was shutdown,I have attached the exception log 
> file,could you please help to analyze?Thanks.
>  
> Best Regards,
>  
> 倪项菲/ David Ni
> 中移德电网络科技有限公司
> Virtue Intelligent Network Ltd, co.
> 
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>  
> 发件人: Jeff Jirsa  
> 发送时间: 2018年3月27日 11:50
> 收件人: Xiangfei Ni 
> 抄送: user@cassandra.apache.org
> 主题: Re: 答复: A node down every day in a 6 nodes cluster
>  
> Only one node having the problem is suspicious. May be that your application 
> is improperly pooling connections, or you have a hardware problem.
>  
> I dont see anything in nodetool that explains it, though you certainly have a 
> data model likely to cause problems over time (the cardinality of 
> rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
> that you have very wide partitions and it'll be difficult to read).
>  
>  
>  
> On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:
> Hi Jeff,
> I need to restart the node manually every time,only one node has this 
> problem.
> I have attached the nodetool output,thanks.
>  
> Best Regards,
>  
> 倪项菲/ David Ni
> 中移德电网络科技有限公司
> Virtue Intelligent Network Ltd, co.
> 
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>  
> 发件人: Jeff Jirsa  
> 发送时间: 2018年3月27日 11:03
> 收件人: user@cassandra.apache.org
> 主题: Re: A node down every day in a 6 nodes cluster
>  
> That warning isn’t sufficient to understand why the node is going down
>  
>  
> Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
> likely a good idea
>  
> Are the nodes coming up on their own? Or are you restarting them?
>  
> Paste the output of nodetool tpstats and nodetool cfstats
>  
>  
>  
> 
> -- 
> Jeff Jirsa
>  
> 
> On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:
> 
> Hi Cassandra experts,
>   I am facing an issue,a node downs every day in a 6 nodes cluster,the 
> cluster is just in one DC,
>   Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
> HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF 
> is 3,a node downs one time every day,the system.log shows below info:
> WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
> CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # nev_tsp_sa> for 
> ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
> QueryMessage.java:128 - Unexpected error during query
> com.google.common.util.concurrent.UncheckedExecutionException: 
> java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
> received only 0 responses.
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
>  ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> 

答复: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Another thins is that I have removed the index which has the wide partition 
rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver as you 
pointed out.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Xiangfei Ni 
发送时间: 2018年3月28日 9:45
收件人: Jeff Jirsa 
抄送: user@cassandra.apache.org
主题: 答复: 答复: A node down every day in a 6 nodes cluster

Hi Jeff,
Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni >
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).





On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]
at 

答复: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Hi Jeff,
Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Jeff Jirsa 
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).





On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
 

Re: Apache Cassandra start up script

[Alex Lourie]

Hi Anumod

Echoing what Christophe said, you should also mention what OS you are
using, so that we would be able to help you get to the solution.

Cheers,
Alex.

On Wed, 28 Mar 2018 at 11:05 Christophe Schmitz 
wrote:

> Hi Anumod,
>
> When you install Cassandra using the tarball install, you will not get a
> service file installed. It becomes your responsability to write one (or to
> take one from the internet).
> When you install Cassandra using a package (.deb, .rpm..), the installer
> should also install the service files, set the conf files in /etc directory
> etc... You should then get access to commands like systemctl start/stop
> Cassandra.
>
>
>
> Hope it helps!
>
> Cheers,
> Christophe
>
> On 28 March 2018 at 11:20, Anumod Mullachery 
> wrote:
>
>> Hi All ,
>>
>> I’ve installed Apache Cassandra( Tarball install )..
>> But there is no way to see the Cassandra status / stop option .
>>
>>
>> Anyone has the Apache Cassandra start / stop script for Cassandra 3.12
>>
>> Can some one help on this ?
>>
>>
>> Thanks,
>>
>> Anumod
>> Mob-718-844-3841
>> PA,USA
>>
>> Sent from my iPhone
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
>
> --
>
> *Christophe Schmitz - **VP Consulting*
>
> AU: +61 4 03751980 / FR: +33 7 82022899 <+33%207%2082%2002%2028%2099>
>
>    
>
>
> Read our latest technical blog posts here
> . This email has been sent on behalf
> of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
> email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
-- 


*Alex Lourie*
*Software Engineer*+61 423177059


   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Execute an external program

[Jeff Jirsa]

CDC may also work for newer versions, but it’ll happen after the mutation is 
applied 

-- 
Jeff Jirsa


> On Mar 27, 2018, at 5:31 PM, Christophe Schmitz  
> wrote:
> 
> Hi Earl,
> 
> You probably want to check Cassandra triggers:
> http://cassandra.apache.org/doc/latest/cql/triggers.html
> You can write arbitrary code that is called for the DML against your table.
> 
> Cheers,
> 
> Christophe
> 
> 
>> On 28 March 2018 at 10:58, Earl Lapus  wrote:
>> Hi All,
>> 
>> I may be over the edge here, but is there a way to execute an external 
>> program if a new row is added or if an existing row is updated on a table?
>> 
>> Cheers,
>> Earl
>> 
>> -- 
>> There are seven words in this sentence.
> 
> 
> 
> -- 
> Christophe Schmitz - VP Consulting
> AU: +61 4 03751980 / FR: +33 7 82022899
> 
> Read our latest technical blog posts here. This email has been sent on behalf 
> of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This email 
> and any attachments may contain confidential and legally privileged 
> information.  If you are not the intended recipient, do not copy or disclose 
> its content, but please reply to this email immediately and highlight the 
> error to the sender and then immediately delete the message.

Re: Apache Cassandra start up script

[Christophe Schmitz]

Hi Anumod,

When you install Cassandra using the tarball install, you will not get a
service file installed. It becomes your responsability to write one (or to
take one from the internet).
When you install Cassandra using a package (.deb, .rpm..), the installer
should also install the service files, set the conf files in /etc directory
etc... You should then get access to commands like systemctl start/stop
Cassandra.



Hope it helps!

Cheers,
Christophe

On 28 March 2018 at 11:20, Anumod Mullachery 
wrote:

> Hi All ,
>
> I’ve installed Apache Cassandra( Tarball install )..
> But there is no way to see the Cassandra status / stop option .
>
>
> Anyone has the Apache Cassandra start / stop script for Cassandra 3.12
>
> Can some one help on this ?
>
>
> Thanks,
>
> Anumod
> Mob-718-844-3841
> PA,USA
>
> Sent from my iPhone
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>


-- 

*Christophe Schmitz - **VP Consulting*

AU: +61 4 03751980 / FR: +33 7 82022899

   


Read our latest technical blog posts here
. This email has been sent on behalf
of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Execute an external program

[Christophe Schmitz]

Hi Earl,

You probably want to check Cassandra triggers:
http://cassandra.apache.org/doc/latest/cql/triggers.html
You can write arbitrary code that is called for the DML against your table.

Cheers,

Christophe


On 28 March 2018 at 10:58, Earl Lapus  wrote:

> Hi All,
>
> I may be over the edge here, but is there a way to execute an external
> program if a new row is added or if an existing row is updated on a table?
>
> Cheers,
> Earl
>
> --
> There are seven words in this sentence.
>



-- 

*Christophe Schmitz - **VP Consulting*

AU: +61 4 03751980 / FR: +33 7 82022899

   


Read our latest technical blog posts here
. This email has been sent on behalf
of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Apache Cassandra start up script

[Anumod Mullachery]

Hi All ,

I’ve installed Apache Cassandra( Tarball install )..
But there is no way to see the Cassandra status / stop option .


Anyone has the Apache Cassandra start / stop script for Cassandra 3.12

Can some one help on this ?


Thanks,

Anumod
Mob-718-844-3841
PA,USA

Sent from my iPhone
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Execute an external program

[Earl Lapus]

Hi All,

I may be over the edge here, but is there a way to execute an external
program if a new row is added or if an existing row is updated on a table?

Cheers,
Earl

-- 
There are seven words in this sentence.

Re: high latency on one node after replacement

[Mike Torra]

thanks for pointing that out, i just found it too :) i overlooked this

On Tue, Mar 27, 2018 at 3:44 PM, Voytek Jarnot 
wrote:

> Have you ruled out EBS snapshot initialization issues (
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html)?
>
> On Tue, Mar 27, 2018 at 2:24 PM, Mike Torra  wrote:
>
>> Hi There -
>>
>> I have noticed an issue where I consistently see high p999 read latency
>> on a node for a few hours after replacing the node. Before replacing the
>> node, the p999 read latency is ~30ms, but after it increases to 1-5s. I am
>> running C* 3.11.2 in EC2.
>>
>> I am testing out using EBS snapshots of the /data disk as a backup, so
>> that I can replace nodes without having to fully bootstrap the replacement.
>> This seems to work ok, except for the latency issue. Some things I have
>> noticed:
>>
>> - `nodetool netstats` doesn't show any 'Completed' Large Messages, only
>> 'Dropped', while this is going on. There are only a few of these.
>> - the logs show warnings like this:
>>
>> WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2018-03-27 18:57:15,655
>> NoSpamLogger.java:94 - Out of 84 commit log syncs over the past 297.28s
>> with average duration of 235.88ms, 86 have exceeded the configured commit
>> interval by an average of 113.66ms
>>   and I can see some slow queries in debug.log, but I can't figure out
>> what is causing it
>> - gc seems normal
>>
>> Could this have something to do with starting the node with the EBS
>> snapshot of the /data directory? My first thought was that this is related
>> to the EBS volumes, but it seems too consistent to be actually caused by
>> that. The problem is consistent across multiple replacements, and multiple
>> EC2 regions.
>>
>> I appreciate any suggestions!
>>
>> - Mike
>>
>
>

Re: high latency on one node after replacement

[Voytek Jarnot]

Have you ruled out EBS snapshot initialization issues (
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html)?

On Tue, Mar 27, 2018 at 2:24 PM, Mike Torra  wrote:

> Hi There -
>
> I have noticed an issue where I consistently see high p999 read latency on
> a node for a few hours after replacing the node. Before replacing the node,
> the p999 read latency is ~30ms, but after it increases to 1-5s. I am
> running C* 3.11.2 in EC2.
>
> I am testing out using EBS snapshots of the /data disk as a backup, so
> that I can replace nodes without having to fully bootstrap the replacement.
> This seems to work ok, except for the latency issue. Some things I have
> noticed:
>
> - `nodetool netstats` doesn't show any 'Completed' Large Messages, only
> 'Dropped', while this is going on. There are only a few of these.
> - the logs show warnings like this:
>
> WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2018-03-27 18:57:15,655
> NoSpamLogger.java:94 - Out of 84 commit log syncs over the past 297.28s
> with average duration of 235.88ms, 86 have exceeded the configured commit
> interval by an average of 113.66ms
>   and I can see some slow queries in debug.log, but I can't figure out
> what is causing it
> - gc seems normal
>
> Could this have something to do with starting the node with the EBS
> snapshot of the /data directory? My first thought was that this is related
> to the EBS volumes, but it seems too consistent to be actually caused by
> that. The problem is consistent across multiple replacements, and multiple
> EC2 regions.
>
> I appreciate any suggestions!
>
> - Mike
>

Re: high latency on one node after replacement

[Christophe Schmitz]

Hi Mike,

Unlike normal EBS volumes for which you don't need to pre-warm, I think
you  need to pre-Warm your EBS volume restored from a snapshot
Have a look at this AWS doc
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html
It says that:
However, storage blocks on volumes that were restored from snapshots must
be initialized (pulled down from Amazon S3 and written to the volume)
before you can access the block. This preliminary action takes time and can
cause a significant increase in the latency of an I/O operation the first
time each block is accessed. For most applications, amortizing this cost
over the lifetime of the volume is acceptable. Performance is restored
after the data is accessed once.

I hope it helps :)

Cheers,

Christophe

On 28 March 2018 at 06:24, Mike Torra  wrote:

> Hi There -
>
> I have noticed an issue where I consistently see high p999 read latency on
> a node for a few hours after replacing the node. Before replacing the node,
> the p999 read latency is ~30ms, but after it increases to 1-5s. I am
> running C* 3.11.2 in EC2.
>
> I am testing out using EBS snapshots of the /data disk as a backup, so
> that I can replace nodes without having to fully bootstrap the replacement.
> This seems to work ok, except for the latency issue. Some things I have
> noticed:
>
> - `nodetool netstats` doesn't show any 'Completed' Large Messages, only
> 'Dropped', while this is going on. There are only a few of these.
> - the logs show warnings like this:
>
> WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2018-03-27 18:57:15,655
> NoSpamLogger.java:94 - Out of 84 commit log syncs over the past 297.28s
> with average duration of 235.88ms, 86 have exceeded the configured commit
> interval by an average of 113.66ms
>   and I can see some slow queries in debug.log, but I can't figure out
> what is causing it
> - gc seems normal
>
> Could this have something to do with starting the node with the EBS
> snapshot of the /data directory? My first thought was that this is related
> to the EBS volumes, but it seems too consistent to be actually caused by
> that. The problem is consistent across multiple replacements, and multiple
> EC2 regions.
>
> I appreciate any suggestions!
>
> - Mike
>



-- 

*Christophe Schmitz - **VP Consulting*

AU: +61 4 03751980 / FR: +33 7 82022899

   


Read our latest technical blog posts here
. This email has been sent on behalf
of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

high latency on one node after replacement

[Mike Torra]

Hi There -

I have noticed an issue where I consistently see high p999 read latency on
a node for a few hours after replacing the node. Before replacing the node,
the p999 read latency is ~30ms, but after it increases to 1-5s. I am
running C* 3.11.2 in EC2.

I am testing out using EBS snapshots of the /data disk as a backup, so that
I can replace nodes without having to fully bootstrap the replacement. This
seems to work ok, except for the latency issue. Some things I have noticed:

- `nodetool netstats` doesn't show any 'Completed' Large Messages, only
'Dropped', while this is going on. There are only a few of these.
- the logs show warnings like this:

WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2018-03-27 18:57:15,655
NoSpamLogger.java:94 - Out of 84 commit log syncs over the past 297.28s
with average duration of 235.88ms, 86 have exceeded the configured commit
interval by an average of 113.66ms
  and I can see some slow queries in debug.log, but I can't figure out what
is causing it
- gc seems normal

Could this have something to do with starting the node with the EBS
snapshot of the /data directory? My first thought was that this is related
to the EBS volumes, but it seems too consistent to be actually caused by
that. The problem is consistent across multiple replacements, and multiple
EC2 regions.

I appreciate any suggestions!

- Mike

Re: Can "data_file_directories" make use of multiple disks?

[Jonathan Haddad]

In Cassandra 3.2 and later, data is partitioned by token range, which
should give you even distribution of data.

If you're going to go into 3.x, please use the latest 3.11, which at this
time is 3.11.2.

On Tue, Mar 27, 2018 at 8:05 AM Venkata Hari Krishna Nukala <
n.v.harikrishna.apa...@gmail.com> wrote:

> Hi,
>
> I am trying to replace machines having HDD with little powerful machines
> having SSD in production. The data present in each node is around 300gb.
> But the newer machines have 2 X 200GB SSDs instead of a single disk.
>
> "data_file_directories" looks like a multi-valued config which I can use.
> Am I looking at the right config?
>
> How does the data is distributed evenly? Leveled Compaction Strategy is
> used for the tables.
>
> Thanks!
>

RE: RE: 答复: A node down every day in a 6 nodes cluster

[Rahul Singh]

It may be that the wife partition is bombarded more than other partitions. 
What’s your RF on that keyspace? If if it’s greater than 1 I’d expect other 
nodes to get the same type of load.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 27, 2018, 5:56 AM -0700, Kenneth Brotman , 
wrote:
> First, anything Jeff Jirsa says is likely very accurate, like it being a 
> really good idea to also get off the version you’re on and onto a version 
> that fixes some of the known problems of the version you’re one.
>
> Replacing a running node:
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceLiveNode.html
>
> Kenneth Brotman
>
>
> From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com]
> Sent: Tuesday, March 27, 2018 5:44 AM
> To: user@cassandra.apache.org
> Subject: Re:RE: 答复: A node down every day in a 6 nodes cluster
>
> Thanks,Kenneth,this is production database,and it is one of three seed 
> nodes,do you have doc for replacing a seed node?
>
>
>
> 发自我的小米手机
> 在 Kenneth Brotman ，2018年3月27日 下午7:45写道：
> David,
>
> Can you replace the misbehaving node to see if that resolves the problem?
>
> Kenneth Brotman
>
> From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com]
> Sent: Tuesday, March 27, 2018 3:27 AM
> To: Jeff Jirsa
> Cc: user@cassandra.apache.org
> Subject: 答复: 答复: A node down every day in a 6 nodes cluster
>
> Thanks Jeff,
>    So your suggestion is to first resolve the data model issue which 
> cause wide partition,right?
>
> Best Regards,
>
> 倪项菲/ David Ni
> 中移德电网络科技有限公司
> Virtue Intelligent Network Ltd, co.
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>
> 发件人: Jeff Jirsa 
> 发送时间: 2018年3月27日 11:50
> 收件人: Xiangfei Ni 
> 抄送: user@cassandra.apache.org
> 主题: Re: 答复: A node down every day in a 6 nodes cluster
>
> Only one node having the problem is suspicious. May be that your application 
> is improperly pooling connections, or you have a hardware problem.
>
> I dont see anything in nodetool that explains it, though you certainly have a 
> data model likely to cause problems over time (the cardinality of
> rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
> that you have very wide partitions and it'll be difficult to read).
>
> On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:
> > quote_type
> > Hi Jeff,
> > I need to restart the node manually every time,only one node has this 
> > problem.
> > I have attached the nodetool output,thanks.
> >
> > Best Regards,
> >
> > 倪项菲/ David Ni
> > 中移德电网络科技有限公司
> > Virtue Intelligent Network Ltd, co.
> > Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
> > Mob: +86 13797007811|Tel: + 86 27 5024 2516
> >
> > 发件人: Jeff Jirsa 
> > 发送时间: 2018年3月27日 11:03
> > 收件人: user@cassandra.apache.org
> > 主题: Re: A node down every day in a 6 nodes cluster
> >
> > That warning isn’t sufficient to understand why the node is going down
> >
> >
> > Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
> > likely a good idea
> >
> > Are the nodes coming up on their own? Or are you restarting them?
> >
> > Paste the output of nodetool tpstats and nodetool cfstats
> >
> >
> >
> > --
> > Jeff Jirsa
> >
> >
> > On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:
> > > Hi Cassandra experts,
> > >   I am facing an issue,a node downs every day in a 6 nodes cluster,the 
> > > cluster is just in one DC,
> > >   Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
> > > HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business 
> > > CF is 3,a node downs one time every day,the system.log shows below info:
> > > WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
> > > CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize 
> > > # for 
> > > ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
> > > QueryMessage.java:128 - Unexpected error during query
> > > com.google.common.util.concurrent.UncheckedExecutionException: 
> > > java.lang.RuntimeException: 
> > > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out 
> > > - received only 0 responses.
> > >     at 
> > > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
> > > ~[guava-18.0.jar:na]
> > >     at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
> > > ~[guava-18.0.jar:na]
> > >     at 
> > > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
> > > ~[guava-18.0.jar:na]
> > >     at 
> > > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> > >  ~[guava-18.0.jar:na]
> > >     at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
> > > ~[apache-cassandra-3.9.jar:3.9]
> > >     at 
> > > 

Re: Can "data_file_directories" make use of multiple disks?

[Rahul Singh]

Yes you can have multiple entries from multiple disks. No guarantee as I can 
see of even distribution. If you want even distribution there are better 
mechanisms for this at the filesystem later.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 27, 2018, 8:05 AM -0700, Venkata Hari Krishna Nukala 
, wrote:
> Hi,
>
> I am trying to replace machines having HDD with little powerful machines 
> having SSD in production. The data present in each node is around 300gb. But 
> the newer machines have 2 X 200GB SSDs instead of a single disk.
>
> "data_file_directories" looks like a multi-valued config which I can use. Am 
> I looking at the right config?
>
> How does the data is distributed evenly? Leveled Compaction Strategy is used 
> for the tables.
>
> Thanks!

Can "data_file_directories" make use of multiple disks?

[Venkata Hari Krishna Nukala]

Hi,

I am trying to replace machines having HDD with little powerful machines
having SSD in production. The data present in each node is around 300gb.
But the newer machines have 2 X 200GB SSDs instead of a single disk.

"data_file_directories" looks like a multi-valued config which I can use.
Am I looking at the right config?

How does the data is distributed evenly? Leveled Compaction Strategy is
used for the tables.

Thanks!

RE: RE: 答复: A node down every day in a 6 nodes cluster

[Kenneth Brotman]

First, anything Jeff Jirsa says is likely very accurate, like it being a really 
good idea to also get off the version you’re on and onto a version that fixes 
some of the known problems of the version you’re one.

 

Replacing a running node:

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceLiveNode.html

 

Kenneth Brotman

 

 

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com] 
Sent: Tuesday, March 27, 2018 5:44 AM
To: user@cassandra.apache.org
Subject: Re:RE: 答复: A node down every day in a 6 nodes cluster

 

Thanks,Kenneth,this is production database,and it is one of three seed nodes,do 
you have doc for replacing a seed node?

 

 

 

发自我的小米手机

在 Kenneth Brotman ，2018年3月27日 下午7:45写道：

David,

 

Can you replace the misbehaving node to see if that resolves the problem?

 

Kenneth Brotman

 

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com] 
Sent: Tuesday, March 27, 2018 3:27 AM
To: Jeff Jirsa
Cc: user@cassandra.apache.org
Subject: 答复: 答复: A node down every day in a 6 nodes cluster

 

Thanks Jeff,

   So your suggestion is to first resolve the data model issue which 
cause wide partition,right?

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

 

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

 

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of 

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).
 
 

 

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

Hi Jeff,

I need to restart the node manually every time,only one node has this 
problem.

I have attached the nodetool output,thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811  |Tel: + 86 27 5024 2516 
 

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

 

That warning isn’t sufficient to understand why the node is going down

 

 

Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

 

Are the nodes coming up on their own? Or are you restarting them?

 

Paste the output of nodetool tpstats and nodetool cfstats

 

 

 

-- 

Jeff Jirsa

 


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:

Hi Cassandra experts,

  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,

  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:

WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 

ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query

com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]

at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]

at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]

at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]

at 

Re:RE: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Thanks,Kenneth,this is production database,and it is one of three seed nodes,do 
you have doc for replacing a seed node?



发自我的小米手机
在 Kenneth Brotman ，2018年3月27日 下午7:45写道：
David,

Can you replace the misbehaving node to see if that resolves the problem?

Kenneth Brotman

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com]
Sent: Tuesday, March 27, 2018 3:27 AM
To: Jeff Jirsa
Cc: user@cassandra.apache.org
Subject: 答复: 答复: A node down every day in a 6 nodes cluster

Thanks Jeff,
   So your suggestion is to first resolve the data model issue which 
cause wide partition,right?

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni >
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).





On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
 ~[apache-cassandra-3.9.jar:3.9]
at 

RE: 答复: A node down every day in a 6 nodes cluster

[Kenneth Brotman]

David,

 

Can you replace the misbehaving node to see if that resolves the problem?

 

Kenneth Brotman

 

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com] 
Sent: Tuesday, March 27, 2018 3:27 AM
To: Jeff Jirsa
Cc: user@cassandra.apache.org
Subject: 答复: 答复: A node down every day in a 6 nodes cluster

 

Thanks Jeff,

   So your suggestion is to first resolve the data model issue which 
cause wide partition,right?

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

 

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

 

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of 

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).
 
 

 

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

Hi Jeff,

I need to restart the node manually every time,only one node has this 
problem.

I have attached the nodetool output,thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811  |Tel: + 86 27 5024 2516 
 

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

 

That warning isn’t sufficient to understand why the node is going down

 

 

Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

 

Are the nodes coming up on their own? Or are you restarting them?

 

Paste the output of nodetool tpstats and nodetool cfstats

 

 

 

-- 

Jeff Jirsa

 


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:

Hi Cassandra experts,

  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,

  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:

WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 

ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query

com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]

at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]

at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]

at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
~[apache-cassandra-3.9.jar:3.9]

at 

答复: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Thanks Jeff,
   So your suggestion is to first resolve the data model issue which 
cause wide partition,right?

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Jeff Jirsa 
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).





On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
 

答复: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Hi Jeff,
This is very strange that the table info that the max partition is 
129557750(123M):

[cid:image001.png@01D3C5E0.0A9F8910]
But why the index max partition is nearly 2G,
 Table (index): 
rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver
SSTable count: 7
Space used (live): 1049948206
Space used (total): 1049948206
Space used by snapshots (total): 0
Off heap memory used (total): 377947
SSTable Compression Ratio: 0.3381407723053012
Number of keys (estimate): 2
Memtable cell count: 9435
Memtable data size: 429878
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 212512
Local write latency: 0.052 ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 112
Bloom filter off heap memory used: 56
Index summary off heap memory used: 91
Compression metadata off heap memory used: 377800
Compacted partition minimum bytes: 785940
Compacted partition maximum bytes: 1996099046
Compacted partition mean bytes: 495191984
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Jeff Jirsa 
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).





On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at