Is Cassandra used in Medical industry?

[Sudhakar Ganesan]

Hi,

Did anyone used Cassandra in medical industry since FDA enforces CFR 21 (part 
11) compliance ?

Regards,
Sudhakar

Legal Disclaimer :
The information contained in this message may be privileged and confidential. 
It is intended to be read only by the individual or entity to whom it is 
addressed 
or by their designee. If the reader of this message is not the intended 
recipient, 
you are on notice that any distribution of this message, in any form, 
is strictly prohibited. If you have received this message in error, 
please immediately notify the sender and delete or destroy any copy of this 
message!

RE: Cassandra upgrade from 2.2.8 to 3.10

[Fd Habash]

Thank you.

In regards to my second inquiry, as we plan for C* upgrades, I did not find the 
NEWS.txt always to be telling of possible upgrade paths. Is there a rule of 
thumb or may be an official reference for upgrade paths?




Thank you

From: Alexander Dejanovski
Sent: Wednesday, March 28, 2018 1:58 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra upgrade from 2.2.8 to 3.10

You can perform an upgrade from 2.2.x straight to 3.11.2, but the op suggests 
adding nodes in 3.10 to a cluster that runs 2.2.8, which is why Jeff says it 
won't work.

I see no reason to upgrade to 3.10 and not 3.11.2 by the way.

On Wed, Mar 28, 2018 at 5:10 PM Fred Habash  wrote:
Hi ...
I'm finding anecdotal evidence on the internet that we are able to upgrade 
2.2.8 to latest 3.11.2. Post below indicates that you can upgrade to latest 3.x 
from 2.1.9 because 3.x no longer requires 'structured upgrade path'.

I just want to confirm that such upgrade is supported. If yes, where can I find 
official documentation showing upgrade path across releases.

https://stackoverflow.com/questions/42094935/apache-cassandra-upgrade-3-x-from-2-1

Thanks 

On Mon, Aug 7, 2017 at 5:58 PM, ZAIDI, ASAD A  wrote:
Hi folks, I’ve question on upgrade method I’m thinking to execute.
 
I’m  planning from apache-Cassandra 2.2.8 to release 3.10.
 
My Cassandra cluster is configured like one rack with two Datacenters like:
 
1.   DC1 has 4 nodes 
2.   DC2 has 16 nodes
 
We’re adding another 12 nodes and would eventually need to remove those 4 nodes 
in DC1.
 
I’m thinking to add another third data center with like DC3 with 12 nodes 
having apache Cassandra 3.10 installed. Then, I start upgrading seed nodes 
first in DC1 & DC2 – once all 20nodes in ( DC1 plus DC2) upgraded – I can 
safely remove 4 DC1 nodes,
can you guys please let me know if this approach would work? I’m concerned if 
having mixed version on Cassandra nodes may  cause any issues like in streaming 
 data/sstables from existing DC to newly created third DC with version 3.10 
installed, will nodes in DC3 join the cluster with data without issues?
 
Thanks/Asad
 
 
 
 




-- 


Thank you ...

Fred Habash, Database Solutions Architect (Oracle OCP 8i,9i,10g,11g)

-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Cassandra upgrade from 2.2.8 to 3.10

[Alexander Dejanovski]

You can perform an upgrade from 2.2.x straight to 3.11.2, but the op
suggests adding nodes in 3.10 to a cluster that runs 2.2.8, which is why
Jeff says it won't work.

I see no reason to upgrade to 3.10 and not 3.11.2 by the way.

On Wed, Mar 28, 2018 at 5:10 PM Fred Habash  wrote:

> Hi ...
> I'm finding anecdotal evidence on the internet that we are able to upgrade
> 2.2.8 to latest 3.11.2. Post below indicates that you can upgrade to latest
> 3.x from 2.1.9 because 3.x no longer requires 'structured upgrade path'.
>
> I just want to confirm that such upgrade is supported. If yes, where can I
> find official documentation showing upgrade path across releases.
>
>
> https://stackoverflow.com/questions/42094935/apache-cassandra-upgrade-3-x-from-2-1
>
> Thanks
>
> On Mon, Aug 7, 2017 at 5:58 PM, ZAIDI, ASAD A  wrote:
>
>> Hi folks, I’ve question on upgrade method I’m thinking to execute.
>>
>>
>>
>> I’m  planning from apache-Cassandra 2.2.8 to release 3.10.
>>
>>
>>
>> My Cassandra cluster is configured like one rack with two Datacenters
>> like:
>>
>>
>>
>> 1.   DC1 has 4 nodes
>>
>> 2.   DC2 has 16 nodes
>>
>>
>>
>> We’re adding another 12 nodes and would eventually need to remove those 4
>> nodes in DC1.
>>
>>
>>
>> I’m thinking to add another third data center with like DC3 with 12 nodes
>> having apache Cassandra 3.10 installed. Then, I start upgrading seed nodes
>> first in DC1 & DC2 – once all 20nodes in ( DC1 plus DC2) upgraded – I can
>> safely remove 4 DC1 nodes,
>>
>> can you guys please let me know if this approach would work? I’m
>> concerned if having mixed version on Cassandra nodes may  cause any issues
>> like in streaming  data/sstables from existing DC to newly created third DC
>> with version 3.10 installed, will nodes in DC3 join the cluster with data
>> without issues?
>>
>>
>>
>> Thanks/Asad
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
> --
>
>
> Thank you ...
> 
> Fred Habash, Database Solutions Architect (Oracle OCP 8i,9i,10g,11g)
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Cassandra upgrade from 2.2.8 to 3.10

[Fred Habash]

Hi ...
I'm finding anecdotal evidence on the internet that we are able to upgrade
2.2.8 to latest 3.11.2. Post below indicates that you can upgrade to latest
3.x from 2.1.9 because 3.x no longer requires 'structured upgrade path'.

I just want to confirm that such upgrade is supported. If yes, where can I
find official documentation showing upgrade path across releases.

https://stackoverflow.com/questions/42094935/apache-cassandra-upgrade-3-x-from-2-1

Thanks

On Mon, Aug 7, 2017 at 5:58 PM, ZAIDI, ASAD A  wrote:

> Hi folks, I’ve question on upgrade method I’m thinking to execute.
>
>
>
> I’m  planning from apache-Cassandra 2.2.8 to release 3.10.
>
>
>
> My Cassandra cluster is configured like one rack with two Datacenters like:
>
>
>
> 1.   DC1 has 4 nodes
>
> 2.   DC2 has 16 nodes
>
>
>
> We’re adding another 12 nodes and would eventually need to remove those 4
> nodes in DC1.
>
>
>
> I’m thinking to add another third data center with like DC3 with 12 nodes
> having apache Cassandra 3.10 installed. Then, I start upgrading seed nodes
> first in DC1 & DC2 – once all 20nodes in ( DC1 plus DC2) upgraded – I can
> safely remove 4 DC1 nodes,
>
> can you guys please let me know if this approach would work? I’m concerned
> if having mixed version on Cassandra nodes may  cause any issues like in
> streaming  data/sstables from existing DC to newly created third DC with
> version 3.10 installed, will nodes in DC3 join the cluster with data
> without issues?
>
>
>
> Thanks/Asad
>
>
>
>
>
>
>
>
>



-- 


Thank you ...

Fred Habash, Database Solutions Architect (Oracle OCP 8i,9i,10g,11g)

RE: Cassandra certification

[Ali Hubail]

It certainly depends on the corporation. I've seen some that value 
certificates, and some others that do not bother with them at all. It also 
depends on the certificate as well. Companies don't care that much about 
Oracle Java Certifications, but they tend to take SAP certificates more 
seriously.

As for DataStax certificates, I find that taking them do not hurt (maybe 
your pocket though). The exam covers a wide range of topics, and their 
online courses are probably the best entry point to c*
DataStax is well recognized in the community. Regardless of the past 
differences between Apache Cassandra and DataStax, DataStax Enterprise is 
to Apache Cassandra as RedHat Enterprise Linux (REHL) to Linux.

Tupshin, from DataStax, about 2 years ago mentioned "multiple vendors that 
had booths at the show made comments amounting to "we'd love to hire 
anybody that passes"
https://www.reddit.com/r/cassandra/comments/3ztbn3/any_thoughtscomments_on_the_datastax_cassandra/

I personally won't judge anyone for not having a DS certificate in an 
interview. I think the experience is what I am looking for, but having the 
certificate at least says that you've been validated by someone else. 
That's only for entry point positions. But for senior positions, I doubt 
people look into such certificates. Experience is what matters. If we 
could afford hiring some of the folks in d...@cassandra.apache.org, the 
last thing we would care about is certificates. I am pretty sure some 
(probably most) of them  are not "certified" but their experience and 
contribution to the community is the black belt people look for.
Again, DS certificates don't hurt, a good starting point, and well 
recognized. See if your company is willing to pay the fees, though

Ali Hubail

Confidentiality warning: This message and any attachments are intended 
only for the persons to whom this message is addressed, are confidential, 
and may be privileged. If you are not the intended recipient, you are 
hereby notified that any review, retransmission, conversion to hard copy, 
copying, modification, circulation or other use of this message and any 
attachments is strictly prohibited. If you receive this message in error, 
please notify the sender immediately by return email, and delete this 
message and any attachments from your system. Petrolink International 
Limited its subsidiaries, holding companies and affiliates disclaims all 
responsibility from and accepts no liability whatsoever for the 
consequences of any unauthorized person acting, or refraining from acting, 
on any information contained in this message. For security purposes, staff 
training, to assist in resolving complaints and to improve our customer 
service, email communications may be monitored and telephone calls may be 
recorded.



Chandan Goel  
03/28/2018 09:50 AM
Please respond to
user@cassandra.apache.org


To
"user@cassandra.apache.org" , 
cc

Subject
RE: Cassandra certification






Thanks for the answer Rahul.
 
I actually wanted to know that if there is any other certification 
authority better than Datastax ? or is it the only one as of now ?
 
From: Rahul Singh [mailto:rahul.xavier.si...@gmail.com] 
Sent: Saturday, March 24, 2018 3:38 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra certification
 
Recognized where? If someone has a Java certification and goes to a .NET 
shop that certificate means nothing. Same goes with a DataStax 
“certificate.” It’s a starting ground — can get you in the door at places 
that uses the technology but doesn’t substitute for experience. Both are 
important.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 24, 2018, 5:38 AM -0700, Chandan Goel , wrote:

Hi,
 
Is the professional certification on Cassandra given by Datastax 
recognized ?
Oreilly has stopped giving any certifications , it seems. Is there any 
other authority giving away certifications on Cassandra post clearing exam 
which is better than datastax? Or Datastax is sufficiently good , any idea 
anyone ?
 
Thanks
Chandan





Disclaimer The information in this email and any attachments may contain 
proprietary and confidential information that is intended for the 
addressee(s) only. If you are not the intended recipient, you are hereby 
notified that any disclosure, copying, distribution, retention or use of 
the contents of this information is prohibited. When addressed to our 
clients or vendors, any information contained in this e-mail or any 
attachments is subject to the terms and conditions in any governing 
contract. If you have received this e-mail in error, please immediately 
contact the sender and delete the e-mail.




 
Disclaimer The information in this email and any attachments may contain 
proprietary and confidential information 

Re: auto_bootstrap for seed node

[Ali Hubail]

"it seems that we still need to keep bootstrap false?"

Could you shed some light on what would happen if the auto_bootstrap is 
removed (or set to true as the default value) in the seed nodes of the 
newly added DC?

What do you have in the seeds param of the new DC nodes (cassandra.yaml)? 
Do you reference the old DC seed nodes there as well?

Ali Hubail

Email: ali.hub...@petrolink.com | www.petrolink.com 
Confidentiality warning: This message and any attachments are intended 
only for the persons to whom this message is addressed, are confidential, 
and may be privileged. If you are not the intended recipient, you are 
hereby notified that any review, retransmission, conversion to hard copy, 
copying, modification, circulation or other use of this message and any 
attachments is strictly prohibited. If you receive this message in error, 
please notify the sender immediately by return email, and delete this 
message and any attachments from your system. Petrolink International 
Limited its subsidiaries, holding companies and affiliates disclaims all 
responsibility from and accepts no liability whatsoever for the 
consequences of any unauthorized person acting, or refraining from acting, 
on any information contained in this message. For security purposes, staff 
training, to assist in resolving complaints and to improve our customer 
service, email communications may be monitored and telephone calls may be 
recorded.



"Peng Xiao" <2535...@qq.com> 
03/28/2018 12:54 AM
Please respond to
user@cassandra.apache.org


To
"user" , 
cc

Subject
Re:  auto_bootstrap for seed node






We followed this 
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_dc_to_cluster_t.html
,
but it does not mention that change bootstrap for seed nodes after the 
rebuild.

Thanks,
Peng Xiao 


-- Original --
From:  "Ali Hubail";
Date:  Wed, Mar 28, 2018 10:48 AM
To:  "user";
Subject:  Re: auto_bootstrap for seed node

You might want to follow DataStax docs on this one: 

For adding a DC to an existing cluster: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAddDCToCluster.html
 

For adding a new node to an existing cluster: 
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAddNodeToCluster.html
 


briefly speaking, 
adding one node to an existing cluster --> use auto_bootstrap 
adding a DC to an existing cluster --> rebuild 

You need to check the version of c* that you're running, and make sure you 
pick the right doc version for that. 

Most of my colleagues miss very important steps while adding/removing 
nodes/cluster, but if they stick to the docs, they always get it done 
right. 

Hope this helps 

Ali Hubail

Confidentiality warning: This message and any attachments are intended 
only for the persons to whom this message is addressed, are confidential, 
and may be privileged. If you are not the intended recipient, you are 
hereby notified that any review, retransmission, conversion to hard copy, 
copying, modification, circulation or other use of this message and any 
attachments is strictly prohibited. If you receive this message in error, 
please notify the sender immediately by return email, and delete this 
message and any attachments from your system. Petrolink International 
Limited its subsidiaries, holding companies and affiliates disclaims all 
responsibility from and accepts no liability whatsoever for the 
consequences of any unauthorized person acting, or refraining from acting, 
on any information contained in this message. For security purposes, staff 
training, to assist in resolving complaints and to improve our customer 
service, email communications may be monitored and telephone calls may be 
recorded. 


"Peng Xiao" <2535...@qq.com> 
03/27/2018 09:39 PM 


Please respond to
user@cassandra.apache.org


To
"user" , 
cc

Subject
auto_bootstrap for seed node








Dear All, 

For adding a new DC ,we need to set auto_bootstrap: false and then run the 
rebuild,finally we need to change auto_bootstrap: true,but for seed 
nodes,it seems that we still need to keep bootstrap false? 
Could anyone please confirm? 

Thanks, 
Peng Xiao 

RE: Cassandra certification

[Chandan Goel]

Thanks for the answer Rahul.

I actually wanted to know that if there is any other certification authority 
better than Datastax ? or is it the only one as of now ?

From: Rahul Singh [mailto:rahul.xavier.si...@gmail.com]
Sent: Saturday, March 24, 2018 3:38 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra certification

Recognized where? If someone has a Java certification and goes to a .NET shop 
that certificate means nothing. Same goes with a DataStax “certificate.” It’s a 
starting ground — can get you in the door at places that uses the technology 
but doesn’t substitute for experience. Both are important.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 24, 2018, 5:38 AM -0700, Chandan Goel 
>, wrote:

Hi,

Is the professional certification on Cassandra given by Datastax recognized ?
Oreilly has stopped giving any certifications , it seems. Is there any other 
authority giving away certifications on Cassandra post clearing exam which is 
better than datastax? Or Datastax is sufficiently good , any idea anyone ?

Thanks
Chandan





Disclaimer The information in this email and any attachments may contain 
proprietary and confidential information that is intended for the addressee(s) 
only. If you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution, retention or use of the contents of this 
information is prohibited. When addressed to our clients or vendors, any 
information contained in this e-mail or any attachments is subject to the terms 
and conditions in any governing contract. If you have received this e-mail in 
error, please immediately contact the sender and delete the e-mail.





Disclaimer The information in this email and any attachments may contain 
proprietary and confidential information that is intended for the addressee(s) 
only. If you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution, retention or use of the contents of this 
information is prohibited. When addressed to our clients or vendors, any 
information contained in this e-mail or any attachments is subject to the terms 
and conditions in any governing contract. If you have received this e-mail in 
error, please immediately contact the sender and delete the e-mail.

Re: Execute an external program

[Earl Lapus]

On Wed, Mar 28, 2018 at 8:39 AM, Jeff Jirsa  wrote:

> CDC may also work for newer versions, but it’ll happen after the mutation
> is applied
>
> --
> Jeff Jirsa
>
>
"after the mutation is applied" means after the query is executed?

RE: 答复: 答复: A node down every day in a 6 nodes cluster

[Kenneth Brotman]

Properly Sizing Your Heap to Prevent OutOfMemoryErrors

https://support.datastax.com/hc/en-us/articles/204225929-Properly-Sizing-Your-Heap-to-Prevent-OutOfMemoryErrors

 

 

From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID] 
Sent: Wednesday, March 28, 2018 5:35 AM
To: user@cassandra.apache.org
Subject: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

If you think that will fix the problem, maybe you could add a little more 
memory to each machine as a short term fix.

 

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com] 
Sent: Wednesday, March 28, 2018 5:24 AM
To: user@cassandra.apache.org
Subject: 答复: 答复: 答复: A node down every day in a 6 nodes cluster

 

Yes ,we discussed and plan to figured out the data model issue and upgrade to 
3.11.3 version.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Kenneth Brotman  
发送时间: 2018年3月28日 20:16
收件人: user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

David, 

 

Did you figure out what to do about the data model problem?  It could be that 
your data files finally grow to the point that the data model problem caused 
the Java heap space issue – in which case everything is actually working as 
it’s supposed to; You just have to fix the data model.

 

Kenneth Brotman

 

 

From: Kenneth Brotman [  
mailto:kenbrot...@yahoo.com] 
Sent: Wednesday, March 28, 2018 4:46 AM
To: 'user@cassandra.apache.org'
Subject: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

Was any change to hardware done around the time the problem started ?

Was any change to the client software done around the time the problem started?

Was any change to the database schema done around the time the problem started?

 

Kenneth Brotman

 

From: Xiangfei Ni [  
mailto:xiangfei...@cm-dt.com] 
Sent: Wednesday, March 28, 2018 4:40 AM
To:   user@cassandra.apache.org
Subject: 答复: 答复: 答复: A node down every day in a 6 nodes cluster

 

Hi Kenneth,

The cluster has been running for 4 months,

The problem occurred from last week,

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Kenneth Brotman <  
kenbrot...@yahoo.com.INVALID> 
发送时间: 2018年3月28日 19:34
收件人:   user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

David,

 

How long has the cluster been operating?

How long has the problem been occurring?

 

Kenneth Brotman

 

From: Jeff Jirsa [  mailto:jji...@gmail.com] 
Sent: Tuesday, March 27, 2018 7:00 PM
To: Xiangfei Ni
Cc:   user@cassandra.apache.org
Subject: Re: 答复: 答复: A node down every day in a 6 nodes cluster

 

 

java.langOutOfMemoryError: Java heap space

 

 

You’re oom’ ing 

 

-- 

Jeff Jirsa

 


On Mar 27, 2018, at 6:45 PM, Xiangfei Ni  wrote:

Hi Jeff,

Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Jeff Jirsa <  jji...@gmail.com> 
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni <  xiangfei...@cm-dt.com>
抄送:   user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

 

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

 

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of 

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).
 
 

 

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

Hi Jeff,

I need to restart the node manually every time,only one node has this 
problem.

I have attached the nodetool output,thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob:   +86 13797007811|Tel:  
 + 86 27 5024 2516

 

发件人: Jeff Jirsa <  jji...@gmail.com> 
发送时间: 2018年3月27日 11:03
收件人:   

Re: Execute an external program

[Earl Lapus]

On Wed, Mar 28, 2018 at 8:31 AM, Christophe Schmitz <
christo...@instaclustr.com> wrote:

>
> You probably want to check Cassandra triggers:
> http://cassandra.apache.org/doc/latest/cql/triggers.html
> You can write arbitrary code that is called for the DML against your table.
>
>
>
 Thanks, I'll check this out.

Re: Apache Cassandra start up script

[Romain Gérard]

Hello Anumod,
If you are using systemd you can use this one.
You may want to hardcode the limitNOFILE value if you don't have a new enough 
version of systemd in order to avoid this issue 
https://github.com/systemd/systemd/issues/6559 
(https://link.getmailspring.com/link/1522241261.local-0405fd00-babe-v1.1.5-5834c...@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fsystemd%2Fsystemd%2Fissues%2F6559=dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw%3D%3D)

# /etc/systemd/system/cassandra.service
[Unit]
Description=cassandra daemon
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
ExecStart=/usr/local/cassandra/bin/cassandra -f -p 
/var/run/cassandra/cassandra.pid
Type=simple
User=cassandra
Group=cassandra
StandardOutput=journal
StandardError=inherit
Environment="CASSANDRA_HOME=/usr/local/cassandra" 
"CASSANDRA_CONF=/usr/local/cassandra/conf"
LimitNOFILE=infinity
TimeoutStopSec=10min
Restart=on-failure
PIDFile=/var/run/cassandra/cassandra.pid
ExecStop=/usr/local/cassandra/bin/nodetool disablegossip
ExecStop=/usr/local/cassandra/bin/nodetool disablethrift
ExecStop=/usr/local/cassandra/bin/nodetool drain
ExecStop=-/usr/local/cassandra/bin/nodetool stopdaemon

On mars 28 2018, at 2:20 am, Anumod Mullachery  
wrote:
>
> Hi All ,
> I’ve installed Apache Cassandra( Tarball install )..
> But there is no way to see the Cassandra status / stop option .
>
>
> Anyone has the Apache Cassandra start / stop script for Cassandra 3.12
> Can some one help on this ?
>
> Thanks,
> Anumod
> Mob-718-844-3841
> PA,USA
>
> Sent from my iPhone
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

RE: 答复: 答复: A node down every day in a 6 nodes cluster

[Kenneth Brotman]

If you think that will fix the problem, maybe you could add a little more 
memory to each machine as a short term fix.

 

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com] 
Sent: Wednesday, March 28, 2018 5:24 AM
To: user@cassandra.apache.org
Subject: 答复: 答复: 答复: A node down every day in a 6 nodes cluster

 

Yes ,we discussed and plan to figured out the data model issue and upgrade to 
3.11.3 version.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Kenneth Brotman  
发送时间: 2018年3月28日 20:16
收件人: user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

David, 

 

Did you figure out what to do about the data model problem?  It could be that 
your data files finally grow to the point that the data model problem caused 
the Java heap space issue – in which case everything is actually working as 
it’s supposed to; You just have to fix the data model.

 

Kenneth Brotman

 

 

From: Kenneth Brotman [  
mailto:kenbrot...@yahoo.com] 
Sent: Wednesday, March 28, 2018 4:46 AM
To: 'user@cassandra.apache.org'
Subject: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

Was any change to hardware done around the time the problem started ?

Was any change to the client software done around the time the problem started?

Was any change to the database schema done around the time the problem started?

 

Kenneth Brotman

 

From: Xiangfei Ni [  
mailto:xiangfei...@cm-dt.com] 
Sent: Wednesday, March 28, 2018 4:40 AM
To:   user@cassandra.apache.org
Subject: 答复: 答复: 答复: A node down every day in a 6 nodes cluster

 

Hi Kenneth,

The cluster has been running for 4 months,

The problem occurred from last week,

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Kenneth Brotman <  
kenbrot...@yahoo.com.INVALID> 
发送时间: 2018年3月28日 19:34
收件人:   user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

David,

 

How long has the cluster been operating?

How long has the problem been occurring?

 

Kenneth Brotman

 

From: Jeff Jirsa [  mailto:jji...@gmail.com] 
Sent: Tuesday, March 27, 2018 7:00 PM
To: Xiangfei Ni
Cc:   user@cassandra.apache.org
Subject: Re: 答复: 答复: A node down every day in a 6 nodes cluster

 

 

java.langOutOfMemoryError: Java heap space

 

 

You’re oom’ ing 

 

-- 

Jeff Jirsa

 


On Mar 27, 2018, at 6:45 PM, Xiangfei Ni  wrote:

Hi Jeff,

Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Jeff Jirsa <  jji...@gmail.com> 
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni <  xiangfei...@cm-dt.com>
抄送:   user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

 

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

 

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of 

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).
 
 

 

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

Hi Jeff,

I need to restart the node manually every time,only one node has this 
problem.

I have attached the nodetool output,thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob:   +86 13797007811|Tel:  
 + 86 27 5024 2516

 

发件人: Jeff Jirsa <  jji...@gmail.com> 
发送时间: 2018年3月27日 11:03
收件人:   user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

 

That warning isn’t sufficient to understand why the node is going down

 

 

Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

 

Are the nodes coming up on their own? Or are you restarting them?

 

Paste the output of nodetool tpstats and nodetool cfstats

答复: 答复: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Yes ,we discussed and plan to figured out the data model issue and upgrade to 
3.11.3 version.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Kenneth Brotman 
发送时间: 2018年3月28日 20:16
收件人: user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

David,

Did you figure out what to do about the data model problem?  It could be that 
your data files finally grow to the point that the data model problem caused 
the Java heap space issue – in which case everything is actually working as 
it’s supposed to; You just have to fix the data model.

Kenneth Brotman


From: Kenneth Brotman [mailto:kenbrot...@yahoo.com]
Sent: Wednesday, March 28, 2018 4:46 AM
To: 'user@cassandra.apache.org'
Subject: RE: 答复: 答复: A node down every day in a 6 nodes cluster

Was any change to hardware done around the time the problem started ?
Was any change to the client software done around the time the problem started?
Was any change to the database schema done around the time the problem started?

Kenneth Brotman

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com]
Sent: Wednesday, March 28, 2018 4:40 AM
To: user@cassandra.apache.org
Subject: 答复: 答复: 答复: A node down every day in a 6 nodes cluster

Hi Kenneth,
The cluster has been running for 4 months,
The problem occurred from last week,

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Kenneth Brotman 
>
发送时间: 2018年3月28日 19:34
收件人: user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

David,

How long has the cluster been operating?
How long has the problem been occurring?

Kenneth Brotman

From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Tuesday, March 27, 2018 7:00 PM
To: Xiangfei Ni
Cc: user@cassandra.apache.org
Subject: Re: 答复: 答复: A node down every day in a 6 nodes cluster




java.langOutOfMemoryError: Java heap space





You’re oom’ ing

--
Jeff Jirsa


On Mar 27, 2018, at 6:45 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni >
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).





On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 

RE: 答复: 答复: A node down every day in a 6 nodes cluster

[Kenneth Brotman]

David, 

 

Did you figure out what to do about the data model problem?  It could be that 
your data files finally grow to the point that the data model problem caused 
the Java heap space issue – in which case everything is actually working as 
it’s supposed to; You just have to fix the data model.

 

Kenneth Brotman

 

 

From: Kenneth Brotman [mailto:kenbrot...@yahoo.com] 
Sent: Wednesday, March 28, 2018 4:46 AM
To: 'user@cassandra.apache.org'
Subject: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

Was any change to hardware done around the time the problem started ?

Was any change to the client software done around the time the problem started?

Was any change to the database schema done around the time the problem started?

 

Kenneth Brotman

 

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com] 
Sent: Wednesday, March 28, 2018 4:40 AM
To: user@cassandra.apache.org
Subject: 答复: 答复: 答复: A node down every day in a 6 nodes cluster

 

Hi Kenneth,

The cluster has been running for 4 months,

The problem occurred from last week,

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Kenneth Brotman  
发送时间: 2018年3月28日 19:34
收件人: user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

David,

 

How long has the cluster been operating?

How long has the problem been occurring?

 

Kenneth Brotman

 

From: Jeff Jirsa [mailto:jji...@gmail.com] 
Sent: Tuesday, March 27, 2018 7:00 PM
To: Xiangfei Ni
Cc: user@cassandra.apache.org
Subject: Re: 答复: 答复: A node down every day in a 6 nodes cluster

 

 

java.langOutOfMemoryError: Java heap space

 

 

You’re oom’ ing 

 

-- 

Jeff Jirsa

 


On Mar 27, 2018, at 6:45 PM, Xiangfei Ni  wrote:

Hi Jeff,

Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

 

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

 

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of 

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).
 
 

 

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

Hi Jeff,

I need to restart the node manually every time,only one node has this 
problem.

I have attached the nodetool output,thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811  |Tel: + 86 27 5024 2516 
 

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

 

That warning isn’t sufficient to understand why the node is going down

 

 

Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

 

Are the nodes coming up on their own? Or are you restarting them?

 

Paste the output of nodetool tpstats and nodetool cfstats

 

 

 

-- 

Jeff Jirsa

 


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:

Hi Cassandra experts,

  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,

  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:

WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 

ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query

com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]

at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-180.jar:na]

at 

RE: 答复: 答复: A node down every day in a 6 nodes cluster

[Kenneth Brotman]

Was any change to hardware done around the time the problem started ?

Was any change to the client software done around the time the problem started?

Was any change to the database schema done around the time the problem started?

 

Kenneth Brotman

 

From: Xiangfei Ni [mailto:xiangfei...@cm-dt.com] 
Sent: Wednesday, March 28, 2018 4:40 AM
To: user@cassandra.apache.org
Subject: 答复: 答复: 答复: A node down every day in a 6 nodes cluster

 

Hi Kenneth,

The cluster has been running for 4 months,

The problem occurred from last week,

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Kenneth Brotman  
发送时间: 2018年3月28日 19:34
收件人: user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

 

David,

 

How long has the cluster been operating?

How long has the problem been occurring?

 

Kenneth Brotman

 

From: Jeff Jirsa [mailto:jji...@gmail.com] 
Sent: Tuesday, March 27, 2018 7:00 PM
To: Xiangfei Ni
Cc: user@cassandra.apache.org
Subject: Re: 答复: 答复: A node down every day in a 6 nodes cluster

 

 

java.langOutOfMemoryError: Java heap space

 

 

You’re oom’ ing 

 

-- 

Jeff Jirsa

 


On Mar 27, 2018, at 6:45 PM, Xiangfei Ni  wrote:

Hi Jeff,

Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

 

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

 

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of 

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).
 
 

 

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

Hi Jeff,

I need to restart the node manually every time,only one node has this 
problem.

I have attached the nodetool output,thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811  |Tel: + 86 27 5024 2516 
 

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

 

That warning isn’t sufficient to understand why the node is going down

 

 

Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

 

Are the nodes coming up on their own? Or are you restarting them?

 

Paste the output of nodetool tpstats and nodetool cfstats

 

 

 

-- 

Jeff Jirsa

 


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:

Hi Cassandra experts,

  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,

  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:

WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 

ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query

com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]

at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-180.jar:na]

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]

at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]

at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]

答复: 答复: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Hi Kenneth,
The cluster has been running for 4 months,
The problem occurred from last week,

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Kenneth Brotman 
发送时间: 2018年3月28日 19:34
收件人: user@cassandra.apache.org
主题: RE: 答复: 答复: A node down every day in a 6 nodes cluster

David,

How long has the cluster been operating?
How long has the problem been occurring?

Kenneth Brotman

From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Tuesday, March 27, 2018 7:00 PM
To: Xiangfei Ni
Cc: user@cassandra.apache.org
Subject: Re: 答复: 答复: A node down every day in a 6 nodes cluster




java.langOutOfMemoryError: Java heap space





You’re oom’ ing

--
Jeff Jirsa


On Mar 27, 2018, at 6:45 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni >
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).





On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-180.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]
at 

RE: 答复: 答复: A node down every day in a 6 nodes cluster

[Kenneth Brotman]

David,

 

How long has the cluster been operating?

How long has the problem been occurring?

 

Kenneth Brotman

 

From: Jeff Jirsa [mailto:jji...@gmail.com] 
Sent: Tuesday, March 27, 2018 7:00 PM
To: Xiangfei Ni
Cc: user@cassandra.apache.org
Subject: Re: 答复: 答复: A node down every day in a 6 nodes cluster

 

 

java.langOutOfMemoryError: Java heap space

 

 

You’re oom’ ing 

 

-- 

Jeff Jirsa

 


On Mar 27, 2018, at 6:45 PM, Xiangfei Ni  wrote:

Hi Jeff,

Today another node was shutdown,I have attached the exception log 
file,could you please help to analyze?Thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni 
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster

 

Only one node having the problem is suspicious. May be that your application is 
improperly pooling connections, or you have a hardware problem.

 

I dont see anything in nodetool that explains it, though you certainly have a 
data model likely to cause problems over time (the cardinality of 

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such 
that you have very wide partitions and it'll be difficult to read).
 
 

 

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

Hi Jeff,

I need to restart the node manually every time,only one node has this 
problem.

I have attached the nodetool output,thanks.

 

Best Regards, 

 

倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811  |Tel: + 86 27 5024 2516 
 

 

发件人: Jeff Jirsa  
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

 

That warning isn’t sufficient to understand why the node is going down

 

 

Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

 

Are the nodes coming up on their own? Or are you restarting them?

 

Paste the output of nodetool tpstats and nodetool cfstats

 

 

 

-- 

Jeff Jirsa

 


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:

Hi Cassandra experts,

  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,

  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:

WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 

ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query

com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]

at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-180.jar:na]

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]

at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]

at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
 ~[apache-cassandra-3.9.jar:3.9]

at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
 ~[apache-cassandra-3.9.jar:3.9]

at