Re: 回复： data loss in different DC
How often are you running repairs? -- Jacob Shadix On Thu, Sep 28, 2017 at 7:53 AM, Reynald Bourtembourg < reynald.bourtembo...@esrf.fr> wrote: > Hi, > > You can write with CL=EACH_QUORUM and read with CL=LOCAL_QUORUM to get > strong consistency. > > Kind regards, > Reynald > > > On 28/09/2017 13:46, Peng Xiao wrote: > > even with CL=QUORUM,there is no guarantee to be sure to read the same data > in DC2,right? > then multi DCs looks make no sense? > > > -- 原始邮件 -- > *发件人:* "DuyHai Doan";<doanduy...@gmail.com> <doanduy...@gmail.com>; > *发送时间:* 2017年9月28日(星期四) 下午5:45 > *收件人:* "user"<user@cassandra.apache.org> <user@cassandra.apache.org>; > *主题:* Re: data loss in different DC > > If you're writing into DC1 with CL = LOCAL_xxx, there is no guarantee to > be sure to read the same data in DC2. Only repair will help you > > On Thu, Sep 28, 2017 at 11:41 AM, Peng Xiao <2535...@qq.com> wrote: > >> Dear All, >> >> We have a cluster with one DC1:RF=3,another DC DC2:RF=1 only for ETL,but >> we found that sometimes we can query records in DC1,while not able not find >> the same record in DC2 with local_quorum.How it happens? >> Could anyone please advise? >> looks we can only run repair to fix it. >> >> Thanks, >> Peng Xiao >> > > >
Re: Understanding Messages in the Debug.log
check out this thread - https://www.mail-archive.com/user@cassandra.apache.org/msg52313.html -- Jacob Shadix On Thu, Sep 21, 2017 at 4:56 AM, Jonathan Baynes < jonathan.bay...@tradeweb.com> wrote: > Good Morning Community, > > > > Can anyone help me understand what these messages (and I have a lot of > them in the log) mean, they must be a symptom of something but I am unsure > how to starting working this out. > > > > I’m using C* 3.0.11, CQL 5.0.1. > > > > I have 6 node ring over 2 DC’s (3 in each DC) each of the nodes has this > message in the log. > > > > The message in the debug log is > > DEBUG [GossipStage:1] 2017-09-21 09:19:52,627 FailureDetector.java:456 - > Ignoring interval time of 2000275419 > > > > > > There are so many of these messages, I’d like to find the root cause of > this, if anyone could help that would be great. Or if they can be ignored, > how do I supress these? > > > > Thanks > > J > > > > *Jonathan Baynes* > > DBA > Tradeweb Europe Limited > > Moor Place • 1 Fore Street Avenue > <https://maps.google.com/?q=1+Fore+Street+Avenue%C2%A0%C2%A0%E2%80%A2%C2%A0%C2%A0London+EC2Y+9DT=gmail=g> > • London EC2Y 9DT > <https://maps.google.com/?q=1+Fore+Street+Avenue%C2%A0%C2%A0%E2%80%A2%C2%A0%C2%A0London+EC2Y+9DT=gmail=g> > P +44 (0)20 77760988 • F +44 (0)20 7776 3201 • M +44 (0) xx > > jonathan.bay...@tradeweb.com > > > > [image: cid:image001.jpg@01CD26AD.4165F110] <http://www.tradeweb.com/> > follow us: *[image: cid:image002.jpg@01CD26AD.4165F110]* > <https://www.linkedin.com/company/tradeweb?trk=top_nav_home> [image: > cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb> > > — > > A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for > electronic fixed income, derivatives and ETF trading > > > > > > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) > please notify the sender immediately and destroy it. Any unauthorized > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. Tradeweb reserves the right to monitor all e-mail > communications through its networks. If you do not wish to receive > marketing emails about our products / services, please let us know by > contacting us, either by email at contac...@tradeweb.com or by writing to > us at the registered office of Tradeweb in the UK, which is: Tradeweb > Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y > 9DT > <https://maps.google.com/?q=number+3912826),+1+Fore+Street+Avenue+London+EC2Y+9DT=gmail=g>. > To see our privacy policy, visit our website @ www.tradeweb.com. >
Re: Timeout while trying to acquire available connection
How many client connections are hitting your cluster? Have you looked at tuning connection pool? https://github.com/datastax/java-driver/tree/3.x/manual/pooling -- Jacob Shadix On Mon, Jun 26, 2017 at 2:50 PM, Ivan Iliev <ivan.iliev.il...@gmail.com> wrote: > Hello everyone! > > I am seeing recent behavior of apps being not able to communicate to the > Cassandra cluster with the following errors: > > All host(s) tried for query failed (tried: cassandra2-test:9042 > (com.datastax.driver.core.exceptions.DriverException: Timeout while > trying to acquire available connection (you may want to increase the driver > number of per-host connections)), cassandra3-test:9042 > (com.datastax.driver.core.exceptions.TransportException: > [cassandra3-test/] Connection has been closed), cassandra1-test:9042 > (com.datastax.driver.core.exceptions.TransportException: > [cassandra1-test/] Connection has been closed)) > > I have enabled start_native_transport already but this does not fix the > problem. > > I would appreciate any feedback on the problem. > > Thanks, > Ivan >
Re: [Cassandra 3.0.9] In Memory table
no, in-memory is only available in DataStax Enterprise 4.0+ -- Jacob Shadix On Thu, Apr 20, 2017 at 3:00 AM, Abhishek Kumar Maheshwari < abhishek.maheshw...@timesinternet.in> wrote: > Hi All, > > > > As Datastax Cassandra version provide a in memory table. Can we achieve > same thing in apache Cassandra? > > > > http://docs.datastax.com/en/archived/datastax_enterprise/ > 4.6/datastax_enterprise/inMemory.html > > > > > > > > > > *Thanks & Regards,* > *Abhishek Kumar Maheshwari* > *+91- 805591 (Mobile)* > > Times Internet Ltd. | A Times of India Group Company > > FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA > > *P** Please do not print this email unless it is absolutely necessary. > Spread environmental awareness.* > > > Attend LEAP Edtech <http://tlabs.in/edtech>, India's largest EdTech > Summit focused on forging partnerships between different ecosystem players. > Register with Discount code LPTBS <https://goo.gl/9RMZtv>to avail 50% > discount on event tickets. >
Re: Multiple nodes decommission
It's still not recommended to start at the same time. Stagger by 2 minutes is what the following documentation suggests; along with additional steps. re. version 2.1 https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html -- Jacob Shadix On Wed, Apr 12, 2017 at 1:48 PM, Vlad <qa23d-...@yahoo.com> wrote: > But it seems OK to add multiple nodes at once, right? > > > On Tuesday, April 11, 2017 8:38 PM, Jacob Shadix <jacobsha...@gmail.com> > wrote: > > > Right! Another reason why I just stick with sequential decommissions. > Maybe someone here could shed some light on what happens under the covers > if parallel decommissions are kicked off. > > -- Jacob Shadix > > On Tue, Apr 11, 2017 at 12:55 PM, benjamin roth <brs...@gmail.com> wrote: > > I did not test it but I'd bet that parallel decommision will lead to > inconsistencies. > Each decommission results in range movements and range reassignments which > becomes effective after a successful decommission. > If you start several decommissions at once, I guess the calculated > reassignments are invalid for at least one node after the first node > finished the decommission process. > > I hope someone will correct me if i am wrong. > > 2017-04-11 18:43 GMT+02:00 Jacob Shadix <jacobsha...@gmail.com>: > > Are you using vnodes? I typically do one-by-one as the decommission will > create additional load/network activity streaming data to the other nodes > as the token ranges are reassigned. > > -- Jacob Shadix > > On Sat, Apr 8, 2017 at 10:55 AM, Vlad <qa23d-...@yahoo.com> wrote: > > Hi, > > how multiple nodes should be decommissioned by "nodetool decommission"- > one by one or in parallel ? > > Thanks. > > > > > > >
Re: Multiple nodes decommission
Right! Another reason why I just stick with sequential decommissions. Maybe someone here could shed some light on what happens under the covers if parallel decommissions are kicked off. -- Jacob Shadix On Tue, Apr 11, 2017 at 12:55 PM, benjamin roth <brs...@gmail.com> wrote: > I did not test it but I'd bet that parallel decommision will lead to > inconsistencies. > Each decommission results in range movements and range reassignments which > becomes effective after a successful decommission. > If you start several decommissions at once, I guess the calculated > reassignments are invalid for at least one node after the first node > finished the decommission process. > > I hope someone will correct me if i am wrong. > > 2017-04-11 18:43 GMT+02:00 Jacob Shadix <jacobsha...@gmail.com>: > >> Are you using vnodes? I typically do one-by-one as the decommission will >> create additional load/network activity streaming data to the other nodes >> as the token ranges are reassigned. >> >> -- Jacob Shadix >> >> On Sat, Apr 8, 2017 at 10:55 AM, Vlad <qa23d-...@yahoo.com> wrote: >> >>> Hi, >>> >>> how multiple nodes should be decommissioned by "nodetool decommission"- >>> one by one or in parallel ? >>> >>> Thanks. >>> >> >> >
Re: Multiple nodes decommission
Are you using vnodes? I typically do one-by-one as the decommission will create additional load/network activity streaming data to the other nodes as the token ranges are reassigned. -- Jacob Shadix On Sat, Apr 8, 2017 at 10:55 AM, Vlad <qa23d-...@yahoo.com> wrote: > Hi, > > how multiple nodes should be decommissioned by "nodetool decommission"- > one by one or in parallel ? > > Thanks. >
Re: cassandra node stops streaming data during nodetool rebuild
I don't see an issue with the size of the data / node. You can attempt the rebuild again and play around with throughput if your network can handle it. It can be changed on-the-fly with nodetool: nodetool setstreamthroughput This article is also worth a read - https://support.datastax.com/hc/en-us/articles/205409646-How-to-performance-tune-data-streaming-activities-like-repair-and-bootstrap -- Jacob Shadix On Fri, Apr 7, 2017 at 9:23 AM, Roland Otta <roland.o...@willhaben.at> wrote: > good point! > > on the source side i can see the following error > > ERROR [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > StreamSession.java:529 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Streaming error occurred on session with peer 10.192.116.1 through 192.168. > 0.114 > org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe > at > org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:145) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.compress. > CompressedStreamWriter.lambda$write$0(CompressedStreamWriter.java:90) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus. > applyToChannel(BufferedDataOutputStreamPlus.java:350) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.compress. > CompressedStreamWriter.write(CompressedStreamWriter.java:90) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages. > OutgoingFileMessage.serialize(OutgoingFileMessage.java:91) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1. > serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1. > serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages. > StreamMessage.serialize(StreamMessage.java:48) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.ConnectionHandler$ > OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.ConnectionHandler$ > OutgoingMessageHandler.run(ConnectionHandler.java:342) > ~[apache-cassandra-3.7.jar:3.7] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] > Caused by: java.io.IOException: Broken pipe > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > ~[na:1.8.0_77] > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > ~[na:1.8.0_77] > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > ~[na:1.8.0_77] > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > ~[na:1.8.0_77] > at > org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:141) > ~[apache-cassandra-3.7.jar:3.7] > ... 10 common frames omitted > DEBUG [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > ConnectionHandler.java:110 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Closing stream connection handler on /10.192.116.1 > INFO [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > StreamResultFuture.java:187 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Session with /10.192.116.1 is complete > WARN [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > StreamResultFuture.java:214 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Stream failed > > > the dataset is approx 300GB / Node. > > does that mean that cassandra does not try to reconnect (for streaming) in > case of short network dropouts? > > On Fri, 2017-04-07 at 08:53 -0400, Jacob Shadix wrote: > > Did you look at the logs on the source DC as well? How big is the dataset? > > -- Jacob Shadix > > On Fri, Apr 7, 2017 at 7:16 AM, Roland Otta <roland.o...@willhaben.at> > wrote: > > Hi! > > we are on 3.7. > > we have some debug messages ... but i guess they are not related to that > issue > DEBUG [GossipStage:1] 2017-04-07 13:11:00,440 FailureDetector.java:456 - > Ignoring interval time of 2002469610 for /192.168.0.27 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2598593732 for /10.192.116.4 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002612298 for /10.192.116.5 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002660534 for /10.192.116.9 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignorin
Re: cassandra node stops streaming data during nodetool rebuild
Did you look at the logs on the source DC as well? How big is the dataset? -- Jacob Shadix On Fri, Apr 7, 2017 at 7:16 AM, Roland Otta <roland.o...@willhaben.at> wrote: > Hi! > > we are on 3.7. > > we have some debug messages ... but i guess they are not related to that > issue > DEBUG [GossipStage:1] 2017-04-07 13:11:00,440 FailureDetector.java:456 - > Ignoring interval time of 2002469610 for /192.168.0.27 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2598593732 for /10.192.116.4 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002612298 for /10.192.116.5 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002660534 for /10.192.116.9 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027212880 for /10.192.116.3 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027279042 for /192.168.0.188 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027313992 for /10.192.116.10 > > beside that the debug.log is clean > > all the mentioned cassandra.yml parameters are the shipped defaults ( > streaming_socket_timeout_in_ms does not exist at all in my cassandra.yml) > i also checked the pending compactions. there are no pending compactions > at the moment. > > bg - roland otta > > On Fri, 2017-04-07 at 06:47 -0400, Jacob Shadix wrote: > > What version are you running? Do you see any errors in the system.log > (SocketTimeout, for instance)? > > And what values do you have for the following in cassandra.yaml: > - - stream_throughput_outbound_megabits_per_sec > - - compaction_throughput_mb_per_sec > - - streaming_socket_timeout_in_ms > > -- Jacob Shadix > > On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta <roland.o...@willhaben.at> > wrote: > > hi, > > we are trying to setup a new datacenter and are initalizing the data > with nodetool rebuild. > > after some hours it seems that the node stopped streaming (at least > there is no more streaming traffic on the network interface). > > nodetool netstats shows that the streaming is still in progress > > Mode: NORMAL > Bootstrap 6918dc90-1ad6-11e7-9f16-51230e2be4e9 > Rebuild 41606030-1ad9-11e7-9f16-51230e2be4e9 > /192.168.0.26 > Receiving 257 files, 145444246572 bytes total. Already received > 1 files, 1744027 bytes total > bds/adcounter_total 76456/47310255 bytes(0%) received from > idx:0/192.168.0.26 > bds/upselling_event 1667571/1667571 bytes(100%) received > from idx:0/192.168.0.26 > /192.168.0.188 > /192.168.0.27 > Receiving 169 files, 79355302464 bytes total. Already received > 1 files, 81585975 bytes total > bds/ad_event_history 81585975/81585975 bytes(100%) received > from idx:0/192.168.0.27 > /192.168.0.189 > Receiving 140 files, 19673034809 bytes total. Already received > 1 files, 5996604 bytes total > bds/adcounter_per_day 5956840/42259846 bytes(14%) received > from idx:0/192.168.0.189 > bds/user_event 39764/39764 bytes(100%) received from > idx:0/192.168.0.189 > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool NameActive Pending Completed Dropped > Large messages n/a 2 3 0 > Small messages n/a 0 68632465 0 > Gossip messages n/a 0 217661 0 > > > > it is in that state for approx 15 hours now > > does it make sense waiting for the streaming to finish or do i have to > restart the node, discard data and restart the rebuild? > > >
Re: cassandra node stops streaming data during nodetool rebuild
What version are you running? Do you see any errors in the system.log (SocketTimeout, for instance)? And what values do you have for the following in cassandra.yaml: - - stream_throughput_outbound_megabits_per_sec - - compaction_throughput_mb_per_sec - - streaming_socket_timeout_in_ms -- Jacob Shadix On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta <roland.o...@willhaben.at> wrote: > hi, > > we are trying to setup a new datacenter and are initalizing the data > with nodetool rebuild. > > after some hours it seems that the node stopped streaming (at least > there is no more streaming traffic on the network interface). > > nodetool netstats shows that the streaming is still in progress > > Mode: NORMAL > Bootstrap 6918dc90-1ad6-11e7-9f16-51230e2be4e9 > Rebuild 41606030-1ad9-11e7-9f16-51230e2be4e9 > /192.168.0.26 > Receiving 257 files, 145444246572 bytes total. Already received > 1 files, 1744027 bytes total > bds/adcounter_total 76456/47310255 bytes(0%) received from > idx:0/192.168.0.26 > bds/upselling_event 1667571/1667571 bytes(100%) received > from idx:0/192.168.0.26 > /192.168.0.188 > /192.168.0.27 > Receiving 169 files, 79355302464 bytes total. Already received > 1 files, 81585975 bytes total > bds/ad_event_history 81585975/81585975 bytes(100%) received > from idx:0/192.168.0.27 > /192.168.0.189 > Receiving 140 files, 19673034809 bytes total. Already received > 1 files, 5996604 bytes total > bds/adcounter_per_day 5956840/42259846 bytes(14%) received > from idx:0/192.168.0.189 > bds/user_event 39764/39764 bytes(100%) received from > idx:0/192.168.0.189 > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool NameActive Pending Completed Dropped > Large messages n/a 2 3 0 > Small messages n/a 0 68632465 0 > Gossip messages n/a 0 217661 0 > > > > it is in that state for approx 15 hours now > > does it make sense waiting for the streaming to finish or do i have to > restart the node, discard data and restart the rebuild? >
Re: upgrade to Cassandra 3.0.12
I installed 2.7 and updated PYTHONPATH, but it is still not finding the newer version. -- Jacob Shadix On Tue, Apr 4, 2017 at 11:22 AM, Voytek Jarnot <voytek.jar...@gmail.com> wrote: > Multiple versions of python can coexist, the cqlsh shell script will > attempt to execute via a python2.7 executable if it finds one. > > On Tue, Apr 4, 2017 at 9:49 AM, Jacob Shadix <jacobsha...@gmail.com> > wrote: > >> I've recently upgraded to 3.0.12 and unable to run CQLSH. >> No appropriate python interpreter found. >> >> The current python version installed is 2.6.6. I realize I need to >> upgrade to 2.7.12 at least, but I also cannot remove the 2.6.6 version. Are >> there any recommendations for installing a newer version of python >> alongside the older release? >> >> -- Jacob Shadix >> > >
upgrade to Cassandra 3.0.12
I've recently upgraded to 3.0.12 and unable to run CQLSH. No appropriate python interpreter found. The current python version installed is 2.6.6. I realize I need to upgrade to 2.7.12 at least, but I also cannot remove the 2.6.6 version. Are there any recommendations for installing a newer version of python alongside the older release? -- Jacob Shadix
Internal Security - Authentication & Authorization
I'm looking for a deeper understanding of how Cassandra interacts with the system_auth keyspace to authenticate/authorize users. Here is what I have pieced together. Please let me know if I am on the right track. A user attempts to connect to Cassandra. Cassandra checks against system_auth for that user @ LOCAL_ONE - - If the user exists, a connection is established. When CQL is executed, C* again checks system_auth for that user @ LOCAL_ONE to determine if it has the correct privileges to perform the CQL. If so, it executes the CQL and the permissions are stored in a cache. During the cache validity timeframe, future requests for ANY user stored in the cache do not require a lookup against system_auth. After the cache validity runs out, any new requests will require a lookup against system_auth. -- Jacob Shadix
Re: OpsCenter w/SSL
If i start the agent on the cluster with encryption, I see lots of these messages in the C* logs - Unexpected exception during request; channel io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record And get an error connecting to the cluster from opscenterd. -- Jacob Shadix On Wed, Feb 22, 2017 at 1:13 PM, Bulat Shakirzyanov < bulat.shakirzya...@datastax.com> wrote: > Hi Jacob, > > OpsCenter supports simultaneous management of Cassandra clusters both with > and without client-to-node encryption enabled. > > The only time you'd need to use SSL everywhere, is when encrypting > OpsCenter Daemon to OpsCenter Agents connections. In that case, you have to > make sure all OpsCenter Agents connecting to a given OpsCenter Daemon use > SSL even if those agents belong to different Cassandra clusters. > > > On Wed, Feb 22, 2017 at 11:18 AM, Jacob Shadix <jacobsha...@gmail.com> > wrote: > >> I have OpsCenter 6.0.7 setup managing multiple clusters. Only one of >> those clusters has encryption turned on (both node-to-node and >> client-to-node). In order to manage that cluster through OpsCenter, do all >> subsequent clusters have to have encryption turned on? >> >> -- Jacob Shadix >> > > > > -- > Cheers, > Bulat Shakirzyanov | @avalanche123 <https://twitter.com/avalanche123> >
OpsCenter w/SSL
I have OpsCenter 6.0.7 setup managing multiple clusters. Only one of those clusters has encryption turned on (both node-to-node and client-to-node). In order to manage that cluster through OpsCenter, do all subsequent clusters have to have encryption turned on? -- Jacob Shadix
Re: UnknownColumnFamilyException after removing all Cassandra data
The node will not bootstrap if it is listed as a seed node. -- Jacob Shadix On Tue, Feb 7, 2017 at 12:16 PM, Simone Franzini <captainfr...@gmail.com> wrote: > To further add to my previous answer, the node in question is a seed node, > so it did not bootstrap. > Should I remove it from the list of seed nodes and then try to restart it? > > Simone Franzini, PhD > > http://www.linkedin.com/in/simonefranzini > > On Tue, Feb 7, 2017 at 9:43 AM, Simone Franzini <captainfr...@gmail.com> > wrote: > >> This is exactly what I did on the second node. If this is not the correct >> / best procedure to adopt in these cases, please advise: >> >> 1. Removed all the data, including the system table (rm -rf data/ >> commitlog/ saved_caches). >> 2. Configured the node to replace itself, by adding the following line to >> cassandra-env.sh: JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=> own IP address>" >> 3. Start the node. >> >> Noticeably, I did not do nodetool decommission or removenode. Is that the >> recommended approach? >> >> Given what I did, I am mystified as to what the problem is. If I query >> the system.schema_columnfamilies on the affected node, all CF IDs are >> there. Same goes for the only other node that is currently up. Also, the >> other node that is currently up has data for all those CF IDs in the data >> folder. >> >> >> Simone Franzini, PhD >> >> http://www.linkedin.com/in/simonefranzini >> >> On Tue, Feb 7, 2017 at 5:39 AM, kurt greaves <k...@instaclustr.com> >> wrote: >> >>> The node is trying to communicate with another node, potentially >>> streaming data, and is receiving files/data for an "unknown column family". >>> That is, it doesn't know about the CF with the id >>> e36415b6-95a7-368c-9ac0-ae0ac774863d. >>> If you deleted some columnfamilies but not all the system keyspace and >>> restarted the node I'd expect this error to occur. Or I suppose if you >>> didn't decommission the node properly before blowing the data away and >>> restarting. >>> >>> You'll have to give us more information on what your exact steps were on >>> this 2nd node: >>> >>> When you say deleted all Cassandra data, did this include the system >>> tables? Were your steps to delete all the data and then just restart the >>> node? Did you remove the node from the cluster prior to deleting the data >>> and restarting it (nodetool decommission/removenode? Did the node rejoin >>> the cluster or did it have to bootstrap? >>> >>> >>> >> >
Re: collecting metrics at column-family level
I found a couple of metrics that should suffice for #2,3 - - > TBL: Live Disk Used & TBL: Local Write Latency. 1. count of number of records inserted within given timeframe 2. data growth 3. write latency -- Jacob Shadix On Mon, Jan 9, 2017 at 2:29 PM, Jacob Shadix <jacobsha...@gmail.com> wrote: > Is it possible to report on the following metrics at table level? > >1. count of number of records inserted within given timeframe >2. data growth >3. write latency > > -- Jacob Shadix >
collecting metrics at column-family level
Is it possible to report on the following metrics at table level? 1. count of number of records inserted within given timeframe 2. data growth 3. write latency -- Jacob Shadix
Python Upgrade to 2.7
I am running Cassandra 2.1.14. Upgraded to Python 2.7 from 2.6.6 and getting the following error with CQLSH. --- Python Cassandra driver not installed, or not on PYTHONPATH. You might try "pip install cassandra-driver". Python: /opt/isv/python27/bin/python Error: can't decompress data; zlib not available --- What am I missing? -- Jacob Shadix
Cipher Suite Cassandra 2.1.14 Encryption
I was testing client encryption w/cqlsh and get the following error when using TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 as the cipher. Any ideas why? Last error: _ssl.c:492: EOF occurred in violation of protocol")}) -- Jacob Shadix
Re: Re : Generic keystore when enabling SSL
I am interested if anyone has taken this approach to share the same keystore across all the nodes with the 3rd party root/intermediate CA existing only in the truststore. If so, please share your experience and lessons learned. Would this impact client-to-node encryption as the certificates used in internode would not have the hostnames represented in CN? -- Jacob Shadix On Wed, Sep 21, 2016 at 11:40 AM, sai krishnam raju potturi < pskraj...@gmail.com> wrote: > hi Evans; >rather than having one individual certificate for every node, we are > looking at getting one Comodo wild-card certificate, and importing that > into the keystore. along with the intermediate CA provided by Comodo. As > far as the trust-store is concerned, we are looking at importing the > intermediate CA provided along with the signed wild-card cert by Comodo. > >So in this case we'll be having just one keystore (generic), and > truststore we'll be copying to all the nodes. We've run into issues > however, and are trying to iron that out. Interested to know if anybody in > the community has taken a similar approach. > >We are pretty much going on the lines of following post by LastPickle > http://thelastpickle.com/blog/2015/09/30/hardening-cassandra- > step-by-step-part-1-server-to-server.html. Instead of creating our own > CA, we are relying on Comodo. > > thanks > Sai > > On Wed, Sep 21, 2016 at 10:30 AM, Eric Evans <john.eric.ev...@gmail.com> > wrote: > >> On Tue, Sep 20, 2016 at 12:57 PM, sai krishnam raju potturi >> <pskraj...@gmail.com> wrote: >> > Due to the security policies in our company, we were asked to use 3rd >> party >> > signed certs. Since we'll require to manage 100's of individual certs, >> we >> > wanted to know if there is a work around with a generic keystore and >> > truststore. >> >> Can you explain what you mean by "generic keystore"? Are you looking >> to create keystores signed by a self-signed root CA (distributed via a >> truststore)? >> >> -- >> Eric Evans >> john.eric.ev...@gmail.com >> > >
