Re: Huge daily outbound network traffic
You could also run tcpdump to inspect the streams. Dinesh On Thursday, August 16, 2018, 1:11:47 PM PDT, Elliott Sims wrote: Since this is cross-node traffic, "nodetool netstats" during the high-traffic period should give you a better idea of what's being sent. On Thu, Aug 16, 2018 at 2:34 AM, Behnam B.Marandi wrote: In case of cronjobs, there is no jobs for that time period and I can see affect of jobs like backups and repairs but traffic that they cause is not comparable. Like 800MB comparing to 2GB. And for this case it is all outbound network on all 3 cluster nodes. On Thu, Aug 16, 2018 at 5:16 PM dinesh.jo...@yahoo.com.INVALID wrote: Since it is predictable, can you check the logs during that period? What do they say? Do you have a cron running on those hosts? Do all the nodes experience this issue? Dinesh On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi wrote: Actually I did. It seems this is a cross node traffic from one node to port 7000 (storage_port) of the other node. On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims wrote: Since it's at a consistent time, maybe just look at it with iftop to see where the traffic's going and what port it's coming from? On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi wrote: I don't have any external process or planed repair in that time period.In case of network, I can see outbound network on Cassandra node network interface but couldn't find any way to check the VPC network to make sure it is not going out of network. Maybe the only way is analysing VPC Flow Log.B. On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh wrote: Are you sure you don’t have an outside process that is doing an export , Spark job, non AWS managed backup process ? Is this network out from Cassandra or from the network? RahulOn Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: Hi,I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 instances with separate EBS volumes for root (gp2), data (gp2) and commitlog (io1).I get daily outbound traffic at a certain time everyday. As you can see in the attached screenshot, whiile my normal networkl oad hardly meets 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is less than 800MB.There is no repair or backup process giong on in that time window, so I am wondering where to look. Any idea? -- -- - To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Huge daily outbound network traffic
Since this is cross-node traffic, "nodetool netstats" during the high-traffic period should give you a better idea of what's being sent. On Thu, Aug 16, 2018 at 2:34 AM, Behnam B.Marandi < behnam.b.mara...@gmail.com> wrote: > In case of cronjobs, there is no jobs for that time period and I can see > affect of jobs like backups and repairs but traffic that they cause is not > comparable. Like 800MB comparing to 2GB. And for this case it is all > outbound network on all 3 cluster nodes. > > On Thu, Aug 16, 2018 at 5:16 PM dinesh.jo...@yahoo.com.INVALID < > dinesh.jo...@yahoo.com.invalid> wrote: > >> Since it is predictable, can you check the logs during that period? What >> do they say? Do you have a cron running on those hosts? Do all the nodes >> experience this issue? >> >> Dinesh >> >> >> On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi < >> behnam.b.mara...@gmail.com> wrote: >> >> >> Actually I did. It seems this is a cross node traffic from one node to >> port 7000 (storage_port) of the other node. >> >> On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims >> wrote: >> >> Since it's at a consistent time, maybe just look at it with iftop to see >> where the traffic's going and what port it's coming from? >> >> On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < >> behnam.b.mara...@gmail.com> wrote: >> >> I don't have any external process or planed repair in that time period. >> In case of network, I can see outbound network on Cassandra node network >> interface but couldn't find any way to check the VPC network to make sure >> it is not going out of network. Maybe the only way is analysing VPC Flow >> Log. >> B. >> >> On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh < >> rahul.xavier.si...@gmail.com> wrote: >> >> Are you sure you don’t have an outside process that is doing an export , >> Spark job, non AWS managed backup process ? >> >> Is this network out from Cassandra or from the network? >> >> >> Rahul >> On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: >> >> Hi, >> I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 >> instances with separate EBS volumes for root (gp2), data (gp2) and >> commitlog (io1). >> I get daily outbound traffic at a certain time everyday. As you can see >> in the attached screenshot, whiile my normal networkl oad hardly meets >> 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is >> less than 800MB. >> There is no repair or backup process giong on in that time window, so I >> am wondering where to look. Any idea? >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> >>
Java 11 support in Cassandra 4.0 + Early Testing and Feedback
Hey folks, As we start to get ready to feature freeze trunk for 4.0, it's going to be important to get a lot of community feedback. This is going to be a big release for a number of reasons. * Virtual tables. Finally a nice way of querying for system metrics & status * Streaming optimizations ( https://cassandra.apache.org/blog/2018/08/07/faster_streaming_in_cassandra.html ) * Groundwork for strongly consistent schema changes * Optimizations to internode communcation * Experimental support for Java 11 I (and many others) would like Cassandra to be rock solid on day one if its release. The best way to ensure that happens is if people provide feedback. One of the areas we're going to need a lot of feedback on is on how things work with Java 11, especially if you have a way of simulating a real world workload on a staging cluster. I've written up instructions here on how to start testing: http://thelastpickle.com/blog/2018/08/16/java11.html Java 11 hasn't been released yet, but that doesn't mean it's not a good time to test. Any bugs we can identify now will help us get to a stable release faster. If you rely on Cassandra for your business, please take some time to participate in the spirit of OSS by helping test & provide feedback to the team. Thanks everyone! --- Jon Haddad Principal Consultant, The Last Pickle
Re: JBOD disk failure
On Wed, Aug 15, 2018 at 3:23 AM kurt greaves wrote: > Yep. It might require a full node replace depending on what data is lost from > the system tables. In some cases you might be able to recover from partially > lost system info, but it's not a sure thing. Ugh, does it really just boil down to what part of `system` happens to be on the disk in question? In my mind, that makes the only sane operational procedure for a failed disk to be: "replace the entire node". IOW, I don't think we can realistically claim you can survive a failed a JBOD device if it relies on happenstance. > On Wed., 15 Aug. 2018, 17:55 Christian Lorenz, > wrote: >> >> Thank you for the answers. We are using the current version 3.11.3 So this >> one includes CASSANDRA-6696. >> >> So if I get this right, losing system tables will need a full node rebuild. >> Otherwise repair will get the node consistent again. > > [ ... ] -- Eric Evans john.eric.ev...@gmail.com - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Reading cardinality from Statistics.db failed
Dinesh, here is the ticket https://issues.apache.org/jira/browse/CASSANDRA-14647 On Thu, Aug 16, 2018 at 10:13 AM dinesh.jo...@yahoo.com.INVALID wrote: > Vitali, > > It doesn't look like there is an existing Jira. It would be helpful if you > could create one with as much information as possible. Can you reduce this > issue to a short, repeatable set of steps that we can reproduce? That'll be > helpful to debug this problem. > > Dinesh > On Wednesday, August 15, 2018, 1:07:21 AM PDT, Vitali Dyachuk < > vdjat...@gmail.com> wrote: > > > I've upgraded to 3.0.17 and the issue is still there, Is there a jira > ticket for that bug or should i create one? > > On Wed, Jul 25, 2018 at 2:57 PM Vitali Dyachuk wrote: > > I'm using 3.0.15. I see that there is some fix for sstable metadata in > 3.0.16 https://issues.apache.org/jira/browse/CASSANDRA-14217 - is that a > fix for "reading cardinalyti from statistics.db" ? > > > On Wed, Jul 25, 2018 at 1:02 PM Hannu Kröger wrote: > > What version of Cassandra are you running? There is a bug in 3.10.0 and > certain 3.0.x that occurs in certain conditions and corrupts that file. > > Hannu > > Vitali Dyachuk kirjoitti 25.7.2018 kello 10.48: > > Hi, > I have noticed in the cassandra system.log that there is some issue with > sstable metadata, the messages says: > WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading > cardinality from Statistics.db failed for > /opt/data/disk5/data/keyspace/table/mc-big-Data.db > Although there is no such file. The message has appeared after i've > changed the compaction strategy from SizeTiered to Leveled. > Currently i'm running nodetool scrub to rebuilt the sstable, and it takes > a lot of time to scrub all sstables. > Reading the code it is said that if this metada is broken, then estimating > the keys will be done using index summary. How expensive it is ? > > https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L245 > > The main question is why has this happened? > > Thanks, > Vitali Djatsuk. > >
Re: Huge daily outbound network traffic
In case of cronjobs, there is no jobs for that time period and I can see affect of jobs like backups and repairs but traffic that they cause is not comparable. Like 800MB comparing to 2GB. And for this case it is all outbound network on all 3 cluster nodes. On Thu, Aug 16, 2018 at 5:16 PM dinesh.jo...@yahoo.com.INVALID wrote: > Since it is predictable, can you check the logs during that period? What > do they say? Do you have a cron running on those hosts? Do all the nodes > experience this issue? > > Dinesh > > > On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi < > behnam.b.mara...@gmail.com> wrote: > > > Actually I did. It seems this is a cross node traffic from one node to > port 7000 (storage_port) of the other node. > > On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims > wrote: > > Since it's at a consistent time, maybe just look at it with iftop to see > where the traffic's going and what port it's coming from? > > On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < > behnam.b.mara...@gmail.com> wrote: > > I don't have any external process or planed repair in that time period. > In case of network, I can see outbound network on Cassandra node network > interface but couldn't find any way to check the VPC network to make sure > it is not going out of network. Maybe the only way is analysing VPC Flow > Log. > B. > > On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh > wrote: > > Are you sure you don’t have an outside process that is doing an export , > Spark job, non AWS managed backup process ? > > Is this network out from Cassandra or from the network? > > > Rahul > On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: > > Hi, > I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 > instances with separate EBS volumes for root (gp2), data (gp2) and > commitlog (io1). > I get daily outbound traffic at a certain time everyday. As you can see in > the attached screenshot, whiile my normal networkl oad hardly meets 200MB, > this outbound (orange) spikes up to 2GB while inbound (purple) is less than > 800MB. > There is no repair or backup process giong on in that time window, so I am > wondering where to look. Any idea? > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > > >
Re: Huge daily outbound network traffic
Since it is predictable, can you check the logs during that period? What do they say? Do you have a cron running on those hosts? Do all the nodes experience this issue? Dinesh On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi wrote: Actually I did. It seems this is a cross node traffic from one node to port 7000 (storage_port) of the other node. On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims wrote: Since it's at a consistent time, maybe just look at it with iftop to see where the traffic's going and what port it's coming from? On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi wrote: I don't have any external process or planed repair in that time period.In case of network, I can see outbound network on Cassandra node network interface but couldn't find any way to check the VPC network to make sure it is not going out of network. Maybe the only way is analysing VPC Flow Log.B. On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh wrote: Are you sure you don’t have an outside process that is doing an export , Spark job, non AWS managed backup process ? Is this network out from Cassandra or from the network? RahulOn Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: Hi,I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 instances with separate EBS volumes for root (gp2), data (gp2) and commitlog (io1).I get daily outbound traffic at a certain time everyday. As you can see in the attached screenshot, whiile my normal networkl oad hardly meets 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is less than 800MB.There is no repair or backup process giong on in that time window, so I am wondering where to look. Any idea? - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Reading cardinality from Statistics.db failed
Vitali, It doesn't look like there is an existing Jira. It would be helpful if you could create one with as much information as possible. Can you reduce this issue to a short, repeatable set of steps that we can reproduce? That'll be helpful to debug this problem. Dinesh On Wednesday, August 15, 2018, 1:07:21 AM PDT, Vitali Dyachuk wrote: I've upgraded to 3.0.17 and the issue is still there, Is there a jira ticket for that bug or should i create one? On Wed, Jul 25, 2018 at 2:57 PM Vitali Dyachuk wrote: I'm using 3.0.15. I see that there is some fix for sstable metadata in 3.0.16 https://issues.apache.org/jira/browse/CASSANDRA-14217 - is that a fix for "reading cardinalyti from statistics.db" ? On Wed, Jul 25, 2018 at 1:02 PM Hannu Kröger wrote: What version of Cassandra are you running? There is a bug in 3.10.0 and certain 3.0.x that occurs in certain conditions and corrupts that file. Hannu Vitali Dyachuk kirjoitti 25.7.2018 kello 10.48: Hi, I have noticed in the cassandra system.log that there is some issue with sstable metadata, the messages says: WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading cardinality from Statistics.db failed for /opt/data/disk5/data/keyspace/table/mc-big-Data.db Although there is no such file. The message has appeared after i've changed the compaction strategy from SizeTiered to Leveled. Currently i'm running nodetool scrub to rebuilt the sstable, and it takes a lot of time to scrub all sstables. Reading the code it is said that if this metada is broken, then estimating the keys will be done using index summary. How expensive it is ? https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L245 The main question is why has this happened? Thanks, Vitali Djatsuk.
Re: Huge daily outbound network traffic
Actually I did. It seems this is a cross node traffic from one node to port 7000 (storage_port) of the other node. On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims wrote: > Since it's at a consistent time, maybe just look at it with iftop to see > where the traffic's going and what port it's coming from? > > On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < > behnam.b.mara...@gmail.com> wrote: > >> I don't have any external process or planed repair in that time period. >> In case of network, I can see outbound network on Cassandra node network >> interface but couldn't find any way to check the VPC network to make sure >> it is not going out of network. Maybe the only way is analysing VPC Flow >> Log. >> B. >> >> On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh < >> rahul.xavier.si...@gmail.com> wrote: >> >>> Are you sure you don’t have an outside process that is doing an export , >>> Spark job, non AWS managed backup process ? >>> >>> Is this network out from Cassandra or from the network? >>> >>> >>> Rahul >>> On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: >>> >>> Hi, >>> I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 >>> instances with separate EBS volumes for root (gp2), data (gp2) and >>> commitlog (io1). >>> I get daily outbound traffic at a certain time everyday. As you can see >>> in the attached screenshot, whiile my normal networkl oad hardly meets >>> 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is >>> less than 800MB. >>> There is no repair or backup process giong on in that time window, so I >>> am wondering where to look. Any idea? >>> >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: user-h...@cassandra.apache.org >>> >>> >> >
