Re: Huge daily outbound network traffic

2018-08-16 Thread dinesh.jo...@yahoo.com.INVALID
You could also run tcpdump to inspect the streams. Dinesh On Thursday, August 16, 2018, 1:11:47 PM PDT, Elliott Sims wrote: Since this is cross-node traffic, "nodetool netstats" during the high-traffic period should give you a better idea of what's being sent. On Thu, Aug 16, 2018

Re: Huge daily outbound network traffic

2018-08-16 Thread Elliott Sims
Since this is cross-node traffic, "nodetool netstats" during the high-traffic period should give you a better idea of what's being sent. On Thu, Aug 16, 2018 at 2:34 AM, Behnam B.Marandi < behnam.b.mara...@gmail.com> wrote: > In case of cronjobs, there is no jobs for that time period and I can

Re: Huge daily outbound network traffic

2018-08-16 Thread Behnam B.Marandi
In case of cronjobs, there is no jobs for that time period and I can see affect of jobs like backups and repairs but traffic that they cause is not comparable. Like 800MB comparing to 2GB. And for this case it is all outbound network on all 3 cluster nodes. On Thu, Aug 16, 2018 at 5:16 PM

Re: Huge daily outbound network traffic

2018-08-16 Thread dinesh.jo...@yahoo.com.INVALID
Since it is predictable, can you check the logs during that period? What do they say? Do you have a cron running on those hosts? Do all the nodes experience this issue? Dinesh On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi wrote: Actually I did. It seems this is a

Re: Huge daily outbound network traffic

2018-08-16 Thread Behnam B.Marandi
Actually I did. It seems this is a cross node traffic from one node to port 7000 (storage_port) of the other node. On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims wrote: > Since it's at a consistent time, maybe just look at it with iftop to see > where the traffic's going and what port it's coming

Re: Huge daily outbound network traffic

2018-08-11 Thread Elliott Sims
Since it's at a consistent time, maybe just look at it with iftop to see where the traffic's going and what port it's coming from? On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < behnam.b.mara...@gmail.com> wrote: > I don't have any external process or planed repair in that time period. > In

Re: Huge daily outbound network traffic

2018-08-10 Thread Behnam B.Marandi
I don't have any external process or planed repair in that time period. In case of network, I can see outbound network on Cassandra node network interface but couldn't find any way to check the VPC network to make sure it is not going out of network. Maybe the only way is analysing VPC Flow Log.

Re: Huge daily outbound network traffic

2018-08-07 Thread Rahul Singh
Are you sure you don’t have an outside process that is doing an export , Spark job, non AWS managed backup process ? Is this network out from Cassandra or from the network? Rahul On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: > Hi, > I have a 3 node Cassandra cluster (version 3.11.1)

Huge daily outbound network traffic

2018-08-07 Thread Behnam B.Marandi
Hi, I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 instances with separate EBS volumes for root (gp2), data (gp2) and commitlog (io1). I get daily outbound traffic at a certain time everyday. As you can see in the attached screenshot, whiile my normal networkl oad hardly meets