Re: Huge daily outbound network traffic
You could also run tcpdump to inspect the streams. Dinesh On Thursday, August 16, 2018, 1:11:47 PM PDT, Elliott Sims wrote: Since this is cross-node traffic, "nodetool netstats" during the high-traffic period should give you a better idea of what's being sent. On Thu, Aug 16, 2018 at 2:34 AM, Behnam B.Marandi wrote: In case of cronjobs, there is no jobs for that time period and I can see affect of jobs like backups and repairs but traffic that they cause is not comparable. Like 800MB comparing to 2GB. And for this case it is all outbound network on all 3 cluster nodes. On Thu, Aug 16, 2018 at 5:16 PM dinesh.jo...@yahoo.com.INVALID wrote: Since it is predictable, can you check the logs during that period? What do they say? Do you have a cron running on those hosts? Do all the nodes experience this issue? Dinesh On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi wrote: Actually I did. It seems this is a cross node traffic from one node to port 7000 (storage_port) of the other node. On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims wrote: Since it's at a consistent time, maybe just look at it with iftop to see where the traffic's going and what port it's coming from? On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi wrote: I don't have any external process or planed repair in that time period.In case of network, I can see outbound network on Cassandra node network interface but couldn't find any way to check the VPC network to make sure it is not going out of network. Maybe the only way is analysing VPC Flow Log.B. On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh wrote: Are you sure you don’t have an outside process that is doing an export , Spark job, non AWS managed backup process ? Is this network out from Cassandra or from the network? RahulOn Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: Hi,I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 instances with separate EBS volumes for root (gp2), data (gp2) and commitlog (io1).I get daily outbound traffic at a certain time everyday. As you can see in the attached screenshot, whiile my normal networkl oad hardly meets 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is less than 800MB.There is no repair or backup process giong on in that time window, so I am wondering where to look. Any idea? -- -- - To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Huge daily outbound network traffic
Since this is cross-node traffic, "nodetool netstats" during the high-traffic period should give you a better idea of what's being sent. On Thu, Aug 16, 2018 at 2:34 AM, Behnam B.Marandi < behnam.b.mara...@gmail.com> wrote: > In case of cronjobs, there is no jobs for that time period and I can see > affect of jobs like backups and repairs but traffic that they cause is not > comparable. Like 800MB comparing to 2GB. And for this case it is all > outbound network on all 3 cluster nodes. > > On Thu, Aug 16, 2018 at 5:16 PM dinesh.jo...@yahoo.com.INVALID < > dinesh.jo...@yahoo.com.invalid> wrote: > >> Since it is predictable, can you check the logs during that period? What >> do they say? Do you have a cron running on those hosts? Do all the nodes >> experience this issue? >> >> Dinesh >> >> >> On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi < >> behnam.b.mara...@gmail.com> wrote: >> >> >> Actually I did. It seems this is a cross node traffic from one node to >> port 7000 (storage_port) of the other node. >> >> On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims >> wrote: >> >> Since it's at a consistent time, maybe just look at it with iftop to see >> where the traffic's going and what port it's coming from? >> >> On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < >> behnam.b.mara...@gmail.com> wrote: >> >> I don't have any external process or planed repair in that time period. >> In case of network, I can see outbound network on Cassandra node network >> interface but couldn't find any way to check the VPC network to make sure >> it is not going out of network. Maybe the only way is analysing VPC Flow >> Log. >> B. >> >> On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh < >> rahul.xavier.si...@gmail.com> wrote: >> >> Are you sure you don’t have an outside process that is doing an export , >> Spark job, non AWS managed backup process ? >> >> Is this network out from Cassandra or from the network? >> >> >> Rahul >> On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: >> >> Hi, >> I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 >> instances with separate EBS volumes for root (gp2), data (gp2) and >> commitlog (io1). >> I get daily outbound traffic at a certain time everyday. As you can see >> in the attached screenshot, whiile my normal networkl oad hardly meets >> 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is >> less than 800MB. >> There is no repair or backup process giong on in that time window, so I >> am wondering where to look. Any idea? >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> >>
Re: Huge daily outbound network traffic
In case of cronjobs, there is no jobs for that time period and I can see affect of jobs like backups and repairs but traffic that they cause is not comparable. Like 800MB comparing to 2GB. And for this case it is all outbound network on all 3 cluster nodes. On Thu, Aug 16, 2018 at 5:16 PM dinesh.jo...@yahoo.com.INVALID wrote: > Since it is predictable, can you check the logs during that period? What > do they say? Do you have a cron running on those hosts? Do all the nodes > experience this issue? > > Dinesh > > > On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi < > behnam.b.mara...@gmail.com> wrote: > > > Actually I did. It seems this is a cross node traffic from one node to > port 7000 (storage_port) of the other node. > > On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims > wrote: > > Since it's at a consistent time, maybe just look at it with iftop to see > where the traffic's going and what port it's coming from? > > On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < > behnam.b.mara...@gmail.com> wrote: > > I don't have any external process or planed repair in that time period. > In case of network, I can see outbound network on Cassandra node network > interface but couldn't find any way to check the VPC network to make sure > it is not going out of network. Maybe the only way is analysing VPC Flow > Log. > B. > > On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh > wrote: > > Are you sure you don’t have an outside process that is doing an export , > Spark job, non AWS managed backup process ? > > Is this network out from Cassandra or from the network? > > > Rahul > On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: > > Hi, > I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 > instances with separate EBS volumes for root (gp2), data (gp2) and > commitlog (io1). > I get daily outbound traffic at a certain time everyday. As you can see in > the attached screenshot, whiile my normal networkl oad hardly meets 200MB, > this outbound (orange) spikes up to 2GB while inbound (purple) is less than > 800MB. > There is no repair or backup process giong on in that time window, so I am > wondering where to look. Any idea? > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > > >
Re: Huge daily outbound network traffic
Since it is predictable, can you check the logs during that period? What do they say? Do you have a cron running on those hosts? Do all the nodes experience this issue? Dinesh On Thursday, August 16, 2018, 12:02:55 AM PDT, Behnam B.Marandi wrote: Actually I did. It seems this is a cross node traffic from one node to port 7000 (storage_port) of the other node. On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims wrote: Since it's at a consistent time, maybe just look at it with iftop to see where the traffic's going and what port it's coming from? On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi wrote: I don't have any external process or planed repair in that time period.In case of network, I can see outbound network on Cassandra node network interface but couldn't find any way to check the VPC network to make sure it is not going out of network. Maybe the only way is analysing VPC Flow Log.B. On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh wrote: Are you sure you don’t have an outside process that is doing an export , Spark job, non AWS managed backup process ? Is this network out from Cassandra or from the network? RahulOn Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: Hi,I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 instances with separate EBS volumes for root (gp2), data (gp2) and commitlog (io1).I get daily outbound traffic at a certain time everyday. As you can see in the attached screenshot, whiile my normal networkl oad hardly meets 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is less than 800MB.There is no repair or backup process giong on in that time window, so I am wondering where to look. Any idea? - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Huge daily outbound network traffic
Actually I did. It seems this is a cross node traffic from one node to port 7000 (storage_port) of the other node. On Sun, Aug 12, 2018 at 2:44 PM Elliott Sims wrote: > Since it's at a consistent time, maybe just look at it with iftop to see > where the traffic's going and what port it's coming from? > > On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < > behnam.b.mara...@gmail.com> wrote: > >> I don't have any external process or planed repair in that time period. >> In case of network, I can see outbound network on Cassandra node network >> interface but couldn't find any way to check the VPC network to make sure >> it is not going out of network. Maybe the only way is analysing VPC Flow >> Log. >> B. >> >> On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh < >> rahul.xavier.si...@gmail.com> wrote: >> >>> Are you sure you don’t have an outside process that is doing an export , >>> Spark job, non AWS managed backup process ? >>> >>> Is this network out from Cassandra or from the network? >>> >>> >>> Rahul >>> On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: >>> >>> Hi, >>> I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 >>> instances with separate EBS volumes for root (gp2), data (gp2) and >>> commitlog (io1). >>> I get daily outbound traffic at a certain time everyday. As you can see >>> in the attached screenshot, whiile my normal networkl oad hardly meets >>> 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is >>> less than 800MB. >>> There is no repair or backup process giong on in that time window, so I >>> am wondering where to look. Any idea? >>> >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: user-h...@cassandra.apache.org >>> >>> >> >
Re: Huge daily outbound network traffic
Since it's at a consistent time, maybe just look at it with iftop to see where the traffic's going and what port it's coming from? On Fri, Aug 10, 2018 at 1:48 AM, Behnam B.Marandi < behnam.b.mara...@gmail.com> wrote: > I don't have any external process or planed repair in that time period. > In case of network, I can see outbound network on Cassandra node network > interface but couldn't find any way to check the VPC network to make sure > it is not going out of network. Maybe the only way is analysing VPC Flow > Log. > B. > > On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh > wrote: > >> Are you sure you don’t have an outside process that is doing an export , >> Spark job, non AWS managed backup process ? >> >> Is this network out from Cassandra or from the network? >> >> >> Rahul >> On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: >> >> Hi, >> I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 >> instances with separate EBS volumes for root (gp2), data (gp2) and >> commitlog (io1). >> I get daily outbound traffic at a certain time everyday. As you can see >> in the attached screenshot, whiile my normal networkl oad hardly meets >> 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is >> less than 800MB. >> There is no repair or backup process giong on in that time window, so I >> am wondering where to look. Any idea? >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >
Re: Huge daily outbound network traffic
I don't have any external process or planed repair in that time period. In case of network, I can see outbound network on Cassandra node network interface but couldn't find any way to check the VPC network to make sure it is not going out of network. Maybe the only way is analysing VPC Flow Log. B. On Tue, Aug 7, 2018 at 11:23 PM, Rahul Singh wrote: > Are you sure you don’t have an outside process that is doing an export , > Spark job, non AWS managed backup process ? > > Is this network out from Cassandra or from the network? > > > Rahul > On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: > > Hi, > I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 > instances with separate EBS volumes for root (gp2), data (gp2) and > commitlog (io1). > I get daily outbound traffic at a certain time everyday. As you can see in > the attached screenshot, whiile my normal networkl oad hardly meets 200MB, > this outbound (orange) spikes up to 2GB while inbound (purple) is less than > 800MB. > There is no repair or backup process giong on in that time window, so I am > wondering where to look. Any idea? > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: Huge daily outbound network traffic
Are you sure you don’t have an outside process that is doing an export , Spark job, non AWS managed backup process ? Is this network out from Cassandra or from the network? Rahul On Aug 7, 2018, 4:09 AM -0400, Behnam B.Marandi , wrote: > Hi, > I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 instances > with separate EBS volumes for root (gp2), data (gp2) and commitlog (io1). > I get daily outbound traffic at a certain time everyday. As you can see in > the attached screenshot, whiile my normal networkl oad hardly meets 200MB, > this outbound (orange) spikes up to 2GB while inbound (purple) is less than > 800MB. > There is no repair or backup process giong on in that time window, so I am > wondering where to look. Any idea? > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org
Huge daily outbound network traffic
Hi, I have a 3 node Cassandra cluster (version 3.11.1) on m4.xlarge EC2 instances with separate EBS volumes for root (gp2), data (gp2) and commitlog (io1). I get daily outbound traffic at a certain time everyday. As you can see in the attached screenshot, whiile my normal networkl oad hardly meets 200MB, this outbound (orange) spikes up to 2GB while inbound (purple) is less than 800MB. There is no repair or backup process giong on in that time window, so I am wondering where to look. Any idea? - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
