Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[Josh Baird]

I haven't seen this on ASE.


On Sun, Nov 20, 2016 at 6:40 PM, George Skorup  wrote:

> ASE = AT Switched Ethernet. Comcast and others pretty much all do the
> same shit for MetroE.
>
> We have three ASE circuits at different locations. All are Ciena 3930s
> with 1G optical hand-off direct to our routers. Each has a different peer
> at the other end of the EVC.
>
> I would get a complete halt in traffic for 2-3 seconds. It was easy to
> catch while running a fast ping interval (50-100ms). I'd see "no route to
> host" exactly when the traffic would quit. We were getting no ARP replies.
> Like an ARP or broadcast rate limiter or filter was mysteriously turned on.
> So I made the ARP entries for the upstream peer static and didn't have any
> more problems.
>
> I've had this problem on all three from time to time, lasting for minutes,
> hours or days and then it just vanishes. Even had two circuits doing this
> simultaneously. So I don't know if it's the 3930s or the ASE core network.
> Although it has been a few months since I've seen it.
>
> I couldn't say if this is your problem, just wanted to give my experience
> because it sounds so similar.
>
>
> On 11/20/2016 4:53 PM, TJ Trout wrote:
>
> Att MIS not Ase, involves also another carriers "ase" product (not at)
> but since all traffic through both peers goes to zero I'm assuming it's
> elsewhere?
>
> On Sun, Nov 20, 2016 at 2:48 PM, George Skorup  wrote:
>
>> Stupid question. Does this involve AT ASE at all?
>>
>>
>> On 11/20/2016 12:47 AM, TJ Trout wrote:
>>
>>> I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that
>>> I've recently installed, prior to installing the 10G switch everything
>>> functioned normally (had a CCR1009, now CCR1036), currently during peak
>>> periods of traffic my traffic will go from 1500mbps to 0mbps for just a
>>> instant then back to 1500mbps, other times it goes from 1500mbps to 750mbps
>>> and back to 1500. Sometimes it happens 30 times a minute sometimes 2 times
>>> per hour.
>>>
>>> I've swapped the switch, sfp's...
>>>
>>> Anyone with any remotely plausible idea would be greatly appreciated
>>>
>>
>>
>
>

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[Faisal Imtiaz]

Have you tried to put a simple queue on your router to limit the bandwidth to 
the subscribed CIR on that circuit ? 

Regards. 

Faisal Imtiaz 
Snappy Internet & Telecom 
7266 SW 48 Street 
Miami, FL 33155 
Tel: 305 663 5518 x 232 

Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net 

> From: "George Skorup" 
> To: af@afmug.com
> Sent: Sunday, November 20, 2016 6:40:46 PM
> Subject: Re: [AFMUG] Traffic dropping to zero for a fraction of a second

> ASE = AT Switched Ethernet. Comcast and others pretty much all do the same
> shit for MetroE.

> We have three ASE circuits at different locations. All are Ciena 3930s with 1G
> optical hand-off direct to our routers. Each has a different peer at the other
> end of the EVC.

> I would get a complete halt in traffic for 2-3 seconds. It was easy to catch
> while running a fast ping interval (50-100ms). I'd see "no route to host"
> exactly when the traffic would quit. We were getting no ARP replies. Like an
> ARP or broadcast rate limiter or filter was mysteriously turned on. So I made
> the ARP entries for the upstream peer static and didn't have any more 
> problems.

> I've had this problem on all three from time to time, lasting for minutes, 
> hours
> or days and then it just vanishes. Even had two circuits doing this
> simultaneously. So I don't know if it's the 3930s or the ASE core network.
> Although it has been a few months since I've seen it.

> I couldn't say if this is your problem, just wanted to give my experience
> because it sounds so similar.

> On 11/20/2016 4:53 PM, TJ Trout wrote:

>> Att MIS not Ase, involves also another carriers "ase" product (not at) but
>> since all traffic through both peers goes to zero I'm assuming it's 
>> elsewhere?

>> On Sun, Nov 20, 2016 at 2:48 PM, George Skorup < geo...@cbcast.com > wrote:

>>> Stupid question. Does this involve AT ASE at all?

>>> On 11/20/2016 12:47 AM, TJ Trout wrote:

 I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that I've
 recently installed, prior to installing the 10G switch everything 
 functioned
 normally (had a CCR1009, now CCR1036), currently during peak periods of 
 traffic
 my traffic will go from 1500mbps to 0mbps for just a instant then back to
 1500mbps, other times it goes from 1500mbps to 750mbps and back to 1500.
 Sometimes it happens 30 times a minute sometimes 2 times per hour.

 I've swapped the switch, sfp's...

 Anyone with any remotely plausible idea would be greatly appreciated

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[George Skorup]

Yup. These are internet circuits which always have traffic on them, both 
in and out. I even set up a 10 second ping using Netwatch as a 
keep-alive. Didn't do shit. Could be a RouterOS bug. But I see this on 
the ASE circuits and nowhere else like internal PTPs. So I don't get it.


On 11/20/2016 5:59 PM, Ken Hohhof wrote:


If there is constant traffic from this IP/MAC pair, shouldn’t that 
keep the ARP cache refreshed so no ARP requests are needed (after the 
first one)?


*From:*Af [mailto:af-boun...@afmug.com] *On Behalf Of *George Skorup
*Sent:* Sunday, November 20, 2016 5:41 PM
*To:* af@afmug.com
*Subject:* Re: [AFMUG] Traffic dropping to zero for a fraction of a second

ASE = AT Switched Ethernet. Comcast and others pretty much all do 
the same shit for MetroE.


We have three ASE circuits at different locations. All are Ciena 3930s 
with 1G optical hand-off direct to our routers. Each has a different 
peer at the other end of the EVC.


I would get a complete halt in traffic for 2-3 seconds. It was easy to 
catch while running a fast ping interval (50-100ms). I'd see "no route 
to host" exactly when the traffic would quit. We were getting no ARP 
replies. Like an ARP or broadcast rate limiter or filter was 
mysteriously turned on. So I made the ARP entries for the upstream 
peer static and didn't have any more problems.


I've had this problem on all three from time to time, lasting for 
minutes, hours or days and then it just vanishes. Even had two 
circuits doing this simultaneously. So I don't know if it's the 3930s 
or the ASE core network. Although it has been a few months since I've 
seen it.


I couldn't say if this is your problem, just wanted to give my 
experience because it sounds so similar.


On 11/20/2016 4:53 PM, TJ Trout wrote:

Att MIS not Ase, involves also another carriers "ase" product (not
at) but since all traffic through both peers goes to zero I'm
assuming it's elsewhere?

On Sun, Nov 20, 2016 at 2:48 PM, George Skorup > wrote:

Stupid question. Does this involve AT ASE at all?



On 11/20/2016 12:47 AM, TJ Trout wrote:

I have 2 peers, one CCR and 2 backhauls connected to a 10G
switch that I've recently installed, prior to installing
the 10G switch everything functioned normally (had a
CCR1009, now CCR1036), currently during peak periods of
traffic my traffic will go from 1500mbps to 0mbps for just
a instant then back to 1500mbps, other times it goes from
1500mbps to 750mbps and back to 1500. Sometimes it happens
30 times a minute sometimes 2 times per hour.

I've swapped the switch, sfp's...

Anyone with any remotely plausible idea would be greatly
appreciated

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[Ken Hohhof]

If there is constant traffic from this IP/MAC pair, shouldn’t that keep the ARP 
cache refreshed so no ARP requests are needed (after the first one)?

 

From: Af [mailto:af-boun...@afmug.com] On Behalf Of George Skorup
Sent: Sunday, November 20, 2016 5:41 PM
To: af@afmug.com
Subject: Re: [AFMUG] Traffic dropping to zero for a fraction of a second

 

ASE = AT Switched Ethernet. Comcast and others pretty much all do the same 
shit for MetroE.

We have three ASE circuits at different locations. All are Ciena 3930s with 1G 
optical hand-off direct to our routers. Each has a different peer at the other 
end of the EVC.

I would get a complete halt in traffic for 2-3 seconds. It was easy to catch 
while running a fast ping interval (50-100ms). I'd see "no route to host" 
exactly when the traffic would quit. We were getting no ARP replies. Like an 
ARP or broadcast rate limiter or filter was mysteriously turned on. So I made 
the ARP entries for the upstream peer static and didn't have any more problems.

I've had this problem on all three from time to time, lasting for minutes, 
hours or days and then it just vanishes. Even had two circuits doing this 
simultaneously. So I don't know if it's the 3930s or the ASE core network. 
Although it has been a few months since I've seen it.

I couldn't say if this is your problem, just wanted to give my experience 
because it sounds so similar.

On 11/20/2016 4:53 PM, TJ Trout wrote:

Att MIS not Ase, involves also another carriers "ase" product (not at) but 
since all traffic through both peers goes to zero I'm assuming it's elsewhere?

 

On Sun, Nov 20, 2016 at 2:48 PM, George Skorup  > wrote:

Stupid question. Does this involve AT ASE at all? 



On 11/20/2016 12:47 AM, TJ Trout wrote:

I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that I've 
recently installed, prior to installing the 10G switch everything functioned 
normally (had a CCR1009, now CCR1036), currently during peak periods of traffic 
my traffic will go from 1500mbps to 0mbps for just a instant then back to 
1500mbps, other times it goes from 1500mbps to 750mbps and back to 1500. 
Sometimes it happens 30 times a minute sometimes 2 times per hour.

I've swapped the switch, sfp's...

Anyone with any remotely plausible idea would be greatly appreciated

 

 

 

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[George Skorup]

ASE = AT Switched Ethernet. Comcast and others pretty much all do the 
same shit for MetroE.


We have three ASE circuits at different locations. All are Ciena 3930s 
with 1G optical hand-off direct to our routers. Each has a different 
peer at the other end of the EVC.


I would get a complete halt in traffic for 2-3 seconds. It was easy to 
catch while running a fast ping interval (50-100ms). I'd see "no route 
to host" exactly when the traffic would quit. We were getting no ARP 
replies. Like an ARP or broadcast rate limiter or filter was 
mysteriously turned on. So I made the ARP entries for the upstream peer 
static and didn't have any more problems.


I've had this problem on all three from time to time, lasting for 
minutes, hours or days and then it just vanishes. Even had two circuits 
doing this simultaneously. So I don't know if it's the 3930s or the ASE 
core network. Although it has been a few months since I've seen it.


I couldn't say if this is your problem, just wanted to give my 
experience because it sounds so similar.


On 11/20/2016 4:53 PM, TJ Trout wrote:
Att MIS not Ase, involves also another carriers "ase" product (not 
at) but since all traffic through both peers goes to zero I'm 
assuming it's elsewhere?


On Sun, Nov 20, 2016 at 2:48 PM, George Skorup > wrote:


Stupid question. Does this involve AT ASE at all?


On 11/20/2016 12:47 AM, TJ Trout wrote:

I have 2 peers, one CCR and 2 backhauls connected to a 10G
switch that I've recently installed, prior to installing the
10G switch everything functioned normally (had a CCR1009, now
CCR1036), currently during peak periods of traffic my traffic
will go from 1500mbps to 0mbps for just a instant then back to
1500mbps, other times it goes from 1500mbps to 750mbps and
back to 1500. Sometimes it happens 30 times a minute sometimes
2 times per hour.

I've swapped the switch, sfp's...

Anyone with any remotely plausible idea would be greatly
appreciated

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[TJ Trout]

Att MIS not Ase, involves also another carriers "ase" product (not at)
but since all traffic through both peers goes to zero I'm assuming it's
elsewhere?

On Sun, Nov 20, 2016 at 2:48 PM, George Skorup  wrote:

> Stupid question. Does this involve AT ASE at all?
>
>
> On 11/20/2016 12:47 AM, TJ Trout wrote:
>
>> I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that
>> I've recently installed, prior to installing the 10G switch everything
>> functioned normally (had a CCR1009, now CCR1036), currently during peak
>> periods of traffic my traffic will go from 1500mbps to 0mbps for just a
>> instant then back to 1500mbps, other times it goes from 1500mbps to 750mbps
>> and back to 1500. Sometimes it happens 30 times a minute sometimes 2 times
>> per hour.
>>
>> I've swapped the switch, sfp's...
>>
>> Anyone with any remotely plausible idea would be greatly appreciated
>>
>
>

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[George Skorup]

Stupid question. Does this involve AT ASE at all?

On 11/20/2016 12:47 AM, TJ Trout wrote:
I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that 
I've recently installed, prior to installing the 10G switch everything 
functioned normally (had a CCR1009, now CCR1036), currently during 
peak periods of traffic my traffic will go from 1500mbps to 0mbps for 
just a instant then back to 1500mbps, other times it goes from 
1500mbps to 750mbps and back to 1500. Sometimes it happens 30 times a 
minute sometimes 2 times per hour.


I've swapped the switch, sfp's...

Anyone with any remotely plausible idea would be greatly appreciated

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[Josh Reynolds]

Sounds like pause frames + microbursts on gear with buffers that are too
small. Didn't we just have this discussion?

On Nov 20, 2016 12:47 AM, "TJ Trout"  wrote:

> I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that
> I've recently installed, prior to installing the 10G switch everything
> functioned normally (had a CCR1009, now CCR1036), currently during peak
> periods of traffic my traffic will go from 1500mbps to 0mbps for just a
> instant then back to 1500mbps, other times it goes from 1500mbps to 750mbps
> and back to 1500. Sometimes it happens 30 times a minute sometimes 2 times
> per hour.
>
> I've swapped the switch, sfp's...
>
> Anyone with any remotely plausible idea would be greatly appreciated
>

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[Roland Houin]

probably 32 bit counter on traffic, rolling over.


Roland

> I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that I've
recently installed, prior to installing the 10G switch everything functioned
normally (had a CCR1009, now CCR1036), currently during peak periods of traffic
my traffic will go from 1500mbps to 0mbps for just a instant then back to
1500mbps, other times it goes from 1500mbps to 750mbps and back to 1500.
Sometimes it happens 30 times a minute sometimes 2 times per hour.

I've swapped the switch, sfp's...

Anyone with any remotely plausible idea would be greatly appreciated <

Re: [AFMUG] Traffic dropping to zero for a fraction of a second

[Stefan Englhardt]

There was a bug regarding simple queues. This happened when enabling/disabling 
queues.Watch the cpu/profiling if something peaks suddenly.

 Ursprüngliche Nachricht 
Von: TJ Trout  
Datum: 20.11.2016  07:47  (GMT+01:00) 
An: af@afmug.com 
Betreff: [AFMUG] Traffic dropping to zero for a fraction of a second 

[AFMUG] Traffic dropping to zero for a fraction of a second

[TJ Trout]

I have 2 peers, one CCR and 2 backhauls connected to a 10G switch that I've
recently installed, prior to installing the 10G switch everything
functioned normally (had a CCR1009, now CCR1036), currently during peak
periods of traffic my traffic will go from 1500mbps to 0mbps for just a
instant then back to 1500mbps, other times it goes from 1500mbps to 750mbps
and back to 1500. Sometimes it happens 30 times a minute sometimes 2 times
per hour.

I've swapped the switch, sfp's...

Anyone with any remotely plausible idea would be greatly appreciated