Re: [c-nsp] Weird Multicast microburst amplification issue
I've had pretty bad luck using flex10 modules with multicast market data. We've had lots of driver related issues and are still fighting with packet loss which, at least for me, is much harder to debug. Our blades were running windows and solaris, so you might have better luck if you're running some flavor of linux. We've got the NIC sliced up into 3 VNICs of 4Gb,4Gb and 2Gb. I've got 4900Ms and 5010s and I don't see issues like this at mkt data rates higher (OPRA) than we're supporting on the flex10s. Regards, --Duane ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
I agree that the 10g-1G is probably the culprit amplifying the burst nature of the packets. At our colo at the NYSE in Mahwah we have implemented HP's flex fabric in blade chassis instead of doing what you are doing with the Arista's and it's working fine. We probably are going to have to do the same at our core datacenter. http://h18004.www1.hp.com/products/quickspecs/13127_div/13127_div.html Basically you connect N number of 10GB uplinks into the flex fabric and you can control how the bandwidth is allocated to each blade. I was hoping to avoid having to do anything, but it looks like the data rates are killing the port buffers. I was hoping that the 6500/sup720 with 6748 would handle 120Mbps, 12k pps multicast, but it doesn't look like it. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp- boun...@puck.nether.net] On Behalf Of Jeff Bacon Sent: Tuesday, December 13, 2011 9:30 AM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Weird Multicast microburst amplification issue It definitely looks like a classic microburst output buffer overflow problem, but with a Sup720 and a 6748 module, I haven't seen this at this volume. Ticker volume has peeked recently, and that might contribute to it. It appears to start happening with more than 120Mbps and/or 12,000 pps output on the port. Other than moving to 10GB, I don't see any solutions. Given the 6748 buffer size, I'm surprised it's overrunning it at this volume. It could very well be. The port buffers on a 6748 are only about so big, after all. The amplification factor may come from the simple fact that you have a 10G pipe between the switches. But the input is only coming in at 1G! you say. Yes, but it's then being intermingled on the 10G pipe. Probably on a 6708/6716 with 200mb port buffers. After going through replication engine. (Ingress mode? Egress? Shouldn't matter though.) So while in theory the traffic should be coming through the 10G at 1G rates, it isn't necessarily, and you have to consider the possibility that you are, yes, facing the ol' 10G-1G neck-down problem. 100 packets @ 1500 bytes == 1.5mbyte == buffer go boom. If the packets are large, you also have serialization delay to consider. What takes 3 micros to get out the 1G pipe only takes 1 micro to come in the 10G pipe. Multiply. I'm not going to point at any of these and say that's it - but I can see where it can happen, as annoying as crap as it might be. Someone suggested running a 1G pipe between the switches to see whether the problem went away - I suspect that is what they were pointing at. I've been moving hosts off the 6500s and onto 10G off aristas fed off the 6500s. Let the 6500 drive the WAN, the aristas handle fan-out. I am actually sitting here debating swapping out a pair of VS720s with sup-2T kit - not even because the hardware is working particularly hard as-is (by the stats, they've got life downright good) but because sledgehammer overkill seems to be about the only safe option in dealing with these kinds of flows, I know it will take me 3 months to swap the sups and cards out, and it might be better to start now, however little thrilled I am at forking $20k more per switch than I had originally intended (the VS720 parts swapped would be used to populate some new chassis in a DR/test facility, the original idea was just to buy VS720 parts, but my vendor came up with better prices on sup2t kit than I'd seen even a couple months ago so now it's just in the range of argh...maybe... instead of no, way too much). Or to quote one of my employees, who knows what MOAR will be asked for next... - or, when will OPRA blow the cap again? I suspect you just helped me decide. G. -bacon ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
This might sound incredibly silly, but in the case I've had better luck with 4900s. They use a single central buffer, so the buffer is effectively considerably deeper, depending on the specific workload of course. I've never been quite able to bring myself to use a dell blade chassis, and none of my people want to do HP. Plus we're playing with different 10G NICs for perf tuning. So, piles of 1Us and Aristas. assuming you've got 6708/6716s (or are you using the 10Gs on the vs720?), you COULD attempt to use SRR QoS to attempt to knock off the edges of the peaks a little - say, cap at 500m-1Gbit for those flows to try and smooth out the microbursts a little on the 10g pipe. or, the gig ports on the supervisor IIRC have somewhat larger buffers. not sure how much help that is though, it's not like there's a lot of 'em. -bacon -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Tuesday, December 13, 2011 9:49 AM To: Jeff Bacon; 'cisco-nsp@puck.nether.net' Subject: RE: Weird Multicast microburst amplification issue I agree that the 10g-1G is probably the culprit amplifying the burst nature of the packets. At our colo at the NYSE in Mahwah we have implemented HP's flex fabric in blade chassis instead of doing what you are doing with the Arista's and it's working fine. We probably are going to have to do the same at our core datacenter. http://h18004.www1.hp.com/products/quickspecs/13127_div/13127_div.h tml Basically you connect N number of 10GB uplinks into the flex fabric and you can control how the bandwidth is allocated to each blade. I was hoping to avoid having to do anything, but it looks like the data rates are killing the port buffers. I was hoping that the 6500/sup720 with 6748 would handle 120Mbps, 12k pps multicast, but it doesn't look like it. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
Do you have any span sessions enabled? Dale Thus spake Matthew Huff (mh...@ox.com) on Fri, Dec 09, 2011 at 01:48:35PM -0500: We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic via the 10gb L3 interface, something is causing the traffic burst to amplify, overrunning the output port. Has anyone seen this, and does anyone know how to mitigate this? Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff| Fax: 914-460-4139 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
No span sessions enabled. It definitely looks like a classic microburst output buffer overflow problem, but with a Sup720 and a 6748 module, I haven't seen this at this volume. Ticker volume has peeked recently, and that might contribute to it. It appears to start happening with more than 120Mbps and/or 12,000 pps output on the port. Other than moving to 10GB, I don't see any solutions. Given the 6748 buffer size, I'm surprised it's overrunning it at this volume. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Dale W. Carder [mailto:dwcar...@wisc.edu] Sent: Monday, December 12, 2011 10:10 AM To: Matthew Huff Cc: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: Re: [c-nsp] Weird Multicast microburst amplification issue Do you have any span sessions enabled? Dale Thus spake Matthew Huff (mh...@ox.com) on Fri, Dec 09, 2011 at 01:48:35PM -0500: We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic via the 10gb L3 interface, something is causing the traffic burst to amplify, overrunning the output port. Has anyone seen this, and does anyone know how to mitigate this? Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff| Fax: 914-460-4139 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
On 12/09/2011 06:48 PM, Matthew Huff wrote: We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. That's very peculiar. Which 10g cards and IOS versions are you using? ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
What if you interconnect the two switches with 1G rather than 10? Frank -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 2:18 PM To: 'Chuck Church'; 'cisco-nsp' Subject: Re: [c-nsp] Weird Multicast microburst amplification issue Yes, only the correct stream. I've opened a case with Cisco. I'm suspecting that the multicast replication engine is doing something that causes it to amplify the bursty nature of the traffic causing the microburst overruns. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 3:01 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Hmmm. If it's not spanning tree, I'd have to say it's something not working right with IGMP, and the server's port is getting more streams than it should. Have you checked the IGMP port association to see what it's subscribed to? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:48 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Yes. The problem only occurs when connected to any other switch than the source switch. We have over 300 servers, it isn't anything with a specific server, but rather the nature of not being on the same switch as the source. The data is a high packet count, bursty traffic. However, the drops only occur on the 6748 module via the layer 2 output on a server subscribing to that multicast traffic. This occurs if we turn layer 2 flowcontrol on or off. No pause packets are generated from the server, rather this is 100% related to the output ASIC queues on the 6748 module. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:43 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Can you move the source server over to switch B to see if the problem still exists on switch B then, or moves to switch A? Anything showing up in the logs? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:25 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Unfortunately, it isn't something simple like that. The output drops are continuously happening. The network is very stable. There are not other issues during this time. It's something amplifying the burst of the stream, either the multicast replication or passing through the layer 3 interface. IF we run a test with a server on switch a, a client on switch a, and a client on switch b, only the client on switch b is seeing the problem. The problem isn't passing the data from switch-a to b, but rather something during the transmission that changes the shape of the data to be a heavier burst. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:11 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Are there multiple streams passing through the switch? A spanning tree recalculation will cause IGMP to flush associations, and flood all streams out all ports until they're relearned. Portfast will fix it, as will a multicast-specific interface command, would need to look it up. Chuck -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 1:49 PM To: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: [c-nsp] Weird Multicast microburst amplification issue We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have
Re: [c-nsp] Weird Multicast microburst amplification issue
Are there multiple streams passing through the switch? A spanning tree recalculation will cause IGMP to flush associations, and flood all streams out all ports until they're relearned. Portfast will fix it, as will a multicast-specific interface command, would need to look it up. Chuck -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 1:49 PM To: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: [c-nsp] Weird Multicast microburst amplification issue We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic via the 10gb L3 interface, something is causing the traffic burst to amplify, overrunning the output port. Has anyone seen this, and does anyone know how to mitigate this? Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff| Fax: 914-460-4139 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
Unfortunately, it isn't something simple like that. The output drops are continuously happening. The network is very stable. There are not other issues during this time. It's something amplifying the burst of the stream, either the multicast replication or passing through the layer 3 interface. IF we run a test with a server on switch a, a client on switch a, and a client on switch b, only the client on switch b is seeing the problem. The problem isn't passing the data from switch-a to b, but rather something during the transmission that changes the shape of the data to be a heavier burst. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:11 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Are there multiple streams passing through the switch? A spanning tree recalculation will cause IGMP to flush associations, and flood all streams out all ports until they're relearned. Portfast will fix it, as will a multicast-specific interface command, would need to look it up. Chuck -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 1:49 PM To: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: [c-nsp] Weird Multicast microburst amplification issue We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic via the 10gb L3 interface, something is causing the traffic burst to amplify, overrunning the output port. Has anyone seen this, and does anyone know how to mitigate this? Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff| Fax: 914-460-4139 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
Can you move the source server over to switch B to see if the problem still exists on switch B then, or moves to switch A? Anything showing up in the logs? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:25 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Unfortunately, it isn't something simple like that. The output drops are continuously happening. The network is very stable. There are not other issues during this time. It's something amplifying the burst of the stream, either the multicast replication or passing through the layer 3 interface. IF we run a test with a server on switch a, a client on switch a, and a client on switch b, only the client on switch b is seeing the problem. The problem isn't passing the data from switch-a to b, but rather something during the transmission that changes the shape of the data to be a heavier burst. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:11 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Are there multiple streams passing through the switch? A spanning tree recalculation will cause IGMP to flush associations, and flood all streams out all ports until they're relearned. Portfast will fix it, as will a multicast-specific interface command, would need to look it up. Chuck -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 1:49 PM To: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: [c-nsp] Weird Multicast microburst amplification issue We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic via the 10gb L3 interface, something is causing the traffic burst to amplify, overrunning the output port. Has anyone seen this, and does anyone know how to mitigate this? Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff| Fax: 914-460-4139 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
Yes. The problem only occurs when connected to any other switch than the source switch. We have over 300 servers, it isn't anything with a specific server, but rather the nature of not being on the same switch as the source. The data is a high packet count, bursty traffic. However, the drops only occur on the 6748 module via the layer 2 output on a server subscribing to that multicast traffic. This occurs if we turn layer 2 flowcontrol on or off. No pause packets are generated from the server, rather this is 100% related to the output ASIC queues on the 6748 module. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:43 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Can you move the source server over to switch B to see if the problem still exists on switch B then, or moves to switch A? Anything showing up in the logs? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:25 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Unfortunately, it isn't something simple like that. The output drops are continuously happening. The network is very stable. There are not other issues during this time. It's something amplifying the burst of the stream, either the multicast replication or passing through the layer 3 interface. IF we run a test with a server on switch a, a client on switch a, and a client on switch b, only the client on switch b is seeing the problem. The problem isn't passing the data from switch-a to b, but rather something during the transmission that changes the shape of the data to be a heavier burst. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:11 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Are there multiple streams passing through the switch? A spanning tree recalculation will cause IGMP to flush associations, and flood all streams out all ports until they're relearned. Portfast will fix it, as will a multicast-specific interface command, would need to look it up. Chuck -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 1:49 PM To: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: [c-nsp] Weird Multicast microburst amplification issue We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic via the 10gb L3 interface, something is causing the traffic burst to amplify, overrunning the output port. Has anyone seen this, and does anyone know how to mitigate this? Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff| Fax: 914-460-4139 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Weird Multicast microburst amplification issue
Hmmm. If it's not spanning tree, I'd have to say it's something not working right with IGMP, and the server's port is getting more streams than it should. Have you checked the IGMP port association to see what it's subscribed to? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:48 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Yes. The problem only occurs when connected to any other switch than the source switch. We have over 300 servers, it isn't anything with a specific server, but rather the nature of not being on the same switch as the source. The data is a high packet count, bursty traffic. However, the drops only occur on the 6748 module via the layer 2 output on a server subscribing to that multicast traffic. This occurs if we turn layer 2 flowcontrol on or off. No pause packets are generated from the server, rather this is 100% related to the output ASIC queues on the 6748 module. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:43 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Can you move the source server over to switch B to see if the problem still exists on switch B then, or moves to switch A? Anything showing up in the logs? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:25 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Unfortunately, it isn't something simple like that. The output drops are continuously happening. The network is very stable. There are not other issues during this time. It's something amplifying the burst of the stream, either the multicast replication or passing through the layer 3 interface. IF we run a test with a server on switch a, a client on switch a, and a client on switch b, only the client on switch b is seeing the problem. The problem isn't passing the data from switch-a to b, but rather something during the transmission that changes the shape of the data to be a heavier burst. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:11 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Are there multiple streams passing through the switch? A spanning tree recalculation will cause IGMP to flush associations, and flood all streams out all ports until they're relearned. Portfast will fix it, as will a multicast-specific interface command, would need to look it up. Chuck -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 1:49 PM To: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: [c-nsp] Weird Multicast microburst amplification issue We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic via the 10gb L3 interface, something is causing the traffic burst to amplify, overrunning the output port. Has anyone seen this, and does anyone know how to mitigate this? Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff| Fax: 914-460-4139 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive
Re: [c-nsp] Weird Multicast microburst amplification issue
Yes, only the correct stream. I've opened a case with Cisco. I'm suspecting that the multicast replication engine is doing something that causes it to amplify the bursty nature of the traffic causing the microburst overruns. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 3:01 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Hmmm. If it's not spanning tree, I'd have to say it's something not working right with IGMP, and the server's port is getting more streams than it should. Have you checked the IGMP port association to see what it's subscribed to? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:48 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Yes. The problem only occurs when connected to any other switch than the source switch. We have over 300 servers, it isn't anything with a specific server, but rather the nature of not being on the same switch as the source. The data is a high packet count, bursty traffic. However, the drops only occur on the 6748 module via the layer 2 output on a server subscribing to that multicast traffic. This occurs if we turn layer 2 flowcontrol on or off. No pause packets are generated from the server, rather this is 100% related to the output ASIC queues on the 6748 module. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:43 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Can you move the source server over to switch B to see if the problem still exists on switch B then, or moves to switch A? Anything showing up in the logs? Chuck -Original Message- From: Matthew Huff [mailto:mh...@ox.com] Sent: Friday, December 09, 2011 2:25 PM To: 'Chuck Church'; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Unfortunately, it isn't something simple like that. The output drops are continuously happening. The network is very stable. There are not other issues during this time. It's something amplifying the burst of the stream, either the multicast replication or passing through the layer 3 interface. IF we run a test with a server on switch a, a client on switch a, and a client on switch b, only the client on switch b is seeing the problem. The problem isn't passing the data from switch-a to b, but rather something during the transmission that changes the shape of the data to be a heavier burst. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 -Original Message- From: Chuck Church [mailto:chuckchu...@gmail.com] Sent: Friday, December 09, 2011 2:11 PM To: Matthew Huff; 'cisco-nsp' Subject: RE: [c-nsp] Weird Multicast microburst amplification issue Are there multiple streams passing through the switch? A spanning tree recalculation will cause IGMP to flush associations, and flood all streams out all ports until they're relearned. Portfast will fix it, as will a multicast-specific interface command, would need to look it up. Chuck -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Matthew Huff Sent: Friday, December 09, 2011 1:49 PM To: 'cisco-nsp (cisco-nsp@puck.nether.net)' Subject: [c-nsp] Weird Multicast microburst amplification issue We have a multicast data stream (real-time ticker data) that by its nature is very bursty. When we connect a source server via gigabit Ethernet to our 6500/sup720 switch via a 6748 module and a destination server via gigabit to the same or different module in the same switch, everything works fine. If the destination server is on a different switch connected by a layer3 10GB connection then we have significant output drops on the Ethernet connected to the destination server. All switches are 6509/sup720 with 6748 line cards. QoS is disabled globally. The servers are identical. The output drops only occur on the Ethernet drop connected to the server. The only thing I can think is happening is that by routing the traffic
