Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

"partial tables" 

There were only 292 IPv4 routes, which was expected. The IPv6 routes were 
expected to be less. There were 118k IPv6 routes in a box that could only 
handle... 4k? 


Fixed that errant IPv6 fill and all is well. Well, seemingly anyway. I won't 
rule out a coincidence, but these seems like a direct line from problem to 
resolution. 





- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mike Hammett"  
To: "NANOG"  
Sent: Monday, April 3, 2023 12:21:29 AM 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Cisco Nexus VXLAN w/vlan-aware-bundle ?

[Adam Thompson]

Does anyone here use VXLAN on Cisco Nexus gear in the "vlan-aware-bundle" mode 
(RFC7432 §6.3), not the "vlan-based" mode (§6.1)?  If so, could you please 
contact me directly to explain how you did so?
Much appreciated,
-Adam

Adam Thompson
Consultant, Infrastructure Services
[cid:image004.png@01D97DB7.68236EA0]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca<https://www.merlin.mb.ca/>
[cid:image003.png@01D97DB7.67506730]Chat with me on 
Teams<https://teams.microsoft.com/l/chat/0/0?users=athomp...@merlin.mb.ca>

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

We have upgraded NX-OS to a new major version and have the same results. 



Apr 20 09:36:05 UTC: %UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 
consistency checker FAILED on slot 1 
Apr 19 13:55:57 UTC: %IPFIB-2-FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM 
exhausted for IPV4 routes in LPM table 

show forwarding inconsistency 
... 
69. slot(1), vrf(default), prefix(198.XXX.XXX.0/YY), Route inconsistent in FIB 
Software. 


Well, those seem like problems. 


EASTGATE401-BGP-02# show ip route summary 
IP Route Table for VRF "default" 
Total number of routes: 292 
Total number of paths: 292 

Unicast paths: 
Best paths per protocol: Backup paths per protocol: 
am : 48 None 
local : 8 
direct : 8 
static : 5 
broadcast : 19 
bgp-X : 204 

Number of routes per mask-length: 
/0 : 1 /8 : 1 /18: 2 /19: 4 /20: 26 
/21: 24 /22: 35 /23: 18 /24: 100 /27: 1 
/28: 1 /29: 1 /30: 4 /32: 74 




It's choking on only 292 routes? 


Error Message FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for 
IPV4 routes in LPM table 
Explanation The TCAM device in the Layer 3 forwarding ASIC has reached its 
system limits for IPv4 entries in the LPM table. 

Recommended Action No action is required. 



TCAM is exhausted and no action is recommended? 


Error Message UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 
consistency checker FAILED on slot [chars] 
Explanation FIB Ipv4 route consistency checker Failed. Route database is 
consistent with hardware 

Recommended Action No action is required. 


The FIB is inconsistent and no action is recommended? 


EASTGATE401-BGP-02# show system internal forwarding route summary 

slot 1 
=== 


IPv4 hosts & routes summary on module 1 
- 

Max host route entries : 8192 
Total number of IPv4 host routes used: 72 
Max LPM table entries : 7167 
Total number of IPv4 LPM routes used : 16 



I seem to be out of my depth here in that 292 is less than 7167, but yet it 
still fails. 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 


From: "Mike Hammett"  
To: "NANOG"  
Sent: Monday, April 3, 2023 12:21:29 AM 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Joelja Bogus]

Sent from my iPhoneOn Apr 16, 2023, at 10:05, Mike Hammett  wrote:We did have our common upstream provider perform maintenance that then afterwards, had the traffic flowing on the right path. Later activity on our direct connection pushed it back to the common upstream. We haven't yet had the opportunity to bump our BGP session with the common upstream provider, but I suspect that will put the traffic back onto the right path. Seems like the router is just hanging onto the oldest BGP session it has, regardless of any other parameter or configuration.This seems like a bug. We do intend on upgrading NX-OS, but that's on someone else's schedule.Not that familiar  with nexus 3k but I would compare the route in the rib and that on the module.  If the platform is exhausting unicast route entries the control plan may  show the route when  module / asic doesn’t have it installed. It’s not always obviously when these things run out of tcam https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/9-x/unicast/configuration/guide/l3_cli_nxos/l3_manage-routes.pdfThis is nexus 9k  But I would expect it to be broadly similar  in terms of diagnostic on the other platformsNexus 3ks are Broadcom merchant silicon (trident/tomahawk so I’d expect the them to run out of fib in the mid tens of thousands of routes..-Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISPFrom: "Mike Hammett" To: "NANOG" Sent: Monday, April 3, 2023 12:21:29 AMSubject: Cisco Nexus 3k Route Selection\Packet Forwarding DebuggingWe have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? -Mike HammettIntelligent Computing Solutionshttp://www.ics-il.comMidwest-IXhttp://www.midwest-ix.com

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

We did have our common upstream provider perform maintenance that then 
afterwards, had the traffic flowing on the right path. Later activity on our 
direct connection pushed it back to the common upstream. We haven't yet had the 
opportunity to bump our BGP session with the common upstream provider, but I 
suspect that will put the traffic back onto the right path. Seems like the 
router is just hanging onto the oldest BGP session it has, regardless of any 
other parameter or configuration. 


This seems like a bug. We do intend on upgrading NX-OS, but that's on someone 
else's schedule. 



- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mike Hammett"  
To: "NANOG"  
Sent: Monday, April 3, 2023 12:21:29 AM 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Nick Olsen]

Not that it's a "Fix" but have you tried rebooting the box? If this is a
bug in the forwarding plane that might clear/rebuild it. And maybe it works
correctly after that.

Friend saw something similar on a Juniper MX with DPC cards that had run
out of FIB space. It would show correctly in all places, but was failing to
install in FIB (Router cut a error about it in the log). So the actual
traffic didn't follow the same path. I saw you specifically referenced
"forwarding" in one of your copy pastes. And it looked right. But I don't
know enough about Cisco to say if that's really what is in FIB. Or just
what it thinks is in FIB.

On Tue, Apr 4, 2023 at 5:47 PM Mike Hammett  wrote:

> Via all mechanisms I could find in the router, it thinks the best path is
> the direct path, the packets just don't go that way.
>
> The in traffic isn't a concern at this time, just the out (from my
> perspective).
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"David Bass" 
> *To: *"Mike Hammett" 
> *Cc: *"Matthew Huff" , "NANOG" 
> *Sent: *Tuesday, April 4, 2023 11:39:26 AM
> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> If you are both connected to the same upstream, but the customer wants
> traffic destined to the upstream to go through you (in and out), then they
> need to do something on their devices to try and affect the inbound path to
> their AS. From the upstream carrier in question they’ll take the best path
> to a prefix, which direct connection is generally going to be preferred
> over a transit AS (basic BGP best path algorithm stuff) unless there is
> some manipulation of the prefix advertisement happening.
>
> To confirm the path being taken you should be able to do a few trace
> routes from various locations as well as use looking glasses.
>
> Now the sflow data is an entirely different thing to analyze.
>
> On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett  wrote:
>
>>
>> sh ip bgp neighbor advertised-routes shows the only routes being
>> advertised to Y are the routes that should be advertised to them. I checked
>> a variety of other peers and have the expected results.
>>
>>
>>
>> From my perspective:
>>
>> Packets come in on port A, supposed to leave on port X, but they leave on
>> port Y. All of the troubleshooting steps I've done on my own (or suggested
>> by mailing lists) say the packets should be leaving on the desired port X.
>>
>>
>> From the customer's perspective, they're supposed to be coming from me on
>> port X, but they're arriving on port Y, another network.
>>
>> Port X in both scenarios is our direct connection, while port Y is a
>> mutual upstream provider.
>>
>>
>> Without knowing more about the specific platform, it seems to me like a
>> bug in the platform. If all indicators (not just configurations, but show
>> commands as well) say the packet should be leaving on X and it leaves on Y,
>> then I'm not sure what else it could be, besides a bug.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>> --
>> *From: *"David Bass" 
>> *To: *"Mike Hammett" 
>> *Cc: *"Matthew Huff" , "NANOG" 
>> *Sent: *Monday, April 3, 2023 9:12:52 PM
>>
>> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>
>> You said that they are seeing traffic from another upstream…are you
>> advertising the prefix to them?  Are you advertising their prefix to your
>> upstream?
>>
>> Looks like the route maps are involved in some dual redistribution…might
>> want to make sure everything is matching correctly, and being advertised
>> like you want.
>>
>> On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett  wrote:
>>
>>> I don't see any route-maps applied to interfaces, so there must not be
>>> any PBR going on. I only see ACLs, setting communities, setting local pref,
>>> etc. in the route maps that are applied to neighbors.
>>>
>>>
>>>
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>> --
>>

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

Via all mechanisms I could find in the router, it thinks the best path is the 
direct path, the packets just don't go that way. 


The in traffic isn't a concern at this time, just the out (from my 
perspective). 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "David Bass"  
To: "Mike Hammett"  
Cc: "Matthew Huff" , "NANOG"  
Sent: Tuesday, April 4, 2023 11:39:26 AM 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


If you are both connected to the same upstream, but the customer wants traffic 
destined to the upstream to go through you (in and out), then they need to do 
something on their devices to try and affect the inbound path to their AS. From 
the upstream carrier in question they’ll take the best path to a prefix, which 
direct connection is generally going to be preferred over a transit AS (basic 
BGP best path algorithm stuff) unless there is some manipulation of the prefix 
advertisement happening. 


To confirm the path being taken you should be able to do a few trace routes 
from various locations as well as use looking glasses. 


Now the sflow data is an entirely different thing to analyze. 



On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett < na...@ics-il.net > wrote: 









sh ip bgp neighbor advertised-routes shows the only routes being advertised to 
Y are the routes that should be advertised to them. I checked a variety of 
other peers and have the expected results. 






>From my perspective: 


Packets come in on port A, supposed to leave on port X, but they leave on port 
Y. All of the troubleshooting steps I've done on my own (or suggested by 
mailing lists) say the packets should be leaving on the desired port X. 




>From the customer's perspective, they're supposed to be coming from me on port 
>X, but they're arriving on port Y, another network . 


Port X in both scenarios is our direct connection, while port Y is a mutual 
upstream provider. 




Without knowing more about the specific platform, it seems to me like a bug in 
the platform. If all indicators (not just configurations, but show commands as 
well) say the packet should be leaving on X and it leaves on Y, then I'm not 
sure what else it could be, besides a bug. 





- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "David Bass" < davidbass...@gmail.com > 
To: "Mike Hammett" < na...@ics-il.net > 
Cc: "Matthew Huff" < mh...@ox.com >, "NANOG" < nanog@nanog.org > 
Sent: Monday, April 3, 2023 9:12:52 PM 




Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


You said that they are seeing traffic from another upstream…are you advertising 
the prefix to them? Are you advertising their prefix to your upstream? 


Looks like the route maps are involved in some dual redistribution…might want 
to make sure everything is matching correctly, and being advertised like you 
want. 



On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett < na...@ics-il.net > wrote: 





I don't see any route-maps applied to interfaces, so there must not be any PBR 
going on. I only see ACLs, setting communities, setting local pref, etc. in the 
route maps that are applied to neighbors. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Mike Hammett" < na...@ics-il.net > 
To: "Matthew Huff" < mh...@ox.com > 
Cc: "NANOG" < nanog@nanog.org > 
Sent: Monday, April 3, 2023 8:26:30 AM 



Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 




Only two VRFs, default and manangement. IIRC, everything I saw before mentioned 
the default VRF. 

I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig 
through this a bit to see what's going on. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Matthew Huff" < mh...@ox.com > 
To: "Mike Hammett" < na...@ics-il.net > 
Cc: "NANOG" < nanog@nanog.org > 
Sent: Monday, April 3, 2023 8:06:51 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

What about VRFs and/or policy based routing? 

switch-core1# show vrf 
VRF-Name VRF-ID State Reason 
default 1 Up -- 
management 2 Up -- 

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
Match clauses: 
interface: Ethernet1/33 
route-type: internal 
Set clauses: 
metric 4000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
Match clauses: 
interface: Ethernet1/34 
route-type: internal

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[David Bass]

If you are both connected to the same upstream, but the customer wants
traffic destined to the upstream to go through you (in and out), then they
need to do something on their devices to try and affect the inbound path to
their AS. From the upstream carrier in question they’ll take the best path
to a prefix, which direct connection is generally going to be preferred
over a transit AS (basic BGP best path algorithm stuff) unless there is
some manipulation of the prefix advertisement happening.

To confirm the path being taken you should be able to do a few trace routes
from various locations as well as use looking glasses.

Now the sflow data is an entirely different thing to analyze.

On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett  wrote:

>
> sh ip bgp neighbor advertised-routes shows the only routes being
> advertised to Y are the routes that should be advertised to them. I checked
> a variety of other peers and have the expected results.
>
>
>
> From my perspective:
>
> Packets come in on port A, supposed to leave on port X, but they leave on
> port Y. All of the troubleshooting steps I've done on my own (or suggested
> by mailing lists) say the packets should be leaving on the desired port X.
>
>
> From the customer's perspective, they're supposed to be coming from me on
> port X, but they're arriving on port Y, another network.
>
> Port X in both scenarios is our direct connection, while port Y is a
> mutual upstream provider.
>
>
> Without knowing more about the specific platform, it seems to me like a
> bug in the platform. If all indicators (not just configurations, but show
> commands as well) say the packet should be leaving on X and it leaves on Y,
> then I'm not sure what else it could be, besides a bug.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"David Bass" 
> *To: *"Mike Hammett" 
> *Cc: *"Matthew Huff" , "NANOG" 
> *Sent: *Monday, April 3, 2023 9:12:52 PM
>
> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> You said that they are seeing traffic from another upstream…are you
> advertising the prefix to them?  Are you advertising their prefix to your
> upstream?
>
> Looks like the route maps are involved in some dual redistribution…might
> want to make sure everything is matching correctly, and being advertised
> like you want.
>
> On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett  wrote:
>
>> I don't see any route-maps applied to interfaces, so there must not be
>> any PBR going on. I only see ACLs, setting communities, setting local pref,
>> etc. in the route maps that are applied to neighbors.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>> --
>> *From: *"Mike Hammett" 
>> *To: *"Matthew Huff" 
>> *Cc: *"NANOG" 
>> *Sent: *Monday, April 3, 2023 8:26:30 AM
>>
>> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>
>> Only two VRFs, default and manangement. IIRC, everything I saw before
>> mentioned the default VRF.
>>
>> I do see a ton of route-maps. It's mostly Greek to me, so I'll have to
>> dig through this a bit to see what's going on.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>> --
>> *From: *"Matthew Huff" 
>> *To: *"Mike Hammett" 
>> *Cc: *"NANOG" 
>> *Sent: *Monday, April 3, 2023 8:06:51 AM
>> *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>
>> What about VRFs and/or policy based routing?
>>
>> switch-core1# show vrf
>> VRF-Name   VRF-ID State   Reason
>>
>> default 1 Up  --
>>
>> management  2 Up  --
>>
>>
>> switch-core1# show route-map
>> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10
>>   Match clauses:
>> interface: Ethernet1/33
>> route-type: internal
>>   Set clauses:
>> metric 4000 10 255 1 1500
>> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20
>>   Match clauses:
>> interface: Ethernet1/34
>> route-type: int

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

sh ip bgp neighbor advertised-routes shows the only routes being advertised to 
Y are the routes that should be advertised to them. I checked a variety of 
other peers and have the expected results. 






>From my perspective: 


Packets come in on port A, supposed to leave on port X, but they leave on port 
Y. All of the troubleshooting steps I've done on my own (or suggested by 
mailing lists) say the packets should be leaving on the desired port X. 




>From the customer's perspective, they're supposed to be coming from me on port 
>X, but they're arriving on port Y, another network . 


Port X in both scenarios is our direct connection, while port Y is a mutual 
upstream provider. 




Without knowing more about the specific platform, it seems to me like a bug in 
the platform. If all indicators (not just configurations, but show commands as 
well) say the packet should be leaving on X and it leaves on Y, then I'm not 
sure what else it could be, besides a bug. 





- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "David Bass"  
To: "Mike Hammett"  
Cc: "Matthew Huff" , "NANOG"  
Sent: Monday, April 3, 2023 9:12:52 PM 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


You said that they are seeing traffic from another upstream…are you advertising 
the prefix to them? Are you advertising their prefix to your upstream? 


Looks like the route maps are involved in some dual redistribution…might want 
to make sure everything is matching correctly, and being advertised like you 
want. 



On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett < na...@ics-il.net > wrote: 





I don't see any route-maps applied to interfaces, so there must not be any PBR 
going on. I only see ACLs, setting communities, setting local pref, etc. in the 
route maps that are applied to neighbors. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Mike Hammett" < na...@ics-il.net > 
To: "Matthew Huff" < mh...@ox.com > 
Cc: "NANOG" < nanog@nanog.org > 
Sent: Monday, April 3, 2023 8:26:30 AM 



Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 




Only two VRFs, default and manangement. IIRC, everything I saw before mentioned 
the default VRF. 

I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig 
through this a bit to see what's going on. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Matthew Huff" < mh...@ox.com > 
To: "Mike Hammett" < na...@ics-il.net > 
Cc: "NANOG" < nanog@nanog.org > 
Sent: Monday, April 3, 2023 8:06:51 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

What about VRFs and/or policy based routing? 

switch-core1# show vrf 
VRF-Name VRF-ID State Reason 
default 1 Up -- 
management 2 Up -- 

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
Match clauses: 
interface: Ethernet1/33 
route-type: internal 
Set clauses: 
metric 4000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
Match clauses: 
interface: Ethernet1/34 
route-type: internal 
Set clauses: 
metric 4000 30 255 1 1500 
route-map rmap_static_to_eigrp, permit, sequence 10 
Match clauses: 
ip address prefix-lists: prefix_static_to_eigrp 
Set clauses: 
route-map rmap_static_to_eigrp_v6, permit, sequence 10 
Match clauses: 
ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp 
Set clauses: 



From: Mike Hammett < na...@ics-il.net > 
Sent: Monday, April 3, 2023 9:00 AM 
To: Matthew Huff < mh...@ox.com > 
Cc: NANOG < nanog@nanog.org > 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

It could be an sFlow bug, but I come at this from a reported problem and 
gathering data on that problem as opposed to looking at data for problems. 

The snmp if index reported by the Nexus matches the if index in ElastiFlow. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

 
From: "Matthew Huff"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Monday, April 3, 2023 7:50:08 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus? 





From: Mike Hammett  
Sent: Monday, April 3, 2023 8:47 AM 
To: Matthew Huff  
Cc: NANOG  
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[David Bass]

You said that they are seeing traffic from another upstream…are you
advertising the prefix to them?  Are you advertising their prefix to your
upstream?

Looks like the route maps are involved in some dual redistribution…might
want to make sure everything is matching correctly, and being advertised
like you want.

On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett  wrote:

> I don't see any route-maps applied to interfaces, so there must not be any
> PBR going on. I only see ACLs, setting communities, setting local pref,
> etc. in the route maps that are applied to neighbors.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Mike Hammett" 
> *To: *"Matthew Huff" 
> *Cc: *"NANOG" 
> *Sent: *Monday, April 3, 2023 8:26:30 AM
>
> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> Only two VRFs, default and manangement. IIRC, everything I saw before
> mentioned the default VRF.
>
> I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig
> through this a bit to see what's going on.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Matthew Huff" 
> *To: *"Mike Hammett" 
> *Cc: *"NANOG" 
> *Sent: *Monday, April 3, 2023 8:06:51 AM
> *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> What about VRFs and/or policy based routing?
>
> switch-core1# show vrf
> VRF-Name   VRF-ID State   Reason
>
> default 1 Up  --
>
> management  2 Up  --
>
>
> switch-core1# show route-map
> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10
>   Match clauses:
> interface: Ethernet1/33
> route-type: internal
>   Set clauses:
> metric 4000 10 255 1 1500
> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20
>   Match clauses:
> interface: Ethernet1/34
> route-type: internal
>   Set clauses:
> metric 4000 30 255 1 1500
> route-map rmap_static_to_eigrp, permit, sequence 10
>   Match clauses:
> ip address prefix-lists: prefix_static_to_eigrp
>   Set clauses:
> route-map rmap_static_to_eigrp_v6, permit, sequence 10
>   Match clauses:
> ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp
>   Set clauses:
>
>
>
> From: Mike Hammett 
> Sent: Monday, April 3, 2023 9:00 AM
> To: Matthew Huff 
> Cc: NANOG 
> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> It could be an sFlow bug, but I come at this from a reported problem and
> gathering data on that problem as opposed to looking at data for problems.
>
> The snmp if index reported by the Nexus matches the if index in ElastiFlow.
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> 
> From: "Matthew Huff" <mailto:mh...@ox.com>
> To: "Mike Hammett" <mailto:na...@ics-il.net>
> Cc: "NANOG" <mailto:nanog@nanog.org>
> Sent: Monday, April 3, 2023 7:50:08 AM
> Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
> SFlow misconfiguration or bug on either the nexus or the sflow monitor? On
> the monitor, can you verify that the snmp interfaces are mapped to the
> correct ones on the nexus?
>
>
>
>
>
> From: Mike Hammett <mailto:na...@ics-il.net>
> Sent: Monday, April 3, 2023 8:47 AM
> To: Matthew Huff <mailto:mh...@ox.com>
> Cc: NANOG <mailto:nanog@nanog.org>
> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> It shows the desired result.
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> 
> From: "Matthew Huff" <mailto:mh...@ox.com>
> To: "Mike Hammett" <mailto:na...@ics-il.net>, "NANOG"  nanog@nanog.org>
> Sent: Monday, April 3, 2023 5:38:23 AM
> Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> switch-core1# sh forwarding route x.x.x.x
>
> slot  1
> ===
>
>
> IPv4 routes for table default/base
>
>
> --+-+--+--

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

I don't see any route-maps applied to interfaces, so there must not be any PBR 
going on. I only see ACLs, setting communities, setting local pref, etc. in the 
route maps that are applied to neighbors. 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 


From: "Mike Hammett"  
To: "Matthew Huff"  
Cc: "NANOG"  
Sent: Monday, April 3, 2023 8:26:30 AM 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

Only two VRFs, default and manangement. IIRC, everything I saw before mentioned 
the default VRF. 

I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig 
through this a bit to see what's going on. 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 


From: "Matthew Huff"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Monday, April 3, 2023 8:06:51 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

What about VRFs and/or policy based routing? 

switch-core1# show vrf 
VRF-Name VRF-ID State Reason 
default 1 Up -- 
management 2 Up -- 

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
Match clauses: 
interface: Ethernet1/33 
route-type: internal 
Set clauses: 
metric 4000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
Match clauses: 
interface: Ethernet1/34 
route-type: internal 
Set clauses: 
metric 4000 30 255 1 1500 
route-map rmap_static_to_eigrp, permit, sequence 10 
Match clauses: 
ip address prefix-lists: prefix_static_to_eigrp 
Set clauses: 
route-map rmap_static_to_eigrp_v6, permit, sequence 10 
Match clauses: 
ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp 
Set clauses: 



From: Mike Hammett  
Sent: Monday, April 3, 2023 9:00 AM 
To: Matthew Huff  
Cc: NANOG  
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

It could be an sFlow bug, but I come at this from a reported problem and 
gathering data on that problem as opposed to looking at data for problems. 

The snmp if index reported by the Nexus matches the if index in ElastiFlow. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

 
From: "Matthew Huff" <mailto:mh...@ox.com> 
To: "Mike Hammett" <mailto:na...@ics-il.net> 
Cc: "NANOG" <mailto:nanog@nanog.org> 
Sent: Monday, April 3, 2023 7:50:08 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus? 





From: Mike Hammett <mailto:na...@ics-il.net> 
Sent: Monday, April 3, 2023 8:47 AM 
To: Matthew Huff <mailto:mh...@ox.com> 
Cc: NANOG <mailto:nanog@nanog.org> 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

It shows the desired result. 


- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

 
From: "Matthew Huff" <mailto:mh...@ox.com> 
To: "Mike Hammett" <mailto:na...@ics-il.net>, "NANOG" <mailto:nanog@nanog.org> 
Sent: Monday, April 3, 2023 5:38:23 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

switch-core1# sh forwarding route x.x.x.x 

slot 1 
=== 


IPv4 routes for table default/base 

--+-+--+-+-
 
Prefix | Next-hop | Interface | Labels | Partial Install 
--+-+--+-+-
 
x.x.x.x/24 x.x.x.250 Ethernet1/29 


switch-core1# show routing hash x.x.x.x y.y.y.y 
Load-share parameters used for software forwarding: 
load-share mode: address source-destination port source-destination 
Hash for VRF "default" 
Hashing to path *y.y.y.y Eth1/29 
For route: 
y.y.y.0/24, ubest/mbest: 1/0 
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal 




From: NANOG <mailto:nanog-bounces+mhuff=ox@nanog.org> On Behalf Of Mike 
Hammett 
Sent: Monday, April 3, 2023 1:21 AM 
To: NANOG <mailto:nanog@nanog.org> 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the def

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

Only two VRFs, default and manangement. IIRC, everything I saw before mentioned 
the default VRF. 

I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig 
through this a bit to see what's going on. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Matthew Huff"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Monday, April 3, 2023 8:06:51 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

What about VRFs and/or policy based routing? 

switch-core1# show vrf 
VRF-Name VRF-ID State Reason 
default 1 Up -- 
management 2 Up -- 

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
Match clauses: 
interface: Ethernet1/33 
route-type: internal 
Set clauses: 
metric 4000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
Match clauses: 
interface: Ethernet1/34 
route-type: internal 
Set clauses: 
metric 4000 30 255 1 1500 
route-map rmap_static_to_eigrp, permit, sequence 10 
Match clauses: 
ip address prefix-lists: prefix_static_to_eigrp 
Set clauses: 
route-map rmap_static_to_eigrp_v6, permit, sequence 10 
Match clauses: 
ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp 
Set clauses: 



From: Mike Hammett  
Sent: Monday, April 3, 2023 9:00 AM 
To: Matthew Huff  
Cc: NANOG  
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

It could be an sFlow bug, but I come at this from a reported problem and 
gathering data on that problem as opposed to looking at data for problems. 

The snmp if index reported by the Nexus matches the if index in ElastiFlow. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

 
From: "Matthew Huff" <mailto:mh...@ox.com> 
To: "Mike Hammett" <mailto:na...@ics-il.net> 
Cc: "NANOG" <mailto:nanog@nanog.org> 
Sent: Monday, April 3, 2023 7:50:08 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus? 





From: Mike Hammett <mailto:na...@ics-il.net> 
Sent: Monday, April 3, 2023 8:47 AM 
To: Matthew Huff <mailto:mh...@ox.com> 
Cc: NANOG <mailto:nanog@nanog.org> 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

It shows the desired result. 


- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

 
From: "Matthew Huff" <mailto:mh...@ox.com> 
To: "Mike Hammett" <mailto:na...@ics-il.net>, "NANOG" <mailto:nanog@nanog.org> 
Sent: Monday, April 3, 2023 5:38:23 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

switch-core1# sh forwarding route x.x.x.x 

slot 1 
=== 


IPv4 routes for table default/base 

--+-+--+-+-
 
Prefix | Next-hop | Interface | Labels | Partial Install 
--+-+--+-+-
 
x.x.x.x/24 x.x.x.250 Ethernet1/29 


switch-core1# show routing hash x.x.x.x y.y.y.y 
Load-share parameters used for software forwarding: 
load-share mode: address source-destination port source-destination 
Hash for VRF "default" 
Hashing to path *y.y.y.y Eth1/29 
For route: 
y.y.y.0/24, ubest/mbest: 1/0 
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal 




From: NANOG <mailto:nanog-bounces+mhuff=ox@nanog.org> On Behalf Of Mike 
Hammett 
Sent: Monday, April 3, 2023 1:21 AM 
To: NANOG <mailto:nanog@nanog.org> 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

What about VRFs and/or policy based routing?  

switch-core1# show vrf
VRF-Name   VRF-ID State   Reason
default 1 Up  --
management  2 Up  --

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
  Match clauses:
interface: Ethernet1/33 
route-type: internal 
  Set clauses:
metric 4000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
  Match clauses:
interface: Ethernet1/34 
route-type: internal 
  Set clauses:
metric 4000 30 255 1 1500 
route-map rmap_static_to_eigrp, permit, sequence 10 
  Match clauses:
ip address prefix-lists: prefix_static_to_eigrp 
  Set clauses:
route-map rmap_static_to_eigrp_v6, permit, sequence 10 
  Match clauses:
ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp 
  Set clauses:



From: Mike Hammett  
Sent: Monday, April 3, 2023 9:00 AM
To: Matthew Huff 
Cc: NANOG 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

It could be an sFlow bug, but I come at this from a reported problem and 
gathering data on that problem as opposed to looking at data for problems.

The snmp if index reported by the Nexus matches the if index in ElastiFlow.




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Matthew Huff" <mailto:mh...@ox.com>
To: "Mike Hammett" <mailto:na...@ics-il.net>
Cc: "NANOG" <mailto:nanog@nanog.org>
Sent: Monday, April 3, 2023 7:50:08 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus?
 
 
 
 
 
From: Mike Hammett <mailto:na...@ics-il.net> 
Sent: Monday, April 3, 2023 8:47 AM
To: Matthew Huff <mailto:mh...@ox.com>
Cc: NANOG <mailto:nanog@nanog.org>
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
 
It shows the desired result.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com
 

From: "Matthew Huff" <mailto:mh...@ox.com>
To: "Mike Hammett" <mailto:na...@ics-il.net>, "NANOG" <mailto:nanog@nanog.org>
Sent: Monday, April 3, 2023 5:38:23 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix            | Next-hop                                | Interface         
   | Labels          | Partial Install 
--+-+--+-+-
x.x.x.x/24      x.x.x.250                            Ethernet1/29        


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
    *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG <mailto:nanog-bounces+mhuff=ox....@nanog.org> On Behalf Of Mike 
Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG <mailto:nanog@nanog.org>
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com
 

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

It could be an sFlow bug, but I come at this from a reported problem and 
gathering data on that problem as opposed to looking at data for problems. 


The snmp if index reported by the Nexus matches the if index in ElastiFlow. 








- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Matthew Huff"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Monday, April 3, 2023 7:50:08 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 



SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus? 







From: Mike Hammett  
Sent: Monday, April 3, 2023 8:47 AM 
To: Matthew Huff  
Cc: NANOG  
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


It shows the desired result. 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -


From: "Matthew Huff" < mh...@ox.com > 
To: "Mike Hammett" < na...@ics-il.net >, "NANOG" < nanog@nanog.org > 
Sent: Monday, April 3, 2023 5:38:23 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

switch-core1# sh forwarding route x.x.x.x 

slot 1 
=== 


IPv4 routes for table default/base 

--+-+--+-+-
 
Prefix | Next-hop | Interface | Labels | Partial Install 
--+-+--+-+-
 
x.x.x.x/24 x.x.x.250 Ethernet1/29 


switch-core1# show routing hash x.x.x.x y.y.y.y 
Load-share parameters used for software forwarding: 
load-share mode: address source-destination port source-destination 
Hash for VRF "default" 
Hashing to path *y.y.y.y Eth1/29 
For route: 
y.y.y.0/24, ubest/mbest: 1/0 
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal 




From: NANOG < nanog-bounces+mhuff=ox@nanog.org > On Behalf Of Mike Hammett 
Sent: Monday, April 3, 2023 1:21 AM 
To: NANOG < nanog@nanog.org > 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus?





From: Mike Hammett 
Sent: Monday, April 3, 2023 8:47 AM
To: Matthew Huff 
Cc: NANOG 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

It shows the desired result.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Matthew Huff" mailto:mh...@ox.com>>
To: "Mike Hammett" mailto:na...@ics-il.net>>, "NANOG" 
mailto:nanog@nanog.org>>
Sent: Monday, April 3, 2023 5:38:23 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix| Next-hop| Interface 
   | Labels  | Partial Install
--+-+--+-+-
x.x.x.x/24  x.x.x.250Ethernet1/29


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG 
mailto:nanog-bounces+mhuff=ox@nanog.org>>
 On Behalf Of Mike Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG mailto:nanog@nanog.org>>
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface?


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

What started this investigation was a client complained of traffic coming from 
another upstream instead of our direct connection. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mike Hammett"  
To: "NANOG"  
Sent: Monday, April 3, 2023 12:21:29 AM 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

It shows the desired result. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Matthew Huff"  
To: "Mike Hammett" , "NANOG"  
Sent: Monday, April 3, 2023 5:38:23 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

switch-core1# sh forwarding route x.x.x.x 

slot 1 
=== 


IPv4 routes for table default/base 

--+-+--+-+-
 
Prefix | Next-hop | Interface | Labels | Partial Install 
--+-+--+-+-
 
x.x.x.x/24 x.x.x.250 Ethernet1/29 


switch-core1# show routing hash x.x.x.x y.y.y.y 
Load-share parameters used for software forwarding: 
load-share mode: address source-destination port source-destination 
Hash for VRF "default" 
Hashing to path *y.y.y.y Eth1/29 
For route: 
y.y.y.0/24, ubest/mbest: 1/0 
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal 




From: NANOG  On Behalf Of Mike Hammett 
Sent: Monday, April 3, 2023 1:21 AM 
To: NANOG  
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

It shows the desired result. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jay Hennigan"  
To: nanog@nanog.org 
Sent: Monday, April 3, 2023 1:02:42 AM 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

On 4/2/23 22:21, Mike Hammett wrote: 
> We have a Nexus 3064 that is setup with partial BGP tables and is 
> routing based on that. 
> 
> 
> I've done a show ip bgp for an IP of interest and it has an expected 
> next hop IP. I show ip arp on that next hop IP and it has the expected 
> interface. 
> 
> 
> However, sFlows show the packets leaving on a different interface, the 
> one that would carry the default route for routes not otherwise known. 
> 
> 
> If the next hop IP is expected and the ARP of that next hop IP is 
> expected, why are packets leaving out an unexpected interface? 

Another route to the destination from a routing protocol with a lower 
distance? 

What does "show ip route [destination]" look like? 

-- 
Jay Hennigan - j...@west.net 
Network Engineering - CCIE #7880 
503 897-8550 - WB6RDV 

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix| Next-hop| Interface 
   | Labels  | Partial Install 
--+-+--+-+-
x.x.x.x/24  x.x.x.250Ethernet1/29


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG  On Behalf Of Mike Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Jay Hennigan]

On 4/2/23 22:21, Mike Hammett wrote:
We have a Nexus 3064 that is setup with partial BGP tables and is 
routing based on that.



I've done a show ip bgp for an IP of interest and it has an expected 
next hop IP. I show ip arp on that next hop IP and it has the expected 
interface.



However, sFlows show the packets leaving on a different interface, the 
one that would carry the default route for routes not otherwise known.



If the next hop IP is expected and the ARP of that next hop IP is 
expected, why are packets leaving out an unexpected interface?


Another route to the destination from a routing protocol with a lower 
distance?


What does "show ip route [destination]" look like?

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Jay Hennigan]

On 4/2/23 22:21, Mike Hammett wrote:
We have a Nexus 3064 that is setup with partial BGP tables and is 
routing based on that.



I've done a show ip bgp for an IP of interest and it has an expected 
next hop IP. I show ip arp on that next hop IP and it has the expected 
interface.



However, sFlows show the packets leaving on a different interface, the 
one that would carry the default route for routes not otherwise known.


What does traceroute to that IP show?


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Joshua Miller]

Is the BGP route getting installed into the rib?




On Mon, Apr 3, 2023, 01:24 Mike Hammett  wrote:

> We have a Nexus 3064 that is setup with partial BGP tables and is routing
> based on that.
>
>
> I've done a show ip bgp for an IP of interest and it has an expected next
> hop IP. I show ip arp on that next hop IP and it has the expected interface.
>
>
>
> However, sFlows show the packets leaving on a different interface, the one
> that would carry the default route for routes not otherwise known.
>
>
> If the next hop IP is expected and the ARP of that next hop IP is
> expected, why are packets leaving out an unexpected interface?
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>

Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Re: Cisco Nexus Odd sFlow Implementation

[Raymond Burkholder]

On 3/20/23 09:43, Mike Hammett wrote:

But then I'm storing and processing twice as much data as I need.





I would say somewhere between 0 - 100% overhead.  Depending upon traffic.

Recall that a flow is packet delivery in one direction only.  An end to 
end flow is an ingress interface with various l2 decorations along with 
an egress interface with its various l2 decorations (which many flavours 
of sflow will report).  So you need to record flows in both directions - 
from point of entry to point of exit.  In Cisco when I have the ability 
to turn on ingress and/or egress.  I turn on both so I can record this 
detail.


To fully record traffic, you need this in both directions.  And recall, 
multihome means that traffic delivery may be asymmetrical across 
interfaces.  So diagnostics will require this full flow info in both 
directions.


And as the disparaged marketing material indicating that a 1G interface 
carries 2G worth of traffic, it isn't a lie.


All this to say that no, you aren't processing twice as much data as you 
need.  You need all that, in both directions, to properly generate 
additional detail reports.






*From: *"Raymond Burkholder" 
*To: *nanog@nanog.org
*Sent: *Monday, March 20, 2023 10:16:37 AM
*Subject: *Re: Cisco Nexus Odd sFlow Implementation


On 3/20/23 08:55, Mike Hammett wrote:

Cisco is sending the in and out packets in their sFlow
implementation on their Nexus switches. Obviously, this double
counts the information.

Are there solutions that process those flows and remove one of
those directions?


Wouldn't you just do that in your reporting?  The report engine 
usually has options for interface, ingress/egress, prefix range, 
protocol, 


Then when you need to aggregate for a specific interface for 
additional troubleshooting, you have all the information for when you 
need it.


Yes, your aggregate numbers for total packets transiting is double, 
but I generally have specific reports for specific traffic patterns I 
focus on anyway.


Raymond Burkholder
One Unified Net Limited




-
Mike Hammett
Intelligent Computing Solutions <http://www.ics-il.com/>

<https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
Midwest Internet Exchange <http://www.midwest-ix.com/>

<https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
The Brothers WISP <http://www.thebrotherswisp.com/>

<https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

Re: Cisco Nexus Odd sFlow Implementation

[Mike Hammett]

But then I'm storing and processing twice as much data as I need. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Raymond Burkholder"  
To: nanog@nanog.org 
Sent: Monday, March 20, 2023 10:16:37 AM 
Subject: Re: Cisco Nexus Odd sFlow Implementation 



On 3/20/23 08:55, Mike Hammett wrote: 



Cisco is sending the in and out packets in their sFlow implementation on their 
Nexus switches. Obviously, this double counts the information. 


Are there solutions that process those flows and remove one of those 
directions? 



Wouldn't you just do that in your reporting? The report engine usually has 
options for interface, ingress/egress, prefix range, protocol,  

Then when you need to aggregate for a specific interface for additional 
troubleshooting, you have all the information for when you need it. 

Yes, your aggregate numbers for total packets transiting is double, but I 
generally have specific reports for specific traffic patterns I focus on 
anyway. 

Raymond Burkholder 
One Unified Net Limited 









- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

Re: Cisco Nexus Odd sFlow Implementation

[Raymond Burkholder]

On 3/20/23 08:55, Mike Hammett wrote:
Cisco is sending the in and out packets in their sFlow implementation 
on their Nexus switches. Obviously, this double counts the information.


Are there solutions that process those flows and remove one of those 
directions?


Wouldn't you just do that in your reporting?  The report engine usually 
has options for interface, ingress/egress, prefix range, protocol, 


Then when you need to aggregate for a specific interface for additional 
troubleshooting, you have all the information for when you need it.


Yes, your aggregate numbers for total packets transiting is double, but 
I generally have specific reports for specific traffic patterns I focus 
on anyway.


Raymond Burkholder
One Unified Net Limited





-
Mike Hammett
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

Cisco Nexus Odd sFlow Implementation

[Mike Hammett]

Cisco is sending the in and out packets in their sFlow implementation on their 
Nexus switches. Obviously, this double counts the information. 


Are there solutions that process those flows and remove one of those 
directions? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

Re: cisco nexus 9000 cctrl ERROR

[Scott Weeks]

--- bs...@teamonesolutions.com wrote:From: Brandon Svec Anyone can create a Cisco login.  I would do that and check the bug tracking tool.  I did a quick search on your error message and came up with this:I was unaware that anyone could do that as I have been away from cisco for a good while now.  Thank you both for the quick response.  It helped a lot!scott

Re: cisco nexus 9000 cctrl ERROR

[Brandon Svec]

Anyone can create a Cisco login.  I would do that and check the bug tracking 
tool.  I did a quick search on your error message and came up with this:

Bug Search <https://bst.cloudapps.cisco.com/bugsearch/>CSCvp48462
Help <>  |  Feedback <>
NXA-PDC-1100W-PI PSU fail log F0411,F0413 in N9K-C9336C-FX2
CSCvp48462
Description <>
Symptom:
NXA-PDC-1100W-PI PSU will occur F0411.F0413 PSU fail log in N9K-C9336C-FX2.

F0411 Power supply failed
F0413 power supply missing

it is not real PSU fail, but only log issue.
in show system internal kernel messages. we can see NACK error on PSU sensor.

[1274386.802885] cctrl ERROR: cctrl_wait_for_pmbio_busy_status@ 35:NACK error 
tmp_data 0x3b58d00 mask = 0x8000 
[1274386.802885] 
[1274386.812815] cctrl ERROR: cctrl2_delay_pmbio_read@ 277:final busy wait 
check failed pid = 8701 (pfmclnt) cs_reg 0x270 win_id 0 dev_addr 0x5a off 0x8d
[1274386.812818] cctrl ERROR: cctrl2_delay_pmbio_read@ 278:write data 
0x82b58d00 read data 0x82b58d00 tmp_maks 0x600 dlen = 2
[1274386.812822] CPU: 1 PID: 8701 Comm: pfmclnt Tainted: PW  O 
3.14.62.0.0insieme-0 #1
[1274386.812824]   88055cd37c18 817ead67 
fffb
[1274386.812829]  82b58d00 88055cd37ca8 c25ad00a 
0002
[1274386.812832]  005a 8805008d c25a5c17 
c9001035c421
[1274386.812836] Call Trace:
[1274386.812844]  [] dump_stack+0x68/0x91
[1274386.812865]  [] cctrl2_delay_pmbio_read+0x28a/0x350 [klm_cctrli2]
[1274386.812873]  [] ? cctrl_read_reg2+0x177/0x190 [klm_cctrli2]
[1274386.812877]  [] ? strstr+0x37/0x90
[1274386.812886]  [] cctrl_psu_handle_sensor+0x16d/0x1e0 [klm_cctrli2]
[1274386.812899]  [] cctrl_tor_scrimshaw_sensor_op+0x1fd/0x240 [klm_cctrli2]
[1274386.812910]  [] sys_srvc_cctrl_sensor_op+0x80/0x190 [klm_cctrli2]
[1274386.812918]  [] sysServices+0x22b/0x880 [klm_sse]
[1274386.812922]  [] ? free_debug_processing+0x17d/0x1c1
[1274386.812928]  [] ? ring_buffer_lock_reserve+0xb3/0xf0
[1274386.812933]  [] sse_compat_ioctl+0x102/0x120 [klm_sse]
[1274386.812938]  [] ? trace_buffer_unlock_commit+0x43/0x60
[1274386.812944]  [] compat_sys_ioctl+0x1dc/0x11f0
[1274386.812948]  [] ? syscall_trace_enter+0x162/0x1b0
[1274386.812951]  [] ia32_do_call+0x13/0x13

Conditions:
NXA-PDC-1100W-PI PSU in N9K-C9336C-FX2

Workaround:
NA

Further Problem Description:
NA

Customer Visible


Notifications

Save Bug

Open Support Case
Was the description about this Bug Helpful?(0)
Details <>
Last Modified:
Jan 13,2020
Status:
Open
Severity:
2 Severe
Product:(1)
Cisco Nexus 9000 Series Switches
Support Cases:
3

> On Jan 17, 2020, at 11:08 AM, Scott Weeks  wrote:
> 
> 
> 
> I don't have a login to cisco to find out what this 
> is and I'm having trouble finding anything about it 
> in search engines that doesn't require a login to 
> cisco.  I guess they only want certain folks to know 
> about it... :(  Does anyone know anything about this 
> and can explain it to me?  If not, I'll go join 
> cisco-nsp and ask there.
> 
> 
> %KERN-3-SYSTEM_MSG: [65292299.903992]  - kernel
> 
> %KERN-3-SYSTEM_MSG: [66730914.839059] cctrl ERROR: 
> cctrl_wait_for_pmbio_busy_status NACK error tmp_data 3b19600 - kernel
> 
> %KERN-3-SYSTEM_MSG: [67511639.312284] cctrl ERROR: 
> cctrl_wait_for_pmbio_busy_status NACK error tmp_data 1b18100 - kernel
> 
> 
> Those last numbers after tmp_data repeat over and over.
> 
> Thanks!
> scott

cisco nexus 9000 cctrl ERROR

[Scott Weeks]

I don't have a login to cisco to find out what this 
is and I'm having trouble finding anything about it 
in search engines that doesn't require a login to 
cisco.  I guess they only want certain folks to know 
about it... :(  Does anyone know anything about this 
and can explain it to me?  If not, I'll go join 
cisco-nsp and ask there.


%KERN-3-SYSTEM_MSG: [65292299.903992]  - kernel

%KERN-3-SYSTEM_MSG: [66730914.839059] cctrl ERROR: 
cctrl_wait_for_pmbio_busy_status NACK error tmp_data 3b19600 - kernel

%KERN-3-SYSTEM_MSG: [67511639.312284] cctrl ERROR: 
cctrl_wait_for_pmbio_busy_status NACK error tmp_data 1b18100 - kernel


Those last numbers after tmp_data repeat over and over.

Thanks!
scott

Re: Cisco Nexus 93180YC Switch Feedback

[Tim Stevenson]

At 03:48 PM 8/8/2017  Tuesday, Simon Lockhart opined:

On Tue Jul 25, 2017 at 08:31:54PM -0700, Tim Stevenson wrote:
> tstevens-92160yc-1# sh int cap | beg 1/49 | eg Eth|Speed
> Ethernet1/49 >   Speed: 4
> Ethernet1/50 >   Speed: 1000,1,25000,4,5,10
> Ethernet1/51 >   Speed: 4
> Ethernet1/52 >   Speed: 1000,1,25000,4,5,10
> Ethernet1/53 >   Speed: 4
> Ethernet1/54 >   Speed: 4

Are you sure?



I'll claim correctness on a technicality ;)

Since the original statement was: "Notably only four of the six 100G 
ports on the 92160YCX will run at 100G, leaving two at 40G."


In the default mode, you get 4 x 40G + 2 x 100G. Good point that 
there is an optional mode which is 4 x 100G - but there is no 40G in 
that mode, ports 53-54 are disabled in that mode.


Thanks,
Tim




xsw-01.ixn# show ver | incl Nexus9
  cisco Nexus9000 C92160YC-X chassis

xsw-01.ixn# sh int cap | beg 1/49 | eg Eth|Speed
Ethernet1/49
  Speed: 4,10
Ethernet1/50
  Speed: 1000,1,25000,4,5,10
Ethernet1/51
  Speed: 1000,1,25000,4,5,10
Ethernet1/52
  Speed: 1000,1,25000,4,5,10

xsw-01.ixn# show run | incl portmode
hardware profile portmode 48x25G+4x100G

xsw-01.ixn(config)# hardware profile portmode ?
  48x25g+2x100g+4x40g  48x25G+2x100G+4x40G port mode
  48x25g+4x100g48x25G+4x100G port mode

You can configure how the ports present themselves...

Simon






Tim Stevenson, tstev...@cisco.com
Routing & Switching CCIE #5561
Distinguished Engineer, Technical Marketing
Data Center Switching
Cisco - http://www.cisco.com
+1(408)526-6759

Re: Cisco Nexus 93180YC Switch Feedback

[Brandon Butterworth]

On Tue Jul 25, 2017 at 08:31:54PM -0700, Tim Stevenson wrote:
> Actually, only two.
> 
> tstevens-92160yc-1# sh int cap | beg 1/49 | eg Eth|Speed
> Ethernet1/49
>   Speed: 4
> Ethernet1/50
>   Speed: 1000,1,25000,4,5,10
> Ethernet1/51
>   Speed: 4
> Ethernet1/52
>   Speed: 1000,1,25000,4,5,10
> Ethernet1/53
>   Speed: 4
> Ethernet1/54
>   Speed: 4
> tstevens-92160yc-1#

same box, cisco Nexus9000 C92160YC-X

is2.ed# sh int cap | beg 1/49 | eg Eth|Speed
Ethernet1/49
  Speed: 4,10
Ethernet1/50
  Speed: 1000,1,25000,4,5,10
Ethernet1/51
  Speed: 1000,1,25000,4,5,10
Ethernet1/52
  Speed: 1000,1,25000,4,5,10

You have to tell it you want more 100G with fewer ports
hardware profile portmode 48x25G+4x100G

brandon

Re: Cisco Nexus 93180YC Switch Feedback

[Simon Lockhart]

On Tue Jul 25, 2017 at 08:31:54PM -0700, Tim Stevenson wrote:
> tstevens-92160yc-1# sh int cap | beg 1/49 | eg Eth|Speed
> Ethernet1/49 >   Speed: 4
> Ethernet1/50 >   Speed: 1000,1,25000,4,5,10
> Ethernet1/51 >   Speed: 4
> Ethernet1/52 >   Speed: 1000,1,25000,4,5,10
> Ethernet1/53 >   Speed: 4
> Ethernet1/54 >   Speed: 4

Are you sure?

xsw-01.ixn# show ver | incl Nexus9
  cisco Nexus9000 C92160YC-X chassis 

xsw-01.ixn# sh int cap | beg 1/49 | eg Eth|Speed
Ethernet1/49
  Speed: 4,10
Ethernet1/50
  Speed: 1000,1,25000,4,5,10
Ethernet1/51
  Speed: 1000,1,25000,4,5,10
Ethernet1/52
  Speed: 1000,1,25000,4,5,10

xsw-01.ixn# show run | incl portmode
hardware profile portmode 48x25G+4x100G

xsw-01.ixn(config)# hardware profile portmode ?
  48x25g+2x100g+4x40g  48x25G+2x100G+4x40G port mode
  48x25g+4x100g48x25G+4x100G port mode

You can configure how the ports present themselves...

Simon

Re: Cisco Nexus 93180YC Switch Feedback

[Tim Stevenson]

At 07:39 PM 7/25/2017  Tuesday, Bill Woodcock opined:

> On Jul 25, 2017, at 7:25 PM, James Braunegg 
 wrote:

>
> Dear Nanog
>
> Just wondering if anyone is using the Cisco 
Nexus 93180YC with the LAN enterprise license 
and has any honest feedback about this 
platform, especially if you are using VXLAN 
routing and eVPN, would love to hear from you.

>
> For those that don't know the Cisco Nexus 
93180YC is a 1RU switch with 48 x 25gbit ports 
SFP+ ports (1G, 10G or 25G) + 6 x 100gbit QSFP uplink ports. (40G, 50G or 100G)


We’ve started testing them, as well as the 
92160YCX and C92300YC, for IXP use.  Notably 
only four of the six 100G ports on the 92160YCX 
will run at 100G, leaving two at 40G.



Actually, only two.

tstevens-92160yc-1# sh int cap | beg 1/49 | eg Eth|Speed
Ethernet1/49
  Speed: 4
Ethernet1/50
  Speed: 1000,1,25000,4,5,10
Ethernet1/51
  Speed: 4
Ethernet1/52
  Speed: 1000,1,25000,4,5,10
Ethernet1/53
  Speed: 4
Ethernet1/54
  Speed: 4
tstevens-92160yc-1#


All 6 on 93180YC EX/FX are 100G capable.

Tim




Anybody had any luck sourcing 2x25G NICs for Cisco UCS servers?

-Bill











Tim Stevenson, tstev...@cisco.com
Routing & Switching CCIE #5561
Distinguished Engineer, Technical Marketing
Data Center Switching
Cisco - http://www.cisco.com
+1(408)526-6759

Re: Cisco Nexus 93180YC Switch Feedback

[Bill Woodcock]

> On Jul 25, 2017, at 7:25 PM, James Braunegg  
> wrote:
> 
> Dear Nanog
> 
> Just wondering if anyone is using the Cisco Nexus 93180YC with the LAN 
> enterprise license and has any honest feedback about this platform, 
> especially if you are using VXLAN routing and eVPN, would love to hear from 
> you.
> 
> For those that don't know the Cisco Nexus 93180YC is a 1RU switch with 48 x 
> 25gbit ports SFP+ ports (1G, 10G or 25G) + 6 x 100gbit QSFP uplink ports. 
> (40G, 50G or 100G)

We’ve started testing them, as well as the 92160YCX and C92300YC, for IXP use.  
Notably only four of the six 100G ports on the 92160YCX will run at 100G, 
leaving two at 40G.

Anybody had any luck sourcing 2x25G NICs for Cisco UCS servers?

-Bill






signature.asc
Description: Message signed with OpenPGP

Cisco Nexus 93180YC Switch Feedback

[James Braunegg]

Dear Nanog

Just wondering if anyone is using the Cisco Nexus 93180YC with the LAN 
enterprise license and has any honest feedback about this platform, especially 
if you are using VXLAN routing and eVPN, would love to hear from you.

For those that don't know the Cisco Nexus 93180YC is a 1RU switch with 48 x 
25gbit ports SFP+ ports (1G, 10G or 25G) + 6 x 100gbit QSFP uplink ports. (40G, 
50G or 100G)

Kindest Regards

James Braunegg



[cid:image001.png@01D280A4.01865B60]

1300 769 972 / 0488 997 207

ja...@micron21.com<mailto:ja...@micron21.com>

www.micron21.com/<http://www.micron21.com/>


[cid:image002.png@01D280A4.01865B60]<http://www.micron21.com/>


[cid:image003.png@01D280A4.01865B60]<https://www.facebook.com/micron21/>

[cid:image004.png@01D280A4.01865B60]<https://twitter.com/micron21>

Follow us on Twitter<https://twitter.com/micron21> for important service and 
system updates.


This message is intended for the addressee named above. It may contain 
privileged or confidential information. If you are not the intended recipient 
of this message you must not use, copy, distribute or disclose it to anyone 
other than the addressee. If you have received this message in error please 
return the message to the sender by replying to it and then delete the message 
from your computer.

Re: Cisco Nexus vPC-VOIP Issues

[Anurag Bhatia]

Hi Santosh


Likely it's disabled arp across broadcast (assuming both servers are on
same broadcast domain). One can comment on it after looking at config of
the port. I have seen similar case in some hosting providers who run shared
vlans across customers and they block direct traffic among those servers.
They usually put a static route of that pool towards gateway.

So e.g you have router on 10.10.10.1 and server 1 on 10.10.10.10, server 2
on 10.10.10.20. Now if direct layer 2 traffic is not allowed by tweaking
broadcast domain, then you can route traffic from say server 1
(10.10.10.10) needs to speak to server 2 (10.10.10.20) then you can put
10.10.10.0/24 static via 10.10.10.1. Whether or not that's a good idea
depends heavily on the use case.

I hope this will help.


On Mon 15 Aug, 2016, 17:26 nico nanog,  wrote:

> Hello,
>
> I cannot see any image in attachment.
>
> If you can ping from outside and not between them, wild guess it's not a
> L2 pbm.
>
> Are you able to see the arp of srv2 from srv1 ( and vice-versa )
>
> Without more info ( or it's maybe on the image I cannot see ) I would
> look in ACL somewhere/firewall on srv
>
>
> Rgd,
> Nico
>
>
> On 08/14/2016 11:59 PM, sathish kumar Ippani wrote:
> > Hello All,
> >
> > Thank you all in advance.
> >
> > We have connected two nexus 3048 Switches and two l2 Switches as below
> > using vPC and LACP.
> >
> > We have not seen any issues apart from one of VOIP server connected to
> > Switch 1 has lost access to VOIP Server connected Switch 2 and vice
> versa.
> >
> > Where I am able to ping both from Global. Can you please let me know what
> > is went wrong here.
> >
> >
> > [image: Inline image 2]
> >
>
>
> --
> Try and fail but never fail to try
>
> --
Anurag Bhatia
http://anuragbhatia.com

Re: Cisco Nexus vPC-VOIP Issues

[nico nanog]

Hello,

I cannot see any image in attachment.

If you can ping from outside and not between them, wild guess it's not a 
L2 pbm.


Are you able to see the arp of srv2 from srv1 ( and vice-versa )

Without more info ( or it's maybe on the image I cannot see ) I would 
look in ACL somewhere/firewall on srv



Rgd,
Nico


On 08/14/2016 11:59 PM, sathish kumar Ippani wrote:

Hello All,

Thank you all in advance.

We have connected two nexus 3048 Switches and two l2 Switches as below
using vPC and LACP.

We have not seen any issues apart from one of VOIP server connected to
Switch 1 has lost access to VOIP Server connected Switch 2 and vice versa.

Where I am able to ping both from Global. Can you please let me know what
is went wrong here.


[image: Inline image 2]




--
Try and fail but never fail to try

Cisco Nexus vPC-VOIP Issues

[sathish kumar Ippani]

Hello All,

Thank you all in advance.

We have connected two nexus 3048 Switches and two l2 Switches as below
using vPC and LACP.

We have not seen any issues apart from one of VOIP server connected to
Switch 1 has lost access to VOIP Server connected Switch 2 and vice versa.

Where I am able to ping both from Global. Can you please let me know what
is went wrong here.


[image: Inline image 2]

-- 
With Regards,

Sathish Kumar Ippani
9177166040

Re: Cisco Nexus

[Ray Soucy]

I have a small setup, Nexus 2 x 5596UP + 12 x 2248TP FEX, 2 x B22DELL,
2 x B22HP, 1 x C2248PQ-10GE.

Been using this setup since 2012, so it's getting a bit long in the
tooth.  It's in an Active-Active setup because there wasn't much
guidance at the time on which way to go.  There are some restrictions
with an AA setup you probably want to avoid.  We currently don't do
any FCoE because we're mostly a NetApp and NFS environment.

The performance and stability have been great.

It works well for a traditional environment with a lot of wired ports
to stand-alone servers.  If you do a lot with virtualization it's not
a great solution.  You really want to avoid connecting VM host servers
to FEX ports because of all the restrictions that come with it.  One
restriction that's a real PITA for me right now is that a FEX port
can't be a promiscuous trunk port if you're using PVLAN.

Using config-sync has been a lot of trouble.  There are a lot of
actions that will verify OK but then fail.  The result is that things
are partially configured and the whole system gets out of sync not
letting you make any other changes; the fix is having to manually go
in to each switch to try and get the configuration to match (which
requires comparing the running-configuration to the running
switch-profile configuration).


On Mon, Feb 2, 2015 at 1:17 PM, Herman, Anthony
 wrote:
> Nanog,
>
> I would like to poll the collective for experiences both positive and 
> negative with the Nexus line. More specifically I am interested in hearing 
> about FEX with N2K at the ToR and if this has indeed made any impact on Opex 
> as well as non-obvious shortcomings to using the fabric extenders. Also if 
> anyone is using any of the Nexus line for I/O convergence (FCoE) I would be 
> interested in hearing your experience with this as well.
>
> Thank you in advance,
>
> -A



-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net

Re: Cisco Nexus

[George Herbert]

> Brandon Ewing  wrote:
> 
>> David Bass wrote:
>> The n2k ToR is not a great design for user or storage interfaces if most of 
>> your traffic is east/west.  It is great as a low cost ilo/drac/choose your 
>> oob port, or if most of your traffic is north/south.  Biggest thing to 
>> remember is that it is not a switch, and has limitations such as not 
>> connecting other switches to it. Like anything else you have to understand 
>> the product so that you don't engineer something that it wasn't designed to 
>> do.
> 
> And remember -- The Nexus 2K performs absolutely ZERO local switching -- all
> frames received from client ports are just copied to the upstream device, so
> it can handle the frame/packet forwarding logic.  
>


What this really does is force you to consider how much of your East-West is 
rack-local, versus off rack.

Rack-local-heavy hurts as badly as off rack, with FEX.

If you want to / can localize E/W tighter than that then you want real TOR 
switching.  If the average E-W is cross rack then the FEX are performance 
equivalent.  For random distributions this comes at a few racks.  For 
intentional distributions it's probably better to TOR switch from day one.


George William Herbert
Sent from my iPhone

Re: Cisco Nexus

[Justin M. Streiner]

On Mon, 2 Feb 2015, Brandon Ewing wrote:


On Mon, Feb 02, 2015 at 12:51:04PM -0600, David Bass wrote:

The n2k ToR is not a great design for user or storage interfaces if most of 
your traffic is east/west.  It is great as a low cost ilo/drac/choose your oob 
port, or if most of your traffic is north/south.  Biggest thing to remember is 
that it is not a switch, and has limitations such as not connecting other 
switches to it. Like anything else you have to understand the product so that 
you don't engineer something that it wasn't designed to do.



And remember -- The Nexus 2K performs absolutely ZERO local switching -- all
frames received from client ports are just copied to the upstream device, so
it can handle the frame/packet forwarding logic.


Also remember that the Nexus (5K, at least) does cut-through switching. 
If you receive an errored frame on one port, the switch can and often will 
happily forward those errored frames once it figures out the destination 
MAC address(es).


jms

Re: Cisco Nexus

[Brandon Ewing]

On Mon, Feb 02, 2015 at 12:51:04PM -0600, David Bass wrote:
> The n2k ToR is not a great design for user or storage interfaces if most of 
> your traffic is east/west.  It is great as a low cost ilo/drac/choose your 
> oob port, or if most of your traffic is north/south.  Biggest thing to 
> remember is that it is not a switch, and has limitations such as not 
> connecting other switches to it. Like anything else you have to understand 
> the product so that you don't engineer something that it wasn't designed to 
> do. 
> 

And remember -- The Nexus 2K performs absolutely ZERO local switching -- all
frames received from client ports are just copied to the upstream device, so
it can handle the frame/packet forwarding logic.

-- 
Brandon Ewing (nicot...@warningg.com)


pgp2sxTQ8wzmN.pgp
Description: PGP signature

Re: Cisco Nexus

[Chris Marget]

There are some unfortunate limitations in classifying incoming traffic.

It's been a while, but I think the rule is that Nexus 2000 devices can only
classify based on incoming 802.1p cos values.

It's a pretty strange and disappointing limitation for an edge device where
you're less likely to have incoming dot1q tags and you're less likely to
trust the other end of the link to mark its own traffic.

/chris

Re: Cisco Nexus

[George Herbert]

I wasn't the implementing engineer but I've been at two places that did that, a 
larger game company and a network gear manufacturer in their engineering 
support computational hubs.  I was there during planning and rollout at the 
game company, very early in the Nexus lifespan.

Both sites brought the FEXes back to 5500s; one used a 6-something for core, 
the other a pair of 7ks.

Game company was more east-west, telco eqpt was very heavy east west.

In both cases it's working fine.

George William Herbert
Sent from my iPhone

> On Feb 2, 2015, at 10:17 AM, "Herman, Anthony" 
>  wrote:
> 
> Nanog,
> 
> I would like to poll the collective for experiences both positive and 
> negative with the Nexus line. More specifically I am interested in hearing 
> about FEX with N2K at the ToR and if this has indeed made any impact on Opex 
> as well as non-obvious shortcomings to using the fabric extenders. Also if 
> anyone is using any of the Nexus line for I/O convergence (FCoE) I would be 
> interested in hearing your experience with this as well.
> 
> Thank you in advance,
> 
> -A

Re: Cisco Nexus

[Daniel Rohan]

I think it depends on what the upstream product from the FEX is and what
your requirements are. Last I checked,  eVPC was not supported on the N7K,
but it was supported as an option on the N5K platform.  eVPC being the dual
homed FEX to two a pair of N7K's running a VPC cluster. I know this is an
old post, but here's a good one that explains precisely what I mean:

If you look at the N5K verified scalability guide, you see this:

Maximum FEXs dual homed to a vPC Cisco Nexus 5500 Series Switch Pair: 24

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/configuration_limits/b_N5500_Config_Limits_602N11_chapter_01.html

If you look at the N7K verified scalability guide, there is *no* mention of
dual-homed fex architectures:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html#reference_E1ED6266546C444093CC27DEB0E1B38E

If I'm wrong and someone is dual homing 2K FEXs to 7K VPC pairs, please
correct me.

If you're interested in latency numbers fex-to-fex, here are some numbers
provided to me by our SE:

http://jumboframe.net/jumboframe/2013/5/5/n2k-fex-to-fex-latency-and-a-reader-follow-up
<http://static1.squarespace.com/static/513c6d36e4b0cc0702f94292/t/51866c97e4b0580e000cf5bd/1367764121313/fex-to-fex-latency.jpg?format=750w>

As you can see, these numbers are decent, but you'd have to be very careful
in choosing your FEX model if you're looking for better than average store
and forward performance out of your FEX.

On Mon, Feb 2, 2015 at 10:17 AM, Herman, Anthony <
anthony.her...@mattersight.com> wrote:

> Nanog,
>
> I would like to poll the collective for experiences both positive and
> negative with the Nexus line. More specifically I am interested in hearing
> about FEX with N2K at the ToR and if this has indeed made any impact on
> Opex as well as non-obvious shortcomings to using the fabric extenders.
> Also if anyone is using any of the Nexus line for I/O convergence (FCoE) I
> would be interested in hearing your experience with this as well.
>
> Thank you in advance,
>
> -A
>

Re: Cisco Nexus

[David Bass]

The n2k ToR is not a great design for user or storage interfaces if most of 
your traffic is east/west.  It is great as a low cost ilo/drac/choose your oob 
port, or if most of your traffic is north/south.  Biggest thing to remember is 
that it is not a switch, and has limitations such as not connecting other 
switches to it. Like anything else you have to understand the product so that 
you don't engineer something that it wasn't designed to do. 

Lots of very large companies using Nexus gear...

That being said I prefer Arista when I'm architecting DCs. 



> On Feb 2, 2015, at 12:17 PM, Herman, Anthony  
> wrote:
> 
> Nanog,
> 
> I would like to poll the collective for experiences both positive and 
> negative with the Nexus line. More specifically I am interested in hearing 
> about FEX with N2K at the ToR and if this has indeed made any impact on Opex 
> as well as non-obvious shortcomings to using the fabric extenders. Also if 
> anyone is using any of the Nexus line for I/O convergence (FCoE) I would be 
> interested in hearing your experience with this as well.
> 
> Thank you in advance,
> 
> -A

RE: Cisco Nexus

[Mann, Jason]

The biggest thing we ran into was no support of spanning-tree on the FEX's. The 
way we are setup, being STATE government, our agency controls the network up to 
the FEX port. Beyond that, the agency's were in control of what they plugged 
into our FEX ports. 

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Herman, Anthony
Sent: Monday, February 02, 2015 11:17 AM
To: nanog@nanog.org
Subject: Cisco Nexus

Nanog,

I would like to poll the collective for experiences both positive and negative 
with the Nexus line. More specifically I am interested in hearing about FEX 
with N2K at the ToR and if this has indeed made any impact on Opex as well as 
non-obvious shortcomings to using the fabric extenders. Also if anyone is using 
any of the Nexus line for I/O convergence (FCoE) I would be interested in 
hearing your experience with this as well.

Thank you in advance,

-A

Cisco Nexus

[Herman, Anthony]

Nanog,

I would like to poll the collective for experiences both positive and negative 
with the Nexus line. More specifically I am interested in hearing about FEX 
with N2K at the ToR and if this has indeed made any impact on Opex as well as 
non-obvious shortcomings to using the fabric extenders. Also if anyone is using 
any of the Nexus line for I/O convergence (FCoE) I would be interested in 
hearing your experience with this as well.

Thank you in advance,

-A

Cisco Nexus 2k EPLD upgrades question.

[Sarpreet Basi]

Hello All,

Looking to deploying 3 x N2K-C2248TP-1GE (2ps 1fan) 4 x10GE slot and 48 
x 10/100/1000, Rather than a 6509.   I'm hesitant with those is the 
maintenance aspect. While ISSUs don't necessarily impact the 2248, EPLD 
upgrades would.  Do you know how upgrades are implemented on the 2248s?  
Is it via the host N7K's own epld process?


Any help on this would be much appreciated.

--
Thank You,

Sarpreet Basi
Knowledge Computers
Email: s...@knowledgecomputers.net
Toll-free: 800.967.6609 x102
International: 250.748.0818 x102
Fax: 250.748.3388
Mobile: 250.709.0336

---
AOL IM: sarpreetb
MSN: sarpr...@hotmail.com
www.knowledgecomputers.net


We Do Maintenance on All EOL Network Product.

RE: Cisco Nexus 5000 with 4G FC module - initialization ?

[Steve Fischer]

Thomas - 

If I'm not mistaken, there is an additional license needed to activate
Fibre-Channel services on the Nexus family of switches.

God bless, 

Steve

-Original Message-
From: Thomas Weible [mailto:thomas.wei...@flexoptix.net] 
Sent: Monday, January 17, 2011 11:14 AM
To: nanog@nanog.org
Subject: Cisco Nexus 5000 with 4G FC module - initialization ?

Hi,

 

I got some trouble to get the 8-port (4/2/1 FC module) up and running in a
Nexus 5000. The module itself is shown in the inventory but when I want to
have a detailed look on an interface than there is no option "Fibre
Channel". 

 

Is there anything else to do with this module (activation, initialization,
etc. ?) 

 

Thanks

Thomas

 

RE: Cisco Nexus 5000 with 4G FC module - initialization ?

[Mike Walter]

When you do a show running, do the interfaces show there at all as fc x/x?  Do 
you have the FCOE feature enabled?

-Mike 
-Original Message-
From: Thomas Weible [mailto:thomas.wei...@flexoptix.net] 
Sent: Monday, January 17, 2011 11:14 AM
To: nanog@nanog.org
Subject: Cisco Nexus 5000 with 4G FC module - initialization ?

Hi,

 

I got some trouble to get the 8-port (4/2/1 FC module) up and running in
a Nexus 5000. The module itself is shown in the inventory but when I
want to have a detailed look on an interface than there is no option
"Fibre Channel". 

 

Is there anything else to do with this module (activation,
initialization, etc. ?) 

 

Thanks

Thomas

 

Cisco Nexus 5000 with 4G FC module - initialization ?

[Thomas Weible]

Hi,

 

I got some trouble to get the 8-port (4/2/1 FC module) up and running in
a Nexus 5000. The module itself is shown in the inventory but when I
want to have a detailed look on an interface than there is no option
"Fibre Channel". 

 

Is there anything else to do with this module (activation,
initialization, etc. ?) 

 

Thanks

Thomas

 

Re: Experience with the Dell PowerConnect 8024F - compare to the Cisco Nexus 5010

[William Pitcock]

Hi,

On Fri, 2010-06-18 at 11:57 -0400, Steven Fischer wrote:
> Does anyone have any experience with the Dell PowerConnect 8024F 10-gig
> switch that they'd be willing to share?  How does it perform?  How reliable
> is it?  My experiences with the Dell switches have been less than favorable
> to this point, but I am willing to concede that some of that may be colored
> by my Cisco bias.  Would you trust this Dell switch in a high-performance
> computing environment, where the ability to move data for sustained
> durations at rates close to line speed is paramount, along with
> high-reliability/high-availability?
> 
> Any feedback is welcomed.
> 

Dell switches are usually Foundry gear relabeled, so it should be ok.
We are using Dell switches alongside actual Foundry gear in a cloud
environment and have had no problems.

Foundry's firmwares have some bugs though as far as SNMP goes.  For
example, our traffic utilization graphs start missing data after about
120 days and we have to reboot them.  This happens on both actual
Foundry gear and the rebranded Dell stuff.  If you're just using the
switches as an interconnect (MPI?), this probably isn't a big deal for
you.  I have heard that newer firmware fixes that problem, but we
haven't had time to test out upgrading so it hasn't been done yet.

The Nexus switch line is also very good, but too expensive for my blood.
I have to eat...  The management is very well done, but the Nexus OS is
feature-lacking in comparison to traditional Cisco IOS.  So, right now,
the Foundry gear is probably a better option.

William

Experience with the Dell PowerConnect 8024F - compare to the Cisco Nexus 5010

[Steven Fischer]

Does anyone have any experience with the Dell PowerConnect 8024F 10-gig
switch that they'd be willing to share?  How does it perform?  How reliable
is it?  My experiences with the Dell switches have been less than favorable
to this point, but I am willing to concede that some of that may be colored
by my Cisco bias.  Would you trust this Dell switch in a high-performance
computing environment, where the ability to move data for sustained
durations at rates close to line speed is paramount, along with
high-reliability/high-availability?

Any feedback is welcomed.

-- 
To him who is able to keep you from falling and to present you before his
glorious presence without fault and with great joy