Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
Actually, I just installed a 10G LR fiber optic module in each of them in order to connect to the 5Ks. I was sure to add the spanning-tree port type edge trunk command on the Nexus so I didn't risk ISSU support. The way I see it, we should be able to keep this working by making sure no hosts go beyond the MTU limit on the smallest switch, since frames don't magically consolidate themselves. :) The iSCSI jumbos have their own switch linked separately in a stack for the jumbo support. Thanks! - Original Message - From:Andrew Jones andrew.jo...@alphawest.com.au To:Conkel, Joshua conk...@wems-llc.com, cisco-nsp@puck.nether.net cisco-nsp@puck.nether.net Sent:9/21/2012 8:39 PM Subject:RE: Are Nexus and per-interface or FEX MTU settings possible? Im assuming your 3560 has gigabit ports to connet the 5ks? Then you can do jumbo frames on those interfaces, as per the following from cisco.com http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c3 You will need to reboot the switch for this to take effect. Andrew Jones -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Conkel, Joshua Sent: Thursday, 20 September 2012 12:57 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible? We just bought a bundle of 2x 5548Ps and 20x 2248s in order to converge our data and storage networks in our new datacenter. After carefully reading the configuration limitations document and designing around the limitations for our migration, we pulled the trigger and bought the material. The document that I used as prep work for the actual deployment was Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels. I planned on being able to hook the Nexus up to our distribution switches and standalone iSCSI network for the migration process and to provide L3 services (because of the limitations with using the L3 routing modules) Now on to our issue: The document states that you can configure per-interface MTU size. Our data network distribution switches are 3560s and run the standard MTU size of 1500. Our iSCSI network, on the other hand, use jumbo frames with a MTU of 9000. After trying the steps outlined in the document, I have only been able to set the MTU globally on the Nexus to jumbo or non-jumbo. After contacting TAC, they have initially confirmed that the MTU qos policy can only be applied at the system level, not per interface. So, any ideas other than not converging our networks and buying a standalone 4900 or 3750x series pair for our iSCSI network? Sorry if it's wordy, but this is the content I gave to TAC as well. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
Haha. For our datacenter migration, we are traveling over about two miles of singlemode fiber with the 10G links. (5 patches total!) I know it's not best practice, but we are doing this to avoid having to readdress servers during the live transition. Once the transition is complete, the Nexus will get a pair of distribution switches of it's own and we will move the existing server subnet routes. We are so small internally that turning off route summarization won't kill us (about 30 routed subnets total). The main focus is being able to migrate all of our VMs to the new storage and hosts with zero downtime. Unfortunately for me, business direction doesn't mesh with best practices if it's not the easiest/cheapest/quickest option. :) - Original Message - From:Andrew Jones andrew.jo...@alphawest.com.au To:Conkel, Joshua conk...@wems-llc.com Cc:cisco-nsp@puck.nether.net cisco-nsp@puck.nether.net Sent:9/21/2012 8:53 PM Subject:RE: Are Nexus and per-interface or FEX MTU settings possible? Also, why LR optics? Unless you need to distance, SR optics are 1/4 the price. Or if you are patching into the same rack, twinax cables even cheaper. Ive generally placed 5k in a central location near core / aggregation switches, then distributed the 2k over the data hall with multimode fibre. Allowing you to use cheap twinax for uplink and cheap fet-10g over multimode for FEX. Cheers, Alphawest -Original Message- From: Conkel, Joshua [mailto:conk...@wems-llc.com] Sent: Saturday, 22 September 2012 10:47 AM To: Andrew Jones Cc: cisco-nsp@puck.nether.net Subject: RE: Are Nexus and per-interface or FEX MTU settings possible? Actually, I just installed a 10G LR fiber optic module in each of them in order to connect to the 5Ks. I was sure to add the spanning-tree port type edge trunk command on the Nexus so I didn't risk ISSU support. The way I see it, we should be able to keep this working by making sure no hosts go beyond the MTU limit on the smallest switch, since frames don't magically consolidate themselves. :) The iSCSI jumbos have their own switch linked separately in a stack for the jumbo support. Thanks! - Original Message - From:Andrew Jones andrew.jo...@alphawest.com.au To:Conkel, Joshua conk...@wems-llc.com, cisco-nsp@puck.nether.net cisco-nsp@puck.nether.net Sent:9/21/2012 8:39 PM Subject:RE: Are Nexus and per-interface or FEX MTU settings possible? Im assuming your 3560 has gigabit ports to connet the 5ks? Then you can do jumbo frames on those interfaces, as per the following from cisco.com http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c3 You will need to reboot the switch for this to take effect. Andrew Jones -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Conkel, Joshua Sent: Thursday, 20 September 2012 12:57 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible? We just bought a bundle of 2x 5548Ps and 20x 2248s in order to converge our data and storage networks in our new datacenter. After carefully reading the configuration limitations document and designing around the limitations for our migration, we pulled the trigger and bought the material. The document that I used as prep work for the actual deployment was Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels. I planned on being able to hook the Nexus up to our distribution switches and standalone iSCSI network for the migration process and to provide L3 services (because of the limitations with using the L3 routing modules) Now on to our issue: The document states that you can configure per-interface MTU size. Our data network distribution switches are 3560s and run the standard MTU size of 1500. Our iSCSI network, on the other hand, use jumbo frames with a MTU of 9000. After trying the steps outlined in the document, I have only been able to set the MTU globally on the Nexus to jumbo or non-jumbo. After contacting TAC, they have initially confirmed that the MTU qos policy can only be applied at the system level, not per interface. So, any ideas other than not converging our networks and buying a standalone 4900 or 3750x series pair for our iSCSI network? Sorry if it's wordy, but this is the content I gave to TAC as well. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
Also, why LR optics? Unless you need to distance, SR optics are 1/4 the price. Or if you are patching into the same rack, twinax cables even cheaper. Ive generally placed 5k in a central location near core / aggregation switches, then distributed the 2k over the data hall with multimode fibre. Allowing you to use cheap twinax for uplink and cheap fet-10g over multimode for FEX. Cheers, Alphawest -Original Message- From: Conkel, Joshua [mailto:conk...@wems-llc.com] Sent: Saturday, 22 September 2012 10:47 AM To: Andrew Jones Cc: cisco-nsp@puck.nether.net Subject: RE: Are Nexus and per-interface or FEX MTU settings possible? Actually, I just installed a 10G LR fiber optic module in each of them in order to connect to the 5Ks. I was sure to add the spanning-tree port type edge trunk command on the Nexus so I didn't risk ISSU support. The way I see it, we should be able to keep this working by making sure no hosts go beyond the MTU limit on the smallest switch, since frames don't magically consolidate themselves. :) The iSCSI jumbos have their own switch linked separately in a stack for the jumbo support. Thanks! - Original Message - From:Andrew Jones andrew.jo...@alphawest.com.au To:Conkel, Joshua conk...@wems-llc.com, cisco-nsp@puck.nether.net cisco-nsp@puck.nether.net Sent:9/21/2012 8:39 PM Subject:RE: Are Nexus and per-interface or FEX MTU settings possible? Im assuming your 3560 has gigabit ports to connet the 5ks? Then you can do jumbo frames on those interfaces, as per the following from cisco.com http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c3 You will need to reboot the switch for this to take effect. Andrew Jones -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Conkel, Joshua Sent: Thursday, 20 September 2012 12:57 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible? We just bought a bundle of 2x 5548Ps and 20x 2248s in order to converge our data and storage networks in our new datacenter. After carefully reading the configuration limitations document and designing around the limitations for our migration, we pulled the trigger and bought the material. The document that I used as prep work for the actual deployment was Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels. I planned on being able to hook the Nexus up to our distribution switches and standalone iSCSI network for the migration process and to provide L3 services (because of the limitations with using the L3 routing modules) Now on to our issue: The document states that you can configure per-interface MTU size. Our data network distribution switches are 3560s and run the standard MTU size of 1500. Our iSCSI network, on the other hand, use jumbo frames with a MTU of 9000. After trying the steps outlined in the document, I have only been able to set the MTU globally on the Nexus to jumbo or non-jumbo. After contacting TAC, they have initially confirmed that the MTU qos policy can only be applied at the system level, not per interface. So, any ideas other than not converging our networks and buying a standalone 4900 or 3750x series pair for our iSCSI network? Sorry if it's wordy, but this is the content I gave to TAC as well. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
Im assuming your 3560 has gigabit ports to connet the 5ks? Then you can do jumbo frames on those interfaces, as per the following from cisco.com http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c3 You will need to reboot the switch for this to take effect. Andrew Jones -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Conkel, Joshua Sent: Thursday, 20 September 2012 12:57 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible? We just bought a bundle of 2x 5548Ps and 20x 2248s in order to converge our data and storage networks in our new datacenter. After carefully reading the configuration limitations document and designing around the limitations for our migration, we pulled the trigger and bought the material. The document that I used as prep work for the actual deployment was Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels. I planned on being able to hook the Nexus up to our distribution switches and standalone iSCSI network for the migration process and to provide L3 services (because of the limitations with using the L3 routing modules) Now on to our issue: The document states that you can configure per-interface MTU size. Our data network distribution switches are 3560s and run the standard MTU size of 1500. Our iSCSI network, on the other hand, use jumbo frames with a MTU of 9000. After trying the steps outlined in the document, I have only been able to set the MTU globally on the Nexus to jumbo or non-jumbo. After contacting TAC, they have initially confirmed that the MTU qos policy can only be applied at the system level, not per interface. So, any ideas other than not converging our networks and buying a standalone 4900 or 3750x series pair for our iSCSI network? Sorry if it's wordy, but this is the content I gave to TAC as well. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
PMTUD should take care of the rest... Andrew Jones -Original Message- From: Conkel, Joshua [mailto:conk...@wems-llc.com] Sent: Saturday, 22 September 2012 10:47 AM To: Andrew Jones Cc: cisco-nsp@puck.nether.net Subject: RE: Are Nexus and per-interface or FEX MTU settings possible? Actually, I just installed a 10G LR fiber optic module in each of them in order to connect to the 5Ks. I was sure to add the spanning-tree port type edge trunk command on the Nexus so I didn't risk ISSU support. The way I see it, we should be able to keep this working by making sure no hosts go beyond the MTU limit on the smallest switch, since frames don't magically consolidate themselves. :) The iSCSI jumbos have their own switch linked separately in a stack for the jumbo support. Thanks! - Original Message - From:Andrew Jones andrew.jo...@alphawest.com.au To:Conkel, Joshua conk...@wems-llc.com, cisco-nsp@puck.nether.net cisco-nsp@puck.nether.net Sent:9/21/2012 8:39 PM Subject:RE: Are Nexus and per-interface or FEX MTU settings possible? Im assuming your 3560 has gigabit ports to connet the 5ks? Then you can do jumbo frames on those interfaces, as per the following from cisco.com http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c3 You will need to reboot the switch for this to take effect. Andrew Jones -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Conkel, Joshua Sent: Thursday, 20 September 2012 12:57 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible? We just bought a bundle of 2x 5548Ps and 20x 2248s in order to converge our data and storage networks in our new datacenter. After carefully reading the configuration limitations document and designing around the limitations for our migration, we pulled the trigger and bought the material. The document that I used as prep work for the actual deployment was Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels. I planned on being able to hook the Nexus up to our distribution switches and standalone iSCSI network for the migration process and to provide L3 services (because of the limitations with using the L3 routing modules) Now on to our issue: The document states that you can configure per-interface MTU size. Our data network distribution switches are 3560s and run the standard MTU size of 1500. Our iSCSI network, on the other hand, use jumbo frames with a MTU of 9000. After trying the steps outlined in the document, I have only been able to set the MTU globally on the Nexus to jumbo or non-jumbo. After contacting TAC, they have initially confirmed that the MTU qos policy can only be applied at the system level, not per interface. So, any ideas other than not converging our networks and buying a standalone 4900 or 3750x series pair for our iSCSI network? Sorry if it's wordy, but this is the content I gave to TAC as well. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
Hi Joshua, You can't set a per interface MTU on the Nexus 5500-series switches. However, you can do it on per Class of Service. So in order to do increased MTU for your iSCSI traffic, you should create a policy-map identifying your iSCSI traffic, place the traffic in a separate queue with increased MTU. This is the way to go forward. Lars Christensen CCIE #20292 Den 20/09/2012 kl. 04.56 skrev Conkel, Joshua conk...@wems-llc.com: We just bought a bundle of 2x 5548Ps and 20x 2248s in order to converge our data and storage networks in our new datacenter. After carefully reading the configuration limitations document and designing around the limitations for our migration, we pulled the trigger and bought the material. The document that I used as prep work for the actual deployment was Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels. I planned on being able to hook the Nexus up to our distribution switches and standalone iSCSI network for the migration process and to provide L3 services (because of the limitations with using the L3 routing modules) Now on to our issue: The document states that you can configure per-interface MTU size. Our data network distribution switches are 3560s and run the standard MTU size of 1500. Our iSCSI network, on the other hand, use jumbo frames with a MTU of 9000. After trying the steps outlined in the document, I have only been able to set the MTU globally on the Nexus to jumbo or non-jumbo. After contacting TAC, they have initially confirmed that the MTU qos policy can only be applied at the system level, not per interface. So, any ideas other than not converging our networks and buying a standalone 4900 or 3750x series pair for our iSCSI network? Sorry if it's wordy, but this is the content I gave to TAC as well. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
I'd like to further clarify this - as I think the subtleties here between layer 2 and layer 3 MTU may be giving a misleading picture. The Layer 2 MTU (AKA frame size) is set globally, in the same way as on a Catalyst floor switch such as a 3560/3750 is done. This is performed by something like this in the config: - policy-map type network-qos enable-jumbo-frames class type network-qos class-default mtu 9216 system qos service-policy type network-qos enable-jumbo-frames -- However the Layer 3 MTU is customisable, per VLAN interface: --- SWT01(config)# int vlan 22 SWT01(config-if)# mtu ? 64-9216 MTU size in bytes SWT01(config-if)# interface Vlan22 no shutdown mtu 9216 - In the majority of cases you will probably want to have a high layer 2 MTU globally, and then perhaps be more careful with your layer 3 MTU settings at your layer 3 boundaries (the default is almost always 1500 so unless you specifically modify it, that's what it'll be, regardless of your layer 2 MTU). If you aren't doing any Layer 3 switching on the device then the layer 3 MTU setting obviously won't be necessary on there. The config above is from a 5548UP running 5.2(1)N1 (that has the L3 daughter card). I'm not aware of any negative side effects on this platform from setting the layer 2 MTU to a high value globally. Things most certainly break badly when the L2 MTU is set too small, but not usually when it's too big. Reuben On 20/09/2012 4:01 PM, Lars Christensen wrote: Hi Joshua, You can't set a per interface MTU on the Nexus 5500-series switches. However, you can do it on per Class of Service. So in order to do increased MTU for your iSCSI traffic, you should create a policy-map identifying your iSCSI traffic, place the traffic in a separate queue with increased MTU. This is the way to go forward. Lars Christensen CCIE #20292 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
On 09/20/2012 07:20 AM, Reuben Farrelly wrote: policy-map type network-qos enable-jumbo-frames class type network-qos class-default mtu 9216 system qos service-policy type network-qos enable-jumbo-frames Interesting. Does anyone know the rationale behind this way of setting the MTU on ths platform? It seems a bit peculiar, and as you note, a too-high L2 MTU is seldom harmful, so you would think per-interface (or even per-device) would suffice. FWIW N7k seems to work normally i.e. set physical interface SVI MTU in the interface config. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Are Nexus and per-interface or FEX MTU settings possible?
We just bought a bundle of 2x 5548Ps and 20x 2248s in order to converge our data and storage networks in our new datacenter. After carefully reading the configuration limitations document and designing around the limitations for our migration, we pulled the trigger and bought the material. The document that I used as prep work for the actual deployment was Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels. I planned on being able to hook the Nexus up to our distribution switches and standalone iSCSI network for the migration process and to provide L3 services (because of the limitations with using the L3 routing modules) Now on to our issue: The document states that you can configure per-interface MTU size. Our data network distribution switches are 3560s and run the standard MTU size of 1500. Our iSCSI network, on the other hand, use jumbo frames with a MTU of 9000. After trying the steps outlined in the document, I have only been able to set the MTU globally on the Nexus to jumbo or non-jumbo. After contacting TAC, they have initially confirmed that the MTU qos policy can only be applied at the system level, not per interface. So, any ideas other than not converging our networks and buying a standalone 4900 or 3750x series pair for our iSCSI network? Sorry if it's wordy, but this is the content I gave to TAC as well. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
