Re: [c-nsp] high cpu usage because of process switching on mlppp interface
I'd like to see the full config of the 72xx. Also, can you do the following: a) disable fragmentation on both sides no ppp multilink fragmentation b) remove all non MLPPP commands from the member links of the bundle You should add those to the MLPPP interface itself You are process switcing all of your traffic inbound on the MLPPP interface of the 72xx: MTP-C72-02#sh inter mul 1 switching Multilink1 Maltepe - Adana MLPPP Throttle count 0 Drops RP 752707 SP 0 SPD Flushes Fast 0SSE 0 SPD Aggress Fast 0 SPD Priority Inputs259 Drops 0 Protocol IP Switching pathPkts In Chars In Pkts Out Chars Out Process 50148189 395174672 261949 26235771 Cache misses 0 - - - Fast 1325 105874 12194 724886 Auton/SSE 0 0 0 0 Protocol Tag Switching pathPkts In Chars In Pkts Out Chars Out Process 79390997 3601460527 5116 538398 Cache misses 0 - - - Fast 0 0 87354606 4071850482 Auton/SSE 0 0 0 0 so that tells me it's another feature configured most likely on the egress interface this traffic will go out of or a bad bug. Notice it's doing it for both ingress ip and MPLS. Rodney On Mon, Sep 03, 2007 at 08:47:04AM +0300, Arda Balkanay wrote: Hi I haven't had a TAC SR yet about this issue, and i couldn't find the related topic previously covered in this group so sory for double opening of this kind of subject. In my scenario i am running mpls over mlppp. this line is my pop uplink. I could'nt get useful information about my problem from the command show cef interface output. According to your previous mail can you please send me the Bug ID, on the other hand i can open a TAC case for this issue, but before i should study cisco switching architecture (like process switching fast switching and so on) MTP-C72-02#sh cef interface mu 1 Multilink1 is up (if_number 55) Corresponding hwidb fast_if_number 55 Corresponding hwidb firstsw-if_number 55 Internet address is 85.29.8.101/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set Interface is marked as point to point interface Hardware idb is Multilink1 Fast switching type 7, interface type 94 IP CEF switching enabled IP CEF Fast switching turbo vector IP Null turbo vector Input fast flags 0x0, Input fast flags2 0x0, Output fast flags 0x0, Output fast flags2 0x0 ifindex 31(31) Slot -1 Slot unit 1 Unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500 ADN-C38-01#sh cef interface Mu1 Multilink1 is up (if_number 21) Corresponding hwidb fast_if_number 21 Corresponding hwidb firstsw-if_number 21 Internet address is 85.29.8.102/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set Interface is marked as point to point interface Hardware idb is Multilink1 Fast switching type 7, interface type 94 IP CEF switching enabled IP CEF Fast switching turbo vector Input fast flags 0x0, Input fast flags2 0x0, Output fast flags 0x0, Output fast flags2 0x0 ifindex 17(17) Slot -1 Slot unit 1 Unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500 ADN-C38-01# My interface configs are as follow: MTP-C72-02#sh run int mu 1 Building configuration... Current configuration : 262 bytes ! interface Multilink1 description Maltepe - Adana MLPPP mtu 1600 ip address 85.29.8.101 255.255.255.252 ip mtu 1500 mpls label protocol ldp tag-switching mtu 1524 tag-switching ip ppp multilink ppp multilink group 1 ppp multilink mrru local 1604 end MTP-C72-02#sh ppp mu Multilink1, bundle name is ADN-C38-01 Endpoint discriminator is ADN-C38-01 Bundle up for 2d01h, 174/255 load Receive buffer limit 25664 bytes, frag timeout 1000 ms 0/0 fragments/bytes in reassembly list 16890 lost fragments, 357474026 reordered 7820/896328 discarded fragments/bytes, 0 lost received 0xD2B4CA received sequence, 0x8EC33D sent sequence Member links: 2 active, 0 inactive (max not set, min not set) Se4/0:0, since 2d01h Se2/3:0, since 2d01h No inactive multilink interfaces MTP-C72-02#sh run int ser 4/0:0 Building
Re: [c-nsp] high cpu usage because of process switching on mlppp interface
Thank you Rodney, i will check my configs again and share the results with you. On 9/4/07, Rodney Dunn [EMAIL PROTECTED] wrote: I'd like to see the full config of the 72xx. Also, can you do the following: a) disable fragmentation on both sides no ppp multilink fragmentation b) remove all non MLPPP commands from the member links of the bundle You should add those to the MLPPP interface itself You are process switcing all of your traffic inbound on the MLPPP interface of the 72xx: MTP-C72-02#sh inter mul 1 switching Multilink1 Maltepe - Adana MLPPP Throttle count 0 Drops RP 752707 SP 0 SPD Flushes Fast 0SSE 0 SPD Aggress Fast 0 SPD Priority Inputs259 Drops 0 Protocol IP Switching pathPkts In Chars In Pkts Out Chars Out Process 50148189 395174672 261949 26235771 Cache misses 0 - - - Fast 1325 105874 12194 724886 Auton/SSE 0 0 0 0 Protocol Tag Switching pathPkts In Chars In Pkts Out Chars Out Process 79390997 3601460527 5116 538398 Cache misses 0 - - - Fast 0 0 87354606 4071850482 Auton/SSE 0 0 0 0 so that tells me it's another feature configured most likely on the egress interface this traffic will go out of or a bad bug. Notice it's doing it for both ingress ip and MPLS. Rodney On Mon, Sep 03, 2007 at 08:47:04AM +0300, Arda Balkanay wrote: Hi I haven't had a TAC SR yet about this issue, and i couldn't find the related topic previously covered in this group so sory for double opening of this kind of subject. In my scenario i am running mpls over mlppp. this line is my pop uplink. I could'nt get useful information about my problem from the command show cef interface output. According to your previous mail can you please send me the Bug ID, on the other hand i can open a TAC case for this issue, but before i should study cisco switching architecture (like process switching fast switching and so on) MTP-C72-02#sh cef interface mu 1 Multilink1 is up (if_number 55) Corresponding hwidb fast_if_number 55 Corresponding hwidb firstsw-if_number 55 Internet address is 85.29.8.101/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set Interface is marked as point to point interface Hardware idb is Multilink1 Fast switching type 7, interface type 94 IP CEF switching enabled IP CEF Fast switching turbo vector IP Null turbo vector Input fast flags 0x0, Input fast flags2 0x0, Output fast flags 0x0, Output fast flags2 0x0 ifindex 31(31) Slot -1 Slot unit 1 Unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500 ADN-C38-01#sh cef interface Mu1 Multilink1 is up (if_number 21) Corresponding hwidb fast_if_number 21 Corresponding hwidb firstsw-if_number 21 Internet address is 85.29.8.102/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set Interface is marked as point to point interface Hardware idb is Multilink1 Fast switching type 7, interface type 94 IP CEF switching enabled IP CEF Fast switching turbo vector Input fast flags 0x0, Input fast flags2 0x0, Output fast flags 0x0, Output fast flags2 0x0 ifindex 17(17) Slot -1 Slot unit 1 Unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500 ADN-C38-01# My interface configs are as follow: MTP-C72-02#sh run int mu 1 Building configuration... Current configuration : 262 bytes ! interface Multilink1 description Maltepe - Adana MLPPP mtu 1600 ip address 85.29.8.101 255.255.255.252 ip mtu 1500 mpls label protocol ldp tag-switching mtu 1524 tag-switching ip ppp multilink ppp multilink group 1 ppp multilink mrru local 1604 end MTP-C72-02#sh ppp mu Multilink1, bundle name is ADN-C38-01 Endpoint discriminator is ADN-C38-01 Bundle up for 2d01h, 174/255 load Receive buffer limit 25664 bytes, frag timeout 1000 ms 0/0 fragments/bytes in reassembly list 16890 lost fragments, 357474026 reordered 7820/896328 discarded fragments/bytes, 0 lost received 0xD2B4CA received sequence,
Re: [c-nsp] high cpu usage because of process switching on mlppp interface
Hi I haven't had a TAC SR yet about this issue, and i couldn't find the related topic previously covered in this group so sory for double opening of this kind of subject. In my scenario i am running mpls over mlppp. this line is my pop uplink. I could'nt get useful information about my problem from the command show cef interface output. According to your previous mail can you please send me the Bug ID, on the other hand i can open a TAC case for this issue, but before i should study cisco switching architecture (like process switching fast switching and so on) MTP-C72-02#sh cef interface mu 1 Multilink1 is up (if_number 55) Corresponding hwidb fast_if_number 55 Corresponding hwidb firstsw-if_number 55 Internet address is 85.29.8.101/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set Interface is marked as point to point interface Hardware idb is Multilink1 Fast switching type 7, interface type 94 IP CEF switching enabled IP CEF Fast switching turbo vector IP Null turbo vector Input fast flags 0x0, Input fast flags2 0x0, Output fast flags 0x0, Output fast flags2 0x0 ifindex 31(31) Slot -1 Slot unit 1 Unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500 ADN-C38-01#sh cef interface Mu1 Multilink1 is up (if_number 21) Corresponding hwidb fast_if_number 21 Corresponding hwidb firstsw-if_number 21 Internet address is 85.29.8.102/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set Interface is marked as point to point interface Hardware idb is Multilink1 Fast switching type 7, interface type 94 IP CEF switching enabled IP CEF Fast switching turbo vector Input fast flags 0x0, Input fast flags2 0x0, Output fast flags 0x0, Output fast flags2 0x0 ifindex 17(17) Slot -1 Slot unit 1 Unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500 ADN-C38-01# My interface configs are as follow: MTP-C72-02#sh run int mu 1 Building configuration... Current configuration : 262 bytes ! interface Multilink1 description Maltepe - Adana MLPPP mtu 1600 ip address 85.29.8.101 255.255.255.252 ip mtu 1500 mpls label protocol ldp tag-switching mtu 1524 tag-switching ip ppp multilink ppp multilink group 1 ppp multilink mrru local 1604 end MTP-C72-02#sh ppp mu Multilink1, bundle name is ADN-C38-01 Endpoint discriminator is ADN-C38-01 Bundle up for 2d01h, 174/255 load Receive buffer limit 25664 bytes, frag timeout 1000 ms 0/0 fragments/bytes in reassembly list 16890 lost fragments, 357474026 reordered 7820/896328 discarded fragments/bytes, 0 lost received 0xD2B4CA received sequence, 0x8EC33D sent sequence Member links: 2 active, 0 inactive (max not set, min not set) Se4/0:0, since 2d01h Se2/3:0, since 2d01h No inactive multilink interfaces MTP-C72-02#sh run int ser 4/0:0 Building configuration... Current configuration : 258 bytes ! interface Serial4/0:0 description Tellcom 2Maltepe-Adana 2Mbps LL DATA NO:0107110068 mtu 1600 no ip address ip mtu 1500 encapsulation ppp load-interval 30 mpls label protocol ldp tag-switching mtu 1524 tag-switching ip ppp multilink group 1 end MTP-C72-02#sh run int ser 2/3:0 Building configuration... Current configuration : 264 bytes ! interface Serial2/3:0 description Tellcom Maltepe-Adana 2Mbps LL DATA NO:0107110060 mtu 1600 no ip address encapsulation ppp load-interval 30 mpls label protocol ldp tag-switching mtu 1524 tag-switching ip fair-queue 64 16 0 ppp multilink group 1 end MTP-C72-02#sh run int ser 4/0:0 Building configuration... Current configuration : 258 bytes ! interface Serial4/0:0 description Tellcom 2Maltepe-Adana 2Mbps LL DATA NO:0107110068 mtu 1600 no ip address ip mtu 1500 encapsulation ppp load-interval 30 mpls label protocol ldp tag-switching mtu 1524 tag-switching ip ppp multilink group 1 end MTP-C72-02#sh run int ser 2/3:0 Building configuration... Current configuration : 264 bytes ! interface Serial2/3:0 description Tellcom Maltepe-Adana 2Mbps LL DATA NO:0107110060 mtu 1600 no ip address encapsulation ppp load-interval 30 mpls label protocol ldp tag-switching mtu 1524 tag-switching ip fair-queue 64 16 0 ppp multilink group 1 end ADN-C38-01#sh run int mu 1 Building configuration... Current configuration : 244 bytes ! interface Multilink1 description Adana - Maltepe MLPPP mtu 1600 ip address 85.29.8.102 255.255.255.252 ip mtu 1500 mpls label protocol ldp mpls ip mpls mtu 1524 ppp multilink ppp multilink group 1 ppp multilink mrru local 1604 end ADN-C38-01#sh ppp mu Multilink1, bundle name is MTP-C72-02 Endpoint discriminator is MTP-C72-02 Bundle up for 2d01h, total bandwidth 4096, load 186/255 Receive buffer limit 25664 bytes, frag timeout 1000 ms 0/0
Re: [c-nsp] high cpu usage because of process switching on mlppp interface
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Arda Balkanay wrote: Hi i have a problem about high cpu usage on a 7206 VXR NPE-G1 because of process switching on a multilink ppp interface. I have two E1 interfaces between 7206 and 3845 (below MTP-C72-02 indicates 7206 and ADN-C38-01 indicates 3845). I've bundled these two E1 interfaces. And when the amount of traffic rised CPU usage also increased about 15 percent more. show proc cpu output points increase at the processes of ip input and tag input. MTP-C72-02#sh proc cpu sorted CPU utilization for five seconds: 78%/64%; one minute: 79%; five minutes: 81% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 6839242000 258183793151 8.92% 8.20% 8.40% 0 IP Input 10925093708 119681214209 6.88% 7.37% 7.79% 0 Tag Input When i looked at the switching status of the interface i saw that 7206 is using process switching while 3845 uses fast switchig (especially for mpls) MTP-C72-02#sh inter mul 1 switching Multilink1 Maltepe - Adana MLPPP Throttle count 0 Drops RP 752707 SP 0 SPD Flushes Fast 0SSE 0 SPD Aggress Fast 0 SPD Priority Inputs259 Drops 0 Protocol IP Switching pathPkts In Chars In Pkts Out Chars Out Process 50148189 395174672 261949 26235771 Cache misses 0 - - - Fast 1325 105874 12194 724886 Auton/SSE 0 0 0 0 Protocol Tag Switching pathPkts In Chars In Pkts Out Chars Out Process 79390997 3601460527 5116 538398 Cache misses 0 - - - Fast 0 0 87354606 4071850482 Auton/SSE 0 0 0 0 ADN-C38-01#sh inter mu 1 switching Multilink1 Adana - Maltepe MLPPP Throttle count 0 Drops RP 0 SP 0 SPD Flushes Fast 0SSE 0 SPD Aggress Fast 0 SPD Priority Inputs242 Drops 0 Protocol IP Switching pathPkts In Chars In Pkts Out Chars Out Process 323942110804 175951260300 Cache misses 0 - - - Fast 559376 60411948154 8439 Auton/SSE 0 0 0 0 Protocol Tag Switching pathPkts In Chars In Pkts Out Chars Out Process 7106 398899 10784 539203 Cache misses 0 - - - Fast 87130476 3140243335 80454345 488241675 Auton/SSE 0 0 0 0 When i checked show ip interface output i can see that fast swithing is enabled but i didn't understand why 7206 uses process switching while 3845 uses fast swithcing with the same multilink interface configuration. ip interface output is as follows: MTP-C72-02#sh ip int mu 1 Multilink1 is up, line protocol is up ... IP fast switching is enabled IP fast switching on the same interface is enabled IP Flow switching is disabled IP CEF switching is enabled IP CEF Fast switching turbo vector IP Null turbo vector IP multicast fast switching is enabled IP multicast distributed fast switching is disabled IP route-cache flags are Fast, CEF ... MTP-C72-02# ADN-C38-01#sh ip int mu 1 Multilink1 is up, line protocol is up ... IP fast switching is enabled IP fast switching on the same interface is enabled IP Flow switching is disabled IP CEF switching is enabled IP CEF Fast switching turbo vector IP multicast fast switching is enabled IP multicast distributed fast switching is disabled IP route-cache flags are Fast, CEF ... ADN-C38-01# 7206 uses ios c7200-jk9s-mz.123-14.T3.bin and 3845 uses c3845-spservicesk9-mz.124-8b.bin Do you have any idea about this situation ? Take a look at the output of show cef interface which will tell you why packets would be dropped into a slower switching path: Dialer1 is up (if_number 6) Corresponding hwidb fast_if_number 6 Corresponding hwidb firstsw-if_number 6 Internet address is 172.25.1.1/24 ICMP redirects are never sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set IP policy routing is disabled BGP based policy accounting on
Re: [c-nsp] high cpu usage because of process switching on mlppp interface
Do you have a TAC SR open on this? I just saw someone ask this a few days ago and I told them it's a bug. We support CEF switching of data over a MLPPP bundle with MPLS. Are you running MPLSoMLPPP or are you just running the MLPPP interface in a VRF? MTP-C72-02#sh inter mul 1 switching Multilink1 Maltepe - Adana MLPPP Throttle count 0 Drops RP 752707 SP 0 SPD Flushes Fast 0SSE 0 SPD Aggress Fast 0 SPD Priority Inputs259 Drops 0 Protocol IP Switching pathPkts In Chars In Pkts Out Chars Out Process 50148189 395174672 261949 26235771 Cache misses 0 - - - Fast 1325 105874 12194 724886 Auton/SSE 0 0 0 0 Protocol Tag Switching pathPkts In Chars In Pkts Out Chars Out Process 79390997 3601460527 5116 538398 Cache misses 0 - - - Fast 0 0 87354606 4071850482 Auton/SSE 0 0 0 0 shows you are process switching a ton of your IP traffic and all of the inbound MPLS traffic. Outbound your mpls traffic is fine so we need to look at the inbound switching vectors coming in the MLPPP interface. As Bruce said can you get 'sh cef in mu 1'? Also, post the configs of the mlppp interface as well as the other interfaces this traffic will go out of once it comes in the MLPPP interface? Rodney On Wed, Aug 29, 2007 at 03:15:19PM +0300, Arda Balkanay wrote: Hi i have a problem about high cpu usage on a 7206 VXR NPE-G1 because of process switching on a multilink ppp interface. I have two E1 interfaces between 7206 and 3845 (below MTP-C72-02 indicates 7206 and ADN-C38-01 indicates 3845). I've bundled these two E1 interfaces. And when the amount of traffic rised CPU usage also increased about 15 percent more. show proc cpu output points increase at the processes of ip input and tag input. MTP-C72-02#sh proc cpu sorted CPU utilization for five seconds: 78%/64%; one minute: 79%; five minutes: 81% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 6839242000 258183793151 8.92% 8.20% 8.40% 0 IP Input 10925093708 119681214209 6.88% 7.37% 7.79% 0 Tag Input When i looked at the switching status of the interface i saw that 7206 is using process switching while 3845 uses fast switchig (especially for mpls) MTP-C72-02#sh inter mul 1 switching Multilink1 Maltepe - Adana MLPPP Throttle count 0 Drops RP 752707 SP 0 SPD Flushes Fast 0SSE 0 SPD Aggress Fast 0 SPD Priority Inputs259 Drops 0 Protocol IP Switching pathPkts In Chars In Pkts Out Chars Out Process 50148189 395174672 261949 26235771 Cache misses 0 - - - Fast 1325 105874 12194 724886 Auton/SSE 0 0 0 0 Protocol Tag Switching pathPkts In Chars In Pkts Out Chars Out Process 79390997 3601460527 5116 538398 Cache misses 0 - - - Fast 0 0 87354606 4071850482 Auton/SSE 0 0 0 0 ADN-C38-01#sh inter mu 1 switching Multilink1 Adana - Maltepe MLPPP Throttle count 0 Drops RP 0 SP 0 SPD Flushes Fast 0SSE 0 SPD Aggress Fast 0 SPD Priority Inputs242 Drops 0 Protocol IP Switching pathPkts In Chars In Pkts Out Chars Out Process 323942110804 175951260300 Cache misses 0 - - - Fast 559376 60411948154 8439 Auton/SSE 0 0 0 0 Protocol Tag Switching pathPkts In Chars In Pkts Out Chars Out Process 7106 398899 10784 539203 Cache misses 0 - - - Fast 87130476 3140243335 80454345 488241675 Auton/SSE 0 0 0 0
