Hi James, Looks like microburst overdriving the interface to me. You are seeing rx misses and rx_overruns on internal driver. There are few misses in the buffers but no failures and I don't think that buffer tuning will help. AFAIK you cannot tune the rx ring on the router but I have been proven wrong before.... Otherwise it could be a software bug so check if the devices' releases are the same and check on bug tracker. You could ask for a shaping to be placed upstream or see if flow control can be used.
Sorry for no good news Brian Il giorno gio 30 gen 2025 alle ore 17:08 james list <[email protected]> ha scritto: > > Hi Brian > thanks for responding. > > I've already done some actions, reboot did not help, the point is that the > same C3925 getting the same feeds in other location doesn't have the same > behaviour. > > One difference: 3925 having the issue is connected 1 Gbs to a C9407 from > the market data feed broker, while the 3925 not having the issue is > connected to an old C4948... > > Unfortunately I have already done some of the actions, not tried > rate-limit to log the exceed: > > 1) reboot doesn't helped > 2) buffer miss doesn't increase a lot > > #sh buffer > Buffer elements: > 603 in free list > 96825593 hits, 0 misses, 1242 created > > Public buffer pools: > Small buffers, 104 bytes (total 1024, permanent 1024): > 1016 in free list (100 min, 2048 max allowed) > 39265134 hits, 0 misses, 0 trims, 0 created > 0 failures (0 no memory) > Middle buffers, 600 bytes (total 1024, permanent 1024): > 1024 in free list (100 min, 2048 max allowed) > 12407239 hits, 0 misses, 0 trims, 0 created > 0 failures (0 no memory) > Big buffers, 1536 bytes (total 512, permanent 512): > 512 in free list (100 min, 1024 max allowed) > 25923483 hits, 0 misses, 0 trims, 0 created > 0 failures (0 no memory) > VeryBig buffers, 4520 bytes (total 150, permanent 150, peak 151 @ 7w0d): > 150 in free list (30 min, 300 max allowed) > 78341 hits, 0 misses, 1 trims, 1 created > 0 failures (0 no memory) > Large buffers, 5024 bytes (total 1, permanent 0, peak 1 @ 7w0d): > 1 in free list (0 min, 10 max allowed) > 0 hits, 0 misses, 486 trims, 487 created > 0 failures (0 no memory) > Huge buffers, 18024 bytes (total 5, permanent 0, peak 8 @ 7w0d): > 5 in free list (4 min, 10 max allowed) > 234041 hits, 972 misses, 2917 trims, 2922 created > 0 failures (0 no memory) > > Interface buffer pools: > CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 7w0d): > 101 in free list (100 min, 200 max allowed) > 0 hits, 0 misses, 486 trims, 487 created > 0 failures (0 no memory) > Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ > 7w0d): > 101 in free list (100 min, 100 max allowed) > 0 hits, 0 misses > CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 7w0d): > 101 in free list (100 min, 200 max allowed) > 0 hits, 0 misses, 486 trims, 487 created > 0 failures (0 no memory) > Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ > 7w0d): > 101 in free list (132 min, 132 max allowed) > 318 hits, 0 misses > CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 7w0d): > 26 in free list (25 min, 50 max allowed) > 0 hits, 0 misses, 486 trims, 487 created > 0 failures (0 no memory) > IPC buffers, 4096 bytes (total 2, permanent 2): > 1 in free list (1 min, 8 max allowed) > 1 hits, 0 fallbacks, 0 trims, 0 created > 0 failures (0 no memory) > CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 7w0d): > 3 in free list (2 min, 4 max allowed) > 0 hits, 0 misses, 486 trims, 487 created > 0 failures (0 no memory) > CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 7w0d): > 2 in free list (1 min, 2 max allowed) > 0 hits, 0 misses, 486 trims, 487 created > 0 failures (0 no memory) > IPC Medium buffers, 16384 bytes (total 2, permanent 2): > 2 in free list (1 min, 8 max allowed) > 0 hits, 0 fallbacks, 0 trims, 0 created > 0 failures (0 no memory) > IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 7w0d): > 17 in free list (16 min, 16 max allowed) > 0 hits, 0 misses, 78809 trims, 78810 created > 0 failures (0 no memory) > > Header pools: > Header buffers, 0 bytes (total 8705, permanent 8192, peak 8705 @ 7w0d): > 513 in free list (512 min, 16384 max allowed) > 8021 hits, 171 misses, 972 trims, 1485 created > 0 failures (0 no memory) > 8192 max cache size, 8192 in cache > 13200 hits in cache, 0 misses in cache > > Particle Clones: > 1024 clones, 19303 hits, 0 misses > > Public particle pools: > F/S buffers, 256 bytes (total 3072, permanent 3072): > 1024 in free list (512 min, 4096 max allowed) > 2048 hits, 0 misses, 0 trims, 0 created > 0 failures (0 no memory) > 2048 max cache size, 2048 in cache > 24526 hits in cache, 0 misses in cache > Normal buffers, 1548 bytes (total 3840, permanent 3840): > 3712 in free list (128 min, 4096 max allowed) > 128 hits, 0 misses, 0 trims, 0 created > 0 failures (0 no memory) > 128 max cache size, 128 in cache > 0 hits in cache, 0 misses in cache > CRYPTO_HW buffers, 1600 bytes (total 12288, permanent 12288): > 0 in free list (0 min, 12288 max allowed) > 12288 hits, 0 misses > 12288 max cache size, 12288 in cache > 234482 hits in cache, 0 misses in cache > > Private particle pools: > HQF Particle buffers, 0 bytes (total 2000, permanent 2000): > 2000 in free list (500 min, 2000 max allowed) > 0 hits, 0 misses, 0 trims, 0 created > 0 failures (0 no memory) > IDS SM buffers, 240 bytes (total 128, permanent 128): > 0 in free list (0 min, 128 max allowed) > 128 hits, 0 fallbacks > 128 max cache size, 128 in cache > 0 hits in cache, 0 misses in cache > GigabitEthernet0/0 buffers, 2304 bytes (total 2561, permanent 2048): > 513 in free list (512 min, 6144 max allowed) > 1877 hits, 171 misses, 972 trims, 1485 created > 0 failures (0 no memory) > 2048 max cache size, 1536 in cache > 31641103 hits in cache, 0 misses in cache > 14 buffer threshold, 0 threshold transitions > GigabitEthernet0/1 buffers, 2304 bytes (total 2561, permanent 2048): > 513 in free list (512 min, 6144 max allowed) > 1877 hits, 171 misses, 972 trims, 1485 created > 0 failures (0 no memory) > 2048 max cache size, 1536 in cache > 2796374359 hits in cache, 0 misses in cache > 14 buffer threshold, 0 threshold transitions > GigabitEthernet0/2 buffers, 2304 bytes (total 2561, permanent 2048): > 513 in free list (512 min, 6144 max allowed) > 1877 hits, 171 misses, 972 trims, 1485 created > 0 failures (0 no memory) > 2048 max cache size, 1536 in cache > 1095825533 hits in cache, 0 misses in cache > 14 buffer threshold, 0 threshold transitions > GigabitEthernet0/3 buffers, 2304 bytes (total 2561, permanent 2048): > 513 in free list (512 min, 6144 max allowed) > 1877 hits, 171 misses, 972 trims, 1485 created > 0 failures (0 no memory) > 2048 max cache size, 1536 in cache > 512 hits in cache, 0 misses in cache > 14 buffer threshold, 0 threshold transitions > > > 3) show controller > > > # sh controllers gigabitEthernet 0/2 > Interface GigabitEthernet0/2 (idb 0x22B2F87C) > Hardware is iGbE > Register base address: 0xFD120000 > PCI REV: A1 > network link is up > loopback type is none > > = Media RJ45 = > > Media Change Count: 1 > > MAC registers: > CTRL =0x000C1A41, STATUS=0x80080783, CTRL_X=0x12800480, IMS > =0x00000091 > EECD =0x00000218, EERD =0x00000000 ETT =0x00000000, > TXDMAC=0x00000000 > RCTL =0x00408032, RDBAL =0x10092000, RDBAH =0x00000000, RDLEN > =0x00002000 > RDH =0x00000000, RDT =0x000001FF, RDTR =0x0000130D, RLPML > =0x000005F4 > RFCTL =0x00800000 > TCTL =0xA40400FA, TDBAL =0x10095000, TDBAH =0x00000000, TDLEN > =0x00004000 > TDH =0x00000178, TDT =0x00000178, TIPG =0x00701808 > TDWAH =0x00000000, TDWAL =0x0FA13803 > FCRTH =0x00000000, FCRTL =0x00000000 > FCAH =0x00000100, FCAL =0x00C28001, FCT =0x00008808, FCTTV > =0x00000000 > RDFH =0x40021A70, RDFT =0x00001A24, RDFPC =0x00000000 > TDFH =0x4002130E, TDFT =0x0000130C, TDFPC =0x00000000 > RDFHS =0x00001AB0, RDFTS =0x00001AB0, RXDCTL=0x02010000 > TDFHS =0x0000130E, TDFTS =0x0000130E, TXDCTL=0x02000000 > RXCSUM=0x00000300, TIDV=0x00000020 > PCS_CFG =0x00000008, PCS_LCTL=0x02110000, PCS_LSTS =0x0061001D > PCS_ANADV =0x00000020, PCS_LPAB=0x0000D800, PCS_NPTX =0x00000000 > PCS_LPABNP=0x00000000, I2CCMD =0x00000000, I2CPARAMS=0x00001746 > > Rxdescriptor addr=10532100 len=48310437, stat=00000000 > Txdescriptor addr=0F8400FC cmd/len=0B00004E, stat=00000000 > RX is enabled, TX is enabled > Device status = full-duplex, link up > PCI width = 32, PCI66=0, PCIX=0, PCIXSPD is 50-66MHz Device Speed = > 1000Mbps > > PCI configuration registers: > bus_no=2, device_no=0, subdevice 0 > DeviceID=0x10C9, VendorID=0x8086, Command=0x0146, Status=0x0010 > Class=0x02/0x00/0x00, Revision=0x01, LatencyTimer=0x00, > CacheLineSize=0x10 > BaseAddr0=0xFD120000, BaseAddr1=0x00000000, MaxLat=0x00,MinGnt=0x00 > SubsysDeviceID=0x0000, SubsysVendorID=0x8086 > Interrupt Line 0x0B, Pin 0x01 > Cap_Ptr=0x00000040 Retry/TRDY Timeout=0x48235001 > PMC=0x00000000 PMCSR=0x00000003 > DEVCTRL=0x00090800 > > Internal Driver Information: > lc_ip_turbo_fs=0x0, ip_routecache=0x11(dfs=0/mdfs=0) > i82576_ds=0x22B30BB8, registers=0xFD120000 > mci_txcount 1024 rx cache size=2048, rx cache end=1536, rx_nobuffer=0 > max_mtu=1528 > ring sizes: RX=512, TX=1024 > rxring=0x10092000, shadow=0x22B3165C, head=0, rx_buf_size=2048 > txring=0x10095000, shadow=0x22B31E90, sw_tdh=376, tail=376 > chip_state=2, pci_rev=1 > congested = 0 tx_count=0, tx_ring_max=1024, tx_limited=0 (cur ring size > 1024) > rx_overrun=58854, rx_seq=0, rx_no_enp=0, rx_discard=0 > throttled=0, enabled=0, disabled=0, bypassed=0 > reset=4(init=1, check=0, restart=3, pci=0), auto_restart=8 > link_reset=0, tx_carrier_loss=4, fatal_tx_err=0 > wait_for_last_tdt=0, rx_stuck=0 > tx_stuck=0, rx_max_spin=1, rr_port_next=1 > > Statistics: > CRC error 0 Symbol error 0 > Missed Packets 58854 Excessive Coll 0 > Multiple Coll 0 Length Error 0 > Late Coll 0 Collision 0 > Defer 0 Receive Length 0 > XON RX 0 > XON TX 0 XOFF RX 0 > XOFF TX 0 FC RX Unsupport 0 > Packet RX (64) 0 Packet RX (127) 5486324 > Packet RX (255) 2568423 Packet RX (511) 764976 > Packet RX (1023) 598611 Packet RX (1522) 3299107 > Good Packet RX 12717441 Broadcast RX 0 > Multicast RX 12688463 Good Packet TX 44078 > Good Octets RX.H 1 Good Octets RX.L 1954032566 > Good Octets TX.H 0 Good Octets TX.L 5414177 > RX No Buff 9421 RX Undersize 0 > RX Fragment 0 RX Oversize 0 > RX Octets High 1 RX Octets Low 1961371318 > TX Octets High 0 TX Octets Low 5414177 > TX Packet 44078 RX Packet 12776868 > TX Broadcast 0 TX Multicast 5968 > Packet TX (64) 3912 Packet TX (127) 32710 > Packet TX (255) 2567 Packet TX (511) 4889 > Packet TX (1023) 0 Packet TX (1522) 0 > TX No CSR 0 > RX Error Count 0 TCP Segmentation 0 > Host Trasmit Discarded packets by MAC Count 0 > Interrupt Assertion Count 5809488 > Receive Packets to Host Count 12717440 > Host Good Packets Transmitted Count 44078 > > > PHY REGISTERS: > Display for CRC check for page:6 > > PACKET_GENERATION reg_addr: 16: 0x0010 > CRC COUNTERS reg_addr: 17: 0xFF00 > > page number:0 > > Copper Contrl @ reg_addr: 0: 0x1140 > Copper Status @ reg_addr: 1: 0x796D > PHY Idf1 @ reg_addr: 2: 0x0141 > PHY Idf2 @ reg_addr: 3: 0x0C97 > Autoneg Adv @ reg_addr: 4: 0x0C01 > Autoneg Lnk Ptr abty @ reg_addr: 5: 0xC001 > Autoneg Exp @ reg_addr: 6: 0x000D > Autoneg Nx Pg Tx @ reg_addr: 7: 0x2801 > Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x4D26 > 1000B-T Ctr @ reg_addr: 9: 0x0E00 > 1000B-T St @ reg_addr: 10: 0x3800 > Copper Ext St @ reg_addr: 15: 0x3000 > Copper Specific Ctl1 @ reg_addr: 16: 0x6060 > Copper Specific St1 @ reg_addr: 17: 0xAC08 > Copper Interrupt En @ reg_addr: 18: 0x0000 > Copper Specific St2 @ reg_addr: 19: 0x0000 > Copper Rcv Er Ctr @ reg_addr: 21: 0x0000 > Copper Specific Ctl2 @ reg_addr: 26: 0x8040 > > page number:1 > > Fiber Contrl @ reg_addr: 0: 0x1140 > Fiber Status @ reg_addr: 1: 0x794D > PHY Idf1 @ reg_addr: 2: 0x0141 > PHY Idf2 @ reg_addr: 3: 0x0C97 > Autoneg Adv @ reg_addr: 4: 0x0060 > Autoneg Lnk Ptr abty @ reg_addr: 5: 0x0000 > Autoneg Exp @ reg_addr: 6: 0x0004 > Autoneg Nx Pg Tx @ reg_addr: 7: 0x2001 > Autoneg Lk Prt Nx Pg @ reg_addr: 8: 0x0000 > Fiber Ext St @ reg_addr: 15: 0x3000 > Fiber Specific Ctl1 @ reg_addr: 16: 0x0000 > Fiber Specific St1 @ reg_addr: 17: 0x0018 > Fiber Interrupt En @ reg_addr: 18: 0x0000 > Fiber Specific St2 @ reg_addr: 19: 0x0000 > Fiber Rcv Er Ctr @ reg_addr: 21: 0x0000 > Fiber Specific Ctl2 @ reg_addr: 26: 0x8085 > > page number:2 > > MAC Contrl @ reg_addr: 0: 0x1040 > PHY Idf1 @ reg_addr: 2: 0x0141 > PHY Idf2 @ reg_addr: 3: 0x0C97 > MAC Specific Ctl1 @ reg_addr: 16: 0x0288 > MAC Specific St1 @ reg_addr: 17: 0x0440 > MAC Interrupt En @ reg_addr: 18: 0x0000 > MAC Specific St2 @ reg_addr: 19: 0x0000 > MAC Specific Ctl2 @ reg_addr: 26: 0x0005 > > HW addr filter: 0x22B33EC4 > > Chip address filter[0] 80000293.16032B44 > Chip address filter[1] 80000E00.00C28001 > Chip address filter[2] 8000CCCC.CC0C0001 > Chip address filter[3] 80000700.00C28001 > Chip address filter[4] 80000200.00C28001 > Chip address filter[5] 03FFFFFF.005E0001 > Chip address filter[6] 03FFFFFF.005E0001 > Chip address filter[7] 03FFFFFF.005E0001 > Chip address filter[8] 03FFFFFF.005E0001 > Chip address filter[9] 03FFFFFF.00000000 > Chip address filter[10] 03FFFFFF.00000000 > Chip address filter[11] 03FFFFFF.00000000 > Chip address filter[12] 03FFFFFF.00000000 > Chip address filter[13] 03FFFFFF.00000000 > Chip address filter[14] 03FFFFFF.00000000 > Chip address filter[15] 03FFFFFF.00000000 > Entry= 0: Addr=442B.0316.9302 > Entry= 1: Addr=0180.C200.000E > Entry= 2: Addr=0100.0CCC.CCCC > Entry= 3: Addr=0180.C200.0007 > Entry= 4: Addr=0180.C200.0002 > (All other entries are empty) > > Software MAC address filter(hash:length/addr/mask/hits) > ------------------------------------------------------- > 0x00: 0 ffff.ffff.ffff 0000.0000.0000 5 > 0x01: 0 442b.0316.9302 0000.0000.0000 1035374 > 0xC0: 0 0100.0ccc.cccc 0000.0000.0000 12399 > 0xC0: 1 0180.c200.0002 0000.0000.0000 0 > 0xC5: 0 0180.c200.0007 0000.0000.0000 0 > 0xCC: 0 0180.c200.000e 0000.0000.0000 2 > 0x00: 0 0100.5e7f.ffff 0000.007f.ffff 1094516179 > > Software filtered frames: 261439 > unicast overflow mode off > multicast overflow mode on > Promiscuous mode off > > > > > > > > > > > > > > Il giorno gio 30 gen 2025 alle ore 16:41 Brian Turnbow < > [email protected]> ha scritto: > >> Hi James >> >> You can use >> sh buffers and sh controller g0/2 for more info and then try and tune >> buffers, but this doesn't help a lot usually. >> Another thing to check would be for microbursts using the old rate limit >> trick setting a policer with exceed action transmit >> Then you can check to see if you are overflowing therate limit bucket as >> sign of microburst. Like maybe due to speed steps 10G/1G. >> Overruns are a sign that the router can't handle the traffic coming in >> and may be due to traffic levels or traffic types that take longer >> to process. >> Flow control could help but I see unsupported listed on the interface >> and admit I am not well versed on the 3925 as a platform >> Have you tried a reboot ? I have seen routers get out of whack that >> magically return to form after a reboot.. >> >> >> >> Brian Turnbow >> >> >> >> Il giorno gio 30 gen 2025 alle ore 14:41 james list via cisco-nsp < >> [email protected]> ha scritto: >> >>> Dear expert >>> I'm trying to get any useful insight and I can also understand not polite >>> comments... >>> >>> We've a customer which is experiencing multicast gaps on feed A and we've >>> identified an old C3925-SPE200 in a remote location (hence difficult to >>> be >>> replaced in a manner) which is experiencing input errors/overrun also on >>> low bandwidth and packet rate. >>> The customer has the same kind of device in another remote location >>> receiving the same multicast feed (feed B) without issues. >>> I think these gaps are due to packet dropped ... customer has also tried >>> to >>> move interface from g0/1 to g0/2 but no improvement has obtained. >>> >>> Any idea what could be the reason ? >>> >>> Thanks in advance >>> James >>> >>> xxx>sh inventory >>> NAME: "CISCO3925-CHASSIS", DESCR: "CISCO3925-CHASSIS" >>> PID: CISCO3925-CHASSIS , VID: V02, SN: FCZxxx >>> >>> NAME: "Cisco Services Performance Engine 200 for Cisco 3900 ISR on Slot >>> 0", >>> DESCR: "Cisco Services Performance Engine 200 for Cisco 3900 ISR" >>> PID: C3900-SPE200/K9 , VID: V04 , SN: FOC160320B7xxx >>> >>> NAME: "C3900 AC Power Supply 1", DESCR: "C3900 AC Power Supply 1" >>> PID: PWR-3900-AC , VID: V03, SN: SNIxxx >>> >>> NAME: "C3900 AC Power Supply 2", DESCR: "C3900 AC Power Supply 2" >>> PID: PWR-3900-AC , VID: V03 , SN: QCSxxx >>> >>> >>> xxx>sh int g0/2 >>> GigabitEthernet0/2 is up, line protocol is up >>> Hardware is iGbE, address is 442b.0316.9302 (bia 442b.0316.9302) >>> Description: To Market >>> MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, >>> reliability 255/255, txload 1/255, rxload 2/255 >>> Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set >>> Keepalive set (10 sec) >>> Full Duplex, 1Gbps, media type is RJ45 >>> output flow-control is unsupported, input flow-control is unsupported >>> ARP type: ARPA, ARP Timeout 04:00:00 >>> Last input 00:00:00, output 00:00:00, output hang never >>> Last clearing of "show interface" counters 01:18:46 >>> Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 >>> Queueing strategy: fifo >>> Output queue: 0/40 (size/max) >>> 30 second input rate 9004000 bits/sec, 2628 packets/sec >>> 30 second output rate 9000 bits/sec, 7 packets/sec >>> 7936911 packets input, 4224448667 bytes, 0 no buffer >>> Received 7914597 broadcasts (7218973 IP multicasts) >>> 0 runts, 0 giants, 0 throttles >>> 56932 input errors, 0 CRC, 0 frame, 56932 overrun, 0 ignored >>> 0 watchdog, 7914597 multicast, 0 pause input >>> 30904 packets output, 2951099 bytes, 0 underruns >>> 0 output errors, 0 collisions, 0 interface resets >>> 158 unknown protocol drops >>> 0 babbles, 0 late collision, 0 deferred >>> 0 lost carrier, 0 no carrier, 0 pause output >>> 0 output buffer failures, 0 output buffers swapped out >>> >>> >>> interface GigabitEthernet0/2 >>> description To Market >>> no ip address >>> load-interval 30 >>> duplex full >>> speed 1000 >>> hold-queue 2000 in >>> ! >>> interface GigabitEthernet0/2.90 >>> description VLAN multicast >>> encapsulation dot1Q 90 >>> ip address xx.xx.221.1 255.255.255.0 >>> ip pim query-interval 5 >>> ip pim bsr-border >>> ip pim dense-mode proxy-register list xx >>> ip multicast boundary xx >>> ip nat outside >>> ip virtual-reassembly in >>> ! >>> interface GigabitEthernet0/2.1683 >>> description To market flex vpn >>> encapsulation dot1Q 1683 >>> ip address 10.xx.xx.85 255.255.255.252 >>> ip access-group CRYPTO-xxx >>> _______________________________________________ >>> cisco-nsp mailing list [email protected] >>> https://puck.nether.net/mailman/listinfo/cisco-nsp >>> archive at http://puck.nether.net/pipermail/cisco-nsp/ >>> >> _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
