So, this was tracked down to be an issue with the ASA doing debug logging. As soon as we changed the logging back down to alert level logging, the issue resolved.
Only caught the issue when watching the Process CPU-Usage and saw that the logger process pop up at 52%, then 64% and then 84% CPU usage before falling off in a 10 second period. vpn-gw# sh proc cpu-usage non-zero Hardware: ASA5516 Cisco Adaptive Security Appliance Software Version 9.16(4)57 ASLR enabled, text region 56474105f000-564744cef285 PC Thread 5Sec 1Min 5Min Process 0x0000564743c1c050 0x00007f2cef4bbe80 1.0% 1.5% 1.0% Unicorn Proxy Thread 0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.2% 0.3% 0x00005647439821d6 0x00007f2cef4bbae0 0.0% 0.2% 0.1% snmp_master_callback_thread 0x0000564743982226 0x00007f2cef4bb740 0.0% 0.4% 0.4% snmp_client_callback_thread 0x00005647437cf58c 0x00007f2cef4be2c0 0.0% 0.1% 0.1% radius_snd 0x00005647421c5dd4 0x00007f2cef4dc4e0 52.1% 11.5% 11.1% Logger 0x00005647427c3cb6 0x00007f2cef4c5e00 0.0% 0.1% 0.1% ARP Thread 0x0000564743c1c050 0x00007f2cef4ebf00 0.0% 0.1% 0.1% aaa_shim_thread 0x0000564741cdf31c 0x00007f2cef4ec640 0.0% 0.1% 0.1% aaa - - 0.8% 2.1% 2.2% DATAPATH-0-1665 - - 2.6% 2.3% 2.4% DATAPATH-1-1666 vpn-gw# sh proc cpu-usage non-zero Hardware: ASA5516 Cisco Adaptive Security Appliance Software Version 9.16(4)57 ASLR enabled, text region 56474105f000-564744cef285 PC Thread 5Sec 1Min 5Min Process 0x0000564743bf433f 0x00007f2cef4bbe80 0.0% 1.2% 1.0% Unicorn Proxy Thread 0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.1% 0.2% 0x00005647439821d6 0x00007f2cef4bbae0 0.0% 0.1% 0.1% snmp_master_callback_thread 0x0000564743982226 0x00007f2cef4bb740 0.0% 0.2% 0.4% snmp_client_callback_thread 0x00005647421c5dd4 0x00007f2cef4dc4e0 64.1% 12.7% 11.3% Logger 0x00005647427c3cb6 0x00007f2cef4c5e00 0.0% 0.1% 0.1% ARP Thread - - 1.3% 2.1% 2.2% DATAPATH-0-1665 - - 4.7% 2.5% 2.4% DATAPATH-1-1666 vpn-gw# sh proc cpu-usage non-zero Hardware: ASA5516 Cisco Adaptive Security Appliance Software Version 9.16(4)57 ASLR enabled, text region 56474105f000-564744cef285 PC Thread 5Sec 1Min 5Min Process 0x0000564743c1c050 0x00007f2cef4bbe80 0.0% 0.9% 0.9% Unicorn Proxy Thread 0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.1% 0.2% 0x00005647439821d6 0x00007f2cef4bbae0 0.0% 0.1% 0.0% snmp_master_callback_thread 0x0000564743982226 0x00007f2cef4bb740 0.0% 0.1% 0.3% snmp_client_callback_thread 0x00005647421c5dd4 0x00007f2cef4dc4e0 84.0% 15.1% 11.8% Logger - - 3.0% 2.3% 2.3% DATAPATH-0-1665 - - 0.4% 2.3% 2.4% DATAPATH-1-1666 vpn-gw# sh proc cpu-usage non-zero Hardware: ASA5516 Cisco Adaptive Security Appliance Software Version 9.16(4)57 ASLR enabled, text region 56474105f000-564744cef285 PC Thread 5Sec 1Min 5Min Process 0x0000564743c1c050 0x00007f2cef4bbe80 1.3% 1.0% 0.9% Unicorn Proxy Thread 0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.1% 0.2% 0x0000564743982226 0x00007f2cef4bb740 0.0% 0.1% 0.3% snmp_client_callback_thread 0x00005647421c5dd4 0x00007f2cef4dc4e0 0.0% 12.8% 11.4% Logger 0x000056474221df82 0x00007f2cef4bc5c0 0.1% 0.1% 0.1% emweb/https 0x00005647427c3cb6 0x00007f2cef4c5e00 0.1% 0.0% 0.0% ARP Thread - - 2.2% 2.3% 2.3% DATAPATH-0-1665 - - 2.4% 2.3% 2.4% DATAPATH-1-1666 vpn-gw# sh proc cpu-usage non-zero Best, -Lee On Wed, Jun 5, 2024 at 2:22 PM Lee Starnes <[email protected]> wrote: > Thank you for the link and info. Unfortunately can['t open a TAC case as > this model (5516-X) is not under support. We have a 5508-X under contract > which is how we are able to get the firmware. > > I will check out the links. Thank you for your help. > > Best, > > -Lee > > On Wed, Jun 5, 2024 at 6:15 AM harbor235 <[email protected]> wrote: > >> Here is an overall performance troubleshooting oc: >> >> >> https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113185-asaperformance.html >> >> Mike >> >> On Wed, Jun 5, 2024 at 9:12 AM harbor235 <[email protected]> wrote: >> >>> If you cannot open a TAc case I would look through your syslog messages >>> looking for errors/critcals/warnings. Also look at all interfaces to ensure >>> there are no input or output errors as well. After that I would verify >>> traffic is hitting your box and is not an upstream problem. While looking >>> at the interfaces for errors here is a good doc: >>> >>> >>> https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115985-asa-overrun-product-tech-note-00.pdf >>> >>> Do you have a router fronting the FW? Make sure you look at all >>> interfaces for high traffic or traffic directed to the FW itself. >>> >>> Hard to tell what it could be without more info. >>> >>> Mike >>> >>> On Tue, Jun 4, 2024 at 7:41 PM harbor235 <[email protected]> wrote: >>> >>>> Hi Lee, >>>> >>>> So logger looks like it is consuming a bit of CPU, it is worth noting >>>> though the data and management plane separation. Logger should not impact >>>> packet forwarding in the data plane. >>>> >>>> Hard to tell what is going on, you must have a support contract you >>>> upgraded the code, did you try to open a TAC case? >>>> >>>> >>>> Mike >>>> >>>> On Tue, Jun 4, 2024 at 7:24 PM Lee Starnes <[email protected]> >>>> wrote: >>>> >>>>> The unit just runs basic NAT-PAT and AnyConnect VPN. No IPS or >>>>> anything setup on this. And only about 9Mbps peak. >>>>> >>>>> -Lee >>>>> >>>>> On Tue, Jun 4, 2024 at 4:03 PM harbor235 <[email protected]> wrote: >>>>> >>>>>> What features do you have enabled? NGFW and/or NGIPS? These features >>>>>> can limit the box to 450Mbps. >>>>>> >>>>>> Mike >>>>>> >>>>>> On Tue, Jun 4, 2024 at 6:53 PM Lee Starnes via cisco-nsp < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hello Everyone, >>>>>>> >>>>>>> I have an odd issue trying to track down. We are seeing issue whereby >>>>>>> traffic just "pauses" through the ASA for about 2-4 seconds before >>>>>>> resuming. >>>>>>> >>>>>>> We started seeing this when the device was low on memory (about 600M >>>>>>> available). we rebooted it and did an firmware update the current >>>>>>> version. >>>>>>> >>>>>>> Still seeing this behavior. >>>>>>> >>>>>>> After another reboot, still seeing this. >>>>>>> >>>>>>> Process: DATAPATH-0-1665, PROC_PC_TOTAL: 407, MAXHOG: 10, >>>>>>> LASTHOG: 5 >>>>>>> MAXHOG At: 15:31:54 PDT Jun 4 2024 >>>>>>> LASTHOG At: 15:37:48 PDT Jun 4 2024 >>>>>>> PC: 0x0000000000000000 (suspend) >>>>>>> >>>>>>> Process: DATAPATH-0-1665, NUMHOG: 385, MAXHOG: 10, LASTHOG: 5 >>>>>>> MAXHOG At: 15:31:54 PDT Jun 4 2024 >>>>>>> LASTHOG At: 15:37:48 PDT Jun 4 2024 >>>>>>> PC: 0x0000000000000000 (suspend) >>>>>>> Call stack: 0x0000564741c98c49 0x0000564742188996 >>>>>>> 0x00005647436c2d28 >>>>>>> 0x00005647436d2abc 0x00005647436e2ae0 >>>>>>> 0x00007f2d2067bff5 >>>>>>> 0x00007f2d1f88416f >>>>>>> >>>>>>> >>>>>>> Process: DATAPATH-1-1666, PROC_PC_TOTAL: 402, MAXHOG: 12, >>>>>>> LASTHOG: 5 >>>>>>> MAXHOG At: 15:31:48 PDT Jun 4 2024 >>>>>>> LASTHOG At: 15:37:41 PDT Jun 4 2024 >>>>>>> PC: 0x0000000000000000 (suspend) >>>>>>> >>>>>>> Process: DATAPATH-1-1666, NUMHOG: 376, MAXHOG: 12, LASTHOG: 5 >>>>>>> MAXHOG At: 15:31:48 PDT Jun 4 2024 >>>>>>> LASTHOG At: 15:37:41 PDT Jun 4 2024 >>>>>>> PC: 0x0000000000000000 (suspend) >>>>>>> Call stack: 0x0000564741c98c49 0x0000564742188996 >>>>>>> 0x00005647436c2d28 >>>>>>> 0x00005647436d2abc 0x00005647436e2ae0 >>>>>>> 0x00007f2d2067bff5 >>>>>>> 0x00007f2d1f88416f >>>>>>> >>>>>>> >>>>>>> >>>>>>> I did disable logging flash-bufferwrap to stop it from writing to >>>>>>> flash. >>>>>>> The logging process stopped using 29% CPU, but still the issue >>>>>>> persists. >>>>>>> >>>>>>> Anyone got any Ideas on what the cause is and how to resolve it? >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> -Lee >>>>>>> _______________________________________________ >>>>>>> cisco-nsp mailing list [email protected] >>>>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp >>>>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/ >>>>>>> >>>>>> _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
