Thanks for all the replies. One of our OC3s had the FREF light on, sometimes flashing too. I started pulling IMA T1s to see if it was customer traffic. None of them fixed the problem. When I pulled that particular OC3 the problem cleared up.
I did see flaps in the log for that particular OC3. They were happening approximately every 60s. Looking at the log a little closer I see that ATM3/0 would come up and then drop within 3-4s, wait 60s, rinse and repeat. I have on that OC3 about 300 PVCs. None of the PVCs have a ubr over 1664. We're using RBE instead of PPPoX. I have ingress & egress ACLs on each PVC as well as uRPF. The vast majority of the PVCs are unnumbered to a loopback, though a few have /30s. Other than that the interface config is basic. There isn't any QoS or shaping going on, other than setting the ubr. When I started disconnecting circuits I was speculating that customer traffic could have been to blame. In the log I noticed repeating ACL drops from RFC1918 IPs to our name servers. There weren't that many hits on the ACL but there were some occasional ACL rate-limit overruns. I've seen far worse though. I can't remember if ACLs happen in HW on the 3660 or not; even if they do ACL logging is a process level function isn't it? Every time that OC3 bounces that's about 300 sub-ints bouncing with it. I imagine repeated bounces would be a painful thing. I've pulled the carrier's OC3 card to get the 3660 back online. I'm planning on shutting down all 300 sub-ints on that OC3 and bring the OC3 back up. If the 3660 is still ok I'll enable about 50 PVCs at a time until the box either flips out or I have all the PVCs back online. I have a spare carrier card but I don't think I have a spare OC3 NM so hopefully that's not the problem. Thanks for all the input. Justin Christian wrote: > the net background proess. sends interface keepalive packets and > processes interface state changes > > On Fri, Apr 4, 2008 at 10:42 AM, Justin Shore <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> wrote: > > I just had a 3660 terminating about 1500 PVCs for DSL die this AM. The > 3660 is running 12.3(22) w/ 256MB of RAM. I'm at the remote POP now, > consoled in and looking at the box. The console hangs for 45-60s before > I get a chance to run a quick command or two. Then it goes back to the > hung state about 5 seconds later. The box keeps dropping my 2 OC3s and > bouncing my IS-IS processes. Syslog didn't appear to capture anything > interesting leading up to this crash. I had the box power cycled while > I was driving down here. > > Overall the CPU is low and neither CPU nor memory were being taxed over > the past number of months. The Net Background process is using a fair > bit of the processor. Actually, that's an understatement. At times > it's only using 9% or so. At other times it's up to 97% as was the case > here: > > 3660-2.brd#sh proc cpu sort 5m > CPU utilization for five seconds: 1%/0%; one minute: 17%; five > minutes: 19% > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process > 28 4715940 611 7731049 97.09% 13.74% 9.23% 0 Net > Background > 31 238608 865 275847 2.65% 8.58% 8.48% 0 > Per-Second Jobs > 5 8336 684 12187 0.00% 0.56% 0.68% 0 Check > heaps > 60 7764 7776 998 0.01% 0.27% 0.43% 0 IP Input > > 18 3856 8741 441 0.00% 0.42% 0.33% 0 ARP > Input > 167 2556 3046 839 0.02% 0.12% 0.13% 0 ISIS Upd > > 140 2732 15 182133 0.00% 0.00% 0.12% 0 Key Proc > > 179 1680 1676 1002 0.00% 0.13% 0.11% 0 OSPF > Router > 47 1912 152 12578 0.00% 0.12% 0.11% 0 > Per-minute Jobs > 82 1792 597 3001 0.01% 0.15% 0.11% 0 IP > Background > 3 1828 238 7680 0.00% 0.11% 0.09% 0 OSPF > Hello > 100 644 1217 529 0.00% 0.13% 0.08% 0 DHCPD > Receive > 51 1700 376 4521 0.00% 0.04% 0.05% 0 ATM > Periodic > 83 700 417 1678 0.00% 0.01% 0.02% 0 IP RIB > Update > 108 500 871 574 0.00% 0.06% 0.02% 0 CEF > process > 124 172 471 365 0.00% 0.04% 0.02% 0 Exec > > 166 620 1908 324 0.00% 0.03% 0.02% 0 ISIS Adj > > 106 420 284 1478 0.00% 0.03% 0.00% 0 Adj > Manager > 6 436 1406 310 0.00% 0.02% 0.00% 0 Pool > Manager > 91 188 556 338 0.00% 0.01% 0.00% 0 ILMI > Timer Proce > 163 92 1795 51 0.00% 0.00% 0.00% 0 CLNS > Input > > > Per-Second Jobs is also using quite a bit of CPU. The log has CPUHOG > entries in it pointing at the Net Background process too: > > 000428: .Apr 4 08:36:31 CDT: %SYS-3-CPUHOG: Task ran for 50332 msec > (10/0), process = Net Background, PC = 60439FFC. > -Traceback= 6043A004 > > What does the Net Background process do? I haven't been able to find > any answers on Cisco.com. > > > We're in the middle of SmartNet renewals and negotiations and > unfortunately this box's contract has expired. I can have our AM bless > a TAC case for us if need be, assuming I can locate him; I think he's in > training this week back East. > > Any ideas? I'm going to go back and scrutinize the log for anything > useful. As of right now the box is intermittently pingable, > corresponding directly to the brief instances that I can use the > console. Customer packets seem to flow about as well. > > Thanks > Justin > > _______________________________________________ > cisco-nsp mailing list [email protected] > <mailto:[email protected]> > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > > _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
