I am seeing a weird problem on one of my aggregation routers. This is a 7206VXR, with a PA-MC-T3 and a PA-MC-4T1 cards. I terminate about 10 T1s on the MC-T3 card (all local), and 2 long-distance T1s on the PA-MC-4T1 card. All services come off the same AT&T multiplexer that's hanging on the wall 20 feet away. On the far end of each of the T1s is a Cisco router (either a 1750 or 1841) with a WIC-1DSU-T1. This setup worked fine for several years.
Few months ago we started seeing line protocol bouncing on the 2 LD circuits, and less frequently, on some of the T3 channels. Most of the time it would go down for ~10seconds and come back up. Long enough for users to complain. Sometimes it would stay down for up to a minute, and sometimes (rarely) would be accompanied by a carrier transition. The far sides are not recording anything that would correlate. The drops would not be systematic, and we could not correlate them to any other activities or events. Sometimes they would accrue 2 drops in a 24-hr period, and sometimes 12 or more. Over time, drops started happening more frequently. Telco, of course, cannot find any problems, even after I made them run an extended (1hr+) test on each circuit. I replaced the PA-MC-4T1 card, the patch cables, and the remote routers. No change. Last week I swapped out the NPE-300 for an NPE-400 (accompanied by IOS switch from 12.4.4.T7 to 12.4.25d). Yes, I know it's not the latest, but I am running it successfully on several other similar routers. The NPE-400 was pulled from a functional router (it was, in turn, upgraded to a NPE-G1). Now the frequency of these drops skyrocketed. Below is the log just from today. My main question is: can these issues be caused by a router component beyond the PA card (i.e. engine or backplane)? As far as I understand the 7200 architecture, L1 and L2 would be terminated on the PA, and since I see same problems on multiple PAs, it shouldn't be the router. In which case the common point would be the AT&T multiplexer, or something beyond it in the telco network. Am I thinking correctly about this? Any thoughts or suggestions would be appreciated. Aug 30 2012 03:13:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 03:13:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Aug 30 2012 03:14:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to down Aug 30 2012 03:14:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to up Aug 30 2012 03:15:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to down Aug 30 2012 03:15:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to up Aug 30 2012 03:16:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 03:16:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 04:36:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 04:36:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 07:19:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to down Aug 30 2012 07:19:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to up Aug 30 2012 07:46:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 07:46:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Aug 30 2012 07:50:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 07:50:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 07:50:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 07:50:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 07:57:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to down Aug 30 2012 07:57:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 07:57:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to up Aug 30 2012 07:57:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Aug 30 2012 08:11:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 08:11:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 08:13:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 08:14:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 08:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 08:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 08:52:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 08:53:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 09:24:09 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to down Aug 30 2012 09:24:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to up Aug 30 2012 09:25:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 09:26:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 09:26:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 09:26:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 09:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 09:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 09:37:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 09:37:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Aug 30 2012 09:38:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to down Aug 30 2012 09:38:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 09:38:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 09:38:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to up Aug 30 2012 09:41:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to down Aug 30 2012 09:41:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 09:41:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Aug 30 2012 09:41:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/10:0, changed state to up Aug 30 2012 09:43:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 09:43:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 09:45:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 09:46:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 09:46:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 09:46:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 09:47:09 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to down Aug 30 2012 09:47:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to up Aug 30 2012 10:11:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 10:11:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 10:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 10:37:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 10:48:47 CST: %CONTROLLER-5-UPDOWN: Controller T1 4/0, changed state to down Aug 30 2012 10:48:49 CST: %LINK-3-UPDOWN: Interface Serial4/0:0, changed state to down Aug 30 2012 10:48:50 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to down Aug 30 2012 10:49:17 CST: %CONTROLLER-5-UPDOWN: Controller T1 4/0, changed state to up Aug 30 2012 10:49:19 CST: %LINK-3-UPDOWN: Interface Serial4/0:0, changed state to up Aug 30 2012 10:49:20 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to up Aug 30 2012 11:36:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 11:36:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 11:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 11:37:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 11:48:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to down Aug 30 2012 11:48:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/7:0, changed state to up Aug 30 2012 11:50:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 11:51:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 11:58:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 11:58:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 11:59:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 11:59:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 12:00:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 12:00:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 12:01:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 12:01:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 12:01:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 12:01:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 12:02:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 12:02:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 12:09:08 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 12:09:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 12:09:39 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to down Aug 30 2012 12:09:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to up Aug 30 2012 12:09:49 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 12:09:59 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Aug 30 2012 12:10:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to down Aug 30 2012 12:10:29 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/0:0, changed state to up Aug 30 2012 12:20:59 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 12:21:09 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Aug 30 2012 12:22:38 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to down Aug 30 2012 12:22:58 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/4:0, changed state to up Aug 30 2012 12:36:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 12:36:48 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 12:37:18 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to down Aug 30 2012 12:37:19 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to down Aug 30 2012 12:37:28 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0/1:0, changed state to up Aug 30 2012 12:37:29 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial4/1:0, changed state to up Sincerely, Michael Malitsky _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
