Hi all,
An update to this issue: we have been able to reproduce the problem in our
labs, but only under specific conditions. At this point, the customer has
not been able to confirm or deny the configuration items in effect during
this problem.
Essentially what we found was that if fast switching was in use, and if
there are multiple equal cost routes for the same destination, the router
will install host routes for each destination to ensure load balancing
across equal cost paths.
Under these conditions, scanning an entire class A network will use up all
of the routers memory in short order.
To avoid this problem, I would recommend using CEF (Cisco Express
Forwarding) which handles equal cost paths differently, and more
efficiently than the fast switching model detailed above. CEF is available
in IOS version 12.0 for most platforms.
Thank you,
Lisa Napier
Product Security Incident Response Team
Cisco Systems
At 07:12 PM 9/7/1999 -0700, Lisa Napier wrote:
Hi all,
Sorry for the delay in response.
At first glance, I had thought that this security advisory may have had
something to do with this issue.
http://cco/warp/customer/770/iossyslog-pub.shtml
This details a specific issue with the routers response to an invalid UDP
packet to the syslog port, however your report details strictly ICMP scan
and tcp port scans.
In speaking with the engineer who is working the case, he was unable to
view the issue 'live'; the people running NMAP had turned it off, just as
he had logged into your network, and functionality was pretty much restored.
We have not yet been able to reproduce the problem in house, but are
still working on it.
The customer support engineer working the case is trying to cause the
problems you saw with the scan parameters that you specified, in that it
was only scanning hosts that responded to a ping. If we presume that the
scan was actually attempting to scan an entire class A network, then the
behavior seen of maxing out the input queues and therefore memory
resources is an expected and nasty side effect, and we have indeed seen
that behavior.
Cisco's product security incident response team is monitoring the case at
this point, and expecting an update within the next few days.
For those on the list unaware of us, the Cisco PSIRT is the best point of
contact for reporting a security issue with any Cisco product. The URL
below gives more details on the Cisco PSIRT.
Thanks,
Lisa Napier
Product Security Incident Response Team
Cisco Systems
408 527-8747
http://www.cisco.com/warp/public/707/sec_incident_response.shtml
At 05:02 PM 8/31/1999 -0700, Lancashire, Andrew wrote:
I don't know if you've ever seen this before. We ran nmap with ICMP
discover and standard tcp scan. We ran the scan against the entire 10.0.0.0
network range. Although we were only looking for 2 ports, we found that the
RSM in our 5500 series (our default route) was running out of memory and
had to be rebooted by our Network Services group multiple times in the 18
hour stretch it took to complete. One of the interesting things is that we
were only generating about 3-5 Mbs and the 5500 can pass Gigabits. I have
not heard of this problem before. We contacted Cisco and sent them the
details. Below is the response to one of our engineers.
Andrew
-Original Message-
From: khollis [SMTP:[EMAIL PROTECTED]]
Sent: Tuesday, August 31, 1999 7:59 AM
To: [EMAIL PROTECTED]
Subject:Regarding Case Number V44290
Hi Dave, as I recall, the symptom we had to work/troubleshoot with was the
router consumed lots of memory. Never heard about packets being dropped. So
it seems like we forwarded everything nmap sent to us. The thing to keep in
mind is that the router will dynamically allocate memory as necessary so
that it can keep up with the load provided to it. Although we did not know
nmap was running at the time, we noticed the memory consumed by the IP Input
process dropped from 40M+ to an acceptable level of (4-5M) after nmap was
shut down. This proves that the router need this much memory to process the
entire load generated by nmap.
I suspect nmap was doing much more than you've been able to calculate. It's
obvious that running nmap continuously for 18-19 hours caused this problem.
One possible explaination is constantly flooding the router w/64 byte
packets for this timeframe could have caused the router's memory to become
seriously fragmented. Also, I guess we can't tell, but another question
would be how many tcp sessions were requested/open on the router after this
timeframe?
Port scanners have a reputation of helping identify potential security
problems. However, they are also known to cause problems...
Hope this helps,
KennyH.
