Back in March I posted the message below. Since then, I've finally been
able to track this down, and wanted to share.
I was able to determine where this traffic was coming from from the tcp
dump. I contacted the school IT administrator, who was helpful in
monitoring his own local traffic. The odd thing was that he wasn't
seeing any traffic, while Lightpath (Optimum) was seeing it.
I went over there yesterday to meet with him and look at his network. I
saw that Lightpath had an additional router, a Cisco 1800, between the
school's firewall and Lightpath's network. This router showed an
extreme amount of traffic, seen by observing the lights.
We powercycled the 1800, and the bad traffic went away.
I suggested that they put in place a local monitoring system, such as
Zabbix (which I use). If he had been able to see bandwidth graphs of
both his router and the Lightpath router, the problem would have been
obvious.
Now we're waiting to see what Lightpath does. I'd like them to replace
the 1800, but a firmware upgrade may solve the problem.
JBB
On 3/9/12 11:42 AM, Jonathan Bayer wrote:
My company has been hit with what we thought was some sort of DDOS
attack. The symptom was that our incoming bandwidth, which is
normally 10-15 Mbit during the day, jumped by about 30 Mbit, and the
30 Mbit stayed fairly constant.
This is the 4th time this has happened. We've finally been able to
get a tcp dump, I'm linking wireshark images.
This are the facts as I've been able to identify them:
1. Our customers are school districts. The districts typically NAT
everything through a single IP, so our normal traffic is usually
several hundred connections from each IP
2. All the excess bandwidth is coming on a single connection
3. The connection seems to always come from the same IP address. The
IP address resolves to one of our customers
4. Wireshark shows that the traffic appears to be a constant stream of
Retransmits. The URL being retransmitted is valid.
5. The traffic passes through our firewall, but doesn't get past the
load balancer
6. When I had our ISP break the connection at the firewall, the
traffic continued to be received at the firewall, but was not passed
through to the load balancer
7. These episodes seem to be getting longer. The first one lasted a
several hours, the last one several days.
This is totally out of my experience.
Is it possible for some mis-configured equipment to cause this to happen?
I'm looking for any sort of suggestions. So far this hasn't hurt us,
but if it gets larger, it will.
I didn't want to attach the capture because of the size, but if
someone want to take a closer look, I'd be happy to send it along.
Thanks in advance.
500 frames captured.
The first one is a normal GET
1 thru 13 are TCP OUT-OF-ORDER
14-500 are Retransmits
here are the image links; they were too big to attach:
http://mrld.info/w1.gif
http://mrld.info/w2.gif
_______________________________________________
Discuss mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Discuss mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss
This list provided by the League of Professional System Administrators
http://lopsa.org/
