Re: Nachi Worm apparently causes "Live Lock" on 4.7 server

[Jim Durham]

On Friday 29 August 2003 01:14 am, paul beard wrote:
> James C. Durham wrote:
> > On Friday 29 August 2003 04:23 am, paul wrote:
> >>James C. Durham wrote:
> >>>It turned out that we had several Windows boxes in the building
> >>> that had been infected with the Nachi worm. This causes some
> >>> kind of DOS or ping probe out onto the internet and the local
> >>> LAN.
> >>>
> >>>Removing the inside interface's ethernet cable caused the ping
> >>> times on the outside interface to go back to the normal .4
> >>> milliseconds to the router.
> >>>
> >>>Apparently, the blast of packets coming from the infected boxes
> >>> managed to cause a "live lock" condition in the server. I
> >>> assume it was interrupt bound servicing the inside interface.
> >>> The packets were ICMP requests to various addresses.
> >>
> >>I could be way off here, but is there any way to isolate machines
> >>that send a sudden blast of packets, either by destination
> >> address (make a firewall rule that drops those packets) or
> >> working out their MAC addresses and dropping their connectivity?
> >> Or scan for open ports and block unsecured systems from
> >> connecting?
> >
> > What I did was go in the switch room and look for pulsing lights
> > on the switch ports and pull the cables. That fixed it, but after
> > much agony.
>
> well, that's a bit draconian, but effective ;-)
>
> >>>My questions is.. what, if any, is a technique for preventing
> >>> this condition? I know, fix the windows boxes, but  I can't
> >>> continually check the status of the virus software and patch
> >>> level of the Windows boxes. There are 250 plus of them and one
> >>> of me. Users won't install upgrades even when warned this worm
> >>> thing was coming. But, i'd like to prevent loss of service when
> >>> one of Bill's boxes goes nuts!
> >>
> >>Where I work, at the University of Washington, the network staff
> >>were dropping as many as 200 machines *per day* off the network.
> >>If a machine was found to have an open RPC port (we run an open
> >>network), that was enough to get your network access cut off.
> >>
> >>I realize these are political solutions more than technical ones,
> >>but they may be of some use.
> >
> > The trouble with that is that my users are largely untechnical
> > and wouldn't have a clue what RPC is and cutting them off is not
> > an option. Welcome to the world of corporate IT! It ain't a
> > pretty job, but it pays the bills...
>
> been there, done that, the bruises have gone down now . . .
>
> One guy to 250 users is a bad ratio.
>
> It seems like there should be some centralized, ie, rule-based
> controls you can put in place. And you should have some leverage
> to force autoupdates on those client machines.
>
> > I got the impression from some reading on Google Groups that
> > there may be a way to tell the xl driver to use polling. I just
> > don't know how.
>
> Well, this is the right place to ask.

The other thing is interrupt priority, maybe ?

-Jim




-- 
-Jim

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Re: Nachi Worm apparently causes "Live Lock" on 4.7 server

[paul beard]

James C. Durham wrote:
On Friday 29 August 2003 04:23 am, paul wrote:

James C. Durham wrote:

It turned out that we had several Windows boxes in the building that had
been infected with the Nachi worm. This causes some kind of DOS or ping
probe out onto the internet and the local LAN.
Removing the inside interface's ethernet cable caused the ping times on
the outside interface to go back to the normal .4 milliseconds to the
router.
Apparently, the blast of packets coming from the infected boxes managed
to cause a "live lock" condition in the server. I assume it was interrupt
bound servicing the inside interface. The packets were ICMP requests to
various addresses.
I could be way off here, but is there any way to isolate machines
that send a sudden blast of packets, either by destination address
(make a firewall rule that drops those packets) or working out
their MAC addresses and dropping their connectivity? Or scan for
open ports and block unsecured systems from connecting?


What I did was go in the switch room and look for pulsing lights on the switch 
ports and pull the cables. That fixed it, but after much agony.
well, that's a bit draconian, but effective ;-)

My questions is.. what, if any, is a technique for preventing this
condition? I know, fix the windows boxes, but  I can't continually check
the status of the virus software and patch level of the Windows boxes.
There are 250 plus of them and one of me. Users won't install upgrades
even when warned this worm thing was coming. But, i'd like to prevent
loss of service when one of Bill's boxes goes nuts!
Where I work, at the University of Washington, the network staff
were dropping as many as 200 machines *per day* off the network.
If a machine was found to have an open RPC port (we run an open
network), that was enough to get your network access cut off.
I realize these are political solutions more than technical ones,
but they may be of some use.


The trouble with that is that my users are largely untechnical and wouldn't 
have a clue what RPC is and cutting them off is not an option. Welcome to the 
world of corporate IT! It ain't a pretty job, but it pays the bills...
been there, done that, the bruises have gone down now . . .

One guy to 250 users is a bad ratio.

It seems like there should be some centralized, ie, rule-based 
controls you can put in place. And you should have some leverage 
to force autoupdates on those client machines.

I got the impression from some reading on Google Groups that there may be a 
way to tell the xl driver to use polling. I just don't know how.
Well, this is the right place to ask.

--
Paul Beard

whois -h whois.networksolutions.com ha=pb202
Receiving a million dollars tax free will make you feel better than
being flat broke and having a stomach ache.
-- Dolph Sharp, "I'm O.K., You're Not So Hot"
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Re: Nachi Worm apparently causes "Live Lock" on 4.7 server

[paul]

Matthew Emmerton wrote:

They were doing the same thing at the IBM location where I work.  It's
brutal if you are in the middle of something, but it's the only way to keep
the latest breed of MS virii/worms/whatever from spreading.
agreed, but if a small subset of hosts can degrade the network -- 
the OP said three could saturate T1 -- I think it's a fair 
position to take.

--
Paul Beard

whois -h whois.networksolutions.com ha=pb202
"Benson, you are so free of the ravages of intelligence"
-- Time Bandits
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Re: Nachi Worm apparently causes "Live Lock" on 4.7 server

[Matthew Emmerton]

> James C. Durham wrote:
>
> >
> > It turned out that we had several Windows boxes in the building that had
been
> > infected with the Nachi worm. This causes some kind of DOS or ping probe
out
> > onto the internet and the local LAN.
> >
> > Removing the inside interface's ethernet cable caused the ping times on
the
> > outside interface to go back to the normal .4 milliseconds to the
router.
> >
> > Apparently, the blast of packets coming from the infected boxes managed
to
> > cause a "live lock" condition in the server. I assume it was interrupt
bound
> > servicing the inside interface. The packets were ICMP requests to
various
> > addresses.
>
> I could be way off here, but is there any way to isolate machines
> that send a sudden blast of packets, either by destination address
> (make a firewall rule that drops those packets) or working out
> their MAC addresses and dropping their connectivity? Or scan for
> open ports and block unsecured systems from connecting?
> >
> > My questions is.. what, if any, is a technique for preventing this
condition?
> > I know, fix the windows boxes, but  I can't continually check the status
of
> > the virus software and patch level of the Windows boxes. There are 250
plus
> > of them and one of me. Users won't install upgrades even when warned
this
> > worm thing was coming. But, i'd like to prevent loss of service when one
of
> > Bill's boxes goes nuts!
>
> Where I work, at the University of Washington, the network staff
> were dropping as many as 200 machines *per day* off the network.
> If a machine was found to have an open RPC port (we run an open
> network), that was enough to get your network access cut off.
>
> I realize these are political solutions more than technical ones,
> but they may be of some use.

They were doing the same thing at the IBM location where I work.  It's
brutal if you are in the middle of something, but it's the only way to keep
the latest breed of MS virii/worms/whatever from spreading.

--
Matt Emmerton

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Re: Nachi Worm apparently causes "Live Lock" on 4.7 server

[paul]

James C. Durham wrote:

It turned out that we had several Windows boxes in the building that had been 
infected with the Nachi worm. This causes some kind of DOS or ping probe out 
onto the internet and the local LAN.

Removing the inside interface's ethernet cable caused the ping times on the 
outside interface to go back to the normal .4 milliseconds to the router.

Apparently, the blast of packets coming from the infected boxes managed to 
cause a "live lock" condition in the server. I assume it was interrupt bound 
servicing the inside interface. The packets were ICMP requests to various 
addresses.
I could be way off here, but is there any way to isolate machines 
that send a sudden blast of packets, either by destination address 
(make a firewall rule that drops those packets) or working out 
their MAC addresses and dropping their connectivity? Or scan for 
open ports and block unsecured systems from connecting?
My questions is.. what, if any, is a technique for preventing this condition? 
I know, fix the windows boxes, but  I can't continually check the status of 
the virus software and patch level of the Windows boxes. There are 250 plus 
of them and one of me. Users won't install upgrades even when warned this 
worm thing was coming. But, i'd like to prevent loss of service when one of 
Bill's boxes goes nuts!
Where I work, at the University of Washington, the network staff 
were dropping as many as 200 machines *per day* off the network. 
If a machine was found to have an open RPC port (we run an open 
network), that was enough to get your network access cut off.

I realize these are political solutions more than technical ones, 
but they may be of some use.
--
Paul Beard

whois -h whois.networksolutions.com ha=pb202

Satellite Safety Tip #14:
If you see a bright streak in the sky coming at you, duck.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Re: Nachi Worm apparently causes "Live Lock" on 4.7 server

[K Anderson]

James C. Durham wrote:
On 8/21, I noticed that internet connectity through our 4.7 FreeBSD gateway 
NAT box was getting REALLY slow. Checking with our T1 provider, there was 
only 128K of data stream (aprox) flowing out the T1. Ping times to the router 
on the external interface yielded times of up to 3 seconds!

This box is a Dell 2350 server with one 500mhz Pent 3 and 512 mg ram.

Running tcpdump on both the internal and external interfaces showed a very 
small number of ICMP packets flowing on either and virtually no IP.

My first conclusion...wrong..was that I had a bad ethernet card. Pulled 
server/gateway box off line and replaced the card. No difference.

It turned out that we had several Windows boxes in the building that had been 
infected with the Nachi worm. This causes some kind of DOS or ping probe out 
onto the internet and the local LAN.

Removing the inside interface's ethernet cable caused the ping times on the 
outside interface to go back to the normal .4 milliseconds to the router.

Apparently, the blast of packets coming from the infected boxes managed to 
cause a "live lock" condition in the server. I assume it was interrupt bound 
servicing the inside interface. The packets were ICMP requests to various 
addresses.

At one point, I substituted a Dell 2650 with 1 gig interfaces and 2 1800 mg 
Xeons at the gateway addresses and it bound up also. Speed seems not to be 
the answer 8-( .

My questions is.. what, if any, is a technique for preventing this condition? 
I know, fix the windows boxes, but  I can't continually check the status of 
the virus software and patch level of the Windows boxes. There are 250 plus 
of them and one of me. Users won't install upgrades even when warned this 
worm thing was coming. But, i'd like to prevent loss of service when one of 
Bill's boxes goes nuts!

The inside interface is the 'xl' driver on a 3Com 3C905. Can it be run in 
polling mode or given lower interrupt priority?

BTW, it seems to only take about 3 infected windows boxes to bring things to a 
halt.

-Jim

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Forget letting the users patch the systems you already pointed out their 
lack of concerne. Set Windows to automaticly retrive the updates (this 
might depend on the version of the OS) and install them. Next you can 
put personal firewalls (PF) on the systems preconfigured to only the 
applications you want to be able to get out on the internet.

If you don't want to go with the PF then watch for unusual activity over 
the network (let's say that anything after hours that causes things to 
get boggy and coming from a Windows machine) then create a rule that 
firewalls the activity so all doesn't get propagated to the wild. You 
can even have it email you this and then from there tunnel in to the 
network and shut down the machine(s) remotely.

As a side note. As the IT person become more proactive in the 
administration of systems. Those CDW commercials may be silly to watch 
but they are absolutely true.

Oh, a final solution...unless a system needs to have Windows then get 
'em off of it.

HTH

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"