On Fri, 13 Mar 2015, Thor Lancelot Simon wrote:
snip
A cursory review from here looks like sysmon should be deleted.
Yeah, sysmon really is pretty ugly. But it does work (more-or-less)
and we don't have any proposals for an alternative sensor monitoring
sub-system.
On Sat, Mar 14, 2015 at 06:53:51AM +0800, Paul Goyette wrote:
On Fri, 13 Mar 2015, Christos Zoulas wrote:
On Mar 13, 6:32pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| What about the attached diff. It adds a counter of busy items
On Fri, 13 Mar 2015, Christos Zoulas wrote:
On Mar 13, 6:32pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| What about the attached diff. It adds a counter of busy items and
| stops enqueueing more work if an item is still busy
On 12 Mar 2015, at 20:59, Christos Zoulas chris...@zoulas.com wrote:
| | Now we have a deadlock, softlck/0 waits for the mutex and therefore
| | callouts will no longer be processed and ciss holds the mutex and waits
| | for a callout through cv_timedwait.
|
| Thanks for looking into it!
On Mar 13, 1:00pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| This would be simple, changing dev/ic/ciss.c like:
|
| sc-sc_sme-sme_name =3D device_xname(sc-sc_dev);
| sc-sc_sme-sme_cookie =3D sc;
| sc-sc_sme
On Mar 13, 11:19am, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| The mutex involved is the sme_mtx protecting the struct sysmon_envsys, so
| our problem doesn't come from missing POLL.
That's what I thought.
| We already have it. If I
On 13 Mar 2015, at 13:03, Christos Zoulas chris...@zoulas.com wrote:
On Mar 13, 1:00pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| This would be simple, changing dev/ic/ciss.c like:
|
| sc-sc_sme-sme_name =3D device_xname
On 13 Mar 2015, at 12:53, Christos Zoulas chris...@zoulas.com wrote:
On Mar 13, 11:19am, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| The mutex involved is the sme_mtx protecting the struct sysmon_envsys, so
| our problem doesn't come
On Mar 13, 4:12pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| Can't it just try to acquire the lock and if it fails it spams, and
| does not deadlock? Or even better, finds the driver that blocks it,
| and bumps its timeout
On Mar 13, 1:08pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| This was just an idea ... Maybe
|
| ...xs..timeout * sc-maxunits + 10
|
| and set xs timeout to 1 .. 5 seconds?
|
| I don't think it is possible to make it self adjusting
On 28 Feb 2015, at 21:05, Christos Zoulas chris...@zoulas.com wrote:
On Feb 28, 8:26pm, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| On Sat, 28 Feb 2015, Christos Zoulas wrote:
|
| Yes, that's
On Mar 12, 8:28pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| Looks like you made it worse.
|
| tick is constant, for HZ == 100 it is 1 so you now have
|
| etick = tick + tohz - etick = 1 + tohz
On 12 Mar 2015, at 20:00, Christos Zoulas chris...@zoulas.com wrote:
On Mar 12, 12:20pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| Now we have a deadlock, softlck/0 waits for the mutex and therefore
| callouts will no longer
On Mar 12, 12:20pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| Now we have a deadlock, softlck/0 waits for the mutex and therefore
| callouts will no longer be processed and ciss holds the mutex and waits
| for a callout through
On Feb 28, 4:46pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
-- Subject: Re: DoS attack against TCP services
| Anyone holding proc_lock? I had a similar problem with fstrans where
| it was a deadlock with proc_lock preventing timer_intr() to succeed and
| therefore all timers stopped
On 28 Feb 2015, at 16:28, Christos Zoulas chris...@zoulas.com wrote:
On Feb 28, 11:37am, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| On Fri, 13 Feb 2015, Christos Zoulas wrote:
|
| I tried
On Fri, 13 Feb 2015, Christos Zoulas wrote:
I tried adding show callout to crash(8) but it is not useful because the
pointers move too quickly. OTOH, next time this happens you can enter ddb
on your machine and type show callout and see if that sheds any light
to the expired and not fired
On Feb 28, 11:37am, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| On Fri, 13 Feb 2015, Christos Zoulas wrote:
|
| I tried adding show callout to crash(8) but it is not useful because the
| pointers move
On Sat, 28 Feb 2015, J. Hannken-Illjes wrote:
This one looks bad. Which thread holds proc_lock?
Helps this?
https://www.ipv6.uni-leipzig.de/proc_lock.png
Regards
Uwe
On Feb 28, 7:39pm, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| On Sat, 28 Feb 2015, J. Hannken-Illjes wrote:
|
| This one looks bad. Which thread holds proc_lock?
|
|
| Helps this?
|
| https
On 28 Feb 2015, at 18:20, 6b...@6bone.informatik.uni-leipzig.de wrote:
On Sat, 28 Feb 2015, Christos Zoulas wrote:
Good idea. You can use crash, ps and see what each process is holding...
christos
Here the output from crash and ps
gate# crash
Crash version 7.0_BETA, image version
On Sat, 28 Feb 2015, Christos Zoulas wrote:
Yes, that's a good start but we need to find which process that
lwp belongs to.
I'm not sure what the best course of action is. The machine is still
running. Should you try to get the information from the current system or
force a dump and analyze
On Sat, 28 Feb 2015, Christos Zoulas wrote:
Good idea. You can use crash, ps and see what each process is holding...
christos
Here the output from crash and ps
gate# crash
Crash version 7.0_BETA, image version 7.99.5.
WARNING: versions differ, you may not be able to examine this image.
On 28 Feb 2015, at 19:39, 6b...@6bone.informatik.uni-leipzig.de wrote:
On Sat, 28 Feb 2015, J. Hannken-Illjes wrote:
This one looks bad. Which thread holds proc_lock?
Helps this?
https://www.ipv6.uni-leipzig.de/proc_lock.png
Looks unlocked -- what about a backtrace of thread 0.5,
bt
On Wed, 4 Feb 2015, Sverre Froyen wrote:
I'd also look at the open descriptors of the named process (although they
should be closed at this time, since TIME_WAIT means closed on this side,
and waiting for the 4 minutes to expire before killing the connection)...
Also I'd record that
In article pine.neb.4.64.1502101617420@6bone.informatik.uni-leipzig.de,
6b...@6bone.informatik.uni-leipzig.de wrote:
The callout code in kern_timeout.c:
if (delta 0)
cc-cc_ev_late.ev_count++;
At the same time, the problem occurs that expired
On 2015-02-04, at 13:08, 6b...@6bone.informatik.uni-leipzig.de wrote:
I picked out one connection:
fe8678e73990 tcp0 0 139.18.25.33.58935199.254.60.1.53
TMWAIT
But 'fstat -n | grep 73990' shows no result. lsof also shows no socket for
this connection.
I
On 2015-02-04, at 12:32, Christos Zoulas chris...@zoulas.com wrote:
On Feb 4, 7:44pm, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| Now the server has over 5000 TIME_WAIT connections
On Fri, 6 Feb 2015, Robert Elz wrote:
What's more, it seems peculiar to your system, as no-one else seems to
be reporting similar problems. So I'd be investigating how the timers
are working (or are not working) in the kernel - perhaps even try
selecting a different timer.
Just to make
On Feb 7, 12:53pm, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| On Fri, 6 Feb 2015, Robert Elz wrote:
|
| What's more, it seems peculiar to your system, as no-one else seems to
| be reporting similar
chris...@zoulas.com (Christos Zoulas) writes:
On Feb 7, 12:53pm, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| On Fri, 6 Feb 2015, Robert Elz wrote:
|
| What's more, it seems peculiar to your
On Sat, 7 Feb 2015, Greg Troxel wrote:
I don't know; I will take look, but in this case the connections are
initiated by the inflicted system.
And so far we don't have any traces showing packets that look like attacks.
There must be no attack, yes. However, it is described that the attack
On Feb 5, 12:29am, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| dmesg reports in loger intervals:
|
| nd6_storelladdr: something odd happens
|
| I do not know if this is the cause for the TIME_WAIT
On Feb 6, 12:49am, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| On Fri, 6 Feb 2015, Robert Elz wrote:
|
| I assume that time (as seen from user processes) is functioning correctly?
|
| ndp -a shows
On Fri, 6 Feb 2015, Robert Elz wrote:
I assume that time (as seen from user processes) is functioning correctly?
ndp -a shows:
...
2001:638:902:2000:290:f5ff:fe39:3815 00:90:f5:39:38:15 vlan14 23h27m30s S
2001:638:902:2000:565:50de:c658:60cc 90:b1:1c:a6:b5:99 vlan14 expired R
On Fri, 6 Feb 2015, Robert Elz wrote:
What's more, it seems peculiar to your system, as no-one else seems to
be reporting similar problems. So I'd be investigating how the timers
are working (or are not working) in the kernel - perhaps even try
selecting a different timer.
I wonder also
Date:Wed, 4 Feb 2015 23:27:39 +0100 (CET)
From:6b...@6bone.informatik.uni-leipzig.de
Message-ID:
pine.neb.4.64.1502042315490@6bone.informatik.uni-leipzig.de
| I'm afraid I have chosen the wrong subject. After some testing, I found
| that all tcp connections
On Feb 4, 7:44pm, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| Now the server has over 5000 TIME_WAIT connections.
|
| netstat -a -n | grep TIME_WAIT
| tcp0 0 139.18.25.33.59256
for a problem report?
Regards
Uwe
On Wed, 4 Feb 2015, Christos Zoulas wrote:
Date: Wed, 4 Feb 2015 15:40:00 + (UTC)
From: Christos Zoulas chris...@astron.com
To: current-users@netbsd.org
Subject: Re: DoS attack against TCP services
In article
pine.neb.4.64.1502041602460
2015 19:54:59 +0100
From: Johnny Billquist b...@update.uu.se
To: 6b...@6bone.informatik.uni-leipzig.de,
Christos Zoulas chris...@astron.com
Cc: current-users@netbsd.org
Subject: Re: DoS attack against TCP services
Are you *sure* the same connections stay around forever, or might it just
...@astron.com
To: current-users@netbsd.org
Subject: Re: DoS attack against TCP services
In article pine.neb.4.64.1502041602460@6bone.informatik.uni-leipzig.de,
6b...@6bone.informatik.uni-leipzig.de wrote:
Hello,
The problem occurred again. The kernel has over 3,000 connections in
TIME_WAIT
for these
connections failing to close the sockets once the connections terminate.
-thanks
-Brian
On Feb 4, 7:44pm, 6b...@6bone.informatik.uni-leipzig.de wrote:
} Subject: Re: DoS attack against TCP services
} Now the server has over 5000 TIME_WAIT connections.
}
} netstat -a -n | grep TIME_WAIT
} tcp
On Feb 4, 11:27pm, 6b...@6bone.informatik.uni-leipzig.de
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: DoS attack against TCP services
| Hello,
|
| I'm afraid I have chosen the wrong subject. After some testing, I found
| that all tcp connections go in the TIME_WAIT state
6b...@6bone.informatik.uni-leipzig.de writes:
the process is the named (version: bind-9.10.1pl1). The outgoing
connections are normal. stopping the named do not remove the TIME_WAIT
connections.
The TIME_WAIT entries aren't connected to a process anymore. That's
normal behaviour.
--
--
...@serpens.de
To: current-users@netbsd.org
Newsgroups: lists.netbsd.current-users
Subject: Re: DoS attack against TCP services
b...@update.uu.se (Johnny Billquist) writes:
Timeout should not depend on distance, and should actually be (at least)
2*MSS, which would be something in the several minutes
In article pine.neb.4.64.1502041602460@6bone.informatik.uni-leipzig.de,
6b...@6bone.informatik.uni-leipzig.de wrote:
Hello,
The problem occurred again. The kernel has over 3,000 connections in
TIME_WAIT state. The compounds are after an hour wait not disappeared.
There are more and more
6b...@6bone.informatik.uni-leipzig.de writes:
Unfortunately, all TCP connections are now in the TIME_WAIT state.
bash-4.3 # netstat -a -n | grep TIME_WAIT | wc -l
34611
Is there a way to remove it without rebooting the server?
tcpdrop(8)?
It works. But why doesn't drop the kernel it
On Mon, 19 Jan 2015, Michael van Elst wrote:
Date: Mon, 19 Jan 2015 09:24:02 + (UTC)
From: Michael van Elst mlel...@serpens.de
To: current-users@netbsd.org
Newsgroups: lists.netbsd.current-users
Subject: Re: DoS attack against TCP services
6b...@6bone.informatik.uni-leipzig.de writes
Hello,
it was launched a DoS attack against my server. The attacker opened ssh
connections from different servers until all sockets are use.
I have stopped the ssh service and terminates all processes.
Unfortunately, all TCP connections are now in the TIME_WAIT state.
bash-4.3 # netstat -a
49 matches
Mail list logo
