On Thu, Aug 06, 2009 at 08:49:42PM -0700, Darren Reed wrote:

Hi Darren,

sorry for the late answer, but vacation ... So going through the mail 
stack took a while as well.

> Jens Elkner wrote:
...
> >>>So the sun case engineer explained, if ipf can not insert an entry into
> >>>the state table, it just _continues_ evaluating the rules that follow. 
...
> >>Well, what would you have it do?
> >
> >As said, at least notify the user about the problem. Since I'm not
...
> >do here. My guts say, drop and stop processing other rules.
> >
> >Another way could be, to lower the TTL of all entries in steps of N hours,
> >cleanup, and check, whether now the insert succeeds. This implies of course
> >a new threshold to be set by the OS/user and a user notification, that this
> >measure has been taken (documentation about what that means would be
> >probably a nice thing, too) ...
  
> It does trigger this to happen but in a slightly more detailed way.
> There's a high and low watermark for the table and it starts with
> the oldest and works its way forward to try and empty things out.

Not sure, whether you mean the same thing: I would do something like this
assuming default TTL is 120h and TTL.low is the lowest TTL allowed:
        if table.load > 98% && TTL > TTL.low
                TTL -= 1h
                foreach con : table
                        if con.TTL <= 1h
                                con.kill && drop con
                        else
                                con.TTL -= 1h
        else if TTL < 120h && table.load < 80%
                TTL += 1h

I guess, this would be sufficient. Usually we've ~ 100..200 IMAP
connections, and ~ 1-2 SMTP conns/s - when bots aka spammers are active, 
~ 5-10 (very seldom 15) conns/s for about 2-3 hours. Since most of the
delivery attemps get NACKd pretty fast, I would say, it is not that much. 
However, not cleanly shutdowned connections probably pollute the state
table (just a guess) ...

> Buf if there are more new connection attempts than there are old
> things that can be removed, packets will get dropped. I'm not
> sure if the improvements in the cleaning were in S10U4, but they
> are in Nevada and the latest S10 updates.

I would guess not, since before U4 we never had such problems.

> >>It is not appropriate to have ipfilter automatically
> >>grow the tables as you suggest through "automatic
> >>tuning" because then your system becomes vulnerable
> >>to a denial of service attack from remote attackers.
> >
> >Isn't it already vunerable to DoS, when the state table is too small?
> 
> Yes & no. But a different denial of service attack.

OK, but growing to a certain degree only should be ok. E.g. if fr_statemax
reached, start cleanup like described above and add some buckets temp.
Max. avail mem. might be the base for calculation (like ARC for ZFS).
  
> If the table just kept growing, it could consume all kernel memory
> and make the machine unresponse/crash. In this situtation, if the table
> fills up then certain packets will get dropped. If you're running a web
> server or some other kind of service where the security of the TCP
> connection itself isn't as important, maybe stateless filtering is a better
> choice?

Hmmm, problem machines are e-mail, subversion and sometimes web servers ....
 
> Or if there are multiple keep-state rules, you can apply individual
> limits to each. (see below)

Ohhh - an undocumented feature ...
  
> >Actually I've checked all of our production servers (most of them
> >already have fr_statemax=40129) and all show paket lost > 100,
> >usually > 5000 even on very very low traffic machines and who wonders, of
> >course on svn machines as well. So this implies, that's something wrong
> >with ipf or at least with its default settings...
> 
> Over what time period?

Depends, not yet monitored (just writing a tool for it), but I would say
wrt. manual checks with 1-2 days. 

...

> The primary purpose of the firewall is to provide security and part of that
> security comes through being able to audit what's going on. In that case,
> letting through packets that matched a "pass" rule but for which the state
> could not be created is arguably incorrect.

Exactly, and that's why ipfilter should not treat the rule in this case
like "not-existing", but stop, block and log.

Appropos log: IMHO it is pretty bad, that log entries are often useless,
since one can't determine, whether a packet 'p'assed or was 'b'locked (for
the normal human being, that's all, what finally counts). Not sure,
but the fact, that the log entry appears somewhere is a clear indication,
that it has been logged - so IMHO 'L' seems to be redundant.  Wrt. 'n'
I think, if no rule was matched, one will see @-1:-1 , so redundant as well.
And 's' (I guess) indicates blocked, but no where explained/documented, so
it is useless as well ...
  
> >Actually can only rely on the things, which are
> >documented, and you probably know, there is a lot room for improvements:
> >E.g. It is documented, how one can obtain certain ipf statistics, but
> >nowhere, what these stats actually telling you/how to interprete those
> >stats.
> 
> I'm aware of that and as we like to say, this is an open source
> project, so if something really bothers you please feel free to
> submit additions or corrections.

Well, I would do that, but I think, guessing games are not helpful and
waste even more valuable time ... And even so I would like to have a
closer look at the code, I'll probably get not the chance to do that
within the next 12 month :(

> We only have so much bandwidth.

Yepp, like every human ;-)

...
> >>syslog message from where?
...
> >What I know is, that e.g. on linux there is a klogd ...
... 
> Yes, but that is Linux.
> There are different rules for OpenSolaris developers.

Yes, of course. But wondering, where I can find them ;-)

> >>Generating messages from within kernel modules
> >>is generally frowned upon.

But having a system, which goes mad and not knowing why, is even worse.

> >>Whilst we can send messages to/via it, we are not allowed to
> >>rely on them as being the only communication channel with the
> >>systems administrator.
> >
> >OK. But did I say that?
> 
> No, but there are different rules for you than there are for me/us
> as a developer(s) of [Open]Solaris.

So you are actually saying, theory is more important than practical use
or a little bit overdone: if the theory is ok, why to care about endusers/
usability?

> >So all of our production machines (S10u7) even the low traffic ones
> >and fr_statemax=40129 have a problem and 'keep state' needs to be
> >considered harmful :(((
> 
> Just to make sure, you have "flags S" with all of your TCP "keep state" 
> rules?

Yes.

> The limit on how many state-table entries can be controlled per-rule:
> 
> pass in on bge0 proto tcp from any to any port = 22 flags S keep state 
> (limit 20)
> 
> ...will always allow upto 20 and no more than 20 entries to be created
> because of that rule matching. Now if there is another rule, like this:
> 
> pass in on bge0 proto tcp all flags S keep state
> 
> ...then the 21st ssh connection will "fail" matching the first rule 
> (limit is reached)
> but will successfully match the second and potentially then cause a new 
> state
> table entry to be created.
> 
> Similarly, the reverse is true also: if the rule without the limit fails 
> to create
> a state table entry because the table is too full then the second, with the
> limit of 20 allows for a state table entry to be created so long as 
> there are
> no more than 19 such entries already in the table.
  
Would be nice to doc this in ipf(4). 
So if I understand that correctly, limit is like a reservation of table
entries for the given rule. The latter case would help at least to get not
locked out of the system, but I think does not really help.
  
> >>>BTW: Why shows 'ipfstat -t' so many entries with negative ttls. It
> >>>    appears, that if the min? value of -59:-59 is reached, ttl gets
> >>>    reset to 0:00 and restarts decrementing it ... - strange
> >>>      
> >>That's a known bug... fixed in the current opensolaris
> >>source code tree and will be fixed in the next release.
> >>A fix is also being considered for Solaris 10.
> 
> They get removed from the state table but not the list of state entries.
> Doing "ipf -Fs -FS" will clean it all out :-D

Still a little bit confused: Is this just another aka stat table?
If not, isn't this perhaps the root cause of fr_statemax reached?

Regards and thanx for your time,
jel.
-- 
Otto-von-Guericke University     http://www.cs.uni-magdeburg.de/
Department of Computer Science   Geb. 29 R 027, Universitaetsplatz 2
39106 Magdeburg, Germany         Tel: +49 391 67 12768
_______________________________________________
networking-discuss mailing list
networking-discuss@opensolaris.org

Reply via email to