On Thu, Aug 06, 2009 at 08:49:42PM -0700, Darren Reed wrote: Hi Darren,
sorry for the late answer, but vacation ... So going through the mail stack took a while as well. > Jens Elkner wrote: ... > >>>So the sun case engineer explained, if ipf can not insert an entry into > >>>the state table, it just _continues_ evaluating the rules that follow. ... > >>Well, what would you have it do? > > > >As said, at least notify the user about the problem. Since I'm not ... > >do here. My guts say, drop and stop processing other rules. > > > >Another way could be, to lower the TTL of all entries in steps of N hours, > >cleanup, and check, whether now the insert succeeds. This implies of course > >a new threshold to be set by the OS/user and a user notification, that this > >measure has been taken (documentation about what that means would be > >probably a nice thing, too) ... > It does trigger this to happen but in a slightly more detailed way. > There's a high and low watermark for the table and it starts with > the oldest and works its way forward to try and empty things out. Not sure, whether you mean the same thing: I would do something like this assuming default TTL is 120h and TTL.low is the lowest TTL allowed: if table.load > 98% && TTL > TTL.low TTL -= 1h foreach con : table if con.TTL <= 1h con.kill && drop con else con.TTL -= 1h else if TTL < 120h && table.load < 80% TTL += 1h I guess, this would be sufficient. Usually we've ~ 100..200 IMAP connections, and ~ 1-2 SMTP conns/s - when bots aka spammers are active, ~ 5-10 (very seldom 15) conns/s for about 2-3 hours. Since most of the delivery attemps get NACKd pretty fast, I would say, it is not that much. However, not cleanly shutdowned connections probably pollute the state table (just a guess) ... > Buf if there are more new connection attempts than there are old > things that can be removed, packets will get dropped. I'm not > sure if the improvements in the cleaning were in S10U4, but they > are in Nevada and the latest S10 updates. I would guess not, since before U4 we never had such problems. > >>It is not appropriate to have ipfilter automatically > >>grow the tables as you suggest through "automatic > >>tuning" because then your system becomes vulnerable > >>to a denial of service attack from remote attackers. > > > >Isn't it already vunerable to DoS, when the state table is too small? > > Yes & no. But a different denial of service attack. OK, but growing to a certain degree only should be ok. E.g. if fr_statemax reached, start cleanup like described above and add some buckets temp. Max. avail mem. might be the base for calculation (like ARC for ZFS). > If the table just kept growing, it could consume all kernel memory > and make the machine unresponse/crash. In this situtation, if the table > fills up then certain packets will get dropped. If you're running a web > server or some other kind of service where the security of the TCP > connection itself isn't as important, maybe stateless filtering is a better > choice? Hmmm, problem machines are e-mail, subversion and sometimes web servers .... > Or if there are multiple keep-state rules, you can apply individual > limits to each. (see below) Ohhh - an undocumented feature ... > >Actually I've checked all of our production servers (most of them > >already have fr_statemax=40129) and all show paket lost > 100, > >usually > 5000 even on very very low traffic machines and who wonders, of > >course on svn machines as well. So this implies, that's something wrong > >with ipf or at least with its default settings... > > Over what time period? Depends, not yet monitored (just writing a tool for it), but I would say wrt. manual checks with 1-2 days. ... > The primary purpose of the firewall is to provide security and part of that > security comes through being able to audit what's going on. In that case, > letting through packets that matched a "pass" rule but for which the state > could not be created is arguably incorrect. Exactly, and that's why ipfilter should not treat the rule in this case like "not-existing", but stop, block and log. Appropos log: IMHO it is pretty bad, that log entries are often useless, since one can't determine, whether a packet 'p'assed or was 'b'locked (for the normal human being, that's all, what finally counts). Not sure, but the fact, that the log entry appears somewhere is a clear indication, that it has been logged - so IMHO 'L' seems to be redundant. Wrt. 'n' I think, if no rule was matched, one will see @-1:-1 , so redundant as well. And 's' (I guess) indicates blocked, but no where explained/documented, so it is useless as well ... > >Actually can only rely on the things, which are > >documented, and you probably know, there is a lot room for improvements: > >E.g. It is documented, how one can obtain certain ipf statistics, but > >nowhere, what these stats actually telling you/how to interprete those > >stats. > > I'm aware of that and as we like to say, this is an open source > project, so if something really bothers you please feel free to > submit additions or corrections. Well, I would do that, but I think, guessing games are not helpful and waste even more valuable time ... And even so I would like to have a closer look at the code, I'll probably get not the chance to do that within the next 12 month :( > We only have so much bandwidth. Yepp, like every human ;-) ... > >>syslog message from where? ... > >What I know is, that e.g. on linux there is a klogd ... ... > Yes, but that is Linux. > There are different rules for OpenSolaris developers. Yes, of course. But wondering, where I can find them ;-) > >>Generating messages from within kernel modules > >>is generally frowned upon. But having a system, which goes mad and not knowing why, is even worse. > >>Whilst we can send messages to/via it, we are not allowed to > >>rely on them as being the only communication channel with the > >>systems administrator. > > > >OK. But did I say that? > > No, but there are different rules for you than there are for me/us > as a developer(s) of [Open]Solaris. So you are actually saying, theory is more important than practical use or a little bit overdone: if the theory is ok, why to care about endusers/ usability? > >So all of our production machines (S10u7) even the low traffic ones > >and fr_statemax=40129 have a problem and 'keep state' needs to be > >considered harmful :((( > > Just to make sure, you have "flags S" with all of your TCP "keep state" > rules? Yes. > The limit on how many state-table entries can be controlled per-rule: > > pass in on bge0 proto tcp from any to any port = 22 flags S keep state > (limit 20) > > ...will always allow upto 20 and no more than 20 entries to be created > because of that rule matching. Now if there is another rule, like this: > > pass in on bge0 proto tcp all flags S keep state > > ...then the 21st ssh connection will "fail" matching the first rule > (limit is reached) > but will successfully match the second and potentially then cause a new > state > table entry to be created. > > Similarly, the reverse is true also: if the rule without the limit fails > to create > a state table entry because the table is too full then the second, with the > limit of 20 allows for a state table entry to be created so long as > there are > no more than 19 such entries already in the table. Would be nice to doc this in ipf(4). So if I understand that correctly, limit is like a reservation of table entries for the given rule. The latter case would help at least to get not locked out of the system, but I think does not really help. > >>>BTW: Why shows 'ipfstat -t' so many entries with negative ttls. It > >>> appears, that if the min? value of -59:-59 is reached, ttl gets > >>> reset to 0:00 and restarts decrementing it ... - strange > >>> > >>That's a known bug... fixed in the current opensolaris > >>source code tree and will be fixed in the next release. > >>A fix is also being considered for Solaris 10. > > They get removed from the state table but not the list of state entries. > Doing "ipf -Fs -FS" will clean it all out :-D Still a little bit confused: Is this just another aka stat table? If not, isn't this perhaps the root cause of fr_statemax reached? Regards and thanx for your time, jel. -- Otto-von-Guericke University http://www.cs.uni-magdeburg.de/ Department of Computer Science Geb. 29 R 027, Universitaetsplatz 2 39106 Magdeburg, Germany Tel: +49 391 67 12768 _______________________________________________ networking-discuss mailing list networking-discuss@opensolaris.org