On Wed, Sep 24, 2003 at 07:34:20PM -0700, Octavian Hornoiu wrote: > How would I go about configuring a 3 network card bridge system to support 1-2 > million states simultaneously? How about 3-4 million? Does the kernel require > any tweaking in order to handle that many networking requests and does pf need > tweaking as well to scale this high? I know using ipfilter i have to recompile > ipfilter with additional states but i am not as familiar with the internals of > pf since it's so new.
You don't need to recompile the kernel with specific options, the pf data structures have no fixed compile-time limits. But 3-4 * 10^6 is a rather high number of concurrent state entries, you'll need plenty of RAM to support that. I don't have specific numbers, but that might mean 2-4GB of RAM, based on the conservative estimate that 1 KB of RAM per state entry is safe (might be less, if you exceed several hundred thousand state entries, I haven't tested, lacking a box with that much RAM). With vmstat -m you can watch the pfstatepl pool where the state entries are stored. With "set limit states" in pf.conf you can limit the state entries. When the limit is set too high for the available RAM, and pf exhausts memory to the point where pool memory allocation fails, the system becomes unstable and usually panics reliably. Nobody has reported safe limits for 2 or 4 GB of RAM yet, so I suggest you try and tell us ;) Another question is whether there are really 3-4 million concurrent active connections. Depending on the timeout values, you'll see a certain percentage more concurrent state entries than concurrent active connections (as states are only removed after a timeout). Reducing the timeouts (making them more aggressive) will reduce that percentage. The gain also depends on how long-lifed your average connection is. Let me know if you have problems dealing with that many states with the maximum amount of RAM the system supports. There's some bloat in the state entry structure that could probably reduce the memory requirements by 10-30% which we haven't compressed yet. As for CPU, state tree lookups are O(log n), so 3-4 million states should only increase the load about 30% compared to 100000 entries. So I guess RAM will be the only real issue. Daniel
