Hi, On Sat, Dec 11, 2021 at 06:04:17PM +0100, Marcel Bischoff wrote:
I've stumbled into this issue some time ago as well. The usual remedies of raising states and table-entries did not help.I could resolve this with a combination of increasing the process memory limit and lowering ZFS memory usage. I'm guessing that you do use ZFS, since you have an 8 GB Pi, if not please disregard.
You're right, I am using zfs
FreeBSD limits the memory available per process to 512 MB by default. It appears that large PF tables cause issues with this default. Raising that limit to 2 GB seems to have done the trick. To attempt this, add the "kern.maxdsiz" tunable to /boot/loader.conf and reboot, like so:kern.maxdsiz="2147483648"
Yes, came to this conclusion shortly before you posted this. I've written what I've done further in
https://forums.freebsd.org/threads/cannot-define-table-cannot-allocate-memory-since-upgrade-to-13-0.80822/#post-546253 I found this thread https://forums.freebsd.org/threads/pf-states-limit-reached.67337/informative too. But the idea of increasing it came from https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html
I note that on recent -current (main-n251261) on aarch64 kern.maxdsiz seems to be 1073741824 by default.
But on amd64 on releng/12.2-p7 and on releng/13.0-p5 and stable/13 it defaults to 34359738368 !
On armv6 r366954 12.2 release it's 536870912. I don't know why it's so low on current (arm64.aarch64). It would explain why i've only seen this issue on this arch.
Sometimes PF still tripped up when replacing one large table with another (via pf-badhosts). Usually this happened when the generated list to replace grew to several MB in size. I've read somewhere that PF needs double the memory to replace a table since for an instant it needs to hold both tables in memory, which makes sense to me. You can verify that this is the case by first dropping and then recreating the pf-badhost table with all entries. That usually works, while replacing (what pf-badhosts does) usually does not.
Yes, can confirm it's the replacing where the errors arose.
In those cases I found the ZFS ARC cache to be the culprit. By default this grabs most of the available memory to cache file access and frees it on application request. Again, PF appears to require the memory quicker than ZFS appears to be able to free it. Usually the default ARC settings are fine but in this case you may want to tune them for testing purposes, again in /boot/loader.conf and reboot:vfs.zfs.arc_max="2000M" vfs.zfs.arc_min="500M"
Thank you for this, I'll do this if required. For now, I've set kern.maxdsiz=4294967296 and am monitoring the pfbsdhost's mailbox for error reports. -- J.
signature.asc
Description: PGP signature
