[pfSense] Suddenly getting pfi_table_update errors [work-around]
I think this issue has been solved:
- issue was errors similar to:
---
[ There were error(s) loading the rules: pfctl: DIOCADDRULE: Invalid argument -
The line in question reads [0]: ]
---
and/or an error indicating that it can't allocate memory (but there's over 50%
of the memory reported as being available via the dashboard widget) -- this is
accompanied by repeating errors like:
---
pfi_table_update cannot set new addresses into table :
---
and regular "I'm completely blocked" delays on the web configurator.
- Chris Buechler suggested trying 64-bit, but my machines are not 64-bit capable
- I can't yet test on pfSense 2.2 because it has a broken cas driver (time to
replace my ol' workhorse Sun 4-port/Gig-e boards)
- when I was writing back to Chris, off-list, I realized that my "System ->
Advanced -> Firewall/Nat" settings for the max firewall tables/entries
(250/250) were more than I needed so I reduced them to 150/15 (actual
items about 100/880K):
+ that caused a more complete failure, causing the 2 "loading firewall rules"
during the boot-up sequence to "show dots" but no "done" and, upon boot-up
completion, there were errors like:
---
02-17-15 23:51:33 [ There were error(s) loading the rules:
/tmp/rules.debug:180: cannot define table NotLAN2lans: Cannot allocate memory -
The line in question reads [180]: table { 172.24.16.0/24 172.24.18.0/24
172.24.32.0/24 172.24.64.0/29 172.24.48.0/29 172.24.48.32/29 172.24.48.64/29
172.24.48.96/29 172.24.48.128/29 } ]
---
and none of the tables were created (and I suspect no rules were actually
loaded, but didn't check).
- given that reducing the max tables/entries values made things worse, it
seemed obvious that I should try increasing them
... and setting max tables/entries to 300/350 appears to have worked around
the issue.
My guess is that there's some static memory allocation that's calculated based
upon the max tables/entries sizes (and other criteria) but that my mix of
tables, with 10 of them having about 70K entries each, isn't anticipated by
that memory-allocation calculation. A static type of allocation wouldn't
surprise me as I'd assume the rules are very performance-critical.
I've subsequently been able to complete the set of alias/rule changes I was
making when things broke so, until it fails again, I'll consider this to be an
OK work-around and hope this write-up will save someone else some
troubleshooting.
Warning: Thus far, I also have found that:
- one port alias was changed to be a duplicate of the port alias just before it
in the list and that new name was propagated where it was used within other
aliases and rules
- one of the URL alias was missing, entirely (and pfSense nicely notified me
about the affected rules)
... so it should be expected that running out of memory for rules/tables is
likely to cause some corruption in the configuration if you don't revert to a
previous config and cause additional memory to be allocated (i.e., don't save
any config changes after experiencing this kind of memory error).
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Suddenly getting pfi_table_update errors
On Tue, Feb 17, 2015 at 10:22 PM, Bryan D. wrote: > I have a relatively low-traffic pfSense 2.1.5 i386 setup on a system with 1.5 > GB of memory that always shows <50% used. > > This setup has normally been reliable but, since upgrading to 2.1.5, today is > the 4th time I've run into a problem after making changes to some aliases. > For some reason that I've been unable to see much pattern to, pfSense will > suddenly report a rash of errors similar to: > --- > [ There were error(s) loading the rules: pfctl: DIOCADDRULE: Invalid argument > - The line in question reads [0]: ] > --- > and/or an error indicating that it can't allocate memory (but there's over > 50% reported as being available). > > > When this happens, the following kind of error will occur during the reboot > while first configuring the firewall ... > --- > pfi_table_update cannot set new addresses into table : > --- > where "" varies, even with the same config being rebooted, and seems to > be either an interface name or "self". The error continues to recur with a > considerable "blocking" pause (up to 10's of seconds) each time it > (apparently) attempts a reload. > It sounds like something in 32 bit isn't happy with very large table sizes. Can't say we've tried large tables on 32 bit, nor do I know of others who have offhand. Where there is a need for large table sizes, you're almost always running 64 bit hardware and the 64 bit version. Is that not a 64 bit CPU? If it is, reinstalling with 64 bit and restoring your backup should be a quick, proven solution. If you wouldn't mind sharing your aliases, email that portion of your config to me off-list. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
[pfSense] Suddenly getting pfi_table_update errors
I have a relatively low-traffic pfSense 2.1.5 i386 setup on a system with 1.5
GB of memory that always shows <50% used.
This setup has normally been reliable but, since upgrading to 2.1.5, today is
the 4th time I've run into a problem after making changes to some aliases. For
some reason that I've been unable to see much pattern to, pfSense will suddenly
report a rash of errors similar to:
---
[ There were error(s) loading the rules: pfctl: DIOCADDRULE: Invalid argument -
The line in question reads [0]: ]
---
and/or an error indicating that it can't allocate memory (but there's over 50%
reported as being available).
When this happens, the following kind of error will occur during the reboot
while first configuring the firewall ...
---
pfi_table_update cannot set new addresses into table :
---
where "" varies, even with the same config being rebooted, and seems to
be either an interface name or "self". The error continues to recur with a
considerable "blocking" pause (up to 10's of seconds) each time it (apparently)
attempts a reload.
I've handled this issue by restoring the most recently saved config.xml (I save
these _very_ often, now!) and it's been "good to go" .. after which I can
remake the changes and all has been good.
However, today that strategy didn't work. After restoring the previously saved
config.xml. which had been running without issues for about a day, the
"pfi_table_update" problems remained after rebooting.
Thinking it might be a disk-corruption and/or hardware issue, I built another
system (with similar resources) and tested it. The same config fails in an
equivalent way.
QUESTION: Can anyone shed some light on how I might troubleshoot this issue?
QUESTION: Does anyone know what's getting loaded when the message
---
There were error(s) loading the rules: pfctl: DIOCADDRULE: Invalid argument -
The line in question reads [0]
---
is being issued? ... if I could see the rule that's giving the problem, maybe
that'd lead somewhere useful.
Other things I've done, without result ...
Of course I asked Mr. Google and searched the pfSense bug tracker for
pfi_table_update, all without results.
I scanned the disk for an operation called pfi_table_update
(find / -type f -exec fgrep -l pfi_table_update {} \;)
but came up empty-handed so I assume this is not a php/pfSense routine.
My first thought when it occurred was that the config.xml file had become
corrupted, but I've never found any evidence of that. I've always compared the
failed config to the successfully reverted config and found no clues (lately,
since I save configs so often, there's only been 2 or 3 changes). The only
thing that's been consistent is that the problem always "pops up" (literally!)
after editing aliases and the rules are being reloaded.
I'm always careful to change aliases in a way that works from the bottom of the
dependencies "up" (when applicable) and, though I do have aliases that include
other aliases, I doubt there's anything unusual in either structure or number
of aliases I have configured (84 host/network aliases, 67 port aliases and 63
URL aliases).
The only thing (related to the aliases) that may be unusual is that I have
about 10 large URL tables (70K entries, each) and have things configured for
250 tables (currently <100) and 2,500,000 table entries (currently about 680K
entries). It's the tables that consume memory, not states, in our case.
Any ideas?#;-)
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold
