[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

Kristof Provost  changed:

   What|Removed |Added

   Assignee|p...@freebsd.org  |freebsd-...@freebsd.org
   See Also||https://bugs.freebsd.org/bu
   ||gzilla/show_bug.cgi?id=2296
   ||44

--- Comment #53 from Kristof Provost  ---
This is an aarch64 (or pine64) problem, not a pf issue.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

Mark Linimon  changed:

   What|Removed |Added

   Assignee|b...@freebsd.org|p...@freebsd.org

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

Kristof Provost  changed:

   What|Removed |Added

   Assignee|freebsd-pf@FreeBSD.org  |freebsd-b...@freebsd.org

--- Comment #50 from Kristof Provost  ---
That pretty much confirms this isn't actually a pf bug, but a time keeping
issue. I'm afraid I can't help with this one.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #49 from noah.bergba...@tum.de ---
At least in my particular case, I eventually tracked this down to a tunable
from loader.conf: kern.timecounter.smp_tsc_adjust=1

Since I removed that, this issue hasn't happened once so I believe it might
have been the reason.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #48 from h...@restart.be ---
(In reply to Kristof Provost from comment #47)

No. I am running on a Pine64+ 2GB

See http://docs.freebsd.org/cgi/mid.cgi?fdf7ea24-6ada-dbb1-146a-0bf6fde11d29

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #47 from Kristof Provost  ---
(In reply to hlh from comment #46)
Are you also using VMWare esxi?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

freebsd.po...@webstyle.ch changed:

   What|Removed |Added

 CC||freebsd.po...@webstyle.ch

--- Comment #45 from freebsd.po...@webstyle.ch ---
We had the exactly same problem after upgrading from 10.3 to 11.1

Since this was the only system with an older vmware esxi, we upgraded the esxi
host to a newer version. Surprise, the problem went away... :-)

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #44 from Kristof Provost  ---
(In reply to hlh from comment #43)
No, that shouldn't matter, unless the clock is completely wrong (i.e. not
running, or running orders of magnitude slower than expected, or running
backwards, ...)

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #43 from h...@restart.be ---
(In reply to Kristof Provost from comment #42)

I upgrade the PINE64 to CURRENT 324563 and now ntpd can't keep the clock for a
long time - after one hour or 2, the clock run too fast. I think that this
clock drift is correlated with the pf problem.
Does pf is sensitive to a clock running too fast and beeing reset backward by
ntpd?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #42 from Kristof Provost  ---
(In reply to hlh from comment #40)
> dtrace: processing aborted: Abort due to systemic unresponsiveness
You'd have to talk to a dtrace specialist about that. I'm not sure what causes
that.

I still don't see this issue on my current boxes, so I don't know what I can do
right now.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #41 from Kristof Provost  ---
(In reply to hlh from comment #38)
> Why ARC at 233% and 1727Mb wired?
The ARC now keeps compressed data, so you can fit more in memory. I suspect
that's a result of that.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #40 from h...@restart.be ---
(In reply to hlh from comment #33)

When I run the dtrace in batch, it end freqently with:

dtrace: script './pf.dtrace2' matched 2 probes
dtrace: buffer size lowered to 2m
dtrace: processing aborted: Abort due to systemic unresponsiveness

and so have no valuable trace when the problem arise.

PS I'm now running 

FreeBSD norquay.restart.bel 12.0-CURRENT FreeBSD 12.0-CURRENT #0
r322941:324563M: Tue Oct 24 16:18:12 CEST 2017
r...@norquay.restart.bel:/usr/obj/usr/src/sys/NORQUAY  arm64

And the problem crop up more freqently.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #39 from h...@restart.be ---
(In reply to Kristof Provost from comment #37)
I'm really sorry because in my previous post I display the tail of
dtrace2.log-after instead of dtrace2.log-before. I have deleted those files;
Shame on me!
Let's wait the next problem...

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #38 from h...@restart.be ---
(In reply to Kristof Provost from comment #37)

The only thing that seems strange to me is the memory used on this system:

top show:

last pid: 37231;  load averages:  0.19,  0.20,  0.16   
up 3+17:01:05  08:04:01
73 processes:  1 running, 71 sleeping, 1 zombie
CPU:  0.1% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.7% idle
Mem: 69M Active, 76M Inact, 57M Laundry, 1727M Wired, 35M Free
ARC: 597M Total, 70M MFU, 195M MRU, 1868K Anon, 4274K Header, 326M Other
 55M Compressed, 212M Uncompressed, 3.84:1 Ratio
Swap: 4096M Total, 356M Used, 3740M Free, 8% Inuse

zfs-stats -M:


ZFS Subsystem ReportMon Oct 23 08:04:34 2017


System Memory:

3.53%   69.41   MiB Active, 3.87%   76.07   MiB Inact
87.95%  1.69GiB Wired,  0.00%   0 Cache
1.72%   33.71   MiB Free,   2.93%   57.48   MiB Gap

Real Installed: 2.00GiB
Real Available: 98.34%  1.97GiB
Real Managed:   97.50%  1.92GiB

Logical Total:  2.00GiB
Logical Used:   94.64%  1.89GiB
Logical Free:   5.36%   109.78  MiB

Kernel Memory:  147.94  MiB
Data:   86.01%  127.24  MiB
Text:   13.99%  20.70   MiB

Kernel Memory Map:  512.00  MiB
Size:   53.68%  274.86  MiB
Free:   46.32%  237.14  MiB



zfs-stats -A


ZFS Subsystem ReportMon Oct 23 08:05:18 2017


ARC Summary: (HEALTHY)
Memory Throttle Count:  0

ARC Misc:
Deleted:4.39m
Recycle Misses: 0
Mutex Misses:   24.67m
Evict Skips:4.15b

ARC Size:   233.77% 598.44  MiB
Target Size: (Adaptive) 100.00% 256.00  MiB
Min Size (Hard Limit):  12.50%  32.00   MiB
Max Size (High Water):  8:1 256.00  MiB

ARC Size Breakdown:
Recently Used Cache Size:   22.91%  137.09  MiB
Frequently Used Cache Size: 77.09%  461.35  MiB

ARC Hash Breakdown:
Elements Max:   41.80k
Elements Current:   38.41%  16.06k
Collisions: 448.17k
Chain Max:  4
Chains: 471



Why ARC at 233% and 1727Mb wired?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #37 from Kristof Provost  ---
I do not understand this at all. There is no reason for the purge thread to get
stuck. There's also no reason for it apparently having run 7 times between
freezing and when you restart the dtrace. (That's based on the purge_idx value
logged by the expired_states probe. 23544 - 21255 == 327 * 7.

(Please just don't stop it at all while you're performing the workaround.
There's no reason for it, and stopping it runs the risk of missing interesting
information).

It still looks like we just don't wake up from the sx_sleep(pf_purge_thread,
_end_lock, ...) call, but that makes no sense. The only way for that to
happen is if someone would be holding the pf_end_lock, but that lock is only
taken when the pf module is unloaded (and by the purge thread, obviously).

Do you see other strange behaviour on the system? If there's something wrong
with the sleep/lock infrastructure I'd expect to see other strange things.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #36 from h...@restart.be ---
I notice that when I do:

echo "set timeout interval 10" | pfctl -mf -

while pf is running without problem, then no more state are created.
I must run `nohup service pf restart`to restore a functional system.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #35 from h...@restart.be ---
(In reply to hlh from comment #33)

The problem crops up again.

Here is the end of the output of your dtrace script when the states are no more
expired:

[root@norquay log]# tail pf.dtrace2.log-after
  3   9078  none:expired_states i 19947 maxentry 327 0
  3   9077thread:wakeup 
  3   9078  none:expired_states i 20274 maxentry 327 0
  0   9077thread:wakeup 
  0   9078  none:expired_states i 20601 maxentry 327 0
  0   9077thread:wakeup 
  0   9078  none:expired_states i 20928 maxentry 327 0
  0   9077thread:wakeup 
  0   9078  none:expired_states i 21255 maxentry 327 0

The dtrace script give no more output.

Then I stop the dtrace and save the previous trace.

I restart a new dtrace which output nothing until I run:

echo "set timeout interval 5" | pfctl -mf -
echo "set timeout interval 5" | pfctl -mf -
nohup service pf restart

The first lines of the dtrace are:

[root@norquay log]# head pf.dtrace2.log-after 
CPU IDFUNCTION:NAME
  0   9077thread:wakeup 
  0   9078  none:expired_states i 23544 maxentry 655 0
  1   9077thread:wakeup 
  1   9078  none:expired_states i 24199 maxentry 655 0
  1   9077thread:wakeup 
  1   9078  none:expired_states i 24854 maxentry 655 0
  1   9077thread:wakeup 
  1   9078  none:expired_states i 25509 maxentry 655 0
  1   9077thread:wakeup

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #34 from h...@restart.be ---
(In reply to Kristof Provost from comment #32)

I check the troubling number of return:

[root@norquay log]# grep entry pf.dtrace1.log-before |wc -l
 2026647
[root@norquay log]# grep return pf.dtrace1.log-before |wc -l
 4021723
[root@norquay log]# bc
2026647*2
4053294
4053294-4021723
31571

Really strange!

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #33 from h...@restart.be ---
(In reply to Kristof Provost from comment #32)

Here is the dtrace script used:

#!/usr/sbin/dtrace -s

fbt:kernel:pf_purge_expired_states:entry
{

}

fbt:kernel:pf_purge_expired_states:return
{

}

fbt:kernel:pf_purge_expired_fragments:entry
{

}

fbt:kernel:pf_purge_thread:entry
{

}

I'm doing the changes you ask.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #32 from Kristof Provost  ---
(In reply to hlh from comment #31)
It's a little odd that you're seeing double pf_purge_expired_states:return
entries.
Any chance you've got two such probes in your dtrace script?

Anyway, let's stick a couple of static probes in and see what's going on:
diff --git a/sys/netpfil/pf/pf.c b/sys/netpfil/pf/pf.c
index 8613a161f0a..f8244a6ef6e 100644
--- a/sys/netpfil/pf/pf.c
+++ b/sys/netpfil/pf/pf.c
@@ -55,6 +55,7 @@ __FBSDID("$FreeBSD$");
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -105,6 +106,14 @@ __FBSDID("$FreeBSD$");

 #defineDPFPRINTF(n, x) if (V_pf_status.debug >= (n)) printf x

+/* DTrace static probes */
+SDT_PROVIDER_DEFINE(pf);
+
+SDT_PROBE_DEFINE(pf, purge, thread, wakeup);
+SDT_PROBE_DEFINE2(pf, purge, , expired_states,
+   "unsigned int",
+   "int");
+
 /*
  * Global variables
  */
@@ -1434,6 +1443,7 @@ pf_purge_thread(void *unused __unused)
sx_xlock(_end_lock);
while (pf_end_threads == 0) {
sx_sleep(pf_purge_thread, _end_lock, 0, "pftm", hz / 10);
+   SDT_PROBE0(pf, purge, thread, wakeup);

VNET_LIST_RLOCK();
VNET_FOREACH(vnet_iter) {
@@ -1680,6 +1690,8 @@ pf_purge_expired_states(u_int i, int maxcheck)

V_pf_status.states = uma_zone_get_cur(V_pf_state_z);

+   SDT_PROBE2(pf, purge, , expired_states, i, maxcheck);
+
/*
 * Go through hash and unlink states that expire now.
 */

You can trace those with:
#!/usr/sbin/dtrace -s

pf:purge:thread:wakeup
{

}

pf:purge::expired_states
{
printf("i %d maxentry %d %d", arg0, arg1, arg2);
}

Hopefully we'll get a clue as to what's going on with this.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #31 from h...@restart.be ---
(In reply to Kristof Provost from comment #27)

When the problem occurs, no more line in the dtrace log.
Here are the last 20 lines:

[root@norquay ~]# tail -n 20 /var/log/pf.dtrace.log-before 
  0   2257pf_purge_expired_states:entry 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 
  0   2257pf_purge_expired_states:entry 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 

then I run:
[root@norquay ~]# echo "set timeout interval 5" | pfctl -mf -

And the dtrace log resume; the first 20 lines are:

[root@norquay ~]# head -n 20 /var/log/pf.dtrace.log
CPU IDFUNCTION:NAME
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  0   2257pf_purge_expired_states:entry 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 
  0   2257pf_purge_expired_states:entry 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 
  0   2257pf_purge_expired_states:entry 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 
  0   2257pf_purge_expired_states:entry 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 
  0   2257pf_purge_expired_states:entry 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 
  0   2258   pf_purge_expired_states:return 

PS I have the 2 log files and we can go back more then 20 lines.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #30 from Kristof Provost  ---
(In reply to noah.bergbauer from comment #29)
Having checked the code, yes, I believe you're correct about that.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #29 from noah.bergba...@tum.de ---
>>I don't understand why your 'pfctl -si' would then only show 31 states.
>I think that the `pfctl -si` shows the counters just when the problem start.

It's the same for me. As far as I remember, this behavior is easily explained
by the fact that these counters are updated regularly by the purge thread, so
... yeah. Frozen purge thread means frozen counters.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #28 from h...@restart.be ---
I think that the `pfctl -si` shows the counters just when the problem start.

I'm running dtrace from now on.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #27 from Kristof Provost  ---
(In reply to hlh from comment #26)
I don't understand why your 'pfctl -si' would then only show 31 states.

Either way, it would be interesting to run the dtrace script constantly (saving
the output), and see what's in that log when the problem occurs.
I'd expect it to stop showing updates when you end up in this state, and
restart when you trigger the workaround. The interesting bit would be the last
line(s) before it blocks, and the first line(s) when it resumes.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #26 from h...@restart.be ---
The first time I detected this problem was when a computer was not allow a
connection to the internet. I check the gateway (the pine64 running CURRENT)
and find the 'PF states limit reached' in /var/log/messages. Then I run pftop
and see that there was a huge number of states.

Reboot the gateway solved the problem. I dig further and find the workaround.

I add
set limit { states 3, src-nodes 2, frags 2 }
to /etc/pf.conf.

Then I regularly check with pftop. For more than one week, no problem. But I
continue to check and it occurs again.

I have to check only from time to time because even when the problem arise, the
limit of 3 is large enough to allow for new connections to be established
for some time...

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #25 from Kristof Provost  ---
(In reply to hlh from comment #24)
Wait? You don't see 'PF states limit reached'? What are you seeing, exactly,
and how do you detect it?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #24 from h...@restart.be ---
I run the pfctl -si after I detected the problem (with pftop) and before
attempting any workaround.

Here is the typical top:

last pid: 38563;  load averages:  0.15,  0.28,  0.25   
up 7+14:37:46  08:20:09
67 processes:  1 running, 64 sleeping, 2 zombie
CPU:  0.0% user,  0.0% nice,  0.2% system,  0.1% interrupt, 99.7% idle
Mem: 49M Active, 84M Inact, 76M Laundry, 1728M Wired, 27M Free
ARC: 571M Total, 63M MFU, 358M MRU, 4934K Anon, 6512K Header, 138M Other
 315M Compressed, 520M Uncompressed, 1.65:1 Ratio
Swap: 4096M Total, 285M Used, 3811M Free, 6% Inuse

I don't see the 'PF states limit reached'.

I run the dtrace when I detected the problem. The dtrace show nothing and start
showing the trace after I run the first workaround.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #23 from Kristof Provost  ---
(In reply to hlh from comment #22)
Did you take that '# pfctl -si' before attempting any of the workarounds? It
shows you've got 31 states, I don't see how you'd hit the state limit that way.

It might be interesting to keep dtrace running while you run the workaround. 
Everything I see so far suggests that we're stuck in the rw_sleep() (which was
changed to an sx_sleep() in CURRENT), but that makes no sense. I don't see how
that could happen, or how the workaround could help there.

Can you confirm you're not running low on memory? That might conceivably also
trigger the 'PF states limit reached' warning (although I'd expect to see many
other warnings too), but I'm not sure how that would look like a frozen purge
thread.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #22 from h...@restart.be ---
The problem crop up:

[root@norquay ~]# pfctl -si
Status: Enabled for 1 days 08:09:42   Debug: Urgent

Interface Stats for ng0   IPv4 IPv6
  Bytes In  33556376980
  Bytes Out  2365865540
  Packets In
Passed 25875320
Blocked   32900
  Packets Out
Passed 23953200
Blocked1090

State Table  Total Rate
  current entries   31   
  searches10992548   94.9/s
  inserts775850.7/s
  removals   770520.7/s
Counters
  match  868050.7/s
  bad-offset 00.0/s
  fragment   00.0/s
  short  00.0/s
  normalize  00.0/s
  memory 00.0/s
  bad-timestamp  00.0/s
  congestion 00.0/s
  ip-option  00.0/s
  proto-cksum00.0/s
  state-mismatch 90.0/s
  state-insert   20.0/s
  state-limit00.0/s
  src-limit  80.0/s
  synproxy 1050.0/s
  map-failed 00.0/s

[root@norquay ~]# pfctl -ss|wc -l
 533

[root@norquay ~]# procstat -kk 7
  PIDTID COMMTDNAME  KSTACK 
7 100084 pf purge-   mi_switch+0x118
sleepq_timedwait+0x40 _sleep+0x268 pf_purge_thread+0xec fork_exit+0x94 

[root@norquay dtrace]# ./pf.dtrace 
dtrace: script './pf.dtrace' matched 4 probes
dtrace: buffer size lowered to 2m

after:

[root@norquay ~]# echo "set timeout interval 5" | pfctl -mf -

CPU IDFUNCTION:NAME
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  1   2257pf_purge_expired_states:entry 
  1   2258   pf_purge_expired_states:return 
  1   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2257pf_purge_expired_states:entry 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2258   pf_purge_expired_states:return 
  3   2257

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #21 from h...@restart.be ---
Here is my pf.conf:

#   $FreeBSD: src/etc/pf.conf,v 1.3 2006/01/27 17:16:20 mlaier Exp $
#   $OpenBSD: pf.conf,v 1.21 2003/09/02 20:38:44 david Exp $
#
# See pf.conf(5) and /usr/share/examples/pf for syntax and examples.
# Required order: options, normalization, queueing, translation, filtering.
# Macros and tables may be defined and used anywhere.
# Note that translation rules are first match while filter rules are last
match.

# Macros: define common values, so they can be referenced and changed easily.
#ext_if="ext0"  # replace with actual external interface name i.e., dc0
#int_if="int0"  # replace with actual internal interface name i.e., dc1
#internal_net="10.1.1.1/8"
#external_addr="192.168.1.1"

#--- RestartSoft ---
int_if="awg0"
int_net="192.168.24.0/24, 2001:41d0:8:bdbe:1::/80"
vdsl_if="ng0"
tunnel_if="gif0"
virtualbox_net="192.168.22.0/24, 2001:41d0:8:bdbe:2::/80"
tignes="5.135.182.190"
ftp_passive="3:31000"
smtp1="24"
http1="8080"
squid="3128"
tcp_services="{" "ftp" $ftp_passive $smtp1 $squid "nntp" "http" $http1 "https"
"}"
bittorrent="6881:6889"
donkey_tcp="{ 4662 6346 6347 6667 6881 6882 7254 21776 }"
donkey_udp="{ 4666 6346 6347 7254 21780 }"

meribel="192.168.24.8"
platon="192.168.24.192"
emule_tcp="{ 4242 4661 4662 4881 }"
emule_udp="{ 4246 4665 4672 }"

# Tables: similar to macros, but more flexible for many addresses.
#table  { 10.0.0.0/8, !10.1.0.0/16, 192.168.0.0/24, 192.168.1.18 }

#--- RestartSoft ---
table  const { 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8
}
table  const { $int_net, $virtualbox_net }
table  persist

# Options: tune the behavior of pf, default values are given.
#set timeout { interval 10, frag 30 }
#set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
#set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
#set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
#set timeout { icmp.first 20, icmp.error 10 }
#set timeout { other.first 60, other.single 30, other.multiple 60 }
#set timeout { adaptive.start 0, adaptive.end 0 }
#--- RestartSoft ---
set limit { states 3, src-nodes 2, frags 2 }
#set loginterface none
#set optimization normal
#set block-policy drop
#set require-order yes
#set fingerprints "/etc/pf.os"

#--- RestartSoft ---
set block-policy drop
set debug urgent
set loginterface $vdsl_if
set state-policy if-bound 

# Normalization: reassemble fragments and resolve or reduce traffic
ambiguities.
#scrub in all

#--- RestartSoft ---
scrub in all

# Queueing: rule-based bandwidth control.
#altq on $ext_if bandwidth 2Mb cbq queue { dflt, developers, marketing }
#queue dflt bandwidth 5% cbq(default)
#queue developers bandwidth 80%
#queue marketing  bandwidth 15%

#--- RestartSoft ---
#- altq on $vdsl_if bandwidth 10Mb cbq queue { dflt, p2p_upload }
#- queue dflt   bandwidth 80% cbq(default)
#- queue p2p_upload bandwidth 20%

# Translation: specify how addresses are to be mapped or redirected.
# nat: packets going out through $ext_if with source address $internal_net will
# get translated as coming from the address of $ext_if, a state is created for
# such packets, and incoming packets will be redirected to the internal
address.
#nat on $ext_if from $internal_net to any -> ($ext_if)

#--- RestartSoft ---
# Translate all internal networks
# Special case for ekiga (voip) on meribel
nat on $vdsl_if proto udp from $meribel   to any -> ($vdsl_if) static-port 
nat on $vdsl_if inet from  to any -> ($vdsl_if)

# rdr: packets coming in on $ext_if with destination $external_addr:1234 will
# be redirected to 10.1.1.1:5678. A state is created for such packets, and
# outgoing packets will be translated as coming from the external address.
#rdr on $ext_if proto tcp from any to $external_addr/32 port 1234 -> 10.1.1.1
port 5678

# rdr outgoing FTP requests to the ftp-proxy
#rdr on $int_if proto tcp from any to any port ftp -> 127.0.0.1 port 8021

# spamd-setup puts addresses to be redirected into table .
#table  persist
#no rdr on { lo0, lo1 } from any to any
#rdr inet proto tcp from  to any port smtp -> 127.0.0.1 port 8025
#--- RestartSoft ---
# Ekiga is running on meribel (192.168.24.8)
rdr on $vdsl_if proto udp from any to ($vdsl_if) port 5000:5100 -> $meribel
rdr on $vdsl_if proto tcp from any to ($vdsl_if) port 1720  -> $meribel

# Filtering: the implicit first two rules are
#pass in all
#pass out all

# block all incoming packets but allow ssh, pass all outgoing tcp and udp
# connections and keep state, logging blocked packets.
#block in log all
#pass  in  on $ext_if proto tcp from any to $ext_if port 22 keep state
#pass  out on $ext_if proto { tcp, udp } all keep state

#--- RestartSoft ---
# Setup a default deny policy
block in  log all
block out log all

# pass incoming packets destined to the addresses given in table .
#pass in on $ext_if proto { tcp, udp } from any to  port 80 keep state

# pass 

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #20 from h...@restart.be ---
The same workaround is working for me:

echo "set timeout interval 5" | pfctl -mf -
echo "set timeout interval 10" | pfctl -mf -

but to be sure I also do:
nohup service pf restart

For more than 1 week (without any config change) the problem don't exist.

Next time it appears I will do the dtrace and procstat.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

h...@restart.be changed:

   What|Removed |Added

 CC||h...@restart.be

--- Comment #18 from h...@restart.be ---
I encounter the same problem with current on arm64 - pine64+ 2GB:

FreeBSD norquay.restart.bel 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r320599M: Sat
Jul  8 18:24:19 CEST 2017
r...@norquay.restart.bel:/usr/obj/usr/src/sys/NORQUAY  arm64

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #17 from Max  ---
(In reply to Kristof Provost from comment #15)
You are right. It is not the problem. But it looks quite similar.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #16 from Max  ---
(In reply to noah.bergbauer from comment #14)
I'll try to reproduce the problem. But I need some starting point. Rules, dead
connections state entries... Anything?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #15 from Kristof Provost  ---
(In reply to noah.bergbauer from comment #14)
Given the nature of your workaround and what we've seen from Dtrace I don't
think that #217997 is the problem.
I'm also pretty sure that that fix is included in 11.1.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #14 from noah.bergba...@tum.de ---
(In reply to Max from comment #13)

Maybe, maybe not. The point of my workaround is to get a mostly functioning
machine. However, the reboot right before this period was necessary because for
some reason not even that workaround works consistently.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #13 from Max  ---
(In reply to noah.bergbauer from comment #12)
> Status: Enabled for 1 days 14:44:53
Have you had any issues during this period?
And do you know which rule produces expired states?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

--- Comment #12 from noah.bergba...@tum.de ---
set limit { states 10, src-nodes 1 }

One of my first attempts to fix this was increasing both limits 10x - didn't
help though.


# pfctl -vsi
No ALTQ support in kernel
ALTQ related functions disabled
Status: Enabled for 1 days 14:44:53   Debug: Urgent

Hostid:   0x4b1e78c2
Checksum: 0x67f2a9cbd7b0d65ce52864ecfc156ebb

State Table  Total Rate
  current entries 3839   
  searches   360179452 2582.1/s
  inserts   5949494.3/s
  removals  5911104.2/s
Source Tracking Table
  current entries0   
  searches   00.0/s
  inserts00.0/s
  removals   00.0/s
Counters
  match 6897824.9/s
  bad-offset 00.0/s
  fragment  160.0/s
  short  00.0/s
  normalize  00.0/s
  memory 00.0/s
  bad-timestamp  00.0/s
  congestion 00.0/s
  ip-option  00.0/s
  proto-cksum  4500.0/s
  state-mismatch   9420.0/s
  state-insert   00.0/s
  state-limit00.0/s
  src-limit  00.0/s
  synproxy   00.0/s
  map-failed 00.0/s
Limit Counters
  max states per rule00.0/s
  max-src-states 00.0/s
  max-src-nodes  00.0/s
  max-src-conn   00.0/s
  max-src-conn-rate  00.0/s
  overload table insertion   00.0/s
  overload flush states  00.0/s

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

Max  changed:

   What|Removed |Added

 CC||maxi...@als.nnov.ru

--- Comment #11 from Max  ---
(In reply to noah.bergbauer from comment #10)
Are there any limits in your ruleset? And what does the "pfctl -vsi" show?
Sounds familiar... just like bug #217997.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"

[Bug 222126] pf is not clearing expired states

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222126

Mark Linimon  changed:

   What|Removed |Added

   Assignee|freebsd-b...@freebsd.org|freebsd-pf@FreeBSD.org

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"