Re: CFT: if_bridge performance improvements
On 4/24/20 06:42, Kristof Provost wrote: > On 22 Apr 2020, at 18:15, Xin Li wrote: >> On 4/22/20 01:45, Kristof Provost wrote: >>> On 22 Apr 2020, at 10:20, Xin Li wrote: Hi, On 4/14/20 02:51, Kristof Provost wrote: > Hi, > > Thanks to support from The FreeBSD Foundation I’ve been able to > work on > improving the throughput of if_bridge. > It changes the (data path) locking to use the NET_EPOCH > infrastructure. > Benchmarking shows substantial improvements (x5 in test setups). > > This work is ready for wider testing now. > > It’s under review here: https://reviews.freebsd.org/D24250 > > Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true > Patches for stable/12: > https://people.freebsd.org/~kp/if_bridge/stable_12/ > > I’m not currently aware of any panics or issues resulting from these > patches. I have observed the following panic with latest stable/12 after applying the stable_12 patchset, it appears like a race condition related NULL pointer deference, but I haven't took a deeper look yet. The box have 7 igb(4) NICs, with several bridge and VLAN configured acting as a router. Please let me know if you need additional information; I can try -CURRENT as well, but it would take some time as the box is relatively slow (it's a ZFS based system so I can create a separate boot environment for -CURRENT if needed, but that would take some time as I might have to upgrade the packages, should there be any ABI breakages). >>> Thanks for the report. I don’t immediately see how this could happen. >>> >>> Are you running an L2 firewall on that bridge by any chance? An earlier >>> version of the patch had issues with a stray unlock in that code path. >> >> I don't think I have a L2 firewall (I assume means filtering based on >> MAC address like what can be done with e.g. ipfw? The bridges were >> created on vlan interfaces though, do they count as L2 firewall?), the >> system is using pf with a few NAT rules: >> > > That backtrace looks identical to the one Peter reported, up to and > including the offset in the bridge_input() function. > Given that there’s no likely way to end up with a NULL mutex either I > have to assume that it’s a case of trying to unlock a locked mutex, and > the most likely reason is that you ran into the same problem Peter ran > into. > > The current version of the patch should resolve it. Thanks, I'd like to report that after applying the patch from Peter the system seems to survive without problem. Cheers, signature.asc Description: OpenPGP digital signature
Re: CFT: if_bridge performance improvements
On 22 Apr 2020, at 18:15, Xin Li wrote: On 4/22/20 01:45, Kristof Provost wrote: On 22 Apr 2020, at 10:20, Xin Li wrote: Hi, On 4/14/20 02:51, Kristof Provost wrote: Hi, Thanks to support from The FreeBSD Foundation I’ve been able to work on improving the throughput of if_bridge. It changes the (data path) locking to use the NET_EPOCH infrastructure. Benchmarking shows substantial improvements (x5 in test setups). This work is ready for wider testing now. It’s under review here: https://reviews.freebsd.org/D24250 Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_12/ I’m not currently aware of any panics or issues resulting from these patches. I have observed the following panic with latest stable/12 after applying the stable_12 patchset, it appears like a race condition related NULL pointer deference, but I haven't took a deeper look yet. The box have 7 igb(4) NICs, with several bridge and VLAN configured acting as a router. Please let me know if you need additional information; I can try -CURRENT as well, but it would take some time as the box is relatively slow (it's a ZFS based system so I can create a separate boot environment for -CURRENT if needed, but that would take some time as I might have to upgrade the packages, should there be any ABI breakages). Thanks for the report. I don’t immediately see how this could happen. Are you running an L2 firewall on that bridge by any chance? An earlier version of the patch had issues with a stray unlock in that code path. I don't think I have a L2 firewall (I assume means filtering based on MAC address like what can be done with e.g. ipfw? The bridges were created on vlan interfaces though, do they count as L2 firewall?), the system is using pf with a few NAT rules: That backtrace looks identical to the one Peter reported, up to and including the offset in the bridge_input() function. Given that there’s no likely way to end up with a NULL mutex either I have to assume that it’s a case of trying to unlock a locked mutex, and the most likely reason is that you ran into the same problem Peter ran into. The current version of the patch should resolve it. Best regards, Kristof ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: CFT: if_bridge performance improvements
Just using pf is enough to provoke this panic. I had the same back trace. This
patch from Kristof fixed it for me.
diff --git a/sys/net/if_bridge.c b/sys/net/if_bridge.c
index 373fa096d70..83c453090bb 100644
--- a/sys/net/if_bridge.c
+++ b/sys/net/if_bridge.c
@@ -2529,7 +2529,6 @@ bridge_input(struct ifnet *ifp, struct mbuf *m)
OR_PFIL_HOOKED_INET6)) { \
if (bridge_pfil(, NULL, ifp, \
PFIL_IN) != 0 || m == NULL) { \
- BRIDGE_UNLOCK(sc); \
return (NULL); \
} \
eh = mtod(m, struct ether_header *);\
> On 22 Apr 2020, at 18:15, Xin Li wrote:
>
> On 4/22/20 01:45, Kristof Provost wrote:
>> On 22 Apr 2020, at 10:20, Xin Li wrote:
>>> Hi,
>>>
>>> On 4/14/20 02:51, Kristof Provost wrote:
Hi,
Thanks to support from The FreeBSD Foundation I’ve been able to work on
improving the throughput of if_bridge.
It changes the (data path) locking to use the NET_EPOCH infrastructure.
Benchmarking shows substantial improvements (x5 in test setups).
This work is ready for wider testing now.
It’s under review here: https://reviews.freebsd.org/D24250
Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true
Patches for stable/12:
https://people.freebsd.org/~kp/if_bridge/stable_12/
I’m not currently aware of any panics or issues resulting from these
patches.
>>>
>>> I have observed the following panic with latest stable/12 after applying
>>> the stable_12 patchset, it appears like a race condition related NULL
>>> pointer deference, but I haven't took a deeper look yet.
>>>
>>> The box have 7 igb(4) NICs, with several bridge and VLAN configured
>>> acting as a router. Please let me know if you need additional
>>> information; I can try -CURRENT as well, but it would take some time as
>>> the box is relatively slow (it's a ZFS based system so I can create a
>>> separate boot environment for -CURRENT if needed, but that would take
>>> some time as I might have to upgrade the packages, should there be any
>>> ABI breakages).
>>>
>> Thanks for the report. I don’t immediately see how this could happen.
>>
>> Are you running an L2 firewall on that bridge by any chance? An earlier
>> version of the patch had issues with a stray unlock in that code path.
>
> I don't think I have a L2 firewall (I assume means filtering based on
> MAC address like what can be done with e.g. ipfw? The bridges were
> created on vlan interfaces though, do they count as L2 firewall?), the
> system is using pf with a few NAT rules:
>
> $ sudo pfctl -s rules
> anchor "miniupnpd" all
> pass in quick inet6 proto tcp from to any flags S/SA keep state
> block drop in quick inet6 proto tcp from ! to flags S/SA
> block drop in quick proto tcp from any os "Linux" to any port = ssh
> pass out on igb6 inet proto tcp from (igb6) to any port = domain flags
> S/SA keep state queue dns
> pass out on igb6 inet proto udp from (igb6) to any port = domain keep
> state queue dns
> pass in on igb6 proto tcp from any to (igb6) port = http flags S/SA
> modulate state queue(web, ack)
> pass in on igb6 proto tcp from any to (igb6) port = https flags S/SA
> modulate state queue(web, ack)
> pass out on igb6 inet proto tcp from (igb6) to any flags S/SA modulate
> state queue bulk
> block drop in quick on igb6 proto tcp from to any port = ssh
> label "ssh bruteforce"
> block drop in on igb6 from to any
>
> Cheers,
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: CFT: if_bridge performance improvements
On 4/22/20 01:45, Kristof Provost wrote: > On 22 Apr 2020, at 10:20, Xin Li wrote: >> Hi, >> >> On 4/14/20 02:51, Kristof Provost wrote: >>> Hi, >>> >>> Thanks to support from The FreeBSD Foundation I’ve been able to work on >>> improving the throughput of if_bridge. >>> It changes the (data path) locking to use the NET_EPOCH infrastructure. >>> Benchmarking shows substantial improvements (x5 in test setups). >>> >>> This work is ready for wider testing now. >>> >>> It’s under review here: https://reviews.freebsd.org/D24250 >>> >>> Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true >>> Patches for stable/12: >>> https://people.freebsd.org/~kp/if_bridge/stable_12/ >>> >>> I’m not currently aware of any panics or issues resulting from these >>> patches. >> >> I have observed the following panic with latest stable/12 after applying >> the stable_12 patchset, it appears like a race condition related NULL >> pointer deference, but I haven't took a deeper look yet. >> >> The box have 7 igb(4) NICs, with several bridge and VLAN configured >> acting as a router. Please let me know if you need additional >> information; I can try -CURRENT as well, but it would take some time as >> the box is relatively slow (it's a ZFS based system so I can create a >> separate boot environment for -CURRENT if needed, but that would take >> some time as I might have to upgrade the packages, should there be any >> ABI breakages). >> > Thanks for the report. I don’t immediately see how this could happen. > > Are you running an L2 firewall on that bridge by any chance? An earlier > version of the patch had issues with a stray unlock in that code path. I don't think I have a L2 firewall (I assume means filtering based on MAC address like what can be done with e.g. ipfw? The bridges were created on vlan interfaces though, do they count as L2 firewall?), the system is using pf with a few NAT rules: $ sudo pfctl -s rules anchor "miniupnpd" all pass in quick inet6 proto tcp from to any flags S/SA keep state block drop in quick inet6 proto tcp from ! to flags S/SA block drop in quick proto tcp from any os "Linux" to any port = ssh pass out on igb6 inet proto tcp from (igb6) to any port = domain flags S/SA keep state queue dns pass out on igb6 inet proto udp from (igb6) to any port = domain keep state queue dns pass in on igb6 proto tcp from any to (igb6) port = http flags S/SA modulate state queue(web, ack) pass in on igb6 proto tcp from any to (igb6) port = https flags S/SA modulate state queue(web, ack) pass out on igb6 inet proto tcp from (igb6) to any flags S/SA modulate state queue bulk block drop in quick on igb6 proto tcp from to any port = ssh label "ssh bruteforce" block drop in on igb6 from to any Cheers, signature.asc Description: OpenPGP digital signature
Re: CFT: if_bridge performance improvements
On 22 Apr 2020, at 10:20, Xin Li wrote: Hi, On 4/14/20 02:51, Kristof Provost wrote: Hi, Thanks to support from The FreeBSD Foundation I’ve been able to work on improving the throughput of if_bridge. It changes the (data path) locking to use the NET_EPOCH infrastructure. Benchmarking shows substantial improvements (x5 in test setups). This work is ready for wider testing now. It’s under review here: https://reviews.freebsd.org/D24250 Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_12/ I’m not currently aware of any panics or issues resulting from these patches. I have observed the following panic with latest stable/12 after applying the stable_12 patchset, it appears like a race condition related NULL pointer deference, but I haven't took a deeper look yet. The box have 7 igb(4) NICs, with several bridge and VLAN configured acting as a router. Please let me know if you need additional information; I can try -CURRENT as well, but it would take some time as the box is relatively slow (it's a ZFS based system so I can create a separate boot environment for -CURRENT if needed, but that would take some time as I might have to upgrade the packages, should there be any ABI breakages). Thanks for the report. I don’t immediately see how this could happen. Are you running an L2 firewall on that bridge by any chance? An earlier version of the patch had issues with a stray unlock in that code path. Best regards, Kristof ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: CFT: if_bridge performance improvements
Hi, On 4/14/20 02:51, Kristof Provost wrote: > Hi, > > Thanks to support from The FreeBSD Foundation I’ve been able to work on > improving the throughput of if_bridge. > It changes the (data path) locking to use the NET_EPOCH infrastructure. > Benchmarking shows substantial improvements (x5 in test setups). > > This work is ready for wider testing now. > > It’s under review here: https://reviews.freebsd.org/D24250 > > Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true > Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_12/ > > I’m not currently aware of any panics or issues resulting from these > patches. I have observed the following panic with latest stable/12 after applying the stable_12 patchset, it appears like a race condition related NULL pointer deference, but I haven't took a deeper look yet. The box have 7 igb(4) NICs, with several bridge and VLAN configured acting as a router. Please let me know if you need additional information; I can try -CURRENT as well, but it would take some time as the box is relatively slow (it's a ZFS based system so I can create a separate boot environment for -CURRENT if needed, but that would take some time as I might have to upgrade the packages, should there be any ABI breakages). === Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x20 fault code = supervisor read data, page not present instruction pointer = 0x20:0x80c286d5 stack pointer = 0x28:0x824cb840 frame pointer = 0x28:0x824cb850 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= resume, IOPL = 0 current process = 0 (if_io_tqg_0) trap number = 12 panic: page fault cpuid = 0 time = 1587541913 KDB: stack backtrace: #0 0x80c117a5 at kdb_backtrace+0x65 #1 0x80bc588e at vpanic+0x17e #2 0x80bc5703 at panic+0x43 #3 0x810d2310 at trap_pfault+0 #4 0x810d235f at trap_pfault+0x4f #5 0x810d19b8 at trap+0x288 #6 0x810aae1c at calltrap+0x8 #7 0x80ba5c96 at __mtx_unlock_sleep+0xb6 #8 0x8248f4c7 at bridge_input+0x877 #9 0x80cd5c47 at ether_nh_input+0x207 #10 0x80cf1e4a at netisr_dispatch_src+0xca #11 0x80cd4f0b at ether_input+0x4b #12 0x80cdf1a3 at vlan_input+0x1f3 #13 0x80cd4ae1 at ether_demux+0x121 #14 0x80cd5d7b at ether_nh_input+0x33b #15 0x80cf1e4a at netisr_dispatch_src+0xca #16 0x80cd4f0b at ether_input+0x4b #17 0x80cee41c at iflib_rxeof+0xadc Uptime: 6m6s Dumping 848 out of 16313 MB:..2%..12%..21%..31%..42%..51%..61%..72%..82%..91% Backtrace: (kgdb) #0 doadump () at src/sys/amd64/include/pcpu_aux.h:55 #1 0x80bc54a5 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451 #2 0x80bc58e6 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:880 #3 0x80bc5703 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:807 #4 0x810d2310 in trap_fatal (frame=, eva=) at /usr/src/sys/amd64/amd64/trap.c:925 #5 0x810d235f in trap_pfault (frame=0x824cb780, usermode=, signo=, ucode=) at src/sys/amd64/include/pcpu_aux.h:55 #6 0x810d19b8 in trap (frame=0x824cb780) at /usr/src/sys/amd64/amd64/trap.c:407 #7 0x810aae1c in calltrap () at /usr/src/sys/amd64/amd64/exception.S:289 #8 0x80c286d5 in turnstile_broadcast (ts=0x0, queue=0) at /usr/src/sys/kern/subr_turnstile.c:880 #9 0x80ba5c96 in __mtx_unlock_sleep (c=0xf80013351430, v=0) at /usr/src/sys/kern/kern_mutex.c:1041 #10 0x8248f4c7 in bridge_input (ifp=, m=) at src/sys/amd64/include/atomic.h:221 #11 0x80cd5c47 in ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:631 #12 0x80cf1e4a in netisr_dispatch_src (proto=5, source=, m=) at /usr/src/sys/net/netisr.c:1124 #13 0x80cd4f0b in ether_input (ifp=0xf800060dc000, m=0x0) at /usr/src/sys/net/if_ethersubr.c:787 #14 0x80cdf1a3 in vlan_input (ifp=0xf800036d6800, m=0xf8001d65fc00) at /usr/src/sys/net/if_vlan.c:1291 #15 0x80cd4ae1 in ether_demux (ifp=0xf800036d6800, m=) at /usr/src/sys/net/if_ethersubr.c:832 #16 0x80cd5d7b in ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:667 #17 0x80cf1e4a in netisr_dispatch_src (proto=5, source=, m=) at /usr/src/sys/net/netisr.c:1124 #18 0x80cd4f0b in ether_input (ifp=0xf800036d6800, m=0xf80013939c00) at /usr/src/sys/net/if_ethersubr.c:787 #19 0x80cee41c in iflib_rxeof (rxq=, budget=) at /usr/src/sys/net/iflib.c:2873 #20 0x80ce87b3 in _task_fn_rx (context=0xf800036d6000) at
Re: CFT: if_bridge performance improvements
Hi, Interesting, maybe ng_eiface + if_bridge is a good idea . Simon 20200422 On 4/21/20 8:51 PM, Jan Bramkamp wrote: On 16.04.20 08:34, Pavel Timofeev wrote: вт, 14 апр. 2020 г., 12:51 Kristof Provost : Hi, Thanks to support from The FreeBSD Foundation I’ve been able to work on improving the throughput of if_bridge. It changes the (data path) locking to use the NET_EPOCH infrastructure. Benchmarking shows substantial improvements (x5 in test setups). This work is ready for wider testing now. It’s under review here: https://reviews.freebsd.org/D24250 Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_12/ I’m not currently aware of any panics or issues resulting from these patches. Do note that if you run a Bhyve + tap on bridges setup the tap code suffers from a similar bottleneck and you will likely not see major improvements in single VM to host throughput. I would expect, but have not tested, improvements in overall throughput (i.e. when multiple VMs send traffic at the same time). Best regards, Kristof Hi! Thank you for your work! Do you know if epair suffers from the same issue as tap? As Kirstof Provost said if_epair locks has per CPU locks, but a problem exists a layer about the epair driver. At leas on FreeBSD 12.0 and 12.1 all the packet processing happens in a single netisr thread that becomes CPU bound and limits how fast useful traffic can move through epair interfaces. Afaik TCP doesn't benifit from multiple netisr threads, but unorderer protocols (e.g. UDP) could profit from multiple threads. I have only tested with iperf (using multiple connections) between the FreeBSD 12.x host and a vnet enabled jail connected via an epair interface and maxed out at about 1-2Gb/s depending on the CPUs single threaded throughput. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: CFT: if_bridge performance improvements
On 16.04.20 08:34, Pavel Timofeev wrote: вт, 14 апр. 2020 г., 12:51 Kristof Provost : Hi, Thanks to support from The FreeBSD Foundation I’ve been able to work on improving the throughput of if_bridge. It changes the (data path) locking to use the NET_EPOCH infrastructure. Benchmarking shows substantial improvements (x5 in test setups). This work is ready for wider testing now. It’s under review here: https://reviews.freebsd.org/D24250 Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_12/ I’m not currently aware of any panics or issues resulting from these patches. Do note that if you run a Bhyve + tap on bridges setup the tap code suffers from a similar bottleneck and you will likely not see major improvements in single VM to host throughput. I would expect, but have not tested, improvements in overall throughput (i.e. when multiple VMs send traffic at the same time). Best regards, Kristof Hi! Thank you for your work! Do you know if epair suffers from the same issue as tap? As Kirstof Provost said if_epair locks has per CPU locks, but a problem exists a layer about the epair driver. At leas on FreeBSD 12.0 and 12.1 all the packet processing happens in a single netisr thread that becomes CPU bound and limits how fast useful traffic can move through epair interfaces. Afaik TCP doesn't benifit from multiple netisr threads, but unorderer protocols (e.g. UDP) could profit from multiple threads. I have only tested with iperf (using multiple connections) between the FreeBSD 12.x host and a vnet enabled jail connected via an epair interface and maxed out at about 1-2Gb/s depending on the CPUs single threaded throughput. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: CFT: if_bridge performance improvements
On 16 Apr 2020, at 8:34, Pavel Timofeev wrote: Hi! Thank you for your work! Do you know if epair suffers from the same issue as tap? I’ve not tested it, but I believe that epair scales significantly better than tap. It has a per-cpu mutex (or more accurately, a mutex in each of its per-cpu structures), so I’d expect much better throughput from epair than you’d see from tap. Best regards, Kristof ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: CFT: if_bridge performance improvements
вт, 14 апр. 2020 г., 12:51 Kristof Provost : > Hi, > > Thanks to support from The FreeBSD Foundation I’ve been able to work > on improving the throughput of if_bridge. > It changes the (data path) locking to use the NET_EPOCH infrastructure. > Benchmarking shows substantial improvements (x5 in test setups). > > This work is ready for wider testing now. > > It’s under review here: https://reviews.freebsd.org/D24250 > > Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true > Patches for stable/12: > https://people.freebsd.org/~kp/if_bridge/stable_12/ > > I’m not currently aware of any panics or issues resulting from these > patches. > > Do note that if you run a Bhyve + tap on bridges setup the tap code > suffers from a similar bottleneck and you will likely not see major > improvements in single VM to host throughput. I would expect, but have > not tested, improvements in overall throughput (i.e. when multiple VMs > send traffic at the same time). > > Best regards, > Kristof > Hi! Thank you for your work! Do you know if epair suffers from the same issue as tap? > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
CFT: if_bridge performance improvements
Hi, Thanks to support from The FreeBSD Foundation I’ve been able to work on improving the throughput of if_bridge. It changes the (data path) locking to use the NET_EPOCH infrastructure. Benchmarking shows substantial improvements (x5 in test setups). This work is ready for wider testing now. It’s under review here: https://reviews.freebsd.org/D24250 Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true Patches for stable/12: https://people.freebsd.org/~kp/if_bridge/stable_12/ I’m not currently aware of any panics or issues resulting from these patches. Do note that if you run a Bhyve + tap on bridges setup the tap code suffers from a similar bottleneck and you will likely not see major improvements in single VM to host throughput. I would expect, but have not tested, improvements in overall throughput (i.e. when multiple VMs send traffic at the same time). Best regards, Kristof ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
