Re: APU2 routing speed NetBSD

[76nemo76]

Finally, I have used a new kernel and it works a little bit faster (90Mbits/sec
instead of 70Mbits/sec).

But after being able to play with the new kernel, I get a lot of trouble 
with Windows 10 by transfering data (either by CIFS or FTP) and with all
OS (Linux, OpenBSD, NetBSD) on the router. 

After a lot of tests, searchs and looking on the Web I have noticed that this 
command (from the PF firewall) solves most of my problems (by avoiding too long 
packets)
   match on $ext_if scrub (max-mss 1454 reassemble tcp random-id)

To continue my tests, I should do more or less the same thing
in NPF. The only documentation I have found is here 
http://www.netbsd.org/~rmind/npf/#_application_level_gateways
but the documentation is quite sparse.

Could somebody give me some pointer how to achieve an equivalent result with
NPF?

Thanks for your help,

Alan

On Wed, Dec 21, 2016 at 05:04:46PM +0100, 76nem...@gmx.ch wrote:
> On Wed, Dec 21, 2016 at 10:09:33AM +, co...@sdf.org wrote:
> > wm(4) had a bunch of work done on it since 7.0, so it's worth
> > trying a newer kernel. I believe most of it will be in 7.1 too.
> How can I do that? My first attempt was to download quickly 
> a kernel from 
> "http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201612210220Z/amd64/binary/sets/;
> but when I try to boot I get the following message:
> 
> > boot hd0a:netbsd-current
>   boot: hd0a:netbsd-current: Inappropriate file type or format
> 
> Should I install a new boot program too? In what set can I find it?
> 
> best regards,
> 
> Alan

Re: APU2 routing speed NetBSD

[Thor Lancelot Simon]

On Wed, Dec 21, 2016 at 11:03:33AM +0100, 76nem...@gmx.ch wrote:
>
> Here is the result of the difference on the router. I have had
> a look on it and I see that no "fast forward" are used. 

It is indeed odd that you're not fast forwarding.  Do you have
firewall rules or IPsec policies installed?

Can you try a test with more than one stream?  If the results are higher,
try a test with larger socket buffers -- much larger -- on both sender and
receiver.  A long time ago, I found that 'wm' forwarding performance was
more sensitive to interrupt latency than I expected, and I did a little work
tuning the interrupt mitigation settings in the driver, but I didn't really
care about single-stream performance.

Finally, if you boot with -1, I am curious whether performance goes up.

A kernel profile -- better, perhaps some data gathered with DTrace FBT --
might be helpful, as might the output of lockstat.

Thor

Re: APU2 routing speed NetBSD

[76nemo76]

On Wed, Dec 21, 2016 at 10:09:33AM +, co...@sdf.org wrote:
> wm(4) had a bunch of work done on it since 7.0, so it's worth
> trying a newer kernel. I believe most of it will be in 7.1 too.
How can I do that? My first attempt was to download quickly 
a kernel from 
"http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201612210220Z/amd64/binary/sets/;
but when I try to boot I get the following message:

> boot hd0a:netbsd-current
  boot: hd0a:netbsd-current: Inappropriate file type or format

Should I install a new boot program too? In what set can I find it?

best regards,

Alan

Re: APU2 routing speed NetBSD

[Manuel Bouyer]

On Wed, Dec 21, 2016 at 02:16:14PM +0100, 76nem...@gmx.ch wrote:
> Here are the result of both commandsi (as far as I can understand, I see
> no drop):

inteed there's no drop at the wm0/wm1 level, that's good.
I expected to see a ipintrq in the netstat -q output, but it seems
to be gone in netbsd-7. I don't know how this queue is controlled now,
but this is a place where there could be drops too.

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: APU2 routing speed NetBSD

[76nemo76]

On Wed, Dec 21, 2016 at 11:04:12AM +0100, Manuel Bouyer wrote:
> On Wed, Dec 21, 2016 at 10:38:35AM +0100, 76nem...@gmx.ch wrote:
> > Hi,
> >   Unfortunately the situation has not really changed after the adding
> > of the computation of the checksums by the card.
> > 
> > I have done a new test with NFS (between a Knoppix 7.2 client and a Ubuntu 
> > 16.04 server).
> > The results are the same: 114Mbyte/sec for OpenBSD with firewall disabled 
> > (it
> > is enabled by default) and 70Mbytes/sec with NetBSD.
> 
> Could you check for drops ? netstat -id  and netstat -q
> 
Here are the result of both commandsi (as far as I can understand, I see
no drop):
Name  Mtu   Network   Address  Ipkts IerrsOpkts Oerrs Colls 
Drops
wm0   1500  00:0d:b9:42:61:9c  6128071 0 24129832 0 0 
0
wm0   1500  fe80::/64 fe80::20d:b9ff:fe  6128071 0 24129832 0 0 
0
wm0   1500  172.16/16 jalouvre   6128071 0 24129832 0 0 
0
wm0   1500  fd4f:bd4e:6d2 fd4f:bd4e:6d27:0:  6128071 0 24129832 0 0 
0
wm0   1500  2a02:168:c000 2a02:168:c000:0:d  6128071 0 24129832 0 0 
0
wm1   1500  00:0d:b9:42:61:9d 24135834 0  6128546 0 0 
0
wm1   1500  fe80::/64 fe80::20d:b9ff:fe 24135834 0  6128546 0 0 
0
wm1   1500  192.168.88/24 192.168.88.2  24135834 0  6128546 0 0 
0
wm1   1500  fd4f:bd4e:6d2 fd4f:bd4e:6d27:2: 24135834 0  6128546 0 0 
0
wm1   1500  2a02:168:c000 2a02:168:c000:2:d 24135834 0  6128546 0 0 
0
wm2*  1500  00:0d:b9:42:61:9e0 00 0 0 
0
lo0   33648  0 00 0 0 
0
lo0   33648 127/8 localhost0 00 0 0 
0
lo0   33648 localhost/128 ::1  0 00 0 0 
0
lo0   33648 fe80::/64 fe80::1  0 00 0 0 
0



arpintrq:
queue length: 0
maximum queue length: 50
packets dropped: 0
atintrq1:
queue length: 0
maximum queue length: 256
packets dropped: 0
atintrq2:
queue length: 0
maximum queue length: 256
packets dropped: 0
ppoediscinq:
queue length: 0
maximum queue length: 256
packets dropped: 0
ppoeinq:
queue length: 0
maximum queue length: 256
packets dropped: 0

Re: APU2 routing speed NetBSD

[76nemo76]

On Sun, Dec 18, 2016 at 07:01:42PM -0500, Greg Troxel wrote:
> 
> 76nem...@gmx.ch writes:
> 
> >  I am working with an APU2 board (4gb, 3 lan and AMD CPu 1Ghz
> > you can see specifications here http://www.pcengines.ch/apu2.htm)
> > to make a router/firewall.
> >
> > Before choosing the OS I want to use I have done some benchmarks.
> > My first benchmark is:
> >  1) Copy of a big file (17GB) compressed with gzip
> > from a Windows 10 machine to a Windows 8.1 machine.
> > This is done by using CIFS on Windows. Each machine
> > is at one side of the router.
> >
> > The results are strange:
> > With OpenBSD and Linux (alpine linux 3.4) I get
> > the maximum speed (112Mbytes/sec) while with NetBSD
> > the speed is limited to 70Mbytes/sec.
> 
> There are basically two things to see about.  One is the raw speed of
> packet forwarding.  The other is whether there is any loss and how that
> interacts with congestion control.  Plus then there are odd things I am
> not thinking of.
> 
> NetBSD itself should be efficient.  However, you are pushing close to 1
> Gb/s.  So, I would want to look at network counters to see if there is
> any trouble.  Do "netstat -s" on the router to a file, before and after,
> and diff -u.  Also, do the equivalent on the test machines.
> 
Here is the result of the difference on the router. I have had
a look on it and I see that no "fast forward" are used. 

Could be that the reason of slowness? I have read somewhere
that fast forward could increase routing speed.

Best regards,

Alan

Differences between file produced by the netstat -s command before and
after the transfer.

--- /tmp/avantCopy.txt  2016-12-21 11:00:32.0 +0100
+++ /tmp/apresCopy.txt  2016-12-21 11:07:48.0 +0100
@@ -84,18 +84,18 @@
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
 udp:
-   203 datagrams received
+   231 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 dropped due to no socket
-   36 broadcast/multicast datagrams dropped due to no socket
+   45 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
-   167 delivered
-   187 PCB hash misses
-   169 datagrams output
+   186 delivered
+   215 PCB hash misses
+   188 datagrams output
 ip:
-   15146914 total packets received
+   30256038 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
@@ -110,13 +110,13 @@
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
-   203 packets for this host
+   231 packets for this host
0 packets for unknown/unsupported protocol
-   15146698 packets forwarded (0 packets fast forwarded)
-   13 packets not forwardable
+   30255791 packets forwarded (0 packets fast forwarded)
+   16 packets not forwardable
0 redirects sent
0 packets no matching gif found
-   169 packets sent from this host
+   188 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
@@ -208,7 +208,7 @@
0 ipcomp input bytes
0 ipcomp output bytes
 ip6:
-   44 total packets received
+   54 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
@@ -218,7 +218,7 @@
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
-   32 packets for this host
+   39 packets for this host
0 packets forwarded
0 packets fast forwarded
0 fast forward flows
@@ -227,18 +227,18 @@
20 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
-   50 output packets discarded due to no route
+   56 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
0 multicast packets which we don't join
Input packet histogram:
-   UDP: 32
-   ICMP6: 12
+   UDP: 39
+   ICMP6: 15
Mbuf statistics:
0 one mbufs
-   44 one ext mbufs
+   54 one ext mbufs
0 two or more ext mbufs
0 packets whose headers are not continuous
0 tunneling packets that can't find gif
@@ -258,7 +258,7 @@
0 bad checksums
0 messages with bad length
Input 

Re: APU2 routing speed NetBSD

[coypu]

wm(4) had a bunch of work done on it since 7.0, so it's worth
trying a newer kernel. I believe most of it will be in 7.1 too.

NPF is not used unless you edit some config files, but it
shouldn't make a difference on a small workload.

Re: APU2 routing speed NetBSD

[Manuel Bouyer]

On Wed, Dec 21, 2016 at 10:38:35AM +0100, 76nem...@gmx.ch wrote:
> Hi,
>   Unfortunately the situation has not really changed after the adding
> of the computation of the checksums by the card.
> 
> I have done a new test with NFS (between a Knoppix 7.2 client and a Ubuntu 
> 16.04 server).
> The results are the same: 114Mbyte/sec for OpenBSD with firewall disabled (it
> is enabled by default) and 70Mbytes/sec with NetBSD.

Could you check for drops ? netstat -id  and netstat -q

> 
> This is annoying while I don't understand what happens. I should still make 
> some test
> with a firewall to see how the transfer speed will decrease. With the slight
> firewall (1 pass, two block rules) of OpenBSD, the speed was about 
> 90Mbytes/sec.
> 
> By the way, is the default kernel from Netbsd SMP aware? 

yes

> Is a firewall enabled by default on NetBSD (like in OpenBSD)?

no

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: APU2 routing speed NetBSD

[Manuel Bouyer]

On Tue, Dec 20, 2016 at 10:00:59AM +0100, 76nem...@gmx.ch wrote:
> Thanks for the info. I suppose that *CSUM is the computation of
> the checksum of packet by the card. How I Configure it in NetBSD?

with ifconfig
e.g.
ifconfig wm0 ip4csum tcp4csum udp4csum ...

> 
> Probably this should be done in "hostname.if" file, but how.

in ifconfig.wm0 and ifconfig.wm1

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: APU2 routing speed NetBSD

[76nemo76]

On Tue, Dec 20, 2016 at 09:44:20AM +0100, Manuel Bouyer wrote:
> On Tue, Dec 20, 2016 at 09:21:59AM +0100, 76nem...@gmx.ch wrote:
> > Hi Martin,
> >  Here are the requested info. Since my last test, I have only added the
> > IPv6 address. The first benchmark was done with IPv4.
> > 
> > Thanks for help
> > 
> > Best regard
> > 
> > Alan
> > 
> > wm0: flags=8843 mtu 1500
> > capabilities=7ff80
> > capabilities=7ff80
> > capabilities=7ff80
> > enabled=0
> 
> You want to enable the *CSUM features here. Other OSes do it by
> default.
> 
Thanks for the info. I suppose that *CSUM is the computation of
the checksum of packet by the card. How I Configure it in NetBSD?

Probably this should be done in "hostname.if" file, but how.

Best regards,

Alan

Re: APU2 routing speed NetBSD

[Manuel Bouyer]

On Tue, Dec 20, 2016 at 09:21:59AM +0100, 76nem...@gmx.ch wrote:
> Hi Martin,
>  Here are the requested info. Since my last test, I have only added the
> IPv6 address. The first benchmark was done with IPv4.
> 
> Thanks for help
> 
> Best regard
> 
> Alan
> 
> wm0: flags=8843 mtu 1500
>   capabilities=7ff80
>   capabilities=7ff80
>   capabilities=7ff80
>   enabled=0

You want to enable the *CSUM features here. Other OSes do it by
default.

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: APU2 routing speed NetBSD

[76nemo76]

Hi Martin,
 Here are the requested info. Since my last test, I have only added the
IPv6 address. The first benchmark was done with IPv4.

Thanks for help

Best regard

Alan

wm0: flags=8843 mtu 1500
capabilities=7ff80
capabilities=7ff80
capabilities=7ff80
enabled=0
ec_capabilities=7
ec_enabled=0
address: 00:0d:b9:42:61:9c
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 172.16.0.10 netmask 0x broadcast 172.16.255.255
inet6 fe80::20d:b9ff:fe42:619c%wm0 prefixlen 64 scopeid 0x1
inet6 fd4f:bd4e:6d27:0:de0b:f387:afeb:7 prefixlen 64
inet6 2a02:168:c000:0:de0b:f387:afeb:7 prefixlen 64


wm1: flags=8843 mtu 1500
capabilities=7ff80
capabilities=7ff80
capabilities=7ff80
enabled=0
ec_capabilities=7
ec_enabled=0
address: 00:0d:b9:42:61:9d
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 192.168.88.2 netmask 0xff00 broadcast 192.168.88.255
inet6 fe80::20d:b9ff:fe42:619d%wm1 prefixlen 64 scopeid 0x2
inet6 fd4f:bd4e:6d27:2:de0b:f387:afeb:2 prefixlen 64
inet6 2a02:168:c000:2:de0b:f387:afeb:2 prefixlen 64


Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015
The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California.  All rights reserved.

NetBSD 7.0.2 (GENERIC.201610210724Z)
total memory = 4079 MB
avail memory = 3943 MB
kern.module.path=/stand/amd64/7.0/modules
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
PC Engines apu2 (1.0)
mainbus0 (root)
ACPI: RSDP 0xf30e0 24 (v02 CORE  )
ACPI: XSDT 0xdffb80e0 5C (v01 CORE   COREBOOT  CORE )
ACPI: FACP 0xdffb96f0 F4 (v04 CORE   COREBOOT  CORE )
ACPI: DSDT 0xdffb8250 001496 (v02 AMDCOREBOOT 00010001 INTL 20140114)
ACPI: FACS 0xdffb8210 40
ACPI: SSDT 0xdffb97f0 45 (v02 CORE   COREBOOT 002A CORE 002A)
ACPI: APIC 0xdffb9840 7E (v01 CORE   COREBOOT  CORE )
ACPI: HEST 0xdffb98c0 0001D0 (v01 CORE   COREBOOT  CORE )
ACPI: SSDT 0xdffb9a90 0048A6 (v02 AMDAGESA0002 MSFT 0400)
ACPI: SSDT 0xdffbe340 0007C8 (v01 AMDAGESA0001 AMD  0001)
ACPI: HPET 0xdffbeb10 38 (v01 CORE   COREBOOT  CORE )
ACPI: All ACPI Tables successfully acquired
ioapic0 at mainbus0 apid 4: pa 0xfec0, version 0x21, 24 pins
ioapic1 at mainbus0 apid 5: pa 0xfec2, version 0x21, 32 pins
cpu0 at mainbus0 apid 0: AMD GX-412TC SOC   , id 
0x730f01
cpu1 at mainbus0 apid 1: AMD GX-412TC SOC   , id 
0x730f01
cpu2 at mainbus0 apid 2: AMD GX-412TC SOC   , id 
0x730f01
cpu3 at mainbus0 apid 3: AMD GX-412TC SOC   , id 
0x730f01
acpi0 at mainbus0: Intel ACPICA 20131218
acpi0: X/RSDT: OemId , AslId 
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
hpet0 at acpi0: high precision event timer (mem 0xfed0-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
acpibut0 at acpi0 (PWRB, PNP0C0C-170): ACPI Power Button
LDRC (PNP0C02) at acpi0 not configured
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
COM1 (PNP0501) at acpi0 not configured
AWR0 (PNP0C02) at acpi0 not configured
ABR0 (PNP0C02) at acpi0 not configured
ABR1 (PNP0C02) at acpi0 not configured
ABR2 (PNP0C02) at acpi0 not configured
ABR3 (PNP0C02) at acpi0 not configured
ABR4 (PNP0C02) at acpi0 not configured
ACPI: Enabled 4 GPEs in block 00 to 1F
attimer1: attached to pcppi1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 0x1022 product 0x1566 (rev. 0x00)
pchb1 at pci0 dev 2 function 0: vendor 0x1022 product 0x156b (rev. 0x00)
ppb0 at pci0 dev 2 function 2: vendor 0x1022 product 0x1439 (rev. 0x00)
ppb0: PCI Express capability version 2  x1 @ 
5.0GT/s
ppb0: link is x1 @ 2.5GT/s
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
wm0 at pci1 dev 0 function 0: I210 Ethernet (FLASH less) (rev. 0x03)

Re: APU2 routing speed NetBSD

[Greg Troxel]

76nem...@gmx.ch writes:

>  I am working with an APU2 board (4gb, 3 lan and AMD CPu 1Ghz
> you can see specifications here http://www.pcengines.ch/apu2.htm)
> to make a router/firewall.
>
> Before choosing the OS I want to use I have done some benchmarks.
> My first benchmark is:
>  1) Copy of a big file (17GB) compressed with gzip
> from a Windows 10 machine to a Windows 8.1 machine.
> This is done by using CIFS on Windows. Each machine
> is at one side of the router.
>
> The results are strange:
> With OpenBSD and Linux (alpine linux 3.4) I get
> the maximum speed (112Mbytes/sec) while with NetBSD
> the speed is limited to 70Mbytes/sec.

There are basically two things to see about.  One is the raw speed of
packet forwarding.  The other is whether there is any loss and how that
interacts with congestion control.  Plus then there are odd things I am
not thinking of.

NetBSD itself should be efficient.  However, you are pushing close to 1
Gb/s.  So, I would want to look at network counters to see if there is
any trouble.  Do "netstat -s" on the router to a file, before and after,
and diff -u.  Also, do the equivalent on the test machines.

This will slow it down, but run tcpudmp and save to a file, and look
over it.  If TCP, use tcpdump2xplot from xplot in pkgsrc (read the docs;
this is not really easy, but very helpful to understand TCP behavior).
If CIFS is over UDP, look at retransmission specs on the clients.

Understand what MTU is being used, with various systems.  Jumbo frames
of 8K data plus header may be faster due to amortizing per-packet
overhead and faster pack/unpack.  THose may be enabled by the others.

Finally, 70 MB/s is still half of GbE, and it may be that other
considerations are more important than the last bit of speed.  But
still, I think this should go faster.



signature.asc
Description: PGP signature

Re: APU2 routing speed NetBSD

[Martin Husemann]

On Sun, Dec 18, 2016 at 03:08:35PM +0100, 76nem...@gmx.ch wrote:
> Hi All,
>  I am working with an APU2 board (4gb, 3 lan and AMD CPu 1Ghz
> you can see specifications here http://www.pcengines.ch/apu2.htm)
> to make a router/firewall.

Can you give a few more details, e.g. the output of dmesg, and ifconfig
for the interfaces used?

Martin