On Sun, Jan 25, 2015 at 07:33:47PM -0800, Peter Grehan wrote:
> Hi,
>
> > I saw that there is an open task of adding Netmap support to bhyve in
> > the latest status report of FreeBSD [1]. So I implement the netmap
> > support for bhyve, and following is the patch. Hope this can be helpful!
>
> Nice work ! Did you manage to get any throughput numbers ?
Thanks! Yeah, I have done some throughput tests. Here are the details:
#1. The parameters I used to launch the VMs:
$ sudo /home/btw/src/usr.sbin/bhyve/bhyve \
-A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2,virtio-blk,./disk-vm1 \
-s 3:0,virtio-net,vale0:vm1 \
-s 3:1,virtio-net,tap1 \
-l com1,stdio \
-m 2G -c 2 vm1
$ sudo /home/btw/src/usr.sbin/bhyve/bhyve \
-A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2,virtio-blk,./disk-vm2 \
-s 3:0,virtio-net,vale0:vm2 \
-s 3:1,virtio-net,tap2 \
-l com1,stdio \
-m 2G -c 1 vm2
#2. The outputs of pkt-gen as the receiver when using tap+bridge:
$ ./pkt-gen -i vtnet1 -f rx
075.499873 main [1686] interface is vtnet1
075.500516 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0
075.500677 extract_ip_range075.501408 [ 407] vtnet_netmap_config vtnet
config txq=1, txd=512 rxq=1, rxd=512
[289] range is 075.503540 [ 407] vtnet_netmap_config vtnet config txq=1,
txd=512 rxq=1, rxd=512
10.1.0.1:0 to 10.1.0.1:0
075.505840 [ 79] vtnet_netmap_free_bufsfreed 1024 mbufs, 0 netmap bufs on
1 queues
075.507838 main [1877] mapped 334980KB at 0x801dff000
Receiving from netmap:vtnet1: 1 queues, 1 threads and 1 cpus.
075.509298 main [1963] Wait 2 secs for phy reset
077.553231 main [1965] Ready...
077.553787 nm_open [456] overriding ifname vtnet1 ringid 0x0 flags 0x1
077.554511 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1,
rxd=512
077.556006 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1,
rxd=512
077.557663 receiver_body [1219] reading from netmap:vtnet1 fd 4 main_fd 3
078.563897 main_thread [1483] 311989 pps (313942 pkts in 1006261 usec)
079.574224 main_thread [1483] 45 pps (336771 pkts in 1010278 usec)
080.584718 main_thread [1483] 307455 pps (310688 pkts in 1010514 usec)
081.600552 main_thread [1483] 332357 pps (337629 pkts in 1015862 usec)
082.604157 main_thread [1483] 342985 pps (344222 pkts in 1003606 usec)
083.614341 main_thread [1483] 327806 pps (331144 pkts in 1010184 usec)
084.634725 main_thread [1483] 320947 pps (327489 pkts in 1020385 usec)
085.645380 main_thread [1483] 328959 pps (332456 pkts in 1010630 usec)
086.654096 main_thread [1483] 313482 pps (316222 pkts in 1008740 usec)
087.671256 main_thread [1483] 307234 pps (312506 pkts in 1017161 usec)
088.673871 main_thread [1483] 311188 pps (312002 pkts in 1002615 usec)
089.686453 main_thread [1483] 322132 pps (326185 pkts in 1012582 usec)
090.694224 main_thread [1483] 331368 pps (333922 pkts in 1007708 usec)
091.714343 main_thread [1483] 290061 pps (295915 pkts in 1020181 usec)
092.724065 main_thread [1483] 316366 pps (319442 pkts in 1009722 usec)
093.749938 main_thread [1483] 322501 pps (330845 pkts in 1025874 usec)
094.758012 main_thread [1483] 292125 pps (294484 pkts in 1008074 usec)
095.765606 main_thread [1483] 338889 pps (341463 pkts in 1007594 usec)
096.781560 main_thread [1483] 327785 pps (333013 pkts in 1015949 usec)
097.789245 main_thread [1483] 330064 pps (332602 pkts in 1007690 usec)
098.794141 main_thread [1483] 328757 pps (330367 pkts in 1004896 usec)
099.804160 main_thread [1483] 302752 pps (305785 pkts in 1010019 usec)
^C100.400690 sigint_h [326] received control-C on thread 0x801806800
100.820895 main_thread [1483] 178717 pps (181708 pkts in 1016735 usec)
Received 7300802 packets, in 22.84 seconds.
Speed: 319.61 Kpps
100.822469 [ 79] vtnet_netmap_free_bufsfreed 0 mbufs, 506 netmap bufs on 1
queues
#3. The outputs of pkt-gen as the receiver when using netmap+vale:
$ ./pkt-gen -i vtnet0 -f rx
139.084350 main [1686] interface is vtnet0
139.085422 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.139.086441 [
407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
1:0
139.085705 139.090250 [ 407] vtnet_netmap_config vtnet config txq=1,
txd=512 rxq=1, rxd=512
extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0
139.094911 [ 79] vtnet_netmap_free_bufsfreed 1024 mbufs, 0 netmap bufs on
1 queues
139.098783 main [1877] mapped 334980KB at 0x801dff000
Receiving from netmap:vtnet0: 1 queues, 1 threads and 1 cpus.
139.099642 main [1963] Wait 2 secs for phy reset
141.142968 main [1965] Ready...
141.143439 nm_op141.143596 [ 407] vtnet_netmap_config vtnet config txq=1,
txd=512 rxq=1, rxd=512
en [456] overrid141.145487 [ 407] vtnet_netmap_config vtnet config txq=1,
txd=512 rxq=1, rxd=512
ing ifname vtnet0 ringid 0x0 flags 0x1
141.147778 receiver_body [1219] reading fr