Re: [PATCH] Add netmap support to bhyve

[Tiwei Bie]

On Sun, Jan 25, 2015 at 07:33:47PM -0800, Peter Grehan wrote:
> Hi,
> 
> > I saw that there is an open task of adding Netmap support to bhyve in
> > the latest status report of FreeBSD [1]. So I implement the netmap
> > support for bhyve, and following is the patch. Hope this can be helpful!
> 
>   Nice work ! Did you manage to get any throughput numbers ?

Thanks! Yeah, I have done some throughput tests. Here are the details:

#1. The parameters I used to launch the VMs:

$ sudo /home/btw/src/usr.sbin/bhyve/bhyve \
   -A -H -P \
   -s 0:0,hostbridge \
   -s 1:0,lpc \
   -s 2,virtio-blk,./disk-vm1 \
   -s 3:0,virtio-net,vale0:vm1 \
   -s 3:1,virtio-net,tap1 \
   -l com1,stdio \
   -m 2G -c 2 vm1

$ sudo /home/btw/src/usr.sbin/bhyve/bhyve \
   -A -H -P \
   -s 0:0,hostbridge \
   -s 1:0,lpc \
   -s 2,virtio-blk,./disk-vm2 \
   -s 3:0,virtio-net,vale0:vm2 \
   -s 3:1,virtio-net,tap2 \
   -l com1,stdio \
   -m 2G -c 1 vm2

#2. The outputs of pkt-gen as the receiver when using tap+bridge:

$ ./pkt-gen -i vtnet1 -f rx
075.499873 main [1686] interface is vtnet1
075.500516 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0
075.500677 extract_ip_range075.501408 [ 407] vtnet_netmap_config   vtnet 
config txq=1, txd=512 rxq=1, rxd=512
 [289] range is 075.503540 [ 407] vtnet_netmap_config   vtnet config txq=1, 
txd=512 rxq=1, rxd=512
10.1.0.1:0 to 10.1.0.1:0
075.505840 [  79] vtnet_netmap_free_bufsfreed 1024 mbufs, 0 netmap bufs on 
1 queues
075.507838 main [1877] mapped 334980KB at 0x801dff000
Receiving from netmap:vtnet1: 1 queues, 1 threads and 1 cpus.
075.509298 main [1963] Wait 2 secs for phy reset
077.553231 main [1965] Ready...
077.553787 nm_open [456] overriding ifname vtnet1 ringid 0x0 flags 0x1
077.554511 [ 407] vtnet_netmap_config   vtnet config txq=1, txd=512 rxq=1, 
rxd=512
077.556006 [ 407] vtnet_netmap_config   vtnet config txq=1, txd=512 rxq=1, 
rxd=512
077.557663 receiver_body [1219] reading from netmap:vtnet1 fd 4 main_fd 3
078.563897 main_thread [1483] 311989 pps (313942 pkts in 1006261 usec)
079.574224 main_thread [1483] 45 pps (336771 pkts in 1010278 usec)
080.584718 main_thread [1483] 307455 pps (310688 pkts in 1010514 usec)
081.600552 main_thread [1483] 332357 pps (337629 pkts in 1015862 usec)
082.604157 main_thread [1483] 342985 pps (344222 pkts in 1003606 usec)
083.614341 main_thread [1483] 327806 pps (331144 pkts in 1010184 usec)
084.634725 main_thread [1483] 320947 pps (327489 pkts in 1020385 usec)
085.645380 main_thread [1483] 328959 pps (332456 pkts in 1010630 usec)
086.654096 main_thread [1483] 313482 pps (316222 pkts in 1008740 usec)
087.671256 main_thread [1483] 307234 pps (312506 pkts in 1017161 usec)
088.673871 main_thread [1483] 311188 pps (312002 pkts in 1002615 usec)
089.686453 main_thread [1483] 322132 pps (326185 pkts in 1012582 usec)
090.694224 main_thread [1483] 331368 pps (333922 pkts in 1007708 usec)
091.714343 main_thread [1483] 290061 pps (295915 pkts in 1020181 usec)
092.724065 main_thread [1483] 316366 pps (319442 pkts in 1009722 usec)
093.749938 main_thread [1483] 322501 pps (330845 pkts in 1025874 usec)
094.758012 main_thread [1483] 292125 pps (294484 pkts in 1008074 usec)
095.765606 main_thread [1483] 338889 pps (341463 pkts in 1007594 usec)
096.781560 main_thread [1483] 327785 pps (333013 pkts in 1015949 usec)
097.789245 main_thread [1483] 330064 pps (332602 pkts in 1007690 usec)
098.794141 main_thread [1483] 328757 pps (330367 pkts in 1004896 usec)
099.804160 main_thread [1483] 302752 pps (305785 pkts in 1010019 usec)
^C100.400690 sigint_h [326] received control-C on thread 0x801806800
100.820895 main_thread [1483] 178717 pps (181708 pkts in 1016735 usec)
Received 7300802 packets, in 22.84 seconds.
Speed: 319.61 Kpps
100.822469 [  79] vtnet_netmap_free_bufsfreed 0 mbufs, 506 netmap bufs on 1 
queues

#3. The outputs of pkt-gen as the receiver when using netmap+vale:

$ ./pkt-gen -i vtnet0 -f rx
139.084350 main [1686] interface is vtnet0
139.085422 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.139.086441 [ 
407] vtnet_netmap_config   vtnet config txq=1, txd=512 rxq=1, rxd=512
1:0
139.085705 139.090250 [ 407] vtnet_netmap_config   vtnet config txq=1, 
txd=512 rxq=1, rxd=512
extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0
139.094911 [  79] vtnet_netmap_free_bufsfreed 1024 mbufs, 0 netmap bufs on 
1 queues
139.098783 main [1877] mapped 334980KB at 0x801dff000
Receiving from netmap:vtnet0: 1 queues, 1 threads and 1 cpus.
139.099642 main [1963] Wait 2 secs for phy reset
141.142968 main [1965] Ready...
141.143439 nm_op141.143596 [ 407] vtnet_netmap_config   vtnet config txq=1, 
txd=512 rxq=1, rxd=512
en [456] overrid141.145487 [ 407] vtnet_netmap_config   vtnet config txq=1, 
txd=512 rxq=1, rxd=512
ing ifname vtnet0 ringid 0x0 flags 0x1
141.147778 receiver_body [1219] reading fr

Re: [PATCH] Add netmap support to bhyve

[Peter Grehan]

Hi,


I saw that there is an open task of adding Netmap support to bhyve in
the latest status report of FreeBSD [1]. So I implement the netmap
support for bhyve, and following is the patch. Hope this can be helpful!


 Nice work ! Did you manage to get any throughput numbers ?

 We've had another submission for this as well. I'll do my best to 
merge the best bits of both into bhyve, though it will have to wait 
until some restructuring work is done to support multiple network device 
emulations.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[PATCH] Add netmap support to bhyve

[Tiwei Bie]

Hello everyone,

I saw that there is an open task of adding Netmap support to bhyve in
the latest status report of FreeBSD [1]. So I implement the netmap
support for bhyve, and following is the patch. Hope this can be helpful!

diff --git a/usr.sbin/bhyve/pci_virtio_net.c b/usr.sbin/bhyve/pci_virtio_net.c
index 5ac9ecd..d7c8e84 100644
--- a/usr.sbin/bhyve/pci_virtio_net.c
+++ b/usr.sbin/bhyve/pci_virtio_net.c
@@ -35,6 +35,10 @@ __FBSDID("$FreeBSD$");
 #include 
 #include 
 #include 
+#ifndef NETMAP_WITH_LIBS
+#define NETMAP_WITH_LIBS
+#endif
+#include 
 
 #include 
 #include 
@@ -132,6 +136,8 @@ struct pci_vtnet_softc {
struct mevent   *vsc_mevp;
 
int vsc_tapfd;
+   struct nm_desc  *vsc_nmd;
+
int vsc_rx_ready;
volatile intresetting;  /* set and checked outside lock */
 
@@ -148,6 +154,10 @@ struct pci_vtnet_softc {
pthread_mutex_t tx_mtx;
pthread_cond_t  tx_cond;
int tx_in_progress;
+
+   void (*pci_vtnet_rx)(struct pci_vtnet_softc *sc);
+   void (*pci_vtnet_tx)(struct pci_vtnet_softc *sc, struct iovec *iov,
+int iovcnt, int len);
 };
 
 static void pci_vtnet_reset(void *);
@@ -369,14 +379,208 @@ pci_vtnet_tap_rx(struct pci_vtnet_softc *sc)
vq_endchains(vq, 1);
 }
 
+static int
+pci_vtnet_netmap_writev(struct nm_desc *nmd, struct iovec *iov, int iovcnt)
+{
+   int r, i;
+   int len = 0;
+
+   for (r = nmd->cur_tx_ring; ; ) {
+   struct netmap_ring *ring = NETMAP_TXRING(nmd->nifp, r);
+   uint32_t cur, idx;
+   char *buf;
+
+   if (nm_ring_empty(ring)) {
+   r++;
+   if (r > nmd->last_tx_ring)
+   r = nmd->first_tx_ring;
+   if (r == nmd->cur_rx_ring)
+   break;
+   continue;
+   }
+   cur = ring->cur;
+   idx = ring->slot[cur].buf_idx;
+   buf = NETMAP_BUF(ring, idx);
+
+   for (i = 0; i < iovcnt; i++) {
+   memcpy(&buf[len], iov[i].iov_base, iov[i].iov_len);
+   len += iov[i].iov_len;
+   }
+   ring->slot[cur].len = len;
+   ring->head = ring->cur = nm_ring_next(ring, cur);
+   nmd->cur_tx_ring = r;
+   ioctl(nmd->fd, NIOCTXSYNC, NULL);
+   break;
+   }
+
+   return (len);
+}
+
+static inline int
+pci_vtnet_netmap_readv(struct nm_desc *nmd, struct iovec *iov, int iovcnt)
+{
+   int len = 0;
+   int i = 0;
+   int r;
+
+   for (r = nmd->cur_rx_ring; ; ) {
+   struct netmap_ring *ring = NETMAP_RXRING(nmd->nifp, r);
+   uint32_t cur, idx;
+   char *buf;
+   size_t left;
+
+   if (nm_ring_empty(ring)) {
+   r++;
+   if (r > nmd->last_rx_ring)
+   r = nmd->first_rx_ring;
+   if (r == nmd->cur_rx_ring)
+   break;
+   continue;
+   }
+   cur = ring->cur;
+   idx = ring->slot[cur].buf_idx;
+   buf = NETMAP_BUF(ring, idx);
+   left = ring->slot[cur].len;
+
+   for (i = 0; i < iovcnt && left > 0; i++) {
+   if (iov[i].iov_len > left)
+   iov[i].iov_len = left;
+   memcpy(iov[i].iov_base, &buf[len], iov[i].iov_len);
+   len += iov[i].iov_len;
+   left -= iov[i].iov_len;
+   }
+   ring->head = ring->cur = nm_ring_next(ring, cur);
+   nmd->cur_rx_ring = r;
+   ioctl(nmd->fd, NIOCRXSYNC, NULL);
+   break;
+   }
+   for (; i < iovcnt; i++)
+   iov[i].iov_len = 0;
+
+   return (len);
+}
+
+/*
+ * Called to send a buffer chain out to the vale port
+ */
 static void
-pci_vtnet_tap_callback(int fd, enum ev_type type, void *param)
+pci_vtnet_netmap_tx(struct pci_vtnet_softc *sc, struct iovec *iov, int iovcnt,
+   int len)
+{
+   static char pad[60]; /* all zero bytes */
+
+   if (sc->vsc_nmd == NULL)
+   return;
+
+   /*
+* If the length is < 60, pad out to that and add the
+* extra zero'd segment to the iov. It is guaranteed that
+* there is always an extra iov available by the caller.
+*/
+   if (len < 60) {
+   iov[iovcnt].iov_base = pad;
+   iov[iovcnt].iov_len = 60 - len;
+   iovcnt++;
+   }
+   (void) pci_vtnet_netmap_writev(sc->vsc_nmd, iov, iovcnt);
+}
+
+static void
+pci_vtnet_netmap_rx(struct pci_vtnet_softc *sc)
+{
+   struct iovec iov[VTNET_MAXSEGS], *riov;
+   struct vqueue_info *vq;
+   void *vrx;
+