Thanks to btw's patch I can now use bhyve and vale.
But I'm hitting a limitation:
I want to start 5 VM connected by a full-meshed network.
This mean:
- VM 1 has 4 interfaces toward each other 4 VMs
- and so one for all others 4 VMs

This give these 5 command lines:

bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \
-s 2:0,virtio-net,vale12:BSDRP_1,mac=58:9c:fc:01:02:01 \
-s 2:1,virtio-net,vale13:BSDRP_1,mac=58:9c:fc:01:03:01 \
-s 2:2,virtio-net,vale14:BSDRP_1,mac=58:9c:fc:01:04:01 \
-s 2:3,virtio-net,vale15:BSDRP_1,mac=58:9c:fc:01:05:01 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_1 \
-l com1,/dev/nmdm1A BSDRP_1

bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \
-s 2:0,virtio-net,vale15:BSDRP_5,mac=58:9c:fc:01:05:05 \
-s 2:1,virtio-net,vale25:BSDRP_5,mac=58:9c:fc:02:05:05 \
-s 2:2,virtio-net,vale35:BSDRP_5,mac=58:9c:fc:03:05:05 \
-s 2:3,virtio-net,vale45:BSDRP_5,mac=58:9c:fc:04:05:05 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_5 \
-l com1,/dev/nmdm5A BSDRP_5

bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \
-s 2:0,virtio-net,vale12:BSDRP_2,mac=58:9c:fc:01:02:02 \
-s 2:1,virtio-net,vale23:BSDRP_2,mac=58:9c:fc:02:03:02 \
-s 2:2,virtio-net,vale24:BSDRP_2,mac=58:9c:fc:02:04:02 \
-s 2:3,virtio-net,vale25:BSDRP_2,mac=58:9c:fc:02:05:02 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_2 \
-l com1,/dev/nmdm2A BSDRP_2

bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc
-s 2:0,virtio-net,vale13:BSDRP_3,mac=58:9c:fc:01:03:03 \
-s 2:1,virtio-net,vale23:BSDRP_3,mac=58:9c:fc:02:03:03 \
-s 2:2,virtio-net,vale34:BSDRP_3,mac=58:9c:fc:03:04:03 \
-s 2:3,virtio-net,vale35:BSDRP_3,mac=58:9c:fc:03:05:03 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_3 \
-l com1,/dev/nmdm3A BSDRP_3

bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc
-s 2:0,virtio-net,vale14:BSDRP_4,mac=58:9c:fc:01:04:04 \
-s 2:1,virtio-net,vale24:BSDRP_4,mac=58:9c:fc:02:04:04 \
-s 2:2,virtio-net,vale34:BSDRP_4,mac=58:9c:fc:03:04:04 \
-s 2:3,virtio-net,vale45:BSDRP_4,mac=58:9c:fc:04:05:04 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_4
-l com1,/dev/nmdm4A BSDRP_4

But host system complains it didn't reach to allocate memory,:

743.527464 nm_open [608] NIOCREGIF failed: Cannot allocate memory
743.527470 nm_open [608] NIOCREGIF failed: Cannot allocate memory
open of netmap device vale35:BSDRP_5 failed
open of netmap device vale45:BSDRP_4 failed
743.561324 nm_open [608] NIOCREGIF failed: Cannot allocate memory
open of netmap device vale45:BSDRP_5 failed
743.561359 nm_open [608] NIOCREGIF failed: Cannot allocate memory
open of netmap device vale35:BSDRP_3 failed

And the vtnet interfaces that failed to allocate memory are correctly in
"status: no carrier mode" in the guests.

But the host system has plenty of free RAM:

root@lame5:~ # top -b
last pid:  2242;  load averages:  0.09,  0.24,  0.22  up 0+04:24:44
64 processes:  1 running, 63 sleeping

Mem: 556M Active, 94M Inact, 5645M Wired, 243G Free
ARC: 1273M Total, 461M MFU, 805M MRU, 16K Anon, 3117K Header, 3332K Other
Swap: 4096M Total, 4096M Free

Can I increase some sysctl dev.netmap for allowing more memory to be used
by netmap/vale ?
I've tried to multiply per 10 dev.netmap.buf_num, but no changes.

freebsd-virtualization@freebsd.org mailing list
To unsubscribe, send any mail to 

Reply via email to