On 2016-03-01 2:57 pm, Nikos Vassiliadis wrote:

On 03/01/16 18:43, dweimer wrote:
I am considering setting up a bhyve virtual machine to run pfSense. Not too thrilled with the CPU heat on the PC Engines APU1D4 when under heavy
load, but don't want to rely entirely on a VM. As I like still having
internet if I would have to take my server offline for disk replacement
or other issues, having web access to search for errors is a big plus.
So in order to avoid spending money on a new piece of hardware I thought
why not do a VM with CARP fail over to the physical. I am not finding
much searching on FreeBSD byhve and CARP, I know its somewhat of an
issue withing VMware on ESX making sure you enable the right options on
the virtual switches and interfaces.

Enable promiscuous mode on the vSwitch
Enable "MAC Address changes"
Enable "Forged transmits"

Before I got started on the setup I was curious if anyone has done
something similar, or know if this isn't possible on bhyve at the
current version? I am running my system currently on 10.3-BETA3.

I am running two postgres VMs with DRBD and not CARP but UCARP which
should be 100% compatible with CARP. Each VM has a tap interface and
each tap is bridged to a bridge interface. There is no need for special
configuration. Everything works as expected.

Well so far I have it mostly working, one issue though, that I can't quite find the source of the problem. I have multiple port forwards setup and use NAT reflection to make those accessible from the same host name internally and externally. I am redirecting ports 80, 443, 7443, and 8443 among others on of the virtual carp IP addresses. 80 and 443 are redirected to my proxy jail running Squid as a reverse proxy, jail is on same host as bhyve. 7443 redirects to Ubiquiti UniFi Video server for HTTPS running on another bhyve Linux virtual machine. 8443 redirects to Ubiquiti UniFi Wireless controller for HTTPS on another jail on the same host as the bhyve virtual machines.

Everything that is running with NAT reflection works except for the port 443 traffic from the bhyve host machine, any jails running on it, and the other bhyve virtual machine. However it works fine from other network clients. Of course the NAT reflection is so that the same certificate can be used on all the HTTPS connections and show as valid.

As near as I can tell the initial request makes it through the pfSense, to the Proxy. The Proxy's response makes it back to the pfSense. The pfSense system sends it to the client, but the client doesn't acknowledge that it received it. I have used tcpdump on the system to verify that it does receive the packets.

I initially suspected something with the HTTPS was rejecting the virtual IPs used with CARP but that doesn't explain why it works on the other HTTPS ports. And failing over to the old physical APU1D4 it all works. As well as it working from other clients.

I plan to add a second HTTPS port to the squid reverse proxy configuration to see if its isolated to the port 443 or if its isolated to the HTTPS on squid. I will also try redirecting straight to the Apache jail that the Proxy forwards to, Squid is only used as a reverse proxy on this setup so that I can test Squid updates here before installing them on the Reverse proxy I maintain at work.

   Dean E. Weimer
freebsd-virtualization@freebsd.org mailing list
To unsubscribe, send any mail to 

Reply via email to