Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests
On Sat, Oct 2, 2010 at 2:44 PM, Adam Thompson wrote: > This started with 4.0, I have upgraded to 4.1 but haven't specifically > tested performance since. Routing from one VLAN to another entirely > inside VMware is still slow, however. AFAIK this is somehow related to > interrupt handling and/or mitigation. The bad news is that since > upgrading to 4.1, the pfSense guest occasionally loses ALL network > interrupts for about 15 minutes at a time - this happens at least once or > twice a week. It starts slowly, performance is merely degraded, then > nothing, then slowly returns to normal - whole event takes ~15min. > > Traffic arriving at or leaving the VMWare HOST shows normal performance > levels, it's only traffic within the host that seems slow: SMB traffic > across the pfSense router, no NAT involved, one pass-all pf rule, runs > between 10Mbit/sec and 100Mbit/sec. I also see lots of TCP badness if I > run a sniffer on either end - dup acks, dup pkts, and missing packets. > That's not the normal experience from what I've seen, sounds specific to something in particular you're doing. I believe every environment I've seen that routes between VLANs within ESX handles the VLANs entirely at the ESX level, with one vswitch per VLAN and the firewall connected to the individual vswitches, maybe that's the difference. Running inside of VMware isn't nearly as fast as running on equivalent bare metal, but most of the time you don't need that kind of performance, 300 Mbps is easily achievable with e1000 NICs and moderately new (anything with VT) server hardware. I've been on dozens of such systems personally this year alone, across numerous different customer environments. It's a common setup, and works well including for routing between VLANs. I know at least a couple setups that route backups between VLANs, maxes out the system at a bit over 300 Mbps, but runs fine every night and the resulting performance degradation for the other interfaces while the firewall VM is pegged isn't an issue in that environment (everything else still works fine). We have customers who run their entire colo environments in vSphere including firewalls, setting the edge CARP pair so the two never get vmotioned to the same host for proper redundancy. To answer the original question, there are numerous environments running that way with great results. Very solid performance and reliability. ESX and ESXi are equivalent, any mentions of ESX here could be ESXi just the same (and many of the environments I'm referring to are ESXi). - To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com For additional commands, e-mail: discussion-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests
Hi folks, I did this for about 6 months to do evaluations of Exchange 2010 and Zimbra. My cluster had two VM hosts, each with 6 nics (2 onboard used for heartbeat, and an an in Intel PCIe quad port). I defined a LAN (vswitch) internal to the cluster only for traffic between all the VM's and the Lan side of the pfsense box. I also added one port from each of the VM hosts and connected to an external switch VLAN which was then directly connected to the internet. DRS and HA worked flawlessly. This worked exceptionally well for the pfsense box. The VM hosts were dual processor dual core P4 Xeon's at 3.0Ghz. The internet connection was 100Mbit and I was easily able to get 80+Mbit across it. CPU use on the VM was never more than 20% of the single vCPU I assigned to it. In the 6 months we had it running it never burped once. It performed exactly like a hardware box. I did not install the VMware tools on pfsense. I would not recommend this for a production scenario though, there are too many unknowns about the footprint that vmware might expose. Especially seeing any only computer will run pfsense very well if all you need is basic routing and NAT'ing. This was on VMware ESXi 4.0 hosts, with a single vSphere manager. We are currently playing with vyatta to do some really neat routing simulations for our larger network which is all cisco at the routing layer. We have several VRF's defined in our cisco's and have been playing with the open source patches to add this to the vyatta project that have not yet been integrated. For us, if we can prove this is stable in vmware, we will consider moving to hardware vyatta boxen. Good luck! Tim
RE: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests
This started with 4.0, I have upgraded to 4.1 but haven't specifically tested performance since. Routing from one VLAN to another entirely inside VMware is still slow, however. AFAIK this is somehow related to interrupt handling and/or mitigation. The bad news is that since upgrading to 4.1, the pfSense guest occasionally loses ALL network interrupts for about 15 minutes at a time - this happens at least once or twice a week. It starts slowly, performance is merely degraded, then nothing, then slowly returns to normal - whole event takes ~15min. Traffic arriving at or leaving the VMWare HOST shows normal performance levels, it's only traffic within the host that seems slow: SMB traffic across the pfSense router, no NAT involved, one pass-all pf rule, runs between 10Mbit/sec and 100Mbit/sec. I also see lots of TCP badness if I run a sniffer on either end - dup acks, dup pkts, and missing packets. I also have a lot (~7Mbyte/sec) of multicast traffic on one of the VLANs, which may contribute to the problem. -Adam > -Original Message- > From: Scott Ullrich [mailto:sullr...@gmail.com] > Sent: Saturday, October 02, 2010 13:37 > To: discussion@pfsense.com > Subject: Re: [pfSense-discussion] pfSense router/firewall in a > Vmware ESXi guest for other guests > > On Sat, Oct 2, 2010 at 2:27 PM, Adam Thompson > wrote: > > It works, but performance is, in my experience, poor. Don't use > trunking > > (802.3ad / LACP) and VLANs together, or inter-vlan routing slows > down > > drastically. This appears to be a VMWare problem, not a pfSense > problem. > > I recommend creating one virtual Ethernet device per network, and > in fact > > mapping each virtual switch (or vlan) to a physical NIC on the > host. > > Basically, keep the networking as simple as possible, don't get > fancy like > > I did. > > Was this with 4.0 or 4.1? 4.1 seems to drastically improved > across > the board in terms of I/O in general. > > Scott > > --- > -- > To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com > For additional commands, e-mail: discussion-h...@pfsense.com > > Commercial support available - https://portal.pfsense.org - To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com For additional commands, e-mail: discussion-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests
On Sat, Oct 2, 2010 at 2:27 PM, Adam Thompson wrote: > It works, but performance is, in my experience, poor. Don't use trunking > (802.3ad / LACP) and VLANs together, or inter-vlan routing slows down > drastically. This appears to be a VMWare problem, not a pfSense problem. > I recommend creating one virtual Ethernet device per network, and in fact > mapping each virtual switch (or vlan) to a physical NIC on the host. > Basically, keep the networking as simple as possible, don't get fancy like > I did. Was this with 4.0 or 4.1? 4.1 seems to drastically improved across the board in terms of I/O in general. Scott - To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com For additional commands, e-mail: discussion-h...@pfsense.com Commercial support available - https://portal.pfsense.org
RE: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests
It works, but performance is, in my experience, poor. Don't use trunking (802.3ad / LACP) and VLANs together, or inter-vlan routing slows down drastically. This appears to be a VMWare problem, not a pfSense problem. I recommend creating one virtual Ethernet device per network, and in fact mapping each virtual switch (or vlan) to a physical NIC on the host. Basically, keep the networking as simple as possible, don't get fancy like I did. -Adam Thompson athom...@athompso.net > -Original Message- > From: Eugen Leitl [mailto:eu...@leitl.org] > Sent: Saturday, October 02, 2010 05:20 > To: discussion@pfsense.com > Subject: [pfSense-discussion] pfSense router/firewall in a Vmware > ESXi guest for other guests > > > A customer needs to run VMWare instances on the cheap, so naturally > I thought > about http://wiki.hetzner.de/index.php/VMware_ESXi_english > > ESXi can't route by itself though, so I thought about putting > pfSense into one VMWare guest instance, and use that for a router/ > firewall for the other guests. > > Anyone here doing that? Works well? Care to share details of > your setup? > > -- > Eugen* Leitl http://leitl.org";>leitl http://leitl.org > __ > ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org > 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE > > --- > -- > To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com > For additional commands, e-mail: discussion-h...@pfsense.com > > Commercial support available - https://portal.pfsense.org - To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com For additional commands, e-mail: discussion-h...@pfsense.com Commercial support available - https://portal.pfsense.org
[pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests
A customer needs to run VMWare instances on the cheap, so naturally I thought about http://wiki.hetzner.de/index.php/VMware_ESXi_english ESXi can't route by itself though, so I thought about putting pfSense into one VMWare guest instance, and use that for a router/ firewall for the other guests. Anyone here doing that? Works well? Care to share details of your setup? -- Eugen* Leitl http://leitl.org";>leitl http://leitl.org __ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE - To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com For additional commands, e-mail: discussion-h...@pfsense.com Commercial support available - https://portal.pfsense.org
