Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

2010-10-07 Thread Chris Buechler
On Thu, Oct 7, 2010 at 3:43 PM, Eugen Leitl eu...@leitl.org wrote:
 On Sat, Oct 02, 2010 at 03:53:54PM -0400, Chris Buechler wrote:

 That's not the normal experience from what I've seen, sounds specific
 to something in particular you're doing. I believe every environment
 I've seen that routes between VLANs within ESX handles the VLANs
 entirely at the ESX level, with one vswitch per VLAN and the firewall
 connected to the individual vswitches, maybe that's the difference.

 Running inside of VMware isn't nearly as fast as running on equivalent
 bare metal, but most of the time you don't need that kind of
 performance, 300 Mbps is easily achievable with e1000 NICs and
 moderately new (anything with VT) server hardware. I've been on dozens

 Chris, how much memory do you recommend for a pfSense ESXi instance,
 which handles 4 guests (one IP address each), 100 MBit/s switched
 setup? Do I need 1+ GByte, or can I risk allocating just 512
 MBytes to the guest?


It depends. Virtual sizing no diff from physical. Depends on
simultaneous connections, what packages and configurations they use,
etc. I use 128 MB RAM and 2 GB disks on most of my test and dev boxes,
they're mostly pretty basic though.

-
To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
For additional commands, e-mail: discussion-h...@pfsense.com

Commercial support available - https://portal.pfsense.org



RE: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

2010-10-07 Thread Greg Hennessy
If I may add one thought to this, 

Chokepoint have recently announced a virtual version of their 'blade' product 
which uses the VMSafe API to enable more efficient inspection of traffic 
travelling between virtual machines and the outside world. 

http://www.networkworld.com/news/2010/090110-check-point-vmware-security.html?hpg1=bn

Dunno what the possibilty of such an approach is with pfSense. 

Given the innards of VMWare is linux based, the ABI is likely to be interesting 
for other operating systems to interface against. 



Greg



From: Chris Buechler [cbuech...@gmail.com]
Sent: 07 October 2010 15:32
To: discussion@pfsense.com
Subject: Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi 
guest for other guests

On Thu, Oct 7, 2010 at 3:43 PM, Eugen Leitl eu...@leitl.org wrote:
 On Sat, Oct 02, 2010 at 03:53:54PM -0400, Chris Buechler wrote:

 That's not the normal experience from what I've seen, sounds specific
 to something in particular you're doing. I believe every environment
 I've seen that routes between VLANs within ESX handles the VLANs
 entirely at the ESX level, with one vswitch per VLAN and the firewall
 connected to the individual vswitches, maybe that's the difference.

 Running inside of VMware isn't nearly as fast as running on equivalent
 bare metal, but most of the time you don't need that kind of
 performance, 300 Mbps is easily achievable with e1000 NICs and
 moderately new (anything with VT) server hardware. I've been on dozens

 Chris, how much memory do you recommend for a pfSense ESXi instance,
 which handles 4 guests (one IP address each), 100 MBit/s switched
 setup? Do I need 1+ GByte, or can I risk allocating just 512
 MBytes to the guest?


It depends. Virtual sizing no diff from physical. Depends on
simultaneous connections, what packages and configurations they use,
etc. I use 128 MB RAM and 2 GB disks on most of my test and dev boxes,
they're mostly pretty basic though.

-
To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
For additional commands, e-mail: discussion-h...@pfsense.com

Commercial support available - https://portal.pfsense.org
-
To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
For additional commands, e-mail: discussion-h...@pfsense.com

Commercial support available - https://portal.pfsense.org



RE: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

2010-10-02 Thread Adam Thompson
It works, but performance is, in my experience, poor.  Don't use trunking 
(802.3ad / LACP) and VLANs together, or inter-vlan routing slows down 
drastically.  This appears to be a VMWare problem, not a pfSense problem. 
I recommend creating one virtual Ethernet device per network, and in fact 
mapping each virtual switch (or vlan) to a physical NIC on the host.
Basically, keep the networking as simple as possible, don't get fancy like 
I did.
-Adam Thompson
 athom...@athompso.net

 -Original Message-
 From: Eugen Leitl [mailto:eu...@leitl.org]
 Sent: Saturday, October 02, 2010 05:20
 To: discussion@pfsense.com
 Subject: [pfSense-discussion] pfSense router/firewall in a Vmware
 ESXi guest for other guests


 A customer needs to run VMWare instances on the cheap, so naturally
 I thought
 about http://wiki.hetzner.de/index.php/VMware_ESXi_english

 ESXi can't route by itself though, so I thought about putting
 pfSense into one VMWare guest instance, and use that for a router/
 firewall for the other guests.

 Anyone here doing that? Works well? Care to share details of
 your setup?

 --
 Eugen* Leitl a href=http://leitl.org;leitl/a http://leitl.org
 __
 ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
 8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE

 ---
 --
 To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
 For additional commands, e-mail: discussion-h...@pfsense.com

 Commercial support available - https://portal.pfsense.org




-
To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
For additional commands, e-mail: discussion-h...@pfsense.com

Commercial support available - https://portal.pfsense.org



Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

2010-10-02 Thread Scott Ullrich
On Sat, Oct 2, 2010 at 2:27 PM, Adam Thompson athom...@c3a.ca wrote:
 It works, but performance is, in my experience, poor.  Don't use trunking
 (802.3ad / LACP) and VLANs together, or inter-vlan routing slows down
 drastically.  This appears to be a VMWare problem, not a pfSense problem.
 I recommend creating one virtual Ethernet device per network, and in fact
 mapping each virtual switch (or vlan) to a physical NIC on the host.
 Basically, keep the networking as simple as possible, don't get fancy like
 I did.

Was this with 4.0 or 4.1?   4.1 seems to drastically improved across
the board in terms of I/O in general.

Scott

-
To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
For additional commands, e-mail: discussion-h...@pfsense.com

Commercial support available - https://portal.pfsense.org



RE: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

2010-10-02 Thread Adam Thompson
This started with 4.0, I have upgraded to 4.1 but haven't specifically 
tested performance since.  Routing from one VLAN to another entirely 
inside VMware is still slow, however.  AFAIK this is somehow related to 
interrupt handling and/or mitigation.  The bad news is that since 
upgrading to 4.1, the pfSense guest occasionally loses ALL network 
interrupts for about 15 minutes at a time - this happens at least once or 
twice a week.  It starts slowly, performance is merely degraded, then 
nothing, then slowly returns to normal - whole event takes ~15min.

Traffic arriving at or leaving the VMWare HOST shows normal performance 
levels, it's only traffic within the host that seems slow: SMB traffic 
across the pfSense router, no NAT involved, one pass-all pf rule, runs 
between 10Mbit/sec and 100Mbit/sec.  I also see lots of TCP badness if I 
run a sniffer on either end - dup acks, dup pkts, and missing packets.

I also have a lot (~7Mbyte/sec) of multicast traffic on one of the VLANs, 
which may contribute to the problem.

-Adam


 -Original Message-
 From: Scott Ullrich [mailto:sullr...@gmail.com]
 Sent: Saturday, October 02, 2010 13:37
 To: discussion@pfsense.com
 Subject: Re: [pfSense-discussion] pfSense router/firewall in a
 Vmware ESXi guest for other guests

 On Sat, Oct 2, 2010 at 2:27 PM, Adam Thompson athom...@c3a.ca
 wrote:
  It works, but performance is, in my experience, poor.  Don't use
 trunking
  (802.3ad / LACP) and VLANs together, or inter-vlan routing slows
 down
  drastically.  This appears to be a VMWare problem, not a pfSense
 problem.
  I recommend creating one virtual Ethernet device per network, and
 in fact
  mapping each virtual switch (or vlan) to a physical NIC on the
 host.
  Basically, keep the networking as simple as possible, don't get
 fancy like
  I did.

 Was this with 4.0 or 4.1?   4.1 seems to drastically improved
 across
 the board in terms of I/O in general.

 Scott

 ---
 --
 To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
 For additional commands, e-mail: discussion-h...@pfsense.com

 Commercial support available - https://portal.pfsense.org




-
To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
For additional commands, e-mail: discussion-h...@pfsense.com

Commercial support available - https://portal.pfsense.org



Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

2010-10-02 Thread Tim Dressel
Hi folks,

I did this for about 6 months to do evaluations of Exchange 2010 and Zimbra.

My cluster had two VM hosts, each with 6 nics (2 onboard used for heartbeat,
and an an in Intel PCIe quad port). I defined a LAN (vswitch) internal to
the cluster only for traffic between all the VM's and the Lan side of the
pfsense box. I also added one port from each of the VM hosts and connected
to an external switch VLAN which was then directly connected to the
internet. DRS and HA worked flawlessly.

This worked exceptionally well for the pfsense box. The VM hosts were dual
processor dual core P4 Xeon's at 3.0Ghz. The internet connection was 100Mbit
and I was easily able to get 80+Mbit across it. CPU use on the VM was never
more than 20% of the single vCPU I assigned to it. In the 6 months we had it
running it never burped once. It performed exactly like a hardware box. I
did not install the VMware tools on pfsense.

I would not recommend this for a production scenario though, there are too
many unknowns about the footprint that vmware might expose. Especially
seeing any only computer will run pfsense very well if all you need is basic
routing and NAT'ing.

This was on VMware ESXi 4.0 hosts, with a single vSphere manager.

We are currently playing with vyatta to do some really neat routing
simulations for our larger network which is all cisco at the routing layer.
We have several VRF's defined in our cisco's and have been playing with the
open source patches to add this to the vyatta project that have not yet been
integrated. For us, if we can prove this is stable in vmware, we will
consider moving to hardware vyatta boxen.

Good luck!

Tim


Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

2010-10-02 Thread Chris Buechler
On Sat, Oct 2, 2010 at 2:44 PM, Adam Thompson athom...@c3a.ca wrote:
 This started with 4.0, I have upgraded to 4.1 but haven't specifically
 tested performance since.  Routing from one VLAN to another entirely
 inside VMware is still slow, however.  AFAIK this is somehow related to
 interrupt handling and/or mitigation.  The bad news is that since
 upgrading to 4.1, the pfSense guest occasionally loses ALL network
 interrupts for about 15 minutes at a time - this happens at least once or
 twice a week.  It starts slowly, performance is merely degraded, then
 nothing, then slowly returns to normal - whole event takes ~15min.

 Traffic arriving at or leaving the VMWare HOST shows normal performance
 levels, it's only traffic within the host that seems slow: SMB traffic
 across the pfSense router, no NAT involved, one pass-all pf rule, runs
 between 10Mbit/sec and 100Mbit/sec.  I also see lots of TCP badness if I
 run a sniffer on either end - dup acks, dup pkts, and missing packets.


That's not the normal experience from what I've seen, sounds specific
to something in particular you're doing. I believe every environment
I've seen that routes between VLANs within ESX handles the VLANs
entirely at the ESX level, with one vswitch per VLAN and the firewall
connected to the individual vswitches, maybe that's the difference.

Running inside of VMware isn't nearly as fast as running on equivalent
bare metal, but most of the time you don't need that kind of
performance, 300 Mbps is easily achievable with e1000 NICs and
moderately new (anything with VT) server hardware. I've been on dozens
of such systems personally this year alone, across numerous different
customer environments. It's a common setup, and works well including
for routing between VLANs. I know at least a couple setups that route
backups between VLANs, maxes out the system at a bit over 300 Mbps,
but runs fine every night and the resulting performance degradation
for the other interfaces while the firewall VM is pegged isn't an
issue in that environment (everything else still works fine). We have
customers who run their entire colo environments in vSphere including
firewalls, setting the edge CARP pair so the two never get vmotioned
to the same host for proper redundancy.

To answer the original question, there are numerous environments
running that way with great results. Very solid performance and
reliability. ESX and ESXi are equivalent, any mentions of ESX here
could be ESXi just the same (and many of the environments I'm
referring to are ESXi).

-
To unsubscribe, e-mail: discussion-unsubscr...@pfsense.com
For additional commands, e-mail: discussion-h...@pfsense.com

Commercial support available - https://portal.pfsense.org