Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[laurent.cli...@gmail.com]

Le 12/05/2011 20:22, Tobias P. Santos a écrit :
> laurent.cli...@gmail.com wrote:
>> Le 12/05/2011 17:02, Tobias P. Santos a écrit :
>>  
>>> Try this:
>>>
>>> ifconfig xn1 -txcsum
>>>
>>> Best regards,
>>> Tobias.
>>> 
>> Hello Tobias,
>>
>> Unfortunately ifconfig xn1 -txcsum didn't give me lot of result.
>> Doing this my network perf through NAT is about 7KiB/s (that is better
>> than before, but still unworkable).
>>
>> Thanks for the idea, maybe we are not that far away from the solution.
>>
>>   
> We had a problem where traffic between VMs was too slow (about 5Mbps
> on a Gigabit network) and disabling TX checksum solved the problem
> (actually, speed increased to more than 1Gbps).
> Your problem is different as you have a nice throughput between VMs on
> xn1, sorry that I missed that on my first message.
>
> Did you try to transfer data from something connected to xn0 to one of
> yours DomU servers connected to xn1 without NAT?
> You also mentioned pf. Did you make any test using natd or anything else?
> Something tells me that's not a nic problem, but that's just a guess.
I already tried setup with natd, as I found on the list archives that PF
can pose problems with XENHVM kernels, but the network perfomance was
the same.
>
> Anyway, on the coming weeks I will have to set up a similar
> environment to yours, so maybe I'll face the same problem (although we
> use natd daemon instead of pf).
>
> Good luck!
I think I will keep my Linux PV VM to do NAT, as it is the only thing on
my Xen setup that cause problems for now. With that I will still be able
to work with FreeBSD for the backend servers.
>
> ___
> freebsd-xen@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[laurent.cli...@gmail.com]

Le 12/05/2011 17:48, Janne Snabb a écrit :
> On Thu, 12 May 2011, Laurent Cligny wrote:
>
>> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
>> Paravirtualized Debian amd64.
> My suggestion would be to try out the same setup with GENERIC kernel
> and the rtl driver (or even better e1000 if your Xen allows it)
> which is easy to do to make a simple comparison.
I tried it following your idea. I compiled GENERIC kernel on both NAT Vm
and Vm behind the NAT, the network perf increased a lot as it is now
about 15Mb/s trough NAT and between the VM on the private network. But
doing this also eat a lot of CPU on "Sys" context (almost 50% on SMP).
So I think it is not a suitable setup for me.
> In one of my recent benchmarks the FreeBSD Xen PV network driver
> performed very well in one direction, but very badly in the other.
> Using rtl or e1000 gave much better TCP throughput if you care
> equally about both directions. This benchmark was done only for
> internal TCP traffic between a Linux dom0 and FreeBSD dumU, the
> traffic never entered a real NIC.
In fact I have great troughput between VM on private NIC with XENHVM
kernel (50 to 70 Mb/s) and also from NAT VM to Internet on public NIC.
The problem appears when I generate traffic from VMs on private network
to and from the Internet trough the NAT VM.
> This might or might not help in your situation...
It has conforted me into the idea that XENHVM kernel doesn't work well
with FreeBSD NAT solutions or with the underlying Xen Cloud Platform Dom0.
> --
> Janne Snabb / EPIPE Communications
> sn...@epipe.com - http://epipe.com/
> ___
> freebsd-xen@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[laurent.cli...@gmail.com]

Le 12/05/2011 20:20, Guillaume Seigneuret a écrit :
> I think you should try totally disable the window sizing both parts to
> isolate it doesn't come from here :
> On your FreeBSD VM : sysctl net.inet.tcp.rfc1323=0
> On your Linux box : sysctl net.ipv4.tcp_window_scaling=0
I set net.inet.tcp.rfc1323=0 to VM Nic and the two NAT VM Nics, but it
changed nothing on the network perf.
>
> Then if the rate is hugely better, try some settings to increase the
> compatibility.
> You can also try an *iperf *test with TCP and UDP packets to see if
> the behavior is generalized, and see what latency you have both part
> of the network.
Iperf is hanging on the test since there is almost not network
troughput. I waited 30min before I cancelled it.
>
> Cordialement,
>
> *Guillaume Seigneuret* 
>
>
>
> **Network and System Security Architect*
> *Mobile *:  +33.6.15.37.35.67
> *Web* :  http://www.omegacube.fr
> *Address* :*
> Hôtel Technologique Marseille Innovation - BP 100
> 45 rue Frederic Joliot Curie
> Technopôle de Château Gombert
> 13382 Marseille Cedex 13
>
>
>
> 2011/5/12 Michael MacLeod  >
>
> Try -rxcsum as well. I have both disabled on my domU.
>
> On Thu, May 12, 2011 at 12:53 PM, laurent.cli...@gmail.com
>  <
> laurent.cli...@gmail.com > wrote:
>
> > Le 12/05/2011 17:02, Tobias P. Santos a écrit :
> > > Try this:
> > >
> > > ifconfig xn1 -txcsum
> > >
> > > Best regards,
> > > Tobias.
> > Hello Tobias,
> >
> > Unfortunately ifconfig xn1 -txcsum didn't give me lot of result.
> > Doing this my network perf through NAT is about 7KiB/s (that is
> better
> > than before, but still unworkable).
> >
> > Thanks for the idea, maybe we are not that far away from the
> solution.
> >
> > >
> > >
> > > Laurent Cligny wrote:
> > >> Hello all,
> > >>
> > >> I want to setup a FreeBSD DomU VM as a network NAT gateway in
> order to
> > >> provide Internet access to other FreeBSD and Linux DomU VMs.
> > >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210
> with 8 cores,
> > >> 16GiB RAM and one NIC.
> > >>
> > >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux
> VM is a
> > >> Paravirtualized Debian amd64.
> > >>
> > >> The NAT gateway is a two-NIC box, with one (xn0) configured
> with a
> > >> public Internet address and the other (xn1) configured with a
> class A
> > >> private address (10.0.0.254).
> > >> The Internet traffic is very good from my FreeBSD NAT gateway
> to the
> > >> Internet (~ 50MiB/s) and also between other VMs on the
> private network
> > >> (~ 70MiB/s trough scp) after applying the patch here
> > >>
> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
> > >> fixing the "too many frags" problem.
> > >>
> > >> Here is a schema of my setup with the observed net speed on it:
> > >>
> > >> -- --- --
> > >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
> > >> -- --- --
> > >>   <--->   <--->
> > >>50MiB/s 70MiB/s
> > >>
> > >>   <--->
> > >>0KiB/s  > >>
> > >> [...]
> > > ___
> > > freebsd-xen@freebsd.org 
> mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> > > To unsubscribe, send any mail to
> "freebsd-xen-unsubscr...@freebsd.org
> "
> >
> > ___
> > freebsd-xen@freebsd.org  mailing
> list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> > To unsubscribe, send any mail to
> "freebsd-xen-unsubscr...@freebsd.org
> "
> >
> ___
> freebsd-xen@freebsd.org  mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to
> "freebsd-xen-unsubscr...@freebsd.org
> "
>
>

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[laurent.cli...@gmail.com]

Le 12/05/2011 19:36, Michael MacLeod a écrit :
> Try -rxcsum as well. I have both disabled on my domU.
I tried with -rxcsum as you mentionned, but unfortunately it changed
nothing on the performance.
>
> On Thu, May 12, 2011 at 12:53 PM, laurent.cli...@gmail.com
>   > wrote:
>
> Le 12/05/2011 17:02, Tobias P. Santos a écrit :
> > Try this:
> >
> > ifconfig xn1 -txcsum
> >
> > Best regards,
> > Tobias.
> Hello Tobias,
>
> Unfortunately ifconfig xn1 -txcsum didn't give me lot of result.
> Doing this my network perf through NAT is about 7KiB/s (that is better
> than before, but still unworkable).
>
> Thanks for the idea, maybe we are not that far away from the solution.
>
> >
> >
> > Laurent Cligny wrote:
> >> Hello all,
> >>
> >> I want to setup a FreeBSD DomU VM as a network NAT gateway in
> order to
> >> provide Internet access to other FreeBSD and Linux DomU VMs.
> >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with
> 8 cores,
> >> 16GiB RAM and one NIC.
> >>
> >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux
> VM is a
> >> Paravirtualized Debian amd64.
> >>
> >> The NAT gateway is a two-NIC box, with one (xn0) configured with a
> >> public Internet address and the other (xn1) configured with a
> class A
> >> private address (10.0.0.254).
> >> The Internet traffic is very good from my FreeBSD NAT gateway
> to the
> >> Internet (~ 50MiB/s) and also between other VMs on the private
> network
> >> (~ 70MiB/s trough scp) after applying the patch here
> >> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
> >> fixing the "too many frags" problem.
> >>
> >> Here is a schema of my setup with the observed net speed on it:
> >>
> >> -- --- --
> >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
> >> -- --- --
> >>   <--->   <--->
> >>50MiB/s 70MiB/s
> >>
> >>   <--->
> >>0KiB/s  >>
> >> [...]
> > ___
> > freebsd-xen@freebsd.org  mailing
> list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> > To unsubscribe, send any mail to
> "freebsd-xen-unsubscr...@freebsd.org
> "
>
> ___
> freebsd-xen@freebsd.org  mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to
> "freebsd-xen-unsubscr...@freebsd.org
> "
>
>

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Janne Snabb]

On Thu, 12 May 2011, Justin T. Gibbs wrote:

> Do you recall which path was slow (rx or tx from the perspective of
> the FreeBSD driver) and what the relative difference in performance
> was between the two approaches?

No, which is really stupid (that is why I haven't published these
numbers before). The numbers are as follows, for simple iperf TCP
test, with all networking, kernel, etc. settings on default values:

input (Mbit/s)  output (Mbit/s)
Xen FreeBSD (rtl) 724 200
Xen FreeBSD (xn)   441700
Xen Linux (xn)   85702340
KVM FreeBSD (e1000)   475 495
KVM FreeBSD (rtl)1100  85
KVM Linux (e1000) 785 890
KVM Linux (virtio)585 715
dom0 <-> dom0   18500   18500

The table above will probably not render correctly, but hopefully
it is somewhat readable. The first column indicates the hypervisor
used, the guest OS and the network driver in the guest. dom0 was
running Debian's 2.6.32-5-amd64 Linux kernel. Xen was Debian's
4.0.1. Linux guests were running Debian's 2.6.32-5-amd64. FreeBSD
was 8.2 amd64 with the "too many frags" and "panic: do something
smart" patches to make it usable with Xen at all.

I did not test multiple concurrent connections, multiple virtual
machines transferring data simultaneously, mixtures of different
kinds of data, CPU load nor anything like that which is also relevant.
Only the raw TCP speed was measured to figure out if there is any
significant difference: and yes there was as can be seen from the
numbers above.

Unfortunately I have no recollection and forgot to write down if
input was input from dom0's perspective or domU's perspective. They
should be the same way around for all the tests though.

My lesson was that it does make sense to select your network drivers
carefully in a virtualized environment.

I also tried two different virtio patches for FreeBSD with KVM, but
one of them did not work at all and the another one gave worse
results than any sensible emulated hardware.

--
Janne Snabb / EPIPE Communications
sn...@epipe.com - http://epipe.com/
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Justin T. Gibbs]

On 5/12/11 11:48 AM, Janne Snabb wrote:

On Thu, 12 May 2011, Laurent Cligny wrote:


All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
Paravirtualized Debian amd64.

My suggestion would be to try out the same setup with GENERIC kernel
and the rtl driver (or even better e1000 if your Xen allows it)
which is easy to do to make a simple comparison.

In one of my recent benchmarks the FreeBSD Xen PV network driver
performed very well in one direction, but very badly in the other.
Using rtl or e1000 gave much better TCP throughput if you care
equally about both directions. This benchmark was done only for
internal TCP traffic between a Linux dom0 and FreeBSD dumU, the
traffic never entered a real NIC.


Do you recall which path was slow (rx or tx from the perspective of
the FreeBSD driver) and what the relative difference in performance
was between the two approaches?

--
Justin
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Tobias P. Santos]

laurent.cli...@gmail.com wrote:

Le 12/05/2011 17:02, Tobias P. Santos a écrit :
  

Try this:

ifconfig xn1 -txcsum

Best regards,
Tobias.


Hello Tobias,

Unfortunately ifconfig xn1 -txcsum didn't give me lot of result.
Doing this my network perf through NAT is about 7KiB/s (that is better
than before, but still unworkable).

Thanks for the idea, maybe we are not that far away from the solution.

  
We had a problem where traffic between VMs was too slow (about 5Mbps on 
a Gigabit network) and disabling TX checksum solved the problem 
(actually, speed increased to more than 1Gbps).
Your problem is different as you have a nice throughput between VMs on 
xn1, sorry that I missed that on my first message.


Did you try to transfer data from something connected to xn0 to one of 
yours DomU servers connected to xn1 without NAT?

You also mentioned pf. Did you make any test using natd or anything else?
Something tells me that's not a nic problem, but that's just a guess.

Anyway, on the coming weeks I will have to set up a similar environment 
to yours, so maybe I'll face the same problem (although we use natd 
daemon instead of pf).


Good luck!

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Guillaume Seigneuret]

I think you should try totally disable the window sizing both parts to
isolate it doesn't come from here :
On your FreeBSD VM : sysctl net.inet.tcp.rfc1323=0
On your Linux box : sysctl net.ipv4.tcp_window_scaling=0

Then if the rate is hugely better, try some settings to increase the
compatibility.
You can also try an *iperf *test with TCP and UDP packets to see if the
behavior is generalized, and see what latency you have both part of the
network.

Cordialement,
*Guillaume Seigneuret*



*Network and System Security Architect
 Mobile :  +33.6.15.37.35.67
 Web :  http://www.omegacube.fr
 Address :*
Hôtel Technologique Marseille Innovation - BP 100
45 rue Frederic Joliot Curie
Technopôle de Château Gombert
13382 Marseille Cedex 13



2011/5/12 Michael MacLeod 

> Try -rxcsum as well. I have both disabled on my domU.
>
> On Thu, May 12, 2011 at 12:53 PM, laurent.cli...@gmail.com <
> laurent.cli...@gmail.com> wrote:
>
> > Le 12/05/2011 17:02, Tobias P. Santos a écrit :
> > > Try this:
> > >
> > > ifconfig xn1 -txcsum
> > >
> > > Best regards,
> > > Tobias.
> > Hello Tobias,
> >
> > Unfortunately ifconfig xn1 -txcsum didn't give me lot of result.
> > Doing this my network perf through NAT is about 7KiB/s (that is better
> > than before, but still unworkable).
> >
> > Thanks for the idea, maybe we are not that far away from the solution.
> >
> > >
> > >
> > > Laurent Cligny wrote:
> > >> Hello all,
> > >>
> > >> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
> > >> provide Internet access to other FreeBSD and Linux DomU VMs.
> > >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8
> cores,
> > >> 16GiB RAM and one NIC.
> > >>
> > >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
> > >> Paravirtualized Debian amd64.
> > >>
> > >> The NAT gateway is a two-NIC box, with one (xn0) configured with a
> > >> public Internet address and the other (xn1) configured with a class A
> > >> private address (10.0.0.254).
> > >> The Internet traffic is very good from my FreeBSD NAT gateway to the
> > >> Internet (~ 50MiB/s) and also between other VMs on the private network
> > >> (~ 70MiB/s trough scp) after applying the patch here
> > >> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
> > >> fixing the "too many frags" problem.
> > >>
> > >> Here is a schema of my setup with the observed net speed on it:
> > >>
> > >> -- --- --
> > >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
> > >> -- --- --
> > >>   <--->   <--->
> > >>50MiB/s 70MiB/s
> > >>
> > >>   <--->
> > >>0KiB/s  > >>
> > >> [...]
> > > ___
> > > freebsd-xen@freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> > > To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
> >
> > ___
> > freebsd-xen@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> > To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
> >
> ___
> freebsd-xen@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
>
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Michael MacLeod]

Try -rxcsum as well. I have both disabled on my domU.

On Thu, May 12, 2011 at 12:53 PM, laurent.cli...@gmail.com <
laurent.cli...@gmail.com> wrote:

> Le 12/05/2011 17:02, Tobias P. Santos a écrit :
> > Try this:
> >
> > ifconfig xn1 -txcsum
> >
> > Best regards,
> > Tobias.
> Hello Tobias,
>
> Unfortunately ifconfig xn1 -txcsum didn't give me lot of result.
> Doing this my network perf through NAT is about 7KiB/s (that is better
> than before, but still unworkable).
>
> Thanks for the idea, maybe we are not that far away from the solution.
>
> >
> >
> > Laurent Cligny wrote:
> >> Hello all,
> >>
> >> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
> >> provide Internet access to other FreeBSD and Linux DomU VMs.
> >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores,
> >> 16GiB RAM and one NIC.
> >>
> >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
> >> Paravirtualized Debian amd64.
> >>
> >> The NAT gateway is a two-NIC box, with one (xn0) configured with a
> >> public Internet address and the other (xn1) configured with a class A
> >> private address (10.0.0.254).
> >> The Internet traffic is very good from my FreeBSD NAT gateway to the
> >> Internet (~ 50MiB/s) and also between other VMs on the private network
> >> (~ 70MiB/s trough scp) after applying the patch here
> >> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
> >> fixing the "too many frags" problem.
> >>
> >> Here is a schema of my setup with the observed net speed on it:
> >>
> >> -- --- --
> >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
> >> -- --- --
> >>   <--->   <--->
> >>50MiB/s 70MiB/s
> >>
> >>   <--->
> >>0KiB/s  >>
> >> [...]
> > ___
> > freebsd-xen@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> > To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
>
> ___
> freebsd-xen@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
>
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[laurent.cli...@gmail.com]

Le 12/05/2011 17:02, Tobias P. Santos a écrit :
> Try this:
>
> ifconfig xn1 -txcsum
>
> Best regards,
> Tobias.
Hello Tobias,

Unfortunately ifconfig xn1 -txcsum didn't give me lot of result.
Doing this my network perf through NAT is about 7KiB/s (that is better
than before, but still unworkable).

Thanks for the idea, maybe we are not that far away from the solution.

>
>
> Laurent Cligny wrote:
>> Hello all,
>>
>> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
>> provide Internet access to other FreeBSD and Linux DomU VMs.
>> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores,
>> 16GiB RAM and one NIC.
>>
>> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
>> Paravirtualized Debian amd64.
>>
>> The NAT gateway is a two-NIC box, with one (xn0) configured with a
>> public Internet address and the other (xn1) configured with a class A
>> private address (10.0.0.254).
>> The Internet traffic is very good from my FreeBSD NAT gateway to the
>> Internet (~ 50MiB/s) and also between other VMs on the private network
>> (~ 70MiB/s trough scp) after applying the patch here
>> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
>> fixing the "too many frags" problem.
>>
>> Here is a schema of my setup with the observed net speed on it:
>>
>> -- --- --
>> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
>> -- --- --
>>   <--->   <--->
>>50MiB/s 70MiB/s
>>
>>   <--->
>>0KiB/s >
>> [...]
> ___
> freebsd-xen@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Laurent Cligny]

Thanks for your quick reply.
> Maybe you should have a look on the tcp window size values.
> I had such a behavior with two gateways who was not playing the same way
> with the TCP window size which made my network damn slow.
> If you have a linux gateway in front of your FreeBSD VM, it should be this.
Well I compared the TCP default values between the FreeBSD Vm and the
Linux PV VM and changed the net.inet.tcp.recvspace and
net.inet.tcp.sendpace to 124928 wich are the values on the Linux VM, but
there's no change on the network performance. Nice idea, much appreciated.
> Command
> Description
> sysctl net.inet.tcp.rfc1323=1 Activate window scaling and timestamp options
> according to RFC 1323.sysctl ipc.maxsockbuf=*[sbmax]* Set maximum size of
> TCP window.sysctl net.inet.tcp.recvspace=*[wstd]* Set default size of TCP
> receive window.sysctl net.inet.tcp.sendspace=*[wstd]* Set default size of
> TCP transmit window.sysctl kern.ipc.nmbclusters View maximum number of mbuf
> clusters. Used for storage of data packets to/from the network interface.
> Can only be set att boot time - see above. sysctl net.inet.tcp.liondmask=7Used
> to enable the "netlion" patch above.
>
> Cordialement,
> *Guillaume Seigneuret*
>
>
> *Network and System Security Architect*
>  *Web* :  http://www.omegacube.fr
>  *Address* :
> Hôtel Technologique Marseille Innovation - BP 100
> 45 rue Frederic Joliot Curie
> Technopôle de Château Gombert
> 13382 Marseille Cedex 13
>
>
>
> 2011/5/12 Laurent Cligny 
>
>> Hello all,
>>
>> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
>> provide Internet access to other FreeBSD and Linux DomU VMs.
>> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores,
>> 16GiB RAM and one NIC.
>>
>> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
>> Paravirtualized Debian amd64.
>>
>> The NAT gateway is a two-NIC box, with one (xn0) configured with a
>> public Internet address and the other (xn1) configured with a class A
>> private address (10.0.0.254).
>> The Internet traffic is very good from my FreeBSD NAT gateway to the
>> Internet (~ 50MiB/s) and also between other VMs on the private network
>> (~ 70MiB/s trough scp) after applying the patch here
>> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
>> fixing the "too many frags" problem.
>>
>> Here is a schema of my setup with the observed net speed on it:
>>
>> -- --- --
>> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
>> -- --- --
>>  <--->   <--->
>>   50MiB/s 70MiB/s
>>
>>  <--->
>>   0KiB/s >
>> I tried doing NAT on the FBSD gateway with PF and also with NATd after
>> shutting down PF and unloaded the kernel modules pf.ko and pflog.ko, but
>> I have the same perf problems with both solutions. Also I tried
>> disabling net.inet.tcp.tso and did ifconfig -tso on all NICs on NAT
>> gateway and other VMs without change.
>>
>> I don't see anything on the error console or in /var/log/message.
>>
>> When i'm trying going though NAT with a Linux paravirt DomU, the network
>> perf is slightly better but don't go over 5KiB/s.
>>
>> Finally I put my Linux paravirt DomU VM as a NAT gateway in place of the
>> FreeBSD one, and the network perf is now normal through it (50MiB/s)
>> from the FreeBSD DomU VMs.
>>
>> As I'm very happy with FreeBSD for my environment I'd like to keep as
>> much as this lovely OS for all my needs, so if anyone have an idea where
>> the problem can lies, and what to test next to have a clue of what's
>> happening, I will be very thankfull.
>>
>> Thanks in advance.
>>
>> --
>> /Regards,/
>>
>> *Laurent Cligny
>> /Founder/*
>> 02 97 37 71 14
>> 06 25 40 86 48
>> laurent.cli...@steadinet.fr
>>
>> Steadinet 
>>
>>
>> ___
>> freebsd-xen@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
>> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
>>
>>
> ___
> freebsd-xen@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Janne Snabb]

On Thu, 12 May 2011, Laurent Cligny wrote:

> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
> Paravirtualized Debian amd64.

My suggestion would be to try out the same setup with GENERIC kernel
and the rtl driver (or even better e1000 if your Xen allows it)
which is easy to do to make a simple comparison.

In one of my recent benchmarks the FreeBSD Xen PV network driver
performed very well in one direction, but very badly in the other.
Using rtl or e1000 gave much better TCP throughput if you care
equally about both directions. This benchmark was done only for
internal TCP traffic between a Linux dom0 and FreeBSD dumU, the
traffic never entered a real NIC.

This might or might not help in your situation...

--
Janne Snabb / EPIPE Communications
sn...@epipe.com - http://epipe.com/
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Guillaume Seigneuret]

Maybe you should have a look on the tcp window size values.
I had such a behavior with two gateways who was not playing the same way
with the TCP window size which made my network damn slow.
If you have a linux gateway in front of your FreeBSD VM, it should be this.

Command
Description
sysctl net.inet.tcp.rfc1323=1 Activate window scaling and timestamp options
according to RFC 1323.sysctl ipc.maxsockbuf=*[sbmax]* Set maximum size of
TCP window.sysctl net.inet.tcp.recvspace=*[wstd]* Set default size of TCP
receive window.sysctl net.inet.tcp.sendspace=*[wstd]* Set default size of
TCP transmit window.sysctl kern.ipc.nmbclusters View maximum number of mbuf
clusters. Used for storage of data packets to/from the network interface.
Can only be set att boot time - see above. sysctl net.inet.tcp.liondmask=7Used
to enable the "netlion" patch above.

Cordialement,
*Guillaume Seigneuret*



*Network and System Security Architect*
 *Web* :  http://www.omegacube.fr
 *Address* :
Hôtel Technologique Marseille Innovation - BP 100
45 rue Frederic Joliot Curie
Technopôle de Château Gombert
13382 Marseille Cedex 13



2011/5/12 Laurent Cligny 

> Hello all,
>
> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
> provide Internet access to other FreeBSD and Linux DomU VMs.
> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores,
> 16GiB RAM and one NIC.
>
> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
> Paravirtualized Debian amd64.
>
> The NAT gateway is a two-NIC box, with one (xn0) configured with a
> public Internet address and the other (xn1) configured with a class A
> private address (10.0.0.254).
> The Internet traffic is very good from my FreeBSD NAT gateway to the
> Internet (~ 50MiB/s) and also between other VMs on the private network
> (~ 70MiB/s trough scp) after applying the patch here
> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
> fixing the "too many frags" problem.
>
> Here is a schema of my setup with the observed net speed on it:
>
> -- --- --
> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
> -- --- --
>  <--->   <--->
>   50MiB/s 70MiB/s
>
>  <--->
>   0KiB/s 
> I tried doing NAT on the FBSD gateway with PF and also with NATd after
> shutting down PF and unloaded the kernel modules pf.ko and pflog.ko, but
> I have the same perf problems with both solutions. Also I tried
> disabling net.inet.tcp.tso and did ifconfig -tso on all NICs on NAT
> gateway and other VMs without change.
>
> I don't see anything on the error console or in /var/log/message.
>
> When i'm trying going though NAT with a Linux paravirt DomU, the network
> perf is slightly better but don't go over 5KiB/s.
>
> Finally I put my Linux paravirt DomU VM as a NAT gateway in place of the
> FreeBSD one, and the network perf is now normal through it (50MiB/s)
> from the FreeBSD DomU VMs.
>
> As I'm very happy with FreeBSD for my environment I'd like to keep as
> much as this lovely OS for all my needs, so if anyone have an idea where
> the problem can lies, and what to test next to have a clue of what's
> happening, I will be very thankfull.
>
> Thanks in advance.
>
> --
> /Regards,/
>
> *Laurent Cligny
> /Founder/*
> 02 97 37 71 14
> 06 25 40 86 48
> laurent.cli...@steadinet.fr
>
> Steadinet 
>
>
> ___
> freebsd-xen@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
>
>
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Tobias P. Santos]

Try this:

ifconfig xn1 -txcsum

Best regards,
Tobias.


Laurent Cligny wrote:

Hello all,

I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
provide Internet access to other FreeBSD and Linux DomU VMs.
My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores,
16GiB RAM and one NIC.

All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
Paravirtualized Debian amd64.

The NAT gateway is a two-NIC box, with one (xn0) configured with a
public Internet address and the other (xn1) configured with a class A
private address (10.0.0.254).
The Internet traffic is very good from my FreeBSD NAT gateway to the
Internet (~ 50MiB/s) and also between other VMs on the private network
(~ 70MiB/s trough scp) after applying the patch here
(http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
fixing the "too many frags" problem.

Here is a schema of my setup with the observed net speed on it:

-- --- --
|Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
-- --- --
  <--->   <--->
   50MiB/s 70MiB/s

  <--->
   0KiB/s 
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Laurent Cligny]

Hello all,

I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
provide Internet access to other FreeBSD and Linux DomU VMs.
My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores,
16GiB RAM and one NIC.

All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
Paravirtualized Debian amd64.

The NAT gateway is a two-NIC box, with one (xn0) configured with a
public Internet address and the other (xn1) configured with a class A
private address (10.0.0.254).
The Internet traffic is very good from my FreeBSD NAT gateway to the
Internet (~ 50MiB/s) and also between other VMs on the private network
(~ 70MiB/s trough scp) after applying the patch here
(http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
fixing the "too many frags" problem.

Here is a schema of my setup with the observed net speed on it:

-- --- --
|Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
-- --- --
  <--->   <--->
   50MiB/s 70MiB/s

  <--->
   0KiB/s http://www.steadinet.fr>

___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"