Re: Is it me or is FreeBSD slower on Xen than Linux?

[Roger Pau Monné]

On Wed, Aug 17, 2016 at 11:29:17AM +0200, rai...@ultra-secure.de wrote:
> Am 2016-08-17 11:12, schrieb Roger Pau Monné:
> 
> > No, I don't think so, this is what I get using a slow USB 2.0 disk as
> > the
> > backend: (on Dom0 I get something between 70-80M/s, so there isn't much
> > difference).
> > 
> > # dc3dd wipe=/dev/ada1
> > 
> > dc3dd 7.2.641 started at 2016-08-17 09:03:26 +
> > compiled options:
> > command line: dc3dd wipe=/dev/ada1
> > device size: 2097152 sectors (probed),1,073,741,824 bytes
> > sector size: 512 bytes (probed)
> >   1073741824 bytes ( 1 G ) copied ( 100% ),   16 s, 65 M/s
> > 
> > input results for pattern `00':
> >2097152 sectors in
> > 
> > output results for device `/dev/ada1':
> >2097152 sectors out
> > 
> > dc3dd completed at 2016-08-17 09:03:42 +
> > 
> > > What may I be doing wrong?
> > 
> > TBH it's hard to tell, I don't know of any option that could cause this
> > disk
> > performance degradation. Do you also have ada* devices apart from the
> > xbd*
> > ones? I don't think it's going to make any difference, but could you try
> > with the ada* block devices instead?
> 
> 
> Strange thing is, I have ada devices for the the other disks, but this one
> didn't show up as ada-device.

The fact that it shows up as ada or xbd depends on what you specify in the 
guest config file (hd* will show up as ada, while xvd* will show up as 
xbd*). I don't know how/if XenServer allows you to specify the vdev in the 
guest configuration.

> On my FreeBSD11 Test VM, the disk didn't show up until I rebooted, even
> though I (believe to) have the xen-guest stuff installed:
> 
> (freebsd11 ) 0 # ps ax |grep xe-d
>  694 v0- I0:00.00 /bin/sh /usr/local/sbin/xe-daemon -p
> /var/run/xe-daemon.pid
> 2202  0  R+   0:00.00 grep xe-d
> (freebsd11 ) 0 # pkg info|grep xe-
> xe-guest-utilities-6.2.0_2 FreeBSD VM tools for Citrix XenServer and XCP
> (freebsd11 ) 0 # pkg info|grep xen
> xen-guest-tools-4.6.1  Xen tools within FreeBSD domU
> 
> I have an ada device there and I got about 10MB/s on wipe. At least in the
> beginning.

Hm, so performance is more or less the same. Having the xen-guest stuff 
should not make a difference regarding disks, this is IIRC only used when 
migrating a VM.

> (freebsd11 ) 0 # sysctl -a |grep xen
> kern.vm_guest: xen
> devicexenpci
> vfs.pfs.vncache.maxentries: 0
> dev.xenbusb_back.0.%parent: xenstore0
> dev.xenbusb_back.0.%pnpinfo:
> dev.xenbusb_back.0.%location:
> dev.xenbusb_back.0.%driver: xenbusb_back
> dev.xenbusb_back.0.%desc: Xen Backend Devices
> dev.xenbusb_back.%parent:
> dev.xn.0.xenstore_peer_path: /local/domain/0/backend/vif/245/0
> dev.xn.0.xenbus_peer_domid: 0
> dev.xn.0.xenbus_connection_state: Connected
> dev.xn.0.xenbus_dev_type: vif
> dev.xn.0.xenstore_path: device/vif/0
> dev.xn.0.%parent: xenbusb_front0
> dev.xbd.1.xenstore_peer_path: /local/domain/0/backend/vbd3/245/768
> dev.xbd.1.xenbus_peer_domid: 0
> dev.xbd.1.xenbus_connection_state: Connected
> dev.xbd.1.xenbus_dev_type: vbd
> dev.xbd.1.xenstore_path: device/vbd/768
> dev.xbd.1.%parent: xenbusb_front0
> dev.xbd.0.xenstore_peer_path: /local/domain/0/backend/vbd3/245/832
> dev.xbd.0.xenbus_peer_domid: 0
> dev.xbd.0.xenbus_connection_state: Connected
> dev.xbd.0.xenbus_dev_type: vbd
> dev.xbd.0.xenstore_path: device/vbd/832
> dev.xbd.0.%parent: xenbusb_front0
> dev.xenbusb_front.0.%parent: xenstore0
> dev.xenbusb_front.0.%pnpinfo:
> dev.xenbusb_front.0.%location:
> dev.xenbusb_front.0.%driver: xenbusb_front
> dev.xenbusb_front.0.%desc: Xen Frontend Devices
> dev.xenbusb_front.%parent:
> dev.xs_dev.0.%parent: xenstore0
> dev.xctrl.0.%parent: xenstore0
> dev.xenballoon.0.%parent: xenstore0
> dev.xenballoon.0.%pnpinfo:
> dev.xenballoon.0.%location:
> dev.xenballoon.0.%driver: xenballoon
> dev.xenballoon.0.%desc: Xen Balloon Device
> dev.xenballoon.%parent:
> dev.debug.0.%parent: xenpv0
> dev.privcmd.0.%parent: xenpv0
> dev.evtchn.0.%parent: xenpv0
> dev.xenstore.0.%parent: xenpv0
> dev.xenstore.0.%pnpinfo:
> dev.xenstore.0.%location:
> dev.xenstore.0.%driver: xenstore
> dev.xenstore.0.%desc: XenStore
> dev.xenstore.%parent:
> dev.xen_et.0.%parent: xenpv0
> dev.xen_et.0.%pnpinfo:
> dev.xen_et.0.%location:
> dev.xen_et.0.%driver: xen_et
> dev.xen_et.0.%desc: Xen PV Clock
> dev.xen_et.%parent:
> dev.granttable.0.%parent: xenpv0
> dev.xenpv.0.%parent: nexus0
> dev.xenpv.0.%pnpinfo:
> dev.xenpv.0.%location:
> dev.xenpv.0.%driver: xenpv
> dev.xenpv.0.%desc: Xen PV bus
> dev.xenpv.%parent:
> dev.xenpci.0.%parent: pci0
> dev.xenpci.0.%pnpinfo: vendor=0x5853 device=0x0001 subvendor=0x5853
> subdevice=0x0001 class=0x01
> dev.xenpci.0.%location: slot=3 function=0 dbsf=pci0:0:3:0
> handle=\_SB_.PCI0.S18_
> dev.xenpci.0.%driver: xenpci
> dev.xenpci.0.%desc: Xen Platform Device
> dev.xenpci.%parent:
> dev.xen.xsd_kva: 18446735281894703104
> dev.xen.xsd_port: 3
> dev.xen.balloon.high_mem: 0
> dev.xen.balloon.low_mem: 0
> dev.xen.balloon.hard_limit: 

Re: Is it me or is FreeBSD slower on Xen than Linux?

[Roger Pau Monné]

On Wed, Aug 17, 2016 at 10:19:05AM +0200, rai...@ultra-secure.de wrote:
> Am 2016-08-16 16:24, schrieb rai...@ultra-secure.de:
> > Am 2016-08-16 16:18, schrieb Roger Pau Monné:
> > 
> > > I'm not saying it's not possible, but are you sure this slowdown is
> > > caused
> > > by the disk? It's certainly a possibility, but I would like to make
> > > sure
> > > it's caused by that before jumping into conclusions.
> > > 
> > > Can you load the full database in RAM and perform the same test.
> > > TBH, I
> > > don't use MariaDB, so I'm not sure what's the best way to achieve
> > > this, but
> > > a quick search on google shows there are multiple ways. In any case,
> > > make
> > > sure with iostat that the database is not read from the disk.
> > 
> > I'll try to do some disk-benchmarks, when I can attach some bigger
> > disks.
> 
> on Ubuntu 14 with HVM:
> 
> dc3dd wipe=/dev/xvdc
> 
> dc3dd 7.1.614 started at 2016-08-17 09:38:17 +0200
> compiled options:
> command line: dc3dd wipe=/dev/xvdc
> device size: 104857600 sectors (probed)
> sector size: 512 bytes (probed)
> 53687091200 bytes (50 G) copied (100%), 464.642 s, 110 M/s
> 
> input results for pattern `00':
>104857600 sectors in
> 
> output results for device `/dev/xvdc':
>104857600 sectors out
> 
> dc3dd completed at 2016-08-17 09:46:01 +0200
> 
> 
> On FreeBSD 10.3-RELEASE-p6 with HVM:
> 
> Aug 17 09:57:52 bla-prod kernel: xbd4: 51200MB  at
> device/vbd/51776 on xenbusb_front0
> Aug 17 09:57:52 bla-prod kernel: xbd4: features: write_barrier
> Aug 17 09:57:52 bla-prod kernel: xbd4: synchronize cache commands enabled.
> 
> dc3dd wipe=/dev/xbd4
> 
> and it's showing 8.something MB/s
> 
> Is this normal?

No, I don't think so, this is what I get using a slow USB 2.0 disk as the 
backend: (on Dom0 I get something between 70-80M/s, so there isn't much 
difference).

# dc3dd wipe=/dev/ada1

dc3dd 7.2.641 started at 2016-08-17 09:03:26 +
compiled options:
command line: dc3dd wipe=/dev/ada1
device size: 2097152 sectors (probed),1,073,741,824 bytes
sector size: 512 bytes (probed)
  1073741824 bytes ( 1 G ) copied ( 100% ),   16 s, 65 M/s

input results for pattern `00':
   2097152 sectors in

output results for device `/dev/ada1':
   2097152 sectors out

dc3dd completed at 2016-08-17 09:03:42 +

> What may I be doing wrong?

TBH it's hard to tell, I don't know of any option that could cause this disk 
performance degradation. Do you also have ada* devices apart from the xbd* 
ones? I don't think it's going to make any difference, but could you try 
with the ada* block devices instead?

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it me or is FreeBSD slower on Xen than Linux?

[Roger Pau Monné]

On Tue, Aug 16, 2016 at 04:05:30PM +0200, rai...@ultra-secure.de wrote:
> Am 2016-08-16 15:48, schrieb Borja Marcos:
> > > On 16 Aug 2016, at 15:41, rai...@ultra-secure.de wrote:
> > > 
> > > Am 2016-08-16 15:38, schrieb Borja Marcos:
> > > > 
> > > > Maybe this is too obvious, my apologies in that case. But, how have
> > > > the filesystems been
> > > > created and mounted? Asynchronous? Synchronous? Journalling?
> > > > Softupdates in the case of
> > > > FreeBSD UFS? It can make quite a difference.
> > > 
> > > 
> > > FreeBSD
> > > 
> > > /dev/ada2p1 on /home/db (ufs, local, soft-updates)
> > > 
> > > 
> > > 
> > > Linux:
> > > /dev/mapper/system-lvm--home /home   ext4defaults
> > > 0   2
> > > 
> > > 
> > > What does "defaults" mean, BTW?
> > 
> > That’s the mother of the lamb, we use to say in Spain ;)
> > 
> > I guess it depends on the particular distribution, not just on being
> > ext4. Is there a tool similar to
> > dumpfs on Linux?
> 
> 
> Apparently, it's in
> cat /proc/mounts
> 
> /dev/mapper/system-lvm--tmp /tmp ext4 rw,relatime,data=ordered 0 0
> /dev/mapper/system-lvm--var /var ext4 rw,relatime,data=ordered 0 0
> /dev/mapper/system-lvm--home /home ext4 rw,relatime,data=ordered 0 0
> /dev/mapper/system-lvm--varlog /var/log ext4 rw,relatime,data=ordered 0 0
> 
> 
> 
> 
> > You can also experiment with the FreeBSD options, maybe it will be a
> > quicker route. Try to mount as asynchronous.
> > In case it  makes a big difference, you got it.
> 
> But I don't really want to mount it asyncronous.
> Would it help to have journaling?
> 
> Or is soft-updates already the "optimum"?

I'm not saying it's not possible, but are you sure this slowdown is caused 
by the disk? It's certainly a possibility, but I would like to make sure 
it's caused by that before jumping into conclusions.

Can you load the full database in RAM and perform the same test. TBH, I 
don't use MariaDB, so I'm not sure what's the best way to achieve this, but 
a quick search on google shows there are multiple ways. In any case, make 
sure with iostat that the database is not read from the disk.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it me or is FreeBSD slower on Xen than Linux?

[Borja Marcos]

> On 16 Aug 2016, at 15:41, rai...@ultra-secure.de wrote:
> 
> Am 2016-08-16 15:38, schrieb Borja Marcos:
>> 
>> Maybe this is too obvious, my apologies in that case. But, how have
>> the filesystems been
>> created and mounted? Asynchronous? Synchronous? Journalling?
>> Softupdates in the case of
>> FreeBSD UFS? It can make quite a difference.
> 
> 
> FreeBSD
> 
> /dev/ada2p1 on /home/db (ufs, local, soft-updates)
> 
> 
> 
> Linux:
> /dev/mapper/system-lvm--home /home   ext4defaults0   2
> 
> 
> What does "defaults" mean, BTW?

That’s the mother of the lamb, we use to say in Spain ;)

I guess it depends on the particular distribution, not just on being ext4. Is 
there a tool similar to
dumpfs on Linux? 

You can also experiment with the FreeBSD options, maybe it will be a quicker 
route. Try to mount as asynchronous.
In case it  makes a big difference, you got it.





Borja.

___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it me or is FreeBSD slower on Xen than Linux?

[rainer]

Am 2016-08-16 15:38, schrieb Borja Marcos:
On 16 Aug 2016, at 15:29, Roger Pau Monné  
wrote:



Could this really be an UFS vs. ext4 thing?


Hm, maybe. There are a lot of moving pieces here that make it quite 
hard to

diagnose the issue properly.

Could you try to run something like UnixBench (or any other general
benchmarking tool) inside of the Linux VM, the FreeBSD VM and a bare 
metal

FreeBSD install? This way we might be able to spot what's causing this
slowdown.


Maybe this is too obvious, my apologies in that case. But, how have
the filesystems been
created and mounted? Asynchronous? Synchronous? Journalling?
Softupdates in the case of
FreeBSD UFS? It can make quite a difference.



FreeBSD

/dev/ada2p1 on /home/db (ufs, local, soft-updates)



Linux:
/dev/mapper/system-lvm--home /home   ext4defaults0   
2



What does "defaults" mean, BTW?

___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it me or is FreeBSD slower on Xen than Linux?

[Borja Marcos]

> On 16 Aug 2016, at 15:29, Roger Pau Monné  wrote:
> 
>> Could this really be an UFS vs. ext4 thing?
> 
> Hm, maybe. There are a lot of moving pieces here that make it quite hard to 
> diagnose the issue properly.
> 
> Could you try to run something like UnixBench (or any other general 
> benchmarking tool) inside of the Linux VM, the FreeBSD VM and a bare metal 
> FreeBSD install? This way we might be able to spot what's causing this 
> slowdown.

Maybe this is too obvious, my apologies in that case. But, how have the 
filesystems been
created and mounted? Asynchronous? Synchronous? Journalling? Softupdates in the 
case of 
FreeBSD UFS? It can make quite a difference.





Borja.

___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it me or is FreeBSD slower on Xen than Linux?

[Roger Pau Monné]

On Tue, Aug 16, 2016 at 03:14:05PM +0200, rai...@ultra-secure.de wrote:
> Am 2016-08-16 13:08, schrieb Roger Pau Monné:
> 
> > And FreeBSD on bare metal is equally fast as Linux then? (ie: the
> > slowdown
> > is only noticeable when running FreeBSD on Xen)
> 
> 
> Yes.
> 
> I should clarify a bit more.
> The task involves 12000 (simple) MySQL-Queries, where the script basically
> selects the numbers 1...4000 from a table (it's stupid, I know) and then
> proceeds to run a for i in ... loop 4000 times which consists of three other
> sql-queries, where the WHERE-clause is constrained by the value from above.
> 
> We've now found that indeed MariaDB is much faster on Xen-Linux than
> Xen-FreeBSD.
> 
> The tables all use innodb and the DB is sitting on UFS (in the
> FreeBSD-on-Xen case, the FreeBSD-on-bare-metal has ZFS).
> Linux is using ext4.

Hm, the fact that FreeBSD on bare metal is using ZFS could also make a 
difference. The ZFS memory caching is quite aggressive, and I expect it 
should speed up database queries (unless the database itself is fully loaded 
into RAM, in which case it doesn't matter much).
 
> Could this really be an UFS vs. ext4 thing?

Hm, maybe. There are a lot of moving pieces here that make it quite hard to 
diagnose the issue properly.

Could you try to run something like UnixBench (or any other general 
benchmarking tool) inside of the Linux VM, the FreeBSD VM and a bare metal 
FreeBSD install? This way we might be able to spot what's causing this 
slowdown.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it me or is FreeBSD slower on Xen than Linux?

[Roger Pau Monné]

On Tue, Aug 16, 2016 at 12:13:55PM +0200, rai...@ultra-secure.de wrote:
> Am 2016-08-16 12:06, schrieb Roger Pau Monné:
> > On Tue, Aug 16, 2016 at 11:29:43AM +0200, rai...@ultra-secure.de wrote:
> > > Am 2016-08-16 10:54, schrieb Roger Pau Monné:
> > > > On Mon, Aug 15, 2016 at 05:54:52PM +0200, rai...@ultra-secure.de wrote:
> > > > > Hi,
> > > > >
> > > > > I've got a problem.
> > > > >
> > > >
> > > > Hello,
> > > >
> > > > >
> > > > > For a customer, I run a VM in Xen that should perform a certain task
> > > > > in PHP
> > > > > (written using the ZendFrameWork).
> > > > >
> > > > > That task takes about 18-20 seconds on FreeBSD 10.3 amd64, MariaDB
> > > > > 5.5.0,
> > > > > php 5.5.37 in a VM that has 8 vCPUs and 16GB of memory
> > > > > The "reference" server that the customer uses is somewhere else and
> > > > > manages
> > > > > to perform the same task in 3s.
> > > > >
> > > > > I've tried this with FreeBSD 10.3, PHP7.0 and MariaDB 10.1 and it
> > > > > takes
> > > > > about 9s.
> > > >
> > > > In the sentence above, are you running it in a Xen VM or on bare metal?
> > > >
> > > 
> > > This is both Xen.
> > > I think the customer is also running it on some sort of
> > > virtualization.
> > 
> > Hm, so a given workload on Xen takes ~9s, and it also takes ~9s when run
> > on
> > bare metal FreeBSD, is that right?
> 
> 
> It only takes 9s with Linux as a Xen-guest.
> With all things equal (PHP-version, MariaDB-version), FreeBSD is essentially
> only half as fast as Linux as a Xen-guest.
> Sorry for the confusion.
> 

And FreeBSD on bare metal is equally fast as Linux then? (ie: the slowdown 
is only noticeable when running FreeBSD on Xen)

> 
> > > > hw.xen.disable_pv_disks=1
> > > > hw.xen.disable_pv_nics=1
> > > 
> > > OK, can I still boot the VM with this or will NICs and disks show up
> > > as
> > > different devices then?
> > 
> > NIC will show up as "re", disks as "ada" (which is what you already
> > have).
> 
> I tried this with the FreeBSD 11 VM mentioned in my other mail and it only
> gets a bit slower.
> Between 5% and 10%, I'd say.

If you can provide me with some way to synthesize this workload that doesn't 
involve setting up the full stack plus your app I can try to reproduce it 
locally and analyze it in order to find the bottlenecks.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"