[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-13 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #59 from rai...@ultra-secure.de ---
BYTE UNIX Benchmarks (Version 4.1.0)
  System -- freebsd11
  Start Benchmark Run: Mon Feb 13 17:28:56 CET 2017
   3 interactive users.
   5:28PM  up  3:06, 3 users, load averages: 0.62, 0.69, 0.71
  -r-xr-xr-x  1 root  wheel  153744 Sep 27 18:03 /bin/sh
  /bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically
linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100122),
FreeBSD-style, stripped
  zroot/ROOT/default11262956 6353528 490942856%/
File Read 4096 bufsize 8000 maxblocks2313874.0 KBps  (30.0 secs, 3 samples)
File Write 4096 bufsize 8000 maxblocks   120869.0 KBps  (30.0 secs, 3 samples)
File Copy 4096 bufsize 8000 maxblocks126921.0 KBps  (30.0 secs, 3 samples)
Shell Scripts (1 concurrent)   2137.7 lpm   (60.0 secs, 3 samples)
Shell Scripts (8 concurrent)314.1 lpm   (60.0 secs, 3 samples)
Shell Scripts (16 concurrent)   154.5 lpm   (60.0 secs, 3 samples)
Arithmetic Test (type = short)1.0 lps   (0.0 secs, 3 samples)
Arithmetic Test (type = int)  1.0 lps   (0.0 secs, 3 samples)
Arithmetic Test (type = long) 1.0 lps   (0.0 secs, 3 samples)
Arithmetic Test (type = float)1.0 lps   (0.0 secs, 3 samples)
Arithmetic Test (type = double)   1.0 lps   (0.0 secs, 3 samples)
Arithoh   1.0 lps   (0.0 secs, 3 samples)
C Compiler Throughput   499.1 lpm   (60.0 secs, 3 samples)
Dc: sqrt(2) to 99 decimal places  78631.1 lpm   (30.0 secs, 3 samples)
Recursion Test--Tower of Hanoi   192648.9 lps   (20.0 secs, 3 samples)


 INDEX VALUES
TESTBASELINE RESULT  INDEX

File Copy 4096 bufsize 8000 maxblocks 5800.0   126921.0  218.8
Shell Scripts (8 concurrent) 6.0  314.1  523.5
 =
 FINAL SCORE 338.5



on a 50G UFS partition.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-13 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #57 from rai...@ultra-secure.de ---
Well, I did do dd test, but they only write on a filesystem.

It was (back then) most likely on ZFS, with compression etc. that changed the
results.
Esp. if I just write zeros from /dev/null.

That's why I switched to dc3dd because it does the same thing on Linux and on
FreeBSD and completely eliminates the filesystem (and caching) layer as well as
any other kinds of write-optimization.


I've now created an empty UFS filesystem on my 50G volume and run unixbench on
it.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-13 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #56 from Roger Pau Monné  ---
(In reply to rainer from comment #55)
Yes, you won't see those tunables in sysctl.

Then again I'm quite lost, because you did test a plain dd, and that was
actually working fine (and yielding results in line with Linux). Can you try a
more complete benchmark, like unixbench (available in ports) without any
tunables and report it's results?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-13 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #52 from rai...@ultra-secure.de ---
Yes.
So, 60%-70% increase.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-13 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #51 from Roger Pau Monné  ---
(In reply to rainer from comment #49)
So performance is slightly better with this patch? (IIRC you where getting
17M/s and with the patch you get 26M/s)

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-11 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #49 from rai...@ultra-secure.de ---
I switched back the OS-type to FreeBSD 10 64bit.
I also booted back into a stock kernel and then the XENTIMER-LAPIC change went
through without a freeze.


I recompiled (a clean source-tree) with your patch and now I get about 26MB/s.

dmesg:
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p7 #2: Sat Feb 11 14:46:26 CET 2017
root@freebsd11:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM
3.8.0)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2593.55-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
 
Features=0x783fbff
 
Features2=0xc3ba2203
  AMD Features=0x28100800
  AMD Features2=0x1
  Structured Extended Features=0x200
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 8585740288 (8188 MB)
avail memory = 8265371648 (7882 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s)
random: unblocking device.
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0  irqs 0-47 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0x8101c970, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0:  on motherboard
cryptosoft0:  on motherboard
acpi0:  on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0:  on acpi0
cpu1:  on acpi0
hpet0:  iomem 0xfed0-0xfed003ff on acpi0
Timecounter "HPET" frequency 6250 Hz quality 950
attimer0:  port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0:  port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
isab0:  at device 1.0 on pci0
isa0:  on isab0
atapci0:  port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc220-0xc22f at device 1.1 on pci0
ata0:  at channel 0 on atapci0
ata1:  at channel 1 on atapci0
uhci0:  port 0xc200-0xc21f irq 23 at
device 1.2 on pci0
usbus0: controller did not stop
usbus0 on uhci0
pci0:  at device 1.3 (no driver attached)
vgapci0:  mem
0xf000-0xf1ff,0xf300-0xf3000fff irq 24 at device 2.0 on pci0
vgapci0: Boot video device
pci0:  at device 3.0 (no driver attached)
re0:  port 0xc100-0xc1ff mem 0xf3001000-0xf30010ff
irq 32 at device 4.0 on pci0
re0: Chip rev. 0x7480
re0: MAC rev. 0x
miibus0:  on re0
rlphy0:  PHY 0 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 02:00:2b:42:00:12
re0: netmap queues/slots: TX 1/64, RX 1/64
atkbdc0:  port 0x60,0x64 irq 1 on acpi0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0:  irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0:  port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ppc0:  port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0:  on ppc0
lpt0:  on ppbus0
lpt0: Interrupt-driven port
ppi0:  on ppbus0
vga0:  at port 0x3c0-0x3df iomem 0xa-0xb on isa0
fdc0: No FDOUT register!
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
nvme cam probe device init
usbus0: 12Mbps Full Speed USB v1.0
ugen0.1:  at usbus0
uhub0:  on usbus0
ada0 at ata0 bus 0 scbus0 target 0 lun 0
ada0:  ATA-7 device
ada0: Serial Number QM1
ada0: 16.700MB/s transfers (WDMA2, PIO 8192bytes)
ada0: 25600MB (52428800 512 byte sectors)
uhub0: 2 ports with 2 removable, self powered
ada1 at ata0 bus 0 scbus0 target 1 lun 0
ada1:  ATA-7 device
ada1: Serial Number QM2
ada1: 16.700MB/s transfers (WDMA2, PIO 8192bytes)
ada1: 51200MB (104857600 512 byte sectors)
ada2 at ata1 bus 0 scbus1 target 0 lun 0
ada2:  ATA-7 device
ada2: Serial Number QM3
ada2: 16.700MB/s transfers (WDMA2, PIO 8192bytes)
ada2: 51200MB (104857600 512 byte sectors)
cd0 at ata1 bus 0 scbus1 target 1 lun 0
cd0:  Removable CD-ROM SCSI device
SMP: AP CPU #1 

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-11 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

Roger Pau Monné  changed:

   What|Removed |Added

 Attachment #179844|0   |1
is obsolete||

--- Comment #47 from Roger Pau Monné  ---
Created attachment 179869
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179869=edit
Disable all the Xen enlightments

Pathc to disable all Xen enlightenments, this time tested.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-11 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #46 from Roger Pau Monné  ---
(In reply to rainer from comment #44)
This panic trace is very disturbing, I'm a little bit confused. Which kind of
guest are you running?

The trace shows xen_start -> hammer_time_xen and this path should _never_ be
used by a HVM guest, this is only used by PVH guests. Can you install addr2line
from ports (it's in binutils) and run the following inside of the VM:

# /usr/local/bin/addr2line -e /usr/lib/debug/boot/kernel/kernel.debug
0x81109cbb

Is also any change that you could get the full boot output from the VM by
connecting to it's serial console (and setting up the console inside of the
guest). See https://www.freebsd.org/doc/handbook/serialconsole-setup.html

Thanks.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-11 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #45 from Roger Pau Monné  ---
(In reply to rainer from comment #43)
Hm, that's certainly not good, switching to the LAPIC timer shouldn't cause the
VM to freeze, I've tried it and it works just fine. Do you see anything in the
console when the VM freezes?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #44 from rai...@ultra-secure.de ---
Created attachment 179858
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179858=edit
Screenshot of panic

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #41 from Roger Pau Monné  ---
Can you try to change the event timer and the time counter to a different one
than the Xen one:

# sysctl -w kern.timecounter.hardware=ACPI-fast
# sysctl -w kern.eventtimer.timer=LAPIC

And finally I'm also attaching a patch that actually disables all the fancy PV
stuff completely, could you also patch your kernel with it (if the above things
don't make a difference) and see if that makes a difference?

Thanks, Roger.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #40 from rai...@ultra-secure.de ---
Here:

(freebsd11 ) 1 # sysctl -a |grep xbd
hw.xbd.xbd_enable_indirect: 0
dev.xbd.2.xenstore_peer_path: /local/domain/0/backend/vbd3/13/768
dev.xbd.2.xenbus_peer_domid: 0
dev.xbd.2.xenbus_connection_state: Connected
dev.xbd.2.xenbus_dev_type: vbd
dev.xbd.2.xenstore_path: device/vbd/768
dev.xbd.2.features: write_barrier
dev.xbd.2.ring_pages: 1
dev.xbd.2.max_request_size: 40960
dev.xbd.2.max_request_segments: 11
dev.xbd.2.max_requests: 32
dev.xbd.2.%parent: xenbusb_front0
dev.xbd.2.%pnpinfo: 
dev.xbd.2.%location: 
dev.xbd.2.%driver: xbd
dev.xbd.2.%desc: Virtual Block Device
dev.xbd.1.xenstore_peer_path: /local/domain/0/backend/vbd3/13/832
dev.xbd.1.xenbus_peer_domid: 0
dev.xbd.1.xenbus_connection_state: Connected
dev.xbd.1.xenbus_dev_type: vbd
dev.xbd.1.xenstore_path: device/vbd/832
dev.xbd.1.features: write_barrier
dev.xbd.1.ring_pages: 1
dev.xbd.1.max_request_size: 40960
dev.xbd.1.max_request_segments: 11
dev.xbd.1.max_requests: 32
dev.xbd.1.%parent: xenbusb_front0
dev.xbd.1.%pnpinfo: 
dev.xbd.1.%location: 
dev.xbd.1.%driver: xbd
dev.xbd.1.%desc: Virtual Block Device
dev.xbd.0.xenstore_peer_path: /local/domain/0/backend/vbd3/13/5632
dev.xbd.0.xenbus_peer_domid: 0
dev.xbd.0.xenbus_connection_state: Connected
dev.xbd.0.xenbus_dev_type: vbd
dev.xbd.0.xenstore_path: device/vbd/5632
dev.xbd.0.features: write_barrier
dev.xbd.0.ring_pages: 1
dev.xbd.0.max_request_size: 40960
dev.xbd.0.max_request_segments: 11
dev.xbd.0.max_requests: 32
dev.xbd.0.%parent: xenbusb_front0
dev.xbd.0.%pnpinfo: 
dev.xbd.0.%location: 
dev.xbd.0.%driver: xbd
dev.xbd.0.%desc: Virtual Block Device
dev.xbd.%parent:

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #39 from Roger Pau Monné  ---
(In reply to rainer from comment #36)
There's clearly something wrong there, you are not receiving as many interrupts
as you should be, this is what I usually see when running dc3dd:

irq808: xen_et0:c0 17545 93
irq809: xen_et0:c1 73460391
irq810: xen_et0:c2 65527349
irq811: xen_et0:c3 73980394
irq814: xbd1  314436   1674

(note that xbd1 is the disk against which the dc3dd is run)

In your case this is:

irq768: xen_et0:c0   4061306 20
irq769: xen_et0:c1   1951430 10
irq773: xbd0 1652038  8
irq774: xbd1  29  0
irq775: xbd2  159503  1

Note the difference in the rate of interrupts (from 1674 in my case).

Can you also post the results of running `sysctl -a | grep xbd` inside the
FreeBSD guest?

Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #38 from rai...@ultra-secure.de ---
Well, for some it works, for some it doesn't.

The 10% I also see when writing to a RAM-disk.

I'd just like to know how I can determine where all the performance is lost.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #37 from Roger Pau Monné  ---
(In reply to Roger Pau Monné from comment #33)
So I've run the dc3dd test on a FreeBSD VM, with 4 vCPUs and 4GB of RAM,
against a block device on a spinning disk, nothing fancy, and this is what I
get:

(FreeBSD guest) # dc3dd wipe=/dev/ada1

dc3dd 7.2.641 started at 2017-02-10 16:19:54 +
compiled options:
command line: dc3dd wipe=/dev/ada1
device size: 20971520 sectors (probed),   10,737,418,240 bytes
sector size: 512 bytes (probed)
 10737418240 bytes ( 10 G ) copied ( 100% ),  125 s, 82 M/s

input results for pattern `00':
   20971520 sectors in

output results for device `/dev/ada1':
   20971520 sectors out

dc3dd completed at 2017-02-10 16:21:58 +

Then I shut down the guest, and ran the same test from Dom0 (Debian Linux 3.16)
against the same exact block device, and this is what I get:

(Linux Dom0) # dc3dd wipe=/dev/dt51-vg/test

dc3dd 7.2.641 started at 2017-02-10 16:31:34 +
compiled options:
command line: dc3dd wipe=/dev/dt51-vg/test
device size: 20971520 sectors (probed),   10,737,418,240 bytes
sector size: 512 bytes (probed)
 10737418240 bytes ( 10 G ) copied ( 100% ),  114 s, 90 M/s

input results for pattern `00':
   20971520 sectors in

output results for device `/dev/dt51-vg/test':
   20971520 sectors out

dc3dd completed at 2017-02-10 16:33:28 +

So there's a < 10M/s difference, which I think it's fine (at the end there's
always some overhead).

I cannot really explain the results that you get, but I cannot also reproduce
your entire setup here. This is using the 12.0-CURRENT snapshot from 20170203.
I'm afraid that unless we find a way for me to reproduce this, there's no way
that I can try to fix it.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #36 from rai...@ultra-secure.de ---
Hi,

I added the output.

As I said, I could give ssh access to the box, if you want.
I would need your ssh key.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #35 from rai...@ultra-secure.de ---
Created attachment 179838
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179838=edit
vmstat -ai while running dc3dd

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #34 from rai...@ultra-secure.de ---
Created attachment 179837
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179837=edit
vmstat -ai before running dc3dd

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-08 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #31 from Roger Pau Monné  ---
You should see the messages in dmesg (if any), just execute:

# dmesg

As root from the console after having run your workload.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-08 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #30 from rai...@ultra-secure.de ---
Hi,

I compiled a new kernel with this.
Where would the messages show up?

Anything special I need to add to GENERIC?
Or a flag at booting?

Sorry to sound so dumb. I stopped paying attention to building FreeBSD once
binary patches became available...

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #29 from Roger Pau Monné  ---
Created attachment 179717
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179717=edit
Initial debug patch

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #28 from Roger Pau Monné  ---
(In reply to rainer from comment #26)
Thanks!

This shows that the guest is mostly inactive (low CPU load), is this correct?

I'm attaching a patch to add some debug to blkfront, please be aware that
things might get noisy. Can you post the dmesg after running your workload with
this patch?

Thanks, Roger.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #27 from rai...@ultra-secure.de ---
(In reply to Roger Pau Monné from comment #25)
(freebsd11 ) 0 # kldload pmc
kldload: can't load pmc: module already loaded or in kernel

(freebsd11 ) 1 # kldstat 
Id Refs AddressSize Name
 1   49 0x8020 1fa7c38  kernel
 21 0x821a9000 30aec0   zfs.ko
 3   11 0x824b4000 adc0 opensolaris.ko
 41 0x824bf000 1620 accf_data.ko
 51 0x824c1000 2710 accf_http.ko
 61 0x824c4000 3a78 cc_htcp.ko
 71 0x82619000 587b fdescfs.ko
 81 0x8261f000 3710 ums.ko
 91 0x82623000 665d nullfs.ko
101 0x8262a000 adec tmpfs.ko
111 0x82635000 1bb42hwpmc.ko
121 0x82651000 848  dtraceall.ko
139 0x82652000 3d890dtrace.ko
141 0x8269 4860 dtmalloc.ko
151 0x82695000 5aef dtnfscl.ko
161 0x8269b000 6832 fbt.ko
171 0x826a2000 585befasttrap.ko
181 0x826fb000 172e sdt.ko
191 0x826fd000 cf3d systrace.ko
201 0x8270a000 cd44 systrace_freebsd32.ko
211 0x82717000 535e profile.ko

Do you have an idea what has to be done on XenServer to enable PMU?

In any case, I cannot do any configuration on Dom0. My coworker would have to
do that tomorrow (CET)


Best Regards
Rainer

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #26 from rai...@ultra-secure.de ---
Created attachment 179716
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=179716=edit
flamegraph dtrace

This is running the example on Brendan's page with dtrace. 
While the VM was processing a dc3dd wipe command.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #24 from rai...@ultra-secure.de ---
ok,

(freebsd11 ) 64 # pmccontrol -L
SOFT
CLOCK.PROF
CLOCK.HARD
CLOCK.STAT
LOCK.FAILED
PAGE_FAULT.ALL
PAGE_FAULT.READ
PAGE_FAULT.WRITE


Is there a way to get virtualized hardware performance counters in a DomU?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #25 from Roger Pau Monné  ---
IIRC this was working fine last time I've tried. Have you loaded the pmc module
(kldload pmc), and which CPU are you using? Note that you also need to enable
the PMU support in Xen [0] by passing vpmu=1 on the Xen command line.

PMC.HASWELL(3) lists RESOURCE_STALLS.ANY as a valid event. If that doesn't work
(or is too complicated to setup) I would try with dtrace, and let's see what we
get.

[0] http://xenbits.xenproject.org/docs/unstable/misc/xen-command-line.html

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #23 from rai...@ultra-secure.de ---
Which event specifier should I use?

I can't even run the sample:

(freebsd11 ) 0 # pmcstat –S RESOURCE_STALLS.ANY -O out.pmcstat sleep 10
pmcstat: [options] [commandline]
 Measure process and/or system performance using hardware
 performance monitoring counters.
 Options include:
 -C  (toggle) show cumulative counts
 -D path create profiles in directory "path"
 -E  (toggle) show counts at process exit
 -F file write a system-wide callgraph (Kcachegrind format) to
"file"
 -G file write a system-wide callgraph to "file"
 -M file print executable/gmon file map to "file"
 -N  (toggle) capture callchains
 -O file send log output to "file"
 -P spec allocate a process-private sampling PMC
 -R file read events from "file"
 -S spec allocate a system-wide sampling PMC
 -T  start in top mode
 -W  (toggle) show counts per context switch
 -a file print sampled PCs and callgraph to "file"
 -c cpu-list set cpus for subsequent system-wide PMCs
 -d  (toggle) track descendants
 -e  use wide history counter for gprof(1) output
 -f spec pass "spec" to as plugin option
 -g  produce gprof(1) compatible profiles
 -k dir  set the path to the kernel
 -l secs set duration time
 -m file print sampled PCs to "file"
 -n rate set sampling rate
 -o file send print output to "file"
 -p spec allocate a process-private counting PMC
 -q  suppress verbosity
 -r fsroot   specify FS root directory
 -s spec allocate a system-wide counting PMC
 -t process-spec attach to running processes matching "process-spec"
 -v  increase verbosity
 -w secs set printing time interval
 -z depthlimit callchain display depth

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #22 from Roger Pau Monné  ---
Hello,

I don't have much time to look into this right now, could you try to create a
flamegraph [0] of this workload, this way we might be able to identify the
bottleneck(s). If possible you should create the flamegraph with pmcstat
instead of dtrace, the result is going to be much more accurate (specially when
running inside of a VM).

[0] http://www.brendangregg.com/blog/2015-03-10/freebsd-flame-graphs.html

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-06 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #21 from rai...@ultra-secure.de ---
Still a problem on FreeBSD 12:

root@f12test:~ # dc3dd wipe=/dev/ada1

dc3dd 7.2.641 started at 2017-02-06 10:12:31 +0100
compiled options:
command line: dc3dd wipe=/dev/ada1
device size: 104857600 sectors (probed),   53,687,091,200 bytes
sector size: 512 bytes (probed)
  1153433600 bytes ( 1.1 G ) copied (  2% ),  131 s, 8.4 M/s

input results for pattern `00':
   2252800 sectors in

output results for device `/dev/ada1':
   2252800 sectors out

dc3dd aborted at 2017-02-06 10:14:42 +0100

root@f12test:~ # uname -a
FreeBSD f12test 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r313113: Fri Feb  3
01:47:24 UTC 2017 r...@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC
 amd64

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2017-02-03 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #20 from rai...@ultra-secure.de ---
Interestingly enough, even when the backend storage is an SSD-backed ScaleIO
volume (PCIe NVMe), it's not faster.

Linux is faster on SSDs.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-11-16 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #19 from rai...@ultra-secure.de ---
Updating to ScaleIO 2.0.3 (and all the latest Hotfixes of XenServer 6.5)
doesn't make a difference.

How would one debug this problem?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-21 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #18 from rai...@ultra-secure.de ---
Hi,

thanks - but it does not make a notable difference.
Neither for the dc3dd -wipe test, nor for my real-world testcase.

I can create a tenant on CloudStack, so you can try it yourself - if you want.
But I have no problem running tests, commands etc. for you, either - I can
devote (almost) as much time on this as it takes.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-16 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #16 from rai...@ultra-secure.de ---
OK, thanks for the clarification.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-16 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

Roger Pau Monné  changed:

   What|Removed |Added

 CC||roy...@freebsd.org

--- Comment #15 from Roger Pau Monné  ---
The fact that you get "ada" or "xbd" devices depend on what you put in the
guest configuration file. If the disk is attached as a "xvd" it will show up as
"xbd" in FreeBSD, and if it's attached as "hd" it will show up as "ada". As
long as you see something like:

xbd0: 51200MB  at device/vbd/832 on xenbusb_front0
xbd0: attaching as adaX

In dmesg it means it's using the PV disks.

I will try to look into this, but it's not going to be now (I hope I will be
able to get to it by the end of the month).

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-15 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #13 from Sydney Meyer  ---
I'm no expert in Cloudstack but perhaps something with the vm template might be
off. Did you tried / is there a possibilty to install FreeBSD with some Linux
template?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-15 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #11 from rai...@ultra-secure.de ---
Well, I chose "FreeBSD 10 64bit" as OS-type.

In dmesg, I see:

xbd0: attaching as ada0
xbd0: features: write_barrier
xbd0: synchronize cache commands enabled.

sysctl -a |grep xen
kern.vm_guest: xen
device  xenpci
vfs.pfs.vncache.maxentries: 0
dev.xctrl.0.%parent: xenstore0
dev.xenbusb_back.0.%parent: xenstore0
dev.xenbusb_back.0.%pnpinfo: 
dev.xenbusb_back.0.%location: 
dev.xenbusb_back.0.%driver: xenbusb_back
dev.xenbusb_back.0.%desc: Xen Backend Devices
dev.xenbusb_back.%parent: 
dev.xn.0.xenstore_peer_path: /local/domain/0/backend/vif/109/0
dev.xn.0.xenbus_peer_domid: 0
dev.xn.0.xenbus_connection_state: Connected
dev.xn.0.xenbus_dev_type: vif
dev.xn.0.xenstore_path: device/vif/0
dev.xn.0.%parent: xenbusb_front0
dev.xbd.0.xenstore_peer_path: /local/domain/0/backend/vbd3/109/768
dev.xbd.0.xenbus_peer_domid: 0
dev.xbd.0.xenbus_connection_state: Connected
dev.xbd.0.xenbus_dev_type: vbd
dev.xbd.0.xenstore_path: device/vbd/768
dev.xbd.0.%parent: xenbusb_front0
dev.xenbusb_front.0.%parent: xenstore0
dev.xenbusb_front.0.%pnpinfo: 
dev.xenbusb_front.0.%location: 
dev.xenbusb_front.0.%driver: xenbusb_front
dev.xenbusb_front.0.%desc: Xen Frontend Devices
dev.xenbusb_front.%parent: 
dev.xenstore.0.%parent: xenpci0
dev.xenstore.0.%pnpinfo: 
dev.xenstore.0.%location: 
dev.xenstore.0.%driver: xenstore
dev.xenstore.0.%desc: XenStore
dev.xenstore.%parent: 
dev.xenpci.0.%parent: pci0
dev.xenpci.0.%pnpinfo: vendor=0x5853 device=0x0001 subvendor=0x5853
subdevice=0x0001 class=0x01
dev.xenpci.0.%location: pci0:0:3:0 handle=\_SB_.PCI0.S18_
dev.xenpci.0.%driver: xenpci
dev.xenpci.0.%desc: Xen Platform Device
dev.xenpci.%parent: 
dev.xen_et.0.%parent: nexus0
dev.xen_et.0.%pnpinfo: 
dev.xen_et.0.%location: 
dev.xen_et.0.%driver: xen_et
dev.xen_et.0.%desc: Xen PV Clock
dev.xen_et.%parent: 
dev.xen.xsd_kva: 18446735281894703104
dev.xen.xsd_port: 17
dev.xen.balloon.high_mem: 0
dev.xen.balloon.low_mem: 0
dev.xen.balloon.hard_limit: 18446744073709551615
dev.xen.balloon.driver_pages: 0
dev.xen.balloon.target: 1048576
dev.xen.balloon.current: 1047552


If I choose "Other 64bit", I get the same (at least in FreeBSD 11RC2, which is
what I could quickly switch-over).

xenbusb_back0:  on xenstore0
xbd0: 51200MB  at device/vbd/832 on xenbusb_front0
xbd0: attaching as ada1
xbd0: features: write_barrier
xbd0: synchronize cache commands enabled.
xn0: backend features: feature-sg feature-gso-tcp4


What would you expect?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-15 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

Sydney Meyer  changed:

   What|Removed |Added

 CC||meyer.syd...@gmail.com

--- Comment #10 from Sydney Meyer  ---
Your disk shows up as using the ada driver, aren't you using some type of
emulated disk device instead of the paravirtualized xbd block device?

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-15 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #9 from rai...@ultra-secure.de ---
I noticed that the local values are very unstable.
Also, I don't really have access to the Xen side (yet).

I will look into how I can debug this further. I'm merely a consumer of it at
this point.

CloudStack does not support Xen-Server 7, unfortunately. And it looks like it's
going to be a while before that happens.

At your request, I will contact you via eMail once I have more debugging
information.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-15 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #8 from k...@pielorz.com ---
(In reply to rainer from comment #7)

Our config here is very similar - HP Proliant DL380 Gen -9- though, with local
SAS disks - which we use for 'Local Storage' for XenServer, and then iSCSI off
to a Synology NAS we use for primary storage of our VM's.

>On local storage, the realworld-test is even slower.

This troubles me, as we don't see that - but we are running the test on
un-contended local storage (you don't say if you are).

The times you show for that last dd run - seem to vary quite a lot:

1342177280 bytes transferred in 22.239869 secs (60350053 bytes/sec)
1342177280 bytes transferred in 38.072567 secs (35253133 bytes/sec)
1342177280 bytes transferred in 5.782933 secs (232092820 bytes/sec)
1342177280 bytes transferred in 7.891797 secs (170072452 bytes/sec)
1342177280 bytes transferred in 12.598706 secs (106532947 bytes/sec)
1342177280 bytes transferred in 7.917661 secs (169516892 bytes/sec)
13421772800 bytes transferred in 147.423047 secs (91042568 bytes/sec)

Especially that last one. How busy is the node? / local disks?

At this stage I'd be tempted to re-run the tests while on FreeBSD looking at
the output of something like:

  iostat -x 1

(which will show what FreeBSD thinks the disk service time, % busy etc. are) -
and on Dom0 providing the local storage running:

  iostat -x 1 | egrep -e "(^Device)|(^sd*)"

Which will do the equivalent for Xen.

Things to look out for are high % busy, and service time / queue depths.

At this point I'm not sure filling the ticket with reams of debug / command
output is helping much - it might be better to close this ticket - and revert
to email to see if we can narrow things down to a possible cause, before
re-opening the ticket again with more specific information.

-Karl

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-15 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #7 from rai...@ultra-secure.de ---
The hardware is HP DL380 Gen 8 servers with 600 or 900 GB SAS disks, running
off a HW RAID controller.

On local storage, the realworld-test is even slower.

I can't run the dc3dd test here and it's apparently a bit more complicated now
to create VMs from a template on local storage now that we have completely
eliminated it from our offerings.

(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480 
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 22.239869 secs (60350053 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.04s user 6.20s system 28% cpu
22.255 total
(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 38.072567 secs (35253133 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.05s user 24.30s system 63%
cpu 38.374 total
(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 5.782933 secs (232092820 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.05s user 2.27s system 38% cpu
6.021 total
(server ) 0 # 
(server ) 0 # 
(server ) 0 # 
(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 7.891797 secs (170072452 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.00s user 2.26s system 27% cpu
8.141 total
(server ) 0 # 
(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 12.598706 secs (106532947 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.02s user 2.37s system 18% cpu
12.845 total
(server ) 0 # 
(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 7.917661 secs (169516892 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.03s user 2.23s system 27% cpu
8.144 total

(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=204800
204800+0 records in
204800+0 records out
13421772800 bytes transferred in 147.423047 secs (91042568 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=204800  0.18s user 22.60s system 15%
cpu 2:27.69 total


dc3dd is just a tool to securely wipe a disk.
I found, because it takes the filesystem out of the equation, that it's a nice
benchmarking tool.
Also, the results of dc3dd correlate directly with the results of the testcase
(in PHP) that the customer has built.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-14 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #6 from k...@pielorz.com ---
(In reply to rainer from comment #5)

Ok, so 'like for like' test see's better performance.

I'm not overly familiar with dc3dd - maybe it's using a "really small block
size" - and  on your setup, this is causing an issue (maybe look at the ssz /
bufsz options)

Do you have any 'local storage' on XenServer (or can you create some) - i.e.
something you can map to a FreeBSD VM - and repeat both the dc3dd and dd test
on - that is backed by a local SATA / SAS disk on XenServer (i.e. not going
through ScaleIO?)

-Karl

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-14 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #5 from rai...@ultra-secure.de ---
10.3-RELEASE-p5:

(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 1.769942 secs (758317078 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.02s user 1.75s system 99% cpu
1.775 total
(server ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=204800
204800+0 records in
204800+0 records out
13421772800 bytes transferred in 17.266468 secs (777331701 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=204800  0.15s user 17.06s system 99%
cpu 17.271 total

This is ZFS.
Probably due to compression on.


root@other-server:/srv# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 14.04.4 LTS
Release:14.04
Codename:   trusty
root@other-server:/srv# time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes (1.3 GB) copied, 2.5559 s, 525 MB/s

real0m2.571s
user0m0.041s
sys 0m2.125s

root@other-server:/srv# time dd if=/dev/zero of=test.dat bs=64k count=204800
204800+0 records in
204800+0 records out
13421772800 bytes (13 GB) copied, 93.6892 s, 143 MB/s

real1m33.940s
user0m0.219s
sys 0m26.136s


root@yet-another-server:/srv# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 16.04.1 LTS
Release:16.04
Codename:   xenial
root@yet-another-server:/srv# time dd if=/dev/zero of=test.dat bs=64k
count=20480
20480+0 records in
20480+0 records out
1342177280 bytes (1.3 GB, 1.2 GiB) copied, 1.62124 s, 828 MB/s

real0m1.652s
user0m0.004s
sys 0m1.616s
root@yet-another-server:/srv# time dd if=/dev/zero of=test.dat bs=64k
count=204800
204800+0 records in
204800+0 records out
13421772800 bytes (13 GB, 12 GiB) copied, 100.348 s, 134 MB/s

real1m40.711s
user0m0.172s
sys 0m25.004s


So, in this particular test, it's actually faster.
But I can assure you, in practical use, it's not.


10.3-RELEASE-p7, UFS:

(freebsd-srv2 ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=20480
20480+0 records in
20480+0 records out
1342177280 bytes transferred in 8.746548 secs (153452229 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=20480  0.02s user 1.65s system 19% cpu
8.756 total
(freebsd-srv2 ) 0 # time dd if=/dev/zero of=test.dat bs=64k count=204800
204800+0 records in
204800+0 records out
13421772800 bytes transferred in 99.078364 secs (135466233 bytes/sec)
dd if=/dev/zero of=test.dat bs=64k count=204800  0.22s user 18.20s system 18%
cpu 1:39.30 total

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-14 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

--- Comment #3 from rai...@ultra-secure.de ---
Our hardware is local disks (not SSDs) networked via ScaleIO.

I've run a dd from one disk of a VM to another one and it was very, very slow.

OS: "Other (64bit)" (which is actually a little bit faster than choosing
"FreeBSD 10"

(freebsd11 ) 0 # dc3dd wipe=/dev/ada1

dc3dd 7.2.641 started at 2016-08-18 13:24:02 +0200
compiled options:
command line: dc3dd wipe=/dev/ada1
device size: 104857600 sectors (probed), 53,687,091,200 bytes
sector size: 512 bytes (probed)
53687091200 bytes ( 50 G ) copied ( 100% ), 3084 s, 17 M/s

input results for pattern `00':
104857600 sectors in

output results for device `/dev/ada1':
104857600 sectors out

dc3dd completed at 2016-08-18 14:15:26 +0200


The question remains: why is this and what can one do?

One of our customers has an application-workload (php+mysql) that takes 3s to
process on an Ubuntu 16 VM with 2 vCPUs and 8GB RAM.
The FreeBSD VM is completely unusable for this because it's bogged down to a
halt, regardless of how many vCPUs and RAM I give it.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-14 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

Daniel Ylitalo  changed:

   What|Removed |Added

 CC||dan...@blodan.se

--- Comment #2 from Daniel Ylitalo  ---
We use XenServer 7 overhere and we can't run FreeBSD vm's on it unfortunately.
(Tried both 11-RC1 and 10.3) (We use local storage)

Just a regular "portsnap fetch extract" takes like 10 times the time it does on
a baremetal server.

So when someone requests a FreeBSD server they get a baremetal one, all
variants of linux gets a vm.

I'm happy to provide a vm if someone is eager to troubleshoot this, I've tried
all solutions out there in the xen I/O mail threads/forum posts but none did
the trick.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"


[Bug 212681] I/O is slow for FreeBSD DOMu on XenServer

2016-09-14 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212681

k...@pielorz.com changed:

   What|Removed |Added

 CC||k...@pielorz.com

--- Comment #1 from k...@pielorz.com ---
Hi,

We run XenServer 6.5 and 7 here - with a range of FreeBSD versions (mostly 10.x
now). We've not noticed any really low I/O performance.

How are you running dc3dd? - So I can try and install / replicate this here.

We have seen that I/O behaves differently under XenServer than it does on bare
metal (which is obvious - i.e. local SATA SSD vs. Multipath iSCSI [or similar])
you will see differences (even just with 'mapped through XenServer' "Local
Storage").

It could be dc3dd / the stuff you're running is a particularly 'bad case' for
the I/O. e.g. In an equally 'synthetic' test - a regular "dd if=/dev/zero
of=test.dat bs=64k count=10240" on our our test pool here (which has two paths
iSCSI over Gigabit) we get around 220Mbyte/sec to a FreeBSD DomU.

That may well be a best case - and you may well have hit some worst cases.

-Karl

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"