Re: FreeBSD/Xen suspend/resume

[Roger Pau Monné]

On Sat, Jan 19, 2019 at 05:00:56PM -0600, Matthew Grooms wrote:
> On 1/19/2019 4:59 AM, Uni Gaia wrote:
> > Does anyone know if suspend/resume works for a FreeBSD/Xen dom0 ? domU?
> 
> Hey Uni,
> 
> I'm not sure how many folks are subscribed to the xen@ mailing list. You may
> get a better answer here:
> 
> FreeBSD dom0 is based on the Xen PVH virtualization mode which has seen two
> major iterations. PVHv1 has been in the tree for some time. I can't remember
> when PVHv2 hit the tree but I'm pretty sure it supplanted v1. It might be in
> 12-RELEASE? Check the release notes. You probably want v2 as it's more
> feature complete. The port was done by Roger who, I believe, works for
> Cirtix Research. My point being that the port should be of very high quality
> and he/they probably want to hear if something isn't working with PVHv2.If
> I'm misrepresented anything, hopefully he'll chime in here.

Thanks for the detailed description! PVHv1 was more similar to classic
PV, while PVHv2 is more similar to HVM.

PVHv1 support has been removed from upstream Xen since 4.10 IIRC, so I
would strongly recommend to use PVHv2. In fact that's the only way to
use the 4.11 Xen packages in the ports tree.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: multi-vCPU networking issues as client OS under Xen

[Roger Pau Monné]

On Mon, Feb 19, 2018 at 10:42:08AM +, Laurence Pawling wrote:
> >When using >1 vCPUs can you set hw.xn.num_queues=1 on
> >/boot/loader.conf and try to reproduce the issue?
> >
> >I'm afraid this is rather related to multiqueue (which is only used
> >if >1 vCPUs).
> >
> >Thanks, Roger.
> 
> Roger - thanks for your quick reply, this is confirmed. Setting 
> hw.xn.num_queues=1 on the server VM when vCPUs > 1 prevents the issue.

I've also been told that in order to discard this being a XenServer
specific issue you should execute the following on Dom0 and reboot the
server:

# xe-switch-network-backend bridge

And then try to reproduce the issue again with >1 vCPUs (and of course
removing the queue limit in loader.conf)

> For reference, please can you comment on the performance impact of this?

I'm afraid I don't have any numbers.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: multi-vCPU networking issues as client OS under Xen

[Roger Pau Monné]

On Mon, Feb 19, 2018 at 09:58:30AM +, Laurence Pawling via freebsd-xen 
wrote:
> Hi all,
> 
>  
> 
> I’m wondering if anyone here has seen this issue before, I’ve spent the last 
> couple of days troubleshooting:
> 
>  
> 
> Platform:
> 
> Host: XenServer 7.0 running on 2 x E2660-v4, 256GB RAM
> 
> Server VM: FreeBSD 11 (tested on 11.0-p15 and 11.1-p6), 2GB RAM (also tested 
> with 32GB RAM), 1x50GB HDD, 1 x NIC, 2 or more vCPUs in any combination (2 
> sockets x 1 core, 1 socket x 2 cores, …)
> 
> Client VM: FreeBSD 11, any configuration of vCPUs, RAM and HDD.
> 
>  
> 
> Behaviour:
> 
> Sporadic interruption of TCP sessions when utilising the above machine as a 
> “server” with “clients” connecting. Looking into the communication with 
> pcap/Wireshark, you see a TCP Dup Ack sent from both ends, followed by the 
> client sending an RST packet, terminating the TCP session. We have also seen 
> evidence of the client sending a Keepalive packet, which is ACK’d by the 
> server before the RST is sent from the client end.
> 
>  
> 
> To recreate:
> 
> On the above VM, perform a vanilla install of nginx:
> 
> pkg install nginx
> 
> service nginx onestart
> 
> Then on a client VM (currently only tested with FreeBSD), run the following 
> (or similar):
> 
> for i in {1..1}; do if [ $(curl -s -o /dev/null -w "%{http_code}" 
> http://10.2.122.71) != 200 ] ; then echo "error"; fi; done
> 
> When vCPUs=1 on the server, I get no errors, when vCPUs>1 I get errors 
> reported. The frequency of errors *seems* to be proportional to the number of 
> vCPUs, but they are sporadic with no clear periodicity or pattern, so that is 
> just anecdotal. Also, the problem seems by far the most prevalent when 
> communicating between two VMs on the same host, in the same VLAN. Xen still 
> sends packets via the switch rather than bridging internally between the 
> interfaces.

When using >1 vCPUs can you set hw.xn.num_queues=1 on
/boot/loader.conf and try to reproduce the issue?

I'm afraid this is rather related to multiqueue (which is only used
if >1 vCPUs).

Thanks, Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Xen Dom0

[Roger Pau Monné]

Hello,

Sorry for the delay, I don't read freebsd-virt as often as I should.
There's also freebsd-xen@ for FreeBSD/Xen specific issues (although
posting here is fine).

On Wed, Jun 28, 2017 at 11:44:27PM -0500, Larry Rosenman wrote:
> Any ideas?

When was the last time that you updated world (or more specifically
/boot/)?

Could you please attempt the following and try again:

cd sys/boot/ && make && make install

I've just tested the port with the current loader code and seems to
work fine.

There was a brief period of time when HEAD couldn't boot a Xen kernel,
IIRC it's ~3month ago or so. That was fixed, but you might just happen
to have a loader from that time.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 11 Stable on a Xen :: bridge0 crashing server

[Roger Pau Monné]

On Fri, Jan 20, 2017 at 07:20:15PM +0300, Andrey V. Elsukov wrote:
> On 20.01.2017 18:57, Trond Endrestøl wrote:
> > > Here is the situation:
> > > I have a VPS server from a well reputed provider (and they deserve the
> > > reputation), running FreeBSD 11 stable x64 under Xen Full Virtualization
> > > (HVM). I have the xn0 interface which is working fine. I intend to use 
> > > VIMAGE,
> > > so I compiled the kernel, added cloned_interface="bridge0" and restarted 
> > > the
> > > server. But as soon as I am attaching the xn0 to bridge0, the kernel is
> > > panicking and the server restarting.
> > > Any suggestion/pointer/test-instruction is highly appreciated.
> > 
> > The code crashes at line 427 of sys/netinet/if_ether.c:
> > 
> >   ARPSTAT_INC(txrequests);
> > 
> > See
> > https://svnweb.freebsd.org/base/stable/11/sys/netinet/if_ether.c?view=annotate#l427
> > 
> > stable/11 has problems accounting the outgoing octets of any xn
> > interface, although this isn't connected to your case.
> > 
> > Just to rule out any uncertainty, try this patch:
> > 
> > https://svnweb.freebsd.org/base?view=revision=308126
> > 
> > See PR 213439,
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213439
> > 
> > Note, I'm not a developer nor a committer, just a humble sysadmin.
> 
> This problem is unrelated. ARP statistics is global and isn't related to
> some specific interface. IMHO, the kernel panics due to missing VNET
> context. As I see from the code in sys/dev/xen, it is not capable with
> VIMAGE.

I cannot really look into this right now due to lack of time, but I'm more than
happy to review/apply patches in order to fix this.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: HEADS UP: Imported Xen 4.7: no blkback

[Roger Pau Monné]

On Mon, Jun 13, 2016 at 07:26:30PM +, Marcin Cieslak wrote:
> On Mon, 13 Jun 2016, Roger Pau Monné wrote:
> 
> > On Fri, Jun 10, 2016 at 09:38:59PM +, Marcin Cieslak wrote:
> > > "?" does not work - it mostly causes a panic, the console is slow, but I 
> > > managed
> > > to switch it to /dev/ada0p2, dmesg below:
> > 
> > This has now been reverted, so when I import the new RC this should be 
> > fixed 
> > and you won't need to change anything.
> 
> I am confused now - so with a new Xen kernel (not yet in ports) I can use
> /dev/xbd* devices again? They are in fact missing - xbd driver says 
> "attaching as ada0"
> and I can mount it only as /dev/ada
> 
> > It seems like Windows PV drivers don't attach at all, or are you running 
> > Windows without the PV drivers?
> 
> Yes, I have. Those Windows partitions used to work properly without changes
> under xen 4.5. But we are too early - the problem is that even ovmf
> does not se them drives now, this is before Windows boots.
> 
> > Since you mention that the console is very slow, if you run 'top' on Dom0, 
> > do you see any process (eg: qemu) taking a lot of CPU time?
> 
> Yes,
> 
>  1635 root  7 1000   241M   101M RUN  7:16  91.14% 
> qemu-system-i386
> 
> or more
> 
> > Also, do you see Dom0 consuming a lot of CPU if you run 'xentop'?
> 
>   Domain-0 -r446  100.04194300   25.2   no limit   n/a
>  10000000  0  
> 00
> 

Hello,

I've just imported Xen 4.7.0-rc6 into the ports tree, could you give it a 
try when you have a moment?

FWIW, I'm going on vacations until the 4th of July, and I will be mostly 
AFK.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: HEADS UP: Imported Xen 4.7: no blkback

[Roger Pau Monné]

On Fri, Jun 10, 2016 at 09:38:59PM +, Marcin Cieslak wrote:
> On Thu, 9 Jun 2016, Roger Pau Monné wrote:
> 
> > On Thu, Jun 09, 2016 at 12:16:59AM +, Marcin Cieslak wrote:
> > > On Fri, 3 Jun 2016, Roger Pau Monné wrote:
> > > 
> > > > One of the more relevant changes in 4.7 regarding FreeBSD is the 
> > > > support for 
> > > > block hotplug scripts. This means that we now have the option to use 
> > > > backends different than simple block or regular files, provided that 
> > > > someone 
> > > > writes the proper hotplug scripts to attach them (I've heard there are 
> > > > some 
> > > > iSCSI hotplug scripts around). This however requires changes in 
> > > > blkback, so 
> > > > if you plan to use the Xen 4.7 port, please make sure that you are 
> > > > running a 
> > > > kernel that contains revision r301269 (or any later version). The same 
> > > > also 
> > > 
> > > I am running it with r301685 and the HVM guests have some trouble with
> > > block devices.
> > > 
> > > SeaBIOS does not find /dev/zvol/zroot/freebsd1,raw,xvda,w to boot FreeBSD
> > > from, after chaging to "hda" I get up to the kernel mountroot prompt
> > > (Xen block devices seem to be detected in dmesg).
> > 
> > Yes, this is intentional, see:
> > 
> > https://marc.info/?l=xen-devel=144482080812353
> 
> those guests worked fine with 4.5, that's why I am surprised.
> 
> xbd0 and xbd1 show up in dmesg, mouting root from xbd0p2 fails, but 
> ada0p2 seems to work.
> 
> I remember that during my previous attempts ZFS ate most of
> my machine's memory (dom0 was too small I think) and the
> symptom was very similar if not identical.
> 
> One thing which struck me is that I was able to fully but
> one Linux HVM domU. I am also toying with OpenFirmware
> which boots from floppy as a HVM guest.
> 
> > Have you checked if you need to change your /etc/fstab to correctly point 
> > to the new device? Does FreeBSD correctly list the disk(s) at the mountroot 
> > prompt when issuing a "?" command?
> 
> "?" does not work - it mostly causes a panic, the console is slow, but I 
> managed
> to switch it to /dev/ada0p2, dmesg below:

This has now been reverted, so when I import the new RC this should be fixed 
and you won't need to change anything.
 
> Copyright (c) 1992-2016 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>   The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 11.0-CURRENT #1 r298620: Tue Apr 26 13:21:50 UTC 2016
> r...@o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 
> 3.8.0)
> WARNING: WITNESS option enabled, expect reduced performance.
> VT(vga): text 80x25
> XEN: Hypervisor version 4.7 detected.
> CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.08-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
>   
> Features=0x17c3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,HTT>
>   
> Features2=0x9fba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
>   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>   AMD Features2=0x1
>   XSAVE Features=0x1
> Hypervisor: Origin = "XenVMMXenVMM"
> real memory  = 2130706432 (2032 MB)
> avail memory = 2018213888 (1924 MB)
> Event timer "LAPIC" quality 400
> ACPI APIC Table: 
> WARNING: L1 data cache covers less APIC IDs than a core
> 0 < 1
> WARNING: L2 data cache covers less APIC IDs than a core
> 0 < 1
> WARNING: L3 data cache covers less APIC IDs than a core
> 0 < 1
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> FreeBSD/SMP: 1 package(s) x 2 core(s)
> random: unblocking device.
> ioapic0: Changing APIC ID to 1
> MADT: Forcing active-low polarity and level trigger for SCI
> ioapic0  irqs 0-47 on motherboard
> random: entropy device external interface
> kbd1 at kbdmux0
> netmap: loaded module
> module_register_init: MOD_LOAD (vesa, 0x80f0ffb0, 0) error 19
> vtvga0:  on motherboard
> cryptosoft0:  on motherboard
> acpi0:  on motherboard
> acpi0: Power Button (fixed)
> acpi0: Sleep Button (fixed)
> cpu0:  on acpi0
> cpu1:  on acpi0
> hpet0:  iomem 0xfed0-0xfed003ff on acpi0
> Timecounter "HPET" frequency 6250 Hz quality 950
> atti

Re: HEADS UP: Imported Xen 4.7 and blkback changes - domU respawning on_crash

[Roger Pau Monné]

On Wed, Jun 08, 2016 at 10:35:22PM +, Marcin Cieslak wrote:
> On Wed, 8 Jun 2016, Marcin Cieslak wrote:
> 
> > On Fri, 3 Jun 2016, Roger Pau Monné wrote:
> > 
> > > Hello,
> > > 
> > > First of all, this message is only relevant to those that use FreeBSD as 
> > > Dom0 (host), not as a DomU (guest), so don't panic.
> > > 
> > > I've imported the latest Xen version (4.7-rc4) into the ports tree, it's 
> > > still not the final version, but it's quite close, so we better start 
> > > testing it to make sure it works fine with FreeBSD.
> 
> One issue maybe unrelated to FreeBSD:
> 
> This domain:
> 
> builder = "hvm"
> memory = 4096
> vcpus = 2
> name = "Windows2016"
> disk = [
> '/dev/zvol/zroot/windows0,raw,hda,w',
> '/dev/zvol/zroot/vs2013,raw,hdb,w',
> #'/root/win/install.iso,raw,hdc:cdrom,r'
> ]
> boot = "c" # Boot to hard disk image
> vnc = 2
> #vnclisten = "0.0.0.0"
> usbdevice = 'tablet'
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
> acpi = 1
> bios = 'ovmf'
> vif = [ 'bridge=bridge0,mac=00:16:3e:5d:0d:48' ]
> videoram=16
> vga = "stdvga"
> 
> crashes because I didn't have ovmf image:
> 
> (d203) HVM Loader
> (d203) Detected Xen v4.7.0-rc
> (d203) Xenbus rings @0xfeffc000, event channel 1
> (d203) Unknown BIOS ovmf, no ROM image found
> (d203) *** HVMLoader bug at hvmloader.c:229
> (d203) *** HVMLoader crashed.
> 
> But I seem unable to kill it with "xl destroy" - it keeps
> respawning again:
> 
> Windows2016211  4079 1 --p---   
> 0.0
> Windows2016213  4096 1 --psc-   
> 0.0
> (disappears)
> Windows2016221  4096 1 --psc-   
> 0.0
> (null) 221   147 1 --psc-   
> 0.0
> ...
> ...
> 
> I have finally managed to snatch it by issuing this a few times, after
> changing the "on_crash" to 'destroy':
> 
> # xl config-update Windows2016 xen/windows-run.cfg
> WARNING: xl now has better capability to manage domain configuration, avoid 
> using this command when possible
> setting dom243 configuration

The problem is that the domain crashed so early on boot that you weren't 
able to destroy it, and kept rebooting due to the "on_crash = 'restart'" 
option. IIRC I've also used the following hacky rune in order to terminate 
this kind of domains: "while [ 1 ]; do xl destroy ; done", but your 
solution seems better.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD/XEN Dom XAPI

[Roger Pau Monné]

On Sat, May 14, 2016 at 04:07:49AM +0200, Outback Dingo wrote:
> seems there is no management interface for XEN/FreeBSD via XAPI, any
> suggestions or pointers for getting XOA, or XenCenter to manage the
> FreeBSD/XEN boxes ?? just a quick inquiry.

You would have to get XAPI working on FreeBSD before trying to use XOA or 
XenCenter, those are specific products that only work with XenServer.

I don't know how much effort it would take to get XAPI working on FreeBSD, 
but I'm quite sure it's not a trivial task.

OTOH, there are other managers that you can probably use with FreeBSD/Xen, 
like OpenStack, OpenNebula or similar managers that are designed to work 
with the open source Xen Project toolstack (xl). Or you can also install 
libvirt and try to use virsh or any manager that has libvirt bindings.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting Solaris 11.3 (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)

[Roger Pau Monné]

On Thu, May 05, 2016 at 10:20:51PM +, Marcin Cieslak wrote:
> On Thu, 5 May 2016, Roger Pau Monné wrote:
> 
> > On Sat, Apr 30, 2016 at 08:56:54PM +, Marcin Cieslak wrote:
> > > > Certainly. I assumed that you meant it crashed the VM, not the whole 
> > > > host. 
> > > > Can you please provide the trace of the crash?
> > > 
> > > Apologies, forgot to let Xen keep VGA:
> > > 
> > > http://marcincieslak.com/tmp/xencrash.png
> > > 
> > > Manual OCR:
> > > 
> > > FreeBSD/amd64 (o.saper.info) (xc0)
> > > 
> > > login: (XEN) vmx.c:2464:d0v0 EPT violation 0x182 (-w-/---), gpa 
> > > 0x010178f000
> > 
> > Hello,
> > 
> > I've been able to debug this and found the issue. I have two patches that 
> > should be applied to FreeBSD in order to fix it, they can be found at:
> > 
> > https://people.freebsd.org/~royger/privcmd_fixes/
> > 
> > Could you please give them a try?
> 
> > I have however been unable to boot a Solaris 11.3 guest under PV mode, but 
> > at least the host is not crashing anymore :).
> 
> Thanks, I can confirm that the patches fix the crash for me.
> The half-life of PV Solaris is still under the microsecond.
> 
> Btw. I have managed to get gdbsx going by replacing "/proc/xen/privcmd"
> with "/dev/xen/privcmd".
> 
> Indeed, the state of Solaris is pretty bad:
> 
> - Solaris 11.3 under HVM boots, but the installer complains there
>   are no local disks (which is not true, even device nodes are there
>   under /dev/dsk/ ... )

That's quite weird, under HVM disks are available using both the emulated 
and the PV interfaces, and Solaris 11.3 should certainly support the 
emulated ones. IIRC they are AHCI. 
 
> - OpenIndiana (OI-hipster-text-20160421.iso as HVM) hangs after printing
>   out the kernel version message
> 
> - SmartOS (smartos-20160428T170316Z.iso) 
>   seems to work at first, but after the initial configuration
>   it cannot bring the network interface up because its link
>   state is unknown (tried model=e1000 and the default Realtek):
> 
> [root@master0 ~]# dladm show-phys
> LINK MEDIASTATE SPEED  DUPLEX   DEVICE
> e1000g0  Ethernet unknown   0  half e1000g0

Hm, AFAICT this doesn't look related to FreeBSD, but just to be sure, could 
you try the same with a Linux Dom0?

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting Solaris 11.3 (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)

[Roger Pau Monné]

On Sat, Apr 30, 2016 at 08:56:54PM +, Marcin Cieslak wrote:
> On Fri, 29 Apr 2016, Roger Pau Monné wrote:
> 
> > On Thu, Apr 28, 2016 at 08:01:23PM +, Marcin Cieslak wrote:
> > > On Tue, 26 Apr 2016, Roger Pau Monné wrote:
> > > 
> > > > On Tue, Apr 26, 2016 at 10:39:06AM +, Marcin Cieslak wrote:
> > > > > By the way, I just managed to crash a whole machine by trying to
> > > > > boot Solaris 11.3:
> > > > > 
> > > > > # builder = "hvm"
> > > > > memory = 2048
> > > > > vcpus = 1
> > > > > name = "Solaris0"
> > > > > disk = [ 'file:/root/sol-11_3-text-x86.iso,6:cdrom,r', 
> > > > > '/dev/zvol/zroot/solaris0,raw,hda,w' ]
> > > > > kernel = "/root/xen/solaris/unix"
> > > > > ramdisk = "/root/xen/solaris/boot_archive"
> > > > > extra = "/platform/i86xpv/kernel/amd64/unix -B 
> > > > > console=ttya,livemode=text"
> > > > > #boot = "c"
> > > > > usbdevice = 'tablet'
> > > > > vnc = 1
> > > > > vnclisten = '0.0.0.0'
> > > > > vif = [ 'bridge=bridge0' ]
> > > > > 
> > > > > /root/xen/solaris/unix and /root/xen/solaris/boot_archive where 
> > > > > extracted from sol-11_3-text-x86.iso
> > > > > ("Intel text-only image").
> > > > 
> > > > I don't think you can boot Solaris as a PV guest anymore, you should 
> > > > instead 
> > > > boot it as a HVM guest. You will have to remove the kernel and ramdisk 
> > > > options and instead add builder="hvm" (that you have left commented 
> > > > out).
> > > 
> > > However supported or not, I think it should not crash a whole host 
> > > system?...
> > 
> > Certainly. I assumed that you meant it crashed the VM, not the whole host. 
> > Can you please provide the trace of the crash?
> 
> Apologies, forgot to let Xen keep VGA:
> 
> http://marcincieslak.com/tmp/xencrash.png
> 
> Manual OCR:
> 
> FreeBSD/amd64 (o.saper.info) (xc0)
> 
> login: (XEN) vmx.c:2464:d0v0 EPT violation 0x182 (-w-/---), gpa 
> 0x010178f000

Hello,

I've been able to debug this and found the issue. I have two patches that 
should be applied to FreeBSD in order to fix it, they can be found at:

https://people.freebsd.org/~royger/privcmd_fixes/

Could you please give them a try?

I have however been unable to boot a Solaris 11.3 guest under PV mode, but 
at least the host is not crashing anymore :).

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting Solaris 11.3 (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)

[Roger Pau Monné]

On Thu, Apr 28, 2016 at 08:01:23PM +, Marcin Cieslak wrote:
> On Tue, 26 Apr 2016, Roger Pau Monné wrote:
> 
> > On Tue, Apr 26, 2016 at 10:39:06AM +, Marcin Cieslak wrote:
> > > By the way, I just managed to crash a whole machine by trying to
> > > boot Solaris 11.3:
> > > 
> > > # builder = "hvm"
> > > memory = 2048
> > > vcpus = 1
> > > name = "Solaris0"
> > > disk = [ 'file:/root/sol-11_3-text-x86.iso,6:cdrom,r', 
> > > '/dev/zvol/zroot/solaris0,raw,hda,w' ]
> > > kernel = "/root/xen/solaris/unix"
> > > ramdisk = "/root/xen/solaris/boot_archive"
> > > extra = "/platform/i86xpv/kernel/amd64/unix -B console=ttya,livemode=text"
> > > #boot = "c"
> > > usbdevice = 'tablet'
> > > vnc = 1
> > > vnclisten = '0.0.0.0'
> > > vif = [ 'bridge=bridge0' ]
> > > 
> > > /root/xen/solaris/unix and /root/xen/solaris/boot_archive where extracted 
> > > from sol-11_3-text-x86.iso
> > > ("Intel text-only image").
> > 
> > I don't think you can boot Solaris as a PV guest anymore, you should 
> > instead 
> > boot it as a HVM guest. You will have to remove the kernel and ramdisk 
> > options and instead add builder="hvm" (that you have left commented out).
> 
> However supported or not, I think it should not crash a whole host system?...

Certainly. I assumed that you meant it crashed the VM, not the whole host. 
Can you please provide the trace of the crash?

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting Windows/UEFI (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)

[Roger Pau Monné]

On Tue, Apr 26, 2016 at 10:39:06AM +, Marcin Cieslak wrote:
> Yes, should be easy. Will try to plug a binary port in.

Thanks :).
 
> By the way, I just managed to crash a whole machine by trying to
> boot Solaris 11.3:
> 
> # builder = "hvm"
> memory = 2048
> vcpus = 1
> name = "Solaris0"
> disk = [ 'file:/root/sol-11_3-text-x86.iso,6:cdrom,r', 
> '/dev/zvol/zroot/solaris0,raw,hda,w' ]
> kernel = "/root/xen/solaris/unix"
> ramdisk = "/root/xen/solaris/boot_archive"
> extra = "/platform/i86xpv/kernel/amd64/unix -B console=ttya,livemode=text"
> #boot = "c"
> usbdevice = 'tablet'
> vnc = 1
> vnclisten = '0.0.0.0'
> vif = [ 'bridge=bridge0' ]
> 
> /root/xen/solaris/unix and /root/xen/solaris/boot_archive where extracted 
> from sol-11_3-text-x86.iso
> ("Intel text-only image").

I don't think you can boot Solaris as a PV guest anymore, you should instead 
boot it as a HVM guest. You will have to remove the kernel and ramdisk 
options and instead add builder="hvm" (that you have left commented out).

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting Windows/UEFI (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)

[Roger Pau Monné]

On Tue, Apr 26, 2016 at 09:24:30AM +, Marcin Cieslak wrote:
> I got lazy and I have downloaded
> 
> https://sourceforge.net/projects/edk2/files/OVMF/OVMF-X64-r15214.zip/download
> 
> recompiled xen-tools by adding to xen-tools/Makefile
> 
> CONFIGURE_ARGS+=--enable-ovmf
> CONFIGURE_ARGS+=--with-system-ovmf=/root/xen/OVMF.fd
> 
> and worked pretty much out of the box.

Oh, thanks for the info, I didn't know OVMF provided pre-compiled binaries. 
This should make adding a OVMF port trivial. Would you like to take a stab 
at adding such a port and wiring it into the xen-tools package?
 
Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting r298488 as Xen Dom0 may break ZFS pool?

[Roger Pau Monné]

On Tue, Apr 26, 2016 at 07:16:25AM +, Marcin Cieslak wrote:
> On Tue, 26 Apr 2016, Roger Pau Monné wrote:
> 
> > > > libxl: error: libxl.c:6121:libxl_xen_console_read_line: reading console 
> > > > ring buffer: Cannot allocate memory
> > > 
> > > It seems that my problem was ...
> > > 
> > > vm.max_wired=1
> > > 
> > > in /boot/loader.conf
> > > 
> > > instead of
> > > 
> > > vm.max_wired=-1
> > 
> > And this also caused the ZFS corruption?
> 
> Looks like it to me... Pretty strange.

I thought the max_wired sysctl node only affected the usage of wired memory 
by userspace applications, but it may have also prevented ZFS from using 
memory.
 
> Btw. is there any way to boot UEFI Windows partition under Xen?
> (it works fine under bhyve)

You will have to boot it using OVMF, which is not compiled with Xen by 
default. I will try to add an OVMF package (like the SeaBIOS one that we 
already have), and wire it into the xen-tools package.

If you want to try it yourself, you can add "--enable-ovmf" to the xen-tools 
package configure and see what breaks ;).

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting r298488 as Xen Dom0 may break ZFS pool?

[Roger Pau Monné]

On Mon, Apr 25, 2016 at 08:50:53PM +, Marcin Cieslak wrote:
> On Mon, 25 Apr 2016, Marcin Cieslak wrote:
> 
> > On Mon, 25 Apr 2016, Marcin Cieslak wrote:
> > 
> > > On Mon, 25 Apr 2016, Roger Pau Monné wrote:
> > > > TBH, I have no idea. Can you also paste the log of the system (Xen + 
> > > > FreeBSD) when it fails to boot? If that's not possible, can you at 
> > > > least 
> > > > paste the output of `xl dmesg` when booted correctly under Xen?
> > 
> > I managed to boot it again...
> > 
> > root@o:~ # xl dmesg
> > xc: error: Could not bounce buffer for version hypercall (35 = Resource 
> > temporarily unavailabl): Internal error
> > xc: error: Could not bounce buffer for version hypercall (35 = Resource 
> > temporarily unavailabl): Internal error
> > xc: error: Could not bounce buffer for version hypercall (35 = Resource 
> > temporarily unavailabl): Internal error
> > xc: error: Could not bounce buffer for version hypercall (35 = Resource 
> > temporarily unavailabl): Internal error
> > xc: error: Could not bounce buffer for version hypercall (35 = Resource 
> > temporarily unavailabl): Internal error
> > xc: error: Could not bounce buffer for version hypercall (35 = Resource 
> > temporarily unavailabl): Internal error
> > libxl: error: libxl.c:6121:libxl_xen_console_read_line: reading console 
> > ring buffer: Cannot allocate memory
> 
> It seems that my problem was ...
> 
> vm.max_wired=1
> 
> in /boot/loader.conf
> 
> instead of
> 
> vm.max_wired=-1

And this also caused the ZFS corruption?

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Booting r298488 as Xen Dom0 may break ZFS pool?

[Roger Pau Monné]

On Sat, Apr 23, 2016 at 10:40:20PM +, Marcin Cieslak wrote:
> On a freshly installed (via upgrade from 10.3 from source)
> -CURRENT on this machine:
> 
> FreeBSD 11.0-CURRENT #0 r298488: Sat Apr 23 11:10:01 UTC 2016
> r...@o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 
> 3.8.0)
> WARNING: WITNESS option enabled, expect reduced performance.
> VT(vga): resolution 640x480
> CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.09-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
>   
> Features=0xbfebfbff
>   
> Features2=0x1fbae3ff
>   AMD Features=0x28100800
>   AMD Features2=0x1
>   XSAVE Features=0x1
>   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>   TSC: P-state invariant, performance statistics
> real memory  = 17179869184 (16384 MB)
> avail memory = 16475140096 (15711 MB)
> Event timer "LAPIC" quality 600
> ACPI APIC Table: 
> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
> FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
> random: unblocking device.
> ioapic0  irqs 0-23 on motherboard
> 
> ... i am trying to boot the system as Dom0 under Xen (4.5.2_2
> installed via pkg install).
> 
> The kernel boots in one of four cases, mostly
> though I don't get a block cursor after Xen messages
> and the machine sits and waits.
> 
> 
> After trying Xen suddenly the system no longer
> boots again:
> 
> Booting from local disk...
> PXE-M0F: Existing Intel Boot Agent.
> 
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> 
> Can't find /boot/zfsloader
> 
> FreeBSD/x86 boot
> Default: zroot:/boot/kernel/kernel
> boot:
> ZFS: i/o error - all block copies unavailable

I'm currently running a r298464 world+kernel with ZFS on root and have seen 
no issues so far.
 
> (...)
> 
> The zpool imports without problems when
> booting from the rescue mfsbsd (10.3):
> 
>   pool: zroot
>  state: ONLINE
>   scan: none requested
> config:
> 
>   NAMESTATE READ WRITE CKSUM
>   zroot   ONLINE   0 0 0
> mirror-0  ONLINE   0 0 0
>   ada0p3  ONLINE   0 0 0
>   ada1p3  ONLINE   0 0 0
> 
> errors: No known data errors
> 
> gpart layout:
> 
> =>34  5860533101  ada0  GPT  (2.7T)
>   341024 1  freebsd-boot  (512K)
> 1058 4194304 2  freebsd-swap  (2.0G)
>  4195362  5856337773 3  freebsd-zfs  (2.7T)
> 
> =>34  5860533101  ada1  GPT  (2.7T)
>   341024 1  freebsd-boot  (512K)
> 1058 4194304 2  freebsd-swap  (2.0G)
>  4195362  5856337773 3  freebsd-zfs  (2.7T)
> 
> I have managed to make zpool boot again by doing voodoo
> similar to this one:
> 
> [root@rescue ~]# zpool import -R /mnt zroot
> [root@rescue ~]# mount -t devfs devfs /mnt/dev
> [root@rescue ~]# chroot /mnt /bin/tcsh
> 
> (... Running make install in /usr/src/sys/boot ...)
> 
> root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada0
> partcode written to ada0p1
> root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada1
> partcode written to ada1p1
> root@rescue:/ # exit
> [root@rescue ~]# umount /mnt/dev
> [root@rescue ~]# zpool export zroot
> [root@rescue ~]# reboot
> 
> Why zpool metadata get corrupted?

TBH, I have no idea. Can you also paste the log of the system (Xen + 
FreeBSD) when it fails to boot? If that's not possible, can you at least 
paste the output of `xl dmesg` when booted correctly under Xen?

What operations did you perform when the system booted correctly using 
FreeBSD/Xen?

Does the disk get corrupted even if the system fails to boot? AFAICT, it 
seems like it's only the bootcode that gets corrupted, is that right?

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Xen from ports won't boot using -CURRENT

[Roger Pau Monné]

On Thu, Apr 14, 2016 at 12:25:41PM -0500, Dustin Marquess wrote:
> AH-HA! Indeed I am.  Is UEFI not supported?

Not at the moment, I'm sorry. I simply haven't got the time to look into 
what's needed to boot FreeBSD + Xen using UEFI (since multiboot is not 
suitable in that case IIRC). If you want to use a FreeBSD Dom0 you will have 
to switch to legacy BIOS for the time being.

Roger
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ports/Xen on CURRENT, error: ELF start or entries are out of bounds.

[Roger Pau Monné]

On Thu, 10 Mar 2016, brahmann wrote:
> Hi all.
> I'm trying to run XEN on latest current as dom0.
> Did all by manual: http://wiki.xen.org/wiki/FreeBSD_Dom0 (except compiling
> from git, cant compile tools, so just used ports xen/xen-tools/xen-kernel)
> And stuck with such an error:
> 
> ---
> (XEN) elf_xen_addr_calc_check: ERROR: ELF start or entries are out of bounds.
> Could not set up DOM0 guest OS
> ---
> 
> This only drops on console while xen kernel boots, all other strings are OK
> according to manual.
> Link to photo of speedy console log: http://imgur.com/a5b9CLE
> If need details pls ask.
> 
> Please, tell me  where to look or what to do or at least guide me in the right
> direction.

Ed committed the fix to this a couple of days ago, as of r297242 this 
should be fixed.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Poor disk performance of FreeBSD under AWS

[Roger Pau Monné]

On Fri, 18 Mar 2016, Julian Elischer wrote:
> On 18/03/2016 3:00 PM, huanghwh wrote:
> > BTW, with 1G ram configuration, vi can exit very quickly.
> > At 2016-03-18 14:26:14, "huanghwh"  wrote:
> > > I have a EC2 server in AWS, 4CPU+16G ram, FreeBSD 10.2R.
> > > 
> > > Use two command dd and vi:
> > > dd if=/dev/zero of=/.swap bs=1M count=8192 &
> > > 
> > > 8589934592 bytes transferred in 117.074462 secs (73371549 bytes/sec)
> > > 
> > > when dd run in background, input vi command  to edit a small txt file
> > > "d.txt" at same time,
> > > 
> > > and then write and quit immediately:
> > > 
> > > 
> > > /usr/bin/time vi d.txt
> > > 
> > > 
> > > 49.82 real 0.00 user 0.00 sys
> > > 
> > > in top command show:
> > > 810 root 1 23 0 12344K 2524K wswbuf 0 0:04 5.76% dd
> > > 821 root 1 20 0 23448K 4092K wdrain 0 0:00 0.00% vi
> > > 
> > > vi need almost 50 seconds to quit.
> I think some people are already looking at this.. it's not limited to AWS.

Is there a PR or Review about this? I don't have a 16GB system at hand, 
but it looks quite weird that it works "better" with 1GB rather than with 
16GB.

Are you using ZFS or UFS as your filesystem?

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ports/Xen on CURRENT, error: ELF start or entries are out of bounds.

[Roger Pau Monné]

On Tue, 22 Mar 2016, Ed Maste wrote:
> On 21 March 2016 at 10:50, Ed Maste  wrote:
> > On 17 March 2016 at 13:30, dfh0522 .  wrote:
> >>
> >> I need to add WITHOUT_ELFCOPY_AS_OBJCOPY after r296096,
> >
> > Sorry for the trouble.
> >
> > I've submitted an ELF Tool Chain ticket
> > (https://sourceforge.net/p/elftoolchain/tickets/524/) for this issue
> > in elfcopy. I expect either Roger or I will have a fix in the next few
> > days, but WITHOUT_ELFCOPY_AS_OBJCOPY will do for now.
> 
> Kai has committed a potential fix for the ticket to the ELF Tool Chain
> repo. I've made a patch for the FreeBSD contrib copy available for
> testing at https://people.freebsd.org/~emaste/patches/elfcopy-lma.diff
> .

Thanks for the notice! I've tried the patch and it indeed fixes the issue 
seen on the FreeBSD kernel, here's the readelf output now:

Program Headers:
  Type   Offset VirtAddr   PhysAddr
 FileSizMemSiz  FlgAlign
  PHDR   0x0040 0x80200040 0x00200040
 0x0150 0x0150  R E0x8
  INTERP 0x0190 0x80200190 0x00200190
 0x000d 0x000d  R  0x1
  [Requesting program interpreter: /red/herring]
  LOAD   0x 0x8020 0x0020
 0x013ed618 0x013ed618  R E0x20
  LOAD   0x013ee000 0x817ee000 0x017ee000
 0x00135c38 0x0060fc40  RW 0x20
  DYNAMIC0x013ee000 0x817ee000 0x017ee000
 0x00d0 0x00d0  RW 0x8
  GNU_STACK  0x 0x 0x
 0x 0x  RWE0x8

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ports/Xen on CURRENT, error: ELF start or entries are out of bounds.

[Roger Pau Monné]

Hello,

On Thu, 10 Mar 2016, brahmann wrote:
> Hi all.
> I'm trying to run XEN on latest current as dom0.
> Did all by manual: http://wiki.xen.org/wiki/FreeBSD_Dom0 (except compiling
> from git, cant compile tools, so just used ports xen/xen-tools/xen-kernel)
> And stuck with such an error:

Yes, using the ports (xen-kernel/xen-tools) is the recommended way, I 
should update the wiki page in order to note this.

> ---
> (XEN) elf_xen_addr_calc_check: ERROR: ELF start or entries are out of bounds.
> Could not set up DOM0 guest OS
> ---
> 
> This only drops on console while xen kernel boots, all other strings are OK
> according to manual.
> Link to photo of speedy console log: http://imgur.com/a5b9CLE
> If need details pls ask.

I've just noticied this issue, it's probably due to an issue with kernel 
linking. It seems like the physical address specified in the ELF program 
headers is wrong. I'm currently trying to bisect it in order to find the 
commit that caused it, in the meantime I can confirm that r295683 and 
older revisions should work fine.

Just to confirm, can you paste the output of the following command:

$ readelf -l /boot/kernel/kernel

With the non-working kernel?

Thanks, Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] [Updated] D5291: hyperv/vmbus: Select an IDT vector has lower priority than LAPIC timer and IPIs vectors

[Roger Pau Monné]

royger added a comment.


  I guess this supersedes https://reviews.freebsd.org/D5254? I would expand the 
commit message so that it's explicitly stated why the hypervisor vector should 
have a lower priority.

REVISION DETAIL
  https://reviews.freebsd.org/D5291

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sepherosa_gmail.com, adrian, delphij, decui_microsoft.com, 
honzhan_microsoft.com, howard0su_gmail.com, royger
Cc: freebsd-virtualization-list
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

HEADS UP: live migration added to Xen 4.5 packages in ports

[Roger Pau Monné]

Hello,

I've just added patches that enable live migration, save and restore
support to the Xen packages in ports:

https://svnweb.freebsd.org/ports?view=revision=398918

This enables live migration and save/restore of HVM and PV guests from a
FreeBSD/Xen PVH Dom0. Remember that in order to use live migration
between hosts the hard drive of the guest must be shared (eg: NFS).

A couple of notes below on it's usage:

I'm not a sysadmin, but save and restore can be used in conjunction with
ZFS snapshots in order to perform checkpoints of guests:

# xl save -p  domain.checkpoint1
# zfs snapshot @checkpoint1
# xl unpause 

This will result in a save file and a ZFS snapshot that can easily be
used to restore the VM to a previous known state.

It should also be possible to migrate guests between a Linux and a
FreeBSD Dom0 (provided there's a common shared disk storage), but I have
not tested it.

NB: if you don't have two hosts but still want to test it, you can do it
locally:

# xl migrate  localhost

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: HEAD UP: notice to FreeBSD/Xen Dom0 users

[Roger Pau Monné]

Hello,

El 15/08/15 a les 15.36, Outback Dingo ha escrit:
 HEAD UP on 10.2 the instructions found at
 http://wiki.xen.org/wiki/FreeBSD_Dom0
  git branch
  master
 * stable-4.5
 
 
 
 no longer appear to work
 
 gmake -j8 install-tools HOSTCC=gcc48 CC=gcc48
 
 fails with
 __SNIP__
 gmake[3]: Entering directory
 '/usr/home/dingo/xen/tools/qemu-xen-dir-remote'
  GEN   config-host.h
  GEN   trace/generated-tracers.h
  GEN   trace/generated-tracers.c
  LINK  qemu-nbd
  LINK  qemu-img
 c++  LINK  qemu-io
 : error: unknown argument: '-fstack-protector-strong'
 c++: error: unknown argument: '-fstack-protector-strong'

I've tried to reproduce this with a fresh checkout without success. I
guess you have some libraries in your system that change the way Qemu is
built. In any case, can you try the following rune:

gmake -j8 install-tools HOSTCC=gcc48 CC=gcc48 HOSTCXX=g++48 CXX=g++48

Also, did you perform a distclean before building? (If the tree is not a
fresh checkout).

Roger.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

HEAD UP: notice to FreeBSD/Xen Dom0 users

[Roger Pau Monné]

Hello,

If you are running a FreeBSD/Xen Dom0 and update FreeBSD to r284870 or
any later revision make sure you are using the xen-kernel-4.5.0_4 and
xen-tools-4.5.0_7 packages (or any later version), or else you won't be
able to boot.

Sorry for the trouble this might cause. Be aware that the Xen PVH
interface (which FreeBSD uses) is still not marked as stable, so other
incompatible changes will happen before this is fully stabilized.

Users of FreeBSD as a Xen guest (PVHVM mode) are not affected and should
ignore this message.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: xen_kernel, console and X11

[Roger Pau Monné]

Hello,

El 04/05/15 a les 16.16, Roman Bogorodskiy ha escrit:
   Roger Pau Monné wrote:
 
 Hello,

 El 02/05/15 a les 17.43, Roman Bogorodskiy ha escrit:
 Hi,

 I'm trying to get Xen running and following these instructions:

 https://wiki.freebsd.org/Xen

 and

 http://wiki.xen.org/wiki/FreeBSD_Dom0

 I'm running two days old -CURRENT and ports. I've installed the
 emulators/xen port and followed instructions in pkg-message.

 I'm having some problems with console. I'm wondering if it's possible to
 have X running on the same box running xen kernel?

 It should be, although I had issues while using the vesa driver with a
 FreeBSD Xen Dom0, mainly because Dom0 doesn't have access to the BDA and
 EBDA, I'm working on fixing this in Xen upstream.

 My setup is as follows:
  
  - Intel i5-4690 that supports IOMMU:
 $ sudo acpidump -t|grep DMAR
   DMAR: Length=128, Revision=1, Checksum=90,
 $
  - vm.max_wired=-1 in /etc/sysctl.conf
  - xc0 /usr/libexec/getty Pc xterm   on  secure in
/etc/ttys

 In loader.conf I have:

 xen_kernel=/boot/xen
 xen_cmdline=dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1
 guest_loglvl=all loglvl=all console=com1

 So you are trying to use the serial console but you are not getting any
 output? If that's not the case, please drop the com1 parameter and set
 console=vga.
 
 Sorry for confusion, actually 'console=com1' works for me.
 Initially I didn't realize that it doesn't display kernel messages and
 after some failed attempts to run startx and hard reboots it took much
 longer to see a login prompt because I have background fsck disabled.
 
 So, 'console=com1' works unless I do 'startx'.

I'm not sure I would call it working if you don't get any kernel
messages while booting. What happens if you set console=vga?

 
 Do you have anything else in your /boot/loader.conf apart from this two
 lines?


 With this setup I get my system booted and at some point I can see a
 login screen. When I type 'startx' the system freezes. Have to hard
 reboot it to get working again.

 On which device do you get a login prompt? Is it xc0, ttyv0 or ttyu0?
 
 I get a login prompt on ttyv0.
 
 BTW, I use the nvidia driver.

Another user also reported a similar problem with Xen and X, and it was
solved by the following patch:

https://people.freebsd.org/~royger/0001-xen-introduce-a-newbus-function-to-allocate-unused-m.patch

Could you apply it and rebuild your kernel to see if that also solves
your issues?

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: xen_kernel, console and X11

[Roger Pau Monné]

El 04/05/15 a les 17.27, Roman Bogorodskiy ha escrit:
   Roger Pau Monné wrote:
 Another user also reported a similar problem with Xen and X, and it was
 solved by the following patch:

 https://people.freebsd.org/~royger/0001-xen-introduce-a-newbus-function-to-allocate-unused-m.patch

 Could you apply it and rebuild your kernel to see if that also solves
 your issues?
 
 Applied this patch on r282416, things didn't change: I do 'startx' in
 ttyv0, X prints that it loads extensions, then a black screen with a
 cursor appears and things hang.

Can you try to enable sshd and see if the system is still responding? Do
you have a serial cable on that box in order to catch any Xen messages?
If not, and the system is still reachable from ssh after running startx
can you paste the output of xl dmesg?

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: available hypervisors in FreeBSD

[Roger Pau Monné]

El 01/04/15 a les 12.30, Udo Rader ha escrit:
 Hi all,
 
 first please excuse if this may be a FAQ, but even though I am a long
 time linux admin (~1996), I am quite new to the *BSD world and I am
 trying to evaluate if FreeBSD fits our virtualization needs.
 
 So, for my many questions:
 
 As far as my homework digging revealed, FreeBSD supports four hypervisors:
 
 * bhyve
 * KVM
 * QEMU
 * VirtualBox

Make that 5:
 * Xen: http://wiki.xen.org/wiki/FreeBSD_Dom0

Altough FreeBSD doesn't run KVM, and I'm not sure whether QEMU fits
under the hypervisor category, it's an emulator instead, so the list
should probably be 3 (Bhyve, VirtualBox and Xen).

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: available hypervisors in FreeBSD

[Roger Pau Monné]

El 01/04/15 a les 14.59, Udo Rader ha escrit:
 On 04/01/2015 12:41 PM, Roger Pau Monné wrote:
 El 01/04/15 a les 12.30, Udo Rader ha escrit:
 As far as my homework digging revealed, FreeBSD supports four hypervisors:

 * bhyve
 * KVM
 * QEMU
 * VirtualBox

 Make that 5:
  * Xen: http://wiki.xen.org/wiki/FreeBSD_Dom0

 Altough FreeBSD doesn't run KVM, and I'm not sure whether QEMU fits
 under the hypervisor category, it's an emulator instead, so the list
 should probably be 3 (Bhyve, VirtualBox and Xen).
 
 thanks for pointing Xen out. I was indeed not aware of Xen running on
 FreeBSD, an intriguing (and well known) alternative.
 
 The wiki says, that migrate/save/restore are missing from the *BSD port.
 Is that still valid?

Yes, I'm currently finishing the patches for Xen. This is not missing
from FreeBSD, but from Xen itself when running Dom0 in PVH mode.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: available hypervisors in FreeBSD

[Roger Pau Monné]

Hello,

El 01/04/15 a les 16.27, Gerd Hafenbrack ha escrit:
 On 2015-04-01 16:19, Roger Pau Monné wrote:
 El 01/04/15 a les 14.59, Udo Rader ha escrit:
 ... thanks for pointing Xen out. I was indeed not aware of Xen
 running on
 FreeBSD, an intriguing (and well known) alternative.

 The wiki says, that migrate/save/restore are missing from the *BSD port.
 Is that still valid?

 Yes, I'm currently finishing the patches for Xen. This is not missing
 from FreeBSD, but from Xen itself when running Dom0 in PVH mode.
 
 The documentation for FreeBSD as Dom0 seems outdated anyway to me.

The document was last modified on the 15th of March 2015. I know things
move fast in the IT industry, but I wouldn't call that outdated.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [Xen-devel] [RFC] Hypervisor RNG and enumeration

[Roger Pau Monné]

Adding the bhyve guys.

El 29/10/14 a les 6.19, Andy Lutomirski ha escrit:
 Here's a draft CommonHV spec.  It's also on github:
 
 https://github.com/amluto/CommonHV
 
 So far, this provides a two-way RNG interface, a way to detect it, and
 a way to detect other hypervisor leaves.  The latter is because, after
 both the enormous public thread and some private discussions, it seems
 that detection of existing CPUID paravirt leaves is annoying and
 inefficient.  If we're going to define some cross-vendor CPUID leaves,
 it seems like it would be useful to offer a way to quickly enumerate
 other leaves.
 
 I've been told the AMD intends to update their manual to match Intel's
 so that hypervisors can use the entire 0x4F?? CPUID range.  I have
 intentionally not fixed an MSR value for the RNG because the range of
 allowed MSRs is very small in both the Intel and AMD manuals.  If any
 given hypervisor wants to ignore that small range and advertise a
 higher-numbered MSR, it is welcome to, but I don't want to codify
 something that doesn't comply with the manuals.
 
 Here's the draft.  Comments?  To the people who work on various
 hypervisors: Would you implement this?  Do you like it?  Is there
 anything, major or minor, that you'd like to see changed?  Do you
 think that this is a good idea at all?
 
 I've tried to get good coverage of various hypervisors.  There are
 Hyper-V, VMWare, KVM, and Xen people on the cc list.
 
 Thanks,
 Andy
 
 
 
 CommonHV, a common hypervisor interface
 ===
 
 This is CommonHV draft 1.
 
 The CommonHV specification is Copyright (c) 2014 Andrew Lutomirski.
 
 Licensing will be determined soon.  The license is expected to be extremely
 liberal.  I am currently leaning towards CC-BY-SA for the specification and
 an explicit license permitting anyone to implement the specification
 with no restrictions whatsoever.
 
 I have not patented, nor do I intend to patent, anything required to implement
 this specification.  I am not aware of any current or future intellectual
 property rights that would prevent a royalty-free implementation of
 this specification.
 
 I would like to find a stable, neutral steward of this specification
 going forward.  Help with this would be much appreciated.
 
 Scope
 -
 
 CommonHV is a simple interface for communication
 between hypervisors and their guests.
 
 CommonHV is intended to be very simple and to avoid interfering with
 existing paravirtual interfaces.  To that end, its scope is limited.
 CommonHV does only two types of things:
 
   * It provides a way to enumerate other paravirtual interfaces.
   * It provides a small, extensible set of paravirtual features that do not
 modify or replace standard system functionality.
 
 For example, CommonHV does not and will not define anything related to
 interrupt handling or virtual CPU management.
 
 For now, CommonHV is only applicable to the x86 platform.
 
 Discovery
 -
 
 A CommonHV hypervisor MUST set the hypervisor bit (bit 31 in CPUID.1H.0H.ECX)
 and provide the CPUID leaf 4F00H, containing:
 
   * CPUID.4F00H.0H.EAX = max_commonhv_leaf
   * CPUID.4F00H.0H.EBX = 0x6D6D6F43
   * CPUID.4F00H.0H.ECX = 0x56486E6F
   * CPUID.4F00H.0H.EDX = 0x66746e49
 
 EBX, ECX, and EDX form the string CommonHVIntf in little-endian ASCII.
 
 max_commonhv_leaf MUST be a number between 0x4F00 and 0x4FFF.  It
 indicates the largest leaf defined in this specification that is provided.
 Any leaves described in this specification with EAX values that exceed
 max_commonhv_leaf MUST be handled by guests as though they contain
 all zeros.
 
 CPUID leaf 4F01H: hypervisor interface enumeration
 --
 
 If max_commonhv_leaf = 0x4F01, CommonHV provides a list of tuples
 (location, signature).  Each tuple indicates the presence of another
 paravirtual interface identified by the signature at the indicated
 CPUID location.  It is expected that CPUID.location.0H will have
 (EBX, ECX, EDX) == signature, although whether this is required
 is left to the specification associated with the given signature.
 
 If the list contains N tuples, then, for each 0 = i  N:
 
   * CPUID.4F01H.i.EBX, CPUID.4F01H.i.ECX, and CPUID.4F01H.i.EDX
 are the signature.
   * CPUID.4F01H.i.EAX is the location.
 
 CPUID with EAX = 0x4F01 and ECX = N MUST return all zeros.
 
 To the extent that the hypervisor prefers a given interface, it should
 specify that interface earlier in the list.  For example, KVM might place
 its KVMKVMKVM signature first in the list to indicate that it should be
 used by guests in preference to other supported interfaces.  Other hypervisors
 would likely use a different order.
 
 The exact semantics of the ordering of the list is beyond the scope of
 this specification.
 
 CPUID leaf 4F02H: miscellaneous features
 
 
 

Re: Directly reserve an interrupt IDT entry for Hyper-V

[Roger Pau Monné]

On 20/08/14 11:19, Wei Hu wrote:
 Hello,
 
 Sending to Xen, drivers and virtualization mailing lists since this might be 
 of interest to the folks on these aliases.
 
 I am working for Microsoft to improve the performance of FreeBSD running on 
 Hyper-V. Right now I am adding a feature in the vmbus driver which could 
 handle the host-guest channel communications on all vCPUs simultaneously. In 
 order to achieve this, the hypervisor will send same interrupt concurrently 
 on all the vCPUs. The traditional way on FreeBSD to set up interrupt handling 
 for devicse, such as calling bus_alloc_resource() to reserve an IRQ line, and 
 then calling bus_setup_intr() to create a vector, doesn't seem to work in 
 this case. It seems if the interrupt is routed via legacy IRQ, it can only be 
 active on one vCPU at a time. In order to allow the same interrupt to be 
 handled on all vCPUs concurrently, all I need is an IDT entry, not an IRQ 
 line.
 
 I checked current FreeBSD code. It looks to me Xen directly uses the vector 
 number IDT_EVTCHN (0x93) to achieve the same purpose. I am proposing both Xen 
 and Hyper-V share this same vector. Following is a little bit detail of my 
 proposal for the changes in the current kernel.
 
 
 1.   In machdep.c:
 
  #ifdef XENHVM
 
 setidt(IDT_EVTCHN, IDTVEC(xen_intr_upcall), SDT_SYSIGT, SEL_UPL, 0);
 
 #else
 
 setidt(IDT_EVTCHN, IDTVEC(hv_vmbus_intr), SDT_SYSIGT, SEL_UPL, 0);
 
 #endif
 
 2.   Apic_vector.S
 
 Add IDTVEC(hv_vmbus_intr) to call Hyper-V vmbus interrupt service routine.
 
 Any  thoughts, objections and feedbacks are all welcome.

Hello,

I don't think using the same IDT vector is the right approach, I would
just pick a different IDT vector and use that for Hyper-V. Using the
same IDT vector (like your suggestion above) would prevent shipping a
kernel with with both Hyper-V and Xen support (like it's done now in
GENERIC).

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Directly reserve an interrupt IDT entry for Hyper-V

[Roger Pau Monné]

On 20/08/14 17:31, John Baldwin wrote:
 On Wednesday, August 20, 2014 9:31:54 am Roger Pau Monné wrote:
 Hello,

 I don't think using the same IDT vector is the right approach, I would
 just pick a different IDT vector and use that for Hyper-V. Using the
 same IDT vector (like your suggestion above) would prevent shipping a
 kernel with with both Hyper-V and Xen support (like it's done now in
 GENERIC).

 Roger.
 
 Hmm, can't you make this a runtime check to only call setidt() if you detect 
 you are under the appropriate hypervisor?
 
 Also, bhyve currently has a hackish way of requesting a free IDT slot.  
 Perhaps it would be best if I added little API to reserve an IDT slot 
 assuming 
 that callers could accept a dynamic IDT vector rather than a static one.

That would work for Xen. The IDT vector doesn't need to be fixed since
it's registered with Xen when the system boots.

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

FreeBSD and memory balloon drivers

[Roger Pau Monné]

Hello,

I've been looking into the Xen balloon driver, because I've experienced
problems when ballooning memory down which AFAICT are also present in
the VirtIO balloon driver. The problem I've experienced is that when
ballooning memory down, we basically allocate a bunch of memory as
WIRED, to make sure nobody tries to swap it do disk, since it will crash
the kernel because the memory is not populated. Due to this massive
amount of memory allocated as WIRED, user-space programs that try to use
mlock will fail because we hit the limit in vm.max_wired.

I'm not sure what's the best way to deal with this limitation, should
vm.max_wired be changed from the balloon drivers when ballooning
down/up? Is there anyway to remove the pages ballooned down from the
memory accounting of wired pages?

Thanks, Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD and memory balloon drivers

[Roger Pau Monné]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 20/06/14 15:28, Konstantin Belousov wrote:
 On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn? wrote:
 Hello,
 
 I've been looking into the Xen balloon driver, because I've
 experienced problems when ballooning memory down which AFAICT are
 also present in the VirtIO balloon driver. The problem I've
 experienced is that when ballooning memory down, we basically
 allocate a bunch of memory as WIRED, to make sure nobody tries to
 swap it do disk, since it will crash the kernel because the
 memory is not populated. Due to this massive amount of memory
 allocated as WIRED, user-space programs that try to use mlock
 will fail because we hit the limit in vm.max_wired.
 
 I'm not sure what's the best way to deal with this limitation,
 should vm.max_wired be changed from the balloon drivers when
 ballooning down/up? Is there anyway to remove the pages ballooned
 down from the memory accounting of wired pages?
 
 You could change the type of pages the ballon driver is
 allocating. Instead of wired pages, you may request unmanaged, by
 passing NULL object to vm_page_alloc().  This would also save on
 the trie nodes for managing the radix trie for the object.  There
 are still plinks or listq to keep track of the allocated pages.

Thanks for the info, I have the following patch which fixes the usage
of WIRED for both the Xen and the VirtIO balloon drivers, could
someone please test the VirtIO side?

Roger.

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (Darwin)

iQEcBAEBAgAGBQJTpFAfAAoJEKXZdqUyumTA1TQH/22YpAGCQ8oa0hmpfE8oovxz
q8EDRyfDoogEswNYwboI8cBP7GSbuBbe1Z0MTiMHtwyHzqGhJM5B7jKioqFsqxvc
/Qfld8z3vDD94/5iaMX64dV2/VKkLwypR2uU5PkN018FTAJ0FFycC336xVjD8eUz
/DCRQIZRUzNlcrZYlOtSALR2M9bM1/f2++e2C6L7kbSsF4BAH2wmRcdtM1uBMO6/
CnD7ctZsnxxdS05eLWMpv6jfcRH8yDM3vPaHgXa223q74TU1Rh7AFw/TPsyBvBY2
cDUnYLZdUx0NQJfPM9MKGLe8P5o3WLVO1hfGmrZHdmjE2B18mNfzg4SS0CUQhQ0=
=hctw
-END PGP SIGNATURE-
From 2ed0f82b16753c96e96acc1e26a75a0fd2ee7d34 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne roger@citrix.com
Date: Fri, 20 Jun 2014 16:34:31 +0200
Subject: [PATCH] xen/virtio: fix balloon drivers to not mark pages as WIRED

Prevent the Xen and VirtIO balloon drivers from marking pages as
wired. This prevents them from increasing the system wired page count,
which can lead to mlock failing because of hitting the limit in
vm.max_wired.

Also, in the Xen case make sure pages are zeroed before giving them
back to the hypervisor, or else we might be leaking data.

Sponsored by: Citrix Systems RD
Reviewed by: xxx
Approved by: xxx

dev/virtio/balloon/virtio_balloon.c:
 - Don't allocate pages with VM_ALLOC_WIRED.

dev/xen/balloon/balloon.c:
 - Don't allocate pages with VM_ALLOC_WIRED.
 - Make sure pages are zeroed before giving them back to the
   hypervisor.
---
 sys/dev/virtio/balloon/virtio_balloon.c |4 +---
 sys/dev/xen/balloon/balloon.c   |   13 ++---
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/sys/dev/virtio/balloon/virtio_balloon.c 
b/sys/dev/virtio/balloon/virtio_balloon.c
index d540099..6d00ef3 100644
--- a/sys/dev/virtio/balloon/virtio_balloon.c
+++ b/sys/dev/virtio/balloon/virtio_balloon.c
@@ -438,8 +438,7 @@ vtballoon_alloc_page(struct vtballoon_softc *sc)
 {
vm_page_t m;
 
-   m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_WIRED |
-   VM_ALLOC_NOOBJ);
+   m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ);
if (m != NULL)
sc-vtballoon_current_npages++;
 
@@ -450,7 +449,6 @@ static void
 vtballoon_free_page(struct vtballoon_softc *sc, vm_page_t m)
 {
 
-   vm_page_unwire(m, PQ_INACTIVE);
vm_page_free(m);
sc-vtballoon_current_npages--;
 }
diff --git a/sys/dev/xen/balloon/balloon.c b/sys/dev/xen/balloon/balloon.c
index fa56c86..a7ca1e4 100644
--- a/sys/dev/xen/balloon/balloon.c
+++ b/sys/dev/xen/balloon/balloon.c
@@ -255,7 +255,6 @@ increase_reservation(unsigned long nr_pages)
 
set_phys_to_machine(pfn, frame_list[i]);
 
-   vm_page_unwire(page, PQ_INACTIVE);
vm_page_free(page);
}
 
@@ -286,18 +285,26 @@ decrease_reservation(unsigned long nr_pages)
for (i = 0; i  nr_pages; i++) {
if ((page = vm_page_alloc(NULL, 0, 
VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ | 
-   VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) {
+   VM_ALLOC_ZERO)) == NULL) {
nr_pages = i;
need_sleep = 1;
break;
}
 
+   if ((page-flags  PG_ZERO) == 0) {
+   /*
+* Zero the page, or else we might be leaking
+* important data to other domains on the same
+* host.
+*/
+   pmap_zero_page(page);
+   }
+
 

Re: Xen PVHVM with FreeBSD10 Guest

[Roger Pau Monné]

On 16/01/14 19:38, Sydney Meyer wrote:
 Well then, thanks for the hint.. dmesg shows the following:
 
 Jan 16 18:22:30 bsd10 kernel: xn0: Virtual Network Interface at 
 device/vif/0 on xenbusb_front0
 Jan 16 18:22:30 bsd10 kernel: xn0: Ethernet address: 00:16:3e:df:1b:5a
 Jan 16 18:22:30 bsd10 kernel: xenbusb_back0: Xen Backend Devices on 
 xenstore0
 Jan 16 18:22:30 bsd10 kernel: xn0: backend features: feature-sg 
 feature-gso-tcp4
 Jan 16 18:22:30 bsd10 kernel: xbd0: 8192MB Virtual Block Device at 
 device/vbd/768 on xenbusb_front0
 Jan 16 18:22:30 bsd10 kernel: xbd0: attaching as ada0
 Jan 16 18:22:30 bsd10 kernel: xbd0: features: flush, write_barrier
 Jan 16 18:22:30 bsd10 kernel: xbd0: synchronize cache commands enabled.
 
 Now i did some tests with raw images and the disk performs very well (10-15% 
 less than native throughput).

So the problem only manifest itself when using block devices as disk
backends?

I've done some tests with fio using direct=1 (and a LVM volume as the
backend), and it shows that disk writes are slower when using PV drivers
instead of the emulated ones. On the other hand disk reads are faster
when using the PV drivers. Have you tried if the 9.x series also show
the same behaviour? (you will have to compile the custom XENHVM kernel)

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Xen PVHVM with FreeBSD10 Guest

[Roger Pau Monné]

On 17/01/14 10:17, Sydney Meyer wrote:
 I’m doing some benchmarks with bonnie and dd on the Variations 
 9.2/10.0;PVHVM/VirtIO;fileio/blockio. I will post the results here to this 
 thread.

By VirtIO I guess you mean emulated IO? That sounds great, I'm eager to
see the results :)

Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Xen PVHVM with FreeBSD10 Guest

[Roger Pau Monné]

On 16/01/14 17:41, Sydney Meyer wrote:
 Hello everyone,
 
 does someone know how to check if the paravirtualized I/O drivers from Xen 
 are loaded/working in FreeBSD 10? To my understanding it isn't necessary 
 anymore to compile a custom kernel with PVHVM enabled, right? In 
 /var/log/messages/ I can see the XN* and XBD* devices and the network 
 performance is very good (saturated Gb) compared to qemu-emulated, but the 
 disk performance is not as well, infact, it is even slower than emulated with 
 qemu (0.10.2). I did some test with dd and bonnie++, turned caching on the 
 host off and tried to directly sync to disk, PVonHVM is averagely 15-20 % 
 slower than QEMU at throughput. Both VM's are running on the same host on a 
 Xen 4.1 Hypervisor with QEMU 0.10.2 on a Debian Linux 3.2 Kernel as Dom0.

PV drivers will be used automatically if Xen is detected. You should see
something like this on dmesg:

xn0: Virtual Network Interface at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 00:16:3e:47:d4:52
xenbusb_back0: Xen Backend Devices on xenstore0
xn0: backend features: feature-sg feature-gso-tcp4
xbd0: 20480MB Virtual Block Device at device/vbd/51712 on xenbusb_front0
xbd0: features: flush, write_barrier
xbd0: synchronize cache commands enabled.

Are you using a raw file as a disk?

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: KVM Clock

[Roger Pau Monné]

On 15/01/14 13:05, Julian Stecklina wrote:
 On 01/14/2014 05:13 PM, Peter Grehan wrote:
 Hi Julian,
 
 is anyone working on KVM clock support for FreeBSD? If not, I
 might take a shot at it.
 
 None I know of: go for it :)
 
 Works for me so far: 
 https://github.com/blitz/freebsd/commit/cdc5f872b3e48cc0dda031fc7d6bdedc65c3148f

Looking
 
at the code it seems some common routines could be shared
between the KVM PV clock and the Xen PV clock
(sys/dev/xen/timer/timer.c). The data passed from the hypervisor to
the guest has exactly the same format (see struct vcpu_time_info in
Xen public headers).

At a first sight the KVM clock can benefit from using scale_delta
(which is going to be faster than the C version implemented in
kvmclock_get_timecount), and xen_fetch_vcpu_tinfo is exactly the same
as kvmclock_fetch.

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: KVM Clock

[Roger Pau Monné]

On 15/01/14 14:45, Julian Stecklina wrote:
 On Mi, 2014-01-15 at 14:08 +0100, Roger Pau Monné wrote:
 On 15/01/14 13:05, Julian Stecklina wrote:
 On 01/14/2014 05:13 PM, Peter Grehan wrote:
 Hi Julian,

 is anyone working on KVM clock support for FreeBSD? If not, I
 might take a shot at it.

 None I know of: go for it :)

 Works for me so far: 
 https://github.com/blitz/freebsd/commit/cdc5f872b3e48cc0dda031fc7d6bdedc65c3148f

 Looking

 at the code it seems some common routines could be shared
 between the KVM PV clock and the Xen PV clock
 (sys/dev/xen/timer/timer.c). The data passed from the hypervisor to
 the guest has exactly the same format (see struct vcpu_time_info in
 Xen public headers).
 
 Yes, I know. Didn't get around to making it pretty yesterday evening. ;)
 I'll post an updated patch, when I have some time. Any suggestions where
 to put the two common functions?
 
 At a first sight the KVM clock can benefit from using scale_delta
 (which is going to be faster than the C version implemented in
 kvmclock_get_timecount),
 
 At least somewhat on amd64. 32-bit looks pretty identical.
 
  and xen_fetch_vcpu_tinfo is exactly the same
 as kvmclock_fetch.
 
 I think xen_fetch_vcpu_tinfo has a subtle bug:
   217 do {
   218 dst-version = src-version;
   219 rmb();
   220 dst-tsc_timestamp = src-tsc_timestamp;
   221 dst-system_time = src-system_time;
   222 dst-tsc_to_system_mul = src-tsc_to_system_mul;
   223 dst-tsc_shift = src-tsc_shift;
   224 rmb();
   225 } while ((src-version  1) | (dst-version ^
 src-version));
 
 In line 225 src-version is potentially read twice. If you consider the
 following schedule:
 
 host starts updating data, sets src-version to 1
 guest reads src-version (1) and stores it into dst-version.
 guest copies inconsistent data
 guest reads src-version (1) and computes xor with dst-version.
 host finishes updating data and sets src-version to 2
 guest reads src-version (2) and checks whether lower bit is not set.
 while loop exits with inconsistent data!
 
 I think the C standard (at least C11) permits this as it does not
 specify in which order the two reads in line 225 need to happen.

According to the operator precedence and associativity in C
(http://en.wikipedia.org/wiki/Operators_in_C_and_C%2B%2B#Operator_precedence),
if I'm reading it right, the condition in the while line will be
evaluated as follows (because of the left-to-right associativity of the
| operator):

1. (src-version  1)
2. (dst-version ^ src-version)
3. result of 1 | result of 2

So AFAICT the flow that you describe could never happen, because
(src-version  1) is always done before (dst-version ^ src-version).

Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Panic starting a bhyve guest after resume

[Roger Pau Monné]

On 14/12/13 03:28, Neel Natu wrote:
 Hi John,
 
 On Fri, Dec 13, 2013 at 2:09 PM, John Baldwin j...@freebsd.org wrote:
 On Thursday, December 12, 2013 4:00:08 pm Neel Natu wrote:
 Hi John,

 On Thu, Dec 12, 2013 at 12:11 PM, John Baldwin j...@freebsd.org wrote:
 If I suspend and resume my laptop and then try to start a guest after the
 resume, I get an odd panic.  It generates a privileged instruction fault 
 (in
 kernel mode) for 'vmclear'.  I've checked CR4 and it claims that VMXE is 
 set.
 I dont have any other ideas off the top of my head on what I should be 
 poking
 at?  It looks like we read a bunch of MSRs in vmx_init(), but we don't 
 write
 to them, and all vmx_enable() does on each CPU is set VMXE in CR4 from 
 what I
 can tell.


 It also does a vmxon on each logical cpu which may also need to be
 done after a resume.

 Ah, yes it does.  That was sufficient both for starting a new guest after
 resume and even doing a suspend/resume while a guest was active (and the
 guest continued to run fine).  I have a hacky patch for this.  One, it
 includes both a suspend and resume hook for VMM, though for my testing I only
 needed a resume hook to invoke vmxon.  Second, the name of vmx_resume2()
 is a total hack (because vmx_resume() was already taken.  I think for now
 if I were to commit this, I'd just add the resme hook and maybe call the
 Intel method vmx_reset() or vmx_restore()?

 http://people.freebsd.org/~jhb/patches/bhyve_resume.patch

 
 There seems to be a race after the APs are restarted and before
 'vmm_resume_p()' where it would be problematic to execute a VMX
 instruction.
 
 Perhaps we should enable VMX on each cpu before they return to the
 interrupted code?

Can you use the hook in cpususpend_handler? It's cpu_ops.cpu_resume, and
gets called on each CPU before returning from the handler.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Running `sysctl -a` causes crash (Hyper-V)

[Roger Pau Monné]

On 11/11/13 15:31, Sergey Kandaurov wrote:
 On 11 November 2013 18:13, Pavel Timofeev tim...@gmail.com wrote:
 [...]
 (kgdb) p vm_guest
 $1 = 3
 Current language:  auto; currently minimal
 (kgdb)

 
 What if you try this change?
 
 Index: sys/kern/subr_param.c
 ===
 --- sys/kern/subr_param.c(revision 257695)
 +++ sys/kern/subr_param.c(working copy)
 @@ -153,6 +153,7 @@
  none,
  generic,
  xen,
 +hv,
  NULL
  };

This is my fault, I've added VM_GUEST_HV to the guest type enum without
knowing there was another array in a completely different file that
contained a description for those values.

If you (or someone else) are going to commit this change, I would
recommend adding a BIG comment in sys/systm.h above the declaration of
VM_GUEST explaining that if you add a new guest type there you also need
to add a description in subr_param.c, this will probably prevent
something like this happening again.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 11/10/13 11:42, Eggert, Lars wrote:
 Hi,
 
 On May 13, 2013, at 20:32, Roger Pau Monné roger@citrix.com wrote:
 Right now the code is in a state where it can be tested by users, so we
 would like to encourage FreeBSD and Xen users to test it and provide
 feedback.
 
 any idea if/when this code will be merged into -CURRENT?

It's already in HEAD, and will hopefully be part of the 10 release.

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Xen (and others Hypervisors) how do they handle IPIs?

[Roger Pau Monné]

On 27/09/13 09:44, Julian Elischer wrote:
 On 9/26/13 5:50 PM, Roger Pau Monné wrote:
 On 26/09/13 03:48, Julian Elischer wrote:
 On 9/25/13 11:04 PM, Roger Pau Monné wrote:
 On 25/09/13 16:56, Julian Elischer wrote:
 If CPUs are mapped around, how are IPIs handled? I assume they must be
 emulated?

 I've noticed that under Xen (on both Amazon EC2 and a Redhat server)
 whenever you schedule a thread it always sits on the run queue for 20
 uSecs before it starts running. It looks to me like it's the IPI
 taking
 a long time to be emulated.
 This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs
 instead of the emulated ones, see r255331. It should be faster than the
 previous emulated implementation.

 I missed that.. thanks!
 Do you (or anyone else) know if this can be used on Amazon EC2?
 And do you need a specific version/configuration of Xen to be able to
 use it?
 
 I ran a new GENERIC kernel on an Amazon Xen 4.2 system and it performed
 a bit better.
 delays in the running of threads reduced from a constant 20uSecs to a
 variable amount between 1.5uSecs and 9uSecs.
 It's still very slow when compared with real hardware.
 (timing viewed using Schedgraph)
 
 Is there a way to see whether PV IPIs are active?
 (it looks it from the results but that could be so many other things too).

The easiest way to see if PV IPIs are active is to run vmstat -i inside
of the guest and search for the following lines:

irq836: cpu6:r 2  0
irq839: cpu6:irg2530 70
irq841: cpu6:b  2175 60

This is the number of the different kinds of PV IPIs received on each
CPU. You can find the correlation between this names and the IPIs on top
of the file x86/xen/hvm.c.

Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Xen (and others Hypervisors) how do they handle IPIs?

[Roger Pau Monné]

On 26/09/13 03:48, Julian Elischer wrote:
 On 9/25/13 11:04 PM, Roger Pau Monné wrote:
 On 25/09/13 16:56, Julian Elischer wrote:
 If CPUs are mapped around, how are IPIs handled? I assume they must be
 emulated?

 I've noticed that under Xen (on both Amazon EC2 and a Redhat server)
 whenever you schedule a thread it always sits on the run queue for 20
 uSecs before it starts running. It looks to me like it's the IPI taking
 a long time to be emulated.
 This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs
 instead of the emulated ones, see r255331. It should be faster than the
 previous emulated implementation.

 I missed that.. thanks!
 Do you (or anyone else) know if this can be used on Amazon EC2?
 And do you need a specific version/configuration of Xen to be able to
 use it?

The PV IPIs require Xen version 4.0 or greater, and the PV timer
requires 4.0.1 or greater if I'm not mistaken.

If you are lucky to get an Amazon instance that uses this Xen version
(or any superior one) they will be activated by default, if not FreeBSD
will switch to the old event delivery method and PV IPIs and PV timer
will be disabled in favour of the emulated ones.

I guess it's just a matter of time before Amazon switches all their
servers to Xen 4.x.

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Xen (and others Hypervisors) how do they handle IPIs?

[Roger Pau Monné]

On 25/09/13 16:56, Julian Elischer wrote:
 If CPUs are mapped around, how are IPIs handled? I assume they must be
 emulated?
 
 I've noticed that under Xen (on both Amazon EC2 and a Redhat server)
 whenever you schedule a thread it always sits on the run queue for 20
 uSecs before it starts running. It looks to me like it's the IPI taking
 a long time to be emulated.

This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs
instead of the emulated ones, see r255331. It should be faster than the
previous emulated implementation.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 22/07/13 09:18, Jeroen van der Ham wrote:
 Hi,
 
 After some more testing I thought it would be good to put this into 
 production for my personal server. I've used pvhvm_v19 and built it without 
 debugging options and installed it on a FreeBSD 9.1 system.
 
 I've run into some hiccups with 9.1 user land and a 10-CURRENT kernel, but 
 that's all solvable[0].
 
 My VPS has some very limited memory (256M), but I've compensated with swap 
 space (1G)

Is your guest running a 32bit or a 64bit kernel?

Could you also provide the config file used to launch your guest and the
Xen and Dom0 kernel versions?

 
 Now anytime I'm putting the system under stress, by building ports or by 
 running a git clone on the kernel repository here, I'm seeing a lot of 
 messages about swap_pager:
 
 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 132545, size: 4096
 
 The system also becomes very sluggish and sometimes unresponsive.
 The weird thing was that one of these messages happened right after a reboot 
 when I rebuilt an outdated port and on the main console was checking the swap 
 memory:
 
 jeroen:~/ $ swapinfo  
 [8:13:29]
 Device  1K-blocks UsedAvail Capacity
 /dev/ada0p2524288 2484   521804 0%
 /dev/md0  1048576 2364  1046212 0%
 Total 1572864 4848  1568016 0%
 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 131424, size: 4096
 
 
 Is anyone else seeing something similar?

Could you also try a HEAD XENHVM kernel (without my patches), to see if
the issue is related to my changes or to some bug already present in HEAD?

 I certainly did not experience something like this on 9.0 with a XENHVM 
 kernel.
 
 If necessary I can rebuild a kernel with debugging support and do some more 
 recording of what is actually going on.
 
 Jeroen.
 
 
 [0]: I have edited bsd.port.mk to always apply the FBSD10_FIX, and for 
 version checking I am  running pkg version with UNAME_r=9.1-RELEASE.
 

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 20/06/13 11:20, Jeroen van der Ham wrote:
 Hi,
 
 I have this running for a day or so now, but I'm noticing that the load 
 averages seem a bit off:
 
 $ uptime
 11:17AM  up 17:14, 1 user, load averages: 0.31, 0.27, 0.21
 
 This is for a clean install, with just enough installed to compile this 
 kernel. In top I'm seeing that the machine is idling 98% of the time. But 
 this does not correlate to the load displayed above.

This is probably due to the fact that we are not properly accounting for
blocked/runnable/offline time. Did you see the same when running the
XENHVM kernel without my patches?

Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 19/06/13 14:16, Jeroen van der Ham wrote:
 
 On 19 Jun 2013, at 13:34, Roger Pau Monné roger@citrix.com wrote:

 Could you provide the boot log of the DomU, backtrace, Xen version and
 Dom0 kernel version?
 
 I did not have a console attached when it rebooted, so I did not have a log 
 of the initial boot. Now that I did, I see that it fails to mount its root 
 volume.
 
 It had been running previously on pvhvm_v10 for about two weeks without 
 problems. I updated my local git, and recompiled the kernel and rebooted. 
 Then this happened.
 
 
 In order:
 
 Booting...
 GDB: no debug ports present
 KDB: debugger backends: ddb
 KDB: current backend: ddb
 Copyright (c) 1992-2013 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 10.0-CURRENT #2 r+6ff8d00-dirty: Tue Jun 18 12:55:16 CEST 2013
 root@image01:/usr/obj/root/freebsd/sys/XENHVM amd64
 FreeBSD clang version 3.3 (trunk 178860) 20130405
 WARNING: WITNESS option enabled, expect reduced performance.
 XEN: Hypervisor version 4.0 detected.
 CPU: Quad-Core AMD Opteron(tm) Processor 2374 HE (2200.07-MHz K8-class CPU)
   Origin = AuthenticAMD  Id = 0x100f42  Family = 0x10  Model = 0x4  
 Stepping = 2
   
 Features=0x1781fbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,MMX,FXSR,SSE,SSE2,HTT
   Features2=0x80802001SSE3,CX16,POPCNT,HV
   AMD Features=0xe2500800SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!
   AMD Features2=0x1f3LAHF,CMP,CR8,ABM,SSE4A,MAS,Prefetch
 real memory  = 536870912 (512 MB)
 avail memory = 472260608 (450 MB)
 Event timer LAPIC quality 400
 ACPI APIC Table: Xen HVM
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 FreeBSD/SMP: 1 package(s) x 2 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  2
 random device not loaded; using insecure entropy
 ioapic0: Changing APIC ID to 1
 MADT: Forcing active-low polarity and level trigger for SCI
 ioapic0 Version 1.1 irqs 0-47 on motherboard
 kbd1 at kbdmux0
 xen_et0: Xen PV Clock on motherboard
 Event timer XENTIMER frequency 10 Hz quality 950
 Timecounter XENTIMER frequency 10 Hz quality 950
 acpi0: Xen on motherboard
 acpi0: Power Button (fixed)
 acpi0: Sleep Button (fixed)
 acpi0: reservation of 0, a (3) failed
 cpu0: ACPI CPU on acpi0
 cpu1: ACPI CPU on acpi0
 attimer0: AT timer port 0x40-0x43 irq 0 on acpi0
 Timecounter i8254 frequency 1193182 Hz quality 0
 Event timer i8254 frequency 1193182 Hz quality 100
 atrtc0: AT realtime clock port 0x70-0x71 irq 8 on acpi0
 Event timer RTC frequency 32768 Hz quality 0
 Timecounter ACPI-fast frequency 3579545 Hz quality 900
 acpi_timer0: 32-bit timer at 3.579545MHz port 0x1f48-0x1f4b on acpi0
 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0
 pci0: ACPI PCI bus on pcib0
 isab0: PCI-ISA bridge at device 1.0 on pci0
 isa0: ISA bus on isab0
 atapci0: Intel PIIX3 WDMA2 controller port 
 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc300-0xc30f at device 1.1 on pci0
 ata0: ATA channel at channel 0 on atapci0
 ata1: ATA channel at channel 1 on atapci0
 pci0: bridge at device 1.3 (no driver attached)
 vgapci0: VGA-compatible display mem 
 0xf000-0xf1ff,0xf300-0xf3000fff at device 2.0 on pci0
 xenpci0: Xen Platform Device port 0xc000-0xc0ff mem 0xf200-0xf2ff 
 irq 28 at device 3.0 on pci0
 xenstore0: XenStore on xenpci0
 atkbdc0: Keyboard controller (i8042) port 0x60,0x64 irq 1 on acpi0
 atkbd0: AT Keyboard irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 psm0: PS/2 Mouse irq 12 on atkbdc0
 psm0: [GIANT-LOCKED]
 psm0: model IntelliMouse Explorer, device ID 4
 fdc0: floppy drive controller port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
 fdc0: does not respond
 device_attach: fdc0 attach returned 6
 uart0: 16550 or compatible port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 uart0: console (9600,n,8,1)
 ppc0: Parallel port port 0x378-0x37f irq 7 on acpi0
 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
 ppbus0: Parallel port bus on ppc0
 lpt0: Printer on ppbus0
 lpt0: Interrupt-driven port
 ppi0: Parallel I/O on ppbus0
 orm0: ISA Option ROM at iomem 0xc9000-0xc97ff on isa0
 sc0: System console at flags 0x100 on isa0
 sc0: VGA 16 virtual consoles, flags=0x300
 vga0: Generic ISA VGA at port 0x3c0-0x3df iomem 0xa-0xb on isa0
 fdc0: No FDOUT register!
 Timecounters tick every 10.000 msec
 xenbusb_front0: Xen Frontend Devices on xenstore0
 cd0 at ata1 bus 0 scbus1 target 0 lun 0
 cd0: QEMU QEMU DVD-ROM 0.10 Removable CD-ROM SCSI-0 device
 cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
 cd0: cd present [360385 x 2048 byte records]
 xn0: Virtual Network Interface at device/vif/0 on xenbusb_front0
 xn0: Ethernet address: 00:16:3e:2f:b7:22
 xn1: Virtual Network Interface at device/vif/1 on xenbusb_front0
 xn1: Ethernet address: 00:16:3e:3e:64:c5
 xenbusb_back0: Xen Backend Devices

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 10/06/13 16:48, Roger Pau Monné wrote:
 Hello,
 
 I've pushed a new branch, pvhvm_v14 that contains support for live
 migration. While there I've also rebased the changes on top of current
 HEAD, so now it contains the recent fixes to blkfront and netfront.
 
 http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/pvhvm_v14

Hello,

There where some issues with the previous branch (pvhvm_v14), I've
pushed a new one (pvhvm_v16) that fixes the following bugs:

 * Make sure there are no IPIs in flight while the VM is migrated,
having in-flight IPIs is a problem because on resume the event channels
are re-initialized, so all pending events are lost, including IPIs.

 * Reset the clock after migration, this prevent clock drifts when the
VM is migrated.

 * blkfront was not correctly freeing the old event channel port.

The following two commits are needed for Xen:

f8e8fd56bd7d5675e8331b4ec74bae76c9dbf24e x86/HVM: fix initialization of
wallclock time for PVHVM on migration

32c864a35ece2c24a336d183869a546798a4b241 x86/vtsc: update vcpu_time in
hvm_set_guest_time

With this branch I've been able to successfully local migrate a busy VM
400 times consecutively.

As usual, the branch can be found here:

http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/pvhvm_v16

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 10/06/13 17:09, Outback Dingo wrote:
 
 
 
 On Mon, Jun 10, 2013 at 10:48 AM, Roger Pau Monné roger@citrix.com
 mailto:roger@citrix.com wrote:
 
 Hello,
 
 I've pushed a new branch, pvhvm_v14 that contains support for live
 migration. While there I've also rebased the changes on top of current
 HEAD, so now it contains the recent fixes to blkfront and netfront.
 
 
 http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/pvhvm_v14
 
 
 looking at your master branch your 2 weeks behind current... so where
 did you rebase your changes to head, or are you referring to your HEAD,
 and not FreeBSD 

No, my HEAD commit from FreeBSD master repository is from 3 days ago:

http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=commit;h=5311e12c931df9b67b64913670eab76a994317b9

This is the commit where I rebased my pvhvm_v14 branch.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 30/05/13 10:50, Jeroen van der Ham wrote:
 Hi,
 
 On 23 May 2013, at 19:41, Roger Pau Monné roger@citrix.com wrote:
 
 Hello,

 I've pushed a new branch, pvhvm_v10 that contains a PV IPI
 implementation for both amd64 and i386. I've also updated the wiki to
 point to the pvhvm_v10 branch:
 
 I've been running a VM with this kernel for about a week now. It ran fine, 
 until about 3:30 in the morning. The only thing I can see is the following 
 cryptic messages in /var/log/messages, followed by a reboot of the system.
 
 May 29 23:42:30 image01 sshd[31227]: error: Received disconnect from 
 150.165.15.175: 11: Bye Bye [preauth]
 May 30 03:30:57 image01 kernel: .
 May 30 03:30:57 image01 ntpd[4436]: ntpd exiting on signal 15
 May 30 03:30:57 image01 kernel: .
 May 30 03:30:58 image01 kernel: .
 May 30 03:31:00 image01 syslogd: exiting on signal 15
 May 30 03:32:52 image01 syslogd: kernel boot file is /boot/kernel/kernel
 May 30 03:32:52 image01 kernel: Copyright (c) 1992-2013 The FreeBSD Project.
 May 30 03:32:52 image01 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 
 1989, 1991, 1992, 1993, 1994
 May 30 03:32:52 image01 kernel: The Regents of the University of California. 
 All rights reserved.
 May 30 03:32:52 image01 kernel: FreeBSD is a registered trademark of The 
 FreeBSD Foundation.
 
 I'm happy to help to gather more information, just tell me what you need.

Hello Jeroen,

So it looks like the system rebooted (but it was not a crash or a
sporadic reboot? the kernel seems to be aware of the reboot request). It
would be interesting if you could provide the output of the serial
console when this happens, that might be helpful. Did you enable
xenconsoled logging?

Also, could you provide more info about your system, Xen version, what
workload was the DomU running, Dom0 kernel version?

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 24/05/13 12:11, Roger Pau Monné wrote:
 On 23/05/13 21:09, Colin Percival wrote:
 On 05/23/13 02:06, Roger Pau Monné wrote:
 On 22/05/13 22:03, Colin Percival wrote:
 Testing on a cr1.8xlarge EC2 instance, I get Xen 4.2, but it ends up with
 a panic -- console output below.  I can get a backtrace and possibly even
 a dump if those would help.

 Thanks for the test, I've been using Xen 4.2 (and 4.3) without problems 
 so far. By looking at the Xen code, the only reason the timer setup 
 could return -22 (EINVAL), is that we try to set the timer for a 
 different vCPU than the one we are running on.

 I've been able to boot a 32 vCPU DomU on my 8way box using Xen 4.2.1 
 (using both qemu-xen and qemu-xen-traditional device models), so I'm 
 unsure if this could be due to some patch Amazon applies to Xen. Could 
 you try the following patch and post the error message? I would like to 
 see if the cpuid reported by kdb and the vCPU that we are trying to set 
 the timer are the same.

 Looks like there's agreement about the cpuids here.  Anything else I should
 try testing?
 
 Thanks for the test, this is what I expected. I'm a little bit out of
 ideas since I'm not able to reproduce this on upstream Xen 4.2. Without
 knowing what's happening inside the hypervisor it's hard to tell what's
 wrong. It would be interesting to try if the same happens with a Linux
 PVHVM (not PV) running on the same instance type.

Hello Matt,

Colin has found an issue on the FreeBSD PVHVM port that I haven't been
able to reproduce using open source Xen, even when using the same
version as the one reported by EC2. Is there anyway you could provide
some help debugging this? Without seeing the Xen code that causes
VCPUOP_set_singleshot_timer to return EINVAL it is quite hard to figure
out what's happening inside the hypervisor.

Thanks, Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [Xen-devel] FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 24/05/13 16:14, Konrad Rzeszutek Wilk wrote:
 On Thu, May 23, 2013 at 07:41:50PM +0200, Roger Pau Monné wrote:
 Hello,

 I've pushed a new branch, pvhvm_v10 that contains a PV IPI
 implementation for both amd64 and i386. I've also updated the wiki to
 point to the pvhvm_v10 branch:
 
 I feel a bit stupid to ask this, but how I install 'gmake'? Doing 'pkg_add -r 
 gmake'
 tells me there is no package (perhaps I am using a too modern version of 
 FreeBSD
 (FreeBSD-10.0-CURRENT-amd64-20130512-r250582-release.iso)?
 
 The Wiki mentions how to install git but that fails b/c it can't find gmake.

Did you install the ports tree during the installation? If so I've
always successfully installed git using:

# whereis git
# cd output of above command
# make install

Maybe the ISO you picked as a broken ports snapshot?
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 22/05/13 22:03, Colin Percival wrote:
 On 05/22/13 04:45, Roger Pau Monné wrote:
 On 18/05/13 17:44, Colin Percival wrote:
 That seems to work.  dmesg is attached.  Are there any particular tests
 you'd like me to run?

 I have not tested ZFS, that might be a good one. If you are running this
 on Xen 3.4 the behaviour should be the same as without this patches, so
 there shouldn't be many differences.
 
 I don't use ZFS personally, so I'm not sure exactly what tests to run on it;
 hopefully someone else can take care of that.
 
 If you could try that on Xen 4.0 at least (if I remember correctly
 that's when the vector callback was introduced), you should see the PV
 timer getting attached, and a performance increase.
 
 Testing on a cr1.8xlarge EC2 instance, I get Xen 4.2, but it ends up with
 a panic -- console output below.  I can get a backtrace and possibly even
 a dump if those would help.

Hello Colin,

Thanks for the test, I've been using Xen 4.2 (and 4.3) without problems 
so far. By looking at the Xen code, the only reason the timer setup 
could return -22 (EINVAL), is that we try to set the timer for a 
different vCPU than the one we are running on.

I've been able to boot a 32 vCPU DomU on my 8way box using Xen 4.2.1 
(using both qemu-xen and qemu-xen-traditional device models), so I'm 
unsure if this could be due to some patch Amazon applies to Xen. Could 
you try the following patch and post the error message? I would like to 
see if the cpuid reported by kdb and the vCPU that we are trying to set 
the timer are the same.

Booting...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #68: Wed May 22 19:00:14 CEST 2013
root@:/usr/obj/usr/src/sys/XENHVM amd64
FreeBSD clang version 3.3 (trunk 178860) 20130405
WARNING: WITNESS option enabled, expect reduced performance.
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU   W3550  @ 3.07GHz (3066.83-MHz K8-class CPU)
  Origin = GenuineIntel  Id = 0x106a5  Family = 0x6  Model = 0x1a  Stepping = 
5
  
Features=0x1783fbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT
  Features2=0x81b82201SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,HV
  AMD Features=0x28100800SYSCALL,NX,RDTSCP,LM
  AMD Features2=0x1LAHF
real memory  = 4286578688 (4088 MB)
avail memory = 3961323520 (3777 MB)
Event timer LAPIC quality 400
ACPI APIC Table: Xen HVM
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 2 package(s) x 16 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  2
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  6
 cpu4 (AP): APIC ID:  8
 cpu5 (AP): APIC ID: 10
 cpu6 (AP): APIC ID: 12
 cpu7 (AP): APIC ID: 14
 cpu8 (AP): APIC ID: 16
 cpu9 (AP): APIC ID: 18
 cpu10 (AP): APIC ID: 20
 cpu11 (AP): APIC ID: 22
 cpu12 (AP): APIC ID: 24
 cpu13 (AP): APIC ID: 26
 cpu14 (AP): APIC ID: 28
 cpu15 (AP): APIC ID: 30
 cpu16 (AP): APIC ID: 32
 cpu17 (AP): APIC ID: 34
 cpu18 (AP): APIC ID: 36
 cpu19 (AP): APIC ID: 38
 cpu20 (AP): APIC ID: 40
 cpu21 (AP): APIC ID: 42
 cpu22 (AP): APIC ID: 44
 cpu23 (AP): APIC ID: 46
 cpu24 (AP): APIC ID: 48
 cpu25 (AP): APIC ID: 50
 cpu26 (AP): APIC ID: 52
 cpu27 (AP): APIC ID: 54
 cpu28 (AP): APIC ID: 56
 cpu29 (AP): APIC ID: 58
 cpu30 (AP): APIC ID: 60
 cpu31 (AP): APIC ID: 62
random device not loaded; using insecure entropy
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 Version 1.1 irqs 0-47 on motherboard
kbd1 at kbdmux0
xen_et0: Xen PV Clock on motherboard
Event timer XENTIMER frequency 10 Hz quality 950
Timecounter XENTIMER frequency 10 Hz quality 950
acpi0: Xen on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
acpi0: reservation of 0, a (3) failed
cpu0: ACPI CPU on acpi0
cpu1: ACPI CPU on acpi0
cpu2: ACPI CPU on acpi0
cpu3: ACPI CPU on acpi0
cpu4: ACPI CPU on acpi0
cpu5: ACPI CPU on acpi0
cpu6: ACPI CPU on acpi0
cpu7: ACPI CPU on acpi0
cpu8: ACPI CPU on acpi0
cpu9: ACPI CPU on acpi0
cpu10: ACPI CPU on acpi0
cpu11: ACPI CPU on acpi0
cpu12: ACPI CPU on acpi0
cpu13: ACPI CPU on acpi0
cpu14: ACPI CPU on acpi0
cpu15: ACPI CPU on acpi0
cpu16: ACPI CPU on acpi0
cpu17: ACPI CPU on acpi0
cpu18: ACPI CPU on acpi0
cpu19: ACPI CPU on acpi0
cpu20: ACPI CPU on acpi0
cpu21: ACPI CPU on acpi0
cpu22: ACPI CPU on acpi0
cpu23: ACPI CPU on acpi0
cpu24: ACPI CPU on acpi0
cpu25: ACPI CPU on acpi0
cpu26: ACPI CPU on acpi0
cpu27: ACPI CPU on acpi0
cpu28: ACPI CPU on acpi0
cpu29: ACPI CPU on acpi0
cpu30: ACPI CPU on acpi0
cpu31: ACPI CPU on acpi0
hpet0: High Precision Event Timer iomem 0xfed0-0xfed003ff on acpi0
Timecounter HPET frequency 6250 Hz quality 950
attimer0

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 23/05/13 15:20, Jeroen van der Ham wrote:
 
 On 13 May 2013, at 20:32, Roger Pau Monné roger@citrix.com wrote:
 Also, I've created a wiki page that explains how to set up a FreeBSD
 PVHVM for testing:

 http://wiki.xen.org/wiki/Testing_FreeBSD_PVHVM
 
 
 You mention on that page that it is easier to install on 10.0-CURRENT 
 snapshots.
 What are the issues with installing this on 9.1? Is it possible?

I don't think it is recommended to use a HEAD (10) kernel with a 9.1
userland. You can always install a 9.1 and then do a full update with
the source on my repository.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 23/05/13 14:57, Jeroen van der Ham wrote:
 Hi,
 
 On 13 May 2013, at 20:32, Roger Pau Monné roger@citrix.com wrote:
 Right now the code is in a state where it can be tested by users, so we
 would like to encourage FreeBSD and Xen users to test it and provide
 feedback.
 
 I've just been able to install it on a VPS using the latest pvhvm_v9 branch.

The branch pvhvm_v9 contains an initial implementation of PV IPIs for
amd64. I've now finished it and I'm going to port it to i386 also, and
push a new branch to the repository.

 This is good news, because the system I had before actually had trouble with 
 the HVM kernel from 9.1 [0].
 
 I'm going to leave this running for a while and do some more tests on it.
 
 Jeroen.
 
 
 [0]: http://www.freebsd.org/cgi/query-pr.cgi?pr=175822
 

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 23/05/13 18:30, Outback Dingo wrote:
 
 
 
 On Thu, May 23, 2013 at 9:33 AM, Roger Pau Monné roger@citrix.com
 mailto:roger@citrix.com wrote:
 
 On 23/05/13 14:57, Jeroen van der Ham wrote:
  Hi,
 
  On 13 May 2013, at 20:32, Roger Pau Monné roger@citrix.com
 mailto:roger@citrix.com wrote:
  Right now the code is in a state where it can be tested by users,
 so we
  would like to encourage FreeBSD and Xen users to test it and provide
  feedback.
 
  I've just been able to install it on a VPS using the latest
 pvhvm_v9 branch.
 
 The branch pvhvm_v9 contains an initial implementation of PV IPIs for
 amd64. I've now finished it and I'm going to port it to i386 also, and
 push a new branch to the repository.
 
  This is good news, because the system I had before actually had
 trouble with the HVM kernel from 9.1 [0].
 
  I'm going to leave this running for a while and do some more tests
 on it.
 
  Jeroen.
 
 
  [0]: http://www.freebsd.org/cgi/query-pr.cgi?pr=175822
 
 
 I built the rev_9 branch on a XCP host and rebooted, however I am seeing
 
 on boot after ugen0.2: QEMU 0.10.2 at usbus0
 
 run_interrupt_driven_hooks: still waiting after 60 seconds for
 xenbus_nop_confighook_cb
 run_interrupt_driven_hooks: still waiting after 120 seconds for
 xenbus_nop_confighook_cb
 run_interrupt_driven_hooks: still waiting after 180 seconds for
 xenbus_nop_confighook_cb
 run_interrupt_driven_hooks: still waiting after 240 seconds for
 xenbus_nop_confighook_cb
 run_interrupt_driven_hooks: still waiting after 300 seconds for
 xenbus_nop_confighook_cb
 panic: run_interrupt_driven_confighooks: waited too long
 cpuid = 0
 KDB: enter: panic
 [ thread pid 0 tid 10 ]
 Stropped at kdb_enter +0x3b: movq $0,0xad6522(%rip)
 db

From what I've read on the list, it seems like you cannot boot the PVHVM
kernel if you have a cdrom attached to the guest, could you try
disabling the cdrom and booting again?

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

Hello,

I've pushed a new branch, pvhvm_v10 that contains a PV IPI
implementation for both amd64 and i386. I've also updated the wiki to
point to the pvhvm_v10 branch:

http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/pvhvm_v10

I've updated my tree to latest HEAD, so now branch pvhvm_v10 is on top
of this commit:

commit b44da0fb82647f2cfb06f65a6695c7e36c98828c
Author: gber g...@freebsd.org
Date:   Thu May 23 12:24:46 2013 +

Rework and organize pmap_enter_locked() function.

pmap_enter_locked() implementation was very ambiguous and confusing.
Rearrange it so that each part of the mapping creation is separated.
Avoid walking through the redundant conditions.
Extract vector_page specific PTE setup from normal PTE setting.

Submitted by:   Zbigniew Bodek z...@semihalf.com
Sponsored by:   The FreeBSD Foundation, Semihalf

Thanks for the testing, Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [Xen-devel] FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 21/05/13 19:40, Konrad Rzeszutek Wilk wrote:
 On Mon, May 13, 2013 at 08:32:56PM +0200, Roger Pau Monné wrote:
 Hello,

 Recently Justin T Gibbs, Will Andrews and myself have been working on
 improving the Xen support in FreeBSD. The main goal of this was to bring
 full PVHVM support to FreeBSD, right now FreeBSD is only using PV
 interfaces for disk and network interfaces when running as a HVM guest.
 The main benefits of this changes are that Xen virtual interrupts (event
 channels) are now delivered to the guest using a vector callback
 injection, that is a per-cpu mechanism that allows each vCPU to have
 different interrupts assigned, so for example network and disk
 interrupts are delivered to different vCPUs in order to improve
 performance. With this changes FreeBSD also uses PV timers when running
 as an HVM guest, which should provide better time keeping and reduce the
 virtualization overhead, since emulated timers are no longer used. PV
 IPIs can also be used inside a HVM guest, but this will be implemented
 later.

 Right now the code is in a state where it can be tested by users, so we
 would like to encourage FreeBSD and Xen users to test it and provide
 feedback.

 The code is available in the following git repository, under the branch
 pvhvm_v5:

 http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=summary

 Also, I've created a wiki page that explains how to set up a FreeBSD
 PVHVM for testing:

 http://wiki.xen.org/wiki/Testing_FreeBSD_PVHVM
 
 I tried on my Linux box to do this:
 
 
 HEAD is now at 9b25356... xen-netfront: fix detach of network interfaces
 konrad@phenom:~/git/freebsd$ make kernel-toolchain  make buildkernel 
 KERNCONF=XENHVM  make installkernel KERNCONF=XENHVM
 Makefile:123: *** missing separator.  Stop.
 
 
 As I thought it would compile the same way you can compile NetBSD - that is
 even on non-BSD distros. Is that not the case? Should I only do this under
 a FreeBSD guest?

I have never tried to run the FreeBSD build system under something
different than FreeBSD, also keep in mind that installkernel will place
a bunch of FreeBSD files on your Linux box if you manage to run it. The
error itself can probably be fixed by installing and using BSD make
(bmake), but anyway you need a FreeBSD HVM DomU in order to test the
kernel, so why not do the compilation on it?

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 18/05/13 17:44, Colin Percival wrote:
 On 05/18/13 02:50, Roger Pau Monné wrote:
 On 17/05/13 05:07, Colin Percival wrote:
 On 05/16/13 17:43, Roger Pau Monné wrote:
 Thanks for testing this on EC2, could you post the full dmesg? So I can
 see the hypervisor version and if the PV timer is loaded or not.

 Here's what I get on a cc2.8xlarge with boot_verbose=YES:

 I've pushed a new branch to my repository, pvhvm_v7 that should work,
 there was a bug with PCI event channel interrupt set up. I've tested
 with 3.4 and seems OK, but of course it doesn't support the vector
 callback injection.
 
 That seems to work.  dmesg is attached.  Are there any particular tests
 you'd like me to run?

I have not tested ZFS, that might be a good one. If you are running this
on Xen 3.4 the behaviour should be the same as without this patches, so
there shouldn't be many differences.

If you could try that on Xen 4.0 at least (if I remember correctly
that's when the vector callback was introduced), you should see the PV
timer getting attached, and a performance increase.

 If anyone else wants to play with this, you can launch ami-e75c358e in the
 EC2 us-east-1 region.
 

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 17/05/13 05:07, Colin Percival wrote:
 On 05/16/13 17:43, Roger Pau Monné wrote:
 Thanks for testing this on EC2, could you post the full dmesg? So I can
 see the hypervisor version and if the PV timer is loaded or not.
 
 Here's what I get on a cc2.8xlarge with boot_verbose=YES:

I've pushed a new branch to my repository, pvhvm_v7 that should work,
there was a bug with PCI event channel interrupt set up. I've tested
with 3.4 and seems OK, but of course it doesn't support the vector
callback injection.

Regards, Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD PVHVM call for testing

[Roger Pau Monné]

On 16/05/13 19:55, Colin Percival wrote:
 On 05/13/13 11:32, Roger Pau Monné wrote:
 Right now the code is in a state where it can be tested by users, so we
 would like to encourage FreeBSD and Xen users to test it and provide
 feedback.

 The code is available in the following git repository, under the branch
 pvhvm_v5:

 http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=summary

 Also, I've created a wiki page that explains how to set up a FreeBSD
 PVHVM for testing:

 http://wiki.xen.org/wiki/Testing_FreeBSD_PVHVM
 
 I built a XENHVM kernel with this code on EC2, and it hanged after
 xenbusb_front0: Xen Frontend Devices on xenstore0
 
 With a XENHVM kernel from FreeBSD HEAD the next line after that is
 xbd0: 10240MB Virtual Block Device at device/vbd/768 on xenbusb_front0
 
 Any ideas?

Hello Colin,

Thanks for testing this on EC2, could you post the full dmesg? So I can
see the hypervisor version and if the PV timer is loaded or not.

Roger.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

FreeBSD PVHVM call for testing

[Roger Pau Monné]

Hello,

Recently Justin T Gibbs, Will Andrews and myself have been working on
improving the Xen support in FreeBSD. The main goal of this was to bring
full PVHVM support to FreeBSD, right now FreeBSD is only using PV
interfaces for disk and network interfaces when running as a HVM guest.
The main benefits of this changes are that Xen virtual interrupts (event
channels) are now delivered to the guest using a vector callback
injection, that is a per-cpu mechanism that allows each vCPU to have
different interrupts assigned, so for example network and disk
interrupts are delivered to different vCPUs in order to improve
performance. With this changes FreeBSD also uses PV timers when running
as an HVM guest, which should provide better time keeping and reduce the
virtualization overhead, since emulated timers are no longer used. PV
IPIs can also be used inside a HVM guest, but this will be implemented
later.

Right now the code is in a state where it can be tested by users, so we
would like to encourage FreeBSD and Xen users to test it and provide
feedback.

The code is available in the following git repository, under the branch
pvhvm_v5:

http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=summary

Also, I've created a wiki page that explains how to set up a FreeBSD
PVHVM for testing:

http://wiki.xen.org/wiki/Testing_FreeBSD_PVHVM
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Difference in event channel implementation for Xen PV vs HVM guests

[Roger Pau Monné]

On 18/03/13 14:08, Justin T. Gibbs wrote:
  Hi Roger,
 
 I know of no reasons why XENHVM cannot use the full event channel
 interface.  In fact, Spectra Logic implemented PV timers and a
 general cleanup of the HVM event channel interface.  I haven't merged
 it back yet because I know the changes break PV and I haven't found
 time to clean up the PV code, merge the HVM and PV event channel
 systems, and move IPI/MSI delivery to event channels.
 
 I've uploaded Spectra's changes here:
 
   http://people.freebsd.org/~gibbs/xen_ev/

Hi Justin,

There's at least one file which seems to be missing:

sys/x86/xen/hvm.c

Could you check if you have that file around?

Thanks, Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Difference in event channel implementation for Xen PV vs HVM guests

[Roger Pau Monné]

On 21/03/13 13:43, Roger Pau Monné wrote:
 On 18/03/13 14:08, Justin T. Gibbs wrote:
   Hi Roger,

 I know of no reasons why XENHVM cannot use the full event channel
 interface.  In fact, Spectra Logic implemented PV timers and a
 general cleanup of the HVM event channel interface.  I haven't merged
 it back yet because I know the changes break PV and I haven't found
 time to clean up the PV code, merge the HVM and PV event channel
 systems, and move IPI/MSI delivery to event channels.

 I've uploaded Spectra's changes here:

  http://people.freebsd.org/~gibbs/xen_ev/
 
 Hi Justin,
 
 There's at least one file which seems to be missing:

Ops, there at least two files missing:

sys/x86/xen/hvm.c
sys/xen/evtchn/evtchnvar.h
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Difference in event channel implementation for Xen PV vs HVM guests

[Roger Pau Monné]

On 18/03/13 14:08, Justin T. Gibbs wrote:
 On Mar 18, 2013, at 6:35 AM, Roger Pau Monné roger@citrix.com wrote:
 
 Hello,

 While working on improving XENHVM (I've been looking at adding PV
 timers), I've realized that the event channel implementation in PV vs
 HVM mode differs greatly. Xen PV port uses sys/xen/evtchn/evtchn.c while
 Xen HVM uses sys/dev/xenpci/evtchn.c, and the Xen HVM implementation is
 greatly reduced (only contains the necessary functions to operate
 backends/frontends).

 To implement PV timers I need to expand the event channel interface for
 XENHVM, and I was wondering why FreeBSD choose to have two different
 implementations, the main difference between PV and HVM is the event
 callback, but I guess this can be abstracted between the two different
 implementations, and then everything else could be reused. Am I missing
 something obvious?

 Is there any known technical problem in modifying XENHVM to use the full
 event channel implementation present in sys/xen/evtchn/evtchn.c that
 prevented XENHVM from using it in the first place?

 (Sorry if I've Cc'ed someone not related)

 Thanks, Roger.
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
 
 Hi Roger,

Hello Justin,

 
 I know of no reasons why XENHVM cannot use the full event channel
 interface.  In fact, Spectra Logic implemented PV timers and a
 general cleanup of the HVM event channel interface.  I haven't merged
 it back yet because I know the changes break PV and I haven't found
 time to clean up the PV code, merge the HVM and PV event channel
 systems, and move IPI/MSI delivery to event channels.

That sounds great, then FreeBSD will have a fully working PVHVM port.

 I've uploaded Spectra's changes here:
 
   http://people.freebsd.org/~gibbs/xen_ev/
 
 The diffs file provides the history of the original checkins to our
 Perforce repository.  The tar file includes all the files that have
 been modified and reflects our efforts to keep our code base in
 sync with stable/9.  Apart from the PV issues outlined above, I
 would expect the code provided to just drop right in to stable/9.

Wow, this is a lot of work to keep in a closet, I will start by rebasing
those on top of HEAD, and probably try to split them into smaller
commits that make logical sense.

 
 Unfortunately, Xen support is not a current priority for Spectra
 so I don't have a lot of day job time to focus on getting this code back
 into FreeBSD.  However, if this code looks like it would suite
 your needs, and you have resources for testing i386/PV, I'd be happy
 to collaborate with you and will make the time to help get this
 committed.

Definitely, you guys did already a lot of work, and it will a shame to
lose it, right now I got my plate quite full, but I expect to get on
with this in a couple of weeks.

Thanks, Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Difference in event channel implementation for Xen PV vs HVM guests

[Roger Pau Monné]

Hello,

While working on improving XENHVM (I've been looking at adding PV
timers), I've realized that the event channel implementation in PV vs
HVM mode differs greatly. Xen PV port uses sys/xen/evtchn/evtchn.c while
Xen HVM uses sys/dev/xenpci/evtchn.c, and the Xen HVM implementation is
greatly reduced (only contains the necessary functions to operate
backends/frontends).

To implement PV timers I need to expand the event channel interface for
XENHVM, and I was wondering why FreeBSD choose to have two different
implementations, the main difference between PV and HVM is the event
callback, but I guess this can be abstracted between the two different
implementations, and then everything else could be reused. Am I missing
something obvious?

Is there any known technical problem in modifying XENHVM to use the full
event channel implementation present in sys/xen/evtchn/evtchn.c that
prevented XENHVM from using it in the first place?

(Sorry if I've Cc'ed someone not related)

Thanks, Roger.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org