Re: help needed with UEFI BIOS and CSM

[Roger Pau Monné via freebsd-xen]

On Mon, May 24, 2021 at 05:15:25PM +0200, lizbethmutterh...@gmail.com wrote:
> I CSMd the BIOS and so I have the start partition, although every system on 
> the 1TB hdd is UEFI (FreeBSD, too). I want to start the hypervisor kernel in 
> advance, made a kernel and builtworld but I'm unsure what to do now?

I'm afraid I'm not sure I understand what you want to achieve.

Please note that FreeBSD HEAD supports booting Xen from UEFI now also,
so you are no longer tied to using legacy BIOS.

If you want to boot Xen you need to install the xen-kernel package and
add the xen_kernel and xen_cmdline options to your loader.conf, see:

https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-xen

You likely also want to install the xen-tools package in order to
manage VMs.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [SUSPECTED SPAM]Re: HEADS UP: FreeBSD/Xen dom0 UEFI support

[Roger Pau Monné via freebsd-xen]

On Sun, May 16, 2021 at 07:17:46PM -0600, Stephen Walker-Weinshenker wrote:
> > > xen_cmdline="dom0_mem=2048M dom0_max_vcpus=1 dom0=pvh com1=115200,8n1
> > > guest_loglvl=all loglvl=all console=vga,com1"
> > Can you try adding 'watchdog' to xen_cmdline and see if that forces
> > the box to reboot after it goes black?
> >
> > If Xen is able to boot, and the problem is with FreeBSD dom0 you might
> > try also adding vga=keep to xen_cmdline, that way we might get some
> > output from dom0 (if that's indeed where the issue is happening).
> >
> > > xen-kernel version = 4.14.1_1
> > > xen-tools version = 4.14.1_1
> > >
> > > Both installed from packages, rather than ports.
> > >
> > > These boxes are Intel NUCs that I normally manage over intel AMT using 
> > > mesh
> > > commander. I tried enabling the virtual serial port through AMT but no
> > > dice.
> >
> > I never tried using the AMT serial port, first thing would be to get
> > it to work with plain FreeBSD, then we can see about getting it
> > working with Xen. FWIW, a lot of the test boxes I work with use SoL,
> > and I would expect the AMT serial to work as well.
> 
> No dice on the AMT serial port, but I have been less than impressed with
> MeshCommander and AMT thus far.
> >
> > > I know the nucs have a hardware serial port, but I don't have the
> > > cable to attach to the header at the moment. I have ordered some headers
> > > and crimp pins to attach to the serial port for more debugging, but they
> > > will not get here for about a week.
> >
> > I did the original EFI work on a NUC7i7DNHE using the serial port.
> > Last time I tried HDMI worked with Xen.
> 
> I was able to get the hardware serial port working. turned out I had a typo in
> my /boot/loader.conf. Commconsole instead of comconsole. That also flagged
> a different error that revealed that vidconsole should be replaced with efi on
> UEFI systems
> >
> > > I also tried connecting directly to the NUC's HDMI port but I just got a
> > > blank screen again, after all the boot messages appeared.
> >
> > Are those messages that you see Xen boot messages or FreeBSD boot
> > messages?
> >
> > Does the box reboot itself?
> 
> No
> 
> >
> > I'm afraid we will require some more information to help diagnose
> > what's going on.
> 
> Tried adding both of the options that you mentioned above to xen_cmdline with 
> no
> appreciable difference.
> The system did not appear to attempt to reboot itself. See serial
> console output below.
> 
> After the Beastie menu, I get the following output on both HDMI and
> serial consoles,
> and then no further output on the serial console, while the HDMI
> console goes blank.
> 
> 
> Loading Xen kernel...
> /boot/xen data=0x26b9a8+0x146658 -
> Loading kernel...
> /boot/kernel/kernel size=0x1debd70
> Loading configured modules...
> /boot/entropy size=0x1000
> /etc/hostid size=0x25
> EFI framebuffer information:
> addr, size 0x40, 0x1d4c00
> dimensions 800 x 600
> stride 800
> masks  0x00ff, 0xff00, 0x00ff, 0xff00
> 
> I am happy to do additional debugging, but I personally am out of my depth 
> here.

So there doesn't seem to be any output at all from Xen.

Can you paste your full /boot/loader.conf?

Can you try to swap com1 with com2 on the xen_cmdline and see if that
makes any difference. Xen doesn't really have any logic to figure out
which is the proper serial port to use.

Also there should now be a binary xen-kernel 4.15 package, so you can
also try with that one to see if it makes a difference.

I've only been able to test multiboot2 EFI loading on a single box, so
it's possible it has some sharp edges, sorry.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Issue with packet framing on xnb(4) connections and NetBSD domus

[Roger Pau Monné via freebsd-xen]

On Wed, Mar 17, 2021 at 01:23:46AM -0700, Brian Buhrow wrote:
>   hello.  I've noticed that on my Xen server, running 
> FreeBSD-12.2/Xen-4.14.0, I see a lot
> of messages like the following from my NetBSD-5 and NetBSD-current domu's: 
> xennet0: discarding oversize frame (len=1518)
> 
> They happen when network traffic is heavy and when traffic is coming from a 
> physical switch in
> the same vlan as the domu's.  Since I don't think packets are  actually 
> oversized on the wire,
> the Cisco switch managing this network does not report any giant packets, I 
> think this is a
> software problem either on the NetBSD domu's, or the FreeBSD dom0.  Since 
> I've been running
> these same NetBSD domu setups on a machine where NetBSD is the dom0, I think 
> this is an issue
> with the xnb(4) or netback.c  driver.  Specifically, it looks like it 
> concatinates multiple
> packets together before notifying the domu that there is traffic available.  
> Perhaps that is by
> design, since I cannot reproduce the issue on a FreeBSD-domu running on the 
> same FreeBSD-dom0
> machine.  In either case, while things are usable, it creates a very noisy 
> log on the
> NetBSD-domu machines.  Does the xnb(4) driver forward multiple packets to the 
> domu front ends
> and set the length of the message to the sum of the lengths of all the 
> packets it's forwarding,
> leaving it to the domu front end driver to separate the packets on reception? 
>  Is there a
> parameter that can be set on a per-domu basis to say whether you want
> multiple-packets-per-transfer or not from the dom0?

I think this is an issue with FreeBSD netback as it doesn't properly
handle frontends that don't understand some of the features
implemented in the FreeBSD backend and are enabled unconditionally.

Could you paste the output of `xenstore-ls -fp` from dom0 when one of
those NetBSD domUs is running?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Strange error messages from xnb on dom0. What do they mean?

[Roger Pau Monné via freebsd-xen]

On Mon, Apr 26, 2021 at 11:41:13AM -0700, Brian Buhrow wrote:
>   hello.  Occasionally I see messages like the following from my 
> FreeBSD-12.2/dom0 machine.
> Is this related to the issue of the NetBSD domu's seeing bursts of oversized 
> packets, which I assume
> is caused by packets getting sent from the dom0 to the domu in batches rather 
> than one at a
> time, each with an interrupt signal, or, the xen equivalent?
> 
> On the domu's I see:
> xennet0: discarding oversize frame (len=1518)
> 
> I see the following messages on the dom0, but not necessarily at the same 
> time as the above
> messages appear in the domu's.
> 
> Any ideas?
> -thanks
> -Brian
> 
> 
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_ring2pkt:1532): Unknown extra 
> info type 255.  Discarding packet
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:302): 
> netif_tx_request index =0
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:303): 
> netif_tx_request.gref  =0
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:304): 
> netif_tx_request.offset=0
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:305): 
> netif_tx_request.flags =8
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:306): 
> netif_tx_request.id=69
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:307): 
> netif_tx_request.size  =1000
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:302): 
> netif_tx_request index =1
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:303): 
> netif_tx_request.gref  =255
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:304): 
> netif_tx_request.offset=0
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:305): 
> netif_tx_request.flags =0
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:306): 
> netif_tx_request.id=0
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_dump_txreq:307): 
> netif_tx_request.size  =0
> Apr 25 14:56:35 xen-lothlorien kernel: xnb(xnb_rxpkt2rsp:2066): Got error -1 
> for hypervisor gnttab_copy status

Those are from netback failing to recognize or process a packet.
Likely netback is being too verbose here, as those should only appear
on debug builds or with boot_verbose. It might help identify what's
going wrong between NetBSD netfront and FreeBSD netback.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [PROMOTIONAL]FreeBSD 14-CURRENT as a guest working?

[Roger Pau Monné via freebsd-xen]

On Fri, May 07, 2021 at 05:12:16AM +, Marcin Cieslak wrote:
> I have put FreeBSD-14.0-CURRENT-amd64-20210422-df456a1fcf7-246266.raw.xz
> on an zvol and tried booting it like this:
> 
> builder = "hvm"
> name = "current0"
> disk = [
> '/dev/zvol/zroot/current0,raw,xvda,w'
> ]
> boot = "c"
> bios = "ovmf"
> usbdevice = 'tablet'
> #nographics = 1
> serial = [ "/dev/nmdm0A" ]
> vnc = 1
> #vnclisten = '0.0.0.0'
> vif = 
> ['bridge=bridge0,mac=00:02:04:15:fd:e1','bridge=bridge1,mac=00:02:04:15:fd:e2']
> memory=8192
> vcpus=6
> vga = "stdvga"
> videoram = 16
> ;xen_platform_pci=1
> 
> it starts to boot but subsequently crashes with various messages related to 
> network front drivers.

The following commit solves the issue in the domU:

https://cgit.freebsd.org/src/commit/?id=5d8fd932e418f03e98b3469c4088a36f0ef34ffe

I don't think there's any snapshot image with this yet.

> I tried to collect some info but now an attempt to start it seems to bog down 
> my Xen host :(
> (it responds to TCP connections but SSH connections are frozen and I can't 
> open new ones)

Hm, that's weird. Do you have a serial attached to the box to know
what's going on?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [SUSPECTED SPAM]Re: HEADS UP: FreeBSD/Xen dom0 UEFI support

[Roger Pau Monné via freebsd-xen]

On Sun, Apr 25, 2021 at 05:10:44PM -0600, Stephen Walker-Weinshenker wrote:
> On Thu, Apr 22, 2021 at 12:58 AM Roger Pau Monné 
> wrote:
> 
> > On Wed, Apr 21, 2021 at 08:32:18PM -0600, Stephen Walker-Weinshenker wrote:
> > > Hello Roger:
> > >
> > > Thank you for your work on this. I am new to Xen and FreeBSD but am
> > willing
> > > to help test this and get it in a release version.
> > >
> > > I have 3 Intel NUC 8v5PNK that I am planning on using for Xen hosts, but
> > > they can only boot in UEFI mode (intel disabled BIOS booting per their
> > > official documentation even though it is still an option in the BIOS
> > > /shrug). Until this patch came along I could not get any Xen distribution
> > > to boot since they all require BIOS mode.
> > >
> > > I have 2 of the 3 NUCS running 14.0-CURRENT at git commit
> > > 55fc118be8f0cd20e2123cfd2728c5f32317a9fa (after your patch was merged)
> > and
> > > I have attempted to follow the instructions at
> > >
> > https://docs.freebsd.org/en_US.ISO8859-1/books/handbook/virtualization-host-xen.html
> > > in order to get the Xen kernel booting. Unfortunately, when I rebooted
> > > after making the changes, I get an error saying:
> > >
> > > >Loading Xen kernel...
> > > >Failed to load Xen kernel '/boot/xen'
> > > >can't load 'kernel'
> > > >
> > > >can't load 'kernel'
> >
> > Did you update the loader on the EFI partition?
> >
> > After an installworld on my box I would usually do:
> >
> > # mount -t msdosfs /dev/ada0p1 /mnt/
> > # cp /boot/loader.efi /mnt/EFI/freebsd/loader.efi
> > # umount /mnt
> >
> > To update the EFI loader. You might have to check the output of
> > `efibootmgr -v`to make sure you are replacing the correct loader. I
> > guess there must also be a way to do this with efibootmgr.
> >
> > For whatever reason my EFI partition nvd0p1 was already mounted at
> /boot/efi,
> so my copy command was
> # cp -f /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
> 
> I did this prior to the last reboot after changing the boot config as
> recommended by
> 
> This fixed the issue of the xen kernel not booting on one of the boxes.
> (did the copy prior to the final reboot) Looks like I will have to
> reinstall the OS on the other one (not a big deal)
> 
> Now, I am just getting a blank screen on boot. I am unable to ssh into the
> Dom0 either.

What's your Xen command line? (ie: xen_cmdline string in
loader.conf)

Which version of Xen are you using?

Do you have serial output available on those boxes? That might help
get some messages if everything else fails.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [SUSPECTED SPAM]Re: HEADS UP: FreeBSD/Xen dom0 UEFI support

[Roger Pau Monné via freebsd-xen]

On Wed, Apr 21, 2021 at 08:32:18PM -0600, Stephen Walker-Weinshenker wrote:
> Hello Roger:
> 
> Thank you for your work on this. I am new to Xen and FreeBSD but am willing
> to help test this and get it in a release version.
> 
> I have 3 Intel NUC 8v5PNK that I am planning on using for Xen hosts, but
> they can only boot in UEFI mode (intel disabled BIOS booting per their
> official documentation even though it is still an option in the BIOS
> /shrug). Until this patch came along I could not get any Xen distribution
> to boot since they all require BIOS mode.
> 
> I have 2 of the 3 NUCS running 14.0-CURRENT at git commit
> 55fc118be8f0cd20e2123cfd2728c5f32317a9fa (after your patch was merged) and
> I have attempted to follow the instructions at
> https://docs.freebsd.org/en_US.ISO8859-1/books/handbook/virtualization-host-xen.html
> in order to get the Xen kernel booting. Unfortunately, when I rebooted
> after making the changes, I get an error saying:
> 
> >Loading Xen kernel...
> >Failed to load Xen kernel '/boot/xen'
> >can't load 'kernel'
> >
> >can't load 'kernel'

Did you update the loader on the EFI partition?

After an installworld on my box I would usually do:

# mount -t msdosfs /dev/ada0p1 /mnt/
# cp /boot/loader.efi /mnt/EFI/freebsd/loader.efi
# umount /mnt

To update the EFI loader. You might have to check the output of
`efibootmgr -v`to make sure you are replacing the correct loader. I
guess there must also be a way to do this with efibootmgr.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Corruption in xenstored tdb file?

[Roger Pau Monné via freebsd-xen]

On Wed, Mar 17, 2021 at 04:13:03PM -0700, Brian Buhrow wrote:
>   hello Roger.  I've successfully compiled the kernel with your patch and 
> installed it.
> Now, both the NetBSD-5 and NetBSD-current VM's boot with full working 
> networks.  
> Also, FreeBSD-12 works as a a pvh guest with full networking.
> I applied your patch to 12-stable, so if you commit this fix, if you could 
> request a pullup to
> FreeBSD-12, that would be great!

Thanks for the testing, will commit early next week. Let's see if I
have some time to look at your other issue also.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: 12.2-STABLE r369477 dom0 creash on xen-kernel-4.14.1_1

[Roger Pau Monné via freebsd-xen]

On Fri, Mar 19, 2021 at 11:59:49PM +, Marcin Cieslak wrote:
> Hello,
> 
> I have just upgrade my machine that used to ran 11.x with Xen 4.7.2_9
> as dom0 for a long time.
> 
> After upgrade to 12.2-STABLE r369477, runs GENERIC kernel just fine.
> 
> Unfortunately it crashes as Xen dom0 with xen-kernel-4.14.1_1

Do you have dom0=pvh set on the Xen command line? Note 4.7 used
dom0pvh=1 instead [0], so you will have to modify your loader.conf.

Thanks, Roger.

[0] 
https://docs.freebsd.org/en_US.ISO8859-1/books/handbook/virtualization-host-xen.html
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Corruption in xenstored tdb file?

[Roger Pau Monné via freebsd-xen]

On Tue, Mar 16, 2021 at 12:06:32AM -0700, Brian Buhrow wrote:
>   hello.  Following up on this further, it seems there may be a timing 
> issue related to this
> after all.  If I bring up a NetBSD-5.2 VM, the VM comes up  without a problem 
> and xennet0 works
> just as it should.  I can do this time and time again without any trouble.
> However, if I bring up a NetBSD-99.77 (current as of January 28 2021), I get 
> the behavior I
> described in the previous message.  It's obviously some kind of race 
> condition, since if I
> reboot the NetBSD-current VM several times, I can get it to come up with a 
> network interface
> occasionally.  However, not enough to make it usable.  
>   Also, since I wrote last, I updated to 12.2-release--p4, just to see if 
> that made things
> better.  It did not.  I suspect, but don't know for sure, that the issue is 
> that NetBSD-current
> is issuing commands on the xenbus faster than it did in NetBSD-5.  If that's 
> true, then I think
> the problem lies with FreeBSD, as, in my view, a VM guest shouldn't be able 
> to trigger a race
> condition in the host side of the server, which is what appears to be 
> happening here.  
>   Is there a way to get a trace of the communications between the domU's 
> and the dom0 so I
> can see the differences between what NetBSD used to do  and what it does 
> today?

So I've taken a look and it seems NetBSD now switches to the
XenbusStateInitialised state without having written some of the
configuration data required by netback. It's not clear to me whether
this is a bug in NetBSD, or a bug in FreeBSD netback. In any case
the patch below should fix it, can you apply it to your kernel
sources, recompile and test?

The above fix changes the behavior of FreeBSD netback to only try to
fetch the data when the frontend switches to the Connected state, this
seems to be inline with what Linux netback does, so in any case I
think it's a change worth making.

Thanks, Roger.
---8<---
diff --git a/sys/dev/xen/netback/netback.c b/sys/dev/xen/netback/netback.c
index 44159f60d996..29efd76430c7 100644
--- a/sys/dev/xen/netback/netback.c
+++ b/sys/dev/xen/netback/netback.c
@@ -1392,8 +1392,8 @@ xnb_frontend_changed(device_t dev, XenbusState 
frontend_state)
 
switch (frontend_state) {
case XenbusStateInitialising:
-   break;
case XenbusStateInitialised:
+   break;
case XenbusStateConnected:
xnb_connect(xnb);
break;

___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [PROMOTIONAL]XEN with UEFI guest

[Roger Pau Monné via freebsd-xen]

On Sun, Mar 14, 2021 at 11:30:33PM +0300, Oleg Ginzburg wrote:
> Hi,
> 
> I recently tried to use the Xen dom0 through UEFI loader and, finally, it
> worked! (Thanks to Roger).
> 
> Now I try to use UEFI boot method in the guest and for some reason, the
> domain does not start.
> The documentation says that this option requires extra config params, so I
> added an option to port:
> 
> --
> root@home2:/usr/ports/sysutils/xen-tools# svnlite diff
> Index: Makefile
> ===
> --- Makefile(revision 568404)
> +++ Makefile(working copy)
> @@ -21,12 +21,14 @@
> BUILD_DEPENDS= seabios>0:misc/seabios
> RUN_DEPENDS=   seabios>0:misc/seabios
> 
> -OPTIONS_DEFINE=DOCS SPICE
> +OPTIONS_DEFINE=DOCS SPICE OVMF
> OPTIONS_DEFAULT=   DOCS
> OPTIONS_SUB=   yes
> 
> SPICE_DESC=Enable SPICE protocol for QEMU
> +OVMF_DESC= Enable OVMF support
> SPICE_CONFIGURE_WITH=  extra-qemuu-configure-args="--enable-spice"
> +OVMF_CONFIGURE_WITH=   extra-qemuu-configure-args="--enable-ovmf"

You are adding the option to the QEMU configure script instead of the
Xen one, you likely want:

OVMF_CONFIGURE_WITH=   enable-ovmf

Albeit I think that won't work out of the box because OVMF will fail
to build with llvm. So you will likely have to build ovmf separately
(like we do for seabios) using gcc and then include it here with
--with-system-ovmf=...

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

HEADS UP: FreeBSD/Xen dom0 UEFI support

[Roger Pau Monné via freebsd-xen]

Hello,

Since commit 97527e9c4fd37140 on main branch FreeBSD should be able to
boot and work as a Xen dom0 from UEFI.

Booting from UEFI also requires the usage of xen-kernel 4.14.1_1,
previous versions of xen-kernel won't boot correctly under UEFI.

The way to setup the system is exactly the same as when using BIOS,
xen_kernel and xen_cmdline options should be set in loader.conf, see:

https://docs.freebsd.org/en_US.ISO8859-1/books/handbook/virtualization-host-xen.html

I don't have plans to backport for 13.0, but will consider backporting
to stable/13 at a later point if there are no issues, so the
functionality could be used in 13.1.

This has been tested on a single Intel box with UEFI, some wider
testing would be appreciated.

Thanks to tsoome, imp and kib for the reviews.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Disruptions to PVH dom0 in main and stable/13 branches

[Roger Pau Monné]

Hello,

If you are not based on the main or the stable/13 branches you can
skip the email below.

There have been some bootloader changes in main and stable/13 branches
that prevented booting a FreeBSD/Xen PVH Dom0. A fix has been committed
to main and should hopefully be backported to stable/13 in due time,
it's:

b6d85a5f51e4 stand/multiboot: adjust the protocol between loader and kernel

That however will require that you also use a Xen package version
greater or equal to 4.14.1_0. I've just added those to the ports
repository:

r563208: emulators/xen-kernel,sysutils/xen-tools: update to 4.14.1

However binary packages will take some time to appear. In the mean
time, if you wish to use a Xen dom0 please make sure to run a build of
FreeBSD that has the above commit applied, and to build the Xen
packages from the ports tree yourself (while a binary one is not
available).

Sorry for the inconvenience.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Corruption in xenstored tdb file?

[Roger Pau Monné via freebsd-xen]

On Thu, Jan 28, 2021 at 05:11:04PM -0800, Brian Buhrow wrote:
>   hello.  Recently One of my FreeBSD-xen servers went down hard due to a 
> power failure.
> When it came back up, one of the domus' came up, but with no network 
> interface.  Dmesg
> messages show the following errors.  To fix the issue, I removed the 
> /var/lib/xenstored/tdb
> file and rebooted.  Before that, I couldn't get this domu to boot with a 
> network interface.  My
> question is, is it reasonable to take this action when such an error occurs 
> and, further, might
> it be better to remove that file before attempting to start xen on reboots, 
> just in case
> something didn't go down cleanly?

Yes, I think it should be placed in a directory that's cleaned on
reboot, it's not expected that people have to go clean this
themselves.

Do you have any recommendation where this should be placed?

/var/run/xen would seem like a suitable place.

Thanks, Roger
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: either xen or custom

[Roger Pau Monné]

Adding freebsd-xen mailing list for reference.

On Tue, Jan 19, 2021 at 05:04:21PM +0100, Lizbeth Mutterhunt, PhD wrote:
> Op Dienstag, 19. Jänner 2021 15:06:45 CET schreef u:
> > On Fri, Jan 15, 2021 at 3:46 PM Lizbeth Mutterhunt, PhD
> > 
> >  wrote:
> > > Op Donnerstag, 14. Jänner 2021 09:21:09 CET schreef u:
> Hi Roger, 
> 
> just to introduce myself a bit:
> 
> !!!thx for your reply and concerns!!! 
> 
> I keep in mind that this certain kind of support isn't self-understanding and 
> will one day be to your advantage, too! For myself: I ask to much and answer 
> too little in forums, etc., as I know what to do in certain kind of matters 
> but still too lazy to give a tipp or to post a reply as many others does. 
> 
> I'm good in linux, especially Arch Linux, doing projects at the reborn BeOS, 
> Linux and now as my main system on 
> 
> [CODE]
> sudo sysctl hw.model hw.machine hw.ncpu
> Wachtwoord: 
> hw.model: Intel(R) Celeron(R) CPU 867 @ 1.30GHz

So your CPU doesn't support VT-d (AKA IOMMU), and won't be able to
run FreeBSD/Xen sadly. This is a hardware limitation and there's
nothing that can be done. The only way to run Xen on that box is to
use Linux in PV mode, which FreeBSD doesn't support.

You can check the CPU features at:

https://ark.intel.com/content/www/us/en/ark/products/63918/intel-celeron-processor-867-2m-cache-1-30-ghz.html

Search for VT-d.

If you want to run FreeBSD/Xen you will need a newer CPU, sadly
there's no way around that.

> > > > Do you have a serial console attached to the box so that we could
> > > > see the log of Xen and FreeBSD booting?
> > > 
> > > how to? every time I boot into xen I have to zpool import -f and mount it
> > > from rescue CD. Is there a log from the last boot somewhere? I
> > > duckduckgo'd a bit but couldn't find anything else than the dmesg. As it
> > > is not a kernel crash there's nothing in /var/crash and on /var/run I
> > > have the running boot inside.
> 
> > You would need to plug a null-modem into the serial port of your box,
> > or alternatively use something like SoL (Serial over LAN) if your box
> > supports it.
> 
> could you plz. explain this a bit more exactly? I don't have a LAN-switch yet 
> but it's planned for connecting the Raspi4 and my upcoming RAPTOR9 machine at 
> easter and to have a "night-light" with blinking RX/TX - LEDS! :-)

Getting serial out of a box is the only way to diagnose early boot
problems (ie: kernel issues). See:

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html

That way you can get the output of FreeBSD or Xen booting in text
mode, so that you can for example paste it on an email when things go
wrong.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Re: either xen or custom

[Roger Pau Monné]

On Fri, Jan 15, 2021 at 3:46 PM Lizbeth Mutterhunt, PhD
 wrote:
>
> Op Donnerstag, 14. Jänner 2021 09:21:09 CET schreef u:
> > On Sun, Dec 13, 2020 at 12:11:40AM +0100, lizbethmutterh...@gmail.com wrote:
> > > to make a long story short: either I get a lock (hard to avoid, not to be
> > > unlimited with ulimit -l unlimited) or a boot of xen and a blank screen
> > > with reboot after a 10 seconds with vga=current.
> >
> > Hello,
> Hello Roger,
>
> > Sorry for the delay on the reply. I'm afraid I will need more info in
> > order to help diagnose your issue.
> no matter, was occupied other ways; using VirtualBox at the moment, slow and
> the hypervisor tends to make long waiting-times...
>
> > How do you get the lock that you mention? Is it booting Xen, or just
> > plain FreeBSD?
> It's booting XEN, but not the CURRENT-kernel afterwards or it boots the
> CURRENT but not xen. I have in /boot/loader.conf
>
> xen_kernel="/boot/xen"
> xen_cmdline="dom0_mem=4048M dom0_max_vcpus=4 dom0=pvh com1=115200,8n1
> guest_loglvl=all loglvl=all"
> iommu="force,no-intremap"

The iommu=... option should either be inside the xen_cmdline or
removed, it's not an option of loader.conf.

> if_tap_load="YES"
> console=vga,com1

console= should also be inside the xen_cmdline= option, not here as a
separate line here.

Your xen_cmdline should be:

xen_cmdline="dom0_mem=4048M dom0_max_vcpus=4 dom0=pvh com1=115200,8n1
guest_loglvl=all loglvl=all console=vga,com1 noreboot"

Note it's all a single line.

> boot_multicons="YES"
> boot_serial="YES"
> console="comconsole,vidconsole"
> vmm_load="YES"
> vga=current
>
> and in /etc/rc.conf:
>
> xencommons_enable="YES"
> cloned_interfaces="bridge0"
> ifconfig_bridge0="addm bge0 SYNCDHCP"
> ifconfig_bge0="up"
>
> sudo ulimit -l
> unlimited
>
> sudo uname -a
> FreeBSD freeBSD-CURRENT 13.0-CURRENT FreeBSD 13.0-CURRENT #5 r368997: Tue Jan
> 12 13:46:33 CET 2021 lizbeth@freeBSD-CURRENT:/usr/obj/usr/src/amd64.amd64/
> sys/LIZBETH  amd64
>
> it's a no-debug kernel!
>
> /etc/ttys
> xc0 "/usr/libexec/getty Pc" xterm   on  secure
>
>
> when booting without XEN I get the following with 'xl dmesg'
>
> xencall: error: Could not obtain handle on privileged command interface /dev/
> xen/privcmd: No such file or directory
> libxl: error: libxl.c:102:libxl_ctx_alloc: cannot open libxc handle: No such
> file or directory
> cannot init xl context

Right, without booting as Xen dom0 /dev/xen/privcmd is simply not there.

> a normal boot without xen is on pastebin (VERBOSE_SYSINIT=2)
>
> https://pastebin.com/RN1SYXdW
>
>
> > Do you have a serial console attached to the box so that we could
> > see the log of Xen and FreeBSD booting?
> how to? every time I boot into xen I have to zpool import -f and mount it from
> rescue CD. Is there a log from the last boot somewhere? I duckduckgo'd a bit
> but couldn't find anything else than the dmesg. As it is not a kernel crash
> there's nothing in /var/crash and on /var/run I have the running boot inside.

You would need to plug a null-modem into the serial port of your box,
or alternatively use something like SoL (Serial over LAN) if your box
supports it.

> Maybe you tell me how to show the log from the serial console; btw, I tried
> both ways, only serial, only bios or both --- no matter, xen's booting,
> screen's blanking for the "normal" kernel but nothing happens but a sudden
> reboot. Is it on the /var/* somewhere when booting from resucue?

There won't be any logs from Xen in the disk, because Xen has no
access to the disk at all.

When booting with Xen, do you see any error message on the screen?

I've added noreboot to the Xen command line above so that you can
maybe take a photo of the crash message if there's any?

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: either xen or custom

[Roger Pau Monné via freebsd-xen]

On Sun, Dec 13, 2020 at 12:11:40AM +0100, lizbethmutterh...@gmail.com wrote:
> to make a long story short: either I get a lock (hard to avoid, not to be 
> unlimited with ulimit -l unlimited) or a boot of xen and a blank screen with 
> reboot after a 10 seconds with vga=current. 

Hello,

Sorry for the delay on the reply. I'm afraid I will need more info in
order to help diagnose your issue.

How do you get the lock that you mention? Is it booting Xen, or just
plain FreeBSD?

Do you have a serial console attached to the box so that we could
see the log of Xen and FreeBSD booting?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Failed to load Xen, kernel '/boot/xen'

[Roger Pau Monné via freebsd-xen]

On Tue, Jan 12, 2021 at 11:12:15AM -0800, Kevin G. Eliuk wrote:

It's kind of hard to tell what's going on without you providing almost
any info, but are you maybe booting using UEFI?

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: ZFS corruption using zvols as backingstore for hvm VM's

[Roger Pau Monné via freebsd-xen]

On Wed, Nov 18, 2020 at 11:58:27AM -0800, Brian Buhrow wrote:
>   hello Roger.  Sorry for the confusion.  I've been running NetBSD, or
> trying to, in both PV and HVM modes.  Here is a mesage I sent to the
> port-...@netbsd.org mailing list a few days ago, detailing what I found
> with running NetBSD in any mode under Xen as a domu.  Unfortunately, I
> don't think I can provide a trace  of a crash to the xen server when the

OK, I'm not really able to reproduce this myself, so if you get those
crashes again can you please make sure you have a serial attached to
the box so that you can provide some trace?

> HVM host misbehaves because I've had to move on in my work to get things
> working and I found a working combination of  NetBSD and ZFS that let's me
> proceed.  I've seen some discussion in the Xen documentation that suggests
> one can run a Xen dom0 in a domu guest, allowing for sand box testing of
> what you're describing without having to devote hardware to the issue.  Am
> I correct in this and where might I find instructions on how to do it?

You can test a nested PV dom0, but not a nested PVH dom0, which is
what FreeBSD uses to run as dom0.

> 
> In any case, here's what I found.

Thanks, I'm sure this will e helpful to others.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: ZFS corruption using zvols as backingstore for hvm VM's

[Roger Pau Monné via freebsd-xen]

On Thu, Nov 12, 2020 at 09:18:59PM -0800, Brian Buhrow wrote:
>   hello Roger.  thanks for engaging with me on this issue.  I think
> I've made progress on the issue and have a better handle on what's going
> wrong.  There seem to be a cascade of bugs here, which I'll try to
> enumerate.  
> 
> 1.  The disk corruption issue seems to be a bug in qemu whereby the
> emulated IDE disk controller issues partial writes instead of full writes
> or no writes with appropriate failure to the disk.  The IDE driver in
> NetBSD-5.2 doesn't play well with this behavior, in fact, NetBSD until May
> of 2020, doesn't play well with this behavior
> See: 
> http://mail-index.NetBSD.org/source-changes/2020/05/24/msg117668.html

Oh great, so it's something specific to NetBSD. This means you are
running NetBSD in HVM mode?

> 2.  This causes memory corruption in the OS itself, which can trigger a xen
> server crash!  (In my view, no matter how badly behaved the guest OS is, it
> shouldn't be able to bring down the xen server.)

Right, we really need the trace from this crash. Do you have a serial
hooked up to the box so that you can provide the panic message?

> 3.  Running NetBSD-5.2/i386 as a domu, which works flawlessly under xen3, 
> gets a 
> panic: HYPERVISOR_mmu_update failed

This would imply that you are running NetBSD in PV mode, in which case
it won't be using the emulated hard disk drive, and hence the commit
you referenced above would be unrelated. Can you assert whether you
are running NetBSD in PV or HVM mode?

>   I suspect this can be worked around using some of the command line
> options under Xen, xpti=true,domu=false, perhaps?  
> Are there others I should consider?

Maybe. I think you should report this to xen-devel and NetBSD/Xen
mailing lists, with a full trace of the crash and the guest config
file.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: ZFS corruption using zvols as backingstore for hvm VM's

[Roger Pau Monné via freebsd-xen]

On Wed, Nov 11, 2020 at 01:13:18AM -0800, Brian Buhrow wrote:
>   hello.  Following up on my own message, I believe I've run into a
> serious problem that exists on FreeBSD-xen with FreeBSD-12.1P10 and
> Xen-4.14.0.  Just in case I was running into an old bug with yesterday's
> post, I updated to xen-4.14.0 and Qemu-5.0.0.  the problem was still there,
> i.e. when writing to a second virtual hard drive on an hvm domu, the drive
> becomes corrupted.  Again, zpool scrub shows no errors.

Are you using volmode=dev when creating the zvol?

# zfs create -V16G -o volmode=dev zroot/foo

This is require when using zvol with bhyve, but shouldn't' be required
for Xen since the lock the guest disks from the kernel so GEOM cannot
taste them.

>   So, I decided it might be some sort of memory error.  I wrote a memory
> test program, shown below, and ran it on my hvm domu.  It not only
> crashed the domu itself, it crashed the entire xen server!  There are some
> dmesg messages that happened before the xen server crash, shown below, which
> suggest a serious problem.  In my view, no matter how badly the domu hvm
> host behaves, it shouldn't be able to crash the xen server itself!  The
> domu is running NetBSD-5.2, an admittedly old version of the operating
> system, but I'm running a fleet of these  machines, both on real hardware
> and on older versions of xen with no stability issues whatsoever!  And, as
> I say, I shouldn't be able to wipe out the xen server from an hvm domu, no
> matter what I do!

Can you please paste the config file of the domain?

> 
>   The memory test program takes one argument, the amount of RAM, 
> in
> megabytes, you want it to test.  It then allocates that memory, and
> sequentially walks through that memory over and over again, writing to it
> and reading from it, checking to make sure the data read matches the data
> written.  this has the effect of causing the resident set size of the
> program to grow slowly over time, as it works.  It was originally written
> to test the paging efficiency of a system, but I modified it to actually
> test the memory along the way.
>   to reproduce the issue, perform the following steps:
> 
> 1.  Set up an hvm host, I think FreeBSD as a domu hvm host will work fine.
> Use zfs zvols as the backingstore for the virtual disk(s) for your host.
> 
> 2.  Compile this program for that host and run it as follows:
> ./testmem 1000
> This should ask the program to allocate 1G of memory and then walk through
> and test it.  It will report each megabyte of memory it's written and
> tested.  My test hvm had 4G of RAM as it was a 32-bit  OS running on the
> domu.  Nothing else was running on either the xen server or the domu host.
> I'm not sure exactly how far the program got in its memory walk before
> things went south, but I think it touched about 100 megabytes of its 1000
> megabyte allocation.
> My program was not running as root, so it had no special privileges, even
> on the domu host.  
> 
>   I'm not sure if the problem is with qemu, xen, or some  combination of
> the two.  
> 
>   It would be great if someone could reproduce this issue and maybe shed
> a bit more light on what's going on.
> 
> -thanks
> -Brian
> 
> 
> 
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_ring2pkt:1534): Unknown extra 
> info type 255.  Discarding packet
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:304): 
> netif_tx_request index =0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:305): 
> netif_tx_request.gref  =0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:306): 
> netif_tx_request.offset=0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:307): 
> netif_tx_request.flags =8
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:308): 
> netif_tx_request.id=69
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:309): 
> netif_tx_request.size  =1000
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:304): 
> netif_tx_request index =1
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:305): 
> netif_tx_request.gref  =255
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:306): 
> netif_tx_request.offset=0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:307): 
> netif_tx_request.flags =0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:308): 
> netif_tx_request.id=0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:309): 
> netif_tx_request.size  =0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_rxpkt2rsp:2068): Got error -1 
> for hypervisor gnttab_copy status
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_ring2pkt:1534): Unknown extra 
> info type 255.  Discarding packet
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:304): 
> netif_tx_request index =0
> Nov 11 00:28:54 xen-lothlorien kernel: xnb(xnb_dump_txreq:305): 
> netif_tx_request.gref  =0
> Nov 11 00:28:54 xen-lothlorien kernel: 

Re: UEFI and FreeBSD/Xen-12.1/4.12

[Roger Pau Monné via freebsd-xen]

On Sun, Nov 08, 2020 at 11:46:34PM -0800, Brian Buhrow wrote:
>   hello.  I'm trying to get an installation of FreeBSD-12.1 / xen 4.12
> running an Intel DQ67SW motherboard with 4 8TB disks attached to it up and
> running.   Everything seems to work fine, except that the BIOS will not
> boot from the hard drives in legacy mode on its own.  If I manually select
> a disk from the boot menu, FreeBSD and Xen load  as they should.  The Intel
> site says that booting from disks larger than 2TB is only supported if UEFI
> booting is enabled.  It's my understanding that Freebsd-Xen will not run
> if one doesn't boot in legacy mode.
>   Any suggestions on how to work around this conflict?  Is it now
> possible to boot FreeBSD as Dom0 in UEFI boot mode?

In order to boot FreeBSD/Xen dom0 using UEFI we need to add support
for the multiboot2 [0] boot protocol, so that the FreeBSD loader can do
the handoff to the Xen kernel. We currently only have multiboot1
implemented in the FreeBSD loader, and hence we are limited to legacy
(BIOS) boot.

I've been meaning to fix this and add support for multiboot2, but so
far I haven't got the time to actually do the work. If someone is
interested in working on this I'm happy to help, but it does require
someone that's slightly familiar with the loader.

Roger.

[0] https://www.gnu.org/software/grub/manual/multiboot2/multiboot.html
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: The vnclisten parameter is ignored with hvm domains under xen-4.12.1 and FreeBSD-12.0

[Roger Pau Monné via freebsd-xen]

On Fri, Aug 28, 2020 at 02:11:07PM -0700, Brian Buhrow wrote:
>   hello Roger.  thanks for the quick reply.  I think, after a careful
> read of the xl.cfg man page plus a careful reading of the xen-tools source
> code, I've figured out the issue.  I'm sending the solution here so folks
> will have it in the future.
> 
> If one is configuring an hvm guest, any vfb specifications  in the domain
> configuration are ignored.

Let me clarify this a bit. vfb is a para-virtualized frame buffer, so
it needs a specific Xen vfb driver in the guest for it to be able to
use the device (ie: it's not an emulated graphics card).

> Instead, the parameters that normally get placed
> in a vfb stanza must be placed as top level configuration items.  For
> example:
> 
> For a PV guest:
> vfb = [ 'vnc=1,vnclisten=10.14.200.200' ]
> 
> Translates to, for an hvm guest:
> 
> vnc = 1
> vnclisten = "10.14.200.200"

OTOH, the global vnc parameter is indeed exclusive to HVM guests and
is related to the emulated graphics card device.

You could have a HVM guest making use of both the emulated graphics
card (the global vnc parameter) and nmultiple para-virtualized vfb
framebuffers, but those would be different outputs (like having more
than one graphics card on a physical box).

You cannot however make use of the global vnc parameter for PV guests,
as they have no emulated graphics card.

> 
> The xl.cfg man page doesn't say that in order for the vnc parameters to be
> picked up for hvm guests with emulated graphics cards, those parameters
> need to be specified as top level parameters in the config file.  It took
> reading the source code of the xen-tools xl code to figure that out.

Sorry, I understand this is quite confusing. Do you think the man page
could be modified to make this easier to understand and create less
confusion?

I'm quite sure the community would be happy to take a patch in order
to clarify stuff.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: The vnclisten parameter is ignored with hvm domains under xen-4.12.1 and FreeBSD-12.0

[Roger Pau Monné via freebsd-xen]

On Fri, Aug 28, 2020 at 08:47:41AM -0700, Brian Buhrow wrote:
>   hello.  I'm trying to configure an hvm domain with a vnc listener  on
> one of the dom0's network addresses, rather than 127.0.0.1.  
> 
> If I use a line like: 
> vfb = [ 'type=vnc,vnc=1,vnclisten=10.14.200.200' ]

type=vnc doesn't seem to be a known option according to the xl.cfg man
page [0].

> 
> in the config file, I still get a vnc listener on 127.0.0.1, rather than
> 10.14.200.200.

Do you also have a VNC sever without using vfb enabled? (ie: the
global xl.cfg vnc option). It might be useful if you post the full
xl.cfg maybe.

> 
>   A google search suggests this is somewhat of a known problem, but I
> don't see a patch in the upstream version of xen-tools that addresses it.
> Is there an easy fix for this issue, or perhaps I need to do what I want in
> a different way?

I think you will likely get more information by posting to xen-devel
directly, as this doesn't seem related to FreeBSD.

Roger.

[0] https://xenbits.xen.org/docs/4.12-testing/man/xl.cfg.5.html
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Slow networking speeds with Freebsd 12.0 and Xen-4.12.1, freebsd-12.0 as dom0

[Roger Pau Monné]

On Fri, Feb 14, 2020 at 10:15:48AM -0800, Brian Buhrow wrote:
>   hello.  Ok.  Perhaps I spoke too soon!  the domu I was testing with,
> which is an entirely pv guest, performs fine with FreeBSD.  However, if I
> take an hvm guest and run that using Intel E1000 emulation, my performance
> drops back to 70 mbits/sec or so of throughput.  For HVM guests, what kind
> of network performance should I expect?

That's hard to tell. My recommendation would be to try to install PV
drivers to HVM guests if possible in order to get better performance.
Performance of emulated nics is always going to be worse.

Is the QEMU process in dom0 consuming lots of CPU when using the
emulated nic?

> I've been running pv guests for
> years, but don't have much experience with hvm guests.  
> -thanks

Guests using PV devices (that applies to HVM/PVH guests also) should
always get better IO performance than those using emulated devices.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Slow networking speeds with Freebsd 12.0 and Xen-4.12.1, freebsd-12.0 as dom0

[Roger Pau Monné]

On Thu, Feb 13, 2020 at 06:58:53PM -0800, Brian Buhrow wrote:
>   Hello.  I'm noticing that network performance on guest VN's whether
> they're pv or hvm is very slow when using FreeBSD-12.0 as dom0 as compared
> with older versions of Xen and NetBSD as dom0.  For example, 
> I have the following two comparisons, both with the same domu running:
> 
> FreeBSD-12.0/Xen 4.12.1, 60 mbits/sec from the vm in question.

So I've done a small test and using a FreeBSD dom0 with a FreeBSD domU
in HVM mode and I get:

# iperf -c 192.168.1.150

Client connecting to 192.168.1.150, TCP port 5001
TCP window size:  113 KByte (default)

[  3] local 192.168.1.101 port 12224 connected with 192.168.1.150 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec   302 MBytes   253 Mbits/sec

This is using my macbook as an iperf server with a USB-ethernet
adapter (so not professional grade hardware at all).

I'm quite sure speed could be better, we certainly have quite a lot of
room for improvement regarding netfront/netback on FreeBSD, as no one
looked at it for quite some time.

Which OS are you using on the domU?

Can you paste the output of ifconfig from the domU? (I'm mostly
interested in knowing which features the domU is attempting to use)

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: FreeBSD 12.1 hypervisor management tools

[Roger Pau Monné]

On Tue, Jan 14, 2020 at 12:47:54AM +0200, Stefan Parvu wrote:
> >> 
> >> libvirtd, virt-manager: no go.
> >> 
> >> * I had to rebuild libvirtd with Xen support
> >> 
> >> * got occasional core dumps from libvirt
> > 
> > Can you report those? (ie: send the dumps here so they can be analyzed
> > and either fixed or forwarded to the appropriate upstream project)
> 
> 
> sure. I should put a list of defects. I need to repeat these. 
> 
> I was in hurry to check on the overall state of the project. But Im curious: 
> has 
> anyone at least tried, meaning really to use virt-manager for Xen on FreeBSD ?

I don't use libvirt at all, so I never tried it myself. I know it
works on Linux and is tested as part of the Xen CI loop, see the
-libvirt jobs in any Xen CI loop output:

http://logs.test-lab.xenproject.org/osstest/logs/146050/

So whatever is broken it's likely to be FreeBSD specific and will
require someone from the FreeBSD community to diagnose and propose
fixes.

The first step should be to report those errors, and figure out
whether someone has interest in getting libvirt working on FreeBSD/Xen
and maintaining it in a sane state. If there's no interest we should
drop the Xen option from the libvirt port to not give users false
expectations.

Thanks, Roger
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen performance on FreeBSD vs Linux

[Roger Pau Monné]

On Mon, Jan 13, 2020 at 02:51:16PM +0100, Juergen Gotteswinter wrote:
> Am So., 12. Jan. 2020 um 20:07 Uhr schrieb Stefan Parvu <
> spa...@kronometrix.org>:
> 
> > >
> > > Also a FreeBSD dom0 can only work in PVH mode, which is faster for
> > > certain operations like page table modifications, but it's slower for
> > > others, like issuing hypercalls, when compared to a PV dom0.
> > >
> > > I don't have figures at hand now, but guest creation is likely slower
> > > on a FreeBSD PVH dom0 than on a Linux PV dom0, and that's because
> > > hypercalls are more expensive on PVH than on PV (has nothing to do
> > > whether Linux or FreeBSD is used).
> >
> > right. thanks a lot for pointers. A bit confusing is that I see mentioned
> > around that Xen of FreeBSD is experimental. Is this true ?
> >
> > I will have 2 servers to test Xen and Bhyve on FreeBSD 12.1 and check how
> > well the hypervisors work for different guest from Linux, BSD and Windows.
> >
> > Im more interested on bhyve vs Xen on FreeBSd than Linux. We will keep
> > FreeBSD as our main virtualization platform.
> >
> > Stefan
> 
> 
> FreeBSD as Dom0 feels (at least ~6 month ago) really somehow experimental
> compared to my linux experience.

Hello,

Can you give us more details regarding this comment, and what did you
feel was missing or could be improved compared to Linux?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen performance on FreeBSD vs Linux

[Roger Pau Monné]

On Sun, Jan 12, 2020 at 09:07:32PM +0200, Stefan Parvu wrote:
> > 
> > Also a FreeBSD dom0 can only work in PVH mode, which is faster for
> > certain operations like page table modifications, but it's slower for
> > others, like issuing hypercalls, when compared to a PV dom0.
> > 
> > I don't have figures at hand now, but guest creation is likely slower
> > on a FreeBSD PVH dom0 than on a Linux PV dom0, and that's because
> > hypercalls are more expensive on PVH than on PV (has nothing to do
> > whether Linux or FreeBSD is used).
> 
> right. thanks a lot for pointers. A bit confusing is that I see mentioned 
> around that Xen of FreeBSD is experimental. Is this true ? 

PVH dom0 support in Xen is experimental, the FreeBSD PVH bits are
mostly stable and I wouldn't expect many changes there except from
bug fixes. The PVH guest ABI is also stable since quite some time
ago.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: FreeBSD 12.1 hypervisor management tools

[Roger Pau Monné]

On Sun, Jan 12, 2020 at 09:12:01PM +0200, Stefan Parvu wrote:
> > 
> > Okay, so there are some options. Will have a look on virt-manager and cbsd. 
> 
> 
> libvirtd, virt-manager: no go.
> 
> * I had to rebuild libvirtd with Xen support
> 
> * got occasional core dumps from libvirt

Can you report those? (ie: send the dumps here so they can be analyzed
and either fixed or forwarded to the appropriate upstream project)

Without us getting detailed bug reports it's very likely that this is
not going to improve.

> 
> * virt-manager is useless, cannot be used to create a simple guests from 
> debian to centos or freebsd
>
> * got all sorts of errors from virt-manager, which does not let me create or 
> boot the guest after configuration  

This should also be properly reported, what did you try, what did you
expect and what did happen instead.

Again, without getting reports it's not likely that this is going to
improve. I'm adding the FreeBSD libvirt maintainer so he is aware.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen performance on FreeBSD vs Linux

[Roger Pau Monné]

On Mon, Jan 06, 2020 at 12:43:51PM +0200, Stefan Parvu wrote:
> Hi,
> 
> How Xen hypervisor compares in terms of performance (guest) on FreeBSD vs 
> Linux
> , using ZFS or UFS ?

There are several factors to take into account here.

On FreeBSD the hypervisor is built with clang, while on Linux it's
usually built with gcc, and hence different optimizations will be
used.

Also a FreeBSD dom0 can only work in PVH mode, which is faster for
certain operations like page table modifications, but it's slower for
others, like issuing hypercalls, when compared to a PV dom0.

I don't have figures at hand now, but guest creation is likely slower
on a FreeBSD PVH dom0 than on a Linux PV dom0, and that's because
hypercalls are more expensive on PVH than on PV (has nothing to do
whether Linux or FreeBSD is used).

> Are there any benchmarks or tests regarding this ?

I don't think so, but I think they will be useful if someone has the
time to perform them :).

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: xen-kernel 4.12.1 unsupported amd family !? FreeBSD

[Roger Pau Monné]

On Fri, Dec 06, 2019 at 09:33:44AM +0100, davidheinrichplanner wrote:
> Hi Royger,
> 
> i want to inform you by freshports page but i forgett my pw and reset doesnt
> work :(.
> 
> first one i don't know which one must be informed about this problem (you or
> direct xen project, xen team by freebsd?). I have a amd epyc system with two
> cpus. And xen tell me its a unsupported amd family. How can be fix this
> problem?

I'm not sure I understand the problem. Does Xen refuse to boot on such
system?

Can you please paste the Xen boot log showing the issue?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen (HVM) and NMI

[Roger Pau Monné]

On Fri, Nov 08, 2019 at 03:15:40PM +0200, Andriy Gapon wrote:
> On 08/11/2019 11:57, Roger Pau Monné wrote:
> > On Fri, Nov 08, 2019 at 10:19:01AM +0200, Andriy Gapon wrote:
> >>> I found this in Linux code:
> >>> HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL);
> >>> It's in xen_send_IPI_one().
> >>> I wonder if that's that or if there is more to this than meets the eye.
> > 
> > Yes, something along this lines should work, we could even use the
> > native NMI signaling using the local APIC, but the hypercall shortcut
> > should be faster in most cases.
> > 
> >> I also found this in an old post.
> >> Ian Campbell wrote:
> >>> You need to register a callback with CALLBACKOP_register
> >>> CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive
> >>> that callback. IIRC you also need to arrange that returning from an NMI
> >>> is always done with HYPERVISOR_iret and not optimised to a direct iret
> >>> as it can be otherwise. This is to allow the hypervisor to implement NMI
> >>> masking correctly.
> > 
> > That's AFAIK for PV guests which use a completely different mechanism
> > in order to receive interrupts. None of this is needed for FreeBSD
> > because there's no classic PV support.
> 
> Perhaps HYPERVISOR_iret is still needed even for us?

No, that's just for classic PV.

> This is said with zero knowledge of the topic.
> 
> > Can you try the patch below?
> 
> I tried it and, unfortunately, it was not a success.  And I don't have much
> diagnostic.

My bad, that's what happens when you send untested patches, the
previous patch was missing an apic_cpuid and was hitting a page-fault
in the kernel. Patch below should work fine.

Roger.
---8<---
diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c
index 7d254ef3f734..8bf2158dbec2 100644
--- a/sys/x86/xen/xen_apic.c
+++ b/sys/x86/xen/xen_apic.c
@@ -72,7 +72,6 @@ static driver_filter_t xen_invlcache;
 static driver_filter_t xen_ipi_bitmap_handler;
 static driver_filter_t xen_cpustop_handler;
 static driver_filter_t xen_cpususpend_handler;
-static driver_filter_t xen_cpustophard_handler;
 #endif
 
 /*-- Macros 
--*/
@@ -96,7 +95,6 @@ static struct xen_ipi_handler xen_ipis[] =
[IPI_TO_IDX(IPI_BITMAP_VECTOR)] = { xen_ipi_bitmap_handler, "b"   },
[IPI_TO_IDX(IPI_STOP)]  = { xen_cpustop_handler,"st"  },
[IPI_TO_IDX(IPI_SUSPEND)]   = { xen_cpususpend_handler, "sp"  },
-   [IPI_TO_IDX(IPI_STOP_HARD)] = { xen_cpustophard_handler,"sth" },
 };
 #endif
 
@@ -259,12 +257,52 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest)
XEN_APIC_UNSUPPORTED;
 }
 
+#define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field)
+static void
+send_nmi(int dest)
+{
+   unsigned int cpu;
+
+   /*
+* NMIs are not routed over event channels, and instead delivered as on
+* native using the exception vector (#2). Triggering them can be done
+* using the local APIC, or an hypercall as a shortcut like it's done
+* below.
+*/
+   switch(dest) {
+   case APIC_IPI_DEST_SELF:
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL);
+   break;
+   case APIC_IPI_DEST_ALL:
+   CPU_FOREACH(cpu)
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+   PCPU_ID_GET(cpu, vcpu_id), NULL);
+   break;
+   case APIC_IPI_DEST_OTHERS:
+   CPU_FOREACH(cpu)
+   if (cpu != PCPU_GET(cpuid))
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+   PCPU_ID_GET(cpu, vcpu_id), NULL);
+   break;
+   default:
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+   PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL);
+   break;
+   }
+}
+#undef PCPU_ID_GET
+
 static void
 xen_pv_lapic_ipi_vectored(u_int vector, int dest)
 {
xen_intr_handle_t *ipi_handle;
int ipi_idx, to_cpu, self;
 
+   if (vector >= IPI_NMI_FIRST) {
+   send_nmi(dest);
+   return;
+   }
+
ipi_idx = IPI_TO_IDX(vector);
if (ipi_idx >= nitems(xen_ipis))
panic("IPI out of range");
@@ -522,14 +560,6 @@ xen_cpususpend_handler(void *arg)
return (FILTER_HANDLED);
 }
 
-static int
-xen_cpustophard_handler(void *arg)
-{
-
-   ipi_nmi_handler();
-   return (FILTER_HANDLED);
-}
-
 /*- XEN PV IPI setup 
-*/
 /*
  * Those functions are provided outside of the Xen PV APIC implementation
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen (HVM) and NMI

[Roger Pau Monné]

On Fri, Nov 08, 2019 at 10:19:01AM +0200, Andriy Gapon wrote:
> On 08/11/2019 10:03, Andriy Gapon wrote:
> > On 07/11/2019 20:08, Andriy Gapon wrote:
> >> For CPUs that do get interrupted I see stack traces like:
> >> cpustop_handler+0x28 ipi_nmi_handler+0x44 xen_cpustophard_handler+0x9
> >> intr_event_handle+0x8b intr_execute_handlers+0x58 
> >> xen_intr_handle_upcall+0x15a
> >> xen_intr_upcall_u+0x96 ...
> >> So, it looks like the NMI is delivered by the same mechanism as normal
> >> interrupts.  If a processor has interrupts disabled then the NMI would not 
> >> get
> >> delivered?

Yes, sorry, I certainly didn't code this correctly regarding NMI
handling.

> >> Is there anything we could do to improve this?
> > 
> > I found this in Linux code:
> > HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL);
> > It's in xen_send_IPI_one().
> > I wonder if that's that or if there is more to this than meets the eye.

Yes, something along this lines should work, we could even use the
native NMI signaling using the local APIC, but the hypercall shortcut
should be faster in most cases.

> I also found this in an old post.
> Ian Campbell wrote:
> > You need to register a callback with CALLBACKOP_register
> > CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive
> > that callback. IIRC you also need to arrange that returning from an NMI
> > is always done with HYPERVISOR_iret and not optimised to a direct iret
> > as it can be otherwise. This is to allow the hypervisor to implement NMI
> > masking correctly.

That's AFAIK for PV guests which use a completely different mechanism
in order to receive interrupts. None of this is needed for FreeBSD
because there's no classic PV support.

Can you try the patch below?

Roger.
---8<---
diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c
index 7d254ef3f734..7a6964d60cdb 100644
--- a/sys/x86/xen/xen_apic.c
+++ b/sys/x86/xen/xen_apic.c
@@ -72,7 +72,6 @@ static driver_filter_t xen_invlcache;
 static driver_filter_t xen_ipi_bitmap_handler;
 static driver_filter_t xen_cpustop_handler;
 static driver_filter_t xen_cpususpend_handler;
-static driver_filter_t xen_cpustophard_handler;
 #endif
 
 /*-- Macros 
--*/
@@ -96,7 +95,6 @@ static struct xen_ipi_handler xen_ipis[] =
[IPI_TO_IDX(IPI_BITMAP_VECTOR)] = { xen_ipi_bitmap_handler, "b"   },
[IPI_TO_IDX(IPI_STOP)]  = { xen_cpustop_handler,"st"  },
[IPI_TO_IDX(IPI_SUSPEND)]   = { xen_cpususpend_handler, "sp"  },
-   [IPI_TO_IDX(IPI_STOP_HARD)] = { xen_cpustophard_handler,"sth" },
 };
 #endif
 
@@ -259,12 +257,52 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest)
XEN_APIC_UNSUPPORTED;
 }
 
+#define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field)
+static void
+send_nmi(int dest)
+{
+   unsigned int cpu;
+
+   /*
+* NMIs are not routed over event channels, and instead delivered as on
+* native using the exception vector (#2). Triggering them can be done
+* using the local APIC, or an hypercall as a shortcut like it's done
+* below.
+*/
+   switch(dest) {
+   case APIC_IPI_DEST_SELF:
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL);
+   break;
+   case APIC_IPI_DEST_ALL:
+   CPU_FOREACH(cpu)
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+   PCPU_ID_GET(cpu, vcpu_id), NULL);
+   break;
+   case APIC_IPI_DEST_OTHERS:
+   CPU_FOREACH(cpu)
+   if (cpu != PCPU_GET(cpuid))
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+   PCPU_ID_GET(cpu, vcpu_id), NULL);
+   break;
+   default:
+   HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(dest, vcpu_id),
+   NULL);
+   break;
+   }
+}
+#undef PCPU_ID_GET
+
 static void
 xen_pv_lapic_ipi_vectored(u_int vector, int dest)
 {
xen_intr_handle_t *ipi_handle;
int ipi_idx, to_cpu, self;
 
+   if (vector >= IPI_NMI_FIRST) {
+   send_nmi(dest);
+   return;
+   }
+
ipi_idx = IPI_TO_IDX(vector);
if (ipi_idx >= nitems(xen_ipis))
panic("IPI out of range");
@@ -522,14 +560,6 @@ xen_cpususpend_handler(void *arg)
return (FILTER_HANDLED);
 }
 
-static int
-xen_cpustophard_handler(void *arg)
-{
-
-   ipi_nmi_handler();
-   return (FILTER_HANDLED);
-}
-
 /*- XEN PV IPI setup 
-*/
 /*
  * Those functions are provided outside of the Xen PV APIC implementation

___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 07:10:44PM +0300, Stefan Parvu wrote:
> > 
> >> Also, IIRC X58 had some interrupt remapping issues, can you try to
> >> boot with iommu=no-intremap? (without any other iommu options)
> > 
> 
> here my /boot/loader.conf
> 
> root@earth:~ # cat /boot/loader.conf 
> kern.geom.label.disk_ident.enable="0"
> kern.geom.label.gptid.enable="0"
> zfs_load="YES"
> if_tap_load="YES"
> xen_kernel="/boot/xen"
> xen_cmdline="dom0_mem=8192M dom0_max_vcpus=4 dom0=pvh console=com1,vga 
> com1=115200,8n1 guest_loglvl=all loglvl=all iommu=no-intremap"
> 
> 
> Nope, same problem 
> 
> and this is what I get now:
> http://www.kronometrix.org/bugs/freebsd_xen2.jpg 
> 

Hm, can you remove iommu=no-intremap and try to capture a video of Xen
booting? I'm hoping I might be able to spot why Xen is refusing to use
the iommu.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 06:59:46PM +0300, Stefan Parvu wrote:
> > Also, IIRC X58 had some interrupt remapping issues, can you try to
> > boot with iommu=no-intremap? (without any other iommu options)
> 
> Let me try quickly that. No serial it is too complicated for this machine. 
> 
> If we can make this running do you think FreeBSD Xen Dom0 will ever support 
> HVM
> linux guests ?

FreeBSD Xen dom0 already supports Linux HVM guests, just as it
supports Windows HVM guests or any kind of HVM guest in general (ie:
whatever guests you can run on a Linux dom0 should also work on a
FreeBSD dom0).

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 06:15:33PM +0300, Stefan Parvu wrote:
> > # acpidump -t|grep DMAR
> >  DMAR: Length=180, Revision=1, Checksum=195,
> > OEMID=A M I, OEM Table ID=OEMDMAR, OEM Revision=0x1,
> > 
> 
> root@earth:~ # acpidump -t | grep DMAR
>   DMAR: Length=288, Revision=1, Checksum=93,
>   OEMID=AMI, OEM Table ID=OEMDMAR, OEM Revision=0x1,

So there's an iommu, then it must be Xen refusing to use it.

Is it possible for you to setup a serial console in order to get the
full Xen boot log?

Also, IIRC X58 had some interrupt remapping issues, can you try to
boot with iommu=no-intremap? (without any other iommu options)

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 02:07:55PM +0300, Konstantin Belousov wrote:
> On Wed, Oct 16, 2019 at 11:28:23AM +0200, Roger Pau Monné wrote:
> > On Wed, Oct 16, 2019 at 12:17:20PM +0300, Stefan Parvu wrote:
> > > > 
> > > > Your box doesn't seem to have an iommu (aka vt-d for Intel).
> > > 
> > > In BIOS I do have VT-D Enabled option. Would that be broken or not 
> > > working ?
> > 
> > Are you sure it's VT-d and not VT-x what you have enabled in the BIOS?
> > 
> > Looking at the details of your CPU:
> > 
> > https://ark.intel.com/content/www/us/en/ark/products/41447/intel-core-i7-930-processor-8m-cache-2-80-ghz-4-80-gt-s-intel-qpi.html
> > 
> > VT-d is not listed in 'Advanced Technologies'. For example looking at
> > a newer model:
> > 
> > https://ark.intel.com/content/www/us/en/ark/products/149091/intel-core-i7-8565u-processor-8m-cache-up-to-4-60-ghz.html
> > 
> > You can see VT-d listed in 'Advanced Technologies'.
> Nehalems do have VT-d, but they have dedicated chip with north bridge
> still, CPU only provided memory controller.  So VT-d is the feature of
> the chipset, e.g. X58, not mentioning Xeon chipsets.  On the other hand,
> VT-d support was relatively buggy, could it be that Xen kernel refuses
> to use it due to the problems ?

It's possible that Xen refuses to enable the iommu due to erratas, I'm
however unable to find the specification document of the X58 chipset
with the erratas, seems to be gone from the Intel site.

Stefan:

When booting plain FreeBSD, can you try the following?

# acpidump -t|grep DMAR
  DMAR: Length=180, Revision=1, Checksum=195,
OEMID=A M I, OEM Table ID=OEMDMAR, OEM Revision=0x1,

If you don't get output it means there's no DMAR ACPI table, and thus
no VT-d support (at least from the OS point of view).

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 01:09:11PM +0300, Stefan Parvu wrote:
> thank you guys for tips.
> 
> > Are you sure it's VT-d and not VT-x what you have enabled in the BIOS?
> 
> Yes, I think the ASUS BIOS had listed VT-d. But will double check tonight. 
> But it might be 
> that the CPU does not support VT-d. Thats very true. Hmmm. Is there any point 
> even to
> update the CPU on this system, would it even support some new CPU models ? 

I'm afraid I have no idea. I don't deal with hardware setup myself.

> > Linux dom0 would likely be in PV mode (note the missing H), which
> > doesn't require an iommu. OTOH FreeBSD only supports PVH dom0, which
> > does require an iommu.
> 
> Right. It could be tat I was using PV mode (paravirtualisation) mode, without 
> HVM
> extensions which PVH uses. Right ? Thats I cant say. It was some time back. 

So in order to provide an overview:

 - PV doesn't require VT-x nor VT-d.
 - PV guests with pci-passthrough devices require VT-d for safety
   reasons.
 - PVH/HVM require VT-x.
 - PVH dom0 or HVM/PVH guests with pci-passthrough devices require
   both VT-d and VT-x.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 12:17:20PM +0300, Stefan Parvu wrote:
> > 
> > Your box doesn't seem to have an iommu (aka vt-d for Intel).
> 
> In BIOS I do have VT-D Enabled option. Would that be broken or not working ?

Are you sure it's VT-d and not VT-x what you have enabled in the BIOS?

Looking at the details of your CPU:

https://ark.intel.com/content/www/us/en/ark/products/41447/intel-core-i7-930-processor-8m-cache-2-80-ghz-4-80-gt-s-intel-qpi.html

VT-d is not listed in 'Advanced Technologies'. For example looking at
a newer model:

https://ark.intel.com/content/www/us/en/ark/products/149091/intel-core-i7-8565u-processor-8m-cache-up-to-4-60-ghz.html

You can see VT-d listed in 'Advanced Technologies'.

> I think I have used Xen a bit older version on Debian some time back if Im 
> not wrong, on the same hdw.

See below.

> >> But you need all boot log. I will try your option tonight. I cant access 
> >> the hardware now.
> > 
> > If you can that would be helpful, but AFAIK this box is simply not
> > capable of running a Xen PVH dom0 (which is the only mode FreeBSD
> > supports in order to work as dom0). Either it has a broken iommu or no
> > iommu at all.
> 
> Hmmm. Thats strange since I do recall I had previously used here Xen but on 
> Linux. But older
> version. Could it be something changed regarding Xen and iommu or ?

Linux dom0 would likely be in PV mode (note the missing H), which
doesn't require an iommu. OTOH FreeBSD only supports PVH dom0, which
does require an iommu.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 12:07:50PM +0300, Stefan Parvu wrote:
> 
> > If your box doesn't have a serial header, you can add noreboot to
> > xen_cmdline and paste a photo of the screen with the panic message,
> > it's not great but at least we would get an idea of what's wrong.
> > 
> 
> Im not sure I can setup a serial console on this wkst. Will check.
> 
> The last thing I see is this:
> http://www.kronometrix.org/bugs/freebsd_xen.jpg 
>  but nothing like a thread 
> dump,
> stack trace etc … 

Your box doesn't seem to have an iommu (aka vt-d for Intel).

> But you need all boot log. I will try your option tonight. I cant access the 
> hardware now.

If you can that would be helpful, but AFAIK this box is simply not
capable of running a Xen PVH dom0 (which is the only mode FreeBSD
supports in order to work as dom0). Either it has a broken iommu or no
iommu at all.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 11:49:50AM +0300, Stefan Parvu wrote:
> > Can you please provide the full boot log? I'm afraid that I'm not able
> > to provide any suggestions without knowing what went wrong.
> 
> In BIOS everything regarding virtualization is turned ON.
> 
> How can I provide the full boot log if my machine is rebooting itself … 
> and have no access to the files? Im not able to control how can I boot again.
> I usually need to reinstall everything from scratch. 

Could you setup a serial console in order to get the boot log?

https://www.freebsd.org/doc/handbook/serialconsole-setup.html
https://www.freebsd.org/doc/handbook/virtualization-host-xen.html

If your box doesn't have a serial header, you can add noreboot to
xen_cmdline and paste a photo of the screen with the panic message,
it's not great but at least we would get an idea of what's wrong.

> Is there any way to configure the boot loader and boot with or without Xen 
> kernel ?

Yes, when you are at the bootloader screen press '3', and then type:

> unload
> unset xen_kernel
> boot

That would make your box boot into plain FreeBSD without Xen.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: cannot install FreeBSD 12.0 Xen 4.12.1 on Intel Core i7 CPU 930

[Roger Pau Monné]

On Wed, Oct 16, 2019 at 12:12:06AM +0300, Stefan Parvu wrote:
> Hi,
> 
> I have a bit of problem, cannot install FreeBSD/Xen on a bit older hardware. 
> The hardware 
> should support the minimal VT-D/IOMMU settings, required by Xen.
> 
> CPU: Intel(R) Core(TM) i7 CPU 930  @ 2.80GHz (2806.42-MHz K8-class 
> CPU)
>   Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
>   
> Features=0xbfebfbff
>   
> Features2=0x98e3bd
>   AMD Features=0x28100800
>   AMD Features2=0x1
>   VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
>   TSC: P-state invariant, performance statistics
> 
> I get always a panic on CPU0 followed by a reboot. My /boot/loader.conf was 
> set as:
> 
> xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0=pvh console=com1,vga 
> com1=115200,8n1 guest_loglvl=all loglvl=all iommu=debug,force”
> 
> Any ideas what I could try ?

Hello,

Can you please provide the full boot log? I'm afraid that I'm not able
to provide any suggestions without knowing what went wrong.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: i/o is very slow on FreeBSD dom0 with Xen-4.12 and Freebsd-12

[Roger Pau Monné]

On Tue, Oct 15, 2019 at 08:56:22AM -0700, Brian Buhrow wrote:
>   hello Roger.  Sorry, my bad.  I didn't even look at the cause of the
> panic.   Rebooting the system caused it to come up clean after another pass
> with fsck.  With the ioapic_ack=old argument, the system comes up right
> away and is quite responsive.
>   Can you explain how to select which argument to use on the command
> line if the autoselection code doesn't work?

Hm, it's not so much that the autodetection is wrong, AFAIK there's
some hardware issue with specific models that make the new ack method
not work properly. This is limited to very specific (and older)
models. Note that you might still see interrupt timeouts with the old
ack method.

I will try to figure out if there's any hardware errata that could be
related to this issue, and try to find a box to reproduce (this might
not be feasible since I'm not sure I have anything similar to your
hardware available).

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: i/o is very slow on FreeBSD dom0 with Xen-4.12 and Freebsd-12

[Roger Pau Monné]

On Mon, Oct 14, 2019 at 02:45:04PM -0700, Brian Buhrow wrote:
>   hello.  Using: ioapic_ack=old causes the dom0 to crash.  Below are the
> logs.

Does the crash happen every time you boot with ioapic_ack=old?

[...]
> dev = gpt/gptroot, block = 1, fs = /
> panic: ffs_blkfree_cg: freeing free block
> cpuid = 2
> time = 1571088217
> KDB: stack backtrace:
> #0 0x80be78d7 at kdb_backtrace+0x67
> #1 0x80b9b4b3 at vpanic+0x1a3
> #2 0x80b9b303 at panic+0x43
> #3 0x80e87345 at ffs_blkrelease_finish+0x6e5
> #4 0x80e8433d at ffs_blkfree+0xad
> #5 0x80eb10af at softdep_get_depcounts+0x48bf
> #6 0x80eb0fe5 at softdep_get_depcounts+0x47f5
> #7 0x80ea195a at softdep_update_inodeblock+0x178a
> #8 0x80eab94a at softdep_request_cleanup+0xa8a
> #9 0x80e95272 at softdep_flushworklist+0x1a2
> #10 0x80e991df at softdep_unmount+0x4af
> #11 0x80b5be83 at fork_exit+0x83
> #12 0x8105061e at fork_trampoline+0xe
> Uptime: 1m59s
> Automatic reboot in 15 seconds - press a key on the console to abort

This looks like a file system error, likely some corruption caused by
previous reboots?

Ie: I'm not sure this is caused by Xen or rather by underlying errors
in the filesystem. At least there are no hardware errors on the log
AFAICT.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: i/o is very slow on FreeBSD dom0 with Xen-4.12 and Freebsd-12

[Roger Pau Monné]

Replying from my phone, sorry for the format.

El ds., 12 d’oct. 2019, 17:32, Brian Buhrow  va escriure:

> Hello.  Here are the logs with remapping enabled, plus the i, m
> and z
> xen console commands.
> Here's hoping you can provide a fix soon!  Or, perhaps, a clue on how to
> fix.
> -thanks
> -Brian
>
>
> 
> boot_serial="YES"   # -h: Use serial console
> comconsole_speed="115200"   # Set the current serial console speed
> comconsole_port="0x2f8" # Set the current serial console port
> #console="vidconsole"   # A comma separated list of console(s)
> console="comconsole"# A comma separated list of console(s)
> geom_mirror_load="YES"  # RAID1 disk driver (see gmirror(8))
> ahci_load="YES" # ahci driver
> ipmi_load="YES" # Ripmi driver
> if_tap_load="YES"   # Load bridge driver.
> if_vlan_load="YES"  # Load vlan driver.
> # Turn on Xen (BB 10/08/2019)
> xen_kernel="/boot/xen"
> xen_cmdline="dom0_mem=4096m dom0_max_vcpus=4 dom0=pvh,verbose
> console=com2,vga iommu=verbose,debug sync_console=true com2=115200,8n1
> guest_loglvl=all loglvl=all"
>
> 
>
> (XEN) Xen version 4.12.1 (buhrow@) (FreeBSD clang version 6.0.1
> (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)) debug=n  Thu Oct  3
> 11:34:20 PDT 2019
> (XEN) Latest ChangeSet:
> (XEN) Console output is synchronous.
> (XEN) Bootloader: FreeBSD Loader
> (XEN) Command line: dom0_mem=4096m dom0_max_vcpus=4 dom0=pvh,verbose
> console=com2,vga iommu=verbose,debug sync_console=true com2=115200,8n1
> guest_loglvl=all loglvl=all
> (XEN) Xen image load base address: 0
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
> (XEN)  EDID info not retrieved because no DDC retrieval method detected
> (XEN) Disc information:
> (XEN)  Found 4 MBR signatures
> (XEN)  Found 4 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)   - 00098400 (usable)
> (XEN)  00098400 - 000a (reserved)
> (XEN)  000e - 0010 (reserved)
> (XEN)  0010 - bf78 (usable)
> (XEN)  bf78e000 - bf79 type 9
> (XEN)  bf79 - bf79e000 (ACPI data)
> (XEN)  bf79e000 - bf7d (ACPI NVS)
> (XEN)  bf7d - bf7e (reserved)
> (XEN)  bf7ec000 - c000 (reserved)
> (XEN)  e000 - f000 (reserved)
> (XEN)  fee0 - fee01000 (reserved)
> (XEN)  ffc0 - 0001 (reserved)
> (XEN)  0001 - 00184000 (usable)
> (XEN) New Xen image base address: 0xbf00
> (XEN) ACPI: RSDP 000FACE0, 0024 (r2 ACPIAM)
> (XEN) ACPI: XSDT BF790100, 008C (r1 SMCI20120803 MSFT   97)
> (XEN) ACPI: FACP BF790290, 00F4 (r3 080312 FACP1521 20120803 MSFT   97)
> (XEN) ACPI: DSDT BF7906A0, 6580 (r1  10600 10600 INTL 20051117)
> (XEN) ACPI: FACS BF79E000, 0040
> (XEN) ACPI: APIC BF790390, 011E (r1 080312 APIC1521 20120803 MSFT   97)
> (XEN) ACPI: MCFG BF7904B0, 003C (r1 080312 OEMMCFG  20120803 MSFT   97)
> (XEN) ACPI: SLIT BF7904F0, 0030 (r1 080312 OEMSLIT  20120803 MSFT   97)
> (XEN) ACPI: OEMB BF79E040, 0086 (r1 080312 OEMB1521 20120803 MSFT   97)
> (XEN) ACPI: SRAT BF79A6A0, 01D0 (r1 080312 OEMSRAT 1 INTL1)
> (XEN) ACPI: HPET BF79A870, 0038 (r1 080312 OEMHPET  20120803 MSFT   97)
> (XEN) ACPI: DMAR BF79E0D0, 0130 (r1AMI  OEMDMAR1 MSFT   97)
> (XEN) ACPI: SSDT BF7A1710, 0363 (r1 DpgPmmCpuPm   12 INTL 20051117)
> (XEN) ACPI: EINJ BF79A8B0, 0130 (r1  AMIER AMI_EINJ 20120803 MSFT   97)
> (XEN) ACPI: BERT BF79AA40, 0030 (r1  AMIER AMI_BERT 20120803 MSFT   97)
> (XEN) ACPI: ERST BF79AA70, 01B0 (r1  AMIER AMI_ERST 20120803 MSFT   97)
> (XEN) ACPI: HEST BF79AC20, 00A8 (r1  AMIER ABC_HEST 20120803 MSFT   97)
> (XEN) System RAM: 98295MB (100654176kB)
> (XEN) SRAT: PXM 0 -> APIC 00 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 02 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 04 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 06 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 01 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 03 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 05 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 07 -> Node 0
> (XEN) SRAT: PXM 1 -> APIC 10 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 12 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 14 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 16 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 11 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 13 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 15 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 17 -> Node 1
> (XEN) SRAT: Node 0 PXM 0 0-a
> (XEN) SRAT: Node 0 PXM 0 10-c000
> (XEN) SRAT: Node 0 PXM 0 1-c4000
> (XEN) SRAT: Node 1 PXM 1 c4000-184000
> (XEN) NUMA: Allocated memnodemap from 183e1b7000 - 183e1d
> (XEN) NUMA: Using 8 for the hash shift.
> (XEN) Domain heap initialised DMA 

Re: i/o is very slow on FreeBSD dom0 with Xen-4.12 and Freebsd-12

[Roger Pau Monné]

On Fri, Oct 11, 2019 at 04:22:32PM -0700, Brian Buhrow wrote:
>   hello.  Following up on this issue yet again, I'm pretty sure I have
> the hardware discussed in the link below.  I tried disabling interrupt
> remapping as the article suggests, but that just makes matters worse.  As I
> understand it, when FreeBSD is running as a dom0, it must have iommu
> interrupt remapping enabled in order to function.

PVH dom0 requires iommu DMA remapping. Interrupt remapping is not
mandated for PVH dom0.

Can you also post the log when booting with
iommu=verbose,debug,no-intremap?

Can you also try booting without the no-intremap option, and then
switch to the hypervisor console (Ctrl-A on the serial line) and
paste the output of the i the M and the z debug keys at the point
where FreeBSD freezes? (together with the full serial log)

> Should I conclude that
> this hardware just cannot run FreeBSD as a dom0 or is there another work
> around that I might use?

I need to look deeper, I'm quite sure the issue is with Xen. There are
also some bugfixes for the iommu in Xen unstable, which I can try to
backport to the FreeBSD 4.12.1 Xen packages, albeit there have been a
lot of changes in the iommu code, so backporting might not be feasible
for some of them.

I'm leaving home today and will be on PTO until Tuesday, and I don't
expect to be able to look into this until then, sorry.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: i/o is very slow on FreeBSD dom0 with Xen-4.12 and Freebsd-12

[Roger Pau Monné]

On Thu, Oct 10, 2019 at 04:23:10PM -0700, Brian Buhrow wrote:
>   hello.  I'm trying to get a FreeBSD-12 xen server up and running and
> have run into the issue that i/o seems to be very slow for the dom0, which
> is FreeBSD-12.0-stable, and Xen is xen-4.12.
>   I followed the instructions for making FreeBSD a dom0 as written in
> 21.8 of the FreeBSD handbook.  All went well until I rebooted the machine
> with the /boot/xen kern loading.  The dom0 boots, but takes about 2 hours
> to get to the login prompt and, on the console, when ever I press
> control-t, it says what ever process is currently running is in biord.
> I'm familiar with NetBSD and how to make it run xen, and am familiar with
> FreeBSD as well.  Stil, I'm sure I'm doing something wrong that's simple,
> or, I missed some instruction that will seem obvious in retrospect.
> If anyone can help me shed light on the trouble, that would be great!

Thanks for testing it out. What you do is correct, I think there's
either an issue in Xen or the firmware.

> Below are my /boot/loader.conf and a transcript of a sample boot.
> Note that it took about  2 hours to get from the initial loading of the
> kernel to the getty error messages.
> -thanks
> -Brian
> 
> 
> boot_serial="YES" # -h: Use serial console
> comconsole_speed="115200" # Set the current serial console speed
> comconsole_port="0x2f8"   # Set the current serial console port
> #console="vidconsole" # A comma separated list of console(s)
> console="comconsole"  # A comma separated list of console(s)
> geom_mirror_load="YES"# RAID1 disk driver (see gmirror(8))
> ipmi_load="YES"   # Ripmi driver
> if_tap_load="YES" # Load bridge driver.
> if_vlan_load="YES"# Load vlan driver.
> # Turn on Xen (BB 10/08/2019)
> xen_kernel="/boot/xen"
> xen_cmdline="dom0_mem=4096m dom0_max_vcpus=4 dom0=pvh console=com2,vga 
> clocksource=hpet com2=115200,8n1 guest_loglvl=all loglvl=all"

Any reason to force the clock source to HPET? Not that's it's wrong,
I'm just curious about why you need that.

Can you change dom0=pvh to dom0=pvh,verbose and also add
iommu=verbose,debug and sync_console and post the boot log again.

There's no need to wait for 2h, just post what you get up to the point
where the box freezes.

[...]
> Trying to mount root from ufs:/dev/gpt/gptroot [rw]...
> uhub0: 6 ports with 6 removable, self powered
> uhub6: 6 ports with 6 removable, self powered
> cd0 at ata4 bus 0 scbus2 target 0 lun 0
> cd0:  Removable CD-ROM SCSI device
> cd0: 150.000MB/s transfers (SATA 1.x, UDMA2, ATAPI 12bytes, PIO 8192bytes)
> cd0: Attempt to query device size failed: NOT READY, Medium not present - 
> tray closed
> usb_alloc_device: set address 2 failed (USB_ERR_TIMEOUT, ignored)
> usbd_setup_device_deusbd_setup_device_desc: getting device descriptor at addr 
> 2 failed, USB_ERR_TIMEOUT
> usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_TIMEOUT, ignored)
> usbd_setup_device_deusbd_setup_device_desc: getting device descriptor at addr 
> 2 failed, USB_ERR_TIMEOUT
> usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_TIMEOUT, ignored)
> usbd_setup_device_desc: getting device descriptor at addr 2 failed, 
> USB_ERR_TIMEOUT
> ugen1.2:  at usbus1 (disconnected)
> uhub_reattach_port: could nGEOM_MIRROR: Device mirror/gm0 launched (1/2).
> GEOM_MIRROR: Device gm0: rebuilding provider ada1.
> WARNING: /mnt was not properly dismounted

The above timeouts point out that at least the usb controller is not
working properly. Let's see if we can get more data with the new
options on the command line.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen 4.12 - systemrescuecd-6.0.3.iso - Baremetal reboots

[Roger Pau Monné]

On Wed, Jul 24, 2019 at 06:49:28PM +, R0me0 *** wrote:
> Roger, I got a machine,  I will perform a fresh install and try to
> reproduce the bug. If possible I will send you

I've just committed a fix to xen-kernel yesterday which I think could
solve your issue. Can you update to xen-kernel 4.12.0_4?

Note you will likely need to build this from the ports tree, since
it's quite likely pkg hasn't got a binary package for this yet.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen 4.12 - systemrescuecd-6.0.3.iso - Baremetal reboots

[Roger Pau Monné]

Hello,

Could you please try to avoid top-posting.

On Tue, Jul 23, 2019 at 04:07:31PM +, R0me0 *** wrote:
> Sorry, but this host is hosted on another country which I do not have
> access to. ( Hetzner )
> 
> So, I think I can't  help.

Can you ask them to provide a serial console for you?

This is quite common practice for servers, since it's the most
reliable way to get backtraces and debug info when something goes
wrong. I'm sure your hosting provider should be able to help you.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen 4.12 - systemrescuecd-6.0.3.iso - Baremetal reboots

[Roger Pau Monné]

On Tue, Jul 23, 2019 at 03:45:49PM +, R0me0 *** wrote:
> Yes the DOM0, the XEN hypervisor completely hangs and the host
> automatically reboots.

You will have to setup a serial console in order to debug then. You
need to find the serial port of your box and hook a null modem [0] to it,
and then plug the other side into another computer and use cu, minicom
or some other program in order to get the serial output.

There's a section on the handbook about how to use the serial console:

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html

You will need to add something like:

console=com1,vga com1=115200,8n1

to your xen_cmdline if you don't have it yet in order to get the
output.

> It seems related with Linux guests using kernel 4.x

Does then same happen if you use a disk line like:

'format=raw, vdev=hda, access=rw, backendtype=qdisk, 
target=/dev/zvol/fbroot/SAND/disk'

Note the 'backendtype' option is set to qdisk.

Roger.

[0] https://en.wikipedia.org/wiki/Null_modem
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen 4.12 - systemrescuecd-6.0.3.iso - Baremetal reboots

[Roger Pau Monné]

On Tue, Jul 23, 2019 at 11:51:51AM +, R0me0 *** wrote:
> I tried to boot kali linux (
> https://cdimage.kali.org/kali-2019.2/kali-linux-2019.2-amd64.iso ) as guest.
> 
> The machine freezes on the same way I am using the latest xen 4.12 on
> freebsd

I think I'm confused by this sentence. When you write the machine
freezes, I assume you mean that the guest freezes, but the host is
responsive?

> here is the screenshot of console:
> 
> [image: 2019-07-23-084718_853x193_scrot.png]
> 
> and here is the xl dmesg
> 
> (d12) Detected Xen v4.12.0
> (d12) xen: copy BIOS tables...
> (d12) Copying SMBIOS entry point from 0x00010020 to 0x000f5e60
> (d12) Copying MPTABLE from 0xfc0011e0/fc0011f0 to 0x000f5d40
> (d12) Copying PIR from 0x00010040 to 0x000f5cc0
> (d12) Copying ACPI RSDP from 0x000100c0 to 0x000f5c90
> (d12) Using pmtimer, ioport 0xb008
> (d12) Scan for VGA option rom
> (d12) Running option rom at c000:0003
> (d12) pmm call arg1=0
> (d12) Turning on vga text mode console
> (d12) SeaBIOS (version 1.11.0-20190516_220714-120amd64-default-job-06)
> (d12) Machine UUID 2e347e67-ad3f-11e9-9209-0cc47a4dbca8
> (d12) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
> (d12) ATA controller 2 at 170/374/0 (irq 15 dev 9)
> (d12) Found 0 lpt ports
> (d12) Found 1 serial ports
> (d12) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (80 GiBytes)
> (d12) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
> (d12) DVD/CD [ata0-1: QEMU DVD-ROM ATAPI-4 DVD/CD]
> (d12) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@1
> (d12) PS2 keyboard initialized
> (d12) All threads complete.
> (d12) Scan for option roms
> (d12) Running option rom at c980:0003
> (d12) pmm call arg1=1
> (d12) pmm call arg1=0
> (d12) pmm call arg1=1
> (d12) pmm call arg1=0
> (d12) Searching bootorder for: /pci@i0cf8/*@4
> (d12)
> (d12) Press ESC for boot menu.
> (d12)
> (d12) Searching bootorder for: HALT
> (d12) drive 0x000f5c20: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
> s=167772160
> (d12) Space available for UMB: ca800-eb000, f5680-f5bc0
> (d12) Returned 258048 bytes of ZoneHigh
> (d12) e820 map has 7 items:
> (d12)   0:  - 0009fc00 = 1 RAM
> (d12)   1: 0009fc00 - 000a = 2 RESERVED
> (d12)   2: 000f - 0010 = 2 RESERVED
> (d12)   3: 0010 - e000 = 1 RAM
> (d12)   4: e000 - f000 = 2 RESERVED
> (d12)   5: fc00 - 0001 = 2 RESERVED
> (d12)   6: 0001 - 00010f80 = 1 RAM
> (d12) enter handle_19:
> (d12)   NULL
> (d12) Booting from Hard Disk...
> (d12) Boot failed: not a bootable disk
> (d12)
> (d12) enter handle_18:
> (d12)   NULL
> (d12) Booting from DVD/CD...
> (d12) Booting from :7c00
> 
> 
> 
> Em qui, 11 de jul de 2019 às 21:02, R0me0 *** 
> escreveu:
> 
> > Hello Roger,
> >
> > >So you are trying to boot systemrescuecd as a guest on the Xen host?
> > Yes.
> >
> > Here is the config I have used:
> >
> > # cat livecd.cfg
> >
> > builder = "hvm"
> > name = "livecd"
> > memory = 2048
> > cpus="all,^0-1"
> > vcpus = 2
> > vif = [ 'type=vif,bridge=sandbox' ]
> > disk = [ '/dev/zvol/fbroot/SAND/disk,raw,xvda,w' ,

Can you try with:

'format=raw, vdev=hda, access=rw, backendtype=qdisk, 
target=/dev/zvol/fbroot/SAND/disk'

Which version of FreeBSD are you running?

Can you also paste the output of `dmesg` when this happens?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Error: xc: error: panic: xc_dom_boot.c:159: xc_dom_boot_domU_map: failed to mmap domU pages 0x100+0x33 [mmap, errno=-6 (Unknown error:-6)]: Internal error

[Roger Pau Monné]

On Mon, Jul 22, 2019 at 11:54:35PM +0900, YAMAMOTO Shigeru wrote:
> 
> Hi, all,
> 
> I'm using Xen Dom0 on FreeBSD/amd64.
> When following error occurs, I can not create new Xen DomU and I need 
> rebooting
> Xen Dom0 to recover from error.
> 
> # xl create jenkins-01.cfg
> Parsing config from jenkins-01.cfg
> xc: error: panic: xc_dom_boot.c:159: xc_dom_boot_domU_map: failed to mmap
> domU pages 0x100+0x33 [mmap, errno=-6 (Unknown error: -6)]: Internal error
> libxl: error: libxl_dom.c:760:libxl__build_dom: xc_dom_build_image failed:
> Invalid argument
> libxl: error: libxl_create.c:1286:domcreate_rebuild_done: Domain 9:cannot
> (re-)build domain: -3
> libxl: error: libxl_domain.c:1038:libxl__destroy_domid: Domain
> 9:Non-existant domain
> libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 9:Unable to
> destroy guest
> libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 9:Destruction of
> domain failed

Thanks for the report!

I've fixed this in upstream Xen quite recently, and haven't backported
the patch to FreeBSD yet. I've now created a differential revision at:

https://reviews.freebsd.org/D21028

With the fix. Feel free to apply it to your ports tree and rebuild
xen-kernel, this should give you a working system.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen 4.12 - systemrescuecd-6.0.3.iso - Baremetal reboots

[Roger Pau Monné]

On Fri, Jun 28, 2019 at 12:02:37PM +, R0me0 *** wrote:
> Guys,
> 
> I'am using FreeBSD 12.0-RELEASE-p6 FreeBSD 12.0-RELEASE-p6 GENERIC  amd64

I guess this is on the host?

> When booting systemrescuecd-6.0.3.iso ( http://www.system-rescue-cd.org )
> ,  my system just hangs and reboot.

So you are trying to boot systemrescuecd as a guest on the Xen host?

Can you paste your guest config file?

> Just tried an older version and it's ok.
> 
> I don't have any log to help. If you guys want and some help, I can provide
> more information.

Could you get a serial hooked up to that box so we can get the trace?

Without some kind of trace it's going to be hard to debug this.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

[Roger Pau Monné]

On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote:
> Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné :
> 
> > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote:
> > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128.
> > VIF
> > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's.
> > >
> > > I feel the throughput is more consistent now, not all over the place as
> > > before, even between runs. But the Retr column (tcp retries) in iperf3
> > has
> > > jumped up considerably from hundreds/s to thousands/s.
> > >
> > > Just a reminder, I have tested this with 11.0-RELEASE also, where the
> > issue
> > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to
> > be,
> > > and 0 retries.
> > >
> > > 12.0-STABLE:
> > >
> > > Connecting to host 172.31.16.128, port 5201
> > > [  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201
> > > [ ID] Interval   Transfer Bitrate Retr  Cwnd
> > > [  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85 KBytes
> > >
> > > [  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0 KBytes
> > >
> > > [  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8 KBytes
> > >
> > > [  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6 KBytes
> > >
> > > [  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6 KBytes
> > >
> > > [  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0 KBytes
> > >
> > > [  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2 KBytes
> > >
> > > [  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85 KBytes
> > >
> > > [  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8 KBytes
> > >
> > > [  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2 KBytes
> > >
> > > [  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3 KBytes
> > >
> > > [  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9 KBytes
> > >
> > > [  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3 KBytes
> > >
> > > [  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1 KBytes
> > >
> > > [  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5 KBytes
> > >
> > > [  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3 KBytes
> > >
> > > [  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8 KBytes
> > >
> > > [  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8 KBytes
> > >
> > > [  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2 KBytes
> > >
> > > [  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9 KBytes
> > >
> > > - - - - - - - - - - - - - - - - - - - - - - - - -
> > > [ ID] Interval   Transfer Bitrate Retr
> > > [  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845
> >
> > Can you paste the output of ifconfig for both the interfaces used in
> > the test?
> >
> > Are you sure all hardware offloading capabilities are turned off on
> > both interfaces?
> >
> > Can you check what's causing those retries?
> >
> > Either using tcpdump, whireshark or some other tool to analyze the
> > network traffic and detect the errors that cause such retries?
> >
> > Thanks, Roger.
> >
> 
> 172.31.16.127 (12.0-STABLE):
> 
> lo0: flags=8049 metric 0 mtu 16384
> options=680003
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> inet 127.0.0.1 netmask 0xff00
> groups: lo
> nd6 options=21
> xn0: flags=8843 metric 0 mtu 1500
> options=501

I would try to disable rxcsum, tso4 and lro also.

> ether 6e:83:99:ed:ce:f7
> inet 172.31.16.127 netmask 0xff00 broadcast 172.31.16.255
> media: Ethernet manual
> status: active
> nd6 options=29
> 
> ethtool -k vif68.0
> Features for vif68.0:
> rx-checksumming: on [fixed]
> tx-checksumming: off
> tx-checksum-ipv4: off
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: off
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: off
> tx-scatter-gather: off
> tx-scatter-gather-fraglist: off
> tcp-segmentation-offload: off
> tx-tcp-segmentation: off
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp6-segmentation: off
&

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

[Roger Pau Monné]

On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote:
> I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF
> cheksum offloading is turned off, and -txcsum for xn0 for both VM's.
> 
> I feel the throughput is more consistent now, not all over the place as
> before, even between runs. But the Retr column (tcp retries) in iperf3 has
> jumped up considerably from hundreds/s to thousands/s.
> 
> Just a reminder, I have tested this with 11.0-RELEASE also, where the issue
> appeared first for me. 10.4-RELEASE is as fast as I could expect it to be,
> and 0 retries.
> 
> 12.0-STABLE:
> 
> Connecting to host 172.31.16.128, port 5201
> [  5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201
> [ ID] Interval   Transfer Bitrate Retr  Cwnd
> [  5]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec  2401   2.85 KBytes
> 
> [  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec  3120   17.0 KBytes
> 
> [  5]   2.00-3.00   sec   121 MBytes  1.02 Gbits/sec  3203   69.8 KBytes
> 
> [  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  3126   15.6 KBytes
> 
> [  5]   4.00-5.00   sec   110 MBytes   921 Mbits/sec  2890   15.6 KBytes
> 
> [  5]   5.00-6.00   sec   108 MBytes   908 Mbits/sec  3308   17.0 KBytes
> 
> [  5]   6.00-7.00   sec   104 MBytes   869 Mbits/sec  3046   48.2 KBytes
> 
> [  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec  2845   2.85 KBytes
> 
> [  5]   8.00-9.00   sec   104 MBytes   874 Mbits/sec  2711   86.8 KBytes
> 
> [  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec  2696   14.2 KBytes
> 
> [  5]  10.00-11.00  sec   103 MBytes   864 Mbits/sec  2660   31.3 KBytes
> 
> [  5]  11.00-12.00  sec  98.8 MBytes   828 Mbits/sec  2476   19.9 KBytes
> 
> [  5]  12.00-13.00  sec  99.9 MBytes   838 Mbits/sec  2857   11.3 KBytes
> 
> [  5]  13.00-14.00  sec   107 MBytes   894 Mbits/sec  2685   24.1 KBytes
> 
> [  5]  14.00-15.00  sec   114 MBytes   953 Mbits/sec  2321   25.5 KBytes
> 
> [  5]  15.00-16.00  sec  93.1 MBytes   781 Mbits/sec  2427   48.3 KBytes
> 
> [  5]  16.00-17.00  sec   107 MBytes   895 Mbits/sec  2219   29.8 KBytes
> 
> [  5]  17.00-18.00  sec  92.5 MBytes   776 Mbits/sec  2441   12.8 KBytes
> 
> [  5]  18.00-19.00  sec   116 MBytes   976 Mbits/sec  2840   38.2 KBytes
> 
> [  5]  19.00-20.00  sec   102 MBytes   853 Mbits/sec  2573   43.9 KBytes
> 
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval   Transfer Bitrate Retr
> [  5]   0.00-20.00  sec  2.05 GBytes   882 Mbits/sec  54845

Can you paste the output of ifconfig for both the interfaces used in
the test?

Are you sure all hardware offloading capabilities are turned off on
both interfaces?

Can you check what's causing those retries?

Either using tcpdump, whireshark or some other tool to analyze the
network traffic and detect the errors that cause such retries?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer

[Roger Pau Monné]

There's a mistake in my reply below.

On Tue, Jun 25, 2019 at 04:56:43PM +0200, Roger Pau Monné wrote:
> On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote:
> > I've made two tests while running tcpdump on the xcp-ng host. I'm not at
> > all qualified to interpret the .pcap files from tcpdump, but I've put them
> > on Google Drive and linked them below the two tests. Perhaps someone more
> > qualified could have a look for anything useful in there. Please note the
> > extremely uneven throughput for test 2 below. It's almost like the
> > throughput increased when running tcpdump simultaneously.
> > 
> > Host: XCP-ng 7.6.0
> > Network: Private Network on host, not connected to any PIF.
> > VM1: 12.0-RELEASE (1 VIF, 172.31.16.125)
> > VM2: 12.0-RELEASE (1 VIF, 172.31.15.126)
> > 
> > On the host I listen with tcpdump on the VIF for VM1 in both tests.
> > 
> > VM1 as client:
> > 
> > On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap
> 
> Can you check the capabilities of vif42.0? (ie: whether csum
> offloading is actually disabled on the host?)
> 
> > xcp-ng-vm1-client.pcap (80M):
> > https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU
> > On VM1: iperf3 -c 172.31.16.126
> > On VM2: iperf3 -s
> 
> I've taken a look at the dump and the checksum is wrong (or maybe
> missing) for all? packets.
> 
> Packets with source 172.31.16.125 all have the TCP checksum set to
> 0x7f80 and all packets with source 172.31.16.125 have the TCP checksum
 ^ 172.31.16.126
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: using virsh and virt-manager for XEN on FreeBSD

[Roger Pau Monné]

On Fri, Feb 15, 2019 at 07:43:54AM +, Eric Bautsch wrote:
> Hi All
> 
> 
> And one more:
> 
> root@bianca # virsh -V
> Virsh command line tool of libvirt 4.10.0
> See web site at https://libvirt.org/
> 
> Compiled with support for:
>  Hypervisors: VMware PHYP VirtualBox ESX Bhyve Test
>  Networking: Remote Network Bridging
>  Storage: Dir ZFS
>  Miscellaneous: Daemon Secrets Debug Readline
> root@bianca # virsh -c xen:///system
> error: failed to connect to the hypervisor
> error: no connection driver available for xen:///system
> 
> 
> Based on this output, I'm assuming that virsh (and virt-manager, too) wasn't
> compiled with XEN support on FreeBSD.
> 
> 
> However, this page (from 2015):
> https://forums.freebsd.org/threads/libvirt-libxl-on-freebsd.51862/
> 
> 
> Would seem to suggest that it should work.
> 
> 
> Should I just stop trying and compile my own virsh and virt-manager?

I haven't looked into building virt-manager with Xen support myself,
there's definitely Xen support in virt-manager that uses libxl IIRC,
but that's not enabled by the virt-manager port.

Would you be willing to look into enabling Xen support for the
virt-manager port?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Issues with XEN and ZFS

[Roger Pau Monné]

Thanks for the testing!

On Fri, Feb 08, 2019 at 07:35:04PM +, Eric Bautsch wrote:
> Hi.
> 
> 
> Brief abstract: I'm having ZFS/Xen interaction issues with the disks being
> declared unusable by the dom0.
>
> 
> The longer bit:
> 
> I'm new to FreeBSD, so my apologies for all the stupid questions. I'm trying
> to migrate from Linux as my virtual platform host (very bad experiences with
> stability, let's leave it at that). I'm hosting mostly Solaris VMs (that
> being my choice of OS, but again, Betamax/VHS, need I say more), as well as
> a Windows VM (because I have to) and a Linux VM (as a future desktop via
> thin clients as and when I have to retire my SunRay solution which also runs
> on a VM for lack of functionality).
> 
> So, I got xen working on FreeBSD now after my newbie mistake was pointed out 
> to me.
> 
> However, I seem to be stuck again:
> 
> I have, in this initial test server, only two disks. They are SATA hanging
> off the on-board SATA controller. The system is one of those Shuttle XPC
> cubes, an older one I had hanging around with 16GB memory and I think 4
> cores.
> 
> I've given the dom0 2GB of memory and 2 core to start with.

2GB might be too low when using ZFS, I would suggest 4G as a minimum
when using ZFS for reasonable performance, even 8G. ZFS is quite
memory hungry.

> The root filesystem is zfs with a mirror between the two disks.
> 
> The entire thing is dead easy to blow away and re-install as I was very
> impressed how easy the FreeBSD automatic installer was to understand and
> pick up, so I have it all scripted. If I need to blow stuff away to test, no
> problem and I can always get back to a known configuration.
> 
> 
> As I only have two disks, I have created a zfs volume for the Xen domU thus:
> 
> zfs create -V40G -o volmode=dev zroot/nereid0
> 
> 
> The domU nereid is defined thus:
> 
> cat - << EOI > /export/vm/nereid.cfg
> builder = "hvm"
> name = "nereid"
> memory = 2048
> vcpus = 1
> vif = [ 'mac=00:16:3E:11:11:51,bridge=bridge0',
> 'mac=00:16:3E:11:11:52,bridge=bridge1',
> 'mac=00:16:3E:11:11:53,bridge=bridge2' ]
> disk = [ '/dev/zvol/zroot/nereid0,raw,hda,rw' ]
> vnc = 1
> vnclisten = "0.0.0.0"
> serial = "pty"
> EOI
> 
> nereid itself also auto-installs, it's a Solaris 11.3 instance.
> 
> 
> As it tries to install, I get this in the dom0:
> 
> Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0):
> WRITE_FPDMA_QUEUED. ACB: 61 18 a0 ef 88 40 46 00 00 00 00 00
> Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
> Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): CAM
> status: CCB request was invalid

That's weird, and I would say it's not related to ZFS, the same could
likely happen with UFS since this is an error message from the
disk controller hardware.

Can you test whether the same happens _without_ Xen running?

Ie: booting FreeBSD without Xen and then doing some kind of disk
stress test, like fio [0].

Thanks, Roger.

[0] https://svnweb.freebsd.org/ports/head/benchmarks/fio/
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Extend documentation (was: Re: Issue getting my first Xen domU up)

[Roger Pau Monné]

On Wed, Feb 06, 2019 at 09:39:47AM -0800, Rodney W. Grimes wrote:
> > Adding Benedict who wrote the Xen handbook documentation, and
> > hopefully would be able to fix it :).
> > 
> > On Tue, Feb 05, 2019 at 08:18:05PM +, Marcin Cieslak wrote:
> > > On Tue, 5 Feb 2019, Eric Bautsch wrote:
> > > 
> > > > And yes, the qemu-dm-titania.log would have been the place to look. I 
> > > > feel
> > > > pretty stupid. It says:
> > > 
> > > Don't feel stupid. I am using Xen with Dom0 on FreeBSD for months now
> > > and I didn't know where the qemu logs go - I was able to figure out
> > > the problem by staring at the configuration file for a long time.
> > > 
> > > Therefore your question was very helpful also for me - thanks!
> > 
> > It seems there's one piece missing in the handbook which I didn't
> > realize, the if_tap module needs to be loaded for QEMU to work.
> > 
> > I suggest the following is added to the handbook:
> > 
> > ```
> > sysrc -f /boot/loader.conf if_tap_load="YES"
> > ```
> > 
> > And then I've also realized the following line:
> > 
> > ```
> > sysrc -f /boot/loader.conf hw.pci.mcfg=0
> > ```
> > 
> > Is only needed for Xen 4.7, but not for Xen 4.11.
> > 
> > Finally it might be interesting to add a reference to the location of
> > the Xen logs. This should likely be added at the end of the document,
> > I think something along the lines of:
> > 
> > ```
> > Xen toolstack related logs can be found at /var/log/xen/ be sure to
> > check the contents of that directory if experiencing issues.
> > ```
> 
> Is there a troubleshooting section in the handbook?
> If not there should be, and this would defanitly be mentioned
> in that section.  

There doesn't seem to be a virtualization troubleshooting section I'm
afraid. I've added something similar to my suggestion above as a tip,
which will make it more visible.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Extend documentation (was: Re: Issue getting my first Xen domU up)

[Roger Pau Monné]

On Wed, Feb 06, 2019 at 11:45:35PM +, Andrew Daugherity wrote:
> > On Feb 6, 2019, at 11:17 AM, Roger Pau Monné  wrote:
> > 
> > Adding Benedict who wrote the Xen handbook documentation, and
> > hopefully would be able to fix it :).
> > 
> > I suggest the following is added to the handbook:
> > […]
> 
> 
> 
> While you’re at it, I think some other updates to that handbook page are in 
> order.  I just read it again for the first time in a good while and the 
> following things caught my eye:

Thanks for the feedback!

> - 21.8.2 mentions using Xen 4.7 on FreeBSD 11 and Xen 4.11 on FreeBSD-CURRENT 
> >= r336475.  As 12.0 has been released, that merits a mention here… while 
> "FreeBSD 12.0-RELEASE r341666 GENERIC" does have a higher rev number, I’m not 
> sure whether or not it contains the necessary stuff for Xen 4.11.

FreeBSD 12.0 or FreeBSD-HEAD are all capable of using Xen 4.11.

> - The /var/log/xen dir is created (and owned) by the xen-tools4{7,11} 
> packages, so no need to manually create it.
> 
> - Wouldn’t 'onifpresent' be a better setting for xc0 in /etc/ttys? Then if 
> you booted back to a vanilla kernel, it won’t try (and fail) to start a getty 
> for a nonexistent device.
> 
> - The DomU example creates a FreeBSD 10.3 VM.  Probably better to showcase a 
> supported release…

Right, let me see about this.

> - Seeing "Xen(TM)" sprinkled throughout the document interrupts the reading 
> flow.  Surely it’s enough to use the trademark symbol for the first mention 
> and then simply "Xen" thereafter?

I don't have an opinion here, I don't know whether using Xen^TM is
mandatory.

> - Is it really necessary to 'xl destroy' the DomU after shutting it down?  
> Once it’s shut down, it shouldn’t be in the list any more, and you just 'xl 
> create' with the new config, right?

I don't see the manual suggesting to perform an `xl shutdown ...`, am
I missing something?

It suggests to destroy the domain after install is finished, there's
no need to perform a graceful shutdown of the installer.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Extend documentation (was: Re: Issue getting my first Xen domU up)

[Roger Pau Monné]

Adding Benedict who wrote the Xen handbook documentation, and
hopefully would be able to fix it :).

On Tue, Feb 05, 2019 at 08:18:05PM +, Marcin Cieslak wrote:
> On Tue, 5 Feb 2019, Eric Bautsch wrote:
> 
> > And yes, the qemu-dm-titania.log would have been the place to look. I feel
> > pretty stupid. It says:
> 
> Don't feel stupid. I am using Xen with Dom0 on FreeBSD for months now
> and I didn't know where the qemu logs go - I was able to figure out
> the problem by staring at the configuration file for a long time.
> 
> Therefore your question was very helpful also for me - thanks!

It seems there's one piece missing in the handbook which I didn't
realize, the if_tap module needs to be loaded for QEMU to work.

I suggest the following is added to the handbook:

```
sysrc -f /boot/loader.conf if_tap_load="YES"
```

And then I've also realized the following line:

```
sysrc -f /boot/loader.conf hw.pci.mcfg=0
```

Is only needed for Xen 4.7, but not for Xen 4.11.

Finally it might be interesting to add a reference to the location of
the Xen logs. This should likely be added at the end of the document,
I think something along the lines of:

```
Xen toolstack related logs can be found at /var/log/xen/ be sure to
check the contents of that directory if experiencing issues.
```

Sorry to bother you Benedict, but do you think you could make the
following changes?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Issue getting my first Xen domU up

[Roger Pau Monné]

On Tue, Feb 05, 2019 at 07:17:55AM +, Eric Bautsch wrote:
> Hi.
> 
> 
> I'm trying to get a Xen domU up and running.
> 
> I'm running FreeBSD 12.0. System is a new install.
> 
> 
> I'm following the instructions here:
> 
> https://www.freebsd.org/doc/handbook/virtualization-host-xen.html
> 
> 
> My config file looks thus:
> 
> builder = "hvm"
> name = "titania"
> memory = 8192
> vcpus = 2
> vif = [ 'mac=00:16:3E:11:11:21,bridge=bridge0' ]
> disk = [ '/dev/zvol/zroot/titania0,raw,hda,rw' ]
> vnc = 1
> vnclisten = "0.0.0.0"
> serial = "pty"

That seems like a valid configuration.

> Upon trying to create the domU, I think it's trying to tell me that it's got
> issues with the storage, but that may be a red herring as the qemu command
> seems to include the storage. Nevertheless, I have also tried with sda and
> messed around with qcow2 (that one didn't work, but probably due to my wrong
> incantation). Anyway here's the output from the above config file.

It's indeed QEMU failing the start the problem here. Do you have the
if_tap module loaded (`kldload if_tap`)?

Can you provide the QEMU log, it should be in
/var/log/xen/qemu-dm-titania.log.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: FreeBSD/Xen suspend/resume

[Roger Pau Monné]

On Sat, Jan 19, 2019 at 10:59:55AM +, Uni Gaia wrote:
> Does anyone know if suspend/resume works for a FreeBSD/Xen dom0 ? domU?

FreeBSD DomU does suspend and resume just fine, or else it's a bug and
should be fixed.

I think FreeBSD Dom0 suspend hasn't been tested (at least I haven't
tested it myself, neither seen any report on the list). That's mostly
because I work with servers. I'm happy to help with this or review
patches if someone wants to make it work.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: ARM/ARM64 FreeBSD/Xen

[Roger Pau Monné]

On Sat, Jan 19, 2019 at 10:59:08AM +, Uni Gaia wrote:
> In https://wiki.xenproject.org/wiki/Xen_Project_Release_Features under
> "Supported Mainline Architectures for the hypervisor (Host)" it is written
> that ARMv7+virt extensions and ARMv8 are supported by Xen.
> 
> Is work ongoing in porting FreeBSD/Xen to ARM/ARM64?

AFAIK there's no current effort to port FreeBSD to run on Xen on ARM,
either as DomU or Dom0.

I'm adding Julien who works on Xen on ARM, he might have more
information than myself, since I mostly work on x86.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: freebsd 11.2-p3 Xen - cant start Xen with dom0_max_vcpus= 2 or more

[Roger Pau Monné]

On Mon, Sep 24, 2018 at 01:52:20PM -0300, R0me0 *** wrote:
> Hello guys,
> 
> I have trying to deploy a production enviroment using FreeBSD
> 
> If I try to boot Xen with more than 1 vcpu allocated to Dom0, the host
> freezes.
> 
> ( also tryied compiled from ports )
> 
> any direction will be appreciated.
> 
> Thanks in Advance

Can you paste the full boot log (from the serial console) when booting
with more than one vCPU?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: HEADS UP: merged PVHv2 support and future plans

[Roger Pau Monné]

On Sat, Aug 25, 2018 at 06:54:54PM +0200, Kai Otto wrote:
> On 13/08/18 19:19, Roger Pau Monné wrote:
> > On Mon, Aug 13, 2018 at 05:24:04PM +0200, Kai Otto wrote:
> >> On 30/07/18 12:19, Roger Pau Monné wrote:
> >>> Packages for Xen 4.11 are now available in pkg.
> >>>
> >>> Remember that in order to use those you need a very recent FreeBSD
> >>> kernel (r336475 or newer) and the Xen command line has slightly changed,
> >>> so dom0=pvh must be used instead of dom0pvh in order to boot. I will
> >>> see about changing this in the handbook.
> >>>
> >>> Roger.
> >>> ___
> >>> freebsd-xen@freebsd.org mailing list
> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-xen
> >>> To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
> >>>
> >>
> >> Hi Roger,
> >> I tried to get some VMs running on
> >> FreeBSD-12.0-CURRENT-amd64-20180802-r337160.
> >> System is a Supermicro X9DRi-LN4F+ mainboard with dual Intel E5-2680v2
> >> (ivy bridge).
> [...]
> >> When using xen-tools411/xen-kernel411, I can boot without problems, but
> >> when running "xl create", neither a PVH nor HVM config work (see configs
> >> and logs attached).
> >> They both throw "libxl: info:
> >> libxl_create.c:109:libxl__domain_build_info_setdefault: qemu-xen is
> >> unavailable, using qemu-xen-traditional instead: No such file or directory"
> > 
> > Hello,
> > 
> > This is a mistake of the port makefile which should be fixed in
> > r477080 that I've just committed. I hope updated packages will be
> > generated soon. Thanks for the testing!
> > 
> 
> Hello again :-)
> 
> Today I gave it another try with FreeBSD 12.0-ALPHA3 r338287 and the
> current packages for xen-tools411 and xen-kernel411. Sadly, I didn't get
> very far, even with the above bug being fixed.
> When attempting to start a HVM config, the console (xl console )
> stays empty and unresponsive.
> Connecting via vnc, the screen only shows this message: "Guest has not
> initialized the display (yet)".

I think this should be fixed by r477953, please make sure you have the
following package installed: xen-tools411-4.11.0_4

> When repeatedly running 'xl list', I can see that the domU almost
> consumes 1 second of CPU per second walltime.
> 
> Trying out PVH, the console also doesn't show any sign of life, and VNC
> says: "This VM has no graphic display device."
> Running 'xl list' shows, that the domU consumed about 1.4s of CPU, and
> then switched to 'blocked' state.

Can you paste the config file you are using to create the PVH guest?

And which OS/version are you trying to boot as PVH?

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Using xen host on freebsd, is com1 the only way to attach to dom0?

[Roger Pau Monné]

On Fri, Aug 31, 2018 at 11:12:30PM -0400, Steven Friedrich wrote:
> I can boot a xen kernel, but it appears it doesn't acquire a DHCP address
> from my router.

Does this mean that the box doesn't boot correctly, or just that the
network card is not working when running under Xen?

> Someone suggested using vga=current,keep on xen_cmdline,
> but I'm unsure of the valid xen_cmdline arguments.

The best option would be to hook a serial cable and get the output
from the serial console.

The list of valid xen_cmdline can be found here for the 4.11 ports:

http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html

Or here if you are using the 4.7 port:

http://xenbits.xen.org/docs/4.7-testing/misc/xen-command-line.html

> Do I have to configure and build a FreeBSD kernel or just use what the xen
> meta-port builds?

You don't have to build a specific kernel to run FreeBSD as Dom0, just
using the GENERIC one from either 11.X or HEAD should work.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: xen+vimage kernel panic

[Roger Pau Monné]

On Sun, Aug 19, 2018 at 10:48:52PM +0200, Marko Zec wrote:
> On Sun, 19 Aug 2018 12:50:55 -0600
> Nathan Friess  wrote:
> 
> > Hi,
> > 
> > While testing out the new PVH support in a domU (which is running 
> > great!), I discovered a kernel panic related to xen and vimage
> > support when trying to add an xn interface into a bridge.
> > 
> > I'm running r337024 from svn.  Removing vimage (which seems to be
> > turned on in 12-CURRENT now) allows using the bridge with no panics.
> > As part of attempting to debug this I enabled vimage in my 11.2 domU
> > and that also panics in the same code.
> > 
> > I'm not sure if the problem is a xen issue or a vimage issue so I 
> > haven't submitted a PR yet.  The kernel output is listed below.
> > 
> > It looks like netfront_backend_changed() calls
> > netfront_send_fake_arp(), which calls arp_ifinit() on the interface.
> > The first line of the call stack with arprequest+0x454 corresponds to
> > a call to ARPSTAT_INC(txrequests) at the end of arprequest, which
> > expands to VNET_PCPUSTAT_ADD().  I tried to debug further and I got a
> > little lost, but that's where I figured out that vimage is involved
> > somehow.
> > 
> > Are there any thoughts on why the xn interface would cause a panic
> > there?
> 
> The xn driver calls arp_ifinit() without setting the vnet context
> first.  Perhaps the attached patch could help (not even compile
> tested...)

I know nothing about VNET, so is this initialization required now that
VNET is enabled? Is this an existing bug in netfront that was harmless
before VNET was activated?

Can you please file a bug report and attach the patch?

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: HEADS UP: merged PVHv2 support and future plans

[Roger Pau Monné]

On Mon, Aug 13, 2018 at 05:24:04PM +0200, Kai Otto wrote:
> On 30/07/18 12:19, Roger Pau Monné wrote:
> > Packages for Xen 4.11 are now available in pkg.
> > 
> > Remember that in order to use those you need a very recent FreeBSD
> > kernel (r336475 or newer) and the Xen command line has slightly changed,
> > so dom0=pvh must be used instead of dom0pvh in order to boot. I will
> > see about changing this in the handbook.
> > 
> > Roger.
> > ___
> > freebsd-xen@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-xen
> > To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
> > 
> 
> Hi Roger,
> I tried to get some VMs running on
> FreeBSD-12.0-CURRENT-amd64-20180802-r337160.
> System is a Supermicro X9DRi-LN4F+ mainboard with dual Intel E5-2680v2
> (ivy bridge).
> 
> When using xen-tools47/xen-kernel47, during boot of dom0, I only get the
> error seen in the attached png ("d0v0 Triple fault").
> 
> 
> When using xen-tools411/xen-kernel411, I can boot without problems, but
> when running "xl create", neither a PVH nor HVM config work (see configs
> and logs attached).
> They both throw "libxl: info:
> libxl_create.c:109:libxl__domain_build_info_setdefault: qemu-xen is
> unavailable, using qemu-xen-traditional instead: No such file or directory"

Hello,

This is a mistake of the port makefile which should be fixed in
r477080 that I've just committed. I hope updated packages will be
generated soon. Thanks for the testing!

> Which doesn't make much sense to me, as PVH shouldn't use qemu at all?

Some PVH guests will use QEMU in order to run the PV disk backend or
the PV framebuffer (vfb), but PVH won't have any of the emulated
devices by QEMU.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: HEADS UP: merged PVHv2 support and future plans

[Roger Pau Monné]

On Thu, Jul 19, 2018 at 11:04:44AM +0200, Roger Pau Monné wrote:
[...]
> After introducing PVHv2 PVHv1 was deprecated and PVHv1 has been
> removed from the hypervisor in recent versions, that's why the Xen
> ports package is still stuck with Xen 4.7, because later versions
> removed PVHv1 support. With the addition of PVHv2 to FreeBSD the port
> can be updated to newer Xen versions and we can move forward.

Packages for Xen 4.11 are now available in pkg.

Remember that in order to use those you need a very recent FreeBSD
kernel (r336475 or newer) and the Xen command line has slightly changed,
so dom0=pvh must be used instead of dom0pvh in order to boot. I will
see about changing this in the handbook.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Recursion in non-recursive mutex when using the grant table free callbacks

[Roger Pau Monné]

On Sun, Jul 29, 2018 at 06:08:56PM +0530, Pratyush Yadav wrote:
> Hi,
> 
> Currently, the grant table free callbacks can not work. This is
> because of a recursion on a non-recursive mutex that causes a kernel
> panic. The cause of the recursion is: check_free_callbacks() is always
> called with the lock gnttab_list_lock held. So, the callback function
> is called with the lock held. So, when the client uses any of the
> grant reference allocation methods get_free_entries() is called, which
> tries to acquire gnttab_list_lock(grant_table.c:77 [0]), causing a
> recursion on the lock.
> 
> I'm not sure what the correct fix would be though. One way I can think
> of is that check_free_callback() should be called without the lock
> held. But with this fix, it is possible for the callback to be called
> even though the grant references it needs are not available. This
> would happen when another thread takes those references while the
> current thread has completed the check if(gnttab_free_count >=
> callback->count) but has not yet called the callback
> (grant_table,c:105 [1]).
> 
> I think a better way to fix this would be to have a check in
> get_free_entries() whether the current thread holds the lock, so it
> does not try to acquire the lock if the current thread already holds
> it.

I agree in the analysis, however I think the proper solution is to use
a recursive lock.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: HEADS UP: merged PVHv2 support and future plans

[Roger Pau Monné]

On Thu, Jul 19, 2018 at 12:46:07PM +, Marcin Cieslak wrote:
> On Thu, 19 Jul 2018, Roger Pau Monné wrote:
> 
> > Hello,
> > 
> > Today I've merged PVHv2 support into FreeBSD, allowing FreeBSD to be
> > used as a PVHv2 DomU and Dom0. While it's not a huge set of changes,
> > I would *really* appreciate if people could test the code starting
> > from r336474 (or any later changeset).
> 
> Hi Roger, I am ready to test it on my (small) setup, should I keep
> existing 4.7 running as is and just upgrade the Dom0? 

Yes, just update the FreeBSD kernel.

> Or should I upgrade Xen too?

I haven't updated the Xen package, so there's no easy way ATM to test
using a PVHv2 Dom0. Let's focus first on making sure I didn't break
existing setups.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: HEADS UP: merged PVHv2 support and future plans

[Roger Pau Monné]

On Thu, Jul 19, 2018 at 02:03:00PM +0200, Miroslav Lachman wrote:
> Roger Pau Monné wrote on 2018/07/19 11:04:
> > Hello,
> > 
> > Today I've merged PVHv2 support into FreeBSD, allowing FreeBSD to be
> > used as a PVHv2 DomU and Dom0. While it's not a huge set of changes,
> > I would *really* appreciate if people could test the code starting
> > from r336474 (or any later changeset).
> > 
> > I expect there's going to be some confusion with PVHv1 vs PVHv2, so I
> > will try to clarify this now. PVHv1 was introduced ~4 years ago, and
> > at the time it seemed like a good way to move forward, allowing Xen to
> > rely more on hardware virtualization. Later on, we sadly discovered
> > that PVHv1 was still too similar to classic PV, and didn't allow Xen
> > to make use of all the possible hardware virtualization extensions, so
> > PVHv2 was introduced ~2 years ago as a replacement for PVHv1. PVHv2
> > ABI however is not compatible with PVHv1, which means that different
> > entry points and interfaces must be used to interact with the
> > hypervisor.
> > 
> > After introducing PVHv2 PVHv1 was deprecated and PVHv1 has been
> > removed from the hypervisor in recent versions, that's why the Xen
> > ports package is still stuck with Xen 4.7, because later versions
> > removed PVHv1 support. With the addition of PVHv2 to FreeBSD the port
> > can be updated to newer Xen versions and we can move forward.
> > 
> > There will be issues however, as newer versions of Xen won't have
> > support for PVHv1. My plan is the following in order to try to make
> > this less painful for users:
> > 
> >   - Wait until FreeBSD 12 is released, which will contain PVHv1 and
> > PVHv2 support.
> >   - Once FreeBSD 12 has been released, update the Xen port to the
> > latest version.
> 
> What about creating new port as xen410 (for version 4.10) or repocopy of the
> old one to xen47 to allow coexist of two different versions in the ports
> tree and allow user to choose the right one for their OS version?

I wondered about that, I will try to do it, but I have to admit my
time is quite limited and I'm not sure I will be able to keep both up
to data.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: HEADS UP: merged PVHv2 support and future plans

[Roger Pau Monné]

On Thu, Jul 19, 2018 at 11:42:24AM +0200, Gerd Hafenbrack wrote:
> Hello,
> 
> Roger Pau Monné  schrieb am Do., 19. Juli 2018, 11:05:
> 
> > ... Later on, we sadly discovered
> > that PVHv1 was still too similar to classic PV, and didn't allow Xen
> > to make use of all the possible hardware virtualization extensions, so
> > PVHv2 was introduced ~2 years ago ...
> >
> 
> Is there a page comparing hardware virtualization extensions between PVHv1
> and PVHv2?

Hm, not really. PVHv2 has some devices (either emulated by software or
hardware) that are not present on PVHv1: LAPIC, IO-APIC. The way to
setup and inject interrupts on PVHv2 is also the same as on native
(IO-APIC pin, MSI or MSI-X).

Also PVHv2 allows to change paging modes (or completely disable
paging), which PVHv1 didn't allow at all.

To sum up, PVHv2 is very, very similar to HVM but it doesn't require a
QEMU device model in order to run.

> What about AMD support? We are getting EPIC machines.

Yes, PVHv2 supports AMD, although I have to admit I haven't done
extensive testing on AMD due to my lack of AMD hardware. I expect
that's going to change soon.

I've tested PVHv2 FreeBSD DomUs on AMD hardware, but I haven't tested
FreeBSD PVHv2 Dom0 on AMD yet.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

HEADS UP: merged PVHv2 support and future plans

[Roger Pau Monné]

Hello,

Today I've merged PVHv2 support into FreeBSD, allowing FreeBSD to be
used as a PVHv2 DomU and Dom0. While it's not a huge set of changes,
I would *really* appreciate if people could test the code starting
from r336474 (or any later changeset).

I expect there's going to be some confusion with PVHv1 vs PVHv2, so I
will try to clarify this now. PVHv1 was introduced ~4 years ago, and
at the time it seemed like a good way to move forward, allowing Xen to
rely more on hardware virtualization. Later on, we sadly discovered
that PVHv1 was still too similar to classic PV, and didn't allow Xen
to make use of all the possible hardware virtualization extensions, so
PVHv2 was introduced ~2 years ago as a replacement for PVHv1. PVHv2
ABI however is not compatible with PVHv1, which means that different
entry points and interfaces must be used to interact with the
hypervisor.

After introducing PVHv2 PVHv1 was deprecated and PVHv1 has been
removed from the hypervisor in recent versions, that's why the Xen
ports package is still stuck with Xen 4.7, because later versions
removed PVHv1 support. With the addition of PVHv2 to FreeBSD the port
can be updated to newer Xen versions and we can move forward.

There will be issues however, as newer versions of Xen won't have
support for PVHv1. My plan is the following in order to try to make
this less painful for users:

 - Wait until FreeBSD 12 is released, which will contain PVHv1 and
   PVHv2 support.
 - Once FreeBSD 12 has been released, update the Xen port to the
   latest version.
 - Send an email to freebsd-xen notifying users of the bump in the
   package.
 - Users running FreeBSD < 12 must stick with the old Xen 4.7
   package.
 - Users running FreeBSD >= 12 must use the new Xen package.

I know this is not ideal, because users wanting to update Xen will
also need to update to FreeBSD 12, but I cannot think of any better
way to deal with this.

Lastly, the Xen CI test system (osstest) has recently gained support
for testing FreeBSD, which should make catching FreeBSD/Xen related
issues easier. This is an example of an initial FreeBSD test flight:

https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01680.html

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: gnttab_end_foreign_access_ref() leaking grant entries?

[Roger Pau Monné]

On Mon, Jun 18, 2018 at 06:11:00PM +0530, Pratyush Yadav wrote:
> Hi everyone,
> 
> I was looking at gnttab_end_foreign_access_ref() and I notice that
> while gnttab_end_foreign_access_ref() ends the foreign access, it does
> not free the grant reference.
> 
> gnttab_end_foreign_access() free the reference by calling put_free_entry(ref).
> 
> gnttab_end_foreign_access_references() also frees the grant entries.
> 
> Shouldn't gnttab_end_foreign_access_ref() also free the grant entry?
> It is an inconsistency at best and a bug at worst.

I think the point of using gnttab_end_foreign_access_ref is that it
doesn't free the reference. The naming convention of grant reference
related functions is just horrible, but I think this one is actually
clear, it just removes the foreign access permissions of this
reference, but it doesn't free it.

I think I would rather rename gnttab_end_foreign_access to
gnttab_end_foreign_access_and_free (and the same with the rest of the
end_foreign_ family of functions). This is more descriptive.

In any case, this is not a bug. Current netfront code makes use of
this function in order to keep a pool of grant references, so by
changing gnttab_end_foreign_access_ref to free the references you will
break netfront.

And now I realize this is something that you will have to take into
account for the busdma grant reference project. You will have to
introduce a flag that allows creating a dmamap with pre-allocated
references that are not freed until the map is destroyed. But let's
focus on that after the initial implementation is done.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Leftover code in gnttab.h?

[Roger Pau Monné]

On Thu, May 24, 2018 at 08:38:30PM +0530, Pratyush Yadav wrote:
> Hi,
> 
> On Thu, May 24, 2018 at 7:36 PM, Roger Pau Monné <roger@citrix.com> wrote:
> > On Thu, May 24, 2018 at 05:53:02PM +0530, Pratyush Yadav wrote:
> >> Hi everyone,
> >>
> >> I was reading the Xen grant table code and I noticed that some code is
> >> wrapped in an #if 0. You can see it in sys/xen/gnttab.h:138. I have
> >> also attached the "commented out" part down below.
> >>
> >> Is it useful? Is it all right if I submit a patch that removes it?
> >
> > This are leftovers from when the code was imported from Linux AFAICT.
> > I guess the original committer thought that those functions would be
> > implemented, so the prototypes where left in place.
> >
> > If you give me a patch to remove them I'm happy to commit it.
> 
> How should I send you the patch? Should I send it through
> reviews.freebsd.org, GitHub pull request or just paste the patch in
> the email?

Let's use reviews.freebsd.org, so that you can get used to it.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Leftover code in gnttab.h?

[Roger Pau Monné]

On Thu, May 24, 2018 at 05:53:02PM +0530, Pratyush Yadav wrote:
> Hi everyone,
> 
> I was reading the Xen grant table code and I noticed that some code is
> wrapped in an #if 0. You can see it in sys/xen/gnttab.h:138. I have
> also attached the "commented out" part down below.
> 
> Is it useful? Is it all right if I submit a patch that removes it?

This are leftovers from when the code was imported from Linux AFAICT.
I guess the original committer thought that those functions would be
implemented, so the prototypes where left in place.

If you give me a patch to remove them I'm happy to commit it.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen kernel fails to boot, d0v1 triple fault looks like the cuplrit

[Roger Pau Monné]

On Wed, May 23, 2018 at 08:27:29PM +0530, Pratyush Yadav wrote:
> Hi,
> 
> On Wed, May 23, 2018 at 1:45 PM, Roger Pau Monné <roger@citrix.com> wrote:
> > It's too early for the logs to be stored anywhere. The point where you
> > get the crash is when the APs are started, which is way before FreeBSD
> > starts proving for disk devices.
> >
> > Can you please try the patch below?
> >
> > Thanks, Roger.
> > ---8<---
> > diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c
> > index 54184898e9bf..52391e2e7c08 100644
> > --- a/sys/x86/xen/pv.c
> > +++ b/sys/x86/xen/pv.c
> > @@ -113,6 +113,7 @@ static int xen_pv_start_all_aps(void);
> >  extern char *doublefault_stack;
> >  extern char *mce_stack;
> >  extern char *nmi_stack;
> > +extern char *dbg_stack;
> >  #endif
> >
> >  /*
> > @@ -329,6 +330,8 @@ start_xen_ap(int cpu)
> > (char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
> > nmi_stack =
> > (char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
> > +   dbg_stack =
> > +   (void *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
> > dpcpu =
> > (void *)kmem_malloc(kernel_arena, DPCPU_SIZE, M_WAITOK | 
> > M_ZERO);
> >
> 
> I think we have different pv.c files. For me, line 113 is:
> /* Xen init_ops implementation. */
> 
> The declarations of doublefault_stach, mce_stack, etc are in line 101.
> 
> Similarly, line 329 for me is:
> {
> 
> in function xen_pv_parse_symtab(void). The declarations your diff
> mentions in line 329 are in line 224.
> 
> This is in sync with the official repository [0]. Maybe you have
> modifications that are not yet upstream?

Sorry, I did indeed have other changes in pv.c. I'm appending the
patch on top of current HEAD.

> Anyway, I manually made the changes. It still does not boot (I used
> make kernel -DKERNFAST, but I don't think that should make a
> difference).

FWIW, I think the recommended way is KERNFAST=1.

Can you paste the error? I think you should no longer get a triple
page fault in mp_machdep.c:307.

Thanks, Roger.
---8<---
diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c
index c7e97c5b2b63..27e98012302b 100644
--- a/sys/x86/xen/pv.c
+++ b/sys/x86/xen/pv.c
@@ -101,6 +101,7 @@ static int xen_pv_start_all_aps(void);
 extern char *doublefault_stack;
 extern char *mce_stack;
 extern char *nmi_stack;
+extern char *dbg_stack;
 #endif
 
 /*
@@ -224,6 +225,8 @@ start_xen_ap(int cpu)
(char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
nmi_stack =
(char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
+   dbg_stack =
+   (void *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
dpcpu =
(void *)kmem_malloc(kernel_arena, DPCPU_SIZE, M_WAITOK | M_ZERO);
 

___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen kernel fails to boot, d0v1 triple fault looks like the cuplrit

[Roger Pau Monné]

On Wed, May 23, 2018 at 12:57:59PM +0530, Pratyush Yadav wrote:
> On Mon, May 21, 2018 at 2:33 PM, Roger Pau Monné <roger@citrix.com>
> wrote:
> > Please try to avoid top posting.
> 
> Sorry, I didn't know. I Googled top posting just now, and realized it is
> inconvenient for the reader.
> 
> > Hm, it seems like dbg_stack is not properly allocated. Can you please
> > try the above debug patch and paste the boot log?
> >
> > ---8<---
> > diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c
> > index 301461420874..8854242b4bf5 100644
> > --- a/sys/amd64/amd64/mp_machdep.c
> > +++ b/sys/amd64/amd64/mp_machdep.c
> > @@ -304,6 +304,7 @@ init_secondary(void)
> >
> > /* Save the per-cpu pointer for use by the DB# handler. */
> > np = ((struct nmi_pcpu *) _stack[PAGE_SIZE]) - 1;
> > +printf("ID %d dbg_stack: %p per-cpu: %p\n", cpu, dbg_stack, np);
> > np->np_pcpu = (register_t) pc;
> >
> > wrmsr(MSR_FSBASE, 0);   /* User value */
> > @@ -403,6 +404,7 @@ native_start_all_aps(void)
> > M_WAITOK | M_ZERO);
> > dbg_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE,
> > M_WAITOK | M_ZERO);
> > +printf("allocated dbg_stack: %p\n", dbg_stack);
> > dpcpu = (void *)kmem_malloc(kernel_arena, DPCPU_SIZE,
> > M_WAITOK | M_ZERO);
> >
> 
> I applied your patch. The boot stops at the same spot. Also, I can't see
> any of the two print messages added in the patch. I booted from a Live CD
> to look at the logs, but they are not stored. I checked dmesg.boot and
> messages, and /var/log/xen/ none of them have the logs. They all have logs
> of the previous boot.

It's too early for the logs to be stored anywhere. The point where you
get the crash is when the APs are started, which is way before FreeBSD
starts proving for disk devices.

Can you please try the patch below?

Thanks, Roger.
---8<---
diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c
index 54184898e9bf..52391e2e7c08 100644
--- a/sys/x86/xen/pv.c
+++ b/sys/x86/xen/pv.c
@@ -113,6 +113,7 @@ static int xen_pv_start_all_aps(void);
 extern char *doublefault_stack;
 extern char *mce_stack;
 extern char *nmi_stack;
+extern char *dbg_stack;
 #endif
 
 /*
@@ -329,6 +330,8 @@ start_xen_ap(int cpu)
(char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
nmi_stack =
(char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
+   dbg_stack =
+   (void *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
dpcpu =
(void *)kmem_malloc(kernel_arena, DPCPU_SIZE, M_WAITOK | M_ZERO);
 
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Linux domU only works with xen_platform_pci=0 ?

[Roger Pau Monné]

On Mon, May 21, 2018 at 01:51:13PM -0600, Nathan Friess wrote:
> On 2018-05-19 02:02 AM, Roger Pau Monné wrote:
> > On Thu, May 17, 2018 at 06:10:15PM -0600, Nathan Friess wrote:
> > > I tried the patch on and I my Linux domU did was not able to complete the
> > > attachment to the FreeBSD backend.  I applied the patch to the 11-RELEASE
> > > kernel. (I couldn't get 11-STABLE to compile.)
> > > 
> > > With xen_platform_pci=0 the frontend and backend vbds were both stuck in
> > > state 1.  With xen_platform_pci=1 the frontend was in state 1 and backend 
> > > in
> > > state 3.
> > 
> > Thanks for the testing! I however think you had some issue applying
> > the patch or building/installing the kernel, with the attached patch
> > the backend should never go into state 3.
> > 
> > Can you confirm that you have the patch correctly applied to the
> > kernel you are running?
> > 
> > And if so, can you try to debug why the backend goes into state 3?
> > 
> > FWIW, the patch works fine for me and I've been able to boot Debian
> > and Alpine Linux guests without having to add xen_platform_pci=0.
> > 
> > Thanks, Roger.
> > 
> 
> Sorry, I don't which run I was looking at there.  Here is the order of
> events.  There are two zvols exposed to the domU,
> /dev/zvol/zroot/vmachines/smyth/rootfs and .../swap, so the messages are
> interspersed between the two.
> 
> xbb(xbb_attach:3738): Attaching to backend/vbd/10/51713
> xbb(xbb_attach:3802): Attach done, set state InitWait
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_attach_disk:3620):   Read physical-device-path failed
> xbb(xbb_frontend_changed:3918): frontend_state=Initialising,
> xbb_state=InitWait
> xbb(xbb_attach:3738): Attaching to backend/vbd/10/51714
> xbb(xbb_attach:3802): Attach done, set state InitWait
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_attach_disk:3620):   Read physical-device-path failed
> xbb(xbb_frontend_changed:3918): frontend_state=Initialising,
> xbb_state=InitWait
> (Shell script) Start block script /local/domain/9/backend/vbd/10/51714 add
> (Shell script) Start block script /local/domain/9/backend/vbd/10/51713 add
> [1] xbb(xbb_frontend_changed:3918): frontend_state=Initialised,
> xbb_state=InitWait
> xbb(xbb_connect:3316): Enter xbb_connect: hotplug=0, get_state=2,
> collect_frontend=0
> [1] xbb(xbb_frontend_changed:3918): frontend_state=Initialised,
> xbb_state=InitWait
> xbb(xbb_connect:3316): Enter xbb_connect: hotplug=0, get_state=2,
> collect_frontend=0
> (Shell script) Write /dev/zvol/zroot/vmachines/smyth/rootfs to
> /local/domain/9/backend/vbd/10/51713/physical-device-path
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_open_backend:2687): opening
> dev=/dev/zvol/zroot/vmachines/smyth/rootfs
> xbb(xbb_open_backend:2757): opened
> dev=/dev/zvol/zroot/vmachines/smyth/rootfs sector_size=512
> media_size=21474836480
> xbb(xbb_attach_disk:3718): Set hotplug done, attach_disk done
> (Shell script) Write /dev/zvol/zroot/vmachines/smyth/swap to
> /local/domain/9/backend/vbd/10/51714/physical-device-path
> xbb(xbb_attach_disk:3612): Enter xbb_attach_disk
> xbb(xbb_open_backend:2687): opening dev=/dev/zvol/zroot/vmachines/smyth/swap
> xbb(xbb_open_backend:2757): opened dev=/dev/zvol/zroot/vmachines/smyth/swap
> sector_size=512 media_size=2147483648
> xbb(xbb_attach_disk:3718): Set hotplug done, attach_disk done
> 
> ... is stuck here...
> 
> At [1] the frontend has transitioned to Initialised but the block script has
> not completed yet so hotplug_done is 0 and the notification about the
> frontend status change is lost.  Calling xbb_connect at the end of
> xbb_attach_disk like in the attached patch will catch this case. Now with
> your recent change and the attached file I can use Linux domUs successfully!

Your Linux kernel is certainly very fast to boot, so that it wins the
race with blkback processing the result from the hotplug script
execution.

I've now committed the fix to HEAD, will MFC in a week if everything
goes fine:

https://svnweb.freebsd.org/base?view=revision=334027

TYVM for the testing and the fix.

> On a related note, did these patches ever make it into 11-stable?  I don't
> see them at svn.freebsd.org/base/stable/11 but I may have missed something.
> 
> https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002918.html

I don't think they are on HEAD either?

I don't have a lot of time ATM, so if you want to pick the patches,
rebase them onto HEAD, test them and give me a branch I'm more than
happy to review and commit them.

> There were 4 patches, the third one being required for x

Re: Xen kernel fails to boot, d0v1 triple fault looks like the cuplrit

[Roger Pau Monné]

Please try to avoid top posting.

On Sat, May 19, 2018 at 03:45:41PM +0530, Pratyush Yadav wrote:
> Hi,
> 
> The line is
> 
> /usr/src/sys/amd64/amd64/mp_machdep.c:307

Hm, it seems like dbg_stack is not properly allocated. Can you please
try the above debug patch and paste the boot log?

---8<---
diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c
index 301461420874..8854242b4bf5 100644
--- a/sys/amd64/amd64/mp_machdep.c
+++ b/sys/amd64/amd64/mp_machdep.c
@@ -304,6 +304,7 @@ init_secondary(void)
 
/* Save the per-cpu pointer for use by the DB# handler. */
np = ((struct nmi_pcpu *) _stack[PAGE_SIZE]) - 1;
+printf("ID %d dbg_stack: %p per-cpu: %p\n", cpu, dbg_stack, np);
np->np_pcpu = (register_t) pc;
 
wrmsr(MSR_FSBASE, 0);   /* User value */
@@ -403,6 +404,7 @@ native_start_all_aps(void)
M_WAITOK | M_ZERO);
dbg_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE,
M_WAITOK | M_ZERO);
+printf("allocated dbg_stack: %p\n", dbg_stack);
dpcpu = (void *)kmem_malloc(kernel_arena, DPCPU_SIZE,
M_WAITOK | M_ZERO);
 
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Xen kernel fails to boot, d0v1 triple fault looks like the cuplrit

[Roger Pau Monné]

On Fri, May 18, 2018 at 05:29:06PM +0530, Pratyush Yadav wrote:
> Hi,
> 
> So I have been trying to get Xen to boot on my laptop. It is getting
> stuck with the following error messages:
> 
> (XEN) Scrubbing free RAM on 1 nodes using 4 CPUs
> (XEN) [VT-D] DMAR: [DMA Read} Request device [:00:1a.0] fault addr
> 9ce13000. iommu reg - 82c000203000
> (XEN) [VT-D] DMAR: reason 06 - PTE read access is not set
> (XEN) ... done
> (XEN) initial low memory virq threshold set at 0x200 pages.
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) Xen is keeping VGA console
> (XEN) Boot video device 00:02.0
> (XEN) ***Serial input -> DOM0 (type 'CTRL-a' three times to switch input to 
> Xen)
> (XEN) Freed 320 kB init memory
> FreeBSD PVH running on xen-3.0-x86_64p
> Copyright (c) 1992-2018 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
> 1994 The regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 12.0-CURRENT #0 r333606: Mon May 14 19:59:08 UTC 2018
> r...@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC 
> amd64
> WARNING: WITNESS option enabled, expect reduced performance.
> VT(vga): text 80x25
> CPU: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz (2494.22-MHz K8-class CPU)
> Origin="GenuineIntel" Id=0x306c3 Family=0x6 Model=0x3c Stepping=3
> 
> Features=0x17c1cbf5 FXSR,SSE,SSE2,HTT>
> 
> Features2=0xf6f83203 POPCNT,AESNI,XSAVE,AVX,F16C,RDRAND,HV>
> AMD Features=0x20101800
> AMD Features2=0x21<
> Structured Extended Features=0x2329
> Hypervisor: Origin = "XenVMMXenVMM"
> real memory  = 9991770112 (9528 MB)
> avail memory = 7936724992 (7569 MB)
> (XEN) d0v1 Triple fault - invoking HVM shutdown action 0
> (XEN) *** Dumping Dom0 vcpu#1 state: ***
> (XEN) [ Xen-4.7.2 x86_64 debug=n Not tainted ]
> (XEN) CPU:1
> (XEN) RIP:0020:[]
> (XEN) RFLAGS:  00010016CONTEXT: hvm guest (d0v1)
> (XEN) rax:    rbx: 8201c100   rcx: 0001
> (XEN) rdx: 3078   rsi: 81b992f8   rdi: fe0003bb6078
> (XEN) rbp: fe90   rsp: fe9fffa0   r8: 
> (XEN) r15: 0400   cr0: 0011   cr4: 0020
> (XEN) cr3: 035a4000   cr2: 0ff0
> (XEN) ds: 0028   es: 0028   fs: 0028   gs: 0028   ss: 0028   cs: 0020
> (XEN) Guest stack trace from rsp=fe9fffa0:
> (XEN)   Fault while accessing guest memory.
> (XEN) Hardware Dom0 halted: halting machine
> 
> Any idea why this is happening and how can I fix this?

Can you execute:

addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 8102042b

This should print a file and line number that would help in order to
debug what's going on.

Since it seems like the crash is caused by a triple fault on an AP,
you can try to boot with dom0_max_vcpus=1 on the xen_cmdline, that
might prevent the panic from happening.

We need however to figure out what's going on and fix it.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Linux domU only works with xen_platform_pci=0 ?

[Roger Pau Monné]

On Thu, May 17, 2018 at 06:10:15PM -0600, Nathan Friess wrote:
> On 2018-05-15 02:08 AM, Roger Pau Monné wrote:
> > On Mon, May 14, 2018 at 08:34:54PM -0600, Nathan Friess wrote:
> > > 
> > > I had similar issues with Linux domUs being unable to detect their disks
> > > when FreeBSD 11.1-RELEASE is the backend:
> > > 
> > > https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002924.html
> > > 
> > > 
> > > What seems to be happening is that on my system the frontend and backend 
> > > may
> > > go from state InitWait to Initialised in different orders and so either 
> > > end
> > > may end up getting stuck waiting for the other side to change state when 
> > > the
> > > other side already has done so.
> > > 
> > > I have been running with the attached patch since my last message above 
> > > and
> > > Linux domUs have been working perfectly since then.  I realize that the 
> > > call
> > > to pause() is not the correct solution but it demonstrates that some fine
> > > tuning of how the states are handled will help.
> > 
> > Can you please give a try to the patch at:
> > 
> > https://lists.freebsd.org/pipermail/freebsd-xen/2018-May/003132.html
> > 
> > I think this is the proper way to solve the issue, and I would like to
> > commit it ASAP in order to MFC it to stable-11 before 11.2 is
> > released, but it could benefit from some more testing.
> > 
> > Thanks, Roger.
> > 
> 
> 
> I tried the patch on and I my Linux domU did was not able to complete the
> attachment to the FreeBSD backend.  I applied the patch to the 11-RELEASE
> kernel. (I couldn't get 11-STABLE to compile.)
> 
> With xen_platform_pci=0 the frontend and backend vbds were both stuck in
> state 1.  With xen_platform_pci=1 the frontend was in state 1 and backend in
> state 3.

Thanks for the testing! I however think you had some issue applying
the patch or building/installing the kernel, with the attached patch
the backend should never go into state 3.

Can you confirm that you have the patch correctly applied to the
kernel you are running?

And if so, can you try to debug why the backend goes into state 3?

FWIW, the patch works fine for me and I've been able to boot Debian
and Alpine Linux guests without having to add xen_platform_pci=0.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it possible to run Xen in a VirtualBox VM?

[Roger Pau Monné]

On Tue, May 15, 2018 at 11:37:43AM +0530, Pratyush Yadav wrote:
> On Mon, May 14, 2018 at 11:25 PM, Roger Pau Monné <roger@citrix.com> 
> wrote:
> > On Mon, May 14, 2018 at 10:49:43PM +0530, Pratyush Yadav wrote:
> >> I am working on the Xen grant table handlers (for my Google Summer of
> >> Code project [0]) , so do you think I would need to run Xen + FreeBSD
> >> when testing?
> >
> > Yes, I think you will need to be able to run Xen + FreeBSD. You can
> > probably manage to complete the first part using Xen + Linux and
> > running FreeBSD as a guest, but you will need Xen + FreeBSD Dom0 for
> > the second part (adding handlers for mapping operations used by the
> > backend).
> >> Also, how can I fix the error I'm currently getting about Intel
> >> processor not being supported?
> >
> > Which error? Can you please paste the full log?
> >
> > Roger.
> 
> Ah, my bad. I misread your message. I thought you were saying we can
> run Xen + FreeBSD Dom0 but not a FreeBSD DomU.
> 
> Anyway, I can't get the logs because they are not outputting to the
> serial port for some reason. I wrote what I could read from the 5
> seconds I get before the reboot in my first email.

You can set noreboot [0] in xen_cmdline in order to prevent Xen from
rebooting automatically. That should give you more time to read the
error message.

> You answered my
> question, but if you want a look at the error message I could read,
> check my first email. I'll also paste it below [0].
> 
> Thanks for your help.
> 
> -- 
> Regards,
> Pratyush Yadav
> 
> [0]:
> xenoprof: Initialization failed. Intel processor family 6 model 60 is
> not supported.
> Dom0 has maximum 600 PIRQs
> 
> ***
> Panic on cpu 0:
> Error creating domain 0
> ***

This seems to be missing some lines. A panic usually contains a
message with the reason of the panic.

Roger.

[0] http://xenbits.xenproject.org/docs/unstable/misc/xen-command-line.html
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it possible to run Xen in a VirtualBox VM?

[Roger Pau Monné]

On Tue, May 15, 2018 at 07:06:49AM +, Marcin Cieslak wrote:
> On Tue, 15 May 2018, Pratyush Yadav wrote:
> 
> > On Mon, May 14, 2018 at 11:25 PM, Roger Pau Monné <roger@citrix.com> 
> > wrote:
> > > Yes, I think you will need to be able to run Xen + FreeBSD. You can
> > > probably manage to complete the first part using Xen + Linux and
> > > running FreeBSD as a guest, but you will need Xen + FreeBSD Dom0 for
> > > the second part (adding handlers for mapping operations used by the
> > > backend).
> > 
> > One more thing, is there any other VM like VirtualBox that can run Xen
> > + FreeBSD as Dom0, or do I have to run it on a different computer.
> 
> I have solved this problem for me by renting a physical server at the hosting 
> company.
> But serious hacking requires having access to the physical console (most
> hosting providers provide something like that).
> 
> I don't know how others are working on this? Anyone running Xen on their 
> laptop for example?

Xen also supports printing to a USB debug port (EHCI debug port) [0],
but some laptops don't even have the USB debug port accessible, and
then you need a special adapter which is impossible to find nowadays.

So, the easier way to debug is to get a box with SOL or a working
serial DB9 port.

Roger.

[0] https://www.coreboot.org/EHCI_Debug_Port
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Linux domU only works with xen_platform_pci=0 ?

[Roger Pau Monné]

On Mon, May 14, 2018 at 08:34:54PM -0600, Nathan Friess wrote:
> On 2018-05-14 07:04 AM, Roger Pau Monné wrote:
> > On Sun, May 13, 2018 at 07:33:03PM +0200, Kai Otto wrote:
> > > On 13/05/18 17:16, Roger Pau Monné wrote:
> > > > On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
> > > > > Hello,
> > > > > 
> > > > > I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> > > > > Following a combination of handbook [1] and wiki [2], I was able to 
> > > > > get
> > > > > get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> > > > > including installation to disk and networking.
> > > > > 
> > > > > It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> > > > > effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> > > > > matter if I set it to 1 or 0.
> > > > > 
> > > > > Do I understand it correctly, that this switch makes the difference
> > > > > between HVM and PVHVM mode?
> > > > > According to xl.cfg(5), it 'enables a guest Operating System [...] to
> > > > > make use of paravirtualization features such as disk and network 
> > > > > devices'.
> > > > > 
> > > > > 
> > > > > Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> > > > > Linux only detected the harddisk with xen_platform_pci=0.
> > > > > 
> > > > > For Alpine Linux, with xen_platform_pci=1, I get the following 
> > > > > messages
> > > > > on the console:
> > > > > 
> > > > > vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
> > > > That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
> > > > backend.
> > > > 
> > > > > vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> > > > > xenbus_dev_probe on device/vbd/5632)
> > > > > 
> > > > > After waiting for a couple minutes it boots, but doesn't detect the 
> > > > > disk.
> 
> I had similar issues with Linux domUs being unable to detect their disks
> when FreeBSD 11.1-RELEASE is the backend:
> 
> https://lists.freebsd.org/pipermail/freebsd-xen/2016-December/002924.html
> 
> 
> What seems to be happening is that on my system the frontend and backend may
> go from state InitWait to Initialised in different orders and so either end
> may end up getting stuck waiting for the other side to change state when the
> other side already has done so.
> 
> I have been running with the attached patch since my last message above and
> Linux domUs have been working perfectly since then.  I realize that the call
> to pause() is not the correct solution but it demonstrates that some fine
> tuning of how the states are handled will help.

Can you please give a try to the patch at:

https://lists.freebsd.org/pipermail/freebsd-xen/2018-May/003132.html

I think this is the proper way to solve the issue, and I would like to
commit it ASAP in order to MFC it to stable-11 before 11.2 is
released, but it could benefit from some more testing.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it possible to run Xen in a VirtualBox VM?

[Roger Pau Monné]

On Mon, May 14, 2018 at 10:49:43PM +0530, Pratyush Yadav wrote:
> On Mon, May 14, 2018 at 10:30 PM, Roger Pau Monné <roger@citrix.com> 
> wrote:
> > On Mon, May 14, 2018 at 10:25:21PM +0530, Pratyush Yadav wrote:
> >> Hi,
> >>
> >> I tried running Xen in a VirtualBox guest and it errors in the boot
> >> process giving this error:
> >>
> >> xenoprof: Initialization failed. Intel processor family 6 model 60 is
> >> not supported.
> >> Dom0 has maximum 600 PIRQs
> >>
> >> ***
> >> Panic on cpu 0:
> >> Error creating domain 0
> >> ***
> >>
> >> So, is it even possible to run Xen in a VM? If it is, how can I fix
> >> this problem?
> >
> > It's likely possible to run Xen + Linux Dom0 inside of VirtualBox, but
> > it's almost certainly not possible to run Xen + FreeBSD inside of
> > VirtualBox because it would require an emulated IOMMU, which I don't
> > think VirtualBox provides.
> 
> I am working on the Xen grant table handlers (for my Google Summer of
> Code project [0]) , so do you think I would need to run Xen + FreeBSD
> when testing?

Yes, I think you will need to be able to run Xen + FreeBSD. You can
probably manage to complete the first part using Xen + Linux and
running FreeBSD as a guest, but you will need Xen + FreeBSD Dom0 for
the second part (adding handlers for mapping operations used by the
backend).

> Also, how can I fix the error I'm currently getting about Intel
> processor not being supported?

Which error? Can you please paste the full log?

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Is it possible to run Xen in a VirtualBox VM?

[Roger Pau Monné]

On Mon, May 14, 2018 at 10:25:21PM +0530, Pratyush Yadav wrote:
> Hi,
> 
> I tried running Xen in a VirtualBox guest and it errors in the boot
> process giving this error:
> 
> xenoprof: Initialization failed. Intel processor family 6 model 60 is
> not supported.
> Dom0 has maximum 600 PIRQs
> 
> ***
> Panic on cpu 0:
> Error creating domain 0
> ***
> 
> So, is it even possible to run Xen in a VM? If it is, how can I fix
> this problem?

It's likely possible to run Xen + Linux Dom0 inside of VirtualBox, but
it's almost certainly not possible to run Xen + FreeBSD inside of
VirtualBox because it would require an emulated IOMMU, which I don't
think VirtualBox provides.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Linux domU only works with xen_platform_pci=0 ?

[Roger Pau Monné]

On Sun, May 13, 2018 at 07:33:03PM +0200, Kai Otto wrote:
> On 13/05/18 17:16, Roger Pau Monné wrote:
> > On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
> >> Hello,
> >>
> >> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> >> Following a combination of handbook [1] and wiki [2], I was able to get
> >> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> >> including installation to disk and networking.
> >>
> >> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> >> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> >> matter if I set it to 1 or 0.
> >>
> >> Do I understand it correctly, that this switch makes the difference
> >> between HVM and PVHVM mode?
> >> According to xl.cfg(5), it 'enables a guest Operating System [...] to
> >> make use of paravirtualization features such as disk and network devices'.
> >>
> >>
> >> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> >> Linux only detected the harddisk with xen_platform_pci=0.
> >>
> >> For Alpine Linux, with xen_platform_pci=1, I get the following messages
> >> on the console:
> >>
> >> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
> > That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
> > backend.
> >
> >> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> >> xenbus_dev_probe on device/vbd/5632)
> >>
> >> After waiting for a couple minutes it boots, but doesn't detect the disk.
> >>
> >> Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
> >> messages:
> >>
> >> xen be core: can't open gnttab device
> >> xen be core: can't open gnttab device
> >> xen be: vkbd-0: initalize() failed
> >> xen be: vkbd-0: initalize() failed
> >> xen be: vkbd-0: initalize() failed
> >>
> >>
> >>
> >> Googling around yielded [3], where someone apparently ran into the same
> >> problem, and used xen_platform_pci=0 as workaround.
> >>
> >>
> >>
> >> So my question is:
> >> Is xen_platform_pci=1 required for PVHVM mode?
> > No, it shouldn't be.
> >
> >> If yes, is PVHVM mode only available for FreeBSD domU's?
> >> If no, how can I enable it for Linux guests?
> > If you boot FreeBSD Dom0 kernel with boot_verbose=YES (in
> > /boot/loader.conf), can you paste the messages you get when trying to
> > boot the PVHVM guest with xen_platform_pci=1?
> >
> > Also, can you paste the output of `xenstore-ls -fp` executed on Dom0
> > after the PVHVM guest has failed to boot but is still running?
> >
> Hello again,
> thanks for your response Roger!
> 
> I attached both. Because I don't know how the mailinglist handles
> attachments, here is a pastebin of xenstore-ls:
> https://pastebin.com/s7wa1ee7
> And a direct paste of dmesg messages:
> 
> random: harvesting attach, 8 bytes (4 bits) from xbbd0
> tap0: bpf attached
> tap0: Ethernet address: 00:bd:ee:99:ff:00
> tap0: link state changed to UP
> tap0: changing name to 'xnb2.0-emu'
> xnb(xnb_probe:1127): Claiming device 0, xnb
> xnb2.0: bpf attached
> xnb(xnb_attach:1271): Attaching to backend/vif/2/0
> random: harvesting attach, 8 bytes (4 bits) from xnb0
> xnb(xnb_frontend_changed:1395): frontend_state=Initialising,
> xnb_state=InitWait
> xnb2.0: link state changed to DOWN
> xnb2.0: link state changed to UP
> xnb2.0: link state changed to DOWN
> xnb2.0: promiscuous mode enabled
> xnb2.0: link state changed to UP
> xnb2.0-emu: promiscuous mode enabled
> nd6_dad_timer: cancel DAD on xnb2.0 because of ND6_IFF_IFDISABLED.
> xnb(xnb_frontend_changed:1395): frontend_state=Connected, xnb_state=InitWait
> xnb(xnb_connect_comms:788): rings connected!

Can you please give a try to the following patch? I think it should
apply cleanly to stable/11 (11.1-RELEASE). You will have to rebuild
and install the kernel.


Thanks, Roger.
---8<---
diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
index 9744c8da73cb..e9c7afe7d18c 100644
--- a/sys/dev/xen/blkback/blkback.c
+++ b/sys/dev/xen/blkback/blkback.c
@@ -806,6 +806,9 @@ struct xbb_softc {
 
/** Watch to wait for hotplug script execution */
struct xs_watch   hotplug_watch;
+
+   /** Got the needed data from hotplug scripts? */
+   bool  hotplug_done;
 };
 
 /* Request Processing 
*/
@@ -3310,10 +3313,9 

Re: Linux domU only works with xen_platform_pci=0 ?

[Roger Pau Monné]

On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
> Hello,
> 
> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> Following a combination of handbook [1] and wiki [2], I was able to get
> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> including installation to disk and networking.
> 
> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> matter if I set it to 1 or 0.
> 
> Do I understand it correctly, that this switch makes the difference
> between HVM and PVHVM mode?
> According to xl.cfg(5), it 'enables a guest Operating System [...] to
> make use of paravirtualization features such as disk and network devices'.
> 
> 
> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> Linux only detected the harddisk with xen_platform_pci=0.
> 
> For Alpine Linux, with xen_platform_pci=1, I get the following messages
> on the console:
> 
> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632

That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
backend.

> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> xenbus_dev_probe on device/vbd/5632)
> 
> After waiting for a couple minutes it boots, but doesn't detect the disk.
> 
> Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
> messages:
> 
> xen be core: can't open gnttab device
> xen be core: can't open gnttab device
> xen be: vkbd-0: initalize() failed
> xen be: vkbd-0: initalize() failed
> xen be: vkbd-0: initalize() failed
> 
> 
> 
> Googling around yielded [3], where someone apparently ran into the same
> problem, and used xen_platform_pci=0 as workaround.
> 
> 
> 
> So my question is:
> Is xen_platform_pci=1 required for PVHVM mode?

No, it shouldn't be.

> If yes, is PVHVM mode only available for FreeBSD domU's?
> If no, how can I enable it for Linux guests?

If you boot FreeBSD Dom0 kernel with boot_verbose=YES (in
/boot/loader.conf), can you paste the messages you get when trying to
boot the PVHVM guest with xen_platform_pci=1?

Also, can you paste the output of `xenstore-ls -fp` executed on Dom0
after the PVHVM guest has failed to boot but is still running?

> Any response would be appreciated, thanks!
> 
> (Unrelated: I also couldn't get PVH guests to boot, neither FreeBSD nor
> Linux. The console stayed blank and CPU went to 100%. But that's a whole
> other story I guess.)

FreeBSD Dom0 runs in PVH mode, so you are indeed able to boot at least
one PVH guest :).

Let's look into the PVHVM issue first.

> My system:
> 
> OS: FreeBSD-11.1-RELEASE amd64
> xen installed via pkg
> Processor: E3-1240 v2, (which supports VT-x, VT-d and EPT)
> 
> 
> My alpine.cfg:
> 
> builder="hvm"
> name="alpine-hvm"
> memory=1024
> vcpus=1
> vif = [ 'bridge=bridge0' ]
> disk = [ '/dev/zvol/zbulk/vm/alpine-hvm_disk0,raw,hda,rw',
>  '/zbulk/mediashare/isos/linux/apline-virt-3.7.0-x86_64.iso,raw,hdb:cdrom,r']
> 
> xen_platform_pci=1 # or 0, see above
> vnc = 1
> vnclisten = "0.0.0.0"
> serial = "pty"
> usbdevice = "tablet"

Your domain config file seems OK to me.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [Bug 224003] xen kernel panics

[Roger Pau Monné]

On Sat, Mar 31, 2018 at 07:38:28PM -0700, Rodney W. Grimes wrote:
> > On 03/31/18 18:11, Chris wrote:
> > > On 03/30/18 18:05, Rodney W. Grimes wrote:
> > >
> > >> I can affirm that a E56xx cpu should be very capable of supporting Xen.
> > >> CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz (3059.07-MHz K8-class CPU)
> > >> Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
> > >> Features=0xbfebfbff
> > >>
> > >> Features2=0x29ee3ff
> > >>
> > >> AMD Features=0x2c100800
> > >> AMD Features2=0x1
> > >> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
> > >> TSC: P-state invariant, performance statistics
> > >>
> > 
> > Looks like this might be the problem:-
> > 
> > > (XEN) [VT-D]Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata
> > > #47, #53
> > 
> > Been in a rush today, but should have looked at that a bit more closely
> > before posting, as istr, saw that  mentioned as a bug eleswhere back
> > from 2015 or so
> 
> https://support.citrix.com/article/CTX136517
> 
> That has some info on how to work around this issue.  You can turn
> of the interrupt remapping and still have an iommu, though it is
> not going be very effective for passthrough of devices.

Just add iommu=no-intremap to your xen_cmdline in loader.conf.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [Bug 224003] xen kernel panics

[Roger Pau Monné]

On Fri, Mar 30, 2018 at 05:43:21PM +, Chris wrote:
> On 03/30/18 12:07, Roger Pau Monné wrote:
> > On Thu, Mar 29, 2018 at 11:32:19PM +, Chris wrote:
> > > I'm having similar problems with xen on 11.1, AMD64, June 2017.
> > > The OS runs fine, but installed xen from package, setup exactly
> > > as per the handbook and get a kernel panic on two machines,
> > > complaining about iommu not being enabled.
> > 
> > FreeBSD/Xen Dom0 requires a working IOMMU, that's documented in the
> > handbook [0] section 21.8.1.
> > 
> > Can you paste the full output that you get when booting under Xen?
> > 
> > > Machines are: a Sun X4170 to start, then  a Proliant DL380 G7
> > > with E5630 cpu. Both have all the virtualisation options
> > > enabled in the bios, but there are no options on either machine
> > >   for iommu.
> > 
> > On Intel hardware the IOMMU is called VT-d. I have no idea if the
> > hardware that you list has an IOMMU, it depends on both the CPU and
> > the motherboard.
> > 
> > Without VT-d (an IOMMU) FreeBSD/Xen Dom0 won't work.
> > 
> > Roger.
> > 
> > [0] https://www.freebsd.org/doc/handbook/virtualization-host-xen.html
> > .
> > 
> 
> Roger,
> 
> Thanks for the reply and clarification on the meaning of iommu. A bit
> more info on the machine:
> 
> Cpu is actually an E5645, hex core, 2.4GHz
> 32 Gb ram, full ecc
> bios is dated 2011, with:
> 
> VT-d, enabled
> Intel virtualisation tech, enabled
> 
> It's difficult to log info on this, as there is just the "needs iommu"
> message on boot, then halt. Live cd and file edits brought the base
> system back, but nothing in /var/log/xen at all.

You can also boot into FreeBSD (without Xen) by dropping into the
loader prompt and typing:

> unload
> unset xen_kernel

I'm afraid you will need to connect to the server using a serial
console and get the output from there. Without the full serial output
it's hard to figure out what's going on.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: [Bug 224003] xen kernel panics

[Roger Pau Monné]

On Thu, Mar 29, 2018 at 11:32:19PM +, Chris wrote:
> I'm having similar problems with xen on 11.1, AMD64, June 2017.
> The OS runs fine, but installed xen from package, setup exactly
> as per the handbook and get a kernel panic on two machines,
> complaining about iommu not being enabled.

FreeBSD/Xen Dom0 requires a working IOMMU, that's documented in the
handbook [0] section 21.8.1.

Can you paste the full output that you get when booting under Xen?

> Machines are: a Sun X4170 to start, then  a Proliant DL380 G7
> with E5630 cpu. Both have all the virtualisation options
> enabled in the bios, but there are no options on either machine
>  for iommu.

On Intel hardware the IOMMU is called VT-d. I have no idea if the
hardware that you list has an IOMMU, it depends on both the CPU and
the motherboard.

Without VT-d (an IOMMU) FreeBSD/Xen Dom0 won't work.

Roger.

[0] https://www.freebsd.org/doc/handbook/virtualization-host-xen.html
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: multi-vCPU networking issues as client OS under Xen

[Roger Pau Monné]

On Mon, Feb 19, 2018 at 10:42:08AM +, Laurence Pawling wrote:
> >When using >1 vCPUs can you set hw.xn.num_queues=1 on
> >/boot/loader.conf and try to reproduce the issue?
> >
> >I'm afraid this is rather related to multiqueue (which is only used
> >if >1 vCPUs).
> >
> >Thanks, Roger.
> 
> Roger - thanks for your quick reply, this is confirmed. Setting 
> hw.xn.num_queues=1 on the server VM when vCPUs > 1 prevents the issue.

I've also been told that in order to discard this being a XenServer
specific issue you should execute the following on Dom0 and reboot the
server:

# xe-switch-network-backend bridge

And then try to reproduce the issue again with >1 vCPUs (and of course
removing the queue limit in loader.conf)

> For reference, please can you comment on the performance impact of this?

I'm afraid I don't have any numbers.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: multi-vCPU networking issues as client OS under Xen

[Roger Pau Monné]

On Mon, Feb 19, 2018 at 09:58:30AM +, Laurence Pawling via freebsd-xen 
wrote:
> Hi all,
> 
>  
> 
> I’m wondering if anyone here has seen this issue before, I’ve spent the last 
> couple of days troubleshooting:
> 
>  
> 
> Platform:
> 
> Host: XenServer 7.0 running on 2 x E2660-v4, 256GB RAM
> 
> Server VM: FreeBSD 11 (tested on 11.0-p15 and 11.1-p6), 2GB RAM (also tested 
> with 32GB RAM), 1x50GB HDD, 1 x NIC, 2 or more vCPUs in any combination (2 
> sockets x 1 core, 1 socket x 2 cores, …)
> 
> Client VM: FreeBSD 11, any configuration of vCPUs, RAM and HDD.
> 
>  
> 
> Behaviour:
> 
> Sporadic interruption of TCP sessions when utilising the above machine as a 
> “server” with “clients” connecting. Looking into the communication with 
> pcap/Wireshark, you see a TCP Dup Ack sent from both ends, followed by the 
> client sending an RST packet, terminating the TCP session. We have also seen 
> evidence of the client sending a Keepalive packet, which is ACK’d by the 
> server before the RST is sent from the client end.
> 
>  
> 
> To recreate:
> 
> On the above VM, perform a vanilla install of nginx:
> 
> pkg install nginx
> 
> service nginx onestart
> 
> Then on a client VM (currently only tested with FreeBSD), run the following 
> (or similar):
> 
> for i in {1..1}; do if [ $(curl -s -o /dev/null -w "%{http_code}" 
> http://10.2.122.71) != 200 ] ; then echo "error"; fi; done
> 
> When vCPUs=1 on the server, I get no errors, when vCPUs>1 I get errors 
> reported. The frequency of errors *seems* to be proportional to the number of 
> vCPUs, but they are sporadic with no clear periodicity or pattern, so that is 
> just anecdotal. Also, the problem seems by far the most prevalent when 
> communicating between two VMs on the same host, in the same VLAN. Xen still 
> sends packets via the switch rather than bridging internally between the 
> interfaces.

When using >1 vCPUs can you set hw.xn.num_queues=1 on
/boot/loader.conf and try to reproduce the issue?

I'm afraid this is rather related to multiqueue (which is only used
if >1 vCPUs).

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Storage 'failover' largely kills FreeBSD 10.x under XenServer?

[Roger Pau Monné]

On Wed, Sep 20, 2017 at 11:35:26AM +0100, Karl Pielorz wrote:
> 
> Hi All,
> 
> We recently experienced an "unplanned storage" fail over on our XenServer
> pool. The pool is 7.1 based (on certified HP kit), and runs a mix of FreeBSD
> (all 10.3 based except for a legacy 9.x VM) - and a few Windows VM's -
> storage is provided by two Citrix certified Synology storage boxes.
> 
> During the fail over - Xen see's the storage paths go down, and come up
> again (re-attaching when they are available again). Timing this - it takes
> around a minute, worst case.
> 
> The process killed 99% of our FreeBSD VM's :(
> 
> The earlier 9.x FreeBSD box survived, and all the Windows VM's survived.
> 
> Is there some 'tuneable' we can set to make the 10.3 boxes more tolerant of
> the I/O delays that occur during a storage fail over?

Do you know whether the VMs saw the disks disconnecting and then
connecting again?

> I've enclosed some of the error we observed below. I realise a full storage
> fail over is a 'stressful time' for VM's - but the Windows VM's, and earlier
> FreeBSD version survived without issue. All the 10.3 boxes logged I/O
> errors, and then panic'd / rebooted.
> 
> We've setup a test lab with the same kit - and can now replicate this at
> will (every time most to all the FreeBSD 10.x boxes panic and reboot, but
> Windows prevails) - so we can test any potential fixes.
> 
> So if anyone can suggest anything we can tweak to minimize the chances of
> this happening (i.e. make I/O more timeout tolerant, or set larger
> timeouts?) that'd be great.

Hm, I have the feeling that part of the problem is that in-flight
requests are basically lost when a disconnect/reconnect happens.

Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Fwd: FreeBSD 11.1 xen trying to create linux domU instance

[Roger Pau Monné]

On Wed, Aug 23, 2017 at 03:49:09PM +0100, Roger Pau Monné wrote:
> On Tue, Aug 22, 2017 at 09:51:47AM -0700, James E. Pace wrote:
> > >> usbdevice=['tablet']
> > >>
> > >
> > This didn't change the behavior; mouse still does not move.  Any other
> > thoughts?
> 
> It works for me, weird.

Forgot to mention, I use tiger-vnc [0] on OS X as the client.

Roger

[0] http://tigervnc.org/
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Re: Fwd: FreeBSD 11.1 xen trying to create linux domU instance

[Roger Pau Monné]

On Tue, Aug 22, 2017 at 09:51:47AM -0700, James E. Pace wrote:
> Thank you for the responses; I appreciate the help.
> 
> On Tue, Aug 22, 2017 at 1:11 AM, Roger Pau Monné <roger@citrix.com>
> > wrote:
> >
> >> On Mon, Aug 21, 2017 at 10:56:33AM -0700, James E. Pace wrote:
> >>
> >
> 
> > > First, each time I boot the (physical) system, I have to tell the FreeBSD
> >> > boot loader to turn on xen.  (That is, hit 6 to set options, then 7 to
> >> > enable xen, then 1 to return to the main menu, then 1 to boot).  Is
> >> there a
> >> > way to make this the default behavior?
> >>
> >> Yes, you need to add the following to /boot/loader.conf:
> >>
> >> xen_kernel="/boot/xen"
> >>
> >
> I have already added this to loader.conf, as suggested in the FreeBSD
> handbook section on Xen, which I followed.I figure this allows the
> possibility of booting xen, but doesn't make it the default?

That makes it the default, you don't need to press any key.

> [Deleted info about not booting Linux.  I added 'raw' for the zfs
> filesystem as hda and that didn't change anything.  However, after leaving
> it for hours, it appears that the domU got further in the boot process.
> Investigation continues.]

If you paste your guest config file + the serial output I might be
able to give you some clues.

> 
> > > Third, on my DomU FreeBSD guest, I'm trying to set up X / Gnome3.  The
> >> > mouse pointer and the little dot are far apart; that is, the mouse isn't
> >> > tracking my movements well.  I found several suggestions to add
> >> > "usbdevice=tablet" to the config file, but that caused it to not
> >> respond to
> >> > my mouse at all.  Suggestions?
> >>
> >> This should be:
> >>
> >> usbdevice=['tablet']
> >>
> >
> This didn't change the behavior; mouse still does not move.  Any other
> thoughts?

It works for me, weird.

Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"