Re: bhyve: svm (amd-v) update
On 15-5-2014 17:56, Anish wrote: Hi Andriy, Thanks for your interest in SVM port of bhyve. I do have patch to sync it to http://svnweb.freebsd.org/base?view=revisionrevision=263780(3/26). If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. Hi, With this patchset I see inb(0x40) calls coming thru to the bhyve VM, where I would expect them to stay down in the VMM dev-driver and emulate the vatpit for that VM. default_inout: port 64, in, bytes 1 default_inout: port 64, in, bytes 1 default_inout: port 67, out, bytes 1 Any particular reason that the IO for that register is not intercepted and ends up in userspace? Any suggestions what to watch for? Not really strong in debuging kernel modules. Thanx, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
On 2014-05-21 6:59, Anish wrote:
Hi Willem,
Thanks for sharing your patch. I see that you have gone further than
what I have. Are you able to boot Linux with these changes?
I'm running ubuntu-{12.x,14.04} with this and I'm able to compile the
linux kernel. Peter helped a lot in getting it going.
Although that is already takeing something like a week... So it is slow.
You might also need my ahci hack/patch, since linux uses a lot of
flushes and there is a 'bug' in the FreeBSD end of ahci that does not
really cooperate with the way ubuntu-14.04 calls it.
So I've emulated a flush to a NO-OP. which then results in a panic
because the ahci command-list is empty. So I 'fixed' that too, and then
things run, albeit slow.
CentOS it tried at the begining, but I'm just not a RH fan, so when that
was too slow, I gave up.
Give some time and I'll try and extract my ahci-hack-patch, and that is
the last part that'll let you get on your way, booting linux.
I do not have a lot of time to look at this, so I'm already a few weeks
stuck on speeding up the linux-kernels.
Also because I do not know enough of the inards of Linux..
--WjW
Thanks and regards,
Anish
On Tue, May 20, 2014 at 2:03 PM, Willem Jan Withagen w...@digiware.nl
mailto:w...@digiware.nl wrote:
On 15-5-2014 17:56, Anish wrote:
Hi Andriy,
Thanks for your interest in SVM port of bhyve. I do have patch
to sync it
to
http://svnweb.freebsd.org/base?view=revisionrevision=263780(3/26). If
patches looks good to you, we can submit it. I have been testing
it on
Phenom box which lacks some of newer SVM features.
I don't quite understand against what this patch is?
Do I run it over head, to get SVM code into head?
Or do I patch against bhyve_SVM, because in the later case I get
complaints that
fatal error: 'vlapic_priv.h' file not found
# locate vlapic_priv.h
/usr/srcs/head/sys/amd64/vmm/io/vlapic_priv.h
So I'm guessing that is against head.
But last time I looked at head, more than just the interrupt stuff was
missing
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
Hi Willem,
Willem Jan Withagen wrote:
I'd be interested in the vlapic to if that helps the speed.
But you can help me a lot if you give me the SVN commands to do what you
described above.
These were my steps:
0) mv /usr/src /usr/src.bak
1) svnlite co svn://svn.freebsd.org/base/projects/bhyve_svm /usr/src
2) cd /usr/src
3) patch -p4 /tmp/bhyve_svm_HEAD_r263780.patch
4) svnlite merge svn://svn.freebsd.org/base/head
one conflict in file amdv.c - enter mf (mine-full); in my previous
post, I mistakenly said theirs-full; what is, of course, wrong.
5) manually patch amdv.c with:
--- SNIP -
Index: sys/amd64/vmm/amd/amdv.c
===
--- sys/amd64/vmm/amd/amdv.c(revision 266491)
+++ sys/amd64/vmm/amd/amdv.c(working copy)
@@ -99,7 +99,7 @@
}
static void
-amd_iommu_add_device(void *domain, int bus, int slot, int func)
+amd_iommu_add_device(void *domain, uint16_t rid)
{
printf(amd_iommu_add_device: not implemented\n);
@@ -106,7 +106,7 @@
}
static void
-amd_iommu_remove_device(void *domain, int bus, int slot, int func)
+amd_iommu_remove_device(void *domain, uint16_t rid)
{
printf(amd_iommu_remove_device: not implemented\n);
--- SNIP -
6) should be fine now to compile and to integrate your patches
Thanks a lot for your work and regards,
Nils
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
On 2014-05-21 11:31, Nils Beyer wrote:
Hi Willem,
Willem Jan Withagen wrote:
I'd be interested in the vlapic to if that helps the speed.
But you can help me a lot if you give me the SVN commands to do what you
described above.
These were my steps:
0) mv /usr/src /usr/src.bak
1) svnlite co svn://svn.freebsd.org/base/projects/bhyve_svm /usr/src
2) cd /usr/src
3) patch -p4 /tmp/bhyve_svm_HEAD_r263780.patch
4) svnlite merge svn://svn.freebsd.org/base/head
one conflict in file amdv.c - enter mf (mine-full); in my previous
post, I mistakenly said theirs-full; what is, of course, wrong.
5) manually patch amdv.c with:
--- SNIP -
Index: sys/amd64/vmm/amd/amdv.c
===
--- sys/amd64/vmm/amd/amdv.c(revision 266491)
+++ sys/amd64/vmm/amd/amdv.c(working copy)
@@ -99,7 +99,7 @@
}
static void
-amd_iommu_add_device(void *domain, int bus, int slot, int func)
+amd_iommu_add_device(void *domain, uint16_t rid)
{
printf(amd_iommu_add_device: not implemented\n);
@@ -106,7 +106,7 @@
}
static void
-amd_iommu_remove_device(void *domain, int bus, int slot, int func)
+amd_iommu_remove_device(void *domain, uint16_t rid)
{
printf(amd_iommu_remove_device: not implemented\n);
--- SNIP -
6) should be fine now to compile and to integrate your patches
Thanx,
Very helpfull... And lets see if we can get amd-v (back) up to speed as
well.
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
On 15-5-2014 17:56, Anish wrote: Hi Andriy, Thanks for your interest in SVM port of bhyve. I do have patch to sync it to http://svnweb.freebsd.org/base?view=revisionrevision=263780(3/26). If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. I don't quite understand against what this patch is? Do I run it over head, to get SVM code into head? Or do I patch against bhyve_SVM, because in the later case I get complaints that fatal error: 'vlapic_priv.h' file not found # locate vlapic_priv.h /usr/srcs/head/sys/amd64/vmm/io/vlapic_priv.h So I'm guessing that is against head. But last time I looked at head, more than just the interrupt stuff was missing --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
Hi Willem, I patch against bhyve_SVM, because in the later case I get complaints that This patch is to sync bhyve_svm project branch with HEAD @263780, so you have to first merge HEAD to bhyve_svm. It will prompt you to resolve conflict in amdv.c, you should accept the changes that are in bhyve_svm and then apply the patch. bhyve HEAD exposed vlapic related interfaces along with some other changes, this patch will enable vlapic interfaces for SVM. Thanks and regards, Anish On Tue, May 20, 2014 at 2:03 PM, Willem Jan Withagen w...@digiware.nlwrote: On 15-5-2014 17:56, Anish wrote: Hi Andriy, Thanks for your interest in SVM port of bhyve. I do have patch to sync it to http://svnweb.freebsd.org/base?view=revisionrevision=263780(3/26). If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. I don't quite understand against what this patch is? Do I run it over head, to get SVM code into head? Or do I patch against bhyve_SVM, because in the later case I get complaints that fatal error: 'vlapic_priv.h' file not found # locate vlapic_priv.h /usr/srcs/head/sys/amd64/vmm/io/vlapic_priv.h So I'm guessing that is against head. But last time I looked at head, more than just the interrupt stuff was missing --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
Thanks for testing it. Your patch applied cleanly to the working copy of the bhyve_svm-project. I was then able to merge with HEAD (using theirs-full on one file) and compile the kernel. So, to me it looks OK to commit. Yes, that's correct. You have to retain changes in sys/amd64/vmm/amd/amdv.c from bhyve_svm branch. Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom 1055T. It produces 200% load on the host CPU, and the emulated machine generates endlessly: Its 200% load because of 2 vcpus to guest. It stuck in loop even with single processor(1 vcpu) after PCI probing[debug messages with linux .earlyprintk=serial debug] [3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) [3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [3.691987] NET: Registered protocol family 1 [3.693382] pci :00:01.0: Activating ISA DMA hang workarounds [3.695214] PCI: CLS 64 bytes, default 64 [3.698176] Trying to unpack rootfs image as initramfs... [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] [3.505631] pnp: PnP ACPI: found 5 devices [3.506417] ACPI: bus type PNP unregistered [3.635781] pci :00:06.0: no compatible bridge window for [mem 0xfe44 -0xfe45 pref] [3.637555] pci :00:06.0: BAR 6: assigned [mem 0x8000-0x8001 pref ] [3.638986] pci :00:01.0: BAR 6: assigned [mem 0x8002-0x800207ff pref ] [3.640416] pci :00:04.0: BAR 6: assigned [mem 0x80020800-0x80020fff pref ] [3.641864] pci :00:05.0: BAR 6: assigned [mem 0x80021000-0x800217ff pref ] [3.643259] pci :00:00.0: not setting up bridge for bus :01 [3.644550] pci_bus :00: resource 4 [io 0x-0x0cf7] [3.645670] pci_bus :00: resource 5 [io 0x0d00-0x] [3.646795] pci_bus :00: resource 6 [mem 0x8000-0xdfff] [3.648031] pci_bus :00: resource 7 [mem 0xd0-0xfc] [3.650970] NET: Registered protocol family 2 [3.661491] TCP established hash table entries: 16384 (order: 6, 262144 bytes ) [3.671854] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [3.681116] TCP: Hash tables configured (established 16384 bind 16384) [3.683335] TCP: reno registered [3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) [3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [3.691987] NET: Registered protocol family 1 [3.693382] pci :00:01.0: Activating ISA DMA hang workarounds [3.695214] PCI: CLS 64 bytes, default 64 [3.698176] Trying to unpack rootfs image as initramfs... [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] [ 30.596366] Modules linked in: Additionally, It produces a lot of MSR requests: Yes, on AMD Linux is touching more MSRs( AMD specific -address 0xC00) compared to Intel. Thanks and regards, Anish On Fri, May 16, 2014 at 2:17 PM, Nils Beyer n...@renzel.net wrote: Hi Anish, Anish wrote: If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. Your patch applied cleanly to the working copy of the bhyve_svm-project. I was then able to merge with HEAD (using theirs-full on one file) and compile the kernel. So, to me it looks OK to commit. Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom 1055T. It produces 200% load on the host CPU, and the emulated machine generates endlessly: === BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1] Modules linked in: CPU 0 Modules linked in: Pid: 1, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1 BHYVE RIP: 0010:[81c5496d] [81c5496d] rc_is_bit_0+0x3a/0x69 RSP: 0018:88013e79dca0 EFLAGS: 0a96 RAX: 009c RBX: 88013e79dcc0 RCX: 880004bdcc7c RDX: 002f9dee RSI: c9000402c538 RDI: 88013e79ddb0 RBP: 8100b9ce R08: c9000402c788 R09: 81de32b8 R10: 0003 R11: R12: 0003 R13: 81157602 R14: 88013e79dc20 R15: 00d2 FS: () GS:88002820() knlGS: CS: 0010 DS: 0018 ES: 0018 CR0: 8005003b CR2: CR3: 01a85000 CR4: 07b0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process swapper (pid: 1, threadinfo 88013e79c000, task 88013e79b500) Stack: c9000402c644 00e5 03b1 88013e79ddb0 d 88013e79dcf0 81c549b7 ffb6 d c9000402c000 88013e79de30 81c554e2 Call Trace: [81c549b7] ? rc_get_bit+0x1b/0x79 [81c554e2] ? unlzma+0xa42/0xc67
Re: bhyve: svm (amd-v) update
Hi Anish, Anish wrote: If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. Your patch applied cleanly to the working copy of the bhyve_svm-project. I was then able to merge with HEAD (using theirs-full on one file) and compile the kernel. So, to me it looks OK to commit. Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom 1055T. It produces 200% load on the host CPU, and the emulated machine generates endlessly: === BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1] Modules linked in: CPU 0 Modules linked in: Pid: 1, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1 BHYVE RIP: 0010:[81c5496d] [81c5496d] rc_is_bit_0+0x3a/0x69 RSP: 0018:88013e79dca0 EFLAGS: 0a96 RAX: 009c RBX: 88013e79dcc0 RCX: 880004bdcc7c RDX: 002f9dee RSI: c9000402c538 RDI: 88013e79ddb0 RBP: 8100b9ce R08: c9000402c788 R09: 81de32b8 R10: 0003 R11: R12: 0003 R13: 81157602 R14: 88013e79dc20 R15: 00d2 FS: () GS:88002820() knlGS: CS: 0010 DS: 0018 ES: 0018 CR0: 8005003b CR2: CR3: 01a85000 CR4: 07b0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process swapper (pid: 1, threadinfo 88013e79c000, task 88013e79b500) Stack: c9000402c644 00e5 03b1 88013e79ddb0 d 88013e79dcf0 81c549b7 ffb6 d c9000402c000 88013e79de30 81c554e2 Call Trace: [81c549b7] ? rc_get_bit+0x1b/0x79 [81c554e2] ? unlzma+0xa42/0xc67 [81c28ab9] ? flush_buffer+0x0/0xa3 [811bb9cb] ? do_utimes+0xdb/0x170 [812827a0] ? nofill+0x0/0x10 [81c29776] ? unpack_to_rootfs+0x167/0x27a [81c28929] ? error+0x0/0x17 [812a6725] ? pci_get_subsys+0x35/0x40 [81c2992b] ? populate_rootfs+0x0/0xd3 [81c29986] ? populate_rootfs+0x5b/0xd3 [8100204c] ? do_one_initcall+0x3c/0x1d0 [81c268e4] ? kernel_init+0x29b/0x2f7 [8100c20a] ? child_rip+0xa/0x20 [81c26649] ? kernel_init+0x0/0x2f7 [8100c200] ? child_rip+0x0/0x20 Code: ff ff ff 00 77 35 48 8b 47 18 48 39 47 08 72 0d 48 89 75 e8 e8 95 ff ff ff 48 8b 75 e8 48 8b 4b 08 c1 63 28 08 8b 53 24 0f b6 01 48 83 c1 01 c1 e2 08 48 89 4b 08 09 d0 89 43 24 0f b7 06 8b 53 Call Trace: [81c549b7] ? rc_get_bit+0x1b/0x79 [81c554e2] ? unlzma+0xa42/0xc67 [81c28ab9] ? flush_buffer+0x0/0xa3 [811bb9cb] ? do_utimes+0xdb/0x170 [812827a0] ? nofill+0x0/0x10 [81c29776] ? unpack_to_rootfs+0x167/0x27a [81c28929] ? error+0x0/0x17 [812a6725] ? pci_get_subsys+0x35/0x40 [81c2992b] ? populate_rootfs+0x0/0xd3 [81c29986] ? populate_rootfs+0x5b/0xd3 [8100204c] ? do_one_initcall+0x3c/0x1d0 [81c268e4] ? kernel_init+0x29b/0x2f7 [8100c20a] ? child_rip+0xa/0x20 [81c26649] ? kernel_init+0x0/0x2f7 [8100c200] ? child_rip+0x0/0x20 === Additionally, It produces a lot of MSR requests: === May 16 09:32:03 10.255.255.96 kernel: emulate_rdmsr 0xc0010015 May 16 09:32:18 10.255.255.96 kernel: emulate_rdmsr 0x1b May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0xc0010112 May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0xc0010048 May 16 09:32:23 10.255.255.96 kernel: emulate_wrmsr 0xc0010048 May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0x8b May 16 09:32:23 10.255.255.96 kernel: emulate_rdmsr 0xc0010140 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc001 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010001 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010002 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010003 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010004 May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010004 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010004 May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010004 May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc001 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0x1b May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0xc0010048 May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010048 May 16 09:32:25 10.255.255.96 kernel: emulate_rdmsr 0x8b May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc0010004 May 16 09:32:25 10.255.255.96 kernel: emulate_wrmsr 0xc001 May 16 09:32:25 10.255.255.96
bhyve: svm (amd-v) update
It seems that the bhyve_svm branch is a bit behind the latest interface changes in head. Is anyone working on sync-ing up the branch with the head? Some examples: - change of init method in vmm_ops - addition of resume, vlapic_init and vlapic_cleanup methods to vmm_ops - replacement of lapic_pending_intr and lapic_intr_accepted with vlapic_pending_intr and vlapic_intr_accepted - changes in struct vm_exit - VMEXIT_EPT_FAULT == VMEXIT_NESTED_FAULT Thanks! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
Hi Andriy, Thanks for your interest in SVM port of bhyve. I do have patch to sync it to http://svnweb.freebsd.org/base?view=revisionrevision=263780(3/26). If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. Thanks and regards, Anish On Thu, May 15, 2014 at 2:10 AM, Andriy Gapon a...@freebsd.org wrote: It seems that the bhyve_svm branch is a bit behind the latest interface changes in head. Is anyone working on sync-ing up the branch with the head? Some examples: - change of init method in vmm_ops - addition of resume, vlapic_init and vlapic_cleanup methods to vmm_ops - replacement of lapic_pending_intr and lapic_intr_accepted with vlapic_pending_intr and vlapic_intr_accepted - changes in struct vm_exit - VMEXIT_EPT_FAULT == VMEXIT_NESTED_FAULT Thanks! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org bhyve_svm_HEAD_r263780.patch Description: Binary data ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
