Re: bhyve win-guest benchmark comparing
> confirm this made a huge effect on a Xeon E3-1220, while it (obviously) made no difference on a Ryzen 7 2700. Bhyve is missing support for AMD AVIC, which is interrupt h/w assist similar to Intel vAPIC. AVIC I think is available only in Ryzen gen1. It is on my TODO list. -Anish On Wed, May 6, 2020 at 12:35 AM Andrea Venturoli wrote: > On 2020-04-20 12:06, Jason Tubnor wrote: > > On Mon., 20 Apr. 2020, 7:09 pm Felix Palmen, wrote: > > > >> > >>> Xeon are not affected, since they have vAPIC, so they should be already > >>> running as fast as possible. Correct? > >> > >> I don't know the details, but on my Xeon E3-1240L v5, it did have a huge > >> effect. > > I confirm this made a huge effect on a Xeon E3-1220, while it > (obviously) made no difference on a Ryzen 7 2700. > > I'll try this ASAP on a Core i5-9400 and a Core(TM) i5-7400 :) > > bye & Thanks > av. > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570
[Resending without screenshot to reduce message size] Hi Christian, This looks like BIOS issue, BIOS must not set Bit4/SVMDIS in VM_CR MSR (0xC001_0114) if SVM is enabled. Bhyve checks for this bit before enabling SVM, otherwise it can cause GP faults. You can read it using the following steps: $kldload cpuctl $ cpucontrol -m 0xc0010114 /dev/cpuctlX <- Read VM_CR MSR MSR 0xc0010114: 0x 0x0008 X is any CPU core number. On my AMD/Ryzen system, bit4 is clear and I can load bhyve. -Anish On Sun, Apr 26, 2020 at 12:07 PM Anish wrote: > > Hi Christian, > > This looks like BIOS issue, BIOS must not set Bit4/SVMDIS in VM_CR MSR > (0xC001_0114) if SVM is enabled. Bhyve checks for this bit before > enabling SVM. You can read it using the following steps: > $kldload cpuctl > > > $ cpucontrol -m 0xc0010114 /dev/cpuctlX <- Read VM_CR MSR > MSR 0xc0010114: 0x 0x0008 > > X is any CPU core number. > > On my AMD/Ryzen system, bit4 is clear and I can load bhyve. I have > attached a screenshot from AMD SDM Vol2, which you may find useful. > > [image: Screen Shot 2020-04-26 at 12.01.14 PM.png] > > -Anish > > > > On Sun, Apr 26, 2020 at 6:27 AM Christian Jeannot > wrote: > >> I have checked MSI support page. The latest BIOS version is already >> installed. >> >> Regards >> >> --Christian >> >> Am 26.04.20 um 15:11 schrieb M. Eisenhardt: >> > I would suggest to update your bios. >> > Kind regards >> > Michael >> > >> >> Am 26.04.2020 um 14:00 schrieb >> freebsd-virtualization-requ...@freebsd.org: >> >> >> >> Send freebsd-virtualization mailing list submissions to >> >> freebsd-virtualization@freebsd.org >> >> >> >> To subscribe or unsubscribe via the World Wide Web, visit >> >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> >> or, via email, send a message with subject or body 'help' to >> >> freebsd-virtualization-requ...@freebsd.org >> >> >> >> You can reach the person managing the list at >> >> freebsd-virtualization-ow...@freebsd.org >> >> >> >> When replying, please edit your Subject line so it is more specific >> >> than "Re: Contents of freebsd-virtualization digest..." >> >> >> >> >> >> Today's Topics: >> >> >> >>1. bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 >> >> (Christian Jeannot) >> >>2. Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 >> >> (Aryeh Friedman) >> >>3. Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 >> >> (Christian Jeannot) >> >> >> >> >> >> -- >> >> >> >> Message: 1 >> >> Date: Sun, 26 Apr 2020 10:57:50 +0200 >> >> From: Christian Jeannot >> >> To: freebsd-virtualization@freebsd.org >> >> Subject: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 >> >> Message-ID: <2535c7f2-ef59-467e-779b-6c9c6db82...@os-plus.org> >> >> Content-Type: text/plain; charset=utf-8; format=flowed >> >> >> >> Hi Community, >> >> >> >> I am testing bhyve from FreeBSD 12.1 RELEASE. >> >> >> >> My environment: >> >> - CPU: AMD Ryze 3900X >> >> - Mainboard: MSI MEG ACE X570, Default settings, latest BIOS 7C35v18 >> >> - RAM: G.Skill RipJaws V DIMM Kit 64GB, DDR4-3200, CL16-18-18-38 >> >> - GPU: Sapphire Nitro RX 5700XT >> >> >> >> I checked the settings in the BIOS. Virtualization options are enabled. >> >> SVM is enabled. >> >> >> >> I follow the instruction from the FreeBSD handbook ?21.7. FreeBSD as a >> >> Host with bhyve?. >> >> >> >> When I run >> >> kldload vmm >> >> I got an output that SVM is disabled by BIOS. When I check the BIOS it >> >> is enabled. >> >> >> >> When I try to start the VM with the example script I got the output >> >> vm_create device not configured >> >> >> >> Did I miss something to configure? >> >> >> >> Does anyone has a Ryzen CPU and a actual MSI mainboard successfully run >> >> bhyve? >> >> >> >> Best regards >> >> >> >> ?Christian >> >> -- >> >> -- >> >> christia
Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570
Hi Christian, This looks like BIOS issue, BIOS must not set Bit4/SVMDIS in VM_CR MSR (0xC001_0114) if SVM is enabled. Bhyve checks for this bit before enabling SVM. You can read it using the following steps: $kldload cpuctl $ cpucontrol -m 0xc0010114 /dev/cpuctlX <- Read VM_CR MSR MSR 0xc0010114: 0x 0x0008 X is any CPU core number. On my AMD/Ryzen system, bit4 is clear and I can load bhyve. I have attached a screenshot from AMD SDM Vol2, which you may find useful. [image: Screen Shot 2020-04-26 at 12.01.14 PM.png] -Anish On Sun, Apr 26, 2020 at 6:27 AM Christian Jeannot wrote: > I have checked MSI support page. The latest BIOS version is already > installed. > > Regards > > --Christian > > Am 26.04.20 um 15:11 schrieb M. Eisenhardt: > > I would suggest to update your bios. > > Kind regards > > Michael > > > >> Am 26.04.2020 um 14:00 schrieb > freebsd-virtualization-requ...@freebsd.org: > >> > >> Send freebsd-virtualization mailing list submissions to > >> freebsd-virtualization@freebsd.org > >> > >> To subscribe or unsubscribe via the World Wide Web, visit > >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > >> or, via email, send a message with subject or body 'help' to > >> freebsd-virtualization-requ...@freebsd.org > >> > >> You can reach the person managing the list at > >> freebsd-virtualization-ow...@freebsd.org > >> > >> When replying, please edit your Subject line so it is more specific > >> than "Re: Contents of freebsd-virtualization digest..." > >> > >> > >> Today's Topics: > >> > >>1. bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> (Christian Jeannot) > >>2. Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> (Aryeh Friedman) > >>3. Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> (Christian Jeannot) > >> > >> > >> -- > >> > >> Message: 1 > >> Date: Sun, 26 Apr 2020 10:57:50 +0200 > >> From: Christian Jeannot > >> To: freebsd-virtualization@freebsd.org > >> Subject: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> Message-ID: <2535c7f2-ef59-467e-779b-6c9c6db82...@os-plus.org> > >> Content-Type: text/plain; charset=utf-8; format=flowed > >> > >> Hi Community, > >> > >> I am testing bhyve from FreeBSD 12.1 RELEASE. > >> > >> My environment: > >> - CPU: AMD Ryze 3900X > >> - Mainboard: MSI MEG ACE X570, Default settings, latest BIOS 7C35v18 > >> - RAM: G.Skill RipJaws V DIMM Kit 64GB, DDR4-3200, CL16-18-18-38 > >> - GPU: Sapphire Nitro RX 5700XT > >> > >> I checked the settings in the BIOS. Virtualization options are enabled. > >> SVM is enabled. > >> > >> I follow the instruction from the FreeBSD handbook ?21.7. FreeBSD as a > >> Host with bhyve?. > >> > >> When I run > >> kldload vmm > >> I got an output that SVM is disabled by BIOS. When I check the BIOS it > >> is enabled. > >> > >> When I try to start the VM with the example script I got the output > >> vm_create device not configured > >> > >> Did I miss something to configure? > >> > >> Does anyone has a Ryzen CPU and a actual MSI mainboard successfully run > >> bhyve? > >> > >> Best regards > >> > >> ?Christian > >> -- > >> -- > >> christian jeannot > >> vogelmauer 17 > >> 86152 augsburg > >> +49 821 81552861 > >> i...@os-plus.org > >> > >> > >> -- > >> > >> Message: 2 > >> Date: Sun, 26 Apr 2020 05:32:36 -0400 > >> From: Aryeh Friedman > >> To: Christian Jeannot > >> Cc: "freebsd-virtualization@freebsd.org" > >> > >> Subject: Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> Message-ID: > >> > > >> Content-Type: text/plain; charset="UTF-8" > >> > >>> On Sun, Apr 26, 2020 at 4:57 AM Christian Jeannot > wrote: > >>> > >>> Hi Community, > >>> > >>> I am testing bhyve from FreeBSD 12.1 RELEASE. > >>> > >>> My environment: > >>> - CPU: AMD Ryze 3900X > >>> - Mainboard: MSI MEG ACE X570, Default settings,
Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570
[Resending without screenshot to reduce message size] Hi Christian, This looks like BIOS issue, BIOS must not set Bit4/SVMDIS in VM_CR MSR (0xC001_0114) if SVM is enabled. Bhyve checks for this bit before enabling SVM otherwise, it can cause GP faults. You can read it using the following steps: $kldload cpuctl $ cpucontrol -m 0xc0010114 /dev/cpuctlX <- Read VM_CR MSR MSR 0xc0010114: 0x 0x0008 X is any CPU core number. On my AMD/Ryzen system, bit4 is clear and I can load bhyve. -Anish On Sun, Apr 26, 2020 at 6:27 AM Christian Jeannot wrote: > I have checked MSI support page. The latest BIOS version is already > installed. > > Regards > > --Christian > > Am 26.04.20 um 15:11 schrieb M. Eisenhardt: > > I would suggest to update your bios. > > Kind regards > > Michael > > > >> Am 26.04.2020 um 14:00 schrieb > freebsd-virtualization-requ...@freebsd.org: > >> > >> Send freebsd-virtualization mailing list submissions to > >> freebsd-virtualization@freebsd.org > >> > >> To subscribe or unsubscribe via the World Wide Web, visit > >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > >> or, via email, send a message with subject or body 'help' to > >> freebsd-virtualization-requ...@freebsd.org > >> > >> You can reach the person managing the list at > >> freebsd-virtualization-ow...@freebsd.org > >> > >> When replying, please edit your Subject line so it is more specific > >> than "Re: Contents of freebsd-virtualization digest..." > >> > >> > >> Today's Topics: > >> > >>1. bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> (Christian Jeannot) > >>2. Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> (Aryeh Friedman) > >>3. Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> (Christian Jeannot) > >> > >> > >> -- > >> > >> Message: 1 > >> Date: Sun, 26 Apr 2020 10:57:50 +0200 > >> From: Christian Jeannot > >> To: freebsd-virtualization@freebsd.org > >> Subject: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> Message-ID: <2535c7f2-ef59-467e-779b-6c9c6db82...@os-plus.org> > >> Content-Type: text/plain; charset=utf-8; format=flowed > >> > >> Hi Community, > >> > >> I am testing bhyve from FreeBSD 12.1 RELEASE. > >> > >> My environment: > >> - CPU: AMD Ryze 3900X > >> - Mainboard: MSI MEG ACE X570, Default settings, latest BIOS 7C35v18 > >> - RAM: G.Skill RipJaws V DIMM Kit 64GB, DDR4-3200, CL16-18-18-38 > >> - GPU: Sapphire Nitro RX 5700XT > >> > >> I checked the settings in the BIOS. Virtualization options are enabled. > >> SVM is enabled. > >> > >> I follow the instruction from the FreeBSD handbook ?21.7. FreeBSD as a > >> Host with bhyve?. > >> > >> When I run > >> kldload vmm > >> I got an output that SVM is disabled by BIOS. When I check the BIOS it > >> is enabled. > >> > >> When I try to start the VM with the example script I got the output > >> vm_create device not configured > >> > >> Did I miss something to configure? > >> > >> Does anyone has a Ryzen CPU and a actual MSI mainboard successfully run > >> bhyve? > >> > >> Best regards > >> > >> ?Christian > >> -- > >> -- > >> christian jeannot > >> vogelmauer 17 > >> 86152 augsburg > >> +49 821 81552861 > >> i...@os-plus.org > >> > >> > >> -- > >> > >> Message: 2 > >> Date: Sun, 26 Apr 2020 05:32:36 -0400 > >> From: Aryeh Friedman > >> To: Christian Jeannot > >> Cc: "freebsd-virtualization@freebsd.org" > >> > >> Subject: Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570 > >> Message-ID: > >> > > >> Content-Type: text/plain; charset="UTF-8" > >> > >>> On Sun, Apr 26, 2020 at 4:57 AM Christian Jeannot > wrote: > >>> > >>> Hi Community, > >>> > >>> I am testing bhyve from FreeBSD 12.1 RELEASE. > >>> > >>> My environment: > >>> - CPU: AMD Ryze 3900X > >>> - Mainboard: MSI MEG ACE X570, Default settings, latest BIOS 7C35v18 > >&
Re: bhyve with AMD Ryzen 3900X and MSI MEG ACE X570
>If SVM is actually disabled, I have no idea what impact this change will have, so beware. As per the AMD document, it will cause GP fault if it has been implemented as per specification. If we are lucky and it works, that will be an interesting finding. -Anish On Sun, Apr 26, 2020 at 10:53 AM Eric van Gyzen wrote: > On 4/26/20 3:57 AM, Christian Jeannot wrote: > > Hi Community, > > > > I am testing bhyve from FreeBSD 12.1 RELEASE. > > > > My environment: > > - CPU: AMD Ryze 3900X > > - Mainboard: MSI MEG ACE X570, Default settings, latest BIOS 7C35v18 > > - RAM: G.Skill RipJaws V DIMM Kit 64GB, DDR4-3200, CL16-18-18-38 > > - GPU: Sapphire Nitro RX 5700XT > > > > I checked the settings in the BIOS. Virtualization options are enabled. > > SVM is enabled. > > > > I follow the instruction from the FreeBSD handbook „21.7. FreeBSD as a > > Host with bhyve“. > > > > When I run > > kldload vmm > > I got an output that SVM is disabled by BIOS. When I check the BIOS it > > is enabled. > > I use bhyve on a Ryzen with an Asus Prime X470-PRO. > > I wonder if your BIOS is enabling SVM but failing to update the MSR to > say it's enabled. Try commenting out the second "return (0);" in > svm_available() in sys/amd64/vmm/amd/svm.c and rebuilding the vmm > module. If SVM is actually disabled, I have no idea what impact this > change will have, so beware. If SVM is actually enabled and this hack > allows you to use it, we could probably commit a tunable to do this more > cleanly. > > Eric > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: BSD11 vs BSD12 KVM performance
Hi Udayan, Intel VM-x and AMD SVM allow hypervisor to intercept TSC access for save/restore of VM migration, so when these intercepts are enabled, rdtsc will require switch between root and non-root mode which is expensive and may cost few hundreds of cpu cycles. Is it possible that one has these intercepts always enabled even when migration is not started? -Anish ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCIe passthrough on AMD Epyc
Hi Kevin, >I'm just pointing out that it's an AMD Epyc because I know the AMD IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is maybe less tested. I tested on Ryzen-I, not sure if anyone tried it on newer Ryzen/Epyc systems. >CPU0: local APIC error 0x40 This is concerning, as per AMD docs, bit6 is for illegal vector which indicate CPU is getting invalid interrupts. Can you share hw.vmm sysctl output? -Anish On Thu, Feb 28, 2019 at 9:59 AM Kevin Day wrote: > > > > On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes < > freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote: > > > >>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes < > freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote: > >>> > >>>> I'm trying to get PCIe device (video card) passthrough working to a > Windows bhyve VM, should this work now? > >>> > >>> No, this should not work now. > >> > >> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now? > > I do not know what would be special about Epyc so can not say > > for certain. > > I'm just pointing out that it's an AMD Epyc because I know the AMD > IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is > maybe less tested. > > >> Just trying the same thing with a simple PCIe USB card gives me: > >> > >> CPU0: local APIC error 0x40 > >> CPU0: local APIC error 0x40 > >> ivhd0: Error: completion failed tail:0xda0, head:0x0. > >> ivhd0: Dump all the commands: > >> [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0 > >> [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7003 > >> [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0 > >> [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0 > >> [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0 > >> [...snip... lots of similar stuff] > >> > >> Then I get a bunch more APIC errors, and the system freezes for about > 60 seconds before recovering. > > > > You do not get any host dmesg errors about unaligned BAR > > ranges or sizes? > > > > Correct. I did with the video card, but not with the xhci USB card. I get > a ton of dmesg errors, and the host system repeatedly freezes until I stop > the VM. (Console is non-responsive, no ICMP replies, etc) > > I didn't actually look at this until now, but Windows actually does see > the USB controller, but it complains that "port reset failed" on any device > I attach to it. > > > Can you get the pciconf -l -b info for the device you passed, something > like: > > # pciconf -l -b uhci0 > > uhci0@pci0:0:26:0: class=0x0c0300 card=0x20f017aa chip=0x29378086 > rev=0x03 hdr=0x00 > >bar [20] = type I/O Port, range 32, base 0x1840, size 32, enabled > > xhci3@pci0:97:0:0: class=0x0c0330 card=0x chip=0x00151912 > rev=0x02 hdr=0x00 > bar [10] = type Memory, range 64, base 0xe1d0, size 8192, enabled > cap 01[50] = powerspec 3 supports D0 D3 current D0 > cap 05[70] = MSI supports 8 messages, 64 bit enabled with 1 message > cap 11[90] = MSI-X supports 8 messages > Table in map 0x10[0x1000], PBA in map 0x10[0x1080] > cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS > link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1) > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0018[150] = LTR 1 > > Also just in case it's relevant, "acpidump" on this system hangs after > allocating ~3GB of ram and just spins at 100% CPU. I am seeing that AMD-Vi > is enabled in the boot dmesg though: > > AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0 > > > > > > > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: on bhyve statistics
>> Currently, bhyve does not expose any of these statistics. All the stats
> available through bhyvectl --get-stats seem to be coming from the VMM,
> not from the userspace emulation.
>That is correct, byhvectl is a diagnostics tool for getting
information from the kernel/vmm module.
bhyvectl provide stats related to processor vmx/svm from vmm.ko and is the
first thing you want to run for performance regression. It will be nice to
include it as part of bhyve perf tool/dashboard that you are intended to
build.
-Anish
On Mon, Aug 27, 2018 at 8:20 AM Rodney W. Grimes <
freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> > Hi list,
> >
> > I'm currently looking at getting the libvirt prometheus exporter[1] to
> > work with libvirt+bhyve. In its current state this doesn't work because
> > at least one of the API calls exposed by libvirt isn't implemented by
> > the libvirt bhyve driver - so I started looking at implementing it.
> >
> > The first API call in question is virDomainBlockStats[2], which returns
> > statistics (number of read and written bytes and ops, respectively).
> >
> > Currently, bhyve does not expose any of these statistics. All the stats
> > available through bhyvectl --get-stats seem to be coming from the VMM,
> > not from the userspace emulation.
>
> That is correct, byhvectl is a diagnostics tool for getting
> information from the kernel/vmm module.
>
> > OTOH, I did see that there are *some*
> > stats being collected in bhyverun.c (see struct bhyvestats {...}
> > stats;). I can't see how these are exposed though - a grep of /usr/src
> > turned up no other uses. Which brings me to the following questions:
> >
> > - are the stats in struct bhyvestats {...} stats exposed or used in any
> > non-obvious way?
>
> Not that I am aware of.
>
> > - architecturally, what would be the best ways to get stats out of the
> > user-space emulations? Off of the top of my head, I could think of the
> > following possibilities:
> > - prometheus exporter
> > - having some socket or pipe to request them
> > - DTrace probes
> >
> > I wouldn't mind implementing any of the above, and so would like to know
> > which of these (or other options) would be the most acceptable, and
> > would appreciate some guidance.
>
> I differ to others on what may be the best way to do this.
>
> > CC'ing novel@ for the libvirt side, and grehan@ for the architectural
> > bhyve questions.
>
> You should replace @grehan with @jhb,@tychon as Peter has moved on,
> and John and Tycho are now the bhyve maintainers. I was going to
> add them, and remove Peter, but I see no cc: anyway, so I am sure
> that they are on the virtualization list though.
>
> > Fabian
> >
> > [1] https://github.com/kumina/libvirt_exporter
> > [2]
> https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockStats
> > ___
> > freebsd-virtualization@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
> --
> Rod Grimes
> rgri...@freebsd.org
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: VMX exit reason=33 and general userboot.so questions
>idtr[0] 0x/0x
IDTR is not set, you can set the value same as in GDTR, from libvmm
/* GDTR, IDTR */
desc_base = 0;
desc_limit = 0x;
desc_access = 0;
error = vm_set_desc(vmctx, vcpu, VM_REG_GUEST_GDTR,
desc_base, desc_limit, desc_access);
if (error != 0)
goto done;
error = vm_set_desc(vmctx, vcpu, VM_REG_GUEST_IDTR,
desc_base, desc_limit, desc_access);
>I’m assuming entry_ctls[0] = 0x91fb is the VM-entry control,
Yes, it is VM entry control. Its value need to be determined based on
MSR_VMX_ENTRY_CTLS
and MSR_VMX_TRUE_ENTRY_CTLS MSRs, see
/* Check support for VM-entry controls */
error = vmx_set_ctlreg(MSR_VMX_ENTRY_CTLS, MSR_VMX_TRUE_ENTRY_CTLS,
VM_ENTRY_CTLS_ONE_SETTING, VM_ENTRY_CTLS_ZERO_SETTING,
_ctls);
-Anish
On Wed, Feb 21, 2018 at 11:08 AM, Fabian Freyer <
fabian.fre...@physik.tu-berlin.de> wrote:
> Hi Peter,
>
> thanks for your response!
>
> On 21 Feb 2018, at 17:59, Peter Grehan wrote:
> > >exit_reason 33
> >
> > From the Intel SDM, vol 3B Appendix C, this error is "VM-entry failure
> due to invalid guest state".
>
> Yes, I’m currently going through SDM, vol 3B, section 26.3, and
>
> > These errors can be difficult to debug given the large amount of guest
> state involved :(
>
> definitely accurately describes the experience. That said, I’m going
> through each check and trying to see what is going wrong.
>
> > However, looking at the state from your dump:
> >
> > > tr desc[0] 0x/0x/0x
> >
> > I believe you will have to set this. Here's the comment and relevant
> code fragment from grub2-bhyve grub-core/kern/emu/bhyve_
> hostif.c:grub_emu_bhyve_boot32()
> >
> > /*
> >* XXX TR is pointing to null selector even though we set the
> >* TSS segment to be usable with a base address and limit of 0.
> >* Has to be 8b or vmenter will fail
> >*/
> > desc_access = 0x008b;
> > assert(vm_set_desc(bhyve_ctx, 0, VM_REG_GUEST_TR, 0x1000, 0x67,
> > desc_access) == 0);
> >
> > grub2-bhyve has been able to load/boot multiboot images, so I suspect
> the register settings in grub_emu_bhyve_boot32() are a good place to start
> from.
>
> Thanks. I’ve tried setting all registers (especially) the segment
> registers as grub2-bhyve does, but I still don’t seem to be able to boot
> correctly. Fixing the TR descriptor didn’t do it.
>
> Here’s a list of the descriptors I set up:
>
> ds desc[0] 0x/0x/0x0093
> es desc[0] 0x/0x/0x0093
> fs desc[0] 0x/0x/0x0093
> gs desc[0] 0x/0x/0x0093
> ss desc[0] 0x/0x/0x0093
> cs desc[0] 0x/0x/0x009b
> tr desc[0] 0x1000/0x0067/0x008b
> ldtr desc[0]0x/0x/0x00010082
>
>
> gdtr points to a default gdt as set up in grub2-bhyve:
>
> static uint16_t bhyve_gdt[] = {
> 0x, 0x, 0x, 0x, /* Null */
> 0x, 0x, 0x, 0x, /* Null #2 */
> 0x, 0x, 0x9a00, 0x00cf, /* code */
> 0x, 0x, 0x9200, 0x00cf, /* data */
> 0x, 0x, 0x8900, 0x0080/* tss */
> };
>
> gdtr[0] 0x00100161/0x0027
> idtr[0] 0x/0x
>
> The rest is set up as by grub2-bhyve:
> cs[0] 0x0008
> ds[0] 0x0010
> es[0] 0x0010
> fs[0] 0x0010
> gs[0] 0x0010
> ss[0] 0x0010
> tr[0] 0x
> ldtr[0] 0x
>
> I’m assuming entry_ctls[0] = 0x91fb is the VM-entry control, but is it the
> value described by Table 24-7 of the SDM Volume 3B? Is this constant for
> the first entry after running bhyveload? That is, when writing some
> compliance checks against section 26.3.1, can I just hardcode that value?
>
> I’m starting to dig through the bhyverun code, but I’m still pretty new to
> VMX stuff, so I am ending up with more questions than answers.
>
> Fabian
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: problem with pass-through on amd
Hi Andriy, >After I fixed the MSI setup problem that I described in the previous email I was finally able to see the real problem. Awesome! fault interrupt handler has been very helpful in debugging AMD-Vi. >I've passed both of the PCI device to bhyve and everything started working. Thanks! Bhyve uses single domain for all the passthrough devices of a guest, that's why even with aliasing it worked. >But it appears to me that FreeBSD currently ignores those aliases. Also, it looks like we are not making any use of IVHD entries beyond printing them. We make use of of very few IVRS entries, see ivhd_dev_parse() but not much for aliasing entries. Found out AMD IOMMU spec has some details on it: "An Alias device type entry is used for each peripheral that does not use its own DeviceID information in bus transactions. For example, peripherals downstream of a bridge device that use the DeviceID of the bridge must have a corresponding Alias Select or Alias Start of Range entry to inform system software which IOMMU Device Table entry will be used for translation information." Thanks a lot for trying out AMD-Vi, let me know if you have any other feedback. Thanks and regards, Anish On Thu, Nov 16, 2017 at 12:51 PM, Andriy Gapon <a...@freebsd.org> wrote: > On 14/11/2017 06:22, Anish wrote: > > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh > > irq256: ivhd0:fault0 0 > > irq257: ivhd1:fault0 0 > > After I fixed the MSI setup problem that I described in the previous email > I was > finally able to see the real problem. > Now I have: > > irq263: ivhd0:fault9 0 > > dev.ivhd.0.event_tail: 240 > dev.ivhd.0.event_head: 240 > dev.ivhd.0.event_intr_count: 9 > > > I've got a bunch of messages like these in the log: > [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x70f70400x30<PR,RW>] > [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x2fdf0400x30<PR,RW>] > (BTW, there seems to be a missing space before 0x30) > > Now it's obvious what the problem is. > The controller has a RID of 0xa00 (its PCI locator is 10:0:0) but the > requests > are coming from 0xa01. The card actually has another PCI device at that > location (10:0:1), it's a vestigial IDE controller (in a sense that it is > not > connected to any ports, so it cannot really provide any functionality). > > I've passed both of the PCI device to bhyve and everything started working. > Thanks! > > I've googled a little bit and it seems that this is not an uncommon > problem. > Linux has a bunch of quirks for this and similar kinds of problems (they > call it > PCI or DMA aliasing): > http://elixir.free-electrons.com/linux/v4.8.16/source/ > drivers/pci/quirks.c#L3701 > The comments say that sometimes requests come from devices that do not > exist at > all (or hidden beyond non-transparent bridges). > > An interesting new world for me :-) > > By the way, my understanding is that the alias IVHD entries in IVRS are > designed > to report such aliasing issues when they are known to the platform. > But it appears to me that FreeBSD currently ignores those aliases. > Also, it looks like we are not making any use of IVHD entries beyond > printing > them. Based on the specification I think that we should use IVHD flags to > set > up appropriate bits in the corresponding device table entries. But we are > not > doing that. That's probably okay since we do not support the pass-though > for > devices with complex properties anyway. > > -- > Andriy Gapon > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: problem with pass-through on amd
Hi Andriy, Newer AMD IOMMU has EFR capability which is missing here, something like: ivhd0: PCI cap 0x190b640f@0x40 feature:19<IOTLB,EFR> Is this an old chipset? Also ivhd has fault interrupt enabled which is very helpful in debugging: [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh irq256: ivhd0:fault0 0 irq257: ivhd1:fault0 0 Regards, Anish On Mon, Nov 13, 2017 at 12:55 PM, Andriy Gapon <a...@freebsd.org> wrote: > On 13/11/2017 22:41, Anish wrote: > > Hi Andriy, > > > >>What's suspicious is that there are no interrupts either for the > > ppt device on the host or the ahci device in the guest. > > > > I think most likely AMD IOMMU/amdvi is not enabled, by default it is > disabled > > and can be confirmed by sysctl. > > > > [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi > > hw.vmm.amdvi.domain_id: 3 > > hw.vmm.amdvi.disable_io_fault: 0 > > hw.vmm.amdvi.ptp_level: 4 > > hw.vmm.amdvi.host_ptp: 1 > > hw.vmm.amdvi.enable: 1 > > hw.vmm.amdvi.count: 2 > > > > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should > work. > > > I followed your how-to, so I didn't make this mistake :-) > > $ sysctl hw.vmm.amdvi > hw.vmm.amdvi.domain_id: 3 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 > hw.vmm.amdvi.count: 1 > > Could it be that AMD-Vi support is broken in the hardware or BIOS? > Are there any checks that I could do? > > Here is what I see in dmesg, but it is not a full dump of IVRS, of course: > $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' > Table 'IVRS' at 0xbdafaf80 > ACPI: IVRS 0xBDAFAF80 F8 (v01 AMDRD890S 00202031 AMD > ) > AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0 > ivhd0: on acpi0 > ivhd0: Unknown dev entry:0xff > ivhd0: Flag:3e<PassPW,ResPassPW,Isoc,IotlbSup,Coherent> > ivhd0: max supported paging level:7 restricting to: 4 > ivhd0: device supported range [0x0 - 0xb01] > ivhd0: device [0xa0 - 0xa0]config:d7<INIT,ExtInt,NMI,LINT0> > ivhd0: PCI cap 0x10b540f@0x40 feature:1 > pci0: at device 0.2 (no driver attached) > > > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <a...@freebsd.org > > <mailto:a...@freebsd.org>> wrote: > > > > > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > > I decided to experiment with PCI pass-through and used an AHCI > controller for > > the experiment. It seems like that partially works. The guest > detects the > > controller and attaches a driver to it, the messages (reported > capabilities, > > etc) look identical to those on the host. But the guest can not > detect any > > disks behind the controller. From what I can tell, the guest has no > problems > > accessing PCI configuration registers and a memory mapped BAR. Not > sure about > > the I/O ports. What's suspicious is that there are no interrupts > either for the > > ppt device on the host or the ahci device in the guest. Maybe that > is what > > causes the failure to see the disks. > > > > When the guest driver attaches to the controller I see these > messages on the > > host: > > ppt0: attempting to allocate 1 MSI vectors (1 supported) > > ppt0: using IRQ 265 for MSI > > and in the guest: > > ahci0: attempting to allocate 1 MSI vectors (1 supported) > > ahci0: using IRQ 279 for MSI > > > > But vmstat -i does not report any interrupts. > > > > Just in case, the controller has some complex topology where the > AHCI PCI device > > is behind two PCI-PCI bridges from the main bus. But I think that > that should > > not matter and the guest needs to see only the AHCI device and the > host handles > > the bridges. > > > > Here are some more messages from the guest: > > ahci0: port > > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef > mem > > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > > ahci0: attempting to allocate 1 MSI vectors (1 supported) > > ahci0: using IRQ 279 for MSI > > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with > FBS > > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA > 2ports > > ahcich0: at channel 0 on ahci0 > > ahcich0: Caps: CPD ESP FBSCP > > ahcich1: at channel 1 on ahci0 > > ahcich1: Caps: CPD ESP FBSCP > > ahcich0: AHCI reset... > > a
Re: problem with pass-through on amd
Hi Andriy, >What's suspicious is that there are no interrupts either for the ppt device on the host or the ahci device in the guest. I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled and can be confirmed by sysctl. [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi hw.vmm.amdvi.domain_id: 3 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 1 hw.vmm.amdvi.count: 2 If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work. Regards, Anish On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <a...@freebsd.org> wrote: > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > I decided to experiment with PCI pass-through and used an AHCI controller > for > the experiment. It seems like that partially works. The guest detects the > controller and attaches a driver to it, the messages (reported > capabilities, > etc) look identical to those on the host. But the guest can not detect any > disks behind the controller. From what I can tell, the guest has no > problems > accessing PCI configuration registers and a memory mapped BAR. Not sure > about > the I/O ports. What's suspicious is that there are no interrupts either > for the > ppt device on the host or the ahci device in the guest. Maybe that is what > causes the failure to see the disks. > > When the guest driver attaches to the controller I see these messages on > the host: > ppt0: attempting to allocate 1 MSI vectors (1 supported) > ppt0: using IRQ 265 for MSI > and in the guest: > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > > But vmstat -i does not report any interrupts. > > Just in case, the controller has some complex topology where the AHCI PCI > device > is behind two PCI-PCI bridges from the main bus. But I think that that > should > not matter and the guest needs to see only the AHCI device and the host > handles > the bridges. > > Here are some more messages from the guest: > ahci0: port > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports > ahcich0: at channel 0 on ahci0 > ahcich0: Caps: CPD ESP FBSCP > ahcich1: at channel 1 on ahci0 > ahcich1: Caps: CPD ESP FBSCP > ahcich0: AHCI reset... > ahcich0: SATA connect time=100us status=0123 > ahcich0: AHCI reset: device found > ahcich1: AHCI reset... > ahcich1: SATA connect time=100us status=0123 > ahcich1: AHCI reset: device found > ahcich1: AHCI reset: device ready after 100ms > ahcich1: Poll timeout on slot 1 port 15 > ahcich1: is cs 0002 ss rs 0002 tfd 150 serr > > cmd 00318017 > ahcich0: AHCI reset: device ready after 200ms > ahcich0: Poll timeout on slot 1 port 15 > ahcich0: is cs 0002 ss rs 0002 tfd 150 serr > > cmd 00318017 > And some more... > > Any ideas or suggestions? > Or maybe some additional information from me? > > Thank you! > > -- > Andriy Gapon > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: PCIe passthrough really that expensive?
Hi Harry, >Any hints highly appreciated! In my setup, I have dual port Intel GigE, one assigned to host and other one is used by guest root@svmhost:~ # pciconf -l |grep ppt ... ppt2@pci0:2:0:0: class=0x02 card=0x125e8086 chip=0x105e8086 rev=0x06 hdr=0x00 root@svmhost:~ # This show up as 'em0' in BSD guest with 2 vcpus. root@bsdguest:~ # pciconf -l |grep em0 em0@pci0:0:21:0:class=0x02 card=0x125e8086 chip=0x105e8086 rev=0x06 hdr=0x00 root@bsdguest:~ # Once guest is booted, ppt2 will claim interrupt resources, in this case just 1 interrupt line #265 root@svmhost:~ # vmstat -i interrupt total rate ... irq263: em0:irq0 1028705634 .. irq265: ppt2 1041187641 Total2835121 1747 Now use cpuset to route IRQ#265 to say core 0 $cpuset -l 0 -x 265 Again use cpuset to force VM[PID 1222] to run on all core except #0 root@svmhost:~ # ps PID TT STATTIME COMMAND 1222 1 I+ 5:59.45 bhyve: vm1 (bhyve) VM can run on all cores except #0. $ cpuset -l 1-3 -p 1222 You can monitor guest due to interrupts using $root@svmhost:~ # bhyvectl --get-stats --vm= --cpu= | grep external vm exits due to external interrupt 27273 root@svmhost:~ # Regards, Anish On Sun, Jun 11, 2017 at 2:51 AM, Harry Schmalzbauer <free...@omnilan.de> wrote: > Bezüglich Harry Schmalzbauer's Nachricht vom 09.06.2017 10:22 (localtime): > > Bezüglich Anish's Nachricht vom 08.06.2017 14:35 (localtime): > >> Hi Harry, > >>> I thought I'd save these expensive VM_Exits by using the passthru path. > >> Completely wrong, is it? > >> > >> It depends on which processor you are using. For example APICv was > >> introduced in IvyBridge which enabled h/w assisted localAPIC rather than > >> using s/w emulated, bhyve supports it on Intel processors. > … > > I'm still usign IvyBridge (E3v2) with this "new" machine, but haven't > > ever heard/thought about APCIv! > > It seems APICv is available on IvyBridge-EP (Xeon E5/E7v2) only, not for > E3v2 :-( > Furthermore, if I didn't miss anything in the datasheets, no currently > available E3 Xeon offers local APIC virtualization. Can somebody of the > xperts confirm that? > > > … > >> Can you run a simple experiment, assign pptdev interrupts to core that's > >> not running guest/vcpu? This will reduce #VMEXIT on vcpu which we know > >> is expensive. > > Interesting approach. But I have no idea how I should assign a PCIe > > specific core to a PCIe dev. Is it pptdev specific? The tunables in > > device.hints(5) can't be used for that, can they? > > I wasn't able to find out how to do that. > Any hints highly appreciated! > > -harry > > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: PCIe passthrough really that expensive?
Hi Harry, >I thought I'd save these expensive VM_Exits by using the passthru path. Completely wrong, is it? It depends on which processor you are using. For example APICv was introduced in IvyBridge which enabled h/w assisted localAPIC rather than using s/w emulated, bhyve supports it on Intel processors. Intel Broadwell introduced PostedInterrupt which enabled interrupt to delivered to guest directly, bypassing hypervisor[2] for passthrough devices. Emulated devices interrupt will still go through hypervisor. You can verify capability using sysctl hw.vmm.vmx. What processor you are using for these performance benchmarking? Can you run a simple experiment, assign pptdev interrupts to core that's not running guest/vcpu? This will reduce #VMEXIT on vcpu which we know is expensive. Regards, Anish On Wed, Jun 7, 2017 at 11:01 AM, Harry Schmalzbauer <free...@omnilan.de> wrote: > Hello, > > some might have noticed my numerous posts recently, mainly in > freebsd-net@, but all around the same story – replacing ESXi. So I hope > nobody minds if I ask for help again to alleviate some of my knowledge > deficiencies about PCIePassThrough. > As last resort for special VMs, I always used to have dedicated NICs via > PCIePassThrough. > But with bhyve (besides other undiscovered strange side effects) I don't > understand the results utilizing bhyve-passthru. > > Simple test: Copy iso image from NFSv4 mount via 1GbE (to null). > > Host, using if_em (hartwell): 4-8kirqs/s (8 @mtu 1500), system idle > ~99-100% > Passing this same hartwell devcie to the guest, running the identical > FreeBSD version like the host, I see 2x8kirqs/s, MTU independent, and > only 80%idle, while almost all cycles are spent in Sys (vmm). > Running the same guest in if_bridge(4)-vtnet(4) or vale(4)-vtnet(4) > deliver identical results: About 80% attainable throughput, only 80% > idle cycles. > > So interrupts triggerd by PCI devices, which are controlled via > bhyve-passthru, are as expensive as interrupts triggered by emulated > devices? > I thought I'd save these expensive VM_Exits by using the passthru path. > Completely wrong, is it? > > I haven't ever done authoritative ESXi measures, but I remember that > there was a significant saving using VMDirectPath. Big enough that I > never felt the need for measuring. Is there any implementation > difference? Some kind of intermediate interrupt moderation maybe? > > Thanks for any hints/links, > > -harry > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D10540: vmx: explicit checking for necessary invvpid/invept types
anish added inline comments. INLINE COMMENTS > ept.c:58-59 > +#define INVVPID_REQUIRED_TYPES_MASK \ > + ((1UL << (INVVPID_TYPE_SINGLE_CONTEXT + 40)) | \ > + (1UL << (INVVPID_TYPE_ALL_CONTEXTS + 40))) > +#define INVVPID_REQUIRED_TYPES_SUPPORTED(cap) \ This is cryptic, better to have separate macros for it #define INVVPID_SINGLE_CTX_SUP(cap) ((cap) & (1UL << 41)) #define INVVPID_ALL_CTX_SUP(cap) ((cap) & (1UL << 42)) REPOSITORY rS FreeBSD src repository REVISION DETAIL https://reviews.freebsd.org/D10540 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: jan.dakinevich_gmail.com, grehan, neel Cc: anish, imp, freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: debugging FreeBSD BHyve guest
I just tested bvmdebug with FreeBSD HEAD and it worked fine. -Anish *./vmrun.sh -g 6466 vm1* ** *root@bsdguest:~ # uname -a* *FreeBSD bsdguest 12.0-CURRENT FreeBSD 12.0-CURRENT #1 r314641M: Sat Apr 15 19:53:23 PDT 2017 root@svmhost:/usr/obj/root/head/sys/BHYVE amd64* *root@bsdguest:~ # sysctl debug.kdb.enter=1* *debug.kdb.enter: 0KDB: enter: sysctl debug.kdb.enter* *Received ^C; trying to switch back to ddb.* *using longjmp, hope it works!* *KDB: reentering* *KDB: stack backtrace:* *...* *Switching to ddb back-end* *[ thread pid 782 tid 100072 ]* *Stopped at kdb_sysctl_enter+0x98: movq$0,kdb_why* *db> gdb* *(ctrl-c will return control to ddb)* *Switching to gdb back-end* - >From other terminal (kgdb) target remote localhost:6466 ... (kgdb) bt #0 0x80a8c6b8 in kdb_sysctl_current (oidp=0x0, arg1=0x80, arg2=-2198296484032, req=0xfe002b51a930) at /root/FreeBSD/base/head/sys/kern/subr_kdb.c:146 #1 0x80a57820 in sysctl_new_user (req=0x1, p=0xfe002b51a930, l=18446741875413067952) at /root/FreeBSD/base/head/sys/kern/kern_sysctl.c:1729 #2 0x80a57016 in sysctl_root (arg1=0x0, arg2=-2120631584) at /root/FreeBSD/base/head/sys/kern/kern_sysctl.c:1911 #3 0x80a575a8 in sys___sysctl (td=, uap=0x3) at /root/FreeBSD/base/head/sys/kern/kern_sysctl.c:1944 #4 0x80a5741f in sysctl_old_user (req=, p=, l=) at /root/FreeBSD/base/head/sys/kern/kern_sysctl.c:1718 #5 0x80ec901a in trap (frame=0x0) at /root/FreeBSD/base/head/sys/amd64/amd64/trap.c:569 #6 0x80eaab1b in Xalign () at /root/FreeBSD/base/head/sys/amd64/amd64/exception.S:164 #7 0x00080097cb6a in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) On Wed, Apr 12, 2017 at 10:14 AM, Julian Elischer <jul...@freebsd.org> wrote: > On 11/4/17 9:30 pm, Hannes Mehnert wrote: > >> Hello, >> >> I want to debug a FreeBSD BHyve guest kernel, and tried to follow the >> instructions on the wiki >> (https://wiki.freebsd.org/bhyve/DebuggingWithGdb) -- but already the >> "device bvmdebug" is no longer recognized by make buildkernel. >> > > I hope that is not true. I use it all the time on 10. > > > >> Is there a way (and a howto?) to kgdb a FreeBSD guest? >> >> >> Thanks, >> >> hannes >> ___ >> freebsd-virtualization@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubs >> cr...@freebsd.org" >> >> > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubs > cr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Assigning PCI ethernet interfaces to VMs in bhyve
Hi Daniel, For Intel cards only condition is that in the VM, these devices should start at function 0 because of PCI enumeration, always starts at function 0. If you want to assign multiple ports to VM, it can be function 0 and any other function, for example: Host PCI dev 4/0/1 and 4/0/2 are to be assigned to guest at virtual PCI slot 10, assignment can be: 4/0/1 -> 10/0 4/0/2 -> 10/1 Regards, Anish On Sun, Apr 2, 2017 at 3:09 AM, Daniel Tihanyi <tetra...@fastmail.fm> wrote: > Hello, > > I have a FreeBSD host running VMs on bhyve. I already have a few PCI-X > ethernet cards assigned to each of them, but only with one ethernet > interface. I would like to purchase a few PCI-X Ethernet cards with 4 > interfaces. My question is if it is possible to assign the individual > interfaces on the card to different VMs? On the bhyve Wiki, there is an > example for this, but it is assigning the interfaces into the same VM, > although to different slots. I am planning to use Intel cards for this > of course. Has anyone experience in this? > Thank you very much in advance! > > Daniel Tihanyi ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: [GSoC 2017] Project "NVMe controller emulator for bhyve"
Hi Shunsuke Mie Awesome! Feel free to also ping me for any question you have. -Anish On 3/28/17 8:27 AM, Shunsuke Mie wrote: Hello bhyve hackers. My name is Shunsuke Mie. I go to graduate school in this April. I'm interesting in kernel, virtual machine, hardware design and open some of code in github ( https://github.com/himtti). These are kernel (WIP, x64 bootable) and micro controller emulator etc. I found theme "NVMe controller emualtor for bhyve" in GSoC idea list. It sounds interesting for me. I'm already contacting with mentor who is Mr. Peter Grehan. I've never experienced commit to FreeBSD and another operating system projects. but I want to join BSD community and contribute. Also after GSoC. Regards, Shunsuke Mie. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: svm/amd-v not working after upgrade from 10.3 to 11
Please share dmesg output. I have not run into this, will give you some patch to try. -Anish Sent from my iPhone > On Oct 24, 2016, at 9:00 PM, Aryeh Friedman <aryeh.fried...@gmail.com> wrote: > > After I did a clean install of 11 on a machine, HP Pavilion p7-1234, that > used to have 10.3 on it vmm and virtualbox both claim that svm/amd-v are > turned off in the BIOS. The BIOS has no settings for virtualization (and > no BIOS upgrades are available from HP). > > How do I force svm/amd-v to be used? > > -- > Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
>ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 Passthrough stub driver, part of vmm, controls the USB controller. Can you share output of /usr/sbin/devinfo from FreeBSD host, highlighting the usb mass/flash device in tree? Regards, Anish On Mon, Dec 21, 2015 at 7:44 PM, Sergey Manucharian <s...@ara-ler.com> wrote: > Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09: > > Hi Sergey, > > > > > This is pretty reproducible: > > > > > > I pass trough a PCI device (USB controller) to a Linux guest. It works > > > properly. Then I halt the VM, make sure that bhyve destroyed it and run > > > Windows guest with the same PCI device passed-through. > > > > > > Windows device manager does show the device, however, e.g. a flash > drive > > > plugged in is not presented to Windows, instead it's being processed by > > > FreeBSD. > > > > > > After that it does not work in Linux guest as well. Kernel module (vmm) > > > unloading and reloading does not help. > > > > The flash drive being processed by FreeBSD would indicate that it has > > ownership of the device. Would you be able to try a 'pciconf -vl' after > > the Linux guest exists, and after the Windows guest exits ? > > OK, I've experimented more and found that even with the same Linux VM > after halting it once, the pass-through device doesn't work when that > Linux guest is restarted again. > > It looks that both host and guest report exactly the same data before and > after restarting the guest: > > Linux - first run: > = > $ lspci -v > 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > Subsystem: Lenovo Device 21f3 > Flags: bus master, medium devsel, latency 0, IRQ 24 > Memory at c001 (64-bit, prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > > $ lsusb > Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub > Bus 001 Device 002: ID 0781:5530 SanDisk Corp. Cruzer > Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub > > FreeBSD when Linux is running first time: > = > $ pciconf -v > ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 > rev=0x04 hdr=0x00 > vendor = 'Intel Corporation' > device = '7 Series/C210 Series Chipset Family USB xHCI Host > Controller' > class = serial bus > subclass = USB > > Linux - second run: > === > $ lspci -v > 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > Subsystem: Lenovo Device 21f3 > Flags: bus master, medium devsel, latency 0, IRQ 24 > Memory at c001 (64-bit, prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > > FreeBSD when Linux is running second time: > == > $ pciconf -lv > ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 > rev=0x04 hdr=0x00 > vendor = 'Intel Corporation' > device = '7 Series/C210 Series Chipset Family USB xHCI Host > Controller' > class = serial bus > subclass = USB > > > -- > Thanks, > Sergey > > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: CDROM options in bhyve
>Is there any way to "change" it when the guest is already running?
You can specify at boot time, for example in vmrun.sh
-s $(($nextslot+1)):0,virtio-blk,${isofile}
\
'isofile' is specified using '-I' argument
$./vmrun.sh -I vm1
Once system is booted, you will see two virtio-block:
1. vtbd0 -> Boot HDD
2. vtbd1 -> CDROM image or ISO file.
On Tue, Dec 15, 2015 at 8:37 PM, Sergey Manucharian wrote:
> Hello,
>
> Currently bhyve supports attaching a cd image when started.
> Is there any way to "change" it when the guest is already running?
> Is there a way to attach the host's CD/DVD drive?
>
> E.g. it would be useful to be able creating symlink to an image file,
> and then re-point it to another one (currently does not work).
>
> Thanks,
> Sergey
>
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Re: Re: AMD processors supported under bhyve
Sorry, you need to have NRIP support in AMD processor for bhyve. I thought of supporting it but all new processors do have NRIP so got pushed out. If someone wants to do it, will be happy to help. Regards, Anish On Wed, Sep 30, 2015 at 10:58 AM, C. L. Martinez <carlopm...@gmail.com> wrote: > On Wed, Sep 30, 2015 at 5:42 PM, Jung-uk Kim <j...@freebsd.org> wrote: > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA256 > > > > On 09/30/2015 13:32, C. L. Martinez wrote: > >> On Wed, Sep 30, 2015 at 5:25 PM, Jung-uk Kim <j...@freebsd.org> > >> wrote: > >>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 > >>> > >>> On 09/30/2015 12:54, C. L. Martinez wrote: > >>>> On Tue, Sep 29, 2015 at 11:03 AM, C. L. Martinez > >>>> <carlopm...@gmail.com> wrote: > >>>>> On Tue, Sep 29, 2015 at 10:43 AM, Jason Tubnor > >>>>> <ja...@tubnor.net> wrote: > >>>>>> On 29 September 2015 at 20:07, C. L. Martinez > >>>>>> <carlopm...@gmail.com> wrote: > >>>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> Maybe a stupid question, but are AMD processors > >>>>>>> supported under FreeBSD 10.2-STABLE for bhyve or are they > >>>>>>> only supported under 11-CURRENT? > >>>>>> > >>>>>> > >>>>>> 10.2-RELEASE is the first release to contain AMD support > >>>>>> (10.1-STABLE did have support however). 11-CURRENT > >>>>>> obviously supports it and is where you'll find all the > >>>>>> latest patches/fixes when reports are made. I use TrueOS > >>>>>> monthly 11-CURRENT snapshots specifically for this reason > >>>>>> (and to have binary updates). > >>>>> > >>>>> > >>>>> Thanks Jason... My idea is to use or FreeBSD or HardenedBSD > >>>>> for this host, and if I have problems maybe I will try TrueOS > >>>>> ... > >>>> > >>>> Uhmm I am installing 10.2-STABLE and I am seeing the following > >>>> error: > >>>> > >>>> module_register_init: MOD_LOAD (vmm, 0x81d914b0, 0) > >>>> error 6 > >>>> > >>>> Does this means AMD is not supported yet or what?? > >>> > >>> We don't support some early generation CPUs. > >>> > >>> https://wiki.freebsd.org/bhyve > >>> > >>> "Barcelona" class and newer AMD processors include the required > >>> RVI extension and as with Intel processors, presence of the > >>> "POPCNT" (POPulation Count) processor feature in dmesg(8) > >>> indicates RVI support. Note some processors such as Kuma core > >>> have POPCNT feature but lacks required "NRIPS" (Next RIP Save) > >>> feature. > >>> > >>> Jung-uk Kim > >> > >> > >> Oops .. Thanks Jung-uk. But, is not currently supported for 10.X or > >> it will never supported? I am asking about using 11-CURRENT. > > > > I believe it won't be supported any time soon unless someone is > > silently working on it. :-( > > > > Jung-uk Kim > > Ok, many thanks to all for your help > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: Unsupported MSI-X configuration: 2/0/0
Another thing I should mention is that acpidump -t | grep DMAR reports nothing. It is mentioned in the documentation but I didn't see until now. DMAR/VT-d has been supported for sometime now by Intel chipset. You want to check your BIOS set-up to enable VT-d. Actually my goal is to use the wifi adapter from a VM. The bge device is just testing and I thought I should report it/ask about it. What is the error you see in VM for enabling wifi adapter? Is the corresponding driver getting attached to WiFi adapter? Then I start bhyve ... -s 10,passthru,2/0/0 ... and it spits: Unsupported MSI-X configuration: 2/0/0 and quits. ppt0@pci0:2:0:0:class=0x02 card=0x06471025 chip=0x16b514e4 rev=0x10 hdr=0x00 I will try to reproduce the problem if I get hold of this device. -Anish On Nov 29, 2014, at 6:44 AM, Nikos Vassiliadis nv...@gmx.com wrote: On 11/29/14 06:02, Anish Gupta wrote: bge0@pci0:2:0:0:class=0x02 card=0x06471025 chip=0x16b514e4 rev=0x10 hdr=0x00 sdhci_pci0@pci0:2:0:1: class=0x080501 card=0x06471025 chip=0x16bc14e4 rev=0x10 hdr=0x00 none2@pci0:2:0:2: class=0x088000 card=0x06471025 chip=0x16be14e4 rev=0x10 hdr=0x00 none3@pci0:2:0:3: class=0x088000 card=0x06471025 chip=0x16bf14e4 rev=0x10 hdr=0x00 Passthrough stub driver is part of vmm.ko and if it was present early in boot, you should see ppt@pciD:B:S:F[Domain:Bus:Slot:Function] in pciocnf list above. Given that bge driver claimed 2/0/0, most likely vmm.ko failed to load or probably was not present. Do you have vmm_load=“YES” in /boot/loader.conf as mentioned in https://wiki.freebsd.org/bhyve/pci_passthru? Yes. To make testing easier: I removed everything from loader.conf and I removed a few drivers(bge and a few others) that were built-in the kernel, so the host kernel won't use them and now I am testing with kenv and loading and unloading vmm.ko. I always see the correct ppt devices in the dmesg when vmm.ko is loaded. If you already have loader.conf configured correctly, you can try to manually load vmm.ko once system is booted and see dmesg for any problem in loading it like kernel mismatch, missing symbol[One common I usually encounter is KTRACE define in sys/modules/vmm/Makefile but kernel config is missing “option KTRACE”] etc. Everything seems ok here as well, I get nothing but the ppt devices. One thing I noticed and seems different is this: ppt0 mem 0xb343-0xb343,0xb344-0xb344 irq 16 at device 0.0 on pci2 ppt1 mem 0xb340-0xb340 irq 17 at device 0.1 on pci2 ppt2 mem 0xb341-0xb341 irq 17 at device 0.2 on pci2 ppt3 mem 0xb342-0xb342 irq 17 at device 0.3 on pci2 so ppt0 (the bge device) has two mem ranges... Another thing I should mention is that acpidump -t | grep DMAR reports nothing. It is mentioned in the documentation but I didn't see until now. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: kernel panic when starting the virtual machine
Can you try cleaning old VM before restarting VM with same name? bhyveload …testvm bhyve……….testvm bhyvectl —destroy —vm=testvm bhyveload …testvm bhyve……….testvm -Anish On Sep 26, 2014, at 10:36 AM, Denis Menshikov da...@inbox.ru wrote: Hello, for greater clarity, will present at once the order of my actions: sudo bhyload -m 256 -d FreeBSD-10.0-RELEASE-amd64-dvd1.iso testvm sudohyve -c 2 -a -A -H -P -I -m 256 -s 0,amd_hostbridge -s 1,lpc -s 2,ahci-cd,FreeBSD-10.0-RELEASE-amd64-dvd1.iso -s 3,virtio-blk,test.img -l com1,/dev/nmdm1A testvm then runs a successful installation, and as written in the manual, I add in /etc/ttys line ttyu0 /usr/libexec/getty 3wire xterm on secure. Then it should restart. In the letter to contain messages when loading during installation and at its completion. Document 1 and 2, respectively. then... sudo bhyveload -m 256 -d test.img testvm sudo bhyve -c 2 -a -A -H -P -m 256 -s 0,amd_hostbridge -s 1,lpc -s 2,virtio-blk,test.img -l com1,/dev/nmdm1A testvm dassi@20:58:~ % sudo cu -l /dev/nmdm1B -s 9600 Connected panic: AP #1 (PHY# 1) failed! cpuid = 0 KDB: stack backtrace: #0 0x808e7dd0 at kdb_backtrace+0x60 #1 0x808af8b5 at panic+0x155 #2 0x80c7e4c5 at cpu_mp_start+0x625 #3 0x808f4e8d at mp_start+0x3d #4 0x80861238 at mi_startup+0x118 #5 0x802d3e0c at btext+0x2c Uptime: 1s It seems to me that I'm doing something wrong. uname -a FreeBSD vmserver 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 -- Denis Menshikov12___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
Hi Willem, I patch against bhyve_SVM, because in the later case I get complaints that This patch is to sync bhyve_svm project branch with HEAD @263780, so you have to first merge HEAD to bhyve_svm. It will prompt you to resolve conflict in amdv.c, you should accept the changes that are in bhyve_svm and then apply the patch. bhyve HEAD exposed vlapic related interfaces along with some other changes, this patch will enable vlapic interfaces for SVM. Thanks and regards, Anish On Tue, May 20, 2014 at 2:03 PM, Willem Jan Withagen w...@digiware.nlwrote: On 15-5-2014 17:56, Anish wrote: Hi Andriy, Thanks for your interest in SVM port of bhyve. I do have patch to sync it to http://svnweb.freebsd.org/base?view=revisionrevision=263780(3/26). If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. I don't quite understand against what this patch is? Do I run it over head, to get SVM code into head? Or do I patch against bhyve_SVM, because in the later case I get complaints that fatal error: 'vlapic_priv.h' file not found # locate vlapic_priv.h /usr/srcs/head/sys/amd64/vmm/io/vlapic_priv.h So I'm guessing that is against head. But last time I looked at head, more than just the interrupt stuff was missing --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve: svm (amd-v) update
Thanks for testing it. Your patch applied cleanly to the working copy of the bhyve_svm-project. I was then able to merge with HEAD (using theirs-full on one file) and compile the kernel. So, to me it looks OK to commit. Yes, that's correct. You have to retain changes in sys/amd64/vmm/amd/amdv.c from bhyve_svm branch. Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom 1055T. It produces 200% load on the host CPU, and the emulated machine generates endlessly: Its 200% load because of 2 vcpus to guest. It stuck in loop even with single processor(1 vcpu) after PCI probing[debug messages with linux .earlyprintk=serial debug] [3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) [3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [3.691987] NET: Registered protocol family 1 [3.693382] pci :00:01.0: Activating ISA DMA hang workarounds [3.695214] PCI: CLS 64 bytes, default 64 [3.698176] Trying to unpack rootfs image as initramfs... [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] [3.505631] pnp: PnP ACPI: found 5 devices [3.506417] ACPI: bus type PNP unregistered [3.635781] pci :00:06.0: no compatible bridge window for [mem 0xfe44 -0xfe45 pref] [3.637555] pci :00:06.0: BAR 6: assigned [mem 0x8000-0x8001 pref ] [3.638986] pci :00:01.0: BAR 6: assigned [mem 0x8002-0x800207ff pref ] [3.640416] pci :00:04.0: BAR 6: assigned [mem 0x80020800-0x80020fff pref ] [3.641864] pci :00:05.0: BAR 6: assigned [mem 0x80021000-0x800217ff pref ] [3.643259] pci :00:00.0: not setting up bridge for bus :01 [3.644550] pci_bus :00: resource 4 [io 0x-0x0cf7] [3.645670] pci_bus :00: resource 5 [io 0x0d00-0x] [3.646795] pci_bus :00: resource 6 [mem 0x8000-0xdfff] [3.648031] pci_bus :00: resource 7 [mem 0xd0-0xfc] [3.650970] NET: Registered protocol family 2 [3.661491] TCP established hash table entries: 16384 (order: 6, 262144 bytes ) [3.671854] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [3.681116] TCP: Hash tables configured (established 16384 bind 16384) [3.683335] TCP: reno registered [3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) [3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [3.691987] NET: Registered protocol family 1 [3.693382] pci :00:01.0: Activating ISA DMA hang workarounds [3.695214] PCI: CLS 64 bytes, default 64 [3.698176] Trying to unpack rootfs image as initramfs... [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] [ 30.596366] Modules linked in: Additionally, It produces a lot of MSR requests: Yes, on AMD Linux is touching more MSRs( AMD specific -address 0xC00) compared to Intel. Thanks and regards, Anish On Fri, May 16, 2014 at 2:17 PM, Nils Beyer n...@renzel.net wrote: Hi Anish, Anish wrote: If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. Your patch applied cleanly to the working copy of the bhyve_svm-project. I was then able to merge with HEAD (using theirs-full on one file) and compile the kernel. So, to me it looks OK to commit. Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom 1055T. It produces 200% load on the host CPU, and the emulated machine generates endlessly: === BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1] Modules linked in: CPU 0 Modules linked in: Pid: 1, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1 BHYVE RIP: 0010:[81c5496d] [81c5496d] rc_is_bit_0+0x3a/0x69 RSP: 0018:88013e79dca0 EFLAGS: 0a96 RAX: 009c RBX: 88013e79dcc0 RCX: 880004bdcc7c RDX: 002f9dee RSI: c9000402c538 RDI: 88013e79ddb0 RBP: 8100b9ce R08: c9000402c788 R09: 81de32b8 R10: 0003 R11: R12: 0003 R13: 81157602 R14: 88013e79dc20 R15: 00d2 FS: () GS:88002820() knlGS: CS: 0010 DS: 0018 ES: 0018 CR0: 8005003b CR2: CR3: 01a85000 CR4: 07b0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process swapper (pid: 1, threadinfo 88013e79c000, task 88013e79b500) Stack: c9000402c644 00e5 03b1 88013e79ddb0 d 88013e79dcf0 81c549b7 ffb6 d c9000402c000 88013e79de30 81c554e2 Call Trace: [81c549b7] ? rc_get_bit+0x1b/0x79 [81c554e2] ? unlzma+0xa42/0xc67 [81c28ab9
Re: bhyve: svm (amd-v) update
Hi Andriy, Thanks for your interest in SVM port of bhyve. I do have patch to sync it to http://svnweb.freebsd.org/base?view=revisionrevision=263780(3/26). If patches looks good to you, we can submit it. I have been testing it on Phenom box which lacks some of newer SVM features. Thanks and regards, Anish On Thu, May 15, 2014 at 2:10 AM, Andriy Gapon a...@freebsd.org wrote: It seems that the bhyve_svm branch is a bit behind the latest interface changes in head. Is anyone working on sync-ing up the branch with the head? Some examples: - change of init method in vmm_ops - addition of resume, vlapic_init and vlapic_cleanup methods to vmm_ops - replacement of lapic_pending_intr and lapic_intr_accepted with vlapic_pending_intr and vlapic_intr_accepted - changes in struct vm_exit - VMEXIT_EPT_FAULT == VMEXIT_NESTED_FAULT Thanks! -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org bhyve_svm_HEAD_r263780.patch Description: Binary data ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: CFT: bhyve AMD snapshot
Hi Aryeh, You need to destroy the VM before creating new one with same name, so between bhyve and bhyveload, you should use bhyvectl --destroy --vm=vm name=t4n1ustl23. This fixed the problem Fatal trap 30: reserved (unknown) fault while in kernel mode which I was able to reproduce on my Phenom box. bhyveload…..vm1 bhyve…..vm1 … bhyvectl --destroy --vm=vm1 bhyveload …vm1 bhyve…..vm1 …. Thanks for your help in testing bhyve_svm and bhyve. -Anish On Feb 17, 2014, at 8:27 PM, Aryeh Friedman aryeh.fried...@gmail.com wrote: Sorry for taking so long (forgot to say orginally this is the guest) On Wed, Feb 12, 2014 at 10:30 AM, Peter Grehan gre...@freebsd.org wrote: Hi Aryeh, kernel panic during boot Can you post a verbose boot log with the panic (i.e. boot -v from the loader prompt) ?OK boot -v Booting... SMAP type=01 base= len=000a SMAP type=01 base=0010 len=1ff0 Table 'APIC' at 0xf0500 APIC: Found table at 0xf0500 APIC: Using the MADT enumerator. MADT: Found CPU APIC ID 0 ACPI ID 0: enabled SMP: Added CPU 0 (AP) Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 Preloaded elf kernel /boot/kernel/kernel at 0x817f2000. Hypervisor: Origin = bhyve bhyve Calibrating TSC clock ... TSC clock: 2595069727 Hz CPU: AMD A6-3650 APU with Radeon(tm) HD Graphics (2595.07-MHz K8-class CPU) Origin = AuthenticAMD Id = 0x300f10 Family = 0x12 Model = 0x1 Stepping = 0 Features=0x783ab7fFPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2 Features2=0x80a02001SSE3,CX16,x2APIC,POPCNT,HV AMD Features=0xe6505880s7,SYSCALL,s12,s14,NX,MMX+,FFXSR,Page1GB,LM,3DNow!+,3DNow! AMD Features2=0x37ffLAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT TSC: P-state invariant L1 2MB data TLB: 48 entries, fully associative L1 2MB instruction TLB: 16 entries, fully associative L1 4KB data TLB: 48 entries, fully associative L1 4KB instruction TLB: 32 entries, fully associative L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative L2 2MB data TLB: 128 entries, 2-way associative L2 2MB instruction TLB: 0 entries, 2-way associative L2 4KB data TLB: 1024 entries, 4-way associative L2 4KB instruction TLB: 1024 entries, 4-way associative L2 unified cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative real memory = 536870912 (512 MB) Physical memory chunk(s): 0x1000 - 0x0009bfff, 634880 bytes (155 pages) 0x0010 - 0x001f, 1048576 bytes (256 pages) 0x0181a000 - 0x1f29, 497573888 bytes (121478 pages) avail memory = 492273664 (469 MB) Event timer LAPIC quality 400 ACPI APIC Table: BHYVE BVMADT APIC: CPU 0 has ACPI ID 0 x86bios: IVT 0x00-0x0004ff at 0xf800 x86bios: SSEG 0x001000-0x001fff at 0xfe002c174000 x86bios: ROM 0x0a-0x0fefff at 0xf80a XEN: CPU 0 has VCPU ID 0 random device not loaded; using insecure entropy ULE: setup cpu 0 ACPI: RSDP 0xf0400 00024 (v02 BHYVE ) ACPI: XSDT 0xf0480 0003C (v01 BHYVE BVXSDT 0001 INTL 20130823) ACPI: APIC 0xf0500 00054 (v01 BHYVE BVMADT 0001 INTL 20130823) ACPI: FACP 0xf0600 0010C (v05 BHYVE BVFACP 0001 INTL 20130823) ACPI: DSDT 0xf0800 0012F (v02 BHYVE BVDSDT 0001 INTL 20130823) ACPI: FACS 0xf0780 00040 ACPI: HPET 0xf0740 00038 (v01 BHYVE BVHPET 0001 INTL 20130823) MADT: Found IO APIC ID 0, Interrupt 0 at 0xfec0 ioapic0: Routing external 8259A's - intpin 0 MADT: Interrupt override: source 0, irq 2 ioapic0: Routing IRQ 0 - intpin 2 MADT: Interrupt override: source 9, irq 9 ioapic0: intpin 9 trigger: level ioapic0: intpin 9 polarity: low ioapic0 Version 1.1 irqs 0-23 on motherboard cpu0 BSP: ID: 0x VER: 0x0011 LDR: 0x DFR: 0x lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff timer: 0x000100ef therm: 0x0001 err: 0x00f0 wlan: 802.11 Link Layer snd_unit_init() u=0x00ff8000 [512] d=0x7c00 [32] c=0x03ff [1024] feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25 Hardware, Intel IvyBridge+ RNG: RDRAND is not present Hardware, VIA Nehemiah Padlock RNG: VIA Padlock RNG not present kbd0 at kbdmux0 mem: memory null: null device, zero device nfslock: pseudo-device Falling back to Software, Yarrow random adaptor random: Software, Yarrow initialized
Re: Running bhyve on a AMD 1075T Phenom
Is it just de vmm driver, or is there a lot more? Yes, it only enable vmm AMD support, nothing else. -Anish On Thu, Oct 10, 2013 at 12:09 PM, Willem Jan Withagen w...@digiware.nlwrote: Op 10 okt. 2013 om 19:00 heeft Peter Grehan gre...@freebsd.org het volgende geschreven: Hi, I'm trying to test bhyve on this testsystem I have. But is does not create /dev/vmm Is this because this processor does not support the right set of features? The bhyve code in 10.0 is Intel-only. If you're willing to build FreeBSD from source, there is a project branch with AMD SVM support at: http://svnweb.freebsd.org/base/projects/bhyve_svm/ Ah, oké, building is no problem. I've been doing that ever since 1.0. :) Is it just de vmm driver, or is there a lot more? --WjW AMD Phenom(tm) II X6 1075T I have one of these - the project branch works fine with this CPU model. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve and AMD
(Anish - is nRIP required ?) Yes, nRIP is required and it will be hard to find AMD processor with SVM support that doesn't have it. We used Phenom II for bhyve port on AMD which is 2+ years old and has RVI and nRIP support. I use http://instlatx64.atw.hu/ to check SVM CPUID capability and looks like after Phenom II, all of them have both of SVM features. CPUID func 800A: 0001-0040--000F ^^ reg[0] LSB = 0x0F indicate that it has NP/RVI(Bit0) and nRIP(bit3) support. Regards, Anish On Wed, Sep 11, 2013 at 2:03 PM, Peter Grehan gre...@freebsd.org wrote: Hi, No, that's the main one. [DN:] When you say that that's the main one, do you mean that there are also other features that an AMD processor must have, besides RVI, to be compatible with bhyve, but that might be absent in some AMD processors with RVI? If so, what other features should I look for? There are some optional features that can be used, but I believe RVI is the only mandatory one (Anish - is nRIP required ?) (I'm now shopping around for dedicated server hosting for a forthcoming demo of the possible bhyve front end I mentioned earlier. To that end, I need to find an otherwise inexpensive server with a processor on which bhyve can run.) As mentioned before, it will be hard to find one that doesn't have the necessary support. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
