Re: 12.0-RELEASE und kvm/qemu using on AMD EPYC
> On Feb 11, 2019, at 6:32 AM, Christian Kratzer wrote: > > I am running freebsd vm on debian 10 buster with libvirt/kvm/qemu. > > I have several kvm hosts in the cluster. Some with various intel xeon and > others with AMD EPYC 7301 cpu. > > FreeBSD vms upto 11.2-RELEASE-p9 boo fine on all systems when passing through > the host cpu using following libvirt xml > > > > Probably not the same issue, but this sounds similar to this bug I reported a few years ago: https://bugs.launchpad.net/qemu/+bug/1329956 It's just as likely to be a bug in Qemu or KVM as it is in FreeBSD IMO. Maybe you can start by determining which CPU feature or features trigger(s) the issue. You'll have to hand-roll either some libvirt XML or qemu command lines to do it. Assuming you want to stick with XML, first grab the CPU model and features list from `virsh capabilities`. Then start with just the model without any extra features (using AMD hardware I have access to as an example, replace "Opteron_G3" as appropriate): Opteron_G3 If that works, then add the other features a few at a time until you break it. Here's an example feature list from my same hardware. Opteron_G3 Once you identify the feature or features that cause things to break, you can report back here, look for open bugs in Qemu or KVM regarding those features, and/or open new bugs. > FreeBSD 12.0-RELEASE and later hang after boot when swithcing to usermode in > start_init: trying /sbin/init > > Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on > Intel CPU > > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on > LLVM 6.0.1) > VT(vga): text 80x25 > CPU: QEMU Virtual CPU version 2.1.0 (2400.13-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x663 Family=0x6 Model=0x6 Stepping=3 > > Features=0x783fbfd > Features2=0x80a02001 > AMD Features=0x20100800 > AMD Features2=0x1 > Hypervisor: Origin = "KVMKVMKVM" > real memory = 1073741824 (1024 MB) > avail memory = 158880 (953 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) x 1 core(s) > ... > > Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on > the qemu virtual cpu > > > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on > LLVM 6.0.1) > VT(vga): text 80x25 > CPU: QEMU Virtual CPU version 2.1.0 (2200.06-MHz K8-class CPU) > Origin="AuthenticAMD" Id=0x663 Family=0x6 Model=0x6 Stepping=3 > > Features=0x783fbfd > Features2=0x80a02001 > AMD Features=0x20100800 > AMD Features2=0x65 > SVM: NAsids=16 > Hypervisor: Origin = "KVMKVMKVM" > real memory = 4294967296 (4096 MB) > avail memory = 4099080192 (3909 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) x 1 core(s) > > Following is dmesg from a succesfull boot of 11.2-RELEASE using host-model on > AMD EPYC > > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.2-RELEASE-p9 #0: Tue Feb 5 15:30:36 UTC 2019 > r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on > LLVM 6.0.0) > VT(vga): text 80x25 > CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU) > Origin="AuthenticAMD" Id=0x800f12 Family=0x17 Model=0x1 Stepping=2 > > Features=0x783fbff > > Features2=0xfff83203 > AMD Features=0x2e500800 > AMD > Features2=0x8003f7 > Structured Extended > Features=0x201c01ab > XSAVE Features=0x7 > AMD Extended Feature Extensions ID EBX=0x2001000 >
Re: The status of docker
> On Jan 22, 2019, at 11:54 PM, Sergey Zakharchenko > wrote: > > Hello there guys, > >> Not quite. I took over the docker freebsd port. Currently I am trying to >> change him to moby project on GH. > > Jochen, I wish you the best of luck. As a couple of cents, and on > behalf of Digital Loggers, Inc., I've uploaded some old patches that > we use to run an ancient version of Docker on FreeBSD: > https://github.com/digitalloggers/docker-zfs-patches . They speed up > building of large containers by not iterating over all container files > at every single stage, using ZFS diffs instead. No warranty, express > or implied, is provided on those patches; I'm sure you'll find some > edge cases where they'll break your container builds; you have been > warned. Also, forgive my Go: that was the first and hopefully the last > time I wrote something in it. > > That's not much; the real problems are with volume (e.g. single-file > "volumes" which are hard links) and networking support; they were > solved (kind of) by us by dynamically generating Dockerfiles and > adding container startup wrappers, to the point that most would say > it's too mutilated to be named Docker, so I'm afraid we aren't sharing > those for the time being. > > My answers to why on earth one would run Docker under FreeBSD instead > of using plain (or wrapped in yet another wrapper unknown to > non-FreeBSD) jails would be uniformity, simplicity, skill reuse, etc. > of quite a broad range of operations. However, Docker/Moby is really > too tied to Linux; there seem to be random attempts at overcoming that > but they don't receive enough mind share. Jetpack > (https://github.com/3ofcoins/jetpack/) could probably also benefit > from the patches (with appropriate adjustments). Interested people > willing to invest time in this should gather and decide how to move > on. Responding to a random message to share a random-ish thought: has anyone looked at Firecracker? https://firecracker-microvm.github.io/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ It's the now-open-source basis of AWS's Fargate service. The idea is to be more secure and flexible than Docker for Kubernetes-like workloads. Linux-only at the moment I'm sure but I don't see any reason that FreeBSD couldn't run inside a Firecracker microVM (using a stripped-down kernel with virtio_blk, if_vtnet, uart and either atkbdc or a custom driver for the 1-button keyboard. It's also feasible that FreeBSD could be a Firecracker host (and able to unmodified pre-packaged Linux or other microVMs) if someone with the right Go skills wanted to port the KVM bits to use VMM/bhyve. JN ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: The status of docker
> On Jan 23, 2019, at 11:26 AM, John Nielsen wrote: > >> On Jan 22, 2019, at 11:54 PM, Sergey Zakharchenko >> wrote: >> >> Hello there guys, >> >>> Not quite. I took over the docker freebsd port. Currently I am trying to >>> change him to moby project on GH. >> >> Jochen, I wish you the best of luck. As a couple of cents, and on >> behalf of Digital Loggers, Inc., I've uploaded some old patches that >> we use to run an ancient version of Docker on FreeBSD: >> https://github.com/digitalloggers/docker-zfs-patches . They speed up >> building of large containers by not iterating over all container files >> at every single stage, using ZFS diffs instead. No warranty, express >> or implied, is provided on those patches; I'm sure you'll find some >> edge cases where they'll break your container builds; you have been >> warned. Also, forgive my Go: that was the first and hopefully the last >> time I wrote something in it. >> >> That's not much; the real problems are with volume (e.g. single-file >> "volumes" which are hard links) and networking support; they were >> solved (kind of) by us by dynamically generating Dockerfiles and >> adding container startup wrappers, to the point that most would say >> it's too mutilated to be named Docker, so I'm afraid we aren't sharing >> those for the time being. >> >> My answers to why on earth one would run Docker under FreeBSD instead >> of using plain (or wrapped in yet another wrapper unknown to >> non-FreeBSD) jails would be uniformity, simplicity, skill reuse, etc. >> of quite a broad range of operations. However, Docker/Moby is really >> too tied to Linux; there seem to be random attempts at overcoming that >> but they don't receive enough mind share. Jetpack >> (https://github.com/3ofcoins/jetpack/) could probably also benefit >> from the patches (with appropriate adjustments). Interested people >> willing to invest time in this should gather and decide how to move >> on. > > Responding to a random message to share a random-ish thought: has anyone > looked at Firecracker? > > https://firecracker-microvm.github.io/ > https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ > > It's the now-open-source basis of AWS's Fargate service. The idea is to be > more secure and flexible than Docker for Kubernetes-like workloads. > Linux-only at the moment I'm sure but I don't see any reason that FreeBSD > couldn't run inside a Firecracker microVM (using a stripped-down kernel with > virtio_blk, if_vtnet, uart and either atkbdc or a custom driver for the > 1-button keyboard. It's also feasible that FreeBSD could be a Firecracker > host (and able to unmodified pre-packaged Linux or other microVMs) if someone > with the right Go skills wanted to port the KVM bits to use VMM/bhyve. S/Go/Rust ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
> On Mar 10, 2016, at 5:31 PM, Paul Vixie wrote: > > Сергей Мамонов wrote: >> Hello! >> >> Yes - zvols looks awesome. But what driver you use for it? > > virtio-blk. > >> And what >> about disk usage overhead in guest? > > ufs on zvol is faster, either in the parent or a bhyve using virtio-blk, than > zfs. at least for writing, which is my dominant work load. i expect that this > is due to zfs's compression logic rather than anything having to do with > creating/extending files to accommodate writes. > >> virtio-blk doesnt support fstrim (ahci-hd support it, but slower? "/At >> this point virtio-blk is indeed faster then ahci-hd on high IOPS/"). >> In linux && kvm we try used virtio-scsi driver with support fstrim, but >> how I see it not availble now in 10-2 stable for bhyve. >> And I not lonely with this question - >> https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-March/003442.html > > i'm just going to live without fstrim until it's supported in virtio-blk. i > know that this option isn't available to everybody, but at the moment storage > is cheap enough to waste. At the risk of getting farther off-topic.. Virtio-blk can't and won't support fstrim. If/when bhyve were to have virtio-scsi support that could work with trim, but last I was aware there wasn't a whole lot of momentum in that direction. The virtual AHCI controller in bhyve does support trim and performs remarkably well. That's what I use for my vol-backed VMs and I haven't had any complaints. Worth testing, IMO. JN ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: FreeBSD 11 - Bhyve - Spoof MAC address
> On Jan 4, 2016, at 9:32 AM, James Lodge wrote: > > Hi All, > > > I'm just getting started with Bhyve. So far everything is working as > expected. My original goal was to be running Ubuntu 12.04 i386 as I need it > for a particular project. One issue I'm having is MAC address spoofing. I'm > aware I can change the MAC address within Ubuntu but I'd like to configure > the tap interface from the host which should be possible according to man > pages. > > > Bhyve Man Page: https://www.freebsd.org/cgi/man.cgi?query=bhyve&sektion=8 > > > > The issue I have is that by setting the below, the vm boots, I can console > via null modem, but there is no eth0 interface, only the loopback. Removing > the static MAC, reboot and everything is present and correct. > > > -s 2:0,virtio-net,tap0,mac=xx:xx:xx:xx:xx:xx It looks like you are setting the MAC correctly on your bhyve command line and bhyve is running; so far so good. Is it possible that Ubuntu has a different MAC saved for its idea of eth0 and is therefore not doing what you expect? (Perhaps udev is renaming the device?) Can you run these two commands within the VM and post the output? ip link show lspci JN ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: error running grub-bhyve with -S
On Oct 22, 2015, at 11:27 AM, Peter Grehan wrote: >> # bhyvectl —vm=vm0 --destroy >> # grub-bhyve -m /images/vm0-device.map -M 1024 -r hd1 -S vm0 >> Could not setup memory for VM >> Error in initializing VM > > The -S option will force allocation of guest memory (required by passthru). > Is there 1G of free mem available on the machine when grub-bhyve was run ? Hi, thanks for the response. Yes, the machine had something like 6GB free (not including cache/buf). I also tried running grub-bhyve with smaller memory values down to and including 32MB (just to see if it would work, not because I expect my VM to run with that), and without a -M flag (which, IIRC, defaults to 256MB). I always got the same error. JN ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
error running grub-bhyve with -S
I’m running bhyve on an Intel machine running FreeBSD 11-CURRENT, updated a few days ago. I have a Debian 8.2 VM that has been running fine but now I’d like to add a PCI pass-through device. Unfortunately, when I add the required “-S” flag to my grub-bhyve command line it doesn’t work: # bhyvectl —vm=vm0 --destroy # grub-bhyve -m /images/vm0-device.map -M 1024 -r hd1 -S vm0 Could not setup memory for VM Error in initializing VM Obviously, running “bhyve” after that fails as well. What would cause that error and what can I do about it? Thanks! JN ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: AMD processors supported under bhyve
On Sep 30, 2015, at 11:22 AM, C. L. Martinez wrote: > On Wed, Sep 30, 2015 at 4:54 PM, C. L. Martinez wrote: >> On Tue, Sep 29, 2015 at 11:03 AM, C. L. Martinez >> wrote: >>> On Tue, Sep 29, 2015 at 10:43 AM, Jason Tubnor wrote: On 29 September 2015 at 20:07, C. L. Martinez wrote: > > Hi all, > > Maybe a stupid question, but are AMD processors supported under > FreeBSD 10.2-STABLE for bhyve or are they only supported under > 11-CURRENT? 10.2-RELEASE is the first release to contain AMD support (10.1-STABLE did have support however). 11-CURRENT obviously supports it and is where you'll find all the latest patches/fixes when reports are made. I use TrueOS monthly 11-CURRENT snapshots specifically for this reason (and to have binary updates). >>> >>> >>> Thanks Jason... My idea is to use or FreeBSD or HardenedBSD for this >>> host, and if I have problems maybe I will try TrueOS ... >> >> Uhmm I am installing 10.2-STABLE and I am seeing the following error: >> >> module_register_init: MOD_LOAD (vmm, 0x81d914b0, 0) error 6 >> >> Does this means AMD is not supported yet or what?? > > Uhmm .. It seems it doesn't works. When I try to launch a FreeBSD > guest install with vmrun.sh script: > > root@tstbhyve:/usr/share/examples/bhyve # sh > /usr/share/examples/bhyve/vmrun.sh -c 1 -m 512M -t tap0 -d > /dev/zvol/zroot/export/vmachines/fbsddnssrv -i -I > /export/isoimages/FreeBSD-10.2-RELEASE-amd64-disc1.iso fbsddnssrv > Launching virtual machine "fbsddnssrv" ... > vm_create: Device not configured > root@tstbhyve:/usr/share/examples/bhyve # > > dmesg out about this processor: > > [1] CPU: Quad-Core AMD Opteron(tm) Processor 1354 (2200.04-MHz K8-class CPU) > [1] Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 > [1] > Features=0x178bfbff > [1] Features2=0x802009 > [1] AMD > Features=0xee500800 > [1] AMD > Features2=0x7ff > [1] SVM: NP,NAsids=64 > [1] TSC: P-state invariant > > As you can see, virtualization support is enabled: "AMD > Features2=0x7ff” While your processor has SVM, it does not have the NRIP feature which is also required by bhyve. See e.g. http://svnweb.freebsd.org/base?view=revision&revision=272926 . That is why the VMM module wouldn’t load successfully. See the “SVM” line in your dmesg output above, versus this one from a Phenom II X4 (which is running 10-STABLE with bhyve VMs successfully): SVM: NP,NRIP,NAsids=64 JN ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve storage improvements
On Mar 27, 2015, at 11:43 AM, Alexander Motin wrote: > On 27.03.2015 18:47, John Nielsen wrote: >> Does anyone have plans (or know about any) to implement virtio-scsi support >> in bhyve? That API does support TRIM and should retain most or all of the >> low-overhead virtio goodness. > > I was thinking about that (not really a plans yet, just some thoughts), > but haven't found a good motivation and understanding of whole possible > infrastructure. > > I am not sure it worth to emulate SCSI protocol in addition to already > done ATA in ahci-hd and simple block in virtio-blk just to get another, > possibly faster then AHCI, block storage with TRIM/UNMAP. Really good > SCSI disk emulation in CTL in kernel takes about 20K lines of code. It > is pointless to duplicate it, and may be complicated for administration > to just interface to it. Indeed I've seen virtio-blk being faster then > ahci-hd in some tests, but those tests were highly synthetic. I haven't > tested it on real workloads, but I have feeling that real difference may > be not that large. If somebody wants to check -- more benchmarks are > highly welcome! From the theoretical side I'd like to notice that both > ATA and SCSI protocols on guests go through additional ATA/SCSI > infrastructure (CAM in FreeBSD), absent in case pure block virtio-blk, > so they have some more overhead by definition. Agreed, more testing is needed to see how big an effect having TRIM remain dependent on AHCI emulation would have on performance. > Main potential benefit I see from using virtio-scsi is a possibility to > pass through to client not a block device, but some real SCSI device. It > can be some local DVD writer, or remote iSCSI storage. The last would be > especially interesting for large production installations. But the main > problem I see here is booting. To make user-level loader boot the kernel > from DVD or iSCSI, bhyve has to implement its own SCSI initiator, like > small second copy of CAM in user-level. Booting kernel from some other > local block storage and then attaching to remote iSCSI storage for data > can be much easier, but it is not convenient. It is possible to nt > connect to iSCSI directly from user-level, but to make kernel CAM do it, > and then make CAM provide both block layer for booting and SCSI layer > for virtio-scsi, but I am not sure that it is very good from security > point to make host system to see virtual disks. Though may be it could > work if CAM could block kernel/GEOM access to them, alike it is done for > ZVOLs now, supporting "geom" and "dev" modes. Though that complicates > CAM and the whole infrastructure. Yes, pass-through of disk devices opens up a number of possibilities. Would it be feasible to just have bhyve broker between a pass(4) device on the host and virtio_scsi(4) in the guest? That would require the guest devices (be they local disks, iSCSI LUNs, etc) be connected to the host but I'm not sure that's a huge concern. The host will always have a high level of access to the guest's data. (Plus, there's nothing preventing a guest from doing its own iSCSI, etc. after it boots). Using the existing kernel infrastructure (CAM, iSCSI initiator, etc) would also remove the need to duplicate any of that in userland, wouldn't it? The user-level loader is necessary for now but once UEFI support exists in bhyve the external loader can go away. Any workarounds like you've described above would similarly be temporary. Using Qemu+KVM on Linux as a comparison point, there are examples of both kernel-level and user-level access by the host to guest disks. Local disk images (be they raw or qcow2) are obviously manipulated by the Qemu process from userland. RBD (Ceph/RADOS network block device) is in userland. SRP (SCSI RDMA Protocol) is in kernel. There are a few ways to do host- and/or kernel-based iSCSI. There is also a userland option if you link Qemu against libiscsi when you build it. If we do ever want userland iSCSI support, libiscsi does claim to be "pure POSIX" and to have been tested on FreeBSD, among others. JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve storage improvements (was: Several bhyve quirks)
On Mar 27, 2015, at 10:47 AM, John Nielsen wrote: > On Mar 27, 2015, at 3:46 AM, Alexander Motin wrote: > >>> I've always assumed virtio driver > emulated driver so it didn't occur >>> to me to try ahci-hd. >> >> I've just merged to FreeBSD stable/10 branch set of bhyve changes that >> should significantly improve situation in the storage area. >> >> virtio-blk driver was fixed to work asynchronously and not block virtual >> CPU, that should fix many problems with performance and interactivity. >> Both virtio-blk and ahci-hd drivers got ability to execute multiple (up >> to 8) requests same time, that should proportionally improve parallel >> random I/O performance on wide storages. At this point virtio-blk is >> indeed faster then ahci-hd on high IOPS, and they both are faster then >> before. >> >> On the other side ahci-hd driver now got TRIM support to allow freeing >> unused space on backing ZVOL. Unfortunately there is no any TRIM/UNMAP >> support in virtio-blk API to allow the same. >> >> Also both virtio-blk and ahci-hd drivers now report to guest logical and >> physical block sizes of underlying storage, that allow guests properly >> align partitions and I/Os for best compatibility and performance. > > Mav, thank you very much for all this great work and for the concise summary. > TRIM on AHCI makes it compelling for a lot of use cases despite the probable > performance hit. > > Does anyone have plans (or know about any) to implement virtio-scsi support > in bhyve? That API does support TRIM and should retain most or all of the > low-overhead virtio goodness. Okay, some belated googling reminded me that this has been listed as an "open task" in the last couple of FreeBSD quarterly status reports and discussed at one or more devsummits. I'd still be interested to know if anyone's actually contemplated or started doing the work though. :) JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve storage improvements (was: Several bhyve quirks)
On Mar 27, 2015, at 3:46 AM, Alexander Motin wrote: >> I've always assumed virtio driver > emulated driver so it didn't occur >> to me to try ahci-hd. > > I've just merged to FreeBSD stable/10 branch set of bhyve changes that > should significantly improve situation in the storage area. > > virtio-blk driver was fixed to work asynchronously and not block virtual > CPU, that should fix many problems with performance and interactivity. > Both virtio-blk and ahci-hd drivers got ability to execute multiple (up > to 8) requests same time, that should proportionally improve parallel > random I/O performance on wide storages. At this point virtio-blk is > indeed faster then ahci-hd on high IOPS, and they both are faster then > before. > > On the other side ahci-hd driver now got TRIM support to allow freeing > unused space on backing ZVOL. Unfortunately there is no any TRIM/UNMAP > support in virtio-blk API to allow the same. > > Also both virtio-blk and ahci-hd drivers now report to guest logical and > physical block sizes of underlying storage, that allow guests properly > align partitions and I/Os for best compatibility and performance. Mav, thank you very much for all this great work and for the concise summary. TRIM on AHCI makes it compelling for a lot of use cases despite the probable performance hit. Does anyone have plans (or know about any) to implement virtio-scsi support in bhyve? That API does support TRIM and should retain most or all of the low-overhead virtio goodness. JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: HEADS UP: Merging projects/bhyve_svm to HEAD
On Oct 26, 2014, at 4:40 PM, Zaphod Beeblebrox wrote: > On Sat, Oct 25, 2014 at 8:20 PM, Neel Natu wrote: > >> On Sat, Oct 25, 2014 at 3:50 PM, Zaphod Beeblebrox >> wrote: >>> I tried to integrate this patch into 10.1_RC3 and I failed. Is there a >>> timeframe to MFC this to 10.1 or 10-STABLE? >> >> It will be MFCed to 10-STABLE but I don't have a specific time frame in >> mind. >> >> I'll guess that it'll be towards the end of November but can be >> accelerated if someone has a need for this in 10-STABLE sooner. > > I would be using such a patch as soon as it was available. On a friend's > advice, I upgraded a ZFS server here at home with an AMD 9590 and 32Gig of > RAM. I'd dearly like to use it to track down the netgraph bug (see my > other recent posts), but it doesn't currently qualify. Ping? I'm also waiting with bated breath for this to be MFCed. :) Thanks for the great work! JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: NATed or Private Network Setups
> On Oct 24, 2014, at 5:08 PM, Pete Wright wrote: > > Hi All, > Has anyone deployed bhyve using NAT'd or private network setups? I've > been able to deploy bridged interfaces, but I was wondering if anyone > has done other network topologies. Is there anything preventing this > from happening code wise? I reckon it could be achieved by creating a > pseudo interface? Rather than supporting something like epair(4) directly, I believe the plan is to allow connecting a bhyve VM to a user-space virtual switch on the host. Neither is currently available to my knowledge. For a NAT setup today you should be able to add your VM's tap(4) interface as the only member of a bridge on the host and assign an IP address to the bridge interface. Services like DHCP for this virtual subnet would need to also be configured on the host in addition to whatever NAT you want to use. For an internal-only network between two or more VMs on the host you could also just use a bridge containing only the VM tap adapters. If you don't want the host to participate in the network then don't put an IP on the bridge. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: can a bhyve instance be resized? adding another virtual disk?
On Oct 15, 2014, at 5:56 AM, freebsd-li...@potato.growveg.org wrote: > Can a bhyve instance be resized? I'm talking about the disk. > Say your end user needs more diskspace. They have 32GB. They need 64GB. > How do you do it? I presume one has to stop the guest, then use truncate. > What about if the guest OS isn't freebsd, and they use say ext2 or 3? Will > ext3 start yelling at me because I've resized it? This isn't specific to FreeBSD or bhyve. Virtio block devices actually can support online resizing, but I don't know if bhyve allows that yet (I'm assuming it doesn't). In which case, yes, stop the guest and resize whatever its volume lives on (if a raw file then truncate would work), then start it up again. That's the easy part. The harder part (but much easier than it used to be) is resizing everything else. If using partitions they need to be extended first (and if using GPT the backup partition table needs to be moved first of all, a la "gpart recover".) On FreeBSD this is pretty straightforward with gpart: sysctl kern.geom.debugflags=16 gpart resize -i $number_of_last_partition $underlying_block_device You should probably reboot at this point so the kernel forgets about the old partition table. Then you get to resize the filesystem. If you are using ZFS or if you have FreeBSD 9.2 or newer and UFS then you can do it while it is mounted. Otherwise you may need to boot from another source to do the resize. For UFS use growfs a la "growfs /dev/$block_special_for_partition". For ZFS use "zpool online -e $zpool $zdev" For ext[234] on Linux, use "resize2fs /dev/$block_special". (If using LVM then you need to first resize the LV with lvextend). For XFS use "xfs_growfs $mountpoint". You can also resize btrfs but I don't know the command off the top of my head. That should be it. > What if they just want another disk? How does one refer to a > newly created virtual disk from a guest? How is it mounted to the guest? Just add a "-d /path/to/new/device" to your vmrun.sh or the corresponding -s to bhyve when you start the VM. It will show up as a new block device in the guest (e.g. /dev/vtbd1), you can partition and/or put filesystems on it as you choose and mount them normally and/or add them to /etc/fstab, etc. JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Using virtio_random
Hi all- I am happy to see that virtio_random(4) will be included in FreeBSD 10.1. To try it out, I loaded up BETA1 in a virtual machine with entropy passthrough enabled. After booting it with the virtio_random module loaded I see this in dmesg: virtio_pci3: port 0xc0a0-0xc0bf irq 10 at device 6.0 on pci0 vtrnd0: on virtio_pci3 virtio_pci3: host features: 0x7100 virtio_pci3: negotiated features: 0 So far, so good. (I think? The 'negotiated features: 0' gives me pause..) Now how do I use the driver as an entropy source (or verify that it's being used)? I don't see any difference in the output of "sysctl kern.random" with or without the driver loaded. Is something missing? Thanks, JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: libvirt and rebooting of a bhyve VM
On Aug 19, 2014, at 9:40 AM, Roman Bogorodskiy wrote: > Craig Rodrigues wrote: > >> Roman, >> >> I am using libvirt and bhyve according to this XML: >> http://libvirt.org/drvbhyve.html >> and it works great. >> I gave a presentation at BAFUG on this: >> http://www.slideshare.net/CraigRodrigues1/libvirt-bhyve >> >> I have one question. If I reboot the bhyve VM started with libvirt >> with "shutdown -r now", >> the VM shuts down, but it does not restart. >> >> How can I get the machine to reboot with "shutdown -r now" when >> started with libvirt? > > Hi Craig, > > Unfortunately, I'm not sure how to get the reboot working. Moreover, I > get the same behaviour when starting bhyve manually -- when I do a > reboot, bhyve(8) exits as soon as the system is ready to restart. > > So looks like that's a default bhyve behaviour or I'm missing something? Wasn't changing this the intention of r267216 (MFCed as r270071)? Roman, was your 10-STABLE built after that revision? JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: consistent VM hang during reboot
On Jun 13, 2014, at 4:23 PM, John Nielsen wrote:
> On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
>> I am trying to solve a problem with amd64 FreeBSD virtual machines running
>> on a Linux+KVM hypervisor. To be honest I'm not sure if the problem is in
>> FreeBSD or the hypervisor, but I'm trying to rule out the OS first.
>>
>> The _second_ time FreeBSD boots in a virtual machine with more than one
>> core, the boot hangs just before the kernel would normally print e.g. "SMP:
>> AP CPU #1 Launched!" (The last line on the console is "usbus0: 12Mbps Full
>> Speed USB v1.0", but the problem persists even without USB). The VM will
>> boot fine a first time, but running either "shutdown -r now" OR "reboot"
>> will lead to a hung second boot. Stopping and starting the host qemu-kvm
>> process is the only way to continue.
...
> Following up on the off chance anyone else is interested. I installed -HEAD
> on a host that was having the problem ("v2" Xeon CPU) and ran a FreeBSD 9 VM
> under bhyve. The problem did _not_ persist. That's not entirely conclusive
> but it does point the finger at Qemu a bit more strongly. I have filed a bug
> with them:
> https://bugs.launchpad.net/qemu/+bug/1329956
With some help from the Qemu and KVM folks I've finally made some headway. The
salient difference between the working and non-working CPUs above seems to be
support for APIC virtualization. Loading the intel_kvm module (on the Linux
host) with "enable_apicv=N" works around the reboot problem I've been having.
Since this now looks like a Linux KVM bug I won't follow up here any more, but
I wanted to wrap up the thread for the archives.
JN
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: consistent VM hang during reboot
On May 13, 2014, at 9:50 AM, John Nielsen wrote:
> On May 9, 2014, at 12:41 PM, John Nielsen wrote:
>
>> On May 8, 2014, at 12:42 PM, Andrew Duane wrote:
>>
>>> From: owner-freebsd-hack...@freebsd.org
>>> [mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of John Nielsen
>>>
>>>> On May 8, 2014, at 11:03 AM, John Baldwin wrote:
>>>>
>>>>> On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
>>>>>> I am trying to solve a problem with amd64 FreeBSD virtual machines
>>>>>> running on a Linux+KVM hypervisor. To be honest I'm not sure if the
>>>>>> problem is in FreeBSD or
>>>>> the hypervisor, but I'm trying to rule out the OS first.
>>>>>>
>>>>>> The _second_ time FreeBSD boots in a virtual machine with more than one
>>>>>> core, the boot hangs just before the kernel would normally print e.g.
>>>>>> "SMP: AP CPU #1
>>>>> Launched!" (The last line on the console is "usbus0: 12Mbps Full Speed
>>>>> USB v1.0", but the problem persists even without USB). The VM will boot
>>>>> fine a first time,
>>>>> but running either "shutdown -r now" OR "reboot" will lead to a hung
>>>>> second boot. Stopping and starting the host qemu-kvm process is the only
>>>>> way to continue.
>>>>>>
>>>>>> The problem seems to be triggered by something in the SMP portion of
>>>>>> cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual
>>>>>> "reset" button the next
>>>>> boot is fine. If I have 'kern.smp.disabled="1"' set for the initial boot
>>>>> then subsequent boots are fine (but I can only use one CPU core, of
>>>>> course). However, if I
>>>>> boot normally the first time then set 'kern.smp.disabled="1"' for the
>>>>> second (re)boot, the problem is triggered. Apparently something in the
>>>>> shutdown code is
>>>>> "poisoning the well" for the next boot.
>>>>>>
>>>>>> The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of
>>>>>> yesterday.
>>>>>>
>>>>>> This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
>>>>>>
>>>>>> --- sys/amd64/amd64/vm_machdep.c.orig2014-05-07 13:19:07.400981580
>>>>>> -0600
>>>>>> +++ sys/amd64/amd64/vm_machdep.c 2014-05-07 17:02:52.416783795 -0600
>>>>>> @@ -593,7 +593,7 @@
>>>>>> void
>>>>>> cpu_reset()
>>>>>> {
>>>>>> -#ifdef SMP
>>>>>> +#if 0
>>>>>> cpuset_t map;
>>>>>> u_int cnt;
>>>>>>
>>>>>> I've tried skipping or disabling smaller chunks of code within the #if
>>>>>> block but haven't found a consistent winner yet.
>>>>>>
>>>>>> I'm hoping the list will have suggestions on how I can further narrow
>>>>>> down the problem, or theories on what might be going on.
>>>>>
>>>>> Can you try forcing the reboot to occur on the BSP (via 'cpuset -l 0
>>>>> reboot')
>>>>> or a non-BSP ('cpuset -l 1 reboot') to see if that has any effect? It
>>>>> might
>>>>> not, but if it does it would help narrow down the code to consider.
>>>>
>>>> Hello jhb, thanks for responding.
>>>>
>>>> I tried your suggestion but unfortunately it does not make any difference.
>>>> The reboot hangs regardless of which CPU I assign the command to.
>>>>
>>>> Any other suggestions?
>>>
>>> When I was doing some early work on some of the Octeon multi-core chips, I
>>> encountered something similar. If I remember correctly, there was an issue
>>> in the shutdown sequence that did not properly halt the cores and set up
>>> the "start jump" vector. So the first core would start, and when it tried
>>> to start the next ones it would hang waiting for the ACK that they were
>>> running (since they didn't have a start vector and hence never started). I
>>> know MIPS, not AMD, so I can't say what the equivalent would be, but I'm
>&
Re: consistent VM hang during reboot
On May 9, 2014, at 12:41 PM, John Nielsen wrote:
> On May 8, 2014, at 12:42 PM, Andrew Duane wrote:
>
>> From: owner-freebsd-hack...@freebsd.org
>> [mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of John Nielsen
>>
>>> On May 8, 2014, at 11:03 AM, John Baldwin wrote:
>>>
>>>> On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
>>>>> I am trying to solve a problem with amd64 FreeBSD virtual machines
>>>>> running on a Linux+KVM hypervisor. To be honest I'm not sure if the
>>>>> problem is in FreeBSD or
>>>> the hypervisor, but I'm trying to rule out the OS first.
>>>>>
>>>>> The _second_ time FreeBSD boots in a virtual machine with more than one
>>>>> core, the boot hangs just before the kernel would normally print e.g.
>>>>> "SMP: AP CPU #1
>>>> Launched!" (The last line on the console is "usbus0: 12Mbps Full Speed USB
>>>> v1.0", but the problem persists even without USB). The VM will boot fine a
>>>> first time,
>>>> but running either "shutdown -r now" OR "reboot" will lead to a hung
>>>> second boot. Stopping and starting the host qemu-kvm process is the only
>>>> way to continue.
>>>>>
>>>>> The problem seems to be triggered by something in the SMP portion of
>>>>> cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual
>>>>> "reset" button the next
>>>> boot is fine. If I have 'kern.smp.disabled="1"' set for the initial boot
>>>> then subsequent boots are fine (but I can only use one CPU core, of
>>>> course). However, if I
>>>> boot normally the first time then set 'kern.smp.disabled="1"' for the
>>>> second (re)boot, the problem is triggered. Apparently something in the
>>>> shutdown code is
>>>> "poisoning the well" for the next boot.
>>>>>
>>>>> The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of
>>>>> yesterday.
>>>>>
>>>>> This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
>>>>>
>>>>> --- sys/amd64/amd64/vm_machdep.c.orig 2014-05-07 13:19:07.400981580
>>>>> -0600
>>>>> +++ sys/amd64/amd64/vm_machdep.c 2014-05-07 17:02:52.416783795 -0600
>>>>> @@ -593,7 +593,7 @@
>>>>> void
>>>>> cpu_reset()
>>>>> {
>>>>> -#ifdef SMP
>>>>> +#if 0
>>>>> cpuset_t map;
>>>>> u_int cnt;
>>>>>
>>>>> I've tried skipping or disabling smaller chunks of code within the #if
>>>>> block but haven't found a consistent winner yet.
>>>>>
>>>>> I'm hoping the list will have suggestions on how I can further narrow
>>>>> down the problem, or theories on what might be going on.
>>>>
>>>> Can you try forcing the reboot to occur on the BSP (via 'cpuset -l 0
>>>> reboot')
>>>> or a non-BSP ('cpuset -l 1 reboot') to see if that has any effect? It
>>>> might
>>>> not, but if it does it would help narrow down the code to consider.
>>>
>>> Hello jhb, thanks for responding.
>>>
>>> I tried your suggestion but unfortunately it does not make any difference.
>>> The reboot hangs regardless of which CPU I assign the command to.
>>>
>>> Any other suggestions?
>>
>> When I was doing some early work on some of the Octeon multi-core chips, I
>> encountered something similar. If I remember correctly, there was an issue
>> in the shutdown sequence that did not properly halt the cores and set up the
>> "start jump" vector. So the first core would start, and when it tried to
>> start the next ones it would hang waiting for the ACK that they were running
>> (since they didn't have a start vector and hence never started). I know
>> MIPS, not AMD, so I can't say what the equivalent would be, but I'm sure
>> there is one. Check that part, setting up the early state.
>>
>> If Juli and/or Adrian are reading this: do you remember anything about that,
>> something like 2 years ago?
>
> That does sound promising, would love more details if anyone can provide them.
>
> Here's another wrinkle:
>
> The KVM machine in q
Re: consistent VM hang during reboot
On May 8, 2014, at 12:42 PM, Andrew Duane wrote:
> From: owner-freebsd-hack...@freebsd.org
> [mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of John Nielsen
>
>> On May 8, 2014, at 11:03 AM, John Baldwin wrote:
>>
>>> On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
>>>> I am trying to solve a problem with amd64 FreeBSD virtual machines running
>>>> on a Linux+KVM hypervisor. To be honest I'm not sure if the problem is in
>>>> FreeBSD or
>>> the hypervisor, but I'm trying to rule out the OS first.
>>>>
>>>> The _second_ time FreeBSD boots in a virtual machine with more than one
>>>> core, the boot hangs just before the kernel would normally print e.g.
>>>> "SMP: AP CPU #1
>>> Launched!" (The last line on the console is "usbus0: 12Mbps Full Speed USB
>>> v1.0", but the problem persists even without USB). The VM will boot fine a
>>> first time,
>>> but running either "shutdown -r now" OR "reboot" will lead to a hung second
>>> boot. Stopping and starting the host qemu-kvm process is the only way to
>>> continue.
>>>>
>>>> The problem seems to be triggered by something in the SMP portion of
>>>> cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual
>>>> "reset" button the next
>>> boot is fine. If I have 'kern.smp.disabled="1"' set for the initial boot
>>> then subsequent boots are fine (but I can only use one CPU core, of
>>> course). However, if I
>>> boot normally the first time then set 'kern.smp.disabled="1"' for the
>>> second (re)boot, the problem is triggered. Apparently something in the
>>> shutdown code is
>>> "poisoning the well" for the next boot.
>>>>
>>>> The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of
>>>> yesterday.
>>>>
>>>> This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
>>>>
>>>> --- sys/amd64/amd64/vm_machdep.c.orig 2014-05-07 13:19:07.400981580
>>>> -0600
>>>> +++ sys/amd64/amd64/vm_machdep.c 2014-05-07 17:02:52.416783795 -0600
>>>> @@ -593,7 +593,7 @@
>>>> void
>>>> cpu_reset()
>>>> {
>>>> -#ifdef SMP
>>>> +#if 0
>>>>cpuset_t map;
>>>>u_int cnt;
>>>>
>>>> I've tried skipping or disabling smaller chunks of code within the #if
>>>> block but haven't found a consistent winner yet.
>>>>
>>>> I'm hoping the list will have suggestions on how I can further narrow down
>>>> the problem, or theories on what might be going on.
>>>
>>> Can you try forcing the reboot to occur on the BSP (via 'cpuset -l 0
>>> reboot')
>>> or a non-BSP ('cpuset -l 1 reboot') to see if that has any effect? It might
>>> not, but if it does it would help narrow down the code to consider.
>>
>> Hello jhb, thanks for responding.
>>
>> I tried your suggestion but unfortunately it does not make any difference.
>> The reboot hangs regardless of which CPU I assign the command to.
>>
>> Any other suggestions?
>
> When I was doing some early work on some of the Octeon multi-core chips, I
> encountered something similar. If I remember correctly, there was an issue in
> the shutdown sequence that did not properly halt the cores and set up the
> "start jump" vector. So the first core would start, and when it tried to
> start the next ones it would hang waiting for the ACK that they were running
> (since they didn't have a start vector and hence never started). I know MIPS,
> not AMD, so I can't say what the equivalent would be, but I'm sure there is
> one. Check that part, setting up the early state.
>
> If Juli and/or Adrian are reading this: do you remember anything about that,
> something like 2 years ago?
That does sound promising, would love more details if anyone can provide them.
Here's another wrinkle:
The KVM machine in question is part of a cluster of identical servers
(hardware, OS, software revisions). The problem is present on all servers in
the cluster.
I also have access to a second homogenous cluster. The OS and software
revisions on this cluster are identical to the first. The hardware is _nearly_
identical--slightly different mainboards from the same manufacturer and
slightly older CPUs. The same VMs (identical di
Re: consistent VM hang during reboot
On May 8, 2014, at 11:03 AM, John Baldwin wrote:
> On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
>> I am trying to solve a problem with amd64 FreeBSD virtual machines running
>> on a Linux+KVM hypervisor. To be honest I'm not sure if the problem is in
>> FreeBSD or
> the hypervisor, but I'm trying to rule out the OS first.
>>
>> The _second_ time FreeBSD boots in a virtual machine with more than one
>> core, the boot hangs just before the kernel would normally print e.g. "SMP:
>> AP CPU #1
> Launched!" (The last line on the console is "usbus0: 12Mbps Full Speed USB
> v1.0", but the problem persists even without USB). The VM will boot fine a
> first time,
> but running either "shutdown -r now" OR "reboot" will lead to a hung second
> boot. Stopping and starting the host qemu-kvm process is the only way to
> continue.
>>
>> The problem seems to be triggered by something in the SMP portion of
>> cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual
>> "reset" button the next
> boot is fine. If I have 'kern.smp.disabled="1"' set for the initial boot then
> subsequent boots are fine (but I can only use one CPU core, of course).
> However, if I
> boot normally the first time then set 'kern.smp.disabled="1"' for the second
> (re)boot, the problem is triggered. Apparently something in the shutdown code
> is
> "poisoning the well" for the next boot.
>>
>> The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of
>> yesterday.
>>
>> This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
>>
>> --- sys/amd64/amd64/vm_machdep.c.orig2014-05-07 13:19:07.400981580
>> -0600
>> +++ sys/amd64/amd64/vm_machdep.c 2014-05-07 17:02:52.416783795 -0600
>> @@ -593,7 +593,7 @@
>> void
>> cpu_reset()
>> {
>> -#ifdef SMP
>> +#if 0
>> cpuset_t map;
>> u_int cnt;
>>
>> I've tried skipping or disabling smaller chunks of code within the #if block
>> but haven't found a consistent winner yet.
>>
>> I'm hoping the list will have suggestions on how I can further narrow down
>> the problem, or theories on what might be going on.
>
> Can you try forcing the reboot to occur on the BSP (via 'cpuset -l 0 reboot')
> or a non-BSP ('cpuset -l 1 reboot') to see if that has any effect? It might
> not, but if it does it would help narrow down the code to consider.
Hello jhb, thanks for responding.
I tried your suggestion but unfortunately it does not make any difference. The
reboot hangs regardless of which CPU I assign the command to.
Any other suggestions?
JN
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
consistent VM hang during reboot
I am trying to solve a problem with amd64 FreeBSD virtual machines running on a
Linux+KVM hypervisor. To be honest I'm not sure if the problem is in FreeBSD or
the hypervisor, but I'm trying to rule out the OS first.
The _second_ time FreeBSD boots in a virtual machine with more than one core,
the boot hangs just before the kernel would normally print e.g. "SMP: AP CPU #1
Launched!" (The last line on the console is "usbus0: 12Mbps Full Speed USB
v1.0", but the problem persists even without USB). The VM will boot fine a
first time, but running either "shutdown -r now" OR "reboot" will lead to a
hung second boot. Stopping and starting the host qemu-kvm process is the only
way to continue.
The problem seems to be triggered by something in the SMP portion of
cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual "reset"
button the next boot is fine. If I have 'kern.smp.disabled="1"' set for the
initial boot then subsequent boots are fine (but I can only use one CPU core,
of course). However, if I boot normally the first time then set
'kern.smp.disabled="1"' for the second (re)boot, the problem is triggered.
Apparently something in the shutdown code is "poisoning the well" for the next
boot.
The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of yesterday.
This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
--- sys/amd64/amd64/vm_machdep.c.orig 2014-05-07 13:19:07.400981580 -0600
+++ sys/amd64/amd64/vm_machdep.c2014-05-07 17:02:52.416783795 -0600
@@ -593,7 +593,7 @@
void
cpu_reset()
{
-#ifdef SMP
+#if 0
cpuset_t map;
u_int cnt;
I've tried skipping or disabling smaller chunks of code within the #if block
but haven't found a consistent winner yet.
I'm hoping the list will have suggestions on how I can further narrow down the
problem, or theories on what might be going on.
Thanks!
JN
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Proposed patch for vmrun.sh
See attached patch for vmrun.sh. It contains two changes: * Use an "err" function that prints to stderr instead of "echo" for text output * Use "file -s" instead of "file" to allow it to check block devices (zvol, etc) Feel free to commit it if you find it useful. Thanks! vmrun.sh.patch Description: Binary data ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: best branch for bhyve on AMD?
On Feb 12, 2014, at 10:31 AM, Peter Grehan wrote: > Hi John, > >> Am I right in thinking that bhyve support for AMD processors is not >> yet in -STABLE? > > Yes, that's correct. > >> If so, is there working code for bhyve under AMD anywhere? Where? >> -HEAD, projects/bhyve_svm or somewhere else? > > projects/bhyve_svm > >> Is it considered experimental, stable, or something in between? > > I'd home somewhere on the stable side of experimental. I've had good success > with it on a Phenom II, but looks like at least 1 user has had issues. > >> Should it work with the above processor? > > I think your model may be predate the h/w nested paging support that bhyve > relies on (EPT, or RVI in early AMD terminology). The list of AMD CPUs with > RVI is at: > > http://support.amd.com/en-us/kb-articles/Pages/GPU120AMDRVICPUsHyperVWin8.aspx > > ... and the Athlon 64 X2 isn't on that list :( Thanks for the quick response. Yes, it looks like only K10-ish and newer have RVI, and my Brisbane is the last of the K8s. Bummer. I'll have to see if my board will take a newer processor. :) JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
best branch for bhyve on AMD?
Greetings, I am running FreeBSD 10.0-STABLE r261733 amd64 on a system with an AMD "Brisbane" CPU: CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 5200+ (2835.16-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x60fb2 Family = 0xf Model = 0x6b Stepping = 2 Features=0x178bfbff Features2=0x2001 AMD Features=0xea500800 AMD Features2=0x11f "kldload vmm" gives these kernel messages: amd_iommu_init: not implemented amdv_init: not implemented amdv_cleanup: not implemented module_register_init: MOD_LOAD (vmm, 0x80f5abf0, 0) error 6 and "# bhyveload -m 256 -d /dev/zvol/bigz/bhyvetest bhyvetest" gives: vm_create: Device not configured Am I right in thinking that bhyve support for AMD processors is not yet in -STABLE? If so, is there working code for bhyve under AMD anywhere? Where? -HEAD, projects/bhyve_svm or somewhere else? Is it considered experimental, stable, or something in between? Should it work with the above processor? Looking forward to trying bhyve on this hardware. Thanks! JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
CentOS in a FreeBSD jail howto
This is a followup to my thread "linux-only jail possible?" of a few months ago. I've been pretty successful in my efforts and now have a jail which contains only CentOS binaries and runs Python, Apache, sshd, PostgreSQL, yum, rpm, etc without problems. Since that's what I plan to use for my current project I haven't tested much else but I expect good results from many other common programs and use cases. I have documented my notes on the FreeBSD wiki. Thanks to bz@ for getting me access. Please look it over and give it a try if you have any interest: http://wiki.freebsd.org/Image/Linux/CentOS55 Please let me know if you encounter anything inaccurate or incomplete in the howto, or if you have comments or additional information about it. Please trim the CC: list on replies. Thanks! JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
