Running bhyve on a AMD 1075T Phenom
I'm trying to test bhyve on this testsystem I have. But is does not create /dev/vmm Is this because this processor does not support the right set of features? Thanx, --WjW This is what dmesg says: FreeBSD 10.0-ALPHA5 #3 r256191M: Wed Oct 9 15:25:35 CEST 2013 .. CPU: AMD Phenom(tm) II X6 1075T Processor (3013.83-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100fa0 Family = 0x10 Model = 0xa Stepping = 0 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x37ff En this is what i get when I try to load the kernel module. Oct 10 17:24:03 freetest kernel: amd_iommu_init: not implemented Oct 10 17:24:03 freetest kernel: amdv_init: not implemented Oct 10 17:24:03 freetest kernel: amdv_cleanup: not implemented Oct 10 17:24:03 freetest kernel: module_register_init: MOD_LOAD (vmm, 0x81413c40, 0) error 6 ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Running bhyve on a AMD 1075T Phenom
Op 10 okt. 2013 om 19:00 heeft Peter Grehan het volgende geschreven: > Hi, > >> I'm trying to test bhyve on this testsystem I have. >> But is does not create /dev/vmm >> >> Is this because this processor does not support the right set of features? > > The bhyve code in 10.0 is Intel-only. > > If you're willing to build FreeBSD from source, there is a project branch > with AMD SVM support at: > > http://svnweb.freebsd.org/base/projects/bhyve_svm/ Ah, oké, building is no problem. I've been doing that ever since 1.0. :) Is it just de vmm driver, or is there a lot more? --WjW > > AMD Phenom(tm) II X6 1075T > > I have one of these - the project branch works fine with this CPU model. > > later, > > Peter. > ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Running bhyve on a AMD 1075T Phenom
Op 10 okt. 2013 om 22:02 heeft Anish het volgende geschreven: > >Is it just de vmm driver, or is there a lot more? > Yes, it only enable vmm AMD support, nothing else. My questionnaire was more like: I already have the sources for head Can I just grab the vmm driver in the tree with the AMD stuff and plug that in the sources that I already have? --WjW > > > On Thu, Oct 10, 2013 at 12:09 PM, Willem Jan Withagen > wrote: > > > Op 10 okt. 2013 om 19:00 heeft Peter Grehan het volgende > geschreven: > > > Hi, > > > >> I'm trying to test bhyve on this testsystem I have. > >> But is does not create /dev/vmm > >> > >> Is this because this processor does not support the right set of features? > > > > The bhyve code in 10.0 is Intel-only. > > > > If you're willing to build FreeBSD from source, there is a project branch > > with AMD SVM support at: > > > > http://svnweb.freebsd.org/base/projects/bhyve_svm/ > > Ah, oké, building is no problem. > I've been doing that ever since 1.0. :) > > Is it just de vmm driver, or is there a lot more? > > --WjW > > > > AMD Phenom(tm) II X6 1075T > > > > I have one of these - the project branch works fine with this CPU model. > > > > later, > > > > Peter. > > > ___ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Running bhyve on a AMD 1075T Phenom
On 2013-10-15 1:51, Peter Grehan wrote: Hi Craig, There was a lot of non-trivial stuff done in the amd64 pmap code and the BHyve intel module for bhyve_npt_pmap, and these changes have not been made for the amd svm code. Right, I looked at it, and decided it was way easier to just run the svm tree Yes, that's right. Does anyone have patches for the svm code? Not yet - Anish is working on it. looking forward to test. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: CFT: bhyve AMD snapshot
On 8-2-2014 21:36, Michael Dexter wrote: > > Hello all, > > I have built and uploaded a bhyve SVM project branch snapshot of r261578 > (February 4th, 2014, MFC @ r259205) that can be found at: > > http://mirrors.nycbug.org/pub/bhyve/r261578-svm/ > > Those with Barcelona class (http://en.wikipedia.org/wiki/AMD_10h) AMD > hardware are invited to give this. So far I have had luck with my humble > Athlon(tm) II Neo N36L Dual-Core Processor (1297.87-MHz K8-class CPU). I was exactly looking for this information, since I upgraded to 10-Stable. And found no bhyve for AMD. Are there any (near future) plans to merge this with the regular stable branch? For now I'll give the above a testshot. Thanx, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: CFT: bhyve AMD snapshot
On 2014-02-10 7:36, Michael Dexter wrote: On 2/9/14 10:33 AM, Willem Jan Withagen wrote: Are there any (near future) plans to merge this with the regular stable branch? It just missed 10.0-RELEASE. :( There was a technical issue at the time and there are some performance patches on their way. Please do test! I usually prefer to build my onw. So I tried that from a both basic 10-stable as well as the svn-url I got from Peter some time ago. But - building 10-stable did not give the amd svm stuff - building the previous svn-url on my 10-stable bombs out due to some missing net/ files. Need to rerun, to get the exact error. But I'll download and try the pre-cooked iso's. And maybe it is just a matter of just loading the "other" kernel-module?? --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: CFT: bhyve AMD snapshot
On Mon, Feb 10, 2014 at 8:19 PM, Michael Dexter mailto:edi...@callfortesting.org>> wrote: Willem, On 2/10/14 7:14 AM, Willem Jan Withagen wrote: > I usually prefer to build my onw. So I tried that from a both basic > 10-stable as well as the svn-url I got from Peter some time ago. But > - building 10-stable did not give the amd svm stuff > - building the previous svn-url on my 10-stable bombs out due to some > missing net/ files. Need to rerun, to get the exact error. The snapshot was built from the SVM projects branch: http://svnweb.freebsd.org/base/projects/bhyve_svm/ I opted to build world with -j4 and kernel on -j2 on a dual-core system as per Glen Barber's suggestion of how he is building the official releases. In my case it was a src.conf which contained WITHOUT_PF = true WITH_IPFW = true After which the buildworld breaks on building iipfw, because of missing pf files. Removing WITHOUT_PF lets me build world. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Bhyve and booting a ZFS-on-root system
Hi, Just for the fun of it, I tried my build zfs-system scripts in a bhyve-vm. I use the 10.0-RELEASE iso to get to a shell, config and interface and download my script. Installing does work, and on a regular system we can go and boot into a ZFS-on-Root system. In bhyve I get the following, on reboot: - freetest# vmrun.sh -d test10zfs -t tap1 -m 2048 test10zfs Launching virtual machine "test10zfs" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (r...@freetest.digiware.nl, Tue Feb 11 10:03:58 CET 2014) \ can't load 'kernel' Type '?' for a list of commands, 'help' for more detailed help. OK -- And there is no known way (to me) to educate the loader to understand zfs disks Which loader is is used in booting? a special bhyve-loader the bootloader in the boot-partition. It seems this is the first one. If so I'm wondering if the grub-bhyve would be a trick to boot the Root-on-ZFS system But then the first question is: where do I find grub-bhyve Any suggestion is welcome... BTW: this is on a AMD system. BTW2: I read that there could be interest for a dedicated opteron-server to do bhyve development on I'm more than willing to put my test server in a slot in our datacenter for people to do testing on. CPU: AMD Phenom(tm) II X6 1075T Processor (3013.84-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100fa0 Family = 0x10 Model = 0xa Stepping = 0 Features=0x178bfbff Features2=0x802009 AMDFeatures=0xee500800 AMDFeatures2=0x37ff --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and booting a ZFS-on-root system
On 22-2-2014 22:43, Craig Rodrigues wrote: > > > > On Sat, Feb 22, 2014 at 12:16 PM, Willem Jan Withagen <mailto:w...@digiware.nl>> wrote: > > > CPU: AMD Phenom(tm) II X6 1075T Processor (3013.84-MHz K8-class CPU) > Origin = "AuthenticAMD" Id = 0x100fa0 Family = 0x10 Model = 0xa > > > Also, as Aryeh has pointed out, > you cannot run BHyve on an AMD processor unless you check out the Aryeh was the one that barged into my thread asking for the most recent snapshot of code for AMD. Which Micheal did indeed answered with the URL. > code from a special branch. Michael Dexter has provided a snapshot > from the branch: > > http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-February/002227.html I am running the vmm AMD code > but that snapshot doesn't have the ZFS on Root change. So you are telling me that ZFS on Root is available in the intel-code or more up to date current?? --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and booting a ZFS-on-root system
Op 22 feb. 2014 om 22:28 heeft Craig Rodrigues het volgende geschreven: > On Sat, Feb 22, 2014 at 12:16 PM, Willem Jan Withagen > wrote: > Hi, > > Just for the fun of it, I tried my build zfs-system scripts in a bhyve-vm. > > I use the 10.0-RELEASE iso to get to a shell, config and interface and > download my script. Installing does work, and on a regular system we can > go and boot into a ZFS-on-Root system. > > ZFS on Root inside a BHyve VM was not working until > today. If you you have a FreeBSD-CURRENT system and then update > to this revision: > http://lists.freebsd.org/pipermail/svn-src-all/2014-February/081210.html > > then you can try it out. I'm running the amdsrc tree, so just upgrading current will probably not work. Peter suggested that he would upgrade the amdsrc-tree shortly. so I'll wait just for that. I'm also looking into running a Ubuntu vm. i want to test/use zoneminder, and all my FreeBSD attempts run into trouble. So I'm also still looking for grub-bhyve. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve FreeBSD 10.0 VM crashed
Hi,
I've kicked off 3 2G vm's which do a continous
while(1) { make -j 4 buidlworld }
And one of those VM's just crashed with the message below.
Observation:
The VM is clearly not able to recover from this. And probably needs to
be destroyed and restarted... Don't know it it got as far as vmrun.sh
restarting it.
Question:
1) I expect this to be due to 3 VM's running in parallel with quite some
load. But is this something that needs to be reported to stable@?
Or does this belong here on this list.
2) Killing the VM and restarting probably also kills the memory but not
the swap? And thus would allow to extract a core? I'll see when I reboot.
--WjW
spin lock 0x814fa030 (smp rendezvous) held by 0xf80079e8d490
(tid 100097) too long
timeout stopping cpus
panic: spin lock held too long
cpuid = 3
KDB: stack backtrace:
#0 0x808e7dd0 at kdb_backtrace+0x60
#1 0x808af8b5 at panic+0x155
#2 0x8089cb71 at _mtx_lock_spin_cookie+0x241
#3 0x80c7ec7a at smp_tlb_shootdown+0x8a
#4 0x80c8066c at pmap_invalidate_range+0x2fc
#5 0x80932f6d at allocbuf+0x55d
#6 0x809317cf at brelse+0x14f
#7 0x80934b18 at bufdone+0x78
#8 0x809346c2 at biodone+0xe2
#9 0x8081c561 at g_io_schedule_up+0x1b1
#10 0x8081ca5d at g_up_procbody+0x6d
#11 0x8088198a at fork_exit+0x9a
#12 0x80c758ce at fork_trampoline+0xe
Uptime: 1h21m3s
Dumping 183 out of 2025 MB:..9%..18%..27%..35%..44%..53%..62%..79%..88%..97%
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
cpu_reset: Stopping other CPUs
timeout stopping cpus
cpu_reset: Restarting BSP
cpu_reset: Failed to restart BSP
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 23-2-2014 15:57, Craig Rodrigues wrote: >> I'm also looking into running a Ubuntu vm. i want to test/use > zoneminder, and all my FreeBSD attempts run into trouble. >> So I'm also still looking for grub-bhyve. > > grub2-bhyve is in ports: http://www.freshports.org/sysutils/grub2-bhyve/ Yup, took me some time to realize that on this experimental box portsnap was not running nightly... > If you do a web search, you'll find several HOWTO articles for using it. > Recently, Rudy wrote a nice one: > http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-February/002305.html I did have that, from his announcement, But trying to follow that, I get: --- freetest# bhyvectl --destroy --vm=ubuntu freetest# grub-bhyve -r cd0 -m ./device.map -M 2048 ubuntu GNU GRUB version 2.00 > [I select to install Ubuntu Server] freetest# bhyve -c 2 -m 2048M -H -P -A \ ? -s 0:0,hostbridge \ ? -s 1:0,lpc -s 2:0,virtio-net,tap3 \ ? -s 3,ahci-cd,ubuntu-13.10.iso -l com1,stdio \ ? -s 4,virtio-blk,/dev/zvol/zfsroot/ubuntu ubuntu vm exit rdmsr 0xc0010015, cpu 0 --- And I've seen discussions in other threads about reading/writing cpu registers where some bits should be 0, but trap when being set (Or something close to this...???) So it could again be due to the amdsrc tree differences? --WjW And again this is running on my AMD-board. /dev/zvol/zfsroot/ubuntu is a 10G zvol tap3 is created and up and ./ubuntu-13.10.iso is the ubuntu server ISO suggested by Rudy. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 23-2-2014 17:40, Peter Grehan wrote: >> vm exit rdmsr 0xc0010015, cpu 0 >> --- >> >> And I've seen discussions in other threads about reading/writing cpu >> registers where some bits should be 0, but trap when being set >> (Or something close to this...???) >> >> So it could again be due to the amdsrc tree differences? > > It's Linux accessing different MSRs on AMD as opposed to Intel. There > is an option to ignore unknown MSRs with bhyve, but that probably > shouldn't be used on AMD systems until we've worked out if the MSRs need > to be emulated or not. Hopefully that will be in the next updated :) It's a toy box here so if anything needs to be tested, worked out. Just let me know. And I'm willing to run the risk of lost work, because using the ignore option bites me So if you want me to see how we fare, .. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 23-2-2014 17:57, Peter Grehan wrote: >> It's a toy box here so if anything needs to be tested, worked out. >> Just let me know. And I'm willing to run the risk of lost work, because >> using the ignore option bites me > > You can try the "-w" option. Aryeh has reported that it works for him. -w does not exist on my bhyve bhyve: illegal option -- w --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 23-2-2014 18:09, Peter Grehan wrote: >> -w does not exist on my bhyve >> bhyve: illegal option -- w > > Hmmm, just checked and you're right. Looks like a mis-merge - this > change went into CURRENT in early December, and the most recent sync for > the SVM branch was from Jan 14 :( > > Now I'm wondering what else hasn't made it in :( Pervious time I tried diff between current and the SVM branch, but that was just a bit too much for me to sort out. If I feel up to it tonight, I'll go and digg some more, but big change it still is too much diff te really grasp. Against what current version did you merge/build the SVM branch? I'll either check that out, or try my subversion skills to see if that reduces the diff. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 23-2-2014 17:35, Aryeh Friedman wrote: > > And I've seen discussions in other threads about reading/writing cpu > registers where some bits should be 0, but trap when being set > (Or something close to this...???) > > So it could again be due to the amdsrc tree differences? > > --WjW > > And again this is running on my AMD-board. > /dev/zvol/zfsroot/ubuntu is a 10G zvol > tap3 is created and up > and ./ubuntu-13.10.iso is the ubuntu server ISO suggested by Rudy. > > > 12.04.3 runs perfectly on the amd patch with no modification. are you > setting up the device map dir correctly? I've verbosely copied the scripts that Rudy has online. (Apart from using other names, and will review them again) Right I'll download that ubuntu one as well. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 23-2-2014 18:26, Peter Grehan wrote: >> Against what current version did you merge/build the SVM branch? >> I'll either check that out, or try my subversion skills to see if that >> reduces the diff. > > My mistake - the sync was a r259205, which was from Dec 10 and > predates the -w option (r259635, Dec 17). I manually added this patch... And then I can get 13.10 to boot. First I get 20* the msr ignore message, but now I in the installer. So you might want to put that in your SVM tree to get more mileage on the code in Linux envs. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 23-2-2014 22:12, Peter Grehan wrote: >> On 23-2-2014 18:26, Peter Grehan wrote: Against what current version did you merge/build the SVM branch? I'll either check that out, or try my subversion skills to see if that reduces the diff. >>> >>> My mistake - the sync was a r259205, which was from Dec 10 and >>> predates the -w option (r259635, Dec 17). >> >> I manually added this patch... >> And then I can get 13.10 to boot. > > Great ! > >> First I get 20* the msr ignore message, but now I in the installer. > > Can you post the MSRs that were hit ? I'd be interested to see what > they are. This is the preamble before we get into the CD-installation. The process is really really slow, and consumes about 150% op CPU. aka 1,5 CPU --WjW rdmsr to register 0xc0010015 on vcpu 0 rdmsr to register 0xc0010112 on vcpu 0 rdmsr to register 0xc0010048 on vcpu 0 wrmsr to register 0xc0010048(0x400) on vcpu 0 rdmsr to register 0xc001102a on vcpu 0 wrmsr to register 0xc001102a(0) on vcpu 0 rdmsr to register 0xc0010140 on vcpu 0 rdmsr to register 0xc0010140 on vcpu 0 rdmsr to register 0xc001 on vcpu 0 rdmsr to register 0xc0010001 on vcpu 0 rdmsr to register 0xc0010002 on vcpu 0 rdmsr to register 0xc0010003 on vcpu 0 rdmsr to register 0xc0010004 on vcpu 0 wrmsr to register 0xc0010004(0x) on vcpu 0 rdmsr to register 0xc0010004 on vcpu 0 rdmsr to register 0xc0010048 on vcpu 1 wrmsr to register 0xc0010048(0x400) on vcpu 1 rdmsr to register 0xc001102a on vcpu 1 wrmsr to register 0xc001102a(0) on vcpu 1 rdmsr to register 0xc0010140 on vcpu 1 rdmsr to register 0xc0010140 on vcpu 1 rdmsr to register 0xc001001f on vcpu 1 wrmsr to register 0xc001001f(0x4000) on vcpu 1 rdmsr to register 0xc001001f on vcpu 0 wrmsr to register 0xc001001f(0x4000) on vcpu 0 rdmsr to register 0xc001103a on vcpu 0 Starting system log daemon: syslogd, klogd. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and booting a ZFS-on-root system
On 22-2-2014 22:28, Craig Rodrigues wrote: > On Sat, Feb 22, 2014 at 12:16 PM, Willem Jan Withagen <mailto:w...@digiware.nl>> wrote: > > Hi, > > Just for the fun of it, I tried my build zfs-system scripts in a > bhyve-vm. > > I use the 10.0-RELEASE iso to get to a shell, config and interface and > download my script. Installing does work, and on a regular system we can > go and boot into a ZFS-on-Root system. > > > ZFS on Root inside a BHyve VM was not working until > today. If you you have a FreeBSD-CURRENT system and then update > to this revision: > http://lists.freebsd.org/pipermail/svn-src-all/2014-February/081210.html Good pointer. Added this patch to my tree, recompiled buildworld, and installed. And I'm up and running a Root on ZFS vm So that works too. Now If only i figured out why my linux install is running horibily slow. I sort of have the feeling that CD/iso access is the crulpit. Every time new things are being loaded it is like molasses. Hopefully the vm without the CD does perform. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve and Ubuntu booting
On 2014-02-23 22:04, Willem Jan Withagen wrote: On 23-2-2014 18:26, Peter Grehan wrote: Against what current version did you merge/build the SVM branch? I'll either check that out, or try my subversion skills to see if that reduces the diff. My mistake - the sync was a r259205, which was from Dec 10 and predates the -w option (r259635, Dec 17). I manually added this patch... And then I can get 13.10 to boot. First I get 20* the msr ignore message, but now I in the installer. So you might want to put that in your SVM tree to get more mileage on the code in Linux envs. std-disclaimer: Running on AMD processor with SVM branch I've started the ubuntu install again, since last night it went into the dark hours. And my laptop shutdown the connection while I used console. So no recovery to the prompts :( Now I'm trying 2 Ubuntu installs: 13.10 12.04 And both run equally slow. And generate msr access that would otherwise be trapped. (running with 2 vCPUs, and 2G mem). From top: 67705 root 5 52 0 2081M 429M CPU1 1 74:38 147.66% bhyve: ubuntu-13.10 67881 root 5 52 0 2077M 194M CPU3 3 67:20 137.16% bhyve: ubuntu-12.04 And the load is always up in the 150% So obviously these 2 installers are doing a form sort of busy waiting? Are other people seeing the same on AMD cpus or on Intel CPUs? Thanx, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: svm (amd-v) update
On 18-5-2014 16:44, Anish wrote: > Thanks for testing it. >> Your patch applied cleanly to the working copy of the "bhyve_svm"-project. > I was then able to merge with HEAD > (using "theirs-full" on one file) and compile the kernel. So, to me it > looks OK to commit. > Yes, that's correct. You have to retain changes in sys/amd64/vmm/amd/amdv.c > from bhyve_svm branch. > >> Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom > 1055T. It produces 200% load on the > host CPU, and the emulated machine generates endlessly: > Its 200% load because of 2 vcpus to guest. It stuck in loop even with > single processor(1 vcpu) after PCI probing[debug messages with linux > .earlyprintk=serial debug] > > [3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) > > [3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) > > [3.691987] NET: Registered protocol family 1 > > [3.693382] pci :00:01.0: Activating ISA DMA hang workarounds > > [3.695214] PCI: CLS 64 bytes, default 64 > > [3.698176] Trying to unpack rootfs image as initramfs... > > [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] > > [3.505631] pnp: PnP ACPI: found 5 devices > > [3.506417] ACPI: bus type PNP unregistered > > [3.635781] pci :00:06.0: no compatible bridge window for [mem > 0xfe44 > > -0xfe45 pref] > > [3.637555] pci :00:06.0: BAR 6: assigned [mem 0x8000-0x8001 > pref > > ] > > [3.638986] pci :00:01.0: BAR 6: assigned [mem 0x8002-0x800207ff > pref > > ] > > [3.640416] pci :00:04.0: BAR 6: assigned [mem 0x80020800-0x80020fff > pref > > ] > > [3.641864] pci :00:05.0: BAR 6: assigned [mem 0x80021000-0x800217ff > pref > > ] > > [3.643259] pci :00:00.0: not setting up bridge for bus :01 > > [3.644550] pci_bus :00: resource 4 [io 0x-0x0cf7] > > [3.645670] pci_bus :00: resource 5 [io 0x0d00-0x] > > [3.646795] pci_bus :00: resource 6 [mem 0x8000-0xdfff] > > [3.648031] pci_bus :00: resource 7 [mem 0xd0-0xfc] > > [3.650970] NET: Registered protocol family 2 > > [3.661491] TCP established hash table entries: 16384 (order: 6, 262144 > bytes > > ) > > [3.671854] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) > > [3.681116] TCP: Hash tables configured (established 16384 bind 16384) > > [3.683335] TCP: reno registered > > [3.684243] UDP hash table entries: 1024 (order: 3, 32768 bytes) > > [3.686484] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) > > [3.691987] NET: Registered protocol family 1 > > [3.693382] pci :00:01.0: Activating ISA DMA hang workarounds > > [3.695214] PCI: CLS 64 bytes, default 64 > > [3.698176] Trying to unpack rootfs image as initramfs... > > [ 30.595279] BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:1] > > [ 30.596366] Modules linked in: >> Additionally, It produces a lot of MSR requests: > Yes, on AMD Linux is touching more MSRs( AMD specific -address 0xC00) > compared to Intel. > > Thanks and regards, > Anish > > > On Fri, May 16, 2014 at 2:17 PM, Nils Beyer wrote: > >> Hi Anish, >> >> Anish wrote: >>> If patches looks good to you, we can submit it. I have been testing it on >>> Phenom box which lacks some of newer SVM features. >> >> Your patch applied cleanly to the working copy of the "bhyve_svm"-project. >> I was then able to merge with HEAD >> (using "theirs-full" on one file) and compile the kernel. So, to me it >> looks OK to commit. >> >> Unfortunately, I am still not able to boot CentOS 6.5 using my Phenom >> 1055T. It produces 200% load on the >> host CPU, and the emulated machine generates endlessly: >> >> === >> BUG: soft lockup - CPU#0 stuck for 67s! [swapper:1] >> Modules linked in: >> CPU 0 >> Modules linked in: >> >> Pid: 1, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1 BHYVE And more... >> I'd love to see CentOS perfectly running on my Phenom as it runs perfectly >> on an Intel i3. >> >> If you need any further information/debug, please let me know... I've been trying to get Ubuntu, CentOS and like to run on AMDs, and currently I'm compiling a kernel, but it goes dirt slow. Attached a patch I have to debug more of the MSRs and it does what I do to get the TSC running It helps, but things are still like molases. For Ubuntu I also needed to fix part of the AHCI code since it bails out on ATA FLUSH. I'm going to take a look at the recently posted diff which should get bhyve_svm in line with head. And see if that speeds up my Ubuntu kernels. --WjW Index: sys/amd64/vmm/amd/svm.c === --- sys/amd64/vmm/amd/svm.c (revision 264582) +++ sys/amd64/vmm/amd/svm.c (working copy) @@ -82,6 +82,8 @@ static
Re: bhyve: svm (amd-v) update
On 15-5-2014 17:56, Anish wrote: > Hi Andriy, > Thanks for your interest in SVM port of bhyve. I do have patch to sync it > to http://svnweb.freebsd.org/base?view=revision&revision=263780(3/26). If > patches looks good to you, we can submit it. I have been testing it on > Phenom box which lacks some of newer SVM features. I don't quite understand against what this patch is? Do I run it over head, to get SVM code into head? Or do I patch against bhyve_SVM, because in the later case I get complaints that fatal error: 'vlapic_priv.h' file not found # locate vlapic_priv.h /usr/srcs/head/sys/amd64/vmm/io/vlapic_priv.h So I'm guessing that is against head. But last time I looked at head, more than just the interrupt stuff was missing --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: svm (amd-v) update
On 2014-05-21 6:59, Anish wrote:
Hi Willem,
Thanks for sharing your patch. I see that you have gone further than
what I have. Are you able to boot Linux with these changes?
I'm running ubuntu-{12.x,14.04} with this and I'm able to compile the
linux kernel. Peter helped a lot in getting it going.
Although that is already takeing something like a week... So it is slow.
You might also need my ahci hack/patch, since linux uses a lot of
flushes and there is a 'bug' in the FreeBSD end of ahci that does not
really cooperate with the way ubuntu-14.04 calls it.
So I've emulated a flush to a NO-OP. which then results in a panic
because the ahci command-list is empty. So I 'fixed' that too, and then
things run, albeit slow.
CentOS it tried at the begining, but I'm just not a RH fan, so when that
was too slow, I gave up.
Give some time and I'll try and extract my ahci-hack-patch, and that is
the last part that'll let you get on your way, booting linux.
I do not have a lot of time to look at this, so I'm already a few weeks
stuck on speeding up the linux-kernels.
Also because I do not know enough of the inards of Linux..
--WjW
Thanks and regards,
Anish
On Tue, May 20, 2014 at 2:03 PM, Willem Jan Withagen mailto:w...@digiware.nl>> wrote:
On 15-5-2014 17:56, Anish wrote:
> Hi Andriy,
> Thanks for your interest in SVM port of bhyve. I do have patch
to sync it
> to
http://svnweb.freebsd.org/base?view=revision&revision=263780(3/26). If
> patches looks good to you, we can submit it. I have been testing
it on
> Phenom box which lacks some of newer SVM features.
I don't quite understand against what this patch is?
Do I run it over head, to get SVM code into head?
Or do I patch against bhyve_SVM, because in the later case I get
complaints that
fatal error: 'vlapic_priv.h' file not found
# locate vlapic_priv.h
/usr/srcs/head/sys/amd64/vmm/io/vlapic_priv.h
So I'm guessing that is against head.
But last time I looked at head, more than just the interrupt stuff was
missing
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: svm (amd-v) update
On 2014-05-21 6:55, Anish wrote: Hi Willem, > I patch against bhyve_SVM, because in the later case I get complaints that This patch is to sync bhyve_svm project branch with HEAD @263780, so you have to first merge HEAD to bhyve_svm. It will prompt you to resolve conflict in amdv.c, you should accept the changes that are in bhyve_svm and then apply the patch. bhyve HEAD exposed vlapic related interfaces along with some other changes, this patch will enable vlapic interfaces for SVM. I'd be interested in the vlapic to if that helps the speed. But you can help me a lot if you give me the SVN commands to do what you described above. I can fetch a clean bhyve_svm brach, but that is as far as my svn goes. I'll see if I can get my patches in as well. Thanx, --WjW Thanks and regards, Anish On Tue, May 20, 2014 at 2:03 PM, Willem Jan Withagen mailto:w...@digiware.nl>> wrote: On 15-5-2014 17:56, Anish wrote: > Hi Andriy, > Thanks for your interest in SVM port of bhyve. I do have patch to sync it > to http://svnweb.freebsd.org/base?view=revision&revision=263780(3/26). If > patches looks good to you, we can submit it. I have been testing it on > Phenom box which lacks some of newer SVM features. I don't quite understand against what this patch is? Do I run it over head, to get SVM code into head? Or do I patch against bhyve_SVM, because in the later case I get complaints that fatal error: 'vlapic_priv.h' file not found # locate vlapic_priv.h /usr/srcs/head/sys/amd64/vmm/io/vlapic_priv.h So I'm guessing that is against head. But last time I looked at head, more than just the interrupt stuff was missing --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: svm (amd-v) update
On 2014-05-21 11:31, Nils Beyer wrote:
Hi Willem,
Willem Jan Withagen wrote:
I'd be interested in the vlapic to if that helps the speed.
But you can help me a lot if you give me the SVN commands to do what you
described above.
These were my steps:
0) mv /usr/src /usr/src.bak
1) svnlite co svn://svn.freebsd.org/base/projects/bhyve_svm /usr/src
2) cd /usr/src
3) patch -p4 < /tmp/bhyve_svm_HEAD_r263780.patch
4) svnlite merge svn://svn.freebsd.org/base/head
one conflict in file "amdv.c" - enter "mf" (mine-full); in my previous
post, I mistakenly said "theirs-full"; what is, of course, wrong.
5) manually patch "amdv.c" with:
<--- SNIP ->
Index: sys/amd64/vmm/amd/amdv.c
===
--- sys/amd64/vmm/amd/amdv.c(revision 266491)
+++ sys/amd64/vmm/amd/amdv.c(working copy)
@@ -99,7 +99,7 @@
}
static void
-amd_iommu_add_device(void *domain, int bus, int slot, int func)
+amd_iommu_add_device(void *domain, uint16_t rid)
{
printf("amd_iommu_add_device: not implemented\n");
@@ -106,7 +106,7 @@
}
static void
-amd_iommu_remove_device(void *domain, int bus, int slot, int func)
+amd_iommu_remove_device(void *domain, uint16_t rid)
{
printf("amd_iommu_remove_device: not implemented\n");
<--- SNIP ->
6) should be fine now to compile and to integrate your patches
Thanx,
Very helpfull... And lets see if we can get amd-v (back) up to speed as
well.
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
bhyve on AMD, linux and high load
Hoi, Just a point on the timeline I think somebody asked why his CPU load was so high on AMD running linux. I've completed compiling a "fresh" linux-kernel on my Ubuntu 14.04 system, installed and rebooted it. And where previously a Linux-kernel would drain the CPUs it got assigned to the max... Aka 2 CPU would drive the load +2 and top would diskplay a 200% cpu. With the new kernel, that is no longer the case. On a idle vm the CPU load is like 6-7% At the moment I'm running: linux-image-3.13.0-24-generic_3.13.0-24.46_amd64 So somewhere in the interaction between bhyve idle detection and the older linux kernel things do not match... --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve on AMD, linux and high load
On 2014-05-22 18:24, Nils Beyer wrote: Hi Willem, Willem Jan Withagen wrote: [...] With the new kernel, that is no longer the case. On a idle vm the CPU load is like 6-7% At the moment I'm running: linux-image-3.13.0-24-generic_3.13.0-24.46_amd64 Now, that looks promising. Is that with or without your own bhyve-/SVM-patches? If that's with your own patches, have you already integrated Anish's patches in them? I've started building a linux kernel like over a week ago So that is with my patches on the "basic" bhyve-svm. I've just completed merging and patching with Anish patches... ONce that seems to run for my freebsd vm's, I'll start merging my patches. And hope that it still works. And best would be that the vlapic stuff gives a extra bit of speed, because we're not quite there yet. I'll let you know how I fare --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve on AMD, linux and high load
On 22-5-2014 21:29, Nils Beyer wrote:
> Hi,
>
> Willem Jan Withagen wrote:
>> I've just completed merging and patching with Anish patches...
>> ONce that seems to run for my freebsd vm's, I'll start merging my
>> patches. And hope that it still works.
>
> Sounds good. I've tried some Linux versions ranging from 2.6.32, 3.10.x,
> 3.11.10 to 3.12.8 using an Anish's patched SVM+HEAD variant. They all
> show the same behaviour being stucked during boot.
>
> I haven't tried your patches yet, though.
Well I'm running Anish SVM+HEAD and I'm no longer able to my old FreeBSD
vm's. And I can't even get vmrun.sh to build new ones
It just doesn't show anything on the commandline output, even if I go:
-l com1,stdio
Strange part is that the Linux 14.04 VM just starts fine
So I have a complete reverse situation.
Linux runs, but I have a hard time getting FreeBSD to run.
>> And best would be that the vlapic stuff gives a extra bit of speed,
>> because we're not quite there yet.
>
> I suppose you start your bhyved Linux instances using the "-A" switch (ACPI
> tables), right? If you feel interested, you can try to start them without the
> "-A" switch. For me, the boot process is now stucked at "Calibrating delay
> loop". As far as I understand, it probably has something to do with a borked
> "jiffies" value and that has probably something to do with a borked "IRQ 0"
> that modifes that "jiffies" value at every tick.
Yes I use -A on all my boots.
> But these are just wild guesses as I understand much too little about
> kernels, virtualizing and stuff.
I'm the last one suggesting that I know more that just a little bit, but
I get by..
> For what it's worth, using an Intel i3 and without the "-A" switch, the
> bhyved Linux instance just boots fine as it does with the "-A" switch.
One of the reasons tinkering with AMD is that I don't have an i{3,5,7}
cpus available.
> BTW: you haven't the oppurtunity to try bhyve SVN on a Barcelona- or later
> class Opteron, have you?
Nope, just got a bunch of boards leftover from a previous project. And
they are all:
CPU: AMD Phenom(tm) II X6 1075T Processor (3013.83-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x100fa0 Family=0x10 Model=0xa Stepping=0
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve on AMD, linux and high load
On 22-5-2014 21:29, Nils Beyer wrote: > I suppose you start your bhyved Linux instances using the "-A" switch (ACPI > tables), right? If you feel interested, you can try to start them without the > "-A" switch. For me, the boot process is now stucked at "Calibrating delay > loop". As far as I understand, it probably has something to do with a borked > "jiffies" value and that has probably something to do with a borked "IRQ 0" > that modifes that "jiffies" value at every tick. Same here --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: svm (amd-v) update
On 15-5-2014 17:56, Anish wrote: > Hi Andriy, > Thanks for your interest in SVM port of bhyve. I do have patch to sync it > to http://svnweb.freebsd.org/base?view=revision&revision=263780(3/26). If > patches looks good to you, we can submit it. I have been testing it on > Phenom box which lacks some of newer SVM features. Hi, With this patchset I see inb(0x40) calls coming thru to the bhyve VM, where I would expect them to stay down in the VMM dev-driver and emulate the vatpit for that VM. default_inout: port 64, in, bytes 1 default_inout: port 64, in, bytes 1 default_inout: port 67, out, bytes 1 Any particular reason that the IO for that register is not intercepted and ends up in userspace? Any suggestions what to watch for? Not really strong in debuging kernel modules. Thanx, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve ahci issue
On 25-5-2014 3:50, Steve Wills wrote: > Hi, > > A bhyve VM I run had this: > > pid 79784 (bhyve), uid 0: exited on signal 6 (core dumped) > > and the only thing on the console was this: > > ahcich0: Timeout on slot 27 port 0 > ahcich0: is cs ss rs tfd 50 serr > cmd 1000c017 > (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 48 ca a6 6a 40 0b 00 00 00 > 00 00 > (ada0:ahcich0:0:0:0): CAM status: Command timeout > (ada0:ahcich0:0:0:0): Retrying command > Assertion failed: (aior != NULL), function ahci_handle_dma, file > /usr/src/usr.sbin/bhyve/pci_ahci.c, line 493. I've converted this Assert into a regular test of an empty list, and up'till I have not run into trouble I needed to remove/convert it, because running Linux with ahci on a AMD cpu ran into ATA FLUSH problems.. So I'm currently ignoring Linux ATA FLUSH. But perhaps others can share some light on why this as to be an Assert in this corner of the code instead of just a regular test. I also do get FPDMA timeouts under Linux with AHCI, but they do not panic, again because of the "fix". --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Trying to run DragonFly under bhyve
When I do this under AMD I get: Copyright (c) 2003-2013 The DragonFly Project. Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. Failed to emulate instruction at 0x8096052c Abort trap (core dumped) To conclude which instruction this is, I need to get at the bytes of that instruction... but that stays hidden in the vmm-driver. Any easy way to get this back into userspace? --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Trying to run DragonFly under bhyve
On 2014-05-28 2:22, Tycho Nightingale wrote: On May 27, 2014, at 8:14 PM, Willem Jan Withagen wrote: When I do this under AMD I get: Copyright (c) 2003-2013 The DragonFly Project. Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. Failed to emulate instruction at 0x8096052c Abort trap (core dumped) To conclude which instruction this is, I need to get at the bytes of that instruction... but that stays hidden in the vmm-driver. Any easy way to get this back into userspace? You could try 'objdump -d' on a copy of the guest's kernel to find the relevant instruction. Would that work? I'd expect things to be reloaded and shuffled around... But I'm going to take a peek at the loader code... Since Peter suggests that DFLY has a loader that is not compatible with what the bhyve loader does at the moment. And further: On 2014-05-28 6:34, Anish wrote:> >Failed to emulate instruction at 0x8096052c > Abort trap (core dumped) > > You can also analyze the coredump of bhyve > $gdb /usr/sbin/bhyve bhyve.core > > Look at vie->inst[] from one of stack frame. > > -Anish I'll give it a spin. Thanx, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Bheve: Slow linux syscalls on AMD
Hi,
In my quest to find why Linux is so slow on AMD I executed
strace -T -c ls -aslR / > /dev/null
On both a Linux running on a real CPU (just a mere Celeron):
% time seconds usecs/call callserrors syscall
-- --- --- - -
25.260.235827 1180706 lstat64
23.060.215235 1255949255942 getxattr
17.650.164726 6 29720 getdents64
13.410.125168 1180698 stat64
12.750.119004 1130339130339 lgetxattr
4.280.039929 2 2265686 readlink
2.080.019399 1 14859 openat
0.700.006569 0 14910 close
0.540.005085 0 14909 fstat64
0.150.001417 0 3135 write
0.100.000896 0 2831 clock_gettime
0.030.000263 38335 open
0.000.00 018 read
0.000.00 0 1 execve
0.000.00 01212 access
0.000.00 0 8 brk
0.000.00 0 3 3 ioctl
0.000.00 035 munmap
0.000.00 012 mprotect
0.000.00 057 _llseek
0.000.00 067 mmap2
0.000.00 0 1 set_thread_area
0.000.00 0 2 2 statfs64
0.000.00 0 4 socket
0.000.00 0 4 4 connect
-- --- --- - -
100.000.933518851019386423 total
And on a virtualized Linux:
% time seconds usecs/call callserrors syscall
-- --- --- - -
55.83 30.198355 666 45331 getdents
17.999.733976 429 22665 5 openat
16.628.988507 396 22674 close
9.104.920301 217 22673 fstat
0.370.201331 223 901 write
0.030.015656 4473523 open
0.020.008853 32827 mmap
0.020.008239 35823 brk
0.010.007008 50114 mprotect
0.010.003540 35410 read
0.010.003146 393 8 8 access
0.000.001808 452 4 munmap
0.000.000660 660 1 mremap
0.000.000615 308 2 2 statfs
0.000.000582 194 3 3 ioctl
0.000.000254 254 1 execve
0.000.000236 236 1 stat
0.000.000193 193 1 arch_prctl
-- --- --- - -
100.00 54.09326011437441 total
One cannot really compare the factual results, but what is significat is
the orders of magnitude difference in syscall time.
So I ripped this from the net:
Which turns out to be a BAD test, since linux caches the getpid value.
#include
#include
#include
#include
int foo(){
return(10);
}
long nanosec(struct timeval t){ /* Calculate nanoseconds in a timeval
structure */
return((t.tv_sec*100+t.tv_usec)*1000);
}
main(){
int i,j,res;
long N_iterations=100; /* A million iterations */
float avgTimeSysCall, avgTimeFuncCall;
struct timeval t1, t2;
/* Find average time for System call */
res=gettimeofday(&t1,NULL); assert(res==0);
for (i=0;iBut even with this bad test, it is clear that the overhead and slowness
in syscalls is killing the performance
So:
1) I'm looking for a better basic syscall in Linux that is not cache,
faked or otherwise tweaked to nog give what I want.
Would really be nice if there was a NOP_syscall, just go in and out of
kernel space.
2) I'm looking for an explanation (or better yet) a fix for the slow
systemcalls
3) Can somebody do the same test on an intel plaform and see what the
results are.
Thanx,
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 30-5-2014 23:27, Nils Beyer wrote:
> Hi Willem,
>
> Willem Jan Withagen wrote:
>> 1) I'm looking for a better basic syscall in Linux that is not cache,
>> faked or otherwise tweaked to nog give what I want.
>> Would really be nice if there was a NOP_syscall, just go in and out of
>> kernel space.
>
> Hmm, I've tried your test with "getuid". Seems not to be cached. Here's the
> diff:
>
> ===
> # diff 0.orig.c 0.c
> 24c24
> < j=getpid();
> ---
>> (void)getuid();
> 38c38
> <printf("Average time for System call getpid : %f\n",avgTimeSysCall);
> ---
>>printf("Average time for System call getuid : %f\n",avgTimeSysCall);
> ===
>
>
> And here is the result:
>
> ===
> # strace -c ./0
> Average time for System call getuid : 10564.581055
> Average time for Function call : 2.285000
> % time seconds usecs/call callserrors syscall
> -- --- --- - -
> 100.000.004590 0 100 getuid
> 0.000.00 0 1 read
> 0.000.00 0 2 write
> 0.000.00 0 2 open
> 0.000.00 0 2 close
> 0.000.00 0 3 fstat
> 0.000.00 0 9 mmap
> 0.000.00 0 3 mprotect
> 0.000.00 0 1 munmap
> 0.000.00 0 1 brk
> 0.000.00 0 1 1 access
> 0.000.00 0 1 execve
> 0.000.00 0 1 arch_prctl
> -- --- --- - -
> 100.000.004590 127 1 total
> ===
Looks a bit like mine on i386 real hardware:
root@ubuntu-i386-14:~/src/tests# strace -c ./syscall-getuid
Average time for System call getuid : 272.023010
Average time for Function call : 7.169000
% time seconds usecs/call callserrors syscall
-- --- --- - -
100.000.018939 0 100 getuid32
0.000.00 0 1 read
0.000.00 0 2 write
0.000.00 0 2 open
0.000.00 0 2 close
0.000.00 0 1 execve
0.000.00 0 3 3 access
0.000.00 0 1 brk
0.000.00 0 4 gettimeofday
0.000.00 0 1 munmap
0.000.00 0 3 mprotect
0.000.00 0 7 mmap2
0.000.00 0 3 fstat64
0.000.00 0 1 set_thread_area
-- --- --- - -
100.000.018939 131 3 total
>> 3) Can somebody do the same test on an intel plaform and see what the
>> results are.
>
> Here is the result from a bhyved CentOS on an Intel i3:
>
> ===
> # strace -c ./0.orig
> Average time for System call getpid : 3.776000
> Average time for Function call : 2.326000
> % time seconds usecs/call callserrors syscall
> -- --- --- - -
> -nan0.00 0 1 read
> -nan0.00 0 2 write
> -nan0.00 0 2 open
> -nan0.00 0 2 close
> -nan0.00 0 3 fstat
> -nan0.00 0 9 mmap
> -nan0.00 0 3 mprotect
> -nan0.00 0 1 munmap
> -nan0.00 0 1 brk
> -nan0.00 0 1 1 access
> -nan0.00 0 1 getpid
> -nan0.00 0 1 execve
> -nan0.00 0 1 arch_prctl
> -- -
Re: Bheve: Slow linux syscalls on AMD
On 31-5-2014 2:13, Peter Grehan wrote: > Hi Willem, > >> So the question remains: >> Why is it taking so long on the AMD platform. > > The time difference looks a lot like a VM-exit roundtrip. My new AMD > box is arriving shortly so I'll have a look into it. Hi Peter, I would expect something like this. The question however what kind of business is done during the roundtrip. Could it be something like a soft-interrupt 0x80 to get from userspace to the kernel? Supossedly this interface is depreciated around 2.4 I'll see if I can augment the exit-code in bhyve. Catching it with bhyvectl --get-all will probably have too much noise from running the rest of the system. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 31-5-2014 2:13, Peter Grehan wrote: > Hi Willem, > >> So the question remains: >> Why is it taking so long on the AMD platform. > > The time difference looks a lot like a VM-exit roundtrip. My new AMD > box is arriving shortly so I'll have a look into it. Before running the getuid test: vm exits due to interrupt window opening0 vm exits due to nmi window opening 0 vm exits due to nested page fault 311094 vm exits for instruction emulation 159415683 number of vm exits for unknown reason 0 number of vm exits handled in userspace 0 number of vm exits due to exceptions0 total number of vm exits353767137 vm exits due to external interrupt 177525597 Then calling getuid() 10^6 times vm exits due to interrupt window opening0 vm exits due to nmi window opening 0 vm exits due to nested page fault 311094 vm exits for instruction emulation 159589889 number of vm exits for unknown reason 0 number of vm exits handled in userspace 0 number of vm exits due to exceptions0 total number of vm exits354157461 vm exits due to external interrupt 177730908 And that gives a difference of: 390324 Which is not the regular amount of vm exits counted in the equal idle time. So there seems to be some relation, but it is not 1:1... --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 8-6-2014 1:06, Peter Grehan wrote: So the question remains: Why is it taking so long on the AMD platform. > > I believe this is now fixed with r267217 - please test out and let us > know how it goes. On Head or on bhyve_svm At the moment I do not have a working bhyve setup, tried to merge too much from head in my own branch... So I'm still figuring out which one is the culprit I'll try and look at the diff. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 8-6-2014 17:27, Anish wrote: > Peter submitted this in bhyve_svm for SVM, Intel VT-x already has this > fix, see vmcs.c file in any branch. > > http://svnweb.freebsd.org/base/projects/bhyve_svm/sys/amd64/vmm/amd/vmcb.c?r1=267217&r2=267216&pathrev=267217 Downloaded bhyve_svm again and figured out as much as this. Working to recompile everything but the kitchen sink --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 2014-06-08 17:30, Willem Jan Withagen wrote: On 8-6-2014 17:27, Anish wrote: Peter submitted this in bhyve_svm for SVM, Intel VT-x already has this fix, see vmcs.c file in any branch. http://svnweb.freebsd.org/base/projects/bhyve_svm/sys/amd64/vmm/amd/vmcb.c?r1=267217&r2=267216&pathrev=267217 Downloaded bhyve_svm again and figured out as much as this. Working to recompile everything but the kitchen sink Right this seems to have fixed the performance problem. At least, compiling kernel file is actually making nice progres. Still seeing that a 2 CPU VM is using about 100% of 1 cpu when idleing, but that is another minor challenge. Thanx Peter --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 8-6-2014 18:23, Peter Grehan wrote: >> Still seeing that a 2 CPU VM is using about 100% of 1 cpu when idleing, >> but that is another minor challenge. > > I know what that one is: should have a fix shortly. I've done some of the syscall tests, like before. getuid(2) is now down to 350 nsec... Running the same on the bare metal with FreeBSD gives 140 nsec So I can't be complaining Comparing umask(2) is more the same: 151 nsec versus VM linux: 364 nsec --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 2014-06-09 23:04, Peter Grehan wrote:
Still seeing that a 2 CPU VM is using about 100% of 1 cpu when idleing,
but that is another minor challenge.
Fixed in r267305
Ack.
There is still a small difference in behaviour between Linux and
FreeBSD. But I'd call that negectable...
PIDSIZE STATE C TIMEWCPU COMMAND
70250 2075M vmidle 5 0:17 2.69% bhyve: ubuntu-14.04 (bhyve){vcpu 1}
70250 2075M vmidle 2 0:22 1.37% bhyve: ubuntu-14.04 (bhyve){vcpu 0}
70296 2075M vmidle 1 0:04 0.00% bhyve: freebsd-10 (bhyve){vcpu 0}
70296 2075M vmidle 4 0:02 0.00% bhyve: freebsd-10 (bhyve){vcpu 1}
Inside the VMs the perception is totally the other way around when just
running top and system processes:
ubuntu14:
load average: 0.01, 0.02, 0.02
FreeBSD:
load averages: 0.11, 0.18, 0.09
So IMHO things get lost tn the margins of (incorrect) measurement.
Next thing to check is how older ubuntu's work. Espacially those with
TLS and a 2.6 kernel. ( I think there is still one release not EOL)
And then start tests for CentOS.
Thanx,
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 2014-06-10 15:59, Peter Grehan wrote:
Hi Willem,
On 2014-06-09 23:04, Peter Grehan wrote:
Still seeing that a 2 CPU VM is using about 100% of 1 cpu when idleing,
but that is another minor challenge.
Fixed in r267305
Ack.
There is still a small difference in behaviour between Linux and
FreeBSD. But I'd call that negectable...
PIDSIZE STATE C TIMEWCPU COMMAND
70250 2075M vmidle 5 0:17 2.69% bhyve: ubuntu-14.04 (bhyve){vcpu 1}
70250 2075M vmidle 2 0:22 1.37% bhyve: ubuntu-14.04 (bhyve){vcpu 0}
Thanks - saw similar behaviour on my tiny Sempron but wasn't sure if
it was the slow CPU/clock aliasing etc. Time to bring out KTR and see
what's going on.
I've got KTR compiled in, but last time I switched it on.
I got swamped in traffic, and I sort of got locked out of the server... :(
Could also be because I was writing it to a file as well.
So you'll have to help/tell me what to do.
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 2014-06-10 17:34, Peter Grehan wrote: Hi Nils, Confirmed. Running a bhyved 3-vCPU-"CentOS 6.5", the host CPU load for "vcpu 0" is around 12% now. Doh, that's not good - haven't given Centos 6.5 a try; will now to investigate this. All Ubuntus I have installed in VMs give more or less perform the same... (13.10, 12.04 LTS) Although the overhead is just 5% and more evenly distibuted over both vPCUs. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: libvirt and bhyve problems
On 2014-06-12 20:33, Craig Rodrigues wrote: On Thu, Jun 12, 2014 at 11:28 AM, Craig Rodrigues wrote: On Thu, Jun 12, 2014 at 1:00 AM, Roman Bogorodskiy wrote: http://people.freebsd.org/~novel/misc/libvirt_port_updated.tgz With this setup, I'm able to get networking (e.g. virsh net-list works) and updated the fix for the previous problem. I was able to start a VM with that setup. I deleted the old port from my system and took your modified port, built it, and installed it. I followed my previous steps in: http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-June/002588.html and got this: 2014-06-12 18:23:54.328+: 34485605376: info : libvirt version: 1.2.5 2014-06-12 18:23:54.328+: 34485605376: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory 2014-06-12 18:23:54.328+: 34485605376: info : networkReloadFirewallRules:1750 : Reloading iptables rules 2014-06-12 18:23:54.328+: 34485605376: info : networkRefreshDaemons:1722 : Refreshing network daemons 2014-06-12 18:23:54.438+: 34485605376: error : virExec:417 : Cannot find 'pm-is-supported' in path: No such file or directory 2014-06-12 18:23:54.439+: 34485605376: warning : virQEMUCapsInit:948 : Failed to get host power management capabilities 2014-06-12 18:23:54.460+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/var/run/libvirt/qemu 2014-06-12 18:23:54.461+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/qemu 2014-06-12 18:23:54.560+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/bhyve 2014-06-12 18:23:54.560+: 34485605376: info : virDomainObjListLoadAllConfigs:18273 : Loading config file 'bhyve.xml' 2014-06-12 18:24:17.940+: 34485598208: error : virNetDevBridgeAddPort:399 : Unable to add bridge tap0 port vnet0: Invalid argument 2014-06-12 18:24:18.056+: 34485598208: error : virCommandWait:2426 : internal error: Child process (/usr/sbin/bhyvectl --destroy --vm=bhyve) unexpected exit status 255 -- Craig After the program fails, these are my tap and bridge devices, as shown by ifconfig: bridge0: flags=8843 metric 0 mtu 1500 ether 02:29:45:c7:8f:00 nd6 options=9 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143 ifmaxaddr 0 port 1 priority 128 path cost 2 member: tap2 flags=143 ifmaxaddr 0 port 7 priority 128 path cost 200 member: tap1 flags=143 ifmaxaddr 0 port 6 priority 128 path cost 200 member: tap0 flags=143 ifmaxaddr 0 port 5 priority 128 path cost 200 tap0: flags=8902 metric 0 mtu 1500 options=8 ether 00:bd:53:27:00:00 nd6 options=29 media: Ethernet autoselect status: no carrier tap1: flags=8902 metric 0 mtu 1500 options=8 ether 00:bd:59:27:00:01 nd6 options=29 media: Ethernet autoselect status: no carrier tap2: flags=8943 metric 0 mtu 1500 options=8 ether 00:bd:5e:27:00:02 nd6 options=29 media: Ethernet autoselect status: active Opened by PID 1506 vnet0: flags=8802 metric 0 mtu 1500 options=8 ether fe:54:00:33:20:8c nd6 options=21 media: Ethernet autoselect status: no carrier This is in my /etc/rc.conf for creating bridge and tap devices on bootup: # # Create tap devices, one tap interface per BHyve VM. # Add the tap interfaces to bridge0 cloned_interfaces="bridge0 tap0 tap1 tap2" autobridge_interfaces="bridge0" autobridge_bridge0="tap* em0" I claim to know nothing about libvirt. But in my scripts I need to delay using the tap?? interfaces for 1-2 secs with a sleep. Because otherwise my bhyve VM fails. Could be the same here? --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bheve: Slow linux syscalls on AMD
On 10-6-2014 20:04, Peter Grehan wrote: > Hi Willem, > >> I've got KTR compiled in, but last time I switched it on. >> I got swamped in traffic, and I sort of got locked out of the >> server... :( >> Could also be because I was writing it to a file as well. >> >> So you'll have to help/tell me what to do. > > For looking at this, I'd use one single-vCPU guest to avoid KTR noise > from other vCPU/guests. The problem shows up with this so no issue there. The trace consists mainly of large number of sequeces like: (ktrdump -trc) 204 137052 vm ubuntu-14.04[0]: pending intr 65 203 139489 vm ubuntu-14.04[0]: VMEXIT halted CPU. 202 153836 vm ubuntu-14.04[0]: SVM:Guest in interrupt shadow. 201 169072 vm ubuntu-14.04[0]: SVM:Enter vmrun RIP:0x8104f596 inst len=0/15 200 137218 vm ubuntu-14.04[0]: pending intr 65 199 139534 vm ubuntu-14.04[0]: VMEXIT halted CPU. intermixed with blocks: 13485 165347 vm ubuntu-14.04[0]: SVM:event injected,vector=65. 13484 141457 vm ubuntu-14.04[0]: vlapic_update_ppr 0x40 13483 153987 vm ubuntu-14.04[0]: vlapic_intr_accepted isr7 0x 13482 152734 vm ubuntu-14.04[0]: vlapic_intr_accepted isr6 0x 13481 152729 vm ubuntu-14.04[0]: vlapic_intr_accepted isr5 0x 13480 152784 vm ubuntu-14.04[0]: vlapic_intr_accepted isr4 0x 13479 152755 vm ubuntu-14.04[0]: vlapic_intr_accepted isr3 0x 13478 152840 vm ubuntu-14.04[0]: vlapic_intr_accepted isr2 0x0002 13477 152709 vm ubuntu-14.04[0]: vlapic_intr_accepted isr1 0x 13476 152669 vm ubuntu-14.04[0]: vlapic_intr_accepted isr0 0x 13475 152722 vm ubuntu-14.04[0]: vlapic_intr_accepted irr7 0x 13474 152694 vm ubuntu-14.04[0]: vlapic_intr_accepted irr6 0x 13473 152701 vm ubuntu-14.04[0]: vlapic_intr_accepted irr5 0x 13472 152705 vm ubuntu-14.04[0]: vlapic_intr_accepted irr4 0x 13471 152692 vm ubuntu-14.04[0]: vlapic_intr_accepted irr3 0x 13470 152884 vm ubuntu-14.04[0]: vlapic_intr_accepted irr2 0x 13469 152839 vm ubuntu-14.04[0]: vlapic_intr_accepted irr1 0x 13468 153055 vm ubuntu-14.04[0]: vlapic_intr_accepted irr0 0x So it looks like the VM goes halted, but interrupt 65 tries to wake it. Dmesg however does not give a irq 65 in the guest. wjw@ubuntu-14:~$ dmesg | grep irq [ 0.00] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) [ 0.00] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.00] nr_irqs_gsi: 40 [ 0.00] NR_IRQS:16640 nr_irqs:256 16 [ 0.321489] hpet: hpet2 irq 40 for MSI [ 0.748675] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 0.770813] 00:02: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 0.778181] virtio-pci :00:02.0: irq 41 for MSI/MSI-X [ 0.778203] virtio-pci :00:02.0: irq 42 for MSI/MSI-X [ 0.778224] virtio-pci :00:02.0: irq 43 for MSI/MSI-X [ 1.393123] ahci :00:03.0: irq 44 for MSI/MSI-X [ 1.424971] ata1: SATA max UDMA/133 abar m1024@0xc0002000 port 0xc0002100 irq 44 [ 1.435304] ahci :00:04.0: irq 45 for MSI/MSI-X [ 1.456767] ata7: SATA max UDMA/133 abar m1024@0xc0002400 port 0xc0002500 irq 45 A larger ktrdump is at: http://smtp.digiware.nl/tmp/ktrdump.log --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: HEADS UP: Merging projects/bhyve_svm to HEAD
On 16-10-2014 5:00, Anish Gupta wrote: > Hi all, > > The projects/bhyve_svm branch is ready to be merged to HEAD. > > This branch contains patches to bhyve to enable it to work on AMD > processors with SVM/AMD-V hardware extensions[1]. Pretty much any AMD > processor since 2010 will have the features required by bhyve. > > bhyve on AMD supports (almost) all the features available with Intel > [2]. All guest OSes supported on Intel are supported on AMD. All the > bhyve-related utilities function similarly on both Intel and AMD > platforms [3]. > > The patch against HEAD revision 273066 is available for review and testing: > https://people.freebsd.org/~neel/bhyve/bhyve_svm.diff [Neel’s web directory] > > [1]: http://en.wikipedia.org/wiki/X86_virtualization > [2]: bhyve doesn't support PCI passthru on AMD at this time > [3]: bhyvectl has grown some processor-specific options Fetched the patch and compiled. Now running: HEAD r273066M and I was able to throw at it all the tests and images that in the past works. And perhaps even better. Great work. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: moving from virtualbox to bhyve - SOLVED
On 22-10-2014 2:20, John wrote: > On Tue, Oct 21, 2014 at 01:55:26PM -0700, Neel Natu wrote: > >> If you are comfortable sharing your disk image then I can try to >> reproduce locally and hopefully get a better grip on what's happening. > > That's very nice of you to offer, but I have to decline as I don't own > the data. I think I might have sorted it in any case. I was thrown by > the login prompt not appearing in the console. > > Really pleased with the performance of ubuntu on bhyve. It doesn't impact > much on the host either. > Why not boot the system, and look in the grub config. And fix the code there by removing most of the graphics code. Or for testing, edit the grub boot line from within grub itself while bhyveload Then you will know if this is the problem, but from old experience trying to get all the linuxcees running in AMD, I think I ran into this as well. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Bhyve building kernel....Just an observation
Hi,
Just out of curiosity I did the following:
Updated 10-STABLE src in both Dom0 and in the Bhyve FreeBSD VM
Rebooted my bhyve AMD testing machine so it was in a fresh state.
Nothing special loaded orhter than ZFS.
And build a 10-STABLE kernel on the raw box, that took about 6 minutes.
no {make,src}.conf, so the full GENERIC KERNEL
That took about 6:45 minutes.
Again rebooted to clean the box up.
And then booted a 10-STABLE bhyve VM with all memory and processors
assigned to the VM. So all the power could be available to the VM.
The build again 10-STABLE in the VM.
That took about 9:30 minutes.
Now it looks like the VM has a 50% overhead.
My expectations were quite a bit better for the performance difference
between the two?
Would people with more experience in doing VM stuff expect such a large
difference?
Or is this a to be expected result?
Thanx,
--WjW
System info:
Dom0:
11.0-CURRENT FreeBSD 11.0-CURRENT #1 r273066M
(110.0 Head with Neel's patch)
CPU: AMD Phenom(tm) II X6 1075T Processor (3013.83-MHz K8-class
16 Gb Ram
ZFS mirror
2* 500Gb WD WD5000AA
cache 2* Samsung SSD 840 PRO Series DXM05B0Q
Dom Bhyve
FreeBSD 10.0-RELEASE-p9
ahci-hd 16GB
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve building kernel....Just an observation
On 24-10-2014 17:03, Allan Jude wrote:
> On 2014-10-24 04:05, Willem Jan Withagen wrote:
>> Hi,
>>
>> Just out of curiosity I did the following:
>>
>> Updated 10-STABLE src in both Dom0 and in the Bhyve FreeBSD VM
>> Rebooted my bhyve AMD testing machine so it was in a fresh state.
>> Nothing special loaded orhter than ZFS.
>>
>> And build a 10-STABLE kernel on the raw box, that took about 6 minutes.
>> no {make,src}.conf, so the full GENERIC KERNEL
>> That took about 6:45 minutes.
>>
>> Again rebooted to clean the box up.
>> And then booted a 10-STABLE bhyve VM with all memory and processors
>> assigned to the VM. So all the power could be available to the VM.
>>
>> The build again 10-STABLE in the VM.
>> That took about 9:30 minutes.
>>
>> Now it looks like the VM has a 50% overhead.
>> My expectations were quite a bit better for the performance difference
>> between the two?
>>
>> Would people with more experience in doing VM stuff expect such a large
>> difference?
>> Or is this a to be expected result?
>>
>> Thanx,
>> --WjW
>>
>> System info:
>> Dom0:
>> 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r273066M
>> (110.0 Head with Neel's patch)
>> CPU: AMD Phenom(tm) II X6 1075T Processor (3013.83-MHz K8-class
>> 16 Gb Ram
>> ZFS mirror
>> 2* 500Gb WD WD5000AA
>> cache 2* Samsung SSD 840 PRO Series DXM05B0Q
>>
>> Dom Bhyve
>> FreeBSD 10.0-RELEASE-p9
>> ahci-hd 16GB
> The big difference here is probably disk performance. If you do
> something that is pure CPU, you'll usually get about the same
> performance as the host, but virtualized disk is not always nearly as good.
Would it make any difference to use virtio instead of ahci-hd?
In any case, to test your assumption:
build a 10G memdisk in Dom0 and DomU
load it with sources and /usr/obj
and see if that helps getting things closer?
Nice thing to do during eveninghours in the weekend.
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve building kernel....Just an observation
On 25-10-2014 1:21, Peter Grehan wrote: > Hi Willem, >> And then booted a 10-STABLE bhyve VM with all memory and processors >> assigned to the VM. So all the power could be available to the VM. > > You'll most likely want to keep some memory and processor resources > available for the host system. ZFS will need memory for ARC and CPU for > operations - if these aren't available, it will compete with bhyve's use > of CPUs, and there will be times when these are conflicting. Hi Peter, Thanx for the hint. The assumption that the performance difference is, is Disk-IO is at least not very obvious from the simple test. Both tests were running from a 6Gb tmpfs Dom0 had 16Gb Ram, and I limited DomU to 12G Ram. Tested it once in Dom0, and it takes ~ 5Gb of tmp store. It holds src and obj, and I see no disk traffic while building. Building kernel with just the default tmpfs (all avail mem + swap) takes 6:30 (zfs with ssd's) versus 5:30 (tmpfs) Running in DomU is get about the same difference: with ahci-hd/zfs backing9:30 with tmpfs 8:30 So I would think the difference is not really in the IO-performance. But as usual: All other opinions more than appreciated. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Looking for a Libvirt example xml file
Hoi I tried the version on the libvirt.org/bhyve page. But that returns: freetest# virsh -c "bhyve:///system" domxml-to-native \\ --format bhyve-argv --xml /root/libvirt-example.xml error: unsupported configuration: unsupported disk device So I was wondering if somebody would like to share his working example? Thanx, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Looking for a Libvirt example xml file
On 26-10-2014 1:37, Conrad Meyer wrote: > On Sat, Oct 25, 2014 at 6:48 PM, Willem Jan Withagen wrote: >> Hoi >> >> I tried the version on the libvirt.org/bhyve page. >> But that returns: >> >> freetest# virsh -c "bhyve:///system" domxml-to-native \\ >> --format bhyve-argv --xml /root/libvirt-example.xml >> error: unsupported configuration: unsupported disk device >> >> So I was wondering if somebody would like to share his working example? > > > Hi Willem, > > What do your sections look like? I am using basically the same > configuration from libvirt's bhyve page as well and have no such issue > (however, I am on recent git which is ~1.2.10, versus the 1.2.6 in > ports). Example: > > > > > > > > For what it's worth, If you do not specify a bus, libvirt will choose > 'ide' and libvirt-bhyve only supports SATA. Seems that my CDrom block was a problem. But the disk is already build, so I just skipped that. But thing do not generate a lot of logging. :( Not even with debug on 4 --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Looking for a Libvirt example xml file
On 26-10-2014 13:35, Conrad Meyer wrote:
> On Sun, Oct 26, 2014 at 5:49 AM, Roman Bogorodskiy wrote:
>> Conrad Meyer wrote:
>>
>>> On Sat, Oct 25, 2014 at 9:42 PM, Willem Jan Withagen
>>> wrote:
>>>> Seems that my CDrom block was a problem.
>>>> But the disk is already build, so I just skipped that.
>>>
>>> Yes, even in git libvirt-bhyve doesn't work well with >1 disk :-(.
>>
>> Hi!
>>
>> Could you please provide more details on the issues you're seeing with
>> that?
>
>
> Hi Roman,
>
> As far as I can tell, libvirt-bhyve will only boot from the first
> disk, and ignores libvirt's boot ordering stuff. Unless libvirt sorts
> the disks array by bootorder before passing to the driver? I had
> trouble getting it to boot a select disk when I included both a cdrom
> and hdd in my testing the other day. E.g. in git master:
>
> 297 virCommandPtr
> 298 virBhyveProcessBuildLoadCmd(virConnectPtr conn,
> 299 virDomainDefPtr def)
> 300 {
> ...
> 310 disk = def->disks[0];
> ...
> 329 cmd = virCommandNew(BHYVELOAD);
> ...
> 337 virCommandAddArg(cmd, "-d");
> 338 virCommandAddArg(cmd, virDomainDiskGetSource(disk));
>
> Libvirt lets you specify some sort of boot ordering in domain XML
> inside /domain/os[0]:
>
>
>
>
> Although now that I am looking more closely it appears that ordering
> is basically ignored for same-bus devices :-(. That may explain my
> test results...
>
> So, I guess the right way to do it is to slap ""
> inside elements. I didn't try that and don't know if libvirt
> sorts those before handing off to driver.
>
> [0]: http://libvirt.org/formatdomain.html#elementsOS
Are there any FBSD directions online to actually get libvirt to build
after you did git clone??
--WjW
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Looking for a Libvirt example xml file
On 26-10-2014 15:27, Conrad Meyer wrote: > On Sun, Oct 26, 2014 at 10:11 AM, Willem Jan Withagen > wrote: >> Are there any FBSD directions online to actually get libvirt to build >> after you did git clone?? > > > See README-hacking[0] and checkout pkgng poudriere logs[1] for options > to ./configure. After that it's just "gmake -j" and "gmake > install". Following another path, but that may very well bomb: next to the requirements from the ports/libvirt/Makefile also installed form ports: gnulib git-merge-changelog And running autogen.sh and then went: gmake And it is still building. But I'm certainly going to take a look at your links as well. Thanx, --WjW > > Best, > Conrad > > [0]: > http://libvirt.org/git/?p=libvirt.git;a=blob;f=README-hacking;h=4e02fd854c4cdbdd40bfc2e29382342419645aae;hb=HEAD > [1]: > http://beefy4.isc.freebsd.org/data/head-amd64-default-ssp/2014-10-22_14h32m42s/logs/libvirt-1.2.6_3.log > ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Looking for a Libvirt example xml file
On 26-10-2014 0:48, Willem Jan Withagen wrote: > Hoi > > I tried the version on the libvirt.org/bhyve page. > But that returns: > > freetest# virsh -c "bhyve:///system" domxml-to-native \\ > --format bhyve-argv --xml /root/libvirt-example.xml > error: unsupported configuration: unsupported disk device > > So I was wondering if somebody would like to share his working example? After building libvirt 1.2.10 I was able to load this XML: /usr/sbin/bhyveload -m 214 -d /home/bhyve/FreeBSD/freebsd-head.disk bhyveCD /usr/sbin/bhyve -c 1 -m 214 -A -I -H -P -s 0:0,hostbridge -s 2:0,virtio-net,tap0,mac=52:54:00:63:43:69 -s 3:0,virtio-blk,/home/bhyve/FreeBSD/freebsd-head.disk -s 4:0,ahci-cd,/home/bhyve/FreeBSD/freebsd-head.iso bhyveCD --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Trying to run an older head in a recent Bhyve.
Hi, I'm trying to run one of my older VM's and get the crash below in the VM when trying to boot... This happens both on an older BSD: FreeBSD 11.0-CURRENT (FREETEST) #1 r273066M: Sun Oct 19 00:59:06 CEST 2014 As well as on a very recent: FreeBSD 11.0-CURRENT (BHYVE00) #0 r274490M: Fri Nov 14 02:42:43 CET 2014 The older 10.0 VM's do boot normally Any suggestions on what this might be, and/or how to debug this.. --WjW DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 3 DHCPOFFER from 192.168.10.75 DHCPREQUEST on vtnet0 to 255.255.255.255 port 67 DHCPACK from 192.168.10.75 panic: ffs_write: type 0xf8000565dce8 0 (0,62) cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00975374d0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe0097537580 vpanic() at vpanic+0x126/frame 0xfe00975375c0 panic() at panic+0x43/frame 0xfe0097537620 ffs_write() at ffs_write+0x68b/frame 0xfe00975376c0 VOP_WRITE_APV() at VOP_WRITE_APV+0x16f/frame 0xfe00975377d0 vn_write() at vn_write+0x2f3/frame 0xfe0097537860 vn_io_fault() at vn_io_fault+0x17b/frame 0xfe00975379f0 dofilewrite() at dofilewrite+0x88/frame 0xfe0097537a40 kern_writev() at kern_writev+0x68/frame 0xfe0097537a90 sys_write() at sys_write+0x63/frame 0xfe0097537ae0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfe0097537bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe0097537bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800de9b4a, rsp = 0x7fffd4c8, rbp = 0x7fffd510 --- KDB: enter: panic [ thread pid 551 tid 100057 ] Stopped at kdb_enter+0x3e: movq$0,kdb_why ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Trying to run an older head in a recent Bhyve.
On 16-11-2014 0:59, Peter Grehan wrote: > Hi Willem, > >> I'm trying to run one of my older VM's and get the crash below in the VM >> when trying to boot... >> >> This happens both on an older BSD: >> FreeBSD 11.0-CURRENT (FREETEST) #1 r273066M: Sun Oct 19 00:59:06 CEST >> 2014 >> As well as on a very recent: >> FreeBSD 11.0-CURRENT (BHYVE00) #0 r274490M: Fri Nov 14 02:42:43 CET 2014 >> >> The older 10.0 VM's do boot normally >> >> Any suggestions on what this might be, and/or how to debug this.. > > Did the disk image backing file change by any chance ? e.g. from file > to zvol ? > > What's the version of the VM that has the issue ? Not sure what you mean by that exactely, but the VM runs a relatively old HEAD: FreeBSD 11.0-CURRENT #1 r262690: Sun Mar 2 21:28:19 CET 2014 root@bhyve-head:/usr/obj/usr/src/sys/GENERIC amd64 It boots in single mode. I'tt try and manually assign an IP number and will see what that brings. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: RFC: Enabling VIMAGE in GENERIC
On 17-11-2014 12:02, Warner Losh wrote: > > On Nov 17, 2014, at 12:46 AM, Craig Rodrigues > wrote: > >> Hi, >> >> PROPOSAL == I would like to get feedback on the following >> proposal. In the head branch (CURRENT), I would like to enable >> VIMAGE with this commit: >> >> >> PATCH == >> >> Index: sys/conf/NOTES >> === >> >> --- sys/conf/NOTES (revision 274300) >> +++ sys/conf/NOTES (working copy) @@ -784,8 +784,8 @@ device >> mn # Munich32x/Falc54 Nx64kbit/sec cards. >> >> # Network stack virtualization. -#options VIMAGE -#options >> VNET_DEBUG # debug for VIMAGE +optionsVIMAGE +options >> VNET_DEBUG # debug for VIMAGE >> >> # # Network interfaces: >> >> >> >> I would like to enable VIMAGE for the following reasons: >> >> REASONS >> >> (1) VIMAGE cannot be enabled off to the side in a separate library >> or kernel module. When enabled, it is a kernel ABI incompatible >> change. This has impact on 3rd party code such as the kernel >> modules which come with VirtualBox. So the time to do it in CURRENT >> is now, otherwise we can't consider doing it until FreeBSD-12 >> timeframe, which is quite a while away. >> >> (2) VIMAGE is used in some 3rd party products, such as FreeNAS. >> These 3rd party products are mostly happy with VIMAGE, but >> sometimes they encounter problems, and FreeBSD doesn't see these >> problems because it is disabled by default. >> >> (3) Most of the major subsystems like ipfw and pf have been fixed >> for VIMAGE, and the only way to shake out the last few issues is to >> make it the default and get feedback from the community. ipfilter >> still needs to be VIMAGE-ified. >> >> >> (4) Not everyone uses bhyve. FreeBSD jails are an excellent >> virtualization platform for FreeBSD. Jails are still very popular >> and performant. VIMAGE makes jails even better by allowing >> per-jail network stacks. >> >> (5) Olivier Cochard-Labbe has provided good network performance >> results in VIMAGE vs. non-VIMAGE kernels: >> >> >> https://lists.freebsd.org/pipermail/freebsd-net/2014-October/040091.html >> >> >> (6) Certain people like Vitaly "wishmaster" have been >> running VIMAGE jails in a production environment for quite a while, >> and would like to see it be the default. >> >> >> ACTION PLAN === >> >> (1) Coordinate/communicate with portmgr, since this has kernel >> ABI implications >> >> (2) Work with clusteradm@, and try to get a test instance of one >> of the PF firewalls in the cluster working with a VIMAGE enabled >> kernel. >> >> (3) Take a pass through http://wiki.freebsd.org/VIMAGE/TODO and >> https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=vimage%20or%20vnet >> >> and try to clean things up. Get help from net@ developers to do >> this. > > And if these don’t get cleaned up? > >> (4) Take a pass on trying to VIMAGE-ify ipfilter. I'll need help >> from the ipfilter maintainers for this and some net@ developers. > > And if this doesn’t happen? > >> (5) Enable VIMAGE by default in CURRENT on January 5, 2015. This >> will *not* be enabled in STABLE. >> >> What do people think? > > How do you plan to address the problems seen by FreeNAS in #2 above? > I don’t see that in the action plan. Without it, we’re enabling an > option that has know, serious issue making 11 potentially a more > unstable release. Hi Warner, I think I understand your critique, but then on the other hand I wonder where the reluctance is As I read it, things are going to be enabled in CURRENT only (for the time). Which is exactly for the reasons you worry about: Is it going to be reliable enough?? But for that it needs exposure... So I would expect it to be turned off as a default IF things are not in a stable state that warrants a default enabling of the options. Things need to move forward, and taking this step is going to be required.. Otherwise I see a big risk of bit-rot somewhere down in the dungeons. --Willem Jan ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: RFC: Enabling VIMAGE in GENERIC
On 17-11-2014 12:42, Bjoern A. Zeeb wrote: > On 17 Nov 2014, at 11:20 , Willem Jan Withagen wrote: > >> I think I understand your critique, but then on the other hand I wonder >> where the reluctance is As I read it, things are going to be enabled >> in CURRENT only (for the time). Which is exactly for the reasons you >> worry about: Is it going to be reliable enough?? > > No, the answer to that still is “no” in it’s current state and we know that. > > I think one of the main problems is that no one has been able to pull the > thing to the end in the last three years. Why should it happen within 6 > weeks now? > > I think it would be a really good idea to do that but the current TODO list, > I think, is by far not sufficing. > > There’s a second problem we’ll hit in that same timeframe: general network > stack breakage; we’ll hit the times when we’ll not be sure if things broke > because of VIMAGE or are also broken in the normal network stack. There’ll > be a lot of regression test writing and debugging to be done. > > > That all said, I’d like to see it happen as well, but I’d love to have a lot > of the issues being addressed first before putting a date on it to enable > it in GENERIC in HEAD. Hi Bjoern, The constraints as you put them are indeed rather tight. There is little to be done about it. I was not aware of the fact that 11.0 is planned for release in such short time. Somewhere in the back op my head is: planning is start release cycle around Q2 2015. And I took the liberty to add some testing & QA time to that, so I expect it after summer holidays in 2015. Now I admit: I don't write code for this, nor do I have the knowledge to so. But do run CURRENT to test exactly all these visualization options... Have not (yet) found a requirement to put VIMAGE to good use, so I never switched it on. The down side of all this, is that if we cannot turn it on during the 11-STABLE lifetime. Then it is going to take a full version release cycle to make it to the front. Which I find a pity, since it misses the opportunity for FreeBSD to add another distinctive element to their visualization possibilities. I prefer not to take this into a debate about the way things should go. Over time I've seen too many of these discussions turn in to shouting wars/bikesheds/and what not So given the fact I'm not going to do it, I'll leave the rest of the discussion to those that are actual doing all the work... (for which already 20 year of thanks) --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: RFC: Enabling VIMAGE in GENERIC
Op 17 nov. 2014 om 16:37 heeft Dag-Erling Smørgrav het volgende geschreven: > Willem Jan Withagen writes: >> The constraints as you put them are indeed rather tight. There is little >> to be done about it. I was not aware of the fact that 11.0 is planned >> for release in such short time. > > It isn't. ISTR that the target is 2015Q4. > So even further in the future than what I expected. But still, somebody(tm) needs to do the actual work. So they get the say. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Trying to run an older head in a recent Bhyve.
On 16-11-2014 19:19, Allan Jude wrote: > On 2014-11-16 12:54, Willem Jan Withagen wrote: >> On 16-11-2014 0:59, Peter Grehan wrote: >>> Hi Willem, >>> >>>> I'm trying to run one of my older VM's and get the crash below in the VM >>>> when trying to boot... >>>> >>>> This happens both on an older BSD: >>>> FreeBSD 11.0-CURRENT (FREETEST) #1 r273066M: Sun Oct 19 00:59:06 CEST >>>> 2014 >>>> As well as on a very recent: >>>> FreeBSD 11.0-CURRENT (BHYVE00) #0 r274490M: Fri Nov 14 02:42:43 CET 2014 >>>> >>>> The older 10.0 VM's do boot normally >>>> >>>> Any suggestions on what this might be, and/or how to debug this.. >>> >>> Did the disk image backing file change by any chance ? e.g. from file >>> to zvol ? >>> >>> What's the version of the VM that has the issue ? >> >> Not sure what you mean by that exactely, but the VM runs a relatively >> old HEAD: >> >> FreeBSD 11.0-CURRENT #1 r262690: Sun Mar 2 21:28:19 CET 2014 >> root@bhyve-head:/usr/obj/usr/src/sys/GENERIC amd64 >> >> It boots in single mode. >> >> I'tt try and manually assign an IP number and will see what that brings. > He is asking how the virtual disk is setup, specifically, if it is a zvol > > Judging by the backtrace, the VM is panicing when trying to write to the > disk and it fails > > From my experience, the two most likely causes are: > > 1) The disk is a zvol, and does not have the volmode set, and GEOM on > the host is grabbing the disk and locking it, preventing writes > > 2) The VM was shutdown ungracefully and the file system needs a fsck. > Since you can get into single user mode, this should be doable. The VM-disk is a file, very early on I tried things with ZVOL but have not yet returned to testing those. Well the system passes the fsck-check during boot. And I can manually config the network, and /etc/resolv.conf Then the system stays up, and I was able to download the new sources and build them without crashing the system... I really crashed right after DHCP answers --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: cu -l /dev/nmdm not setting rows and columns
On 24-11-2014 3:04, Peter Grehan wrote: > Hi Craig, > >> # stty -a >> speed 9600 baud; 0 rows; 0 columns; >> # echo $TERM >> dialup >> >> Any idea how I can fix this? The console inside the VM >> is quite unusable when it does not have the correct >> rows/colums set. > > Not sure how you're getting 'dialup' as the terminal type: the default > ttys file for 10.1 shouldn't need to be edited, and has > > ttyu0 "/usr/libexec/getty std.9600" vt100 onifconsole secure > > The rows/columns is always 0 for uart-style serial lines since it's not > possible to know what's on the other end. That's why $TERM has to be set > correctly for these. > > Or, you can network-login to the guest in which case xterm works fine :) Or, like in the old days, just assign a TERM env value in your shell-login script based on the device the shell is connected too. That'll give you a basic working setup. Resizing will probably still not automagically work, and will require extra intervention. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: cu -l /dev/nmdm not setting rows and columns
On 27-11-2014 12:23, Nikos Vassiliadis wrote: > On 11/27/14 02:02, Craig Rodrigues wrote: >> On Wed, Nov 26, 2014 at 3:44 PM, John-Mark Gurney >> wrote: >> >>> >>> So, what exactly is the problem again? >>> >> >> https://lists.freebsd.org/pipermail/freebsd-virtualization/2014-November/003173.html >> > > Set the correct $TERM and set also rows and columns. Something like > this, will suffice: > TERM=xterm; export TERM > stty rows 80 > stty columns 80 > > Then the system you are connecting to, will know what to send back to > your terminal. And then wrap this into a piece of shell init. for tcsh in .tcshrc, but you get the drift: if (@`tty`@ == @/dev/ttyd0@) ( setenv TERM vt220; stty rows 60 ) if (@`tty`@ == @/dev/ttyv6@) /usr/bin/systat -vm 1 if (@`tty`@ == @/dev/ttyv7@) /usr/bin/top Just try it once an check the tty with /usr/bin/tty --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Native Linux guest in Bhyve (no grub2-bhyve) status?
On 23-10-2014 1:00, Peter Grehan wrote: > Hi Conrad, > >> Is the work being done in public? > > We're working on getting it released. > Peter, Would it it be possible in the meantime to enhance grub2-bhyve with the same possibility as bhyve itself has for the console? So I can redirect grub screen access to /dev/nmdm12041, and one can even use this channel to see the grubscreen during rebooting. Or if it is not that hard, give some pointers to write it myself. Regards, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Native Linux guest in Bhyve (no grub2-bhyve) status?
On 2015-01-05 0:28, Peter Grehan wrote: Hi Willem, Would it it be possible in the meantime to enhance grub2-bhyve with the same possibility as bhyve itself has for the console? So I can redirect grub screen access to /dev/nmdm12041, and one can even use this channel to see the grubscreen during rebooting. Or if it is not that hard, give some pointers to write it myself. Conrad contributed this to grub-bhyve with https://github.com/grehan-freebsd/grub2-bhyve/pull/2 It's available in v0.30 which has been in ports for a while now: https://github.com/grehan-freebsd/grub2-bhyve/releases/tag/v0.30 Right, Did miss that, I guess. I'll upgrade my bhyve-grub. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ipv4 routing from bhyve
On 11-1-2015 22:32, williamecow...@hush.ai wrote: > Hello, I hope I can have some assistance. > > I am trying to get networking via wlan0 but without NAT or bridging (doesn't > work on wifi unless WDS). > > say my my main network is 10.10.2.0/24, gateway/internet is 10.10.2.1, my ip > is 10.10.2.252. > > I started to config my bhyve network on 172.16.32.0/24 > > I added a bridge interface with an ip of 172.16.32.1 > > enable forwarding and fastforwarding. from my understanding of the handbook > chapter things should work when I type: > > # route add -net 172.16.32.0/24 10.10.2.252 > route: writing to routing socket: File exists > add net 172.16.32.0: gateway 10.10.2.252 fib 0: route already in table > # > > # netstat -4nr > Routing tables > > Internet: > DestinationGatewayFlags Netif Expire > default10.10.2.1UGS lagg0 > 127.0.0.1 link#3 UH lo0 > 10.10.2.0/24 link#5 U lagg0 > 10.10.2.252 link#5 UHS lo0 > 172.16.32.0/24link#4 U bridge0 > 172.16.32.1 link#4 UHS lo0 > # > > bridge0: flags=8843 metric 0 mtu 1500 > ether 00:bd:0f:fc:01:10 > inet 172.16.32.1 netmask 0xff00 broadcast 172.16.32.255 > nd6 options=9 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 6 priority 128 path cost 200 > lagg0: flags=8843 metric 0 mtu 1500 > ... > inet netmask 0xff00 broadcast 10.10.2.255 > nd6 options=9 > media: Ethernet autoselect > status: active > laggproto failover lagghash l2,l3,l4 > laggport: alc0 flags=1 > laggport: wlan0 flags=4 > tap0: flags=8903 metric 0 mtu 1500 > options=8 > ether 00:bd:8f:62:67:10 > nd6 options=9 > media: Ethernet autoselect > status: no carrier > wlan0: flags=8843 metric 0 mtu 1500 > ... > pflog0: flags=141 metric 0 mtu 33160 > tap9: flags=8802 metric 0 mtu 1500 > options=8 > ether 00:bd:cb:46:02:09 > nd6 options=1 > media: Ethernet autoselect > status: no carrier > tap1: flags=8802 metric 0 mtu 1500 > options=8 > ether 00:bd:58:61:02:01 > nd6 options=1 > media: Ethernet autoselect > status: no carrier Well one of the things of concern is the fact that your tap interfaces have: status: no carrier My connected bhyve vm's have, amongst others: status: active groups: tap Opened by PID 20763 And my bridge device tells me: bridge0: flags=8843 metric 0 mtu 1500 ether 02:76:2d:3d:9c:00 inet xxx.xxx.xxx.xxx netmask 0xff00 broadcast 37.255.255.255 nd6 options=9 groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap651 flags=143 ifmaxaddr 0 port 11 priority 128 path cost 200 member: tap6 flags=143 ifmaxaddr 0 port 10 priority 128 path cost 55 member: tap14041 flags=143 ifmaxaddr 0 port 9 priority 128 path cost 200 member: tap13101 flags=143 ifmaxaddr 0 port 8 priority 128 path cost 200 member: tap12041 flags=143 ifmaxaddr 0 port 6 priority 128 path cost 200 member: tap13 flags=143 ifmaxaddr 0 port 4 priority 128 path cost 200 member: em0 flags=143 ifmaxaddr 0 port 1 priority 128 path cost 2 So I think you first need to connect your VM's, before anything else will start to work. Like adding the tap-ifs to the bridge. And on the host itself you don't really need to add routing for the VM's because everything is actually already connected. Which is what the netstat output tells you. The routing table tells you that traffic for 172.16.32.0/24link#4 U bridge0 is send into the the bridge0 devices, which is directly connected. And ip-nrs in that range should appear in the the arp table. And the host then knows how to get to them directly. Routing for 172.16.32.0/24, if any needed, will be required on other hosts on you network on lagg0. Unless all hosts there have 10.10.2.252 as their default route. Regards, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
How to connect ssh and /dev/nmdm devices Was:( Re: Native Linux guest in Bhyve (no grub2-bhyve) status? )
On 2015-01-05 9:41, Willem Jan Withagen wrote: On 2015-01-05 0:28, Peter Grehan wrote: Hi Willem, Would it it be possible in the meantime to enhance grub2-bhyve with the same possibility as bhyve itself has for the console? So I can redirect grub screen access to /dev/nmdm12041, and one can even use this channel to see the grubscreen during rebooting. Or if it is not that hard, give some pointers to write it myself. Conrad contributed this to grub-bhyve with https://github.com/grehan-freebsd/grub2-bhyve/pull/2 It's available in v0.30 which has been in ports for a while now: https://github.com/grehan-freebsd/grub2-bhyve/releases/tag/v0.30 Right, Did miss that, I guess. I'll upgrade my bhyve-grub. Yup, works really nice. Now in a continuance from this... What is the easiest way to "propagate" the full-duplex tty stream from a SSH-login to a /dev/nmdmA. This will give easy access to the console screens. one would type: ssh -p user@management-ip and end up at the console on /dev/nmdmA Currently I'm using tip for this, but that also requires hacking new VMs into /etc/remote. So a very basic TIP, without the serial/ACU/speed stuff, would really be useful. Preferably all signals are sent thru as well. And if this might not be 100% secure, it would require an ssh-jailed setup. A bit like sftp. hints and or suggestion welcome. --WjW --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: How to connect ssh and /dev/nmdm devices
On 2015-01-20 18:48, Paul Vixie wrote: Willem Jan Withagen <mailto:w...@digiware.nl> Tuesday, January 20, 2015 8:37 AM What is the easiest way to "propagate" the full-duplex tty stream from a SSH-login to a /dev/nmdmA. somewhat expectedly, i use <http://www.freshports.org/sysutils/rtty/> for this. with a restricted shell, i can offer console access to guests via ssh. "cu" also works, but lacks rtty's background logging. HI, I got several suggestions, and in the end I went for this one. Why? The code base is small and simple it does exactly what I required, no more, no less. Getting it to work requires a bit of careful configuration but one you've done that you end up with SSH users which can do multiple logins and all see the same console. Input and things all the same. So no exclusion because you've left your terminal open at work and went home. Nifty stuff. Basic install: 1) pkg install rtty. This populates the /usr/local/rtty tree. And that is where just almost everything happens. 2) In /usr/local/rtty/dev create links to the nmdm-devices you'd like to connect to the console: root@bhyve00:/usr/local/rtty # ls -l dev total 1 lrwxr-xr-x 1 root wheel 14 Jan 22 11:13 bh1101A -> /dev/nmdm1101B 3) Create a "shell" to connect to this console: root@bhyve00:/usr/local/rtty # cat bin/rtty-shell #!/bin/sh exec /usr/local/rtty/bin/console "$LOGNAME" # exec /usr/bin/cu -t -l /dev/nmdm1101A # exec /usr/bin/tip vm1101 exit And make it executable: root@bhyve00:/usr/local/rtty # ls -l bin/rtty-shell -rwxr-xr-x 1 root wheel 127 Jan 22 11:15 bin/rtty-shell As you can see you can use other commands as well, but both tip and cu are all about exclusive access. Trick here is that LOGNAME is what the user is using for login, and that is also the name of the device in ../dev to which he/she is connected. 4) Create the user in /etc/passwd: root@bhyve00:/usr/local/rtty # grep bh1101A /etc/passwd bh1101A:*:10001:10001:Bhyve dome remote access user:/home/bhyvedemo:/usr/local/rtty/bin/rtty-shell And put him in /etc/group to make things complete: root@bhyve00:/usr/local/rtty # grep bh1101A /etc/group bh1101A:*:10001: Group is needed for the rights on the socket-device in step 6. 5) Which also requires an entry in /etc/shells to work: root@bhyve00:/usr/local/rtty # grep rtty /etc/shells /usr/local/rtty/bin/rtty-shell 6) Having done all this the we need to make sure that the accessrights on the linked device are set correctly: root@bhyve00:/usr/local/rtty # cat owner/bh1101A.sock root:bh1101A This will prevent another user from hijacking a console which is not his, in case of access by other means. 7) Last ting to do is to start the server deamon that does the multiplexing/sharing stuff: /usr/sbin/daemon -cf /usr/local/rtty/bin/Startup Which you could dump into /etc/rc.local, or the more brave would write a rc.d compliant startup script. 8) Test: root@bhyve00:/usr/local/rtty # ssh bh1101A@localhost Password for bh1101A@bhyve00: Last login: Wed Jan 21 12:42:37 2015 from mail.ip6.digiware.nl FreeBSD 11.0-CURRENT (GENERIC) #19 r277452: Wed Jan 21 08:01:54 UTC 2015 Welcome to FreeBSD! connected (use (CR)~? for minimal help; also (CR)~q? and (CR)~s?) [authorized] [bh1101A@/dev/pts/4 connected] FreeBSD/amd64 (FreeBSD-11) (ttyu0) login: Even if the VM is not yet running you can connect to the server, and then start the VM, to all of its booting output. Or interact with the loader. It also works to actually install a VM from CD. Not that on 10.0 installs the default console is OFF, en needs to be edited in /etc/ttys to ttyu0 "/usr/libexec/getty std.9600" vt100 on secure or ttyu0 "/usr/libexec/getty std.9600" vt100 onifconsole secure For more recent relesase where init(1) understand onifconsole Probably there are some more things to do, like getting rtty and ssh to be silent on login, etc But these are the essentials. Rinse and repeat steps 2, 4 and 6 to create a new access. After which you net to restart the server to get it to learn a new device. And even though I'm a CLI junky Next step would to see if we can use all kinds of java(script) sshlibs to connect, so you can embed this in a management webconsole. Enjoy, --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Running grub-bhyve in the background???
Hi (Peter), I'm trying to run grub-bhyve completely automated but when I run my version of vmrun.sh like ../bin/bhyve-run -f /usr/local/etc/ezbhyve/Ubuntu1204A/rc.conf & Thegrub-bhyve loader waits for me to forground it again, because it wants to write to the output. Probably this is due to the use of curses, which only continues if it is actually connected to the foreground. + /usr/local/sbin/grub-bhyve -e -vvv -r hd0,msdos1 -m ./ubuntu-12.04.map -M 2048M ubuntu-12.04.1 [2] + Suspended (tty output)../bin/bhyve-run -f /usr/local/etc/ezbhyve/Ubuntu1204A/rc.conf Is it possible on the commandline to get grub-bhyve to continue regardless the backgrounded state? Regards, --Willem Jan ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Running grub-bhyve in the background???
On 8-2-2015 2:35, Allan Jude wrote: > On 2015-02-07 20:04, Willem Jan Withagen wrote: >> Hi (Peter), >> >> I'm trying to run grub-bhyve completely automated but when I run my >> version of vmrun.sh like >> ../bin/bhyve-run -f /usr/local/etc/ezbhyve/Ubuntu1204A/rc.conf & >> >> Thegrub-bhyve loader waits for me to forground it again, because it >> wants to write to the output. Probably this is due to the use of curses, >> which only continues if it is actually connected to the foreground. >> >> >> + /usr/local/sbin/grub-bhyve -e -vvv -r hd0,msdos1 -m ./ubuntu-12.04.map >> -M 2048M ubuntu-12.04.1 >> >> [2] + Suspended (tty output)../bin/bhyve-run -f >> /usr/local/etc/ezbhyve/Ubuntu1204A/rc.conf >> >> >> Is it possible on the commandline to get grub-bhyve to continue >> regardless the backgrounded state? > In newer versions of the bhyve loader, you can redirect the output to a > nmdm (null modem) device, so it can run unattended bhyveloader does work like this. And even if there is nobody to listen to the loader, things to continue. It just takes 10 sec. But once in a while I have trouble getting grub-bhyve to act the same. If I connect it to a nmdm-device it just waits for me to connect. And if I do not specify a device, it is nog possible to run it in the background. A inbetween sulution at the moment is to run grub-bhyve -c /dev/null. That continues, dus does not offer the possibility to interfere in the boot process. At least not for my ubuntu-12.04 VMs. So a flag with grub-bhyve to not require a forground process, or even better: ignore flowcontrol on the -c device, would be nice. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Running grub-bhyve in the background???
On 8-2-2015 21:04, Yamagi Burmeister wrote: > On Sun, 08 Feb 2015 15:53:45 +0100 > Willem Jan Withagen wrote: > >> A inbetween sulution at the moment is to run grub-bhyve -c /dev/null. >> That continues, dus does not offer the possibility to interfere in the >> boot process. At least not for my ubuntu-12.04 VMs. > > I hacked around that problem by writing one bit into the nmdm device. > Or to say it in code: > > true > $NMDMB & > sleep 0.5 > /usr/local/sbin/grub-bhyve -r $BOOT -m $MAP -M $MEMORY -c $NMDMA $NAME & > > It's not a nice solution but at least it works reliable. Hi Yamagi, Nice trick. The advantage is probably that you can use the nmdm device in the grub-session if things do go south? I've starting ripping the guts out of Grub2 But it really looks like a huge swiss-army knive and has several stacked layers of routines. Even ignoring all languages, modules and odd platforms.. I've got it loading now, but very little output trace. Curses is taking most of it. :( --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Running grub-bhyve in the background???
On 8-2-2015 21:53, Jason Tubnor wrote: > For my OpenBSD guests, I just re-direct the grub-bhyve output to > /dev/null while feeding in my step-through configuration for > grub-bhyve. > > If I know how grub-bhyve is going to behave, I don't really need to > see what is happening at that level and just hook bhyve to nmdm. I agree with the later... And connecting stdout of grub-bhyve to /dev/null would probably work as well Will give it a shot. --WjW > On 9 February 2015 at 01:53, Willem Jan Withagen > wrote: >> On 8-2-2015 2:35, Allan Jude wrote: >>> On 2015-02-07 20:04, Willem Jan Withagen wrote: >>>> Hi (Peter), >>>> >>>> I'm trying to run grub-bhyve completely automated but when I >>>> run my version of vmrun.sh like ../bin/bhyve-run -f >>>> /usr/local/etc/ezbhyve/Ubuntu1204A/rc.conf & >>>> >>>> Thegrub-bhyve loader waits for me to forground it again, >>>> because it wants to write to the output. Probably this is due >>>> to the use of curses, which only continues if it is actually >>>> connected to the foreground. >>>> >>>> + /usr/local/sbin/grub-bhyve -e -vvv -r hd0,msdos1 -m >>>> ./ubuntu-12.04.map -M 2048M ubuntu-12.04.1 >>>> >>>> [2] + Suspended (tty output)../bin/bhyve-run -f >>>> /usr/local/etc/ezbhyve/Ubuntu1204A/rc.conf >>>> >>>> Is it possible on the commandline to get grub-bhyve to >>>> continue regardless the backgrounded state? >> >>> In newer versions of the bhyve loader, you can redirect the >>> output to a nmdm (null modem) device, so it can run unattended >> >> bhyveloader does work like this. And even if there is nobody to >> listen to the loader, things to continue. It just takes 10 sec. >> >> But once in a while I have trouble getting grub-bhyve to act the >> same. If I connect it to a nmdm-device it just waits for me to >> connect. And if I do not specify a device, it is nog possible to >> run it in the background. >> >> A inbetween sulution at the moment is to run grub-bhyve -c >> /dev/null. That continues, dus does not offer the possibility to >> interfere in the boot process. At least not for my ubuntu-12.04 >> VMs. >> >> So a flag with grub-bhyve to not require a forground process, or >> even better: ignore flowcontrol on the -c device, would be nice. >> >> --WjW ___ >> freebsd-virtualization@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To >> unsubscribe, send any mail to >> "freebsd-virtualization-unsubscr...@freebsd.org" > > > ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve Suspend/Resume
On 07/03/2015 23:51, Allan Jude wrote: > On 2015-03-07 17:46, Manas Bhatnagar wrote: >> Hello, >> >> I am trying to find out if Bhyve in 10-STABLE has support for suspending >> and resuming guests. Please let me know. A google search returned a >> mailing list thread from 2013 that shows it is a feature being worked >> on. Has this work completed or is it still ongoing? Please let me know. >> >> Thanks, >> Manas >> ___ >> freebsd-virtualization@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to >> "freebsd-virtualization-unsubscr...@freebsd.org" >> > > No, bhyve in stable/10 does not support suspend/resume. > This sort hints the suggestion that it is possible on -CURRENT. Is this the case? And if yes, how? --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Bhyve Suspend/Resume
On 09/03/2015 23:17, Neel Natu wrote: > Hi Willem, > > On Mon, Mar 9, 2015 at 2:45 PM, Willem Jan Withagen wrote: >> On 07/03/2015 23:51, Allan Jude wrote: >>> On 2015-03-07 17:46, Manas Bhatnagar wrote: >>>> Hello, >>>> >>>> I am trying to find out if Bhyve in 10-STABLE has support for suspending >>>> and resuming guests. Please let me know. A google search returned a >>>> mailing list thread from 2013 that shows it is a feature being worked >>>> on. Has this work completed or is it still ongoing? Please let me know. >>>> >>>> Thanks, >>>> Manas >>>> ___ >>>> freebsd-virtualization@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>>> To unsubscribe, send any mail to >>>> "freebsd-virtualization-unsubscr...@freebsd.org" >>>> >>> >>> No, bhyve in stable/10 does not support suspend/resume. >>> >> This sort hints the suggestion that it is possible on -CURRENT. >> Is this the case? > > Unfortunately its not available in -current as well. > > best > Neel > >> And if yes, how? Too bad, I hoped I just missed the news... Did find the way to send a APCI shutdown, by sending kill to the bhyve proces. Would be nice if suspend and resume would work more or less the same... --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve 20150316 crashed
Assertion failed: (err == 0), function pci_vtblk_proc, file /usr/srcs/src11/src/usr.sbin/bhyve/pci_virtio_block.c, line 276. I noticed some work by several people on the virtio part of the tree. This is with todays tree: bhyve00 11.0-CURRENT FreeBSD 11.0-CURRENT #72 r280010: En VM crashing is an older FreeBSD-head, whilest running svlite up. And err is actually 7, type = 1 --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve 20150316 crashed
On 16/03/2015 09:59, Alexander Motin wrote: > On 16.03.2015 10:56, Florian Smeets wrote: >> On 3/16/2015 1:12 AM, Willem Jan Withagen wrote: >>> Assertion failed: (err == 0), function pci_vtblk_proc, file >>> /usr/srcs/src11/src/usr.sbin/bhyve/pci_virtio_block.c, line 276. >>> >>> I noticed some work by several people on the virtio part of the tree. >>> >>> This is with todays tree: >>> bhyve00 11.0-CURRENT FreeBSD 11.0-CURRENT #72 r280010: >>> En VM crashing is an older FreeBSD-head, whilest running >>> svlite up. >>> >>> And err is actually 7, >>> type = 1 >>> >> >> I was about to report the same thing. I'm still running r280041 and >> wanted to repro with the latest changes before reporting. >> >> I've just now updates 7 bhyves to r280120 and am putting some load on it. > > I also hit this problem, and I hope it should be fixed by r280126. > Thanx, I'll svnup again, and build the new stuff. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve 20150316 crashed
On 16-3-2015 09:59, Alexander Motin wrote: On 16.03.2015 10:56, Florian Smeets wrote: On 3/16/2015 1:12 AM, Willem Jan Withagen wrote: Assertion failed: (err == 0), function pci_vtblk_proc, file /usr/srcs/src11/src/usr.sbin/bhyve/pci_virtio_block.c, line 276. I noticed some work by several people on the virtio part of the tree. This is with todays tree: bhyve00 11.0-CURRENT FreeBSD 11.0-CURRENT #72 r280010: En VM crashing is an older FreeBSD-head, whilest running svlite up. And err is actually 7, type = 1 I was about to report the same thing. I'm still running r280041 and wanted to repro with the latest changes before reporting. I've just now updates 7 bhyves to r280120 and am putting some load on it. I also hit this problem, and I hope it should be fixed by r280126. Alexander, Updated /usr/src, and then successfully updated the FreeBSD-head VM /usr/src. And it it now slugging away on building world. Which has not generated any problems. Thanx for the fast response. --WjW ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Program for dynamically making taps/bridge topologies.
On 11-10-2015 19:43, Alfred Perlstein wrote: > Hello, > > Here at Norse we are using bhyve to run our appliance in a test > environment. > > We have a need for a test network to be made on demand, so this weekend > I took some time and wrote a tool that allows you to specify a topology > of bridges and taps that you can then use to create your virtual network. > > In our case our test servers have a bridge0 that is static and bridges > to our private test network. However we also need a bridge to be > dynamically that we will generate traffic over in isolation. We then > need taps created and "assigned" or labeled properly for our QA suite to > attach to our VMs properly. > > We then also need the ability to query the topology and emit the correct > taps to assign to each vm. > > Finally we need the ability to tear down the virtual network once it's > no longer needed. > > This tool provides all three functions. > > The tool is available here: > https://github.com/splbio/netmanager > > Comments, pull requests and questions are welcome. Sounds like cool stuff This should fit nicely into what is now fashionably called SDN... Now I got to figure out a way to glue this to my home-grown vmrun.sh stuff. And the ultimate step could be to see if it matches the things that openstack trieds to do for networking. Looking at it right away. But seems that it is getting harder and harder not to want to understand/write python. --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Running a FreeBSD bhyve instance on a Ceph cluster
Hi, Just an info point. I'm preparing for a lecture tomorrow, and thought why not do an actual demo Like to be friends with Murphy :) So after I started the cluster: 5 jails with 7 OSDs This what I manually needed to do to boot a memory stick # Start een Bhyve instance rbd --dest-pool rbd_data --no-progress import memstick.img memstick rbd-ggate map rbd_data/memstick # ggate-devvice is available on /dev/ggate1 kldload vmm kldload nmdm kldload if_tap kldload if_bridge kldload cpuctl sysctl net.link.tap.up_on_open=1 ifconfig bridge0 create ifconfig bridge0 addm em0 up ifconfig ifconfig tap11 create ifconfig bridge0 addm tap11 ifconfig tap11 up # load the GGate disk in bhyve bhyveload -c /dev/nmdm11A -m 2G -d /dev/ggate1 FB11 # and boot a single from it. bhyve -H -P -A -c 1 -m 2G -l com1,/dev/nmdm11A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap11 -s 4,ahci-hd,/dev/ggate1 FB11 & bhyvectl --vm=FB11 --get-stats # Connect to the VM cu -l /dev/nmdm11B = And that'll give you a bhyve VM running on an RBD image over ggate. In the installer I tested reading from the bootdisk: root@:/ # dd if=/dev/ada0 of=/dev/null bs=32M 21+1 records in 21+1 records out 734077952 bytes transferred in 5.306260 secs (138341865 bytes/sec) which is a nice 138Mb/sec. Hope the demonstration does work out tomorrow. --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Adding a different type of blockstore to Bhyve
Hi,
One of the ways to run backing blockstore with KVM/Qemu is thru
the Ceph Rados Block Device (RBD).
https://github.com/qemu/qemu/blame/master/block/rbd.c
And is make it possible use as boot-image or other blockdevice. Where
the virtual machine using this image can migrate to another Dom0 host.
I've been working on Ceph for quite some time, and one of the ways to
offer a block device on FreeBSD is with rbd-ggate.
This works thru geom-gate and will give a /dev/ggate# device that is
mapped to an image in a rados pool.
And I not into migration for Bhyve, but I would like to integrate RBD
into Bhyve as an alternative backing store
Something like:
bhyve -s 1,virtio-blk,rbd:poolname/imagename[@snapshotname] \
[:option1=value1[:option2=value2...]]
So started browsing the bhyve code, and end up in block_if.{hc}.
But code there is rather strongly targeted towards a local
filesystem storage
I also ran into net_backends.{ch}, and I guess it would be a nicer
solution to create a block_backends.{ch} as well for interfacing to
more than just one blockstore provider.
And then load the RBD provider in the chain of blockstore providers.
That way would it even be possible to make that code dl-loadable in
case the LGPL ceph code is not directly importable in the usr.sbin
tree. (Which I suspect it is)
The alternative is to start using the /dev/ggate# devices but then
we probably lose the option of live migration.
And performance takes a serious hit:
A block write/read would go from the vm kernel
to the bhyve process in userspace.
Then it would go to/dev/ggate# and again end up in the kernel
only to have geom-gate send it back to userspace
where rbd-ggate sends it to the cluster.
Just typing this data flow is a lot of steps, showing that this
might not be the best architecture.
So the questions are:
1) Is the abstraction of block_backends.{ch} the way to go?
1.1) And would the extra indirection there be acceptable?
(For network devices it seems no problem)
2) Does anybody already have such a framework for blockdevs?
(Otherwise I'll try to morph the net_backends.{ch}
3) Other suggestions I need to consider?
Thanx,
--WjW
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Adding a different type of blockstore to Bhyve
On 30-12-2019 19:06, Willem Jan Withagen wrote:
Hi,
One of the ways to run backing blockstore with KVM/Qemu is thru
the Ceph Rados Block Device (RBD).
https://github.com/qemu/qemu/blame/master/block/rbd.c
And is make it possible use as boot-image or other blockdevice. Where
the virtual machine using this image can migrate to another Dom0 host.
I've been working on Ceph for quite some time, and one of the ways to
offer a block device on FreeBSD is with rbd-ggate.
This works thru geom-gate and will give a /dev/ggate# device that is
mapped to an image in a rados pool.
And I not into migration for Bhyve, but I would like to integrate RBD
into Bhyve as an alternative backing store
Something like:
bhyve -s 1,virtio-blk,rbd:poolname/imagename[@snapshotname] \
[:option1=value1[:option2=value2...]]
So started browsing the bhyve code, and end up in block_if.{hc}.
But code there is rather strongly targeted towards a local
filesystem storage
I also ran into net_backends.{ch}, and I guess it would be a nicer
solution to create a block_backends.{ch} as well for interfacing to
more than just one blockstore provider.
And then load the RBD provider in the chain of blockstore providers.
That way would it even be possible to make that code dl-loadable in
case the LGPL ceph code is not directly importable in the usr.sbin
tree. (Which I suspect it is)
The alternative is to start using the /dev/ggate# devices but then
we probably lose the option of live migration.
And performance takes a serious hit:
A block write/read would go from the vm kernel
to the bhyve process in userspace.
Then it would go to/dev/ggate# and again end up in the kernel
only to have geom-gate send it back to userspace
where rbd-ggate sends it to the cluster.
Just typing this data flow is a lot of steps, showing that this
might not be the best architecture.
So the questions are:
1) Is the abstraction of block_backends.{ch} the way to go?
1.1) And would the extra indirection there be acceptable?
(For network devices it seems no problem)
2) Does anybody already have such a framework for blockdevs?
(Otherwise I'll try to morph the net_backends.{ch}
3) Other suggestions I need to consider?
Looking for reviewers of:
https://reviews.freebsd.org/D23010
In the days after newyear I made a first attempt to refactor the
block_if stuff into a generic backend: blockbe_ and and implementation
of the local storage: lockblk_
I've submitted it to phabricator, and I'm seeking reviews and ultimately
somebody that will commit this when all issues are worked out.
--WjW
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: Adding a different type of blockstore to Bhyve
On 31-12-2019 00:48, Paul Vixie wrote:
On Monday, 30 December 2019 18:06:11 UTC Willem Jan Withagen wrote:
Something like:
bhyve -s 1,virtio-blk,rbd:poolname/imagename[@snapshotname] \
[:option1=value1[:option2=value2...]]
this is approximately how i'd hope to do object-store level ZFS integration,
so as to avoid the zvol abstraction. i know you're working on Ceph not ZFS but
the concepts and facilities are similar enough to warrant cooperative
thinking.
So the questions are:
1) Is the abstraction of block_backends.{ch} the way to go?
1.1) And would the extra indirection there be acceptable?
(For network devices it seems no problem)
2) Does anybody already have such a framework for blockdevs?
(Otherwise I'll try to morph the net_backends.{ch}
3) Other suggestions I need to consider?
i think you're hitting an architectural limit, and that the bhyve design team
should be thinking about a third way, one which would also solve my own
loopback and mmap requirements as i've described variously. what you want to
do should not only be possible, it should be clean and performant.
Hi Paul,
Thanx for the encouraging words
I've missed the discussion on loopback and mmap, I just don't have the
time to read all FreeBSD and other groups. So once in a while I just do
major catch up by delete. ;-)
In the mean time I have submitted:
https://reviews.freebsd.org/D23010
To give block_if a generic interface.
Feel free to comment on it, and suggest changes that will help you with
what you are looking at.
--WjW
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
[RFC] Adding a Rados block driver to bhyve
Hi all, And sorry for crosspoing three groups, but the answer can/could be a mix of things to do in these three areas. I have a prototype of bhyve running on Rados/Ceph working: https://github.com/freebsd/freebsd/pull/426 But there are a few catches on how to get it in the FreeBSd sources... 1) Easiest would be to just compile it in with the code of the current bhyve. That will require librados/librbd libraries... Ceph of this purpose is LGPL2/3 and could go into contrib. In this case bhyve will hold the rbd-driver by default and a user does not need to do anything by himself But I have the feeling that this is the most unwanted scenario 2) User first installs a Ceph package and FreeBSD sources, and then recompiles bhyve with the option BHYVE_RBD. And then reinstalls this new version as bhyve or bhyve-rbd in /usr/sbin 3) Create a bhyve-rbd port. Problem with that is that it will require the FreeBSD source tree for the bhyve sources, but there is no Ports option for that? Or bhyve sources are manually copied into the port. And then try to keep these sources up to date. Then compile and install a bhyve-rbd into /usr/local/sbin 4) Create a bhyve-blockrbd port. This is much like 3) but instead of building a bhyve-rbd executable, it delivers a libblockrbd.so that is dynamically loadable by the standaard bhyve that comes with base. For this bhyve needs to be extended with dynamic loadable driver modules. This is reasonably doable, but is this acceptable for the bhyve maintainers? For building the port, the bhyve-blockrbd code will only need a limited set of files from /usr/src/usr.bin/bhyve thus limiting the chance of running out sequence with the bhyve from base. Looking over these 4 options, I think that 4 is the most desirable one? But 2 would parhaps be workable for users as well, but the project might think otherwise. Are there other options? And/or is 4 the best way to go, with 2 as a nice intermediate? Thanx, --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: [RFC] Adding a Rados block driver to bhyve
On 9-3-2020 14:46, Alan Somers wrote: On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen <mailto:w...@digiware.nl>> wrote: Hi all, And sorry for crosspoing three groups, but the answer can/could be a mix of things to do in these three areas. I have a prototype of bhyve running on Rados/Ceph working: https://github.com/freebsd/freebsd/pull/426 .. 4) Create a bhyve-blockrbd port. This is much like 3) but instead of building a bhyve-rbd executable, it delivers a libblockrbd.so that is dynamically loadable by the standaard bhyve that comes with base. > Great work! I also agree that option 4 sounds like the best. There's precedent for ports that > require the FreeBSD Sources. For example, see devel/py-libzfs or emulators/virtualbox-ose. > You just need to define the SRC_BASE variable. Hi Alan, Thanx for the hint, and it made me check what is actually available within the poudriere jail And that does have full source, so the Makefile code is mainly for those that build in a different way. I've got a proto version working when compiling stuff with `make buildworld`, but run in the problem that libblock_rbd.so is stripped in such a way that the symbol I need is removed. Using the unstripped version does work. Is there an incantation for the SRC Makefiles that builds a dynamical loadable lib?? And I'm still looking for a PORTS example of building a dynamical loadable lib. Or is there no generic code for that in the PORTS Mk files? --WjW BTW: Still haven't worked in your AIO code :( ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: [RFC] Adding a Rados block driver to bhyve
On 10-3-2020 16:15, Alan Somers wrote:
On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen <mailto:w...@digiware.nl>> wrote:
On 9-3-2020 14:46, Alan Somers wrote:
On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen
mailto:w...@digiware.nl>> wrote:
Hi all,
And sorry for crosspoing three groups, but the answer
can/could be a mix
of things to do in these three areas.
I have a prototype of bhyve running on Rados/Ceph working:
https://github.com/freebsd/freebsd/pull/426
..
4) Create a bhyve-blockrbd port.
This is much like 3) but instead of building a bhyve-rbd
executable,
it delivers a libblockrbd.so that is dynamically
loadable by the
standaard bhyve that comes with base.
> Great work! I also agree that option 4 sounds like the best.
There's precedent for ports that
> require the FreeBSD Sources. For example, see devel/py-libzfs
or emulators/virtualbox-ose.
> You just need to define the SRC_BASE variable.
Hi Alan,
Thanx for the hint, and it made me check what is actually
available within the poudriere jail
And that does have full source, so the Makefile code is mainly for
those that build in a different way.
I've got a proto version working when compiling stuff with `make
buildworld`, but run in the
problem that libblock_rbd.so is stripped in such a way that the
symbol I need is removed.
Using the unstripped version does work.
Is there an incantation for the SRC Makefiles that builds a
dynamical loadable lib??
And I'm still looking for a PORTS example of building a dynamical
loadable lib.
Or is there no generic code for that in the PORTS Mk files?
--WjW
BTW: Still haven't worked in your AIO code :(
There are plenty of dynamic libraries built with the SRC makefiles.
For example,
https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup
.
That looks dangerously close to what I have for libblock_rbd.
===
> cat Makefile-librbd
#
# $FreeBSD$
#
PACKAGE=lib${LIB}
.include
LIB= block_rbd
SHLIB_MAJOR= 1
SRCS= block_rbd.c
CFLAGS+=-I${SRCTOP}/sys
CFLAGS+=-g -O0 -fPIC -rdynamic
LDFLAGS+=-Wl,-export-dynamic,-Bdynamic
CFLAGS+=-DWITHOUT_CAPSICUM
LOCALBASE?= /usr/local
CFLAGS+= -I${LOCALBASE}/include
LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd
WARNS?= 2
===
This is the code that mk.lib.bsd runs:
objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug
objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug
libblock_rbd.so.1.full libblock_rbd.so.1
So still I get a stripped lib in /usr/lib. And then the one and only
symbol I need to load
is not found. Copying libblock_rbd.so.1.full actually works for me.
So either I'm doing it the wrong way, like special options on the
symbols oid.
Or mk.lib.bsd cannot deliver dlopen/dlsym-able files?
And there are plenty of ports that build shared libraries too, just look
at /usr/local/lib/*.so. However, the ports framework doesn't have much
special code just to support building libraries. Instead the hard work
is always done by the ports themselves. Some use autotools, some cmake,
etc etc. The simplest port I can find that uses both SRC_BASE and
INSTALL_LIB is this one:
https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ .
Oke thanx, I'll have a look at it, and given that I can see most of the
compile build stuff
in the SRC_BASE version I'll get it to work.
--WjW
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [RFC] Adding a Rados block driver to bhyve
On 10-3-2020 17:21, Alan Somers wrote:
On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen <mailto:w...@digiware.nl>> wrote:
On 10-3-2020 16:15, Alan Somers wrote:
On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen
mailto:w...@digiware.nl>> wrote:
On 9-3-2020 14:46, Alan Somers wrote:
On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen
mailto:w...@digiware.nl>> wrote:
Hi all,
And sorry for crosspoing three groups, but the answer
can/could be a mix
of things to do in these three areas.
I have a prototype of bhyve running on Rados/Ceph working:
https://github.com/freebsd/freebsd/pull/426
..
4) Create a bhyve-blockrbd port.
This is much like 3) but instead of building a
bhyve-rbd executable,
it delivers a libblockrbd.so that is dynamically
loadable by the
standaard bhyve that comes with base.
> Great work! I also agree that option 4 sounds like the
best. There's precedent for ports that
> require the FreeBSD Sources. For example, see
devel/py-libzfs or emulators/virtualbox-ose.
> You just need to define the SRC_BASE variable.
Hi Alan,
Thanx for the hint, and it made me check what is actually
available within the poudriere jail
And that does have full source, so the Makefile code is
mainly for those that build in a different way.
I've got a proto version working when compiling stuff with
`make buildworld`, but run in the
problem that libblock_rbd.so is stripped in such a way that
the symbol I need is removed.
Using the unstripped version does work.
Is there an incantation for the SRC Makefiles that builds a
dynamical loadable lib??
And I'm still looking for a PORTS example of building a
dynamical loadable lib.
Or is there no generic code for that in the PORTS Mk files?
--WjW
BTW: Still haven't worked in your AIO code :(
There are plenty of dynamic libraries built with the SRC
makefiles. For example,
https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup
.
That looks dangerously close to what I have for libblock_rbd.
===
> cat Makefile-librbd
#
# $FreeBSD$
#
PACKAGE=lib${LIB}
.include http://src.opts.mk>>
LIB= block_rbd
SHLIB_MAJOR= 1
SRCS= block_rbd.c
CFLAGS+=-I${SRCTOP}/sys
CFLAGS+=-g -O0 -fPIC -rdynamic
LDFLAGS+=-Wl,-export-dynamic,-Bdynamic
CFLAGS+=-DWITHOUT_CAPSICUM
LOCALBASE?= /usr/local
CFLAGS+= -I${LOCALBASE}/include
LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd
WARNS?= 2
===
This is the code that mk.lib.bsd runs:
objcopy --only-keep-debug libblock_rbd.so.1.full
libblock_rbd.so.1.debug
objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug
libblock_rbd.so.1.full libblock_rbd.so.1
So still I get a stripped lib in /usr/lib. And then the one and
only symbol I need to load
is not found. Copying libblock_rbd.so.1.full actually works for me.
So either I'm doing it the wrong way, like special options on the
symbols oid.
Or mk.lib.bsd cannot deliver dlopen/dlsym-able files?
And there are plenty of ports that build shared libraries too,
just look at /usr/local/lib/*.so. However, the ports framework
doesn't have much special code just to support building
libraries. Instead the hard work is always done by the ports
themselves. Some use autotools, some cmake, etc etc. The simplest
port I can find that uses both SRC_BASE and INSTALL_LIB is this
one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ .
Oke thanx, I'll have a look at it, and given that I can see most
of the compile build stuff
in the SRC_BASE version I'll get it to work.
--WjW
Try setting "STRIP= " in your makefile. That should prevent the
stripping. However, I think there's something wrong with your
library, too. The library should be usable even if it's stripped.
I checked with objdump, and the symbol that I need is definitly not
present in the stripped version.
And it does not really matter if I declare it static or not.
But I'll give it a few more itterations to try it out.
Including 'STRIP= '
Thanx,
--WjW
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: [RFC] Adding a Rados block driver to bhyve
On 10-3-2020 17:48, Alan Somers wrote:
On Tue, Mar 10, 2020 at 10:28 AM Willem Jan Withagen
wrote:
On 10-3-2020 17:21, Alan Somers wrote:
On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen
wrote:
On 10-3-2020 16:15, Alan Somers wrote:
On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen
wrote:
On 9-3-2020 14:46, Alan Somers wrote:
On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen
wrote:
Hi all,
And sorry for crosspoing three groups, but the answer can/could be a mix
of things to do in these three areas.
I have a prototype of bhyve running on Rados/Ceph working:
https://github.com/freebsd/freebsd/pull/426
..
4) Create a bhyve-blockrbd port.
This is much like 3) but instead of building a bhyve-rbd
executable,
it delivers a libblockrbd.so that is dynamically loadable by the
standaard bhyve that comes with base.
Great work! I also agree that option 4 sounds like the best. There's
precedent for ports that
require the FreeBSD Sources. For example, see devel/py-libzfs or
emulators/virtualbox-ose.
You just need to define the SRC_BASE variable.
Hi Alan,
Thanx for the hint, and it made me check what is actually available
within the poudriere jail
And that does have full source, so the Makefile code is mainly for those
that build in a different way.
I've got a proto version working when compiling stuff with `make
buildworld`, but run in the
problem that libblock_rbd.so is stripped in such a way that the symbol I
need is removed.
Using the unstripped version does work.
Is there an incantation for the SRC Makefiles that builds a dynamical
loadable lib??
And I'm still looking for a PORTS example of building a dynamical
loadable lib.
Or is there no generic code for that in the PORTS Mk files?
--WjW
BTW: Still haven't worked in your AIO code :(
There are plenty of dynamic libraries built with the SRC makefiles. For
example,
https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup
.
That looks dangerously close to what I have for libblock_rbd.
===
cat Makefile-librbd
#
# $FreeBSD$
#
PACKAGE=lib${LIB}
.include
LIB=block_rbd
SHLIB_MAJOR=1
SRCS= block_rbd.c
CFLAGS+=-I${SRCTOP}/sys
CFLAGS+=-g -O0 -fPIC -rdynamic
LDFLAGS+=-Wl,-export-dynamic,-Bdynamic
CFLAGS+=-DWITHOUT_CAPSICUM
LOCALBASE?= /usr/local
CFLAGS+=-I${LOCALBASE}/include
LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd
WARNS?= 2
===
This is the code that mk.lib.bsd runs:
objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug
objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug
libblock_rbd.so.1.full libblock_rbd.so.1
So still I get a stripped lib in /usr/lib. And then the one and only
symbol I need to load
is not found. Copying libblock_rbd.so.1.full actually works for me.
So either I'm doing it the wrong way, like special options on the symbols
oid.
Or mk.lib.bsd cannot deliver dlopen/dlsym-able files?
And there are plenty of ports that build shared libraries too, just look
at /usr/local/lib/*.so. However, the ports framework doesn't have much
special code just to support building libraries. Instead the hard work is
always done by the ports themselves. Some use autotools, some cmake, etc
etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB
is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ .
Oke thanx, I'll have a look at it, and given that I can see most of the
compile build stuff
in the SRC_BASE version I'll get it to work.
--WjW
Try setting "STRIP=" in your makefile. That should prevent the
stripping. However, I think there's something wrong with your library,
too. The library should be usable even if it's stripped.
I checked with objdump, and the symbol that I need is definitly not
present in the stripped version.
And it does not really matter if I declare it static or not.
But I'll give it a few more itterations to try it out.
Including 'STRIP= '
Thanx,
--WjW
What does "nm --dynamic libblock_rbd.so.1" show?
Don't know what has changed, but I just rebuild everything.
And now it seems to work ;-)
One thing I'm wondering if th DATA_SET(block_backend_set, )
gets updated when dyn-loading the library it is in?
Or if not, how do I add an entry to this DATA_SET, once we are running
the program?
I'm dlsym() looking for block_backend_rbd:
w _Jv_RegisterClasses
U __assert
w __cxa_finalize
U __error
U __stack_chk_fail
U __stack_chk_guard
6098 D __start_set_block_backend_set
U __stderrp
60a0 D __stop_set_block_backend_set
475c T _fini
474c T _init
6008 D block_backend_rbd
6270 B blocklocal_backend
2170 T blockrbd_cleanup
0
Re: [RFC] Adding a Rados block driver to bhyve
On 10-3-2020 19:08, Conrad Meyer wrote: On Tue, Mar 10, 2020 at 9:28 AM Willem Jan Withagen wrote: problem that libblock_rbd.so is stripped in such a way that the symbol I need is removed. So either I'm doing it the wrong way, like special options on the symbols oid. However, I think there's something wrong with your library, too. The library should be usable even if it's stripped. Yes, strip only removes local symbols (.symtab), not exported ones (.dynsym). I checked with objdump, and the symbol that I need is definitly not present in the stripped version. How are you defining the symbol intended for export? Is the symbol a function or data? Does the compiler flag -fvisibility=hidden get used? Which symbol is missing and what are the symptoms? And it does not really matter if I declare it static or not. Not declaring it "static" is necessary, if not sufficient. Looking at your code on github, here are some issues: * In block_if.h, you define an object blocklocal_backend. This is a header, and every compilation unit that pulls in the header will get its own copy of blocklocal_backend. You probably want 'extern block_backend_t blocklocal_backend;' instead. I have fixed this in a different way, so I do not need to play tricks with blocklocal_backend to handle the legacy case.So that is removed. * You SET_DECLARE block_backend_set in block_if.c, but I think it needs to be in block_if.h. Looked at the Macro definition, and it is `extern` so it can go into block_if.h * There is some weirdness around linker sets being removed by the linker if they are empty, so you may want to add blockbackend_local to the linker set in the main program. (It's unclear to me why blockbackend_local is treated specially regardless.) However, I'm not quite sure why DATA_SET() in block_rbd.c is not creating __start_set_block_backend_set / __stop_set_block_backend_set exported symbols. As Alan asked, please provide 'nm -D foo.so'. Best, Conrad It is dynamic loaded with dlopen()/dlsym at the moment. But are you suggesting that a `DATA_SET()` actually adds to the set during dynamic loading? Note that the __start_set_block_backend_set / __stop_set_block_backend_setis avialable in the library. --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
On 22-1-2021 11:09, Matt Churchyard wrote: Hello, all, We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. [1] https://reviews.freebsd.org/D28270 Thank you, Elena I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") The current process (from a mile high overview) effectively seems to be the following - * Start guest on host2 pointing at a shared disk, and halt any execution * use bhyvectl to pause and begin migration on host1 * Start the guest on host2 What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - * send a ZFS snapshot of the running machine to host2 * start the guest in migrate recv mode on host2 * pause the guest on host1 * send a new snapshot * initiate the migration of memory/device data * start guest on host2 Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? Shared storage is great but becomes complex and expensive if you want high performance and reliability (seeing as the storage quickly becomes a major single point of failure without enterprise active/active kit). I suspect the most common deployment of bhyve is independent hosts with local ZFS pools as this is easy/cheap and gives great performance. Most hypervisors only had "shared storage" migration for a long time but the big ones now also support transferring disk data live. It would be great to be able to do this out of the gate. Great work to see this landing. This is actually one of the reasons I started porting Ceph and build a bhyve backend that interfaces to Ceph RBD. So you do not need to migrate the disk data. You are right that Ceph does not come cheap in hardware, but the designers try to jump through hoops to make it reliable. And it will also take some performance, but I'm very happy running my FreeBSD-vm's on our Openstack/Ceph cluster. Warm migration has been on the horizon for quite some time, so I very happy to see it getting there. Unfortunately I have limited time ATM to actually see if warm migration could be made to work with Ceph a diskstorage. --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Warm Migration feature for bhyve - review on Phabricator
On 25-1-2021 19:42, John-Mark Gurney wrote: Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +: -Original Message- From: John-Mark Gurney Sent: 25 January 2021 06:21 To: Matt Churchyard Cc: Elena Mihailescu ; freebsd-virtualization@freebsd.org Subject: Re: Warm Migration feature for bhyve - review on Phabricator Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +: Hello, all, We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. [1] https://reviews.freebsd.org/D28270 Thank you, Elena I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") The current process (from a mile high overview) effectively seems to be the following - * Start guest on host2 pointing at a shared disk, and halt any execution * use bhyvectl to pause and begin migration on host1 * Start the guest on host2 What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - * send a ZFS snapshot of the running machine to host2 * start the guest in migrate recv mode on host2 * pause the guest on host1 * send a new snapshot * initiate the migration of memory/device data * start guest on host2 Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? There's also hastd which can aid with this... Thanks for the reply. I've always been wary of the additional complexity of HAST and ZFS, as it doesn't seem to have widespread usage or support, and things get ugly fast when storage is involved. Totally agree... However, the idea of using HAST on top of zvols to provide network mirrored storage for a guest is interesting. It adds a lot of extra complexity, and probably performance impact though if it's just for the ability to move a guest between systems that may only happen every now and then. I'm also not sure it would help (or would at least be even more complex) if I have 4 hosts and want to be able to move guests anywhere. gmirror + ggate is another option as well... I did try all kinds of these solutions that are based on a 2 system backend storage, and it was relatively easy for me to get the state of the storage in a split-brain situation which is not really repairable without a lot of manual user intervention. So I gave up on HAST and gmirror over ggate. --WjW ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
