bhyve: regression in virtio-9p option parsing
Hi,
I'm noticing a behavior that looks like a regression in virtio-9p option
parsing.
I have a command line that has been working for me for a while, relevant
part is:
bhyve ... -s 7:0,virtio-9p,distfiles=/workspace/distfiles ...
At some point it started to fail with:
virtio-9p: more than one share name given
Even though there's only one share name.
As I can see, the relevant code for that is:
static int
pci_vt9p_legacy_config(nvlist_t *nvl, const char *opts)
{
char *sharename, *tofree, *token, *tokens;
if (opts == NULL)
return (0);
tokens = tofree = strdup(opts);
while ((token = strsep(, ",")) != NULL) {
if (strchr(token, '=') != NULL) {
if (sharename != NULL) {
EPRINTLN(
"virtio-9p: more than one share name given");
return (-1);
}
sharename = strsep(, "=");
set_config_value_node(nvl, "sharename", sharename);
set_config_value_node(nvl, "path", token);
} else
set_config_bool_node(nvl, token, true);
}
free(tofree);
return (0);
}
And it fails at the first iteration, likely because the sharename was
not initialised and points to some arbitrary non-NULL value.
Explicitly setting it to NULL like this:
diff --git a/usr.sbin/bhyve/pci_virtio_9p.c b/usr.sbin/bhyve/pci_virtio_9p.c
index e27159eb22cb..830e13878a71 100644
--- a/usr.sbin/bhyve/pci_virtio_9p.c
+++ b/usr.sbin/bhyve/pci_virtio_9p.c
@@ -232,7 +232,7 @@ pci_vt9p_notify(void *vsc, struct vqueue_info *vq)
static int
pci_vt9p_legacy_config(nvlist_t *nvl, const char *opts)
{
- char *sharename, *tofree, *token, *tokens;
+ char *sharename = NULL, *tofree, *token, *tokens;
if (opts == NULL)
return (0);
appears to fall back to the normal behavior: it accepts a single share
name just fine, but shows that error if I try to specify more than one
share name, e.g.:
-s 7:0,virtio-9p,distfiles=/workspace/distfiles,foo=bar
It looks like the regression was introduced by the global variables
commit (https://reviews.freebsd.org/D26035). The old code had
"char *sharename = NULL".
Roman Bogorodskiy
signature.asc
Description: PGP signature
Re: RFC: bhyve supported features reporting
Marcelo Araujo wrote:
> Hi Roman,
>
> I think it is doable, my only concern is the complexity and
> maintainability, as along the way we will add more devices and would be
> nice to have something transparent for those additions or at least
> something easy to be extended.
Yeah, it's true, I think it would be good to make it more compact
somehow. I'll play around with macros to see if I can make it easier to
add new items.
> As an example this patch that you made:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210111
>
> It has much less complexity, and seemed for me to be much easier to
> maintain, however doesn't give us a detailed information like the latest
> one you are proposing, I'm not sure if we really need so extensive details,
> I didn't see any bhyve device changing its options along the past years,
> but that doesn't means it would not in the future.
Yeah, this patch makes things easier, but doesn't cover all the cases.
For example, at some point VNC authentication was added [1] through
adding a 'password' keyword, but with an approach like that it's not
possible to know if the given bhyve version supports that or not.
1: https://svnweb.freebsd.org/base?view=revision=319487
> I do believe if we remove the complexity or at least makes it easy to be
> extend, IMHO it is a good idea.
>
> Best,
>
>
> 2018-06-24 18:13 GMT+08:00 Roman Bogorodskiy :
>
> > Bhyve evolves over time, and new features get added, such as new
> > devices support, new guest configurations and so on. However, it's not
> > really straight-forward to figure out what features a given bhyve binary
> > supports. This makes it harder to develop tools on top of bhyve,
> > specifically error reporting.
> >
> > For example, libvirt's bhyve driver [1] probes bhyve capabilities [2]
> > using:
> >
> > * Running 'bhyve -h' and parsing output,
> > * For detecting devices, it runs 'bhyve -s 0,dev' and parses error
> >output to see if the device is supported.
> >
> > It's not very effective because 'bhyve' binary has to be executed
> > multiple times just to check things. Additionally, it's harder to check
> > things on a deeper level, e.g. specific device parameters. Also, this is
> > error-prone because help output generally is not designed for
> > machine-parsing, and this could easily break on slight formatting
> > change.
> >
> > I would like to discuss introducing a general way of reporting features
> > bhyve supports. To start a discussion, I've created a simple draft/PoC
> > which adds '-f' (as for features) command line switch to bhyve. This
> > switch makes bhyve print its supported features. I'll use JSON as
> > example, however, as it's done using libxo, XML is also supported.
> > Example JSON output with inline comments below:
> >
> > "bhyve": {
> >
> > // 'features' schema version using the common versioning pattern:
> > //major version increments on incompatible changes,
> > //minor version increments on compatible changes (extensions).
> > "schema_version": "1.0",
> >
> > // there could also go some general properties like max_cpus etc
> >
> > // list of specific features, mostly should be self-descriptive
> > "features": {
> > "rtc_utc": {
> > "description": "RTC keeps UTC time",
> > "cmd": {
> > "switch": "-u"
> > }
> > },
> > "wire_guest_memory": {
> > "description": "Wire guest memory",
> > "cmd": {
> > "switch": "-S"
> > }
> > },
> > "devices": {
> > "description": "Devices support",
> > "cmd": {
> > "switch": "-s",
> > "arguments": [
> >
> > // Probably, it'd be better to make this more
> > organized,
> > // e.g. instead of a string with all arguments and
> > arg/value paris,
> > // use a list of objects which would include
> > possible values,
> > // required/optional, etc
> >
> > {"options": "virtio-net,tapN,mac=xx:xx:xx:xx:xx:xx",
> >"description": "Virtio network device"
> >
RFC: bhyve supported features reporting
Bhyve evolves over time, and new features get added, such as new
devices support, new guest configurations and so on. However, it's not
really straight-forward to figure out what features a given bhyve binary
supports. This makes it harder to develop tools on top of bhyve,
specifically error reporting.
For example, libvirt's bhyve driver [1] probes bhyve capabilities [2] using:
* Running 'bhyve -h' and parsing output,
* For detecting devices, it runs 'bhyve -s 0,dev' and parses error
output to see if the device is supported.
It's not very effective because 'bhyve' binary has to be executed
multiple times just to check things. Additionally, it's harder to check
things on a deeper level, e.g. specific device parameters. Also, this is
error-prone because help output generally is not designed for
machine-parsing, and this could easily break on slight formatting
change.
I would like to discuss introducing a general way of reporting features
bhyve supports. To start a discussion, I've created a simple draft/PoC
which adds '-f' (as for features) command line switch to bhyve. This
switch makes bhyve print its supported features. I'll use JSON as
example, however, as it's done using libxo, XML is also supported.
Example JSON output with inline comments below:
"bhyve": {
// 'features' schema version using the common versioning pattern:
//major version increments on incompatible changes,
//minor version increments on compatible changes (extensions).
"schema_version": "1.0",
// there could also go some general properties like max_cpus etc
// list of specific features, mostly should be self-descriptive
"features": {
"rtc_utc": {
"description": "RTC keeps UTC time",
"cmd": {
"switch": "-u"
}
},
"wire_guest_memory": {
"description": "Wire guest memory",
"cmd": {
"switch": "-S"
}
},
"devices": {
"description": "Devices support",
"cmd": {
"switch": "-s",
"arguments": [
// Probably, it'd be better to make this more organized,
// e.g. instead of a string with all arguments and
arg/value paris,
// use a list of objects which would include possible
values,
// required/optional, etc
{"options": "virtio-net,tapN,mac=xx:xx:xx:xx:xx:xx",
"description": "Virtio network device"
},
{"options":
"virtio-blk,path,nocache,direct,ro,sectorsize=logical/physical",
"description": "Virtio block device"
},
{"options":
"fbuf,rfb,rfb=IP:port,w=width,h=heigh,vga=vgaconf,wait,password=password",
"description": "Framebuffer device"
}
]
}
}
}
}
Sample code is here: https://reviews.freebsd.org/D15992. At this point
it's just an excuse to start a discussion; it needs some macros to make
items creation easier, and it needs to have all the other
features/devices populated.
1: https://libvirt.org/drvbhyve.html
2:
https://github.com/libvirt/libvirt/blob/master/src/bhyve/bhyve_capabilities.c
Roman Bogorodskiy
signature.asc
Description: PGP signature
Re: Call for testing bhyve cpu topology additions
Rodney W. Grimes wrote: > There is a new version of the CPU topology review up at > http://reviews.freebsd.org/D9930 > > I would like to ask that if people can test this and provide > feedback that they do so. > > It has some undesired side effects on vm-bhyve and probably > other down stream tools, I am in the process of contacting > the vm-bhyve author to see what can be done to clean up > this output (if you are here please respond to this thread): > > src-topo # vm list > NAMEDATASTORE LOADER CPUMEMORYVNC > AUTOSTARTSTATE > issd30g1default bhyveload > cpus=8,sockets=2,cores=4,threads=1 1024M -No > Stopped > > Notice that due to the new CPU string being much more complicated than > the normal int it makes the output format ugly. I have another change > to vm-bhyve that I would like to see too, and that is to move the NAME > to the end of the line and remove the 16 character limit. I have done > that in my local copy already. This string and its parsing are designed > to be Qemu compatible. > > Here is a sample of my local vm-bhyve vm list output (no topology used): > /tmp # vm list > DATASTORE LOADER CPUMEMORYVNC AUTOSTART > STATENAME > default bhyveload 1 128M -No > Stopped fb-bld-10-amd64 > default bhyveload 1 128M -No > Stopped fb-bld-11-amd64 > default bhyveload 1 128M -No > Stopped fb-bld-11.0-p1-amd64 > default bhyveload 1 128M -No > Stopped fb-bld-11.0-p1-i386 > default bhyveload 4 512M -No > Stopped fb-bld-11_1-amd64 > default bhyveload 4 512M -No > Stopped fb-bld-11_1-i386 > default bhyveload 2 256M -No > Running (30227) fb-bld-head-amd64 > > Thoughts are to teach vm-bhyve to parse the string just as bhyve does > and only output the vCPU count. Other thoughts I have are to have > it parse the string and output either vCPU if cores/threads is 1, > or a simple S/C/T string. > > If you are a downstream maintainer of one of the other vm management packages > I am open to input. The implemntation I have done should allow any existing > tool that treated -c as a string to use the new topology without changes. > This includes the in base vmrun.sh. My general input on adding new features to bhyve is that it would be nice to have a way to detect if the given bhyve version supports this specific feature. In this specific case it looks like this could be checked by running bhyve -c gibberish and checking if the error message contains 'invalid cpu topology' string. But generally it would be good to have some more convenient way of doing that because running bhyve to probe each feature is pretty expensive. By the way, looking at the DR, it seems that the usage output (bhyve -h) wasn't updated, but maybe I'm missing that without context. > Also people using the sysctls: > hw.vmm.topology.cores_per_package: 1 > hw.vmm.topology.threads_per_core: 1 > can continue to do so at this time, but there is work in process to > deprecate these, that work includes making stable/11 emit a warning > message if they are used, and remove them in head/12. > > If I can get some significant test results back I plan to commit > D9930 to ^head and merge it back to stable/11 3 days later. > > Thanks, > -- > Rod Grimes rgri...@freebsd.org Roman Bogorodskiy signature.asc Description: PGP signature
bhyve / libvmmapi usage
Hi, A couple of questions on the libvmmapi lib: - Is that a "public" library intended for a wide audience or sort of an internal lib to be used by bhyve(8) and friends? - Somewhat continuation of the first question: any expectations on libvmmapi API/ABI stability? Thanks, Roman Bogorodskiy signature.asc Description: PGP signature
Re: bhyve graphics support
Peter Grehan wrote: > Hi Roman, > > > I've just tried to do a Fedora installation and it worked like a charm. > > > > A couple of questions: > > > > * There was a limitation that AHCI devices must use slots 3-6. [1] > >Is it still there? If yes, any plans to get rid of it? > > It's not a limitation for guests that can use MSI for AHCI. Older > versions of Windows use legacy interrupts, but I noticed that 2k16 (and > maybe recent builds of 10) are using now using MSI. > > The real fix is to have more flexible ACPI DSDT generation from inside > of UEFI, but that's a non-trivial project. > > > * It *seems* that now it's OK to cycle a VM this way: > > > >host# bhyve ... > >guest# reboot # guest goes away > >host# bhyve ... # run it again > > > >Previously (with bhyveload) it didn't work (for me at least) without > >doing "bhyvectl --destroy". Is it safe now not to call "bhyvectl > >--destroy" before the second run now? > > Yes, except for the case when you modify the amount of memory given to > the guest - then you will need to delete prior to the run with the new > config. Got it, thanks! Roman Bogorodskiy signature.asc Description: PGP signature
Re: Vmm.ko to kernel
kakawkaization kaka wrote: > Hello, I can't find instruction how to include module vmm to kernel. > Instruction like : > device vmm > Is not works. > > Please help me to solve this problem, thanks. Hi, I suggest to follow this instruction: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html If you still want to build a custom kernel, please refer to this guide: https://www.freebsd.org/doc/handbook/kernelconfig.html If this does not help still, please be more specific, "is not works" is not really helpful, please post some more details on what you are doing and exact errors you get. Roman Bogorodskiy signature.asc Description: PGP signature
Re: Virt-manager issue (Solved)
Luciano Mannucci wrote: > On Mon, 16 May 2016 19:16:46 +0300 > Roman Bogorodskiy <no...@freebsd.org> wrote: > > > Luciano Mannucci wrote: > > > > > > > > Hello all! I hpoe this is the right list for that kind of things... > > > > > > After last upgrade to libvirt 1.3.4, I get an error when I try to > > > open a virtual host detail from virt-manager: > > > > > > Error launching details: Requiring namespace 'Gtk' version '2.0', but > > > '3.0' > > > is already loaded > > > > > > I'm I missing something obvious? > > > > Hi Luciano, > > > > Thanks for the report. A couple of questions: > > > > 1. By "open virtual host" you mean right click on VM -> context menu -> > >Open, right? > Yes. > > > 2. Could you run virt-manager from console with the "--debug" flag and > > copy/paste complete error message (with traceback if it prints one)? > Well, afer looking at help->about I noticed it was sayin 1.2 instead of > 1.3. I thig the pkg upgrade didn't remove all old bits of the previous > version, so I removed virt-manager via "pkg delete" and reinstalled. > Now everything works as expected... :-) > > BTW, it was on a 10.1-RELEASE-p26 amd64. Hm, it's strange that upgrade didn't work as expected, but if the logs are gone there's nothing that could be done to debug that. Anyway, good that it's working finally. > Cheers && Many Thanks, > > luciano. > -- > /"\ /Via A. Salaino, 7 - 20144 Milano (Italy) > \ / ASCII RIBBON CAMPAIGN / PHONE : +39 2 485781 FAX: +39 2 48578250 > X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org > / \ AND POSTINGS/ WWW: http://www.lesassaie.IT/ Roman Bogorodskiy ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Understanding Bhyve shutdown
Matt Churchyard wrote: > As I understand it > > 1) shutdown from guest > 2) 'kill ' -> pressing the power button once. > 3) --force-poweroff -> holding power button in > 4) --force-reset -> pressing the reset button > 5) bhyvectl --destroy -> same as 3? (although vmm is destroyed as well) > > 1 or 2 are obviously preferred. I've found however that some guests don't > respond that well to the apci event. CentOS 6 seems to need apci installing > and enabling?!, and my Win2012R2 machine will only shutdown if I send the > signal twice. > > The rest are all hard shutdowns that should not be seen as a way to stop the > guest, just a last resort if it can't be stopped correctly. I don't know the > practical difference between poweroff vs just destroy. I CCed Peter and Neel, probably they'll shed some light on 'bhyvectl --force-poweroff && bhyvectl --destroy' vs just 'bhyvectl --destroy'. > I see no reason why the documentation suggests reboot rather than shutdown. > Bhyve exits either way and the only difference is the exit code. When using > one of the bhyve management tools that support reboots (such as > vmrun.sh/vm-bhyve/iohyve) however, a "restart" exit code (0) will cause the > guest to restart automatically; If a guest is locked up for example, a > --force-reset will cause it to immediately reboot. > > From a host, the only safe choice is 'kill ' (possibly multiple times) > and wait for bhyve process to (hopefully) exit. If that doesn't work either > the acpi issue needs fixing or ideally the guest needs to be stopped using > its built-in shutdown function. > > The devs will have to confirm why they're implemented the way they are. My > instinct is that bhyvectl works on the vmm device, and reset/poweroff are > just functions that are possible via that interface, and so have been > exposed. The ACPI shutdown event likely needs to be initiated by the bhyve > process itself, hence using a signal to it. /end-speculation > > I think most users will want to use a bhyve tool so the raw specifics of the > bhyve/bhyvectl commands are glossed over, although that doesn't mean the > handbook documentation of the base commands shouldn't be as complete/correct > as possible of course. FWIW, I've created a patch: https://reviews.freebsd.org/D5982 which at least documents exit codes and the SIGTERM thing. Roman Bogorodskiy signature.asc Description: PGP signature
Understanding Bhyve shutdown
Hi, I was trying to get better understanding of how to properly shutdown VMs in bhyve, but unfortunately the documentation does not provide much details on that. Specifically, handbook [I] suggests to reboot a machine and then run bhyvectl --destroy on it. The bhyvectl(8) manpage mentions the '--force-reset' and '--force-poweroff' switches, but does not give details on those. I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html I tried all the options I know and wrote down the results. I also have some questions, hopefully you'll be able to answer some of them. 1. bhyvectl --vm=$name --destroy * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm/ entry goes away In my experience, it's a dangerous way to shutdown a VM because sometimes it appears it damages the image and VM fails to boot with something like this: --- Starting devd. mode = 0100600, inum = 170269, fs = / panic: ffs_valloc: dup alloc cpuid = 0 KDB: stack backtrace: #0 0x80984e30 at kdb_backtrace+0x60 #1 0x809489e6 at vpanic+0x126 #2 0x809488b3 at panic+0x43 #3 0x80b74a6e at ffs_valloc+0x84e #4 0x80bb60ad at ufs_makeinode+0x7d #5 0x80bb24fd at ufs_create+0x2d #6 0x80e71841 at VOP_CREATE_APV+0xa1 #7 0x809cd9e6 at uipc_bindat+0x346 #8 0x809c5488 at kern_bindat+0x108 #9 0x809c52a7 at sys_bind+0x77 #10 0x80d4b3f7 at amd64_syscall+0x357 #11 0x80d30adb at Xfast_syscall+0xfb Uptime: 3s Dump failed. Partition too small. --- 2. kill -SIGTERM $bhyve_pid If guest supports ACPI shutdown: * guest shuts down cleanly * the corresponding bhyve(8) process terminates * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup If guest does not support ACPI shutdown (such as doing sysctl hw.acpi.power_button_state=NONE): * Nothing happens Q1: Is there a way to know if a guest reacted to power button but waiting for the bhyve process to terminate? Q2: Why it's not done via bhyvectl (it seems that it's easier for users + don't have to overload a useful SIGTERM signal) 3. bhyvectl --vm=$name --force-poweroff * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup Q: what's the practical difference with just doing --destroy right away? 4. bhyvectl --vm=$name --force-reset Looks very similar to item #3 with just different exit code (reboot appears to be using 0, while shutdown and halt use 1 and 2). Q: what's the practical use of it? Would greatly appreciate if somebody could provide more details on that. I guess we'll need to update Handbook with this information as well because it needs to mention SIGTERM for ACPI shutdown at least. Roman Bogorodskiy ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Illumos support in bhyve
Nicolas Gilles wrote: > SmartOS is built to work that way, booting from an "static" ISO or USB > or PXE, and just using the hard-drives in the system as store for the > zones + vms, the os "drives" is loaded into memory. > > In a VM, it's not going to change you much to boot form a virtual DVD > drive vs virtual HD install, what troubles you booting from the ISO? Sorry, I should have made myself more clear: my question about self-contained image that does not need an ISO to boot was not SmartOS specific, but general, mainly regarding FreeBSD images or, say, Linux images. > On Fri, Nov 6, 2015 at 4:33 PM, Roman Bogorodskiy <no...@freebsd.org> wrote: > > gre...@freebsd.org wrote: > > > >> Hi Roman, > >> > >> > I tried this instruction and I was able to get into smartos > >> > installation. I choose all the default options there, it reported > >> > successful installation. Then I dropped 'ahci-cd' part, but I cannot > >> > get > >> > smartos boot. > >> > >> For smartos, the CD should always be left there since that's the boot > >> media. The first boot just detects that the hard drive is empty which > >> triggers the install. > > > > Ah, I see. It works now, thanks! > > > >> > And, by the way, is it possible to run FreeBSD using UEFI? > >> > > >> > I encountered problems with that as well: > >> > > >> > https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-November/003904.html > >> > > >> > Any hints how to debug this are appreciated. > >> > >> The VM images don't include the UEFI loader. You'll have to boot from > >> the ISO for that, and force serial console output at the loader prompt. > > > > Is it possible to prepare an image that would not require booting from > > the ISO? > > > > Roman Bogorodskiy > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" Roman Bogorodskiy signature.asc Description: PGP signature
Re: Illumos support in bhyve
gre...@freebsd.org wrote: > Hi Roman, > > > I tried this instruction and I was able to get into smartos > > installation. I choose all the default options there, it reported > > successful installation. Then I dropped 'ahci-cd' part, but I cannot > > get > > smartos boot. > > For smartos, the CD should always be left there since that's the boot > media. The first boot just detects that the hard drive is empty which > triggers the install. Ah, I see. It works now, thanks! > > And, by the way, is it possible to run FreeBSD using UEFI? > > > > I encountered problems with that as well: > > > > https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-November/003904.html > > > > Any hints how to debug this are appreciated. > > The VM images don't include the UEFI loader. You'll have to boot from > the ISO for that, and force serial console output at the loader prompt. Is it possible to prepare an image that would not require booting from the ISO? Roman Bogorodskiy signature.asc Description: PGP signature
Re: Illumos support in bhyve
Peter Grehan wrote: > To follow up on the previous post, with the UEFI support introduced in > r288524, bhyve can now boot Illumos since there is a working BIOS. > > This has been tested with SmartOS and also OpenIndiana and OmniOS, > though I've not been able to work out how to enable serial console > support for the latter two post-install. > > The best bet is the most recent SmartOS: > > https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos-latest.iso > > Grab the UEFI build with CSM/BIOS enabled: > http://people.freebsd.org/~grehan/bhyve_uefi/BHYVE_UEFI_CSM_20151002.fd > > An example command line is: > > bhyve \ > -c 2 \ > -s 3,ahci-cd,/path/to/smartos-20151001T070028Z.iso \ > -s 4,ahci-hd,/path/to/smartos.img \ > -s 5,virtio-net,tap0 \ > -s 31,lpc \ > -l com1,/dev/nmdm0A \ > -l com2,/dev/nmdm1A \ > -l bootrom,/path/to/BHYVE_UEFI_CSM_20151002.fd \ > -m 2G -H smartos > > UEFI/CSM debug output is on com1, grub1 is on com2, install is on com1. > On a reboot after install, login sessions are enabled for both com1 and > com2. > > Many thanks to Andriy Gapon for helping out with debug, and Ben Perrault > for giving it a good shakeout. Hi Peter, I tried this instruction and I was able to get into smartos installation. I choose all the default options there, it reported successful installation. Then I dropped 'ahci-cd' part, but I cannot get smartos boot. I can see on nmdm0B the following: Legacy INT19 Boot... And that's all. And there's nothing on nmdm1B. My command is: bhyve -c 2 -s 4,ahci-hd,/home/novel/smartos.img -s 5,virtio-net,tap0 -s 31,lpc -l com1,/dev/nmdm0A -l com2,/dev/nmdm1A -l bootrom,/home/novel/uefi/BHYVE_UEFI_CSM_20151002.fd -m 2G -H smartos ^T reports: load: 1.17 cmd: bhyve 4821 [running] 2461.38r 347.75u 2109.83s 100% 7580k And, by the way, is it possible to run FreeBSD using UEFI? I encountered problems with that as well: https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-November/003904.html Any hints how to debug this are appreciated. Thanks, Roman Bogorodskiy signature.asc Description: PGP signature
Re: bhyve -l bootrom problem
Trent Thompson wrote: > According to the bhyve/Windows wiki: https://wiki.freebsd.org/bhyve/Windows > The lpc slot needs to be in slot 31, and the block devices bust be in 3, 4, > 5, and 6. > I have had luck* booting into the Debian install environment using this > method, as long as you pass "console=ttyS0" to grub at boot time. I believe > you must do the same as well on FreeBSD, with adding the ability to use a > console in /boot/loader.conf > > *I have installed Debian VMs utilizing UEFI, however I have not had success > after destroying the VM. > They fail to boot and drop you to that fancy EFI shell. Hm... changing pci slot numbers doesn't seem to change anything: sudo /usr/sbin/bhyve -l bootrom,/home/novel/uefi/BHYVE_UEFI_20151002.fd -c 2 -m 512 -A -I -u -H -P -s 0:0,hostbridge -s 3:0,ahci-hd,/home/novel/bhyve_freebsd.img.bak -s 31,lpc -l com1,/dev/nmdm0A testvm and I get: Boot Failed. EFI Hard Drive However, when I do: sudo bhyveload -m 512 -d /home/novel/bhyve_freebsd.img.bak testvm and then: sudo /usr/sbin/bhyve -c 2 -m 512 -A -I -u -H -P -s 0:0,hostbridge -s 3:0,ahci-hd,/home/novel/bhyve_freebsd.img.bak -s 31,lpc -l com1,/dev/nmdm0A testvm I can see a console: (It fails to mount root though because my image is configured to virtio, but I guess it's not relevant to this situation). > > On Mon, Nov 2, 2015 at 10:13 AM, Roman Bogorodskiy <no...@freebsd.org> > wrote: > > > Hi, > > > > I'm facing a strange problem with -l bootrom. > > > > I try to boot a VM this way: > > > > sudo /usr/sbin/bhyve -l bootrom,/home/novel/uefi/BHYVE_UEFI_20151002.fd > > -c 2 -m 214 -A -I -u -H -P -s 0:0,hostbridge -s > > 2:0,virtio-net,tap0,mac=52:54:00:56:9a:77 -s > > 3:0,ahci-hd,/home/novel/FreeBSD-11.0-CURRENT-amd64.raw -s 1,lpc -l > > com1,/dev/nmdm0A testvm > > > > BHYVE_UEFI_20151002.fd is downloaded from [1]. > > FreeBSD-11.0-CURRENT-amd64.raw is downloaded from [2]. > > > > VMs fail to boot with 'Boot Failed. EFI Hard Drive' and brings into UEFI > > shell. > > > > Changing ahci-hd to virtio-blk changes error message to: > > > > 'Boot Failed. EFI Misc Device'. > > > > Dropping 'hostbridge' and 'virtio-net' doesn't seem to change anything. > > > > Also, before trying FreeBSD-11.0-CURRENT-amd64.raw I tried my old images > > I used with bhyveload and they fail in the same way, though they're > > still booting using bhyveload. > > > > I'm running -CURRENT updated last weekend on Intel i5-2310. > > > > What could be wrong with this? > > > > 1: http://people.freebsd.org/~grehan/bhyve_uefi/BHYVE_UEFI_20151002.fd > > 2: > > > > http://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/11.0-CURRENT/amd64/Latest/FreeBSD-11.0-CURRENT-amd64.raw.xz > > > > Roman Bogorodskiy > > Roman Bogorodskiy signature.asc Description: PGP signature
Re: virtualization ports
Jason Helfman wrote: > On Thu, Sep 24, 2015 at 12:32 AM, Roman Bogorodskiy <bogorods...@gmail.com> > wrote: > > > Jason Helfman wrote: > > > > > I have a number of ports that I maintain, that I really never use but at > > > one time I did. Initially, I put them into the portstree to help out the > > > community, with the idea that I would be using them one day, however I > > work > > > at a VMware shop, and the likelihood of me using this software is looking > > > far less likely. I was wondering if it would be okay to assign these > > ports > > > to virtualization group? > > > > > > deskutils/virt-manager > > > devel/libvirt > > > devel/libvirt-glib > > > devel/libvirt-java > > > devel/py-libvirt > > > devel/spice-protocol > > > net-mgmt/virt-viewer > > > > FWIW, I could take care of libvirt, py-libvirt, libvirt-glib and > > virt-manager. > > > > Roman Bogorodskiy > > > > > Sounds good, thanks! Additionally, if you are going to take virt-manager, > you really should take spice-protocol and virt-viewer as they are > dependencies and require updates for virt-manager. > Thoughts? Yes, it looks reasonable. Roman Bogorodskiy pgp3l_WEY6j5a.pgp Description: PGP signature
Re: xen_kernel, console and X11
Roman Bogorodskiy wrote: With this setup I get my system booted and at some point I can see a login screen. When I type 'startx' the system freezes. Have to hard reboot it to get working again. Update: I was bothering Roger off-list and he helped to figure out that the problem with X11 is closely related to nvidia-driver (the one from x11/nvidia-driver port). There's no solution how to make it work so far. Roman Bogorodskiy pgpHtEZXu_jj8.pgp Description: PGP signature
Re: xen_kernel, console and X11
Roger Pau Monné wrote: Hello, El 04/05/15 a les 16.16, Roman Bogorodskiy ha escrit: Roger Pau Monné wrote: Hello, El 02/05/15 a les 17.43, Roman Bogorodskiy ha escrit: Hi, I'm trying to get Xen running and following these instructions: https://wiki.freebsd.org/Xen and http://wiki.xen.org/wiki/FreeBSD_Dom0 I'm running two days old -CURRENT and ports. I've installed the emulators/xen port and followed instructions in pkg-message. I'm having some problems with console. I'm wondering if it's possible to have X running on the same box running xen kernel? It should be, although I had issues while using the vesa driver with a FreeBSD Xen Dom0, mainly because Dom0 doesn't have access to the BDA and EBDA, I'm working on fixing this in Xen upstream. My setup is as follows: - Intel i5-4690 that supports IOMMU: $ sudo acpidump -t|grep DMAR DMAR: Length=128, Revision=1, Checksum=90, $ - vm.max_wired=-1 in /etc/sysctl.conf - xc0 /usr/libexec/getty Pc xterm on secure in /etc/ttys In loader.conf I have: xen_kernel=/boot/xen xen_cmdline=dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all console=com1 So you are trying to use the serial console but you are not getting any output? If that's not the case, please drop the com1 parameter and set console=vga. Sorry for confusion, actually 'console=com1' works for me. Initially I didn't realize that it doesn't display kernel messages and after some failed attempts to run startx and hard reboots it took much longer to see a login prompt because I have background fsck disabled. So, 'console=com1' works unless I do 'startx'. I'm not sure I would call it working if you don't get any kernel messages while booting. What happens if you set console=vga? Do you have anything else in your /boot/loader.conf apart from this two lines? With this setup I get my system booted and at some point I can see a login screen. When I type 'startx' the system freezes. Have to hard reboot it to get working again. On which device do you get a login prompt? Is it xc0, ttyv0 or ttyu0? I get a login prompt on ttyv0. BTW, I use the nvidia driver. Another user also reported a similar problem with Xen and X, and it was solved by the following patch: https://people.freebsd.org/~royger/0001-xen-introduce-a-newbus-function-to-allocate-unused-m.patch Could you apply it and rebuild your kernel to see if that also solves your issues? Applied this patch on r282416, things didn't change: I do 'startx' in ttyv0, X prints that it loads extensions, then a black screen with a cursor appears and things hang. Roman Bogorodskiy pgpdijBRhhlSg.pgp Description: PGP signature
Re: libvirt improvements for bhyve
Craig Rodrigues wrote: On Sat, Oct 11, 2014 at 11:05 AM, Roman Bogorodskiy no...@freebsd.org wrote: Frankly speaking, it's a hard topic for me and currently I don't have a complete idea how to design that. The thing I'm worried about is that as far as I understand that exit codes are sort of temporary solution and will be dropped in the future. Also, it seems the behaviour is changing already, e.g. IIRC on -CURRENT the bhyve proccess doesn't exit on reboots (unlike in 10-R). My goal is to support 10-R and I'm not sure how to probe if exit codes are available or not (and how to probe if e.g. if new exit code will be added in some future version). Any reason why you are stuck on 10-R? Unlike other parts of FreeBSD, bhyve is not mature, and is evolving over time. Serious users of bhyve are following CURRENT or 10-STABLE. 10-STABLE has the same error status codes as CURRENT, and 10.1-R is coming out in a few weeks. libvirt for bhyve is even less mature than bhyve itself, so tracking bhyve beyond the release should be OK, but it's up to you. Well, it's not an ultimate goal to support 10-R, but while it's not very painful, I think it's a nice thing to do. Also, it doesn't mean that I'm not adding features available only in -STABLE, I just want user to have a possibility to have a meaningful error messages if certain feature is not support. Also, I forgot to mention one more important issue with the exit code approach. Libvirt doesn't serve as a parent process for the bhyve processes. The reason for that is to allow libvirt restart without having to interrupt the VMs (that's useful for upgrades or e.g. libvirtd crash). I'm not sure how I could track exit codes in such case. IMHO, the good way to solve this problem is to introduce some sort of control socket on the bhyve side that could be used to query VM status and push events about its change. I've been wanting to implement a PoC for that for quite some time, but, unfortunately, haven't got to it yet. In its current form, my experience has been that bhyve and libvirt are unusable for serious work, based on the issues which I've described before for starting/stopping VM's. bhyve + custom scripting is the only way to go. I tried to use libvirt + bhyve for jenkins.freebsd.org, but found it unusable, even though Jenkins has libvirt support for starting VM's on demand. I backed away from libvirt, and ended up writing my own scripts. I am trying to convince some people to go with libvirt + bhyve for a simulation/testing system, but am having a hard time because of all the rough edges, and Linux + KVM + libvirt looks a lot more mature and easier to use. It's too bad, because I think if you can incorporate a few minor enhancements to libvirt for bhyve, it can definitely be a replacement for Linux + KVM + libvirt Neel, Peter, can you comment on the status exit codes from bhyve? Are those temporary, or will you keep them? There is no other way for external scripts to detect reset/poweroff. It looks like the confusion over this issue is blocking Roman from moving forward with libvirt. Thanks. -- Craig Roman Bogorodskiy pgpF0qdEbZyie.pgp Description: PGP signature
Re: libvirt improvements for bhyve
Craig Rodrigues wrote: Roman, I am using bhyve a lot these days, but I find that I need to write scripts on top of bhyveload / bhyve in order to work with VM's. I would rather use libvirt, because there are many 3rd party programs to work with VM's that are written for libvirt (especially for KVM). How hard would it be to implement the following: (1) The last time I tried libvirt 1.2.7, it did not seem to properly keep track if a VM was powered on or shutdown. If I started a VM with libvirt, then did shutdown -r now, I couldn't seem to restart the VM, unless I destroyed the VM in libvirt, and restarted. This is very inconvenient. If you see the latest version of vmrun.sh ( https://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=markup ) you will see that I added some comments to tell when the VM has been reset, powered off, etc. by looking at the exit status of bhyve. Can we add this logic to libvirt, so that it can properly tell if a VM is up or not? Frankly speaking, it's a hard topic for me and currently I don't have a complete idea how to design that. The thing I'm worried about is that as far as I understand that exit codes are sort of temporary solution and will be dropped in the future. Also, it seems the behaviour is changing already, e.g. IIRC on -CURRENT the bhyve proccess doesn't exit on reboots (unlike in 10-R). My goal is to support 10-R and I'm not sure how to probe if exit codes are available or not (and how to probe if e.g. if new exit code will be added in some future version). On the other hand, I think I need to check cases when bhyve process vanishes away and probably set VM status to something like 'shutdown / error' or something like that. Also, I think I need to support 'bhyvectl --force-reset' if that's available. I guess that should be safe for all versions of bhyve and I'll add it to my todo list. (2) bhyveload -e allows specifying environment variables to loader. Can we have a way to specify loader environment variables, in the libvirt XML config? (3) grub-bhyve is needed to boot VM's for Linux. Can we specify an option to choose between bhyveload and grub-bhyve? For 2-3 I think bootloader should be honored from the XML file: http://libvirt.org/formatdomain.html#elementsOSBootloader I'll add it to my todo list as well. (4) bhyve has many options for specifying which PCI slots to use, and also PCI passthru ( https://wiki.freebsd.org/bhyve/pci_passthru ) Can this be specified in libvirt? Currently you can specify PCI slot and function by using 'address' tag in the device element (e.g. disk or interface), for example: interface type='bridge' mac address='52:54:00:f6:41:4e'/ source bridge='virbr0'/ model type='virtio'/ address type='pci' domain='0x' bus='0x00' slot='0x02' function='0x0'/ /interface PCI passthrough is not supported as I've never looked at that, will need to take a look. For me, without (1), libvirt is mostly unusable for bhyve. (2) - (4) would be nice to have. However, if we could fix libvirt with these things, then a lot of things I do now in scripts, I could do in libvirt. I would much rather use libvirt, because then I could take advantage of a lot of libvirt software targeted towards KVM. Thanks. -- Craig Roman Bogorodskiy pgpSxXSIXNmRB.pgp Description: PGP signature
Re: Perl, libvirt and bhyve
Denis Menshikov wrote: Good day! As far as I know, libvirt does not provide API for Perl. Can I use this Perl module for managing bhyve virtual machines https://metacpan.org/pod/Sys::Virt? I think Yes, but I want to clarify. Thank you! Hi, You are right, that's the official binding for libvirt as mentioned on http://libvirt.org/bindings.html. Most probably it works fine with bhyve as well. Roman Bogorodskiy pgppfUMhqe78E.pgp Description: PGP signature
Re: FreeBSD with qemu/kvm -kernel option
Assaf Gordon wrote: Hello, I have FreeBSD 8.4, 9.3 and 10 guests VM running under KVM (on amd64 host). They work great with virtio disk and network. Now I'm trying to take it to the next level: Instead of using the bootloader inside the VM, I want to specify the kernel directly using qemu's -kernel option. Generally, the usage is supposed to be this: kvm -drive file=freebsd.qcow2,media=disk,if=virtio,index=0 \ -kernel ./kernel Where ./kernel is the FreeBSD kernel ELF file (stored as a file on the host machine). Using the generic /boot/kernel/kernel kernel does not work (VM doesn't boot at all). I assume that more options are needed (e.g. -initrd and -append), and perhaps also a kernel with different options. Does anyone have an idea of how to achieve this ? Is rebuilding a kernel image from source with special options needed ? I guess, qemu's -kernel requires kernel to be multiboot compatible and FreeBSD kernel is not. Roman Bogorodskiy pgp_0eosSM0rb.pgp Description: PGP signature
Re: Does anyone use the FreeBSD VM qcow2 images
Glen Barber wrote: Hi, I'm talking about the images available here: http://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/ Since mkimg(1) in head/ and stable/10 supports VMDK and VHD formats, I would prefer to remove the need to use qemu-img to generate the VM images. This would mean that the qcow2 format would no longer be produced. The qcow2 image format is very popular among Linux KVM/qemu-kvm users, maybe the most popular even. So I think it'd be useful to keep producing these images. Roman Bogorodskiy pgpP1gOtFTDxg.pgp Description: PGP signature
Re: libvirt and rebooting of a bhyve VM
Allan Jude wrote: On 2014-08-19 11:50, John Nielsen wrote: On Aug 19, 2014, at 9:40 AM, Roman Bogorodskiy no...@freebsd.org wrote: Craig Rodrigues wrote: Roman, I am using libvirt and bhyve according to this XML: http://libvirt.org/drvbhyve.html and it works great. I gave a presentation at BAFUG on this: http://www.slideshare.net/CraigRodrigues1/libvirt-bhyve I have one question. If I reboot the bhyve VM started with libvirt with shutdown -r now, the VM shuts down, but it does not restart. How can I get the machine to reboot with shutdown -r now when started with libvirt? Hi Craig, Unfortunately, I'm not sure how to get the reboot working. Moreover, I get the same behaviour when starting bhyve manually -- when I do a reboot, bhyve(8) exits as soon as the system is ready to restart. So looks like that's a default bhyve behaviour or I'm missing something? Wasn't changing this the intention of r267216 (MFCed as r270071)? Roman, was your 10-STABLE built after that revision? JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org Yes, this revision adds the ability to 'reboot'. This does not exit bhyve at all, so there is no exit level You can 'reboot' a bhyve externally using: bhyvectl --force-reset --vm=xxx or force a poweroff with: bhyvectl --force-reset --vm=xxx SIGTERM simulates an ACPI shutdown the bhyvectl --destroy is to clean up, the instance remains even after a clean shutdown. In the past, it had to be destroyed and recreated to start the VM again, although now with reboot support, that may not be required. I have updated to 10-STABLE that includes commit r270071. I can see bhyveload has --force-reset flag. However, when I boot a VM and do 'reboot' there, bhyve(8) exits with exit code 0. The same happens when I do bhyvectl --force-reset. Any ideas how to debug why bhyve(8) exits in this case? Roman Bogorodskiy pgpuzYzrwjScz.pgp Description: PGP signature
Re: libvirt and rebooting of a bhyve VM
Kurt Lidl wrote: On Tue, Aug 19, 2014, Craig Rodriques wrote: On Tue, Aug 19, 2014 at 12:18 PM, Allan Jude allanjude at freebsd.org wrote: The name of the vm is in the title of the bhyve process, but yes, it Hmm, OK, that would require me to grep the processes and send SIGTERM to the right pid. That's not ideal, especially if I start lots of VM's, but it is workable. I would think that hacking in support to the bhyve program to use pidfile_write() in libutil, with something like: /var/run/bhyve.vmname.pid as the default pathname for the pidfile would make this a lot easier to manage from a script. IMHO, as for the pidfile, it's the thing that could be easily done in the script itself. What would be really good from my point of view is to have a control socket (Unix socket) and support for some sort of a protocol encoded in a machine readable format. So it could be used to request VM information, perform actions on it, doing a dynamic configuration (e.g. plugging devices etc). Also, it'd be easier to monitor a socket presence than a pid. Of course, all this could be done by extending bhyvectl as well. This way seems less convenient from me because from the scripting point of view command line application features are much harder to probe than doing the same via a defined protocol. Roman Bogorodskiy pgp1BzDQvTI3v.pgp Description: PGP signature
Re: libvirt and rebooting of a bhyve VM
John Nielsen wrote: On Aug 19, 2014, at 9:40 AM, Roman Bogorodskiy no...@freebsd.org wrote: Craig Rodrigues wrote: Roman, I am using libvirt and bhyve according to this XML: http://libvirt.org/drvbhyve.html and it works great. I gave a presentation at BAFUG on this: http://www.slideshare.net/CraigRodrigues1/libvirt-bhyve I have one question. If I reboot the bhyve VM started with libvirt with shutdown -r now, the VM shuts down, but it does not restart. How can I get the machine to reboot with shutdown -r now when started with libvirt? Hi Craig, Unfortunately, I'm not sure how to get the reboot working. Moreover, I get the same behaviour when starting bhyve manually -- when I do a reboot, bhyve(8) exits as soon as the system is ready to restart. So looks like that's a default bhyve behaviour or I'm missing something? Wasn't changing this the intention of r267216 (MFCed as r270071)? Roman, was your 10-STABLE built after that revision? Thanks for pointing at this commit. My -STABLE is older than that so I'll need to update. Anyway, if it works without restarting bhyve(8) then it should work in libvirt 'for free'. :) Roman Bogorodskiy pgp8ArppQPyZL.pgp Description: PGP signature
Re: py-libvirt problems with bhyve
Craig Rodrigues wrote:
On Wed, Jun 25, 2014 at 11:01 AM, Roman Bogorodskiy no...@freebsd.org
wrote:
Craig Rodrigues wrote:
conn = libvirt.openReadOnly(None)
^^^
I think here the URI should be 'bhyve:///system'.
Or you could tweak libvirt.conf and assign the bhyve url value to the
'uri_default' parameter.
http://libvirt.org/uri.html#URI_default
Hi
OK, I changed my steps a little bit.
I did the following:
(1) Set up libvirt, taking Roman's latest port, and configuring bhyve
support:
http://people.freebsd.org/~rodrigc/libvirt/libvirt-bhyve.html
(2) Installed the py-libvirt port.
I wrote this code:
import libvirt
import sys
# Note we need to specify the URL here
conn = libvirt.openReadOnly(bhyve:///system)
if conn == None:
print 'Failed to open connection to the hypervisor'
sys.exit(1)
try:
print(All domains: ,conn.listAllDomains())
print(Defined domains: , conn.listDefinedDomains())
#print dir(conn)
dom0 = conn.lookupByName(bhyve)
except:
print 'Failed to find the main domain'
sys.exit(1)
print(Domain 0: id %d running % (dom0.ID()))
print(dom0.info())
I got this:
('All domains: ', [libvirt.virDomain object at 0x801f50550])
('Defined domains: ', ['bhyve'])
Domain 0: id -1 running
('Domain info: ', [5, 2147483L, 0L, 1, 0L])
So that's an improvement.
I started reading this tutorial on libvirt and KVM scripting:
https://www.ibm.com/developerworks/linux/library/os-python-kvm-scripting1/
In that tutorial, they give an example where it is possible
to configure the MAC and IP address of the VM in an XML config file.
Is it possible to do that with libvirt and bhyve?
Generally, the dnsmasq related code is the same, so it should work.
The script in this example seem to operate only on the network
definition file, it misses the actual applying of the changes (or maybe
I missed that in the article).
I didn't try the script itself, but tried the suggested changes for the
network definition, i.e. adding stuff like:
host mac='de:af:de:af:00:03' name='vm-2' ip='192.168.100.3' /
do the dhcp section.
I did 'net-destroy default', then 'net-edit default', made this change,
stopped the dnsmasq and did 'net-start default' (I'm not sure if it's a
bug or a feature that dnsmasq didn't stop, need to look closer into that)
and rebooted a domain. It seemed to work, it got the configured IP
address and the hostname.
Roman Bogorodskiy
pgpgeTeWkBubD.pgp
Description: PGP signature
Re: libvirt, bhyve, and Jenkins configuration problems
Craig Rodrigues wrote: On Wed, Jun 25, 2014 at 11:08 AM, Roman Bogorodskiy no...@freebsd.org wrote: Does that mean that I need to configure my VM, so that it comes up with a known IP, so that it can be entered on this configuration page? Correct. Though, normally you don't need to perform any special configuration for the IP address as long as the networking configuration is correct. The VM just needs to be configured to obtain an IP address via DHCP. Later on it should be persistent in dnsmasq. An IP address could be obtained either from the dnsmasq leases file or from arp cache for example. But how does the Jenkins libvirt plugin know the IP address for the VM, unless you enter it in this field in the Jenkins UI? Sorry for confusion, yes, it's required to enter the IP address into Jenkins configuration dialog, I was just talking about libvirt / guest configuration side of things. Roman Bogorodskiy ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: libvirt, bhyve, and Jenkins configuration problems
Craig Rodrigues wrote: On Sat, Jun 21, 2014 at 6:52 AM, Roman Bogorodskiy no...@freebsd.org wrote: When you configure the connection in the 'Add a new Cloud' dialog, does 'test connection' work for you? I don't see a host and credentials specified, I'm not sure if jenkins libvirt plugin works with the local connection. OK, following your example, I have gotten further along. I have updated my setup instructions here: http://people.freebsd.org/~rodrigc/libvirt/libvirt-bhyve-jenkins.html In Step 3, when I click on Test Connection, I see OK: bhyve+ssh://root@localhost:22/system?no_tty=1 version=1100 so that seems to work. For Step 5, when I enter the Host and Credentials information, does this information need to be the IP address and login information for the VM that has been spawned, and not the info for libvirt, as specified in Step 3? Correct. Does that mean that I need to configure my VM, so that it comes up with a known IP, so that it can be entered on this configuration page? Correct. Though, normally you don't need to perform any special configuration for the IP address as long as the networking configuration is correct. The VM just needs to be configured to obtain an IP address via DHCP. Later on it should be persistent in dnsmasq. An IP address could be obtained either from the dnsmasq leases file or from arp cache for example. Also, the VM, that is spawned, it must have openjdk installed? Does that mean that after libvirt starts the VM, Jenkins will try to SSH into the VM and start a Jenkins job by invoking java? Yes, it's true, it starts its agent which does the job. PS Maybe there are other ways to push Jobs to Jenkins nodes, but I didn't look close into that. Roman Bogorodskiy pgpIUPdF05ZE6.pgp Description: PGP signature
Re: libvirt, bhyve, and Jenkins configuration problems
Craig Rodrigues wrote: Hi, I am trying to configure Jenkins to spawn bhyve VM's on demand using libvirt. I did the following: (1) Looked at Roman's blog article: http://empt1e.blogspot.com/2014/03/using-jenkins-libvirt-slave-plugin-with.html (2) Followed the libvirt configuration steps here: http://people.freebsd.org/~rodrigc/libvirt/libvirt-bhyve.html Here are the screenshots of my Jenkins setp for libvirt: http://people.freebsd.org/~rodrigc/libvirt/libvirt-bhyve-jenkins.html However, after going through all that, when I tried to configure a job to run on the libvirt node, it did not start. Any ideas what I am doing wrong? Hi Craig, Some questions: When you configure the connection in the 'Add a new Cloud' dialog, does 'test connection' work for you? I don't see a host and credentials specified, I'm not sure if jenkins libvirt plugin works with the local connection. In my setup, I have specified the host and the credentials like this: http://people.freebsd.org/~novel/misc/jenkins-libvirt-01.png Then, the new node screen doesn't look complete as well. There, one has to choose one of the pre-defined domains, specify its IP address and credentials so Jenkins could ssh to it and run its agent. Also, one has to install JDK as the agent is Java-based. As I've described in my post, I did it using: pkg install java/openjdk7 command in the guest. Here's a config of my sample node: http://people.freebsd.org/~novel/misc/jenkins-libvirt-02.png After that in the node page one could start the agent and if ssh credentials and IP address of the VM are correct, the agent will successfully start: http://people.freebsd.org/~novel/misc/jenkins-libvirt-03.png Then in the nodes list it should be up and ready for the builds: http://people.freebsd.org/~novel/misc/jenkins-libvirt-04.png Note: depending on the policy specified for node, it might be turned off when idle so it'll be reported down in this list until when new jobs for it appear. Roman Bogorodskiy pgpn_ghmwNmb7.pgp Description: PGP signature
Re: Request for testers: virt-manager, libvirt, and bhyve
Bartek Rutkowski wrote: I am happy to look at updating the virt-manager - any chance you could point me to what from the below list was the dependency causing issues? dbus=0.83.2 libxml2=2.7.8 virtinst=0.600.4 vte=0.26.2 gtk-vnc=0.3.10 librsvg2=2.32.1 libvirt=0 Hi, I've looked at updating it to 1.0.1. Apparently, everything seemed to work fine, either something was updated since I've looked at it last time or I missed something. :-) Could you please review it: http://people.freebsd.org/~novel/misc/virt-manager-101.diff I did a basic testing and it seemed to work fine. I haven't yet backported my patch from the git master to add 'bhyve' into menu list, but one could still connect using: $ virt-manager -c bhyve+ssh://root@localhost/system Roman Bogorodskiy pgpxFp3SR5fCY.pgp Description: PGP signature
Re: Request for testers: virt-manager, libvirt, and bhyve
Craig Rodrigues wrote: Hi, I created a patch for the deskutils/virt-manager port to add support for bhyve. virt-manager depends on libvirt, so the work that Roman has done with libvirt is critical for it to work. I have instructions for setting it up here, plus a screenshot of the virt-manager UI: http://people.freebsd.org/~rodrigc/libvirt/virt-manager.html Is there anyone interested in testing this and providing feedback? Also, since virt-manager is in Python, is there anyone out there interested in improving this patch, and sending it upstream to the virt-manager developers? Hi Craig, Back in April bhyve support was added into virt-manager master [1]. So it should be there in the next release. I've looked at updating the desktuils/virt-manager port to 1.0.x, but at that time it was blocked by some outdated dependency. 1: https://git.fedorahosted.org/cgit/virt-manager.git/commit/?id=05df5a64843f2bd4e9a5197d97608d41b2e6dc43 Roman Bogorodskiy pgpTECtGIQNYI.pgp Description: PGP signature
Re: Request for Testers: libvirt and bhyve
Craig Rodrigues wrote: Hi, I've had some success with getting libvirt and bhyve to work. I wrote up a list of the steps that I followed here: http://people.freebsd.org/~rodrigc/libvirt-bhyve/libvirt-bhyve.html If anyone is interested, please try it out, and provide feedback to this list. It would be great to get more testers. Roman, when do you think you will be able to commit all your latest fixes into the devel/libvirt port Makefile? It would greatly increase the usability of the port. Thanks for doing all the work to add bhyve support to libvirt!! Hi! I hope to get a clean and ready patch for the port this weekend and will send it to jgh@. Thanks a lot for your testing and the feedback! Roman Bogorodskiy pgpKFUSQgnPK2.pgp Description: PGP signature
Re: libvirt and bhyve problems
Craig Rodrigues wrote: On Sun, Jun 15, 2014 at 8:20 AM, Roman Bogorodskiy no...@freebsd.org wrote: I have updated the port: http://people.freebsd.org/~novel/misc/libvirt_port2.tgz OK, I deinstall the old port, took your port, built it, and installed it. When I started libvirtd, I got this: 2014-06-15 18:47:07.236+: 34485605376: info : libvirt version: 1.2.5 2014-06-15 18:47:07.236+: 34485605376: info : dnsmasqCapsSetFromBuffer:685 : dnsmasq version is 2.71, --bind-dynamic is present, SO_BINDTODEVICE is in use 2014-06-15 18:47:07.237+: 34485605376: info : networkReloadFirewallRules:1750 : Reloading iptables rules 2014-06-15 18:47:07.237+: 34485605376: info : networkRefreshDaemons:1722 : Refreshing network daemons 2014-06-15 18:47:07.395+: 34485605376: error : virCommandWait:2426 : internal error: Child process (/usr/local/sbin/dnsmasq --conf-file=/usr/local/var/lib/libvirt/dnsmasq/default.conf) unexpected exit status 2: dnsmasq: failed to create listening socket for 127.0.0.1: Address already in use 2014-06-15 18:47:07.484+: 34485605376: error : virExec:417 : Cannot find 'pm-is-supported' in path: No such file or directory 2014-06-15 18:47:07.484+: 34485605376: warning : virQEMUCapsInit:948 : Failed to get host power management capabilities 2014-06-15 18:47:07.495+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/var/run/libvirt/qemu 2014-06-15 18:47:07.495+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/qemu 2014-06-15 18:47:07.583+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/bhyve 2014-06-15 18:47:07.584+: 34485605376: info : virDomainObjListLoadAllConfigs:18273 : Loading config file 'bhyve.xml' Trying to start bhyve did not work. After some investigation, I did the following: service local_unbound stop Then I started libvirtd again: 2014-06-15 18:47:07.236+: 34485605376: info : libvirt version: 1.2.5 2014-06-15 18:47:07.236+: 34485605376: info : dnsmasqCapsSetFromBuffer:685 : dnsmasq version is 2.71, --bind-dynamic is present, SO_BINDTODEVICE is in use 2014-06-15 18:47:07.237+: 34485605376: info : networkReloadFirewallRules:1750 : Reloading iptables rules 2014-06-15 18:47:07.237+: 34485605376: info : networkRefreshDaemons:1722 : Refreshing network daemons 2014-06-15 18:47:07.395+: 34485605376: error : virCommandWait:2426 : internal error: Child process (/usr/local/sbin/dnsmasq --conf-file=/usr/local/var/lib/libvirt/dnsmasq/default.conf) unexpected exit status 2: dnsmasq: failed to create listening socket for 127.0.0.1: Address already in use 2014-06-15 18:47:07.484+: 34485605376: error : virExec:417 : Cannot find 'pm-is-supported' in path: No such file or directory 2014-06-15 18:47:07.484+: 34485605376: warning : virQEMUCapsInit:948 : Failed to get host power management capabilities 2014-06-15 18:47:07.495+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/var/run/libvirt/qemu 2014-06-15 18:47:07.495+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/qemu 2014-06-15 18:47:07.583+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/bhyve 2014-06-15 18:47:07.584+: 34485605376: info : virDomainObjListLoadAllConfigs:18273 : Loading config file 'bhyve.xml' At this point, I could see the virbr0 device: virbr0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 ether 02:b8:18:5f:0f:00 inet 192.168.122.1 netmask 0xff00 broadcast 192.168.122.255 nd6 options=1PERFORMNUD id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 4 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 I was then able to do: virsh -c 'bhyve:///system' start bhyve and start a VM. This is good progress! Once thing I observed is that if I killed the libvirtd process, it did not clean up the dnsmasq process that it started: /usr/local/sbin/dnsmasq --conf-file=/usr/local/var/lib/libvirt/dnsmasq/default.conf and a subsequent restart of libvirtd would fail. I think it's intentional not to kill dnsmasq. There are persistent and transient objects in libvirt (such as domains and networks), and persistent objects live forever, while transient objects live until manually destroyed or host is restarted. So libvirt doesn't kill persistent VMs and, I guess, it doesn't stop dnsmasq to keep persistent networks functional. It should not affect restart. I'll fail to start the network, but as it's here already, we're fine. BTW, I'm still yet to write the code to catch up
Re: libvirt and bhyve problems
Craig Rodrigues wrote: On Thu, Jun 12, 2014 at 12:00 PM, Roman Bogorodskiy no...@freebsd.org wrote: Ah. Actually, libvirt manages bridges and taps on its own. So, the preferred flow is that libvirt starts its networks on startup automatically. By default it has a network with the 'virbr0' device which it creates on its own. Later on it creates vnet devices (which are tap devices) automatically as well on VM startup and plugs them to the bridge. The configuration you have probably confuses it, could you consider trying without that? OK, I removed all autobridge and cloned_interfaces entries from my rc.conf and rebooted. I am still using your version of the libvirt port. I tried the steps again and got this: 2014-06-13 19:09:10.482+: 34485605376: info : libvirt version: 1.2.5 2014-06-13 19:09:10.482+: 34485605376: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory 2014-06-13 19:09:10.482+: 34485605376: info : networkReloadFirewallRules:1750 : Reloading iptables rules 2014-06-13 19:09:10.482+: 34485605376: info : networkRefreshDaemons:1722 : Refreshing network daemons 2014-06-13 19:09:10.593+: 34485605376: error : virExec:417 : Cannot find 'pm-is-supported' in path: No such file or directory 2014-06-13 19:09:10.593+: 34485605376: warning : virQEMUCapsInit:948 : Failed to get host power management capabilities 2014-06-13 19:09:10.615+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/var/run/libvirt/qemu 2014-06-13 19:09:10.615+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/qemu 2014-06-13 19:09:10.716+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/bhyve 2014-06-13 19:09:10.716+: 34485605376: info : virDomainObjListLoadAllConfigs:18273 : Loading config file 'bhyve.xml' 2014-06-13 19:09:25.938+: 34485598208: error : virNetDevTapCreate:385 : Unable to create tap device: Invalid argument 2014-06-13 19:09:26.063+: 34485598208: error : virCommandWait:2426 : internal error: Child process (/usr/sbin/bhyvectl --destroy --vm=bhyve) unexpected exit status 255 If I do ifconfig, I have no bridge or tap devices. Also, do you have working 'net-list' command in virsh with my version of port? I get this: virsh # net-list Name State Autostart Persistent -- I have updated the port: http://people.freebsd.org/~novel/misc/libvirt_port2.tgz Changes are: * added RUN_DEPENDS on dnsmasq (required for networking) * added installation on network definition files Hopefully this will allow to bring up network on start. When the daemon starts, virsh should show the following: virsh # net-list Name State Autostart Persistent -- default active yes yes virsh # If that's not the case, then something is still going wrong. And yes, qemu support should be enabled (networks defs seem to be installed with qemu support). Even if it doesn't work, and 'net-list --all' is empty, we could try to do it manually (but still, if it's the case, then it's something wrong with the port that needs to be fixed): in virsh # net-define /usr/local/etc/libvirt/qemu/networks/default.xml # net-start default Roman Bogorodskiy pgpZGtuqlk5zW.pgp Description: PGP signature
Re: libvirt and bhyve problems
Craig Rodrigues wrote: On Wed, Jun 11, 2014 at 12:25 PM, Roman Bogorodskiy no...@freebsd.org wrote: I've attached a patch which should fix the segfault. Could you please let me know if it fixes the problem? I have incorporated your patch into the libvirt port, and provided a tarball of the port here: http://people.freebsd.org/~rodrigc/libvirt_port.tgz I can confirm that your patch eliminates the coredump for me. So now I can do the following: (1) Start libvirtd: /usr/local/sbin/libvirtd (2) Start virsh: virsh -c bhyve:///system (3) Execute following inside virsh: define bhyve.xml start bhyve ( using bhyve.xml file from http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-June/002588.html ) In the libvirtd console, I see this: 2014-06-11 22:47:16.357+: 34485605376: info : libvirt version: 1.2.5 2014-06-11 22:47:16.357+: 34485605376: error : virExec:417 : Cannot find 'pm-is-supported' in path: No such file or directory 2014-06-11 22:47:16.357+: 34485605376: warning : virQEMUCapsInit:948 : Failed to get host power management capabilities 2014-06-11 22:47:21.050+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/var/run/libvirt/qemu 2014-06-11 22:47:21.063+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/qemu 2014-06-11 22:47:21.168+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/bhyve 2014-06-11 22:47:21.168+: 34485605376: info : virDomainObjListLoadAllConfigs:18273 : Loading config file 'bhyve.xml' 2014-06-11 22:47:43.469+: 34485598208: error : virNetDevBridgeAddPort:399 : Unable to add bridge tap0 port vnet18: Invalid argument 2014-06-11 22:47:43.573+: 34485598208: error : virCommandWait:2426 : internal error: Child process (/usr/sbin/bhyvectl --destroy --vm=bhyve) unexpected exit status 255 For the last line, it looks like it is trying to do bhyvectl --destroy --vm=bhyve on a VM which doesn't exist, i.e. there is no entry in /dev/vmm/bhyve It should ignore the error at this point and try to spawn the VM, but it fails. This actually happens because it fails to spawn a VM (because there's no bridge) and tries to make sure to unload previously loaded VM in order not to leave it in a stale state. I have adjusted the port: http://people.freebsd.org/~novel/misc/libvirt_port_updated.tgz With this setup, I'm able to get networking (e.g. virsh net-list works) and updated the fix for the previous problem. I was able to start a VM with that setup. Roman Bogorodskiy pgpfn5YmLJaTs.pgp Description: PGP signature
Re: libvirt and bhyve problems
Craig Rodrigues wrote: On Thu, Jun 12, 2014 at 11:28 AM, Craig Rodrigues rodr...@freebsd.org wrote: On Thu, Jun 12, 2014 at 1:00 AM, Roman Bogorodskiy no...@freebsd.org wrote: http://people.freebsd.org/~novel/misc/libvirt_port_updated.tgz With this setup, I'm able to get networking (e.g. virsh net-list works) and updated the fix for the previous problem. I was able to start a VM with that setup. I deleted the old port from my system and took your modified port, built it, and installed it. I followed my previous steps in: http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-June/002588.html and got this: 2014-06-12 18:23:54.328+: 34485605376: info : libvirt version: 1.2.5 2014-06-12 18:23:54.328+: 34485605376: error : dnsmasqCapsRefreshInternal:726 : Cannot check dnsmasq binary dnsmasq: No such file or directory 2014-06-12 18:23:54.328+: 34485605376: info : networkReloadFirewallRules:1750 : Reloading iptables rules 2014-06-12 18:23:54.328+: 34485605376: info : networkRefreshDaemons:1722 : Refreshing network daemons 2014-06-12 18:23:54.438+: 34485605376: error : virExec:417 : Cannot find 'pm-is-supported' in path: No such file or directory 2014-06-12 18:23:54.439+: 34485605376: warning : virQEMUCapsInit:948 : Failed to get host power management capabilities 2014-06-12 18:23:54.460+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/var/run/libvirt/qemu 2014-06-12 18:23:54.461+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/qemu 2014-06-12 18:23:54.560+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/bhyve 2014-06-12 18:23:54.560+: 34485605376: info : virDomainObjListLoadAllConfigs:18273 : Loading config file 'bhyve.xml' 2014-06-12 18:24:17.940+: 34485598208: error : virNetDevBridgeAddPort:399 : Unable to add bridge tap0 port vnet0: Invalid argument 2014-06-12 18:24:18.056+: 34485598208: error : virCommandWait:2426 : internal error: Child process (/usr/sbin/bhyvectl --destroy --vm=bhyve) unexpected exit status 255 -- Craig After the program fails, these are my tap and bridge devices, as shown by ifconfig: bridge0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 ether 02:29:45:c7:8f:00 nd6 options=9PERFORMNUD,IFDISABLED id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143LEARNING,DISCOVER,AUTOEDGE,AUTOPTP ifmaxaddr 0 port 1 priority 128 path cost 2 member: tap2 flags=143LEARNING,DISCOVER,AUTOEDGE,AUTOPTP ifmaxaddr 0 port 7 priority 128 path cost 200 member: tap1 flags=143LEARNING,DISCOVER,AUTOEDGE,AUTOPTP ifmaxaddr 0 port 6 priority 128 path cost 200 member: tap0 flags=143LEARNING,DISCOVER,AUTOEDGE,AUTOPTP ifmaxaddr 0 port 5 priority 128 path cost 200 tap0: flags=8902BROADCAST,PROMISC,SIMPLEX,MULTICAST metric 0 mtu 1500 options=8LINKSTATE ether 00:bd:53:27:00:00 nd6 options=29PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL media: Ethernet autoselect status: no carrier tap1: flags=8902BROADCAST,PROMISC,SIMPLEX,MULTICAST metric 0 mtu 1500 options=8LINKSTATE ether 00:bd:59:27:00:01 nd6 options=29PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL media: Ethernet autoselect status: no carrier tap2: flags=8943UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST metric 0 mtu 1500 options=8LINKSTATE ether 00:bd:5e:27:00:02 nd6 options=29PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL media: Ethernet autoselect status: active Opened by PID 1506 vnet0: flags=8802BROADCAST,SIMPLEX,MULTICAST metric 0 mtu 1500 options=8LINKSTATE ether fe:54:00:33:20:8c nd6 options=21PERFORMNUD,AUTO_LINKLOCAL media: Ethernet autoselect status: no carrier This is in my /etc/rc.conf for creating bridge and tap devices on bootup: # # Create tap devices, one tap interface per BHyve VM. # Add the tap interfaces to bridge0 cloned_interfaces=bridge0 tap0 tap1 tap2 autobridge_interfaces=bridge0 autobridge_bridge0=tap* em0 Ah. Actually, libvirt manages bridges and taps on its own. So, the preferred flow is that libvirt starts its networks on startup automatically. By default it has a network with the 'virbr0' device which it creates on its own. Later on it creates vnet devices (which are tap devices
Re: libvirt and bhyve problems
is the best way to fix this? Thanks for the analysis! I'll think what would be a proper fix for that problem. Also, while thinking how to reproduce it, I did 'kldunload if_bridge' and started to see the same problem. So, as a temporary fix, could you check if you have bridge support available (in kernel or via module)? Bridges are needed to get networking anyway. Thanks, Roman Bogorodskiy pgptL5WguNh6w.pgp Description: PGP signature
Re: libvirt and bhyve problems
Craig Rodrigues wrote: On Wed, Jun 11, 2014 at 11:21 AM, Roman Bogorodskiy no...@freebsd.org wrote: Thanks for the analysis! I'll think what would be a proper fix for that problem. Also, while thinking how to reproduce it, I did 'kldunload if_bridge' and started to see the same problem. So, as a temporary fix, could you check if you have bridge support available (in kernel or via module)? Bridges are needed to get networking anyway. Hi, My output of kldstat is: 1 29 0x8020 19d6218 kernel 21 0x81bd7000 2e44b0 zfs.ko 32 0x81ebc000 8208 opensolaris.ko 41 0x81ec5000 4d68 nmdm.ko 51 0x81eca000 1a2110 vmm.ko 61 0x82211000 53e7 fdescfs.ko 71 0x82217000 9aff if_bridge.ko 81 0x82221000 5851 bridgestp.ko 91 0x82227000 5673 if_tap.ko 101 0x8222d000 2a94 uhid.ko 111 0x8223 3592 ums.ko so I do have if_bridge.ko loaded. I've attached a patch which should fix the segfault. Could you please let me know if it fixes the problem? However, we still need to figure out why networking is not being enabled. The root cause of that is that libvirt should start the default network when the daemon starts up. This network should create a bridge device (virbr0). Later on, when one starts a guest, it should create tap device and add it to the bridge. Obviously, it fails at this step as there's no bridge device. Could you please send me config.log file and also a port directory with your fixes so I could test it as well? As for your previous question on my setup, I'm using master branch of the libvirt git repo and my configure args are: $ ./configure --with-bhyve --without-polkit --with-hal \ CFLAGS=-g -O0 -I/usr/local/include LDFLAGS=-L/usr/local/lib \ --with-xml-catalog-file=/usr/local/share/xml/catalog I'm working on 10-STABLE. However, it's really bad if it works only this way. :-( I would love to fix the port. If you are doing development on CURRENT, make sure that you have WITH_LLDB=yes in /etc/src.conf when you rebuild the system. Looks like gdb in base cannot debug the cores. Thanks for the hint! Roman Bogorodskiy pgpqI5uoZ_yOU.pgp Description: PGP signature
Re: libvirt and bhyve problems
Craig Rodrigues wrote: On Tue, Jun 10, 2014 at 8:58 AM, Craig Rodrigues rodr...@freebsd.org wrote: On Tue, Jun 10, 2014 at 4:43 AM, Roman Bogorodskiy no...@freebsd.org wrote: Hi Craig, It's a little unfortunate that there's no stacktrace, though we can try to see what's going on without it. Could you please edit libvirtd.conf (should be something like /usr/local/etc/libvirtd.conf if installed from port) to add: log_level = 2 then run libvirtd by hand in foreground: OK, I added log_level = 2 to /usr/local/etc/libvirt/libvirtd.conf and ran libvirtd in the foreground: 2014-06-10 15:53:26.846+: 34485593088: info : libvirt version: 1.2.5 2014-06-10 15:53:26.846+: 34485593088: warning : virDriverLoadModule:63 : Module /usr/local/lib/libvirt/connection-driver/libvirt_driver_uml.so not accessible I saw the above warning, so I tried patching the port with this: --- pkg-plist.orig 2014-06-10 10:36:22.084757789 -0700 +++ pkg-plist 2014-06-10 09:27:25.280043239 -0700 @@ -274,6 +274,8 @@ lib/libvirt/connection-driver/libvirt_driver_secret.la lib/libvirt/connection-driver/libvirt_driver_storage.so lib/libvirt/connection-driver/libvirt_driver_storage.la +lib/libvirt/connection-driver/libvirt_driver_uml.so +lib/libvirt/connection-driver/libvirt_driver_uml.la lib/libvirt/connection-driver/libvirt_driver_vbox.so lib/libvirt/connection-driver/libvirt_driver_vbox.la This warning is not critical. Actually, UML should not be built on FreeBSD and I would need to disable it by default. For now I think the proper fix is to configure with '--without-uml' flag. As for the segfault, I think I have a guess. Could you please drop '--without-network' from CONFIGURE_ARGS in the port's Makefile and try to rebuild? Roman Bogorodskiy pgpzBHVhXENdD.pgp Description: PGP signature
Re: libvirt and bhyve problems
Craig Rodrigues wrote: On Tue, Jun 10, 2014 at 10:54 AM, Roman Bogorodskiy no...@freebsd.org wrote: This warning is not critical. Actually, UML should not be built on FreeBSD and I would need to disable it by default. For now I think the proper fix is to configure with '--without-uml' flag. As for the segfault, I think I have a guess. Could you please drop '--without-network' from CONFIGURE_ARGS in the port's Makefile and try to rebuild? Roman Bogorodskiy OK, I dropped --without-network from CONFIGURE_ARGS and rebuilt the port. I am still getting the coredump, but am getting different errors: 2014-06-10 18:59:22.713+: 34485593088: info : libvirt version: 1.2.5 2014-06-10 18:59:22.713+: 34485593088: warning : virDriverLoadModule:63 : Module /usr/local/lib/libvirt/connection-driver/libvirt_driver_network.so not accessible Hmmm, this sounds weird. I think I need to try the port version myself to see if I could reproduce it this way (I'm usually using git version of libvirt instead of port). In the meantime, I'd really appreciate if you could build the port with WITH_DEBUG flag and try to get a stacktrace. 2014-06-10 18:59:22.715+: 34485593088: error : virDriverLoadModule:71 : failed to load module /usr/local/lib/libvirt/connection-driver/libvirt_driver_qemu.so /usr/local/lib/libvirt/connection-driver/libvirt_driver_qemu.so: Undefined symbol networkAllocateActualDevice 2014-06-10 18:59:22.716+: 34485593088: error : virDriverLoadModule:71 : failed to load module /usr/local/lib/libvirt/connection-driver/libvirt_driver_uml.so /usr/local/lib/libvirt/connection-driver/libvirt_driver_uml.so: Undefined symbol inotify_add_watch 2014-06-10 18:59:22.845+: 34485605376: info : virDomainObjListLoadAllConfigs:18249 : Scanning for configs in /usr/local/etc/libvirt/bhyve 2014-06-10 18:59:22.845+: 34485605376: info : virDomainObjListLoadAllConfigs:18273 : Loading config file 'bhyve.xml' 2014-06-10 18:59:22.847+: 34485605376: error : do_open:1166 : no connection driver available for qemu:///system 2014-06-10 18:59:42.273+: 34485598208: error : virNetDevBridgeAddPort:399 : Unable to add bridge tap0 port vnet7: Invalid argument This last line also sounds strange. I'm wondering if you execute 'net-list --all' will you see the 'default' network? And if it's there, will 'net-show default' should that Autostart is enabled for it? Segmentation fault (core dumped) Thanks, Roman Bogorodskiy pgp0SZ86gpFPj.pgp Description: PGP signature
Re: [PATCH] Flexible vcpu pinning configuration
Neel Natu wrote:
Hi Roman,
On Sun, Apr 27, 2014 at 3:45 AM, Roman Bogorodskiy no...@freebsd.org wrote:
I've created an initial version of the patch which allows more flexible
vcpu pinning configuration.
Current schema is:
bhyve -p N
pins vcpu i to hostcpu N + i.
The propsed extension is:
bhyve -p N:M -p 0:1 -p 3:5
which pins vcpu N to host pcpu M. Option needs to be specified
individually for each vcpu.
So it works like that for me:
sudo /usr/sbin/bhyve -p 0:0 -p 1:3 -c 2 ...
# sudo cpuset -g -t 100262
tid 100262 mask: 0
# sudo cpuset -g -t 100264
tid 100264 mask: 3
PS I used cpumat_t* array to store these values instead of int, because
if the idea is OK, I'll extend it to support ranges like e.g. cpuset(1)
supports, e.g.: 1:2-5.
The questions are:
- Is it OK to chance '-p' arg syntax or it's better to introduce a new
one?
I think we can reuse the -p option unless anybody objects vociferously.
- Is the syntax OK (currently: 'vcpu:pcpu', later
'vcpu:pcpuN-pcpuM,pcpuX)?
Yup, I think that works fine.
The patch looks good in general but I have a few comments:
- Scope of 'vcpupmap[]' should be restricted to 'static'.
- usage() and man page need to be updated.
- pincpu_parse():
The option parsing can be made much easier by using:
if (sscanf(str, %d:%d, vcpu, pcpu) == 2) {
/* success */
} else {
return (-1);
}
If the same vcpu is specified multiple times then we should
malloc(sizeof(cpuset_t)) only the first time:
if (vcpumap[vcpu] != NULL)
mask = vcpumap[vcpu];
else
mask = malloc(sizeof(cpuset_t));
We need to range-check 'vcpu' before using it as an index into the
'vcpumap[]' array.
best
Neel
Attached an updated patch.
I'm still inclined to use something like parselist() from
usr.bin/cpuset/cpuset.c, but I don't want to copy/paste and I don't know
where it'd make sense to move it so it was usable outside of cpuset?
PS While reading bhyverun.c, I think I spotted a typo: in fbsdrun_deletecpu()
error message says fprintf(stderr, addcpu: Should it be deletecpu:
instead?
Roman Bogorodskiy
Index: bhyve.8
===
--- bhyve.8 (revision 264921)
+++ bhyve.8 (working copy)
@@ -78,12 +78,14 @@
allow a remote kernel kgdb to be relayed to the guest kernel gdb stub
via a local IPv4 address and this port.
This option will be deprecated in a future version.
-.It Fl p Ar pinnedcpu
+.It Fl p Ar n:m
Force guest virtual CPUs to be pinned to host CPUs.
Virtual CPU
.Em n
is pinned to host CPU
-.Em pinnedcpu+n .
+.Em m .
+This option could be specified multiple times to set pinning for each
+Virtual CPU and to pin Virtual CPU to more than one host CPU.
.It Fl P
Force the guest virtual CPU to exit when a PAUSE instruction is detected.
.It Fl W
Index: bhyverun.c
===
--- bhyverun.c (revision 264921)
+++ bhyverun.c (working copy)
@@ -83,7 +83,6 @@
int guest_ncpus;
-static int pincpu = -1;
static int guest_vmexit_on_hlt, guest_vmexit_on_pause, disable_x2apic;
static int virtio_msix = 1;
@@ -119,6 +118,8 @@
int mt_vcpu;
} mt_vmm_info[VM_MAXCPU];
+static cpuset_t *vcpumap[VM_MAXCPU] = { NULL };
+
static void
usage(int code)
{
@@ -125,12 +126,12 @@
fprintf(stderr,
Usage: %s [-aehwAHIPW] [-g gdb port] [-s pci] [-S pci]\n
- %*s [-c vcpus] [-p pincpu] [-m mem] [-l lpc] vm\n
+ %*s [-c vcpus] [-p vcpu:hostcpu] [-m mem] [-l lpc] vm\n
-a: local apic is in XAPIC mode (default is X2APIC)\n
-A: create an ACPI table\n
-g: gdb port\n
-c: # cpus (default 1)\n
- -p: pin vcpu 'n' to host cpu 'pincpu + n'\n
+ -p: n:m pin vcpu 'n' to host cpu 'm'\n
-H: vmexit from the guest on hlt\n
-P: vmexit from the guest on pause\n
-W: force virtio to use single-vector MSI\n
@@ -146,6 +147,53 @@
exit(code);
}
+static int
+pincpu_parse(const char *cpupin)
+{
+char *string;
+int vcpu = -1, pcpu = -1, ret = -1;
+
+if ((string = strdup(cpupin)) == NULL) {
+fprintf(stderr, strdup failed: %d\n, errno);
+return -1;
+}
+
+if (sscanf(string, %d:%d, vcpu, pcpu) != 2) {
+fprintf(stderr, invalid format: %s\n, string);
+goto cleanup;
+}
+
+if (vcpu 0 || vcpu VM_MAXCPU - 1) {
+fprintf(stderr, invalid vcpu value '%d',
+should be from 0 to %d\n,
+vcpu, VM_MAXCPU - 1);
+goto cleanup;
+}
+
+if (pcpu 0 || pcpu CPU_SETSIZE) {
+fprintf(stderr, invalid pcpu value '%d',
+should be from 0 to %d\n,
+pcpu, CPU_SETSIZE);
+goto cleanup;
+}
+
+if (vcpumap[vcpu] == NULL) {
+if ((vcpumap[vcpu] = malloc(sizeof(cpuset_t))) == NULL
Re: Understanding CPU and memory usage in Bhyve
Peter Grehan wrote: Hi Roman, But the summary is: is there a way to figure out how much CPU time bhyve and the guest spends on host CPUs N (N = 0, 1, ...)? I don't think FreeBSD records that for a process, and bhyve doesn't record the guest vCPU time on individual host CPUs (it's an aggregate over all CPUs). Do you know if Linux supports that ? As far as I understand, Linux supports that through Cgroups, specifically, quoting: http://docs.fedoraproject.org/en-US/Fedora/15/html/Resource_Management_Guide/sec-cpuacct.html cpuacct.usage_percpu reports the CPU time (in nanoseconds) consumed on each CPU by all tasks in this cgroup (including tasks lower in the hierarchy). Roman Bogorodskiy pgppooiAZ8GBU.pgp Description: PGP signature
Understanding CPU and memory usage in Bhyve
Hi,
I'm trying to understand how to obtain resource usage information for a
bhyve guest.
For the CPU, there's some info provided by bhyvectl, e.g.:
/usr/sbin/bhyvectl --vm=bhyve --get-stats --cpu=1
in the field 'vcpu total runtime', for example:
vcpu total runtime 8178870653
Two question about that:
1. What are the units used here? Looks like it's ticks, so if I want to
convert this number, to, say, nanoseconds, would it be right to do
something like:
X * 1000**3 / kern.clockrate.stathz
(X beeing the value returned by bhyvectl and kern.clockrate is a
sysctl name)?
2. This value shows only vcpu time, without hypervisor time?
For the memory stats, it's less obvious. Does guest always use an amount
of memory specified at startup with '-m' (i.e. bhyve -m 1G) or it could
use less?
What is the format of bhyvectl --get-{low,high}mem output?
PS I found it a little confusing that bhyvectl displays vcpu0 stats by
default if --cpu is not provided, expected it provide info of all vcpus
in the guest by default. Also, didn't find a way to get a number of
vcpus in a running guest.
Roman Bogorodskiy
pgpdf09dm2nHF.pgp
Description: PGP signature
Re: [SOLVED] Re: lazy bhyve doesn't boot until I conenct to console
Peter Grehan wrote: I'm seeing something odd... I am running this command: bhyve -c 4 -m 2048M -H -P -A -s 0:0,hostbridge -s 1:0,lpc -s 2,virtio-net,tap1 -s 3,virtio-blk,/dev/zvol/data/vm1 -l com1,/dev/nmdm1A vm1 but I the VM is being lazy... A number of people have reported this, and looking at it, I think it's a bug in bhyve. Fixed in r262884 Hi Peter, Do you plan to MFC that? The original commit set MFC to 3 weeks and now more than 4 weeks passed since. That would be a very useful fix to have in STABLE. Thanks, Roman Bogorodskiy pgpt5CDTqkcxg.pgp Description: PGP signature
Re: bhyve: allow specifiying tty by fd
Peter Grehan wrote: Hi Roman, Currently bhyve(8) allows to specify either stdio or device path as a TTY device. However, it could be useful to specify a TTY device by file descriptor, e.g. when bhyve is being executed in an automated way by other application, so a result of openpty(3) could be passed to it. Attached a poc patch for that. It allows to specify fd this way: bhyve snip -s 31,lpc -l com1,fd=19 vm0 Yes, useful idea, and will become more relevant in the capsicum world of passing fd's to a sandboxed bhyve. Thanks: I'll get this submitted. Hi Peter! Any news on that? Thanks, Roman Bogorodskiy pgpL082wXlZUG.pgp Description: PGP signature
Bhyve support in Libvirt
Hi! I was suggested it'd be useful to share that info on this list. The news is that libvirt 1.2.2 released around a month ago was the first libvirt version with Bhyve support! Libvirt is the virtualization library which aims to provide a unified API for hypervisors (and actually even more things, storage for example). Libvirt webpage is here: http://libvirt.org/ I've provided some more details in a blog post I wrote couple of weeks ago: http://empt1e.blogspot.ru/2014/03/bhyve-in-libvirt.html also, there is some info on the Qemu driver status as well in the blog, if you're interested. Roman Bogorodskiy pgpTgbNt36J7l.pgp Description: PGP signature
Re: Bhyve support in Libvirt
Craig Rodrigues wrote: On Mon, Mar 31, 2014 at 10:00 AM, Roman Bogorodskiy no...@freebsd.org wrote: http://empt1e.blogspot.ru/2014/03/bhyve-in-libvirt.html This is a huge step forward for bhyve! Thank you! There is a sizable ecosystem of software for managing hypervisors built on top of libvirt. Have you tested any of this software, such as virsh? Are there any GUI or web UI utilities built on top of libvirt which work with bhyve? virsh is an official client that is a part of libvirt. I use it for development and testing and also it's the main tool I use to manage VMs with libvirt. I know there's a GUI application called virt-manager (deskutils/virt-manager port), but I didn't try that. I'm not aware of web UI applications, but I didn't looks specifically. You might want to consider submitting a status report at: https://www.freebsd.org/news/status/status.html I'll take a look, thanks for a suggestion. Roman Bogorodskiy ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve + OpenStack
Craig Rodrigues wrote: On Sat, Mar 8, 2014 at 6:42 PM, Marc Fournier scra...@hub.org wrote: Has anyone started lookign into this? FreeBSD + bhyve would, Semihalf is doing work Openstack in FreeBSD. See: http://www.freebsd.org/news/status/report-2013-10-2013-12.html#FreeBSD-Host-Support-for-OpenStack-and-OpenContrail Roman Bogorodskiy no...@freebsd.org is working on libvirt and FreeBSD. However, his initial focus has been on QEMU, not bhyve See: http://empt1e.blogspot.hu/2013/11/running-qemu-vms-on-freebsd.html Yes, a lot of people would like to see Openstack + FreeBSD + BHyve. Hi, Actually, I'm working on the bhyve driver for libvirt as well. libvirt 1.2.2 released this March includes initial support for the bhyve driver. It's in a basic state so far though. OpenStack upstream prefers to see new hypervisor driver support through libvirt, so once libvirt bhyve driver would be suitable for that, I'm going to work in that direction. Roman Bogorodskiy pgp3kiq6YCQYI.pgp Description: PGP signature
bhyve: allow specifiying tty by fd
Hi,
Currently bhyve(8) allows to specify either stdio or device path as a
TTY device. However, it could be useful to specify a TTY device by file
descriptor, e.g. when bhyve is being executed in an automated way by
other application, so a result of openpty(3) could be passed to it.
Attached a poc patch for that. It allows to specify fd this way:
bhyve snip -s 31,lpc -l com1,fd=19 vm0
Roman Bogorodskiy
Index: bhyve.8
===
--- bhyve.8 (revision 262780)
+++ bhyve.8 (working copy)
@@ -184,6 +184,8 @@
the bhyve process.
.It Pa /dev/xxx
Use the host TTY device for serial port I/O.
+.It Li fd=N
+Use a specified file descriptor of the TTY device.
.El
.Pp
Pass-through devices:
Index: uart_emul.c
===
--- uart_emul.c (revision 262780)
+++ uart_emul.c (working copy)
@@ -585,7 +585,11 @@
retval = -1;
- fd = open(opts, O_RDWR);
+ if (!strncmp(opts, fd=, 3)) {
+ fd = atoi(opts+3);
+ } else {
+ fd = open(opts, O_RDWR);
+ }
if (fd 0 isatty(fd)) {
sc-tty.fd = fd;
sc-tty.opened = true;
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: BHyVe: vm_setup_memory(highmem): Cannot allocate memory
Peter Grehan wrote: Hi Roman, When I was seing this 'Failed to emulate instruction at 0x80594f3a', it happened right after boot loader menu (the one with the ascii-art). At that time I had 'hw.physmem=0x18000' in my host's loader.conf. Also the problem wasn't 100% reproducible, it started to show up after creating/destroying several VMs. OK - I'm now wondering if this is a guest phys memory leak in the hypervisor, and it's not reporting the error correctly. How many times do you have to create/destroy vm's for this to occurr ? I'm writing some sort of automation scripts so I created and destroy vms quite often, I guess maybe 20 times or so... BTW, any ideas why I stared loosing a console after: GDB: debug ports: bvm GDB: current port: bvm KDB: debugger backends: ddb gdb KDB: current backend: ddb ? Roman Bogorodskiy pgpUjmUjfkc0d.pgp Description: PGP signature
Re: BHyVe: vm_setup_memory(highmem): Cannot allocate memory
Peter Grehan wrote: Hi Roman, I start a VM like that: sudo /usr/sbin/bhyveload -m 256 -M 0 -h /var/run/bhyve/test1/vm1 test1 sudo /usr/bin/cpuset -l 0-3 /usr/sbin/bhyve -m 256 -M 0 -s 1,virtio-net,tap0 test1 The last commands give me: Failed to emulate instruction at 0x80594f3a What could be wrong with that? Was this during boot or later ? Do you have a console log up to where it occurred ? Hm, I see strange things happening. When I was seing this 'Failed to emulate instruction at 0x80594f3a', it happened right after boot loader menu (the one with the ascii-art). At that time I had 'hw.physmem=0x18000' in my host's loader.conf. Also the problem wasn't 100% reproducible, it started to show up after creating/destroying several VMs. I replaced the physmem value with hw.physmem=0x1. Now I have 100% reproducibility and it fails here: GDB: debug ports: bvm GDB: current port: bvm KDB: debugger backends: ddb gdb KDB: current backend: ddb (20:06) novel@kloomba:~/bhyve/vm1 % I can still see this VM in /dev/vmm though. Roman Bogorodskiy pgpAO7U4jmgnB.pgp Description: PGP signature
Re: BHyVe: vm_setup_memory(highmem): Cannot allocate memory
Andreas Nilsson wrote: On Sun, Sep 2, 2012 at 6:56 PM, Roman Bogorodskiy no...@freebsd.org wrote: Roman Bogorodskiy wrote: Peter Grehan wrote: Hi Roman, Yeah, but I'd like to leave more memory for the host system, so I specified 6GB. I have 8GB at all, so 2GB left for VMs. Should it cause any problems? No, that should be fine. BTW, I encountered a problem with the tap0 device. I create a tap device and assign an address to it, using 'ifconfig tap0 192.168.1.1 up'. I boot a VM and everything goes fine. When I do 'reboot' in the guest, it reboots, but tap0 on the host goes down and its address is dropped. Is that an expected behaviour? Yes - we probably need to fix the tap device to not do that :( Yeah, it would be great to get this fixed. And one question: is there a way to get a list of all running VMs on the host? Doesn't seem like vmmctl being able to do that. Oh, I've started reading the code and figured out that I can just do `ls /dev/vmm`! Roman Bogorodskiy IMHO: Please don't introduce reading stuff in /dev as a standard way to get info ( at least not for end user). sysctl is so much nicer to work with. Yes, reading from /dev is not very convenient, but appears that sysctl is not supported. I didn't read the code though, but I don't see anything looking like that in sysctl -a. Probably the most handy way of doing that is via 'vmmctl' or a tool like that. Anyways, I started spotting one more problem: I start a VM like that: sudo /usr/sbin/bhyveload -m 256 -M 0 -h /var/run/bhyve/test1/vm1 test1 sudo /usr/bin/cpuset -l 0-3 /usr/sbin/bhyve -m 256 -M 0 -s 1,virtio-net,tap0 test1 The last commands give me: Failed to emulate instruction at 0x80594f3a What could be wrong with that? Roman Bogorodskiy pgpfM2RxGRdE6.pgp Description: PGP signature
Re: BHyVe: vm_setup_memory(highmem): Cannot allocate memory
Peter Grehan wrote: Hi Roman, Yeah, but I'd like to leave more memory for the host system, so I specified 6GB. I have 8GB at all, so 2GB left for VMs. Should it cause any problems? No, that should be fine. BTW, I encountered a problem with the tap0 device. I create a tap device and assign an address to it, using 'ifconfig tap0 192.168.1.1 up'. I boot a VM and everything goes fine. When I do 'reboot' in the guest, it reboots, but tap0 on the host goes down and its address is dropped. Is that an expected behaviour? Yes - we probably need to fix the tap device to not do that :( Yeah, it would be great to get this fixed. And one question: is there a way to get a list of all running VMs on the host? Doesn't seem like vmmctl being able to do that. Thanks, Roman Bogorodskiy pgpkXhR9oLPFb.pgp Description: PGP signature
BHyVe: vm_setup_memory(highmem): Cannot allocate memory
Hi, I'm running into a problem. I did a fresh checkout of 'bhyve' branch from svn and followed the instruction here: http://people.freebsd.org/~neel/bhyve/bhyve_instructions.txt In my /boot/loader.conf I have: debug.witness.watch=0 hw.physmem=0x18000 and dmesg says: real memory = 8589934592 (8192 MB) avail memory = 5643694080 (5382 MB) When I try to start a VM I get this error: (19:46) novel@kloomba:~/bhyve/vm1 % sudo ./vmrun.sh -m 256 vm1 Launching virtual machine vm1 with 256MB memory below 4GB and 2048MB memory above 4GB ... vm_setup_memory(highmem): Cannot allocate memory (19:46) novel@kloomba:~/bhyve/vm1 % This page: http://callfortesting.org/bhyve/ describes that reloading 'vmm' would help, but doing it changes nothing. Any ideas what's wrong with that? Also, I am curious, what do these numbers mean: 256MB memory below 4GB and 2048MB memory above 4GB. I get that 256 is the amount of mem to give to VM, but what do 4GB and 2048MB stand for? Thanks Roman Bogorodskiy pgpHiqI81lbEq.pgp Description: PGP signature
