Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On 04/09/18 11:02, Kashyap Chamarthy wrote: > - It also provides a persistent command history in a convenient file: > '~/.qmp-shell_history' I noticed it used readline, but I didn't know about the dedicated history file. Nice! Thanks! Laszlo -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On Fri, Apr 06, 2018 at 08:21:00PM +0200, Laszlo Ersek wrote:
> On 04/06/18 20:10, Eric Blake wrote:
[...]
> # key=value pairs also support Python or JSON object literal subset notations,
> # without spaces. Dictionaries/objects {} are supported as are arrays [].
> #
> #example-command arg-name1={'key':'value','obj'={'prop':"value"}}
> #
> # Both JSON and Python formatting should work, including both styles of
> # string literal quotes. Both paradigms of literal values should work,
> # including null/true/false for JSON and None/True/False for Python.
A couple of examples of the the key-value pairs and using JSON dicts
with 'qmp-shell'.
Key-value pairs:
(QEMU) blockdev-snapshot node=node-Base snapshot-file=./overlay1.qcow2
overlay=node-Overlay1
[...]
A combination key-value pairs and JSON dicts:
(QEMU) blockdev-add driver=qcow2 node-name=node-overlay1
file={"driver":"file","filename":"overlay1.qcow2"}
[...]
> This looks awesome, because it should let me provide messy nested input
> (which I'll obviously compose in my $EDITOR and then paste it), and then
> the QMP shell will both validate and pretty print that. I'm going to try
> this.
A couple of notes when using the 'qmp-shell':
- Assuming you started QEMU with: `[...] -qmp
unix:/tmp/qmp-sock,server,nowait`, you might want to use the '-p'
command-line option to prett-print the JSON:
$ qmp-shell -v -p /tmp/qmp-sock
- It also provides a persistent command history in a convenient file:
'~/.qmp-shell_history'
--
/kashyap
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On 04/06/18 20:10, Eric Blake wrote:
> On 04/06/2018 12:28 PM, Laszlo Ersek wrote:
>
>> I've created an RFC-level "qapi/firmware.json" schema file, based on
>> this discussion. It "builds", and the generated documentation looks
>> acceptable, superficially speaking.
>>
>> Before I post "qapi/firmware.json" for getting comments, I'd like to
>> write JSON text that (a) describes firmware that I use, and (b)
>> conforms to the schema. IOW, I'd like to validate whether the schema
>> is good enough for describing at least such firmware that I know.
>>
>> Is there a tool that generates example JSON objects from a given
>> schema?
>
> I know the QMP shell (scripts/qmp/qmp-shell) lets you enter commands
> with a lot less typing than full JSON, and has a mode where it will
> then echo the full JSON command it constructed from what you typed. To
> be able to quickly validate examples, it may be sufficient to
> temporarily add a new QMP command 'check-firmware':
>
> { 'command': 'check-firmware', 'boxed': true, 'data': 'Firmware' }
>
> assuming 'Firmware' is your top-level 'struct' in the QAPI file, then
> implement a trivial:
>
> qmp_check_firmware(Firmware *obj, Error **errp) {
> return 0;
> }
>
> so that you can then run QMP shell, and type:
>
> check-firmware arg1=foo arg2=bar ...
>
> which will then generate the corresponding JSON, then either
> successfully do nothing (what you typed validated, AND you have the
> JSON output printed), or print an error (what you typed failed QAPI
> validation, perhaps because it had an unrecognized key, was missing a
> required key, used a wrong type, etc).
>
>> I vaguely recall there used to be one. Otherwise, writing the
>> examples manually looks arduous (and I wouldn't know how to verify
>> them against the schema).
>
> Similarly, if you generate a command the produces a 'Firmware' as the
> return value, then you can populate the generated C struct (since you
> did manage to run the QAPI generator over your new file, you should be
> able to look at the C struct it generated), then output that over QMP
> to show the counterpart JSON that matches the struct as populated.
>
The top level structure is complex / nested, but that doesn't appear to
be an issue. According to the script,
# key=value pairs also support Python or JSON object literal subset notations,
# without spaces. Dictionaries/objects {} are supported as are arrays [].
#
#example-command arg-name1={'key':'value','obj'={'prop':"value"}}
#
# Both JSON and Python formatting should work, including both styles of
# string literal quotes. Both paradigms of literal values should work,
# including null/true/false for JSON and None/True/False for Python.
This looks awesome, because it should let me provide messy nested input
(which I'll obviously compose in my $EDITOR and then paste it), and then
the QMP shell will both validate and pretty print that. I'm going to try
this.
Thank you, Eric!
Laszlo
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On 04/06/2018 12:28 PM, Laszlo Ersek wrote:
> I've created an RFC-level "qapi/firmware.json" schema file, based on
> this discussion. It "builds", and the generated documentation looks
> acceptable, superficially speaking.
>
> Before I post "qapi/firmware.json" for getting comments, I'd like to
> write JSON text that (a) describes firmware that I use, and (b) conforms
> to the schema. IOW, I'd like to validate whether the schema is good
> enough for describing at least such firmware that I know.
>
> Is there a tool that generates example JSON objects from a given schema?
I know the QMP shell (scripts/qmp/qmp-shell) lets you enter commands
with a lot less typing than full JSON, and has a mode where it will then
echo the full JSON command it constructed from what you typed. To be
able to quickly validate examples, it may be sufficient to temporarily
add a new QMP command 'check-firmware':
{ 'command': 'check-firmware', 'boxed': true, 'data': 'Firmware' }
assuming 'Firmware' is your top-level 'struct' in the QAPI file, then
implement a trivial:
qmp_check_firmware(Firmware *obj, Error **errp) {
return 0;
}
so that you can then run QMP shell, and type:
check-firmware arg1=foo arg2=bar ...
which will then generate the corresponding JSON, then either
successfully do nothing (what you typed validated, AND you have the JSON
output printed), or print an error (what you typed failed QAPI
validation, perhaps because it had an unrecognized key, was missing a
required key, used a wrong type, etc).
> I vaguely recall there used to be one. Otherwise, writing the examples
> manually looks arduous (and I wouldn't know how to verify them against
> the schema).
Similarly, if you generate a command the produces a 'Firmware' as the
return value, then you can populate the generated C struct (since you
did manage to run the QAPI generator over your new file, you should be
able to look at the C struct it generated), then output that over QMP to
show the counterpart JSON that matches the struct as populated.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
signature.asc
Description: OpenPGP digital signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On 03/08/18 11:17, Daniel P. Berrangé wrote: > On Thu, Mar 08, 2018 at 08:52:45AM +0100, Gerd Hoffmann wrote: >> Hi, >> [*] Open question: Who, between QEMU and libvirt, should define the said firmware metadata format and file? >>> >>> IMHO QEMU should be defining the format, because the file will contain >>> info about certain QEMU features associated with the firmware (eg smm). >>> Also there are potentially other non-libvirt mgmt apps that spawn QEMU >>> which would like this info (eg libguestfs), so having libvirt define the >>> format is inappropriate. >>> >>> I'd suggest we just need something in docs/specs/firmware-metadata.rst >>> for QEMU source tree. >>> >>> Potentially QEMU could even use the metadata files itself for finding >>> the default firmeware images, instead of compiling this info into its >>> binaries. I wouldn't suggest we need todo that right away, but bear it >>> in mind as a potential use case. >> >> With qemu using this itself in mind it probably makes sense to specify >> this as qapi schema. That'll simplify parsing and using these files in >> qemu, and possibly simplifies things on the libvirt side too. > > I was thinking of an 'ini' style format, similar to that used by systemd > unit files, but a JSON format file is a nicer fit with QEMU & Libvirt if > we describe it with qapi. I've created an RFC-level "qapi/firmware.json" schema file, based on this discussion. It "builds", and the generated documentation looks acceptable, superficially speaking. Before I post "qapi/firmware.json" for getting comments, I'd like to write JSON text that (a) describes firmware that I use, and (b) conforms to the schema. IOW, I'd like to validate whether the schema is good enough for describing at least such firmware that I know. Is there a tool that generates example JSON objects from a given schema? I vaguely recall there used to be one. Otherwise, writing the examples manually looks arduous (and I wouldn't know how to verify them against the schema). Thanks! Laszlo -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On Thu, Mar 08, 2018 at 08:52:45AM +0100, Gerd Hoffmann wrote: > Hi, > > > > [*] Open question: Who, between QEMU and libvirt, should define the said > > > firmware metadata format and file? > > > > IMHO QEMU should be defining the format, because the file will contain > > info about certain QEMU features associated with the firmware (eg smm). > > Also there are potentially other non-libvirt mgmt apps that spawn QEMU > > which would like this info (eg libguestfs), so having libvirt define the > > format is inappropriate. > > > > I'd suggest we just need something in docs/specs/firmware-metadata.rst > > for QEMU source tree. > > > > Potentially QEMU could even use the metadata files itself for finding > > the default firmeware images, instead of compiling this info into its > > binaries. I wouldn't suggest we need todo that right away, but bear it > > in mind as a potential use case. > > With qemu using this itself in mind it probably makes sense to specify > this as qapi schema. That'll simplify parsing and using these files in > qemu, and possibly simplifies things on the libvirt side too. Yeah, FWIW, I too find using the QAPI schema for the metadata file format more appealing (rather than a separate file format). And libvirt already has infrastructure to handle QAPI. -- /kashyap -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On Thu, Mar 08, 2018 at 08:52:45AM +0100, Gerd Hoffmann wrote: > Hi, > > > > [*] Open question: Who, between QEMU and libvirt, should define the said > > > firmware metadata format and file? > > > > IMHO QEMU should be defining the format, because the file will contain > > info about certain QEMU features associated with the firmware (eg smm). > > Also there are potentially other non-libvirt mgmt apps that spawn QEMU > > which would like this info (eg libguestfs), so having libvirt define the > > format is inappropriate. > > > > I'd suggest we just need something in docs/specs/firmware-metadata.rst > > for QEMU source tree. > > > > Potentially QEMU could even use the metadata files itself for finding > > the default firmeware images, instead of compiling this info into its > > binaries. I wouldn't suggest we need todo that right away, but bear it > > in mind as a potential use case. > > With qemu using this itself in mind it probably makes sense to specify > this as qapi schema. That'll simplify parsing and using these files in > qemu, and possibly simplifies things on the libvirt side too. I was thinking of an 'ini' style format, similar to that used by systemd unit files, but a JSON format file is a nicer fit with QEMU & Libvirt if we describe it with qapi. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On Thu, Mar 08, 2018 at 08:45:07AM +0100, Gerd Hoffmann wrote: > > Suggested approach > > -- > > > > Based on an upstream discussion on 'virt-tools'[1] mailing list and some > > Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion > > to define a firmware metadata format and file (example in [1]): > > > > - For each firmware file we need a metadata file in a well defined > > location, e.g. /usr/share/qemu/bios/ that lists stuff like: > > > > - Path to the firmware binary > > - Path to the pre-built OVMF 'vars' file (if any) > > How to load the binary (using -bios, -pflash, possibly also -kernel, for > uboot @ arm). I wonder if there's value in also using this for describing secondary device specific ROMs like iPXE and friends. > > > - Support architectures - associated QEMU feature flags (Secure > > Boot) > > Also machine types. ovmf builds with smm don't boot on pc. coreboot > has hardware-specific roms too, so the pc build wouldn't boot on q35 and > visa versa. Same on arm, where the firmware typically is board-specific. > > > - If the binary provides / requires SMM (System Management Mode) > > Possibly a more generic "flags" or "properties" thing, I can easily > imagine that simliar requirements show up on other platforms too. > > Also a "name" and a "description" field would be useful. > > cheers, > Gerd > Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
Hi, > > [*] Open question: Who, between QEMU and libvirt, should define the said > > firmware metadata format and file? > > IMHO QEMU should be defining the format, because the file will contain > info about certain QEMU features associated with the firmware (eg smm). > Also there are potentially other non-libvirt mgmt apps that spawn QEMU > which would like this info (eg libguestfs), so having libvirt define the > format is inappropriate. > > I'd suggest we just need something in docs/specs/firmware-metadata.rst > for QEMU source tree. > > Potentially QEMU could even use the metadata files itself for finding > the default firmeware images, instead of compiling this info into its > binaries. I wouldn't suggest we need todo that right away, but bear it > in mind as a potential use case. With qemu using this itself in mind it probably makes sense to specify this as qapi schema. That'll simplify parsing and using these files in qemu, and possibly simplifies things on the libvirt side too. cheers, Gerd -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
> Suggested approach > -- > > Based on an upstream discussion on 'virt-tools'[1] mailing list and some > Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion > to define a firmware metadata format and file (example in [1]): > > - For each firmware file we need a metadata file in a well defined > location, e.g. /usr/share/qemu/bios/ that lists stuff like: > > - Path to the firmware binary > - Path to the pre-built OVMF 'vars' file (if any) How to load the binary (using -bios, -pflash, possibly also -kernel, for uboot @ arm). > - Support architectures - associated QEMU feature flags (Secure > Boot) Also machine types. ovmf builds with smm don't boot on pc. coreboot has hardware-specific roms too, so the pc build wouldn't boot on q35 and visa versa. Same on arm, where the firmware typically is board-specific. > - If the binary provides / requires SMM (System Management Mode) Possibly a more generic "flags" or "properties" thing, I can easily imagine that simliar requirements show up on other platforms too. Also a "name" and a "description" field would be useful. cheers, Gerd -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
On Wed, Mar 07, 2018 at 03:49:51PM +0100, Kashyap Chamarthy wrote: > Problem background > -- > > The various OVMF binary file names and paths are slightly different[+] > for each Linux distribution. And each high-level management tool > (libguestfs, oVirt, `virt-manager` and OpenStack Nova) is inventing its > own approach to detect and configure the said OVMF files. This email > thread is about arriving at some common understanding to make this a bit > more "unified" by addressing these needs in QEMU and libvirt. Let me expand on the problem & desired solution from libvirt's POV Currently, if you are happy with the default firmware for a particular machine type, life is easy. QEMU knows the right path to use automatically. This was fine until people started wanting to use UEFI firemware with x86_64 Q35 and/or aarch64 virt machine types, because Q35 defaults to SeaBIOS and aarch64 defaults to no firmware. So mgmt applications wanting to use UEFI have to provide a full path to the firmware to libvirt. >From a libvirt POV, we want to make it possible to just ask for a firmware type eg for x86_64 we want to allow apps to just do either or and then have libvirt automatically provide the best firmware image path to QEMU. We'll still allow apps to provide an explicit path themselves if they really want to, because if nothing else that's useful for people who need to test out ad-hoc firmware builds. IOW, in general if you (sysadmin/mgmt app) want to enable EFI for a guest, you should never need to care about firmware paths. This will give libvirt QEMU driver parity with what's possible for vmware guests in this area. With this in mind, when we talk about providing metadata files for firmware below, we should bear in mind that we likely want this metadata to be general purpose, not something specific to OVMF. IOW all existing QEMU firmware images, for all architetures should be covered & whatever custom firmware users might have. > Suggested approach > -- > > Based on an upstream discussion on 'virt-tools'[1] mailing list and some > Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion > to define a firmware metadata format and file (example in [1]): > > - For each firmware file we need a metadata file in a well defined > location, e.g. /usr/share/qemu/bios/ that lists stuff like: > > - Path to the firmware binary > - Path to the pre-built OVMF 'vars' file (if any) Not neccessarily just OVMF - anything that uses the same pflash config mechanism. > - Support architectures - associated QEMU feature flags (Secure > Boot) We need to have ability to list machine types too, since historically there have been cases where we screwed up and needed to provide differnt builds of the same firmware to different machine types. > - If the binary provides / requires SMM (System Management Mode) I think we just need a well defined list of properties / features here as it might need to track morethan just SMM. > Essentially, QEMU would define[*] the file format, and provide > metadata files from any ROMs it ships directly. If Linux > distributions / vendors ship extra ROMs like OVMF, etc then they > should provide suitable metadata files. We need to define what order the metadata files are processed in, so that if there are multuple firemware images that satisfy a given guest config, we have a predictable / determinstic choice. This could be as simple as recommending each filename start with a 2 digit number. eg 50-ovmf.json That lets end user drop in a 20-ovmf.json if they want to provide a different ovmf by default. > - Libvirt can read these metadata files and then pick the correct > firmware binary based on the settings for the guest. IOW, we'll look at various things like firmware=efi|bios, and whether secure boot is requested for the guest. I don't think we actually have an explicit attribute for requesting secureboot=on|off, so we'd probably want to add one. THis is then used to pick the default firmware. As a specific non-goal: If multiple metadata files are all satisfactory for a given guest config, libvirt is just going to pick the first acceptable match. If users/apps want to have finer grained control, they can still pass an explicit path themselves as done today to override the default choice. > - Management tools can then wire up the libvirt-based OVMF SB (Secure > Boot) configuration. NB, mgmt tools should never need to read the metadata files - they'll just request libvirt to use firmware=efi|bios, etc > [*] Open question: Who, between QEMU and libvirt, should define the said > firmware metadata format and file? IMHO QEMU should be defining the format, because the file will contain info about certain QEMU features associated with the firmware (eg smm). Also there are potentially other non-libvirt mgmt apps that spawn QEMU which would like this
[libvirt] [RFC] Defining firmware (OVMF, et al) metadata format & file
Problem background -- The various OVMF binary file names and paths are slightly different[+] for each Linux distribution. And each high-level management tool (libguestfs, oVirt, `virt-manager` and OpenStack Nova) is inventing its own approach to detect and configure the said OVMF files. This email thread is about arriving at some common understanding to make this a bit more "unified" by addressing these needs in QEMU and libvirt. Suggested approach -- Based on an upstream discussion on 'virt-tools'[1] mailing list and some Bugzillas, Gerd Hoffmann, Laszlo Ersek and Dan Berrangé had a suggestion to define a firmware metadata format and file (example in [1]): - For each firmware file we need a metadata file in a well defined location, e.g. /usr/share/qemu/bios/ that lists stuff like: - Path to the firmware binary - Path to the pre-built OVMF 'vars' file (if any) - Support architectures - associated QEMU feature flags (Secure Boot) - If the binary provides / requires SMM (System Management Mode) Essentially, QEMU would define[*] the file format, and provide metadata files from any ROMs it ships directly. If Linux distributions / vendors ship extra ROMs like OVMF, etc then they should provide suitable metadata files. - Libvirt can read these metadata files and then pick the correct firmware binary based on the settings for the guest. - Management tools can then wire up the libvirt-based OVMF SB (Secure Boot) configuration. [*] Open question: Who, between QEMU and libvirt, should define the said firmware metadata format and file? References -- [1] A past proposal from Gerd to create a sort of a "firmware registry" https://www.redhat.com/archives/virt-tools-list/2014-September/msg00145.html [2] A libvirt upstream RFE bug requesting to simplify handling UEFI https://bugzilla.redhat.com/show_bug.cgi?id=1295146 -- RFE: provide a bios=uefi XML convenience option [3] DanPB wrote a PoC in libvirt: https://www.redhat.com/archives/libvir-list/2016-October/msg00045.html -- [libvirt] [PATCH 0/3] Make UEFI firmware config simpler But later came to the conclusion that it is flawed, and said we go the route of "Suggested approach" mentioned earlier). [*] OVMF package path names for different distributions --- - Debian (https://packages.debian.org/stretch/all/ovmf/filelist) - /usr/share/OVMF/OVMF_CODE.fd - /usr/share/OVMF/OVMF_VARS.fd - OpenSUSE (package: "qemu-ovmf-x86_64" -- and it has *35* files in that package): - /usr/share/qemu/ovmf-x86_64-opensuse-code.bin - /usr/share/qemu/ovmf-x86_64-opensuse-vars.bin [...] Their RPM spec file gives some hints (where all the files with 'ms' means signed with MS keys; the files with 'opensuse-4096' means signed with OpenSUSE 4096 bit CA keys -- I think that's one of the points of Secure Boot, to let people have control over system keys): https://build.opensuse.org/package/view_file/openSUSE:Factory/ovmf/ovmf.spec?expand=1 - Fedora 27 (package: "edk2-ovmf", x86_64): - /usr/share/edk2/ovmf/OVMF_CODE.fd - /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd - /usr/share/edk2/ovmf/OVMF_VARS.fd - RHEL-7.4 (package: "OVMF", x86_64): - /usr/share/OVMF/OVMF_CODE.secboot.fd - /usr/share/OVMF/OVMF_VARS.fd - Gerd's firmware repo from Git: - /usr/share/edk2.git/ovmf-x64/OVMF_VARS-need-smm.fd - /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd -- /kashyap -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
