Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
On 8/11/2018 12:50 AM, Chris Apsey wrote: This sounds promising and there seems to be a feasible way to do this, but it also sounds like a decent amount of effort and would be a new feature in a future release rather than a bugfix - am I correct in that assessment? Yes I'd say it's a blueprint and not a bug fix - it's not something we'd backport to stable branches upstream, for example. -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
This sounds promising and there seems to be a feasible way to do this, but it also sounds like a decent amount of effort and would be a new feature in a future release rather than a bugfix - am I correct in that assessment? On August 9, 2018 13:30:31 "Daniel P. Berrangé" wrote: On Thu, Aug 09, 2018 at 12:14:56PM -0500, Matt Riedemann wrote: On 8/9/2018 6:03 AM, Chris Apsey wrote: Exactly. And I agree, it seems like hw_architecture should dictate which emulator is chosen, but as you mentioned its currently not. I'm not sure if this is a bug and it's supposed to 'just work', or just something that was never fully implemented (intentionally) and would be more of a feature request/suggestion for a later version. The docs are kind of sparse in this area. What are your thoughts? I can open a bug if you think the scope is reasonable. I'm not sure if this is a bug or a feature, or if there are reasons why it's never been done. I'm gonna have to rope in Kashyap and danpb since they'd likely know more. Dan/Kaskyap: tl;dr why doesn't the nova libvirt driver, configured for qemu, set the guest.arch based on the hw_architecture image property so that you can run ppc guests in an x86 host? Yes, it should do exactly that IMHO ! The main caveat is that a hell of alot of code in libvirt assumes that guest arch == host arch. ie when building guest XML there's lots of code that looks at caps.host.cpu.arch to decide how to configure the guest. This all needs fixing to look at the guest.arch value instead, having set that from hw_architecture prop. Nova libvirt driver is already reporting that it is capable of running guest with multiple arches (the _get_instance_capaiblities method in nova/virt/libvirt/driver.py). The only other thing is that you likely want to distinguish between hosts that can do PPC64 via KVM vs those that can only do it via emulation, so you don't get guests randomly placed on slow vs fast hosts. Some kind of scheduler filter / weighting can do that based on info already reported from the compute host I expect. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
Exactly. And I agree, it seems like hw_architecture should dictate which emulator is chosen, but as you mentioned its currently not. I'm not sure if this is a bug and it's supposed to 'just work', or just something that was never fully implemented (intentionally) and would be more of a feature request/suggestion for a later version. The docs are kind of sparse in this area. What are your thoughts? I can open a bug if you think the scope is reasonable. --- v/r Chris Apsey bitskr...@bitskrieg.net https://www.bitskrieg.net On 2018-08-08 06:40 PM, Matt Riedemann wrote: On 8/8/2018 2:42 PM, Chris Apsey wrote: qemu-system-arm, qemu-system-ppc64, etc. in our environment are all x86 packages, but they perform system-mode emulation (via dynamic instruction translation) for those target environments. So, you run qemu-system-ppc64 on an x86 host in order to get a ppc64-emulated VM. Our use case is specifically directed at reverse engineering binaries and fuzzing for vulnerabilities inside of those architectures for things that aren't built for x86, but there are others. If you were to apt-get install qemu-system and then hit autocomplete, you'd get a list of archiectures that qemu can emulate on x86 hardware - that's what we're trying to do incorporate. We still want to run normal qemu-x86 with KVM virtualization extensions, but we ALSO want to run the other emulators without the KVM virtualization extensions in order to have more choice for target environments. So to me, openstack would interpret this by checking to see if a target host supports the architecture specified in the image (it does this correctly), then it would choose the correct qemu-system-xx for spawning the instance based on the architecture flag of the image, which it currently does not (it always choose qemu-system-x86_64). Does that make sense? OK yeah now I'm following you - running ppc guests on an x86 host (virt_type=qemu rather than kvm right?). I would have thought the hw_architecture image property was used for this somehow to configure the arch in the guest xml properly, like it's used in a few places [1][2][3]. See [4], I'd think we'd set the guest.arch but don't see that happening. We do set the guest.os_type though [5]. [1] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4649 [2] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4927 [3] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/blockinfo.py#L257 [4] https://libvirt.org/formatcaps.html#elementGuest [5] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L5196 ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
On 8/8/2018 2:42 PM, Chris Apsey wrote: qemu-system-arm, qemu-system-ppc64, etc. in our environment are all x86 packages, but they perform system-mode emulation (via dynamic instruction translation) for those target environments. So, you run qemu-system-ppc64 on an x86 host in order to get a ppc64-emulated VM. Our use case is specifically directed at reverse engineering binaries and fuzzing for vulnerabilities inside of those architectures for things that aren't built for x86, but there are others. If you were to apt-get install qemu-system and then hit autocomplete, you'd get a list of archiectures that qemu can emulate on x86 hardware - that's what we're trying to do incorporate. We still want to run normal qemu-x86 with KVM virtualization extensions, but we ALSO want to run the other emulators without the KVM virtualization extensions in order to have more choice for target environments. So to me, openstack would interpret this by checking to see if a target host supports the architecture specified in the image (it does this correctly), then it would choose the correct qemu-system-xx for spawning the instance based on the architecture flag of the image, which it currently does not (it always choose qemu-system-x86_64). Does that make sense? OK yeah now I'm following you - running ppc guests on an x86 host (virt_type=qemu rather than kvm right?). I would have thought the hw_architecture image property was used for this somehow to configure the arch in the guest xml properly, like it's used in a few places [1][2][3]. See [4], I'd think we'd set the guest.arch but don't see that happening. We do set the guest.os_type though [5]. [1] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4649 [2] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4927 [3] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/blockinfo.py#L257 [4] https://libvirt.org/formatcaps.html#elementGuest [5] https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L5196 -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
Matt, qemu-system-arm, qemu-system-ppc64, etc. in our environment are all x86 packages, but they perform system-mode emulation (via dynamic instruction translation) for those target environments. So, you run qemu-system-ppc64 on an x86 host in order to get a ppc64-emulated VM. Our use case is specifically directed at reverse engineering binaries and fuzzing for vulnerabilities inside of those architectures for things that aren't built for x86, but there are others. If you were to apt-get install qemu-system and then hit autocomplete, you'd get a list of archiectures that qemu can emulate on x86 hardware - that's what we're trying to do incorporate. We still want to run normal qemu-x86 with KVM virtualization extensions, but we ALSO want to run the other emulators without the KVM virtualization extensions in order to have more choice for target environments. So to me, openstack would interpret this by checking to see if a target host supports the architecture specified in the image (it does this correctly), then it would choose the correct qemu-system-xx for spawning the instance based on the architecture flag of the image, which it currently does not (it always choose qemu-system-x86_64). Does that make sense? Chris --- v/r Chris Apsey bitskr...@bitskrieg.net https://www.bitskrieg.net On 2018-08-08 03:07 PM, Matt Riedemann wrote: On 8/7/2018 8:54 AM, Chris Apsey wrote: We don't actually have any non-x86 hardware at the moment - we're just looking to run certain workloads in qemu full emulation mode sans KVM extensions (we know there is a huge performance hit - it's just for a few very specific things). The hosts I'm talking about are normal intel-based compute nodes with several different qemu packages installed (arm, ppc, mips, x86_64 w/ kvm extensions, etc.). Is nova designed to work in this kind of scenario? It seems like many pieces are there, but they're just not quite tied together quite right, or there is some config option I'm missing. As far as I know, nova doesn't make anything arch-specific for QEMU. Nova will execute some qemu commands like qemu-img but as far as the virt driver, it goes through the libvirt-python API bindings which wrap over libvirtd which interfaces with QEMU. I would expect that if you're on an x86_64 arch host, that you can't have non-x86_64 packages installed on there (or they are noarch packages). Like, I don't know how your packaging works (are these rpms or debs, or other?) but how do you have ppc packages installed on an x86 system? ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
On 8/7/2018 8:54 AM, Chris Apsey wrote: We don't actually have any non-x86 hardware at the moment - we're just looking to run certain workloads in qemu full emulation mode sans KVM extensions (we know there is a huge performance hit - it's just for a few very specific things). The hosts I'm talking about are normal intel-based compute nodes with several different qemu packages installed (arm, ppc, mips, x86_64 w/ kvm extensions, etc.). Is nova designed to work in this kind of scenario? It seems like many pieces are there, but they're just not quite tied together quite right, or there is some config option I'm missing. As far as I know, nova doesn't make anything arch-specific for QEMU. Nova will execute some qemu commands like qemu-img but as far as the virt driver, it goes through the libvirt-python API bindings which wrap over libvirtd which interfaces with QEMU. I would expect that if you're on an x86_64 arch host, that you can't have non-x86_64 packages installed on there (or they are noarch packages). Like, I don't know how your packaging works (are these rpms or debs, or other?) but how do you have ppc packages installed on an x86 system? -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
Hey Matt, We don't actually have any non-x86 hardware at the moment - we're just looking to run certain workloads in qemu full emulation mode sans KVM extensions (we know there is a huge performance hit - it's just for a few very specific things). The hosts I'm talking about are normal intel-based compute nodes with several different qemu packages installed (arm, ppc, mips, x86_64 w/ kvm extensions, etc.). Is nova designed to work in this kind of scenario? It seems like many pieces are there, but they're just not quite tied together quite right, or there is some config option I'm missing. Thanks! --- v/r Chris Apsey bitskr...@bitskrieg.net https://www.bitskrieg.net On 2018-08-07 09:32 AM, Matt Riedemann wrote: On 8/5/2018 1:43 PM, Chris Apsey wrote: Trying to enable some alternate (non-x86) architectures on xenial + queens. I can load up images and set the property correctly according to the supported values (https://docs.openstack.org/nova/queens/configuration/config.html) in image_properties_default_architecture. From what I can tell, the scheduler works correctly and instances are only scheduled on nodes that have the correct qemu binary installed. However, when the instance request lands on this node, it always starts it with qemu-system-x86_64 rather than qemu-system-arm, qemu-system-ppc, etc. If I manually set the correct binary, everything works as expected. Am I missing something here, or is this a bug in nova-compute? image_properties_default_architecture is only used in the scheduler filter to pick a compute host, it doesn't do anything about the qemu binary used in nova-compute. mnaser added the config option so maybe he can share what he's done on his computes. Do you have qemu-system-x86_64 on non-x86 systems? Seems like a package/deploy issue since I'd expect x86 packages shouldn't install on a ppc system and vice versa, and only one qemu package should provide the binary. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][glance] nova-compute choosing incorrect qemu binary when scheduling 'alternate' (ppc64, armv7l) architectures?
On 8/5/2018 1:43 PM, Chris Apsey wrote: Trying to enable some alternate (non-x86) architectures on xenial + queens. I can load up images and set the property correctly according to the supported values (https://docs.openstack.org/nova/queens/configuration/config.html) in image_properties_default_architecture. From what I can tell, the scheduler works correctly and instances are only scheduled on nodes that have the correct qemu binary installed. However, when the instance request lands on this node, it always starts it with qemu-system-x86_64 rather than qemu-system-arm, qemu-system-ppc, etc. If I manually set the correct binary, everything works as expected. Am I missing something here, or is this a bug in nova-compute? image_properties_default_architecture is only used in the scheduler filter to pick a compute host, it doesn't do anything about the qemu binary used in nova-compute. mnaser added the config option so maybe he can share what he's done on his computes. Do you have qemu-system-x86_64 on non-x86 systems? Seems like a package/deploy issue since I'd expect x86 packages shouldn't install on a ppc system and vice versa, and only one qemu package should provide the binary. -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
