[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Executive summary for kernel team: What makes both libvirt and Nova unhappy about the Cavium Thunder X NIC is the fact that they are denied with "Operation not supported" when attempting to read from sysfs node phys_port_id from its virtual functions. Example:

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

** Changed in: charm-nova-compute Assignee: (unassigned) => Frode Nordahl (fnordahl) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node device not found: no node

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

1) The 'No compute node record for host phanpy: ComputeHostNotFound_Remote: Compute host phanpy could not be found.' message is benign, this message appears on first start of the `nova- compute` service. It keeps appearing in the log here due to failure to register available resources. See 3)

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

** Attachment added: "libvirt-debug.log" https://bugs.launchpad.net/charm-nova-compute/+bug/1771662/+attachment/5157735/+files/libvirt-debug.log ** Changed in: charm-nova-compute Status: Incomplete => Invalid ** Also affects: nova Importance: Undecided Status: New -- You

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

To be clear, on our lab machines (gigabyte arm64), we don't observe this issue with Bionic + Queens, hence the request to try to triage on the specific kit involved. Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Incomplete in libvirt pending debug from live system by openstack team. ** Changed in: libvirt (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title:

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Escalated due to delay in triage and fix given our contract with ARM -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node device not found: no node device with matching

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

@raharper - I concur, there is a workflow gap in the nova-compute charm with regard to hypervisor registration success with nova, and I've raised a separate bug to address that generically. However, that won't fix this bug, it will just make it more visible by blocking the juju charm unit and

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

In order to make progress from the charm front, I would need access to at least one machine with the hardware which is specific to this bug, plus two adjacent machines for control/data plane. Can we arrange that access for openstack charms engineering? ** Changed in: charm-nova-compute

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

I'm not certain we can rule out the charm; the observant behavior is that the compute nodes do not get enrolled. Certainly the lack of a nova-compute node being registered has some touch point to the charms. The follow-up I think comes from the Openstack team to walk through where the charm leaves

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

This defect seems to have stalled somewhat. Is there more information we can gather for this to move forward again? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

If this is a bug on the OpenStack side, it's not in the charm. It would be in nova proper. ** Changed in: charm-nova-compute Status: New => Opinion -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

After comparing the sysfs data, I don't see any differences w.r.t the physical paths in sysfs for the thunder nic. I wonder if there is something that detects "xenial" and does one thing, vs "bionic" despite the xenial host using the same kernel level. The apparmor denied on the namespaces only

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

ls -alR /sys on bionic http://paste.ubuntu.com/p/nrxyRGP3By/ The bionic kernel has also bumped: Linux aurorus 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:14:36 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux On Tue, May 22, 2018 at 7:10 PM, Ryan Harper <1771...@bugs.launchpad.net> wrote: > Looks

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Looks like the ls -aLR contains more data; we can compare bionic. On Tue, May 22, 2018 at 6:53 PM, Jason Hobbs wrote: > cd /sys/bus/pci/devices && grep -nr . * > > xenial: > http://paste.ubuntu.com/p/F5qyvN2Qrr/ > > On Tue, May 22, 2018 at 5:27 PM, Jason Hobbs

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

cd /sys/bus/pci/devices && grep -nr . * xenial: http://paste.ubuntu.com/p/F5qyvN2Qrr/ On Tue, May 22, 2018 at 5:27 PM, Jason Hobbs wrote: > Do you really want a tar? How about ls -alR? xenial: > > http://paste.ubuntu.com/p/wyQ3kTsyBB/ > > On Tue, May 22, 2018 at 5:14

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Do you really want a tar? How about ls -alR? xenial: http://paste.ubuntu.com/p/wyQ3kTsyBB/ On Tue, May 22, 2018 at 5:14 PM, Jason Hobbs wrote: > ok; looks like that 4.15.0-22-generic just released and wasn't what I > used in the first reproduction... I doubt that's

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

ok; looks like that 4.15.0-22-generic just released and wasn't what I used in the first reproduction... I doubt that's it. On Tue, May 22, 2018 at 4:58 PM, Ryan Harper <1771...@bugs.launchpad.net> wrote: > Comparing the kernel logs, on Xenial, the second nic comes up: > > May 22 15:00:27 aurorus

Re: [Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Comparing the kernel logs, on Xenial, the second nic comes up: May 22 15:00:27 aurorus kernel: [ 24.840500] IPv6: ADDRCONF(NETDEV_UP): enP2p1s0f2: link is not ready May 22 15:00:27 aurorus kernel: [ 25.472391] thunder-nicvf 0002:01:00.2 enP2p1s0f2: Link is Up 1 Mbps Full duplex But on

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

marked new on nova-compute-charm due to rharper's comment #18, and new on libvirt because I've posted all the requested logs now. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title:

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

@rharper, here are the logs you requested from the xenial deploy. ** Attachment added: "xenial-logs-1771662.tgz" https://bugs.launchpad.net/charm-nova-compute/+bug/1771662/+attachment/5142976/+files/xenial-logs-1771662.tgz ** Changed in: charm-nova-compute Status: Invalid => New **

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Christian, thanks for digging in. Yes, I really just setup base openstack and hit this condition. I'm not doing anything to setup devices as passthrough or anything along those lines, and I'm not trying to start instances. -- You received this bug notification because you are a member of Ubuntu

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Newly deployed Cavium System with 18.04 to get my own view onto this (without openstack/charms in the way) 1. start a basic guest $ sudo apt install uvtool-libvirt qemu-efi-aarch64 $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=arm64

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Thanks for the logs. I generally don't see anything *fatal* to libvirt. In the nova logs, I can see that virsh capabilities returns host information. It certainly is failing to find the VFs on the SRIOV device; it's not clear if that's because the device is misbehaving (we can see the kernel

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

all of /var/log and /etc from the bionic deploy. ** Attachment added: "bionic-var-log-and-etc.tgz" https://bugs.launchpad.net/charm-nova-compute/+bug/1771662/+attachment/5141000/+files/bionic-var-log-and-etc.tgz -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

@rharper here are the logs you asked for from the bionic deploy ** Attachment added: "bionic-logs.tgz" https://bugs.launchpad.net/charm-nova-compute/+bug/1771662/+attachment/5140998/+files/bionic-logs.tgz -- You received this bug notification because you are a member of Ubuntu Bugs, which

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Some package level deltas that may be relevant: ii linux-firmware 1.173 ii linux-firmware 1.157.18 ii pciutils 1:3.3.1-1.1ubuntu1.2 ii pciutils 1:3.5.2-1ubuntu libvirt0:arm644.0.0-1ubuntu7~cloud0 libvirt0:arm644.0.0-1ubuntu8

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

@rharper still working on getting the other stuff you've asked for, but here is the uname -a output from xenial vs bionic: http://paste.ubuntu.com/p/rJDpK5SyW9/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

To make it more clear; the hardware SRIOV device is different that normal: TL;DR this special device has VFs that have NO PF associated software doesn't understand this Though per comment #3; it seems odd that a Xenial/Queens with the same kernel (HWE) works OK. So some tracing in libvirt/nova

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

And for the xenial deployment version, can we get what's in /etc/network/interfaces* (including the .d)? I'm generally curious w.r.t what interfaces are managed by the OS, and which ones are being delegated to the guests. -- You received this bug notification because you are a member of Ubuntu

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Please capture: 1) cloud-init collect-logs (writes cloud-init.tar to $CWD) 2) the journal /var/log/journal 3) /etc/netplan and /run/systemd 4) /etc/udev/rules.d -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

steve captured what I meant in #8 better than I did: 17:46 < slangasek> one could as accurately say "I'm suspicious this is related to us replacing the whole networking stack in Ubuntu" ;-) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

> I'm suspicious of netplan here. netplan is only the messenger here, between cloud-init+juju and networkd. Can you show the complete netplan yaml as it's been laid down on the system in question? -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

This looks like it is specific to this hardware and the way it does VFs and PFs, so I'm removing field-high. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node device

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

given it works with the same libvirt and kernel on 16.04 but not 18.04, I'm suspicious of netplan here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node device not

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

The deploy works fine with juju 2.4 beta 2 and xenial/queens. package versions: http://paste.ubuntu.com/p/PF7Jb7gxnX/ we do see this in nova-compute.log, but it's not fatal: http://paste.ubuntu.com/p/Dh4ZGVTtH8/ -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

Further information: Using juju 2.4 beta2 I was able to deploy magpie on bionic in lxd and baremetal via MAAS. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node device

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

We think this is an issue in libvirt, related to how it handles the sriov hardware in these machines. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node device not

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

** Changed in: charm-nova-compute Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1771662 Title: libvirtError: Node device not found: no node device with matching name

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

** Description changed: After deploying openstack on arm64 using bionic and queens, no hypervisors show upon. On my compute nodes, I have an error like: 2018-05-16 19:23:08.165 282170 ERROR nova.compute.manager libvirtError: Node device not found: no node device with matching name

[Bug 1771662] Re: libvirtError: Node device not found: no node device with matching name

What puzzles me is Xenial-Queens working and Bionic showing issues. Because it seems like libvirt being unable to cope with this type of HW, but since it works in one but not the other ... Yet versions are: - xenial-queens libvirt 4.0.0-1ubuntu7~cloud0 qemu 1:2.11+dfsg-1ubuntu7~cloud0 -