Public bug reported: With libvirt 6.8.0 introduced virt-ssh-helper:
+ * remote: ``virt-ssh-helper`` replaces ``nc`` for SSH tunnelling + + Libvirt now provides a ``virt-ssh-helper`` binary on the server + side. The libvirt remote client will use this binary for setting + up an SSH tunnelled connection to hosts. If not present, it will + transparently fallback to the traditional ``nc`` tunnel. The new + binary makes it possible for libvirt to transparently connect + across hosts even if libvirt is built with a different installation + prefix on the client vs server. It also enables remote access to + the unprivileged per-user libvirt daemons(eg using a URI such as + ``qemu+ssh://hostname/session``. The only requirement is that + ``virt-ssh-helper`` is present in $PATH of the remote host. Libvirt first checks for the `virt-ssh-helper` binary, if it's not present, then it falls back to `nc`. The code where the 'nova-migration-wrapper' script looks for the "nc" binary is here[1] libvirt used to first check for `nc` (netcat). But these two libvirt commits[2][3] -- which are present in the libvirt build used in this bug -- have now changed it to first look for `virt-ssh-helper`, if it not available, then fall back to `nc` (see David's comment#18 for the debug logging). So this trips up the 'nova-migration-wrapper' Mar 08 16:52:39 overcloud-novacompute-1 nova_migration_wrapper[240622]: Denying connection='192.168.24.18 54668 192.168.24.9 2022' command=['sh', '-c', "'which", 'virt-ssh-helper', '1>/dev/null', '2>&1;', 'if', 'test', '$?', '=', '0;', 'then', '', '', '', '', 'virt- ssh-helper', "'qemu:///system';", 'else', '', '', '', 'if', "'nc'", '-q', '2>&1', '|', 'grep', '"requires', 'an', 'argument"', '>/dev/null', '2>&1;', 'then', 'ARG=-q0;else', "ARG=;fi;'nc'", '$ARG', '-U', '/var/run/libvirt/libvirt-sock;', "fi'"] A possible workaround is to force-use "netcat" (`nc`) by appending to the migration URI: "&proxy=netcat", so the `diff` of the URL: - qemu+ssh://[email protected]:2022/system?keyfile=/etc/nova/migration/identity + qemu+ssh://[email protected]:2022/system?keyfile=/etc/nova/migration/identity&proxy=netcat But longer term we want to allow the virt-ssh-helper, because that's needed to work properly with the split daemons as the socket path has changed [1] https://github.com/rdo-packages/nova-distgit/blob/rpm-master/nova- migration-wrapper#L32 [2] https://libvirt.org/git/?p=libvirt.git;a=commit;h=f8ec7c842d (rpc: use new virt-ssh-helper binary for remote tunnelling, 2020-07-08) [3] https://libvirt.org/git/?p=libvirt.git;a=commit;h=7d959c302d (rpc: Fix virt-ssh-helper detection, 2020-10-27) ** Affects: nova Importance: Undecided Status: New ** Affects: tripleo Importance: Undecided Assignee: Martin Schuppert (mschuppert) Status: In Progress ** Also affects: nova Importance: Undecided Status: New ** Changed in: tripleo Assignee: (unassigned) => Martin Schuppert (mschuppert) ** Changed in: tripleo Status: New => In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918250 Title: live migration is failing with libvirt >= 6.8.0 Status in OpenStack Compute (nova): New Status in tripleo: In Progress Bug description: With libvirt 6.8.0 introduced virt-ssh-helper: + * remote: ``virt-ssh-helper`` replaces ``nc`` for SSH tunnelling + + Libvirt now provides a ``virt-ssh-helper`` binary on the server + side. The libvirt remote client will use this binary for setting + up an SSH tunnelled connection to hosts. If not present, it will + transparently fallback to the traditional ``nc`` tunnel. The new + binary makes it possible for libvirt to transparently connect + across hosts even if libvirt is built with a different installation + prefix on the client vs server. It also enables remote access to + the unprivileged per-user libvirt daemons(eg using a URI such as + ``qemu+ssh://hostname/session``. The only requirement is that + ``virt-ssh-helper`` is present in $PATH of the remote host. Libvirt first checks for the `virt-ssh-helper` binary, if it's not present, then it falls back to `nc`. The code where the 'nova-migration-wrapper' script looks for the "nc" binary is here[1] libvirt used to first check for `nc` (netcat). But these two libvirt commits[2][3] -- which are present in the libvirt build used in this bug -- have now changed it to first look for `virt-ssh-helper`, if it not available, then fall back to `nc` (see David's comment#18 for the debug logging). So this trips up the 'nova-migration-wrapper' Mar 08 16:52:39 overcloud-novacompute-1 nova_migration_wrapper[240622]: Denying connection='192.168.24.18 54668 192.168.24.9 2022' command=['sh', '-c', "'which", 'virt-ssh- helper', '1>/dev/null', '2>&1;', 'if', 'test', '$?', '=', '0;', 'then', '', '', '', '', 'virt-ssh-helper', "'qemu:///system';", 'else', '', '', '', 'if', "'nc'", '-q', '2>&1', '|', 'grep', '"requires', 'an', 'argument"', '>/dev/null', '2>&1;', 'then', 'ARG=-q0;else', "ARG=;fi;'nc'", '$ARG', '-U', '/var/run/libvirt /libvirt-sock;', "fi'"] A possible workaround is to force-use "netcat" (`nc`) by appending to the migration URI: "&proxy=netcat", so the `diff` of the URL: - qemu+ssh://[email protected]:2022/system?keyfile=/etc/nova/migration/identity + qemu+ssh://[email protected]:2022/system?keyfile=/etc/nova/migration/identity&proxy=netcat But longer term we want to allow the virt-ssh-helper, because that's needed to work properly with the split daemons as the socket path has changed [1] https://github.com/rdo-packages/nova-distgit/blob/rpm-master/nova- migration-wrapper#L32 [2] https://libvirt.org/git/?p=libvirt.git;a=commit;h=f8ec7c842d (rpc: use new virt-ssh-helper binary for remote tunnelling, 2020-07-08) [3] https://libvirt.org/git/?p=libvirt.git;a=commit;h=7d959c302d (rpc: Fix virt-ssh-helper detection, 2020-10-27) To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918250/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
