[Yahoo-eng-team] [Bug 1919357] Re: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option
** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1919357 Title: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: Fix Released Status in OpenStack Compute (nova) ussuri series: Fix Released Status in OpenStack Compute (nova) victoria series: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: In Progress Bug description: - [x] This doc is inaccurate in this way: __ I followed the guide to setup qemu native tls for live migration. After checking, that libvirt is able to use tls using tcpdump to listen on the port for tls, I also wanted to check that it works when I live migrate an instance. Apparently it didn't. But it used the port for unencrypted TCP [1]. After digging through documentation and code afterwards I found that in this code part: https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120 @staticmethod def _live_migration_uri(dest): uris = { 'kvm': 'qemu+%(scheme)s://%(dest)s/system', 'qemu': 'qemu+%(scheme)s://%(dest)s/system', 'xen': 'xenmigr://%(dest)s/system', 'parallels': 'parallels+tcp://%(dest)s/system', } dest = oslo_netutils.escape_ipv6(dest) virt_type = CONF.libvirt.virt_type # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike uri = CONF.libvirt.live_migration_uri if uri: return uri % dest uri = uris.get(virt_type) if uri is None: raise exception.LiveMigrationURINotAvailable(virt_type=virt_type) str_format = { 'dest': dest, 'scheme': CONF.libvirt.live_migration_scheme or 'tcp', } return uri % str_format the uri is calculated using the config parameter 'live_migration_scheme' or using the hard coded tcp parameter. Coming from the guide for qemu native tls, there was no hint that this config option needs to be set. In fact without setting this 'live_migration_scheme' config option to tls, there is no way to see, that the live migration still uses the unencrypted tcp connection - one has to use tcpdump and listen for tcp or tls to recognize it. Neither in the logs nor in any debug output there is any hint that it is still unencrypted! Thus I conclude there might be OpenStack deployments which are configured as the guide say but these config changes have no effect! - [x] This is a doc addition request. To fix this the config parameter 'live_migration_scheme' should be set to tls and maybe there should be a warning in the documentation, that without doing this, the traffic is still unencrypted. - [ ] I have a fix to the document that I can paste below including example: input and output. [1] without setting 'live_migration_scheme' in the nova.conf $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184 0x: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..pZ 0x0020: 8018 01f6 2669 0101 080a 1948 0297 0x0030: 6fc6 f589 1828 2000 8086 0001 o..( 0x0040: 012f 0009 .../ 0x0050: 0001 000f 6465 7374 696e 6174 destinat 0x0060: 696f 6e5f 786d 6c00 0007 129b ion_xml. 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 ...inst 0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-2e99...7e2 0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45- 0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d 0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8...> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206 0x:
[Yahoo-eng-team] [Bug 1919357] Re: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option
** Changed in: nova/ussuri Status: New => Fix Released ** Changed in: nova/victoria Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1919357 Title: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: Fix Released Status in OpenStack Compute (nova) ussuri series: Fix Released Status in OpenStack Compute (nova) victoria series: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: In Progress Bug description: - [x] This doc is inaccurate in this way: __ I followed the guide to setup qemu native tls for live migration. After checking, that libvirt is able to use tls using tcpdump to listen on the port for tls, I also wanted to check that it works when I live migrate an instance. Apparently it didn't. But it used the port for unencrypted TCP [1]. After digging through documentation and code afterwards I found that in this code part: https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120 @staticmethod def _live_migration_uri(dest): uris = { 'kvm': 'qemu+%(scheme)s://%(dest)s/system', 'qemu': 'qemu+%(scheme)s://%(dest)s/system', 'xen': 'xenmigr://%(dest)s/system', 'parallels': 'parallels+tcp://%(dest)s/system', } dest = oslo_netutils.escape_ipv6(dest) virt_type = CONF.libvirt.virt_type # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike uri = CONF.libvirt.live_migration_uri if uri: return uri % dest uri = uris.get(virt_type) if uri is None: raise exception.LiveMigrationURINotAvailable(virt_type=virt_type) str_format = { 'dest': dest, 'scheme': CONF.libvirt.live_migration_scheme or 'tcp', } return uri % str_format the uri is calculated using the config parameter 'live_migration_scheme' or using the hard coded tcp parameter. Coming from the guide for qemu native tls, there was no hint that this config option needs to be set. In fact without setting this 'live_migration_scheme' config option to tls, there is no way to see, that the live migration still uses the unencrypted tcp connection - one has to use tcpdump and listen for tcp or tls to recognize it. Neither in the logs nor in any debug output there is any hint that it is still unencrypted! Thus I conclude there might be OpenStack deployments which are configured as the guide say but these config changes have no effect! - [x] This is a doc addition request. To fix this the config parameter 'live_migration_scheme' should be set to tls and maybe there should be a warning in the documentation, that without doing this, the traffic is still unencrypted. - [ ] I have a fix to the document that I can paste below including example: input and output. [1] without setting 'live_migration_scheme' in the nova.conf $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184 0x: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..pZ 0x0020: 8018 01f6 2669 0101 080a 1948 0297 0x0030: 6fc6 f589 1828 2000 8086 0001 o..( 0x0040: 012f 0009 .../ 0x0050: 0001 000f 6465 7374 696e 6174 destinat 0x0060: 696f 6e5f 786d 6c00 0007 129b ion_xml. 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 ...inst 0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-2e99...7e2 0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45- 0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d 0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8...> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options
[Yahoo-eng-team] [Bug 1919357] Re: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option
** Changed in: nova/train Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1919357 Title: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: Fix Released Status in OpenStack Compute (nova) ussuri series: New Status in OpenStack Compute (nova) victoria series: New Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: In Progress Bug description: - [x] This doc is inaccurate in this way: __ I followed the guide to setup qemu native tls for live migration. After checking, that libvirt is able to use tls using tcpdump to listen on the port for tls, I also wanted to check that it works when I live migrate an instance. Apparently it didn't. But it used the port for unencrypted TCP [1]. After digging through documentation and code afterwards I found that in this code part: https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120 @staticmethod def _live_migration_uri(dest): uris = { 'kvm': 'qemu+%(scheme)s://%(dest)s/system', 'qemu': 'qemu+%(scheme)s://%(dest)s/system', 'xen': 'xenmigr://%(dest)s/system', 'parallels': 'parallels+tcp://%(dest)s/system', } dest = oslo_netutils.escape_ipv6(dest) virt_type = CONF.libvirt.virt_type # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike uri = CONF.libvirt.live_migration_uri if uri: return uri % dest uri = uris.get(virt_type) if uri is None: raise exception.LiveMigrationURINotAvailable(virt_type=virt_type) str_format = { 'dest': dest, 'scheme': CONF.libvirt.live_migration_scheme or 'tcp', } return uri % str_format the uri is calculated using the config parameter 'live_migration_scheme' or using the hard coded tcp parameter. Coming from the guide for qemu native tls, there was no hint that this config option needs to be set. In fact without setting this 'live_migration_scheme' config option to tls, there is no way to see, that the live migration still uses the unencrypted tcp connection - one has to use tcpdump and listen for tcp or tls to recognize it. Neither in the logs nor in any debug output there is any hint that it is still unencrypted! Thus I conclude there might be OpenStack deployments which are configured as the guide say but these config changes have no effect! - [x] This is a doc addition request. To fix this the config parameter 'live_migration_scheme' should be set to tls and maybe there should be a warning in the documentation, that without doing this, the traffic is still unencrypted. - [ ] I have a fix to the document that I can paste below including example: input and output. [1] without setting 'live_migration_scheme' in the nova.conf $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184 0x: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..pZ 0x0020: 8018 01f6 2669 0101 080a 1948 0297 0x0030: 6fc6 f589 1828 2000 8086 0001 o..( 0x0040: 012f 0009 .../ 0x0050: 0001 000f 6465 7374 696e 6174 destinat 0x0060: 696f 6e5f 786d 6c00 0007 129b ion_xml. 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 ...inst 0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-2e99...7e2 0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45- 0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d 0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8...> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206 0x: 4500 0102 a605 4000
[Yahoo-eng-team] [Bug 1919357] Re: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option
** Changed in: nova Status: New => In Progress ** Changed in: nova Assignee: (unassigned) => Josephine Seifert (josei) ** Changed in: nova Importance: Undecided => High ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/victoria Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/ussuri Importance: Undecided Status: New ** Changed in: nova/stein Importance: Undecided => High ** Changed in: nova/train Importance: Undecided => High ** Changed in: nova/ussuri Importance: Undecided => High ** Changed in: nova/victoria Importance: Undecided => High ** Tags added: tls ** Tags added: live-migration -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1919357 Title: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Status in OpenStack Compute (nova) ussuri series: New Status in OpenStack Compute (nova) victoria series: New Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: - [x] This doc is inaccurate in this way: __ I followed the guide to setup qemu native tls for live migration. After checking, that libvirt is able to use tls using tcpdump to listen on the port for tls, I also wanted to check that it works when I live migrate an instance. Apparently it didn't. But it used the port for unencrypted TCP [1]. After digging through documentation and code afterwards I found that in this code part: https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120 @staticmethod def _live_migration_uri(dest): uris = { 'kvm': 'qemu+%(scheme)s://%(dest)s/system', 'qemu': 'qemu+%(scheme)s://%(dest)s/system', 'xen': 'xenmigr://%(dest)s/system', 'parallels': 'parallels+tcp://%(dest)s/system', } dest = oslo_netutils.escape_ipv6(dest) virt_type = CONF.libvirt.virt_type # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike uri = CONF.libvirt.live_migration_uri if uri: return uri % dest uri = uris.get(virt_type) if uri is None: raise exception.LiveMigrationURINotAvailable(virt_type=virt_type) str_format = { 'dest': dest, 'scheme': CONF.libvirt.live_migration_scheme or 'tcp', } return uri % str_format the uri is calculated using the config parameter 'live_migration_scheme' or using the hard coded tcp parameter. Coming from the guide for qemu native tls, there was no hint that this config option needs to be set. In fact without setting this 'live_migration_scheme' config option to tls, there is no way to see, that the live migration still uses the unencrypted tcp connection - one has to use tcpdump and listen for tcp or tls to recognize it. Neither in the logs nor in any debug output there is any hint that it is still unencrypted! Thus I conclude there might be OpenStack deployments which are configured as the guide say but these config changes have no effect! - [x] This is a doc addition request. To fix this the config parameter 'live_migration_scheme' should be set to tls and maybe there should be a warning in the documentation, that without doing this, the traffic is still unencrypted. - [ ] I have a fix to the document that I can paste below including example: input and output. [1] without setting 'live_migration_scheme' in the nova.conf $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184 0x: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..pZ 0x0020: 8018 01f6 2669 0101 080a 1948 0297 0x0030: 6fc6 f589 1828 2000 8086 0001 o..( 0x0040: 012f 0009 .../ 0x0050: 0001 000f 6465 7374 696e 6174 destinat 0x0060: 696f 6e5f 786d 6c00 0007 129b ion_xml. 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 ...inst 0x0090:
[Yahoo-eng-team] [Bug 1919357] Re: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option
We discussed this at some length earlier today in the #openstack-nova IRC channel: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack- nova.2021-03-16.log.html#t2021-03-16T14:31:57 Given that the information is already public, and the bug in this case is incomplete documentation (it fails to mention one of the two configuration options necessary to enable TLS), there's no need for us to keep an embargo on the report. Per the VMT's taxonomy, this is a class D report (misleading documentation), but may warrant publication of a security note: https://security.openstack.org/vmt-process.html#incident-report-taxonomy https://wiki.openstack.org/wiki/Security/Security_Note_Process ** Information type changed from Private Security to Public ** Also affects: ossa Importance: Undecided Status: New ** Changed in: ossa Status: New => Won't Fix ** Also affects: ossn Importance: Undecided Status: New ** Tags added: security -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1919357 Title: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option Status in OpenStack Compute (nova): New Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: - [x] This doc is inaccurate in this way: __ I followed the guide to setup qemu native tls for live migration. After checking, that libvirt is able to use tls using tcpdump to listen on the port for tls, I also wanted to check that it works when I live migrate an instance. Apparently it didn't. But it used the port for unencrypted TCP [1]. After digging through documentation and code afterwards I found that in this code part: https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120 @staticmethod def _live_migration_uri(dest): uris = { 'kvm': 'qemu+%(scheme)s://%(dest)s/system', 'qemu': 'qemu+%(scheme)s://%(dest)s/system', 'xen': 'xenmigr://%(dest)s/system', 'parallels': 'parallels+tcp://%(dest)s/system', } dest = oslo_netutils.escape_ipv6(dest) virt_type = CONF.libvirt.virt_type # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike uri = CONF.libvirt.live_migration_uri if uri: return uri % dest uri = uris.get(virt_type) if uri is None: raise exception.LiveMigrationURINotAvailable(virt_type=virt_type) str_format = { 'dest': dest, 'scheme': CONF.libvirt.live_migration_scheme or 'tcp', } return uri % str_format the uri is calculated using the config parameter 'live_migration_scheme' or using the hard coded tcp parameter. Coming from the guide for qemu native tls, there was no hint that this config option needs to be set. In fact without setting this 'live_migration_scheme' config option to tls, there is no way to see, that the live migration still uses the unencrypted tcp connection - one has to use tcpdump and listen for tcp or tls to recognize it. Neither in the logs nor in any debug output there is any hint that it is still unencrypted! Thus I conclude there might be OpenStack deployments which are configured as the guide say but these config changes have no effect! - [x] This is a doc addition request. To fix this the config parameter 'live_migration_scheme' should be set to tls and maybe there should be a warning in the documentation, that without doing this, the traffic is still unencrypted. - [ ] I have a fix to the document that I can paste below including example: input and output. [1] without setting 'live_migration_scheme' in the nova.conf $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184 0x: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..pZ 0x0020: 8018 01f6 2669 0101 080a 1948 0297 0x0030: 6fc6 f589 1828 2000 8086 0001 o..( 0x0040: 012f 0009 .../ 0x0050: 0001 000f 6465 7374 696e 6174 destinat 0x0060: 696f 6e5f 786d 6c00 0007 129b ion_xml. 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 ...inst 0x0090: 616e 6365 2d30 3030 3032 6539