Re: [openstack-dev] [nova] Native QEMU LUKS decryption review overview ahead of FF

[Matthew Booth]

On 24 January 2018 at 22:57, Matt Riedemann  wrote:

> On 1/22/2018 8:22 AM, Lee Yarwood wrote:
>
>> Hello,
>>
>> With M3 and FF rapidly approaching this week I wanted to post a brief
>> overview of the QEMU native LUKS series.
>>
>> The full series is available on the following topic, I'll go into more
>> detail on each of the changes below:
>>
>> https://review.openstack.org/#/q/topic:bp/libvirt-qemu-nativ
>> e-luks+status:open
>>
>> libvirt: Collocate encryptor and volume driver calls
>> https://review.openstack.org/#/c/460243/ (Missing final +2 and +W)
>>
>> This refactor of the Libvirt driver connect and disconnect volume code
>> has the added benefit of also correcting a number of bugs around the
>> attaching and detaching of os-brick encryptors. IMHO this would be
>> useful in Queens even if the rest of the series doesn't land.
>>
>> libvirt: Introduce disk encryption config classes
>> https://review.openstack.org/#/c/464008/ (Missing final +2 and +W)
>>
>> This is the most straight forward change of the series and simply
>> introduces the required config classes to wire up native LUKS decryption
>> within the domain XML of an instance. Hopefully nothing controversial.
>>
>> libvirt: QEMU native LUKS decryption for encrypted volumes
>> https://review.openstack.org/#/c/523958/ (Missing both +2s and +W)
>>
>> This change carries the bulk of the implementation, wiring up encrypted
>> volumes during their initial attachment. The commit message has a
>> detailed run down of the various upgrade and LM corner cases we attempt
>> to handle here, such as LM from a P to Q compute, detaching a P attached
>> encrypted volume after upgrading to Q etc.
>>
>> Upgrade and LM testing is enabled by the following changes:
>>
>> fixed_key: Use a single hardcoded key across devstack deployments
>> https://review.openstack.org/#/c/536343/
>>
>> compute: Introduce an encrypted volume LM test
>> https://review.openstack.org/#/c/536177/
>>
>> This is being tested by tempest-dsvm-multinode-live-migration and
>> grenade-dsvm-neutron-multinode-live-migration in the following DNM Nova
>> change, enabling volume backed LM tests:
>>
>> DNM: Test LM with encrypted volumes
>> https://review.openstack.org/#/c/536350/
>>
>> Hopefully that covers everything but please feel free to ping if you
>> would like more detail, background etc. Thanks in advance,
>>
>> Lee
>>
>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> The patch is already approved, and I asked melwitt to write a release
> note, at which point it was noted that swap volume will not work with
> native luks encrypted volumes. That's a regression.
>

It's only a regression since swap_volume with encrypted volumes was fixed
in https://review.openstack.org/#/c/460243/, which landed on Monday as part
of this series. Prior to Monday, swap_volume with encrypted volumes would
result in the raw encrypted volume being presented to the guest after the
swap.

We need to at least report a nova bug for this so we can work on some kind
> of fallback to the non-native decryption until there is a libvirt/qemu fix
> upstream and we can put version conditionals in place for when we can
> support swap volume with a native luks-encrypted volume.
>

In the context of the above, I don't think this is a priority as clearly
nobody is currently doing it. There's already a bug to track the problem in
libvirt, which is linked in a code comment. Admittedly that BZ is
unnecessarily private, which I noted in review, but we've reached out to
the author to ask them to open it up as there's nothing sensitive going on
in there.

In general, anything qemu can do natively makes Nova both simpler and more
robust because we don't have to modify the host configuration. This
eliminates a whole class of error states and race conditions, because when
we kill qemu there's nothing left to clean up.

Matt
-- 
Matthew Booth
Red Hat OpenStack Engineer, Compute DFG

Phone: +442070094448 (UK)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Native QEMU LUKS decryption review overview ahead of FF

[Matt Riedemann]

On 1/22/2018 8:22 AM, Lee Yarwood wrote:

Hello,

With M3 and FF rapidly approaching this week I wanted to post a brief
overview of the QEMU native LUKS series.

The full series is available on the following topic, I'll go into more
detail on each of the changes below:

https://review.openstack.org/#/q/topic:bp/libvirt-qemu-native-luks+status:open

libvirt: Collocate encryptor and volume driver calls
https://review.openstack.org/#/c/460243/ (Missing final +2 and +W)

This refactor of the Libvirt driver connect and disconnect volume code
has the added benefit of also correcting a number of bugs around the
attaching and detaching of os-brick encryptors. IMHO this would be
useful in Queens even if the rest of the series doesn't land.

libvirt: Introduce disk encryption config classes
https://review.openstack.org/#/c/464008/ (Missing final +2 and +W)

This is the most straight forward change of the series and simply
introduces the required config classes to wire up native LUKS decryption
within the domain XML of an instance. Hopefully nothing controversial.

libvirt: QEMU native LUKS decryption for encrypted volumes
https://review.openstack.org/#/c/523958/ (Missing both +2s and +W)

This change carries the bulk of the implementation, wiring up encrypted
volumes during their initial attachment. The commit message has a
detailed run down of the various upgrade and LM corner cases we attempt
to handle here, such as LM from a P to Q compute, detaching a P attached
encrypted volume after upgrading to Q etc.

Upgrade and LM testing is enabled by the following changes:

fixed_key: Use a single hardcoded key across devstack deployments
https://review.openstack.org/#/c/536343/

compute: Introduce an encrypted volume LM test
https://review.openstack.org/#/c/536177/

This is being tested by tempest-dsvm-multinode-live-migration and
grenade-dsvm-neutron-multinode-live-migration in the following DNM Nova
change, enabling volume backed LM tests:

DNM: Test LM with encrypted volumes
https://review.openstack.org/#/c/536350/

Hopefully that covers everything but please feel free to ping if you
would like more detail, background etc. Thanks in advance,

Lee



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



The patch is already approved, and I asked melwitt to write a release 
note, at which point it was noted that swap volume will not work with 
native luks encrypted volumes. That's a regression.


We need to at least report a nova bug for this so we can work on some 
kind of fallback to the non-native decryption until there is a 
libvirt/qemu fix upstream and we can put version conditionals in place 
for when we can support swap volume with a native luks-encrypted volume.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Native QEMU LUKS decryption review overview ahead of FF

[Lee Yarwood]

On 23-01-18 13:44:49, Lee Yarwood wrote:
> A breif progress update in-line below.
> 
> On 22-01-18 14:22:12, Lee Yarwood wrote:
> > Hello,
> > 
> > With M3 and FF rapidly approaching this week I wanted to post a brief
> > overview of the QEMU native LUKS series.
> > 
> > The full series is available on the following topic, I'll go into more
> > detail on each of the changes below:
> > 
> > https://review.openstack.org/#/q/topic:bp/libvirt-qemu-native-luks+status:open
> > 
> > libvirt: Collocate encryptor and volume driver calls
> > https://review.openstack.org/#/c/460243/ (Missing final +2 and +W)
> > 
> > This refactor of the Libvirt driver connect and disconnect volume code
> > has the added benefit of also correcting a number of bugs around the
> > attaching and detaching of os-brick encryptors. IMHO this would be
> > useful in Queens even if the rest of the series doesn't land.
> > 
> > libvirt: Introduce disk encryption config classes
> > https://review.openstack.org/#/c/464008/ (Missing final +2 and +W)
> > 
> > This is the most straight forward change of the series and simply
> > introduces the required config classes to wire up native LUKS decryption
> > within the domain XML of an instance. Hopefully nothing controversial.
> 
> Both of these have landed, my thanks to jaypipes for his reviews!
> 
> > libvirt: QEMU native LUKS decryption for encrypted volumes
> > https://review.openstack.org/#/c/523958/ (Missing both +2s and +W)
> > 
> > This change carries the bulk of the implementation, wiring up encrypted
> > volumes during their initial attachment. The commit message has a
> > detailed run down of the various upgrade and LM corner cases we attempt
> > to handle here, such as LM from a P to Q compute, detaching a P attached
> > encrypted volume after upgrading to Q etc.
> 
> Thanks to melwitt and mdbooth for your reviews! I've respun to address
> the various nits and typos pointed out in this change. Ready and waiting
> to respin again if any others crop up.

My thanks again to melwitt for another review on this final patch. I'm
going to be offline for most of Thursday ahead of the FF deadline so if any
non-RH core reviewers are able to look at this today I'll do my best to
address any nits, concerns, facepalms etc ASAP. 

Cheers,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Native QEMU LUKS decryption review overview ahead of FF

[Lee Yarwood]

On 23-01-18 16:52:30, Corey Bryant wrote:
> On Tue, Jan 23, 2018 at 8:44 AM, Lee Yarwood  wrote:
>> grenade-dsvm-neutron-multinode-live-migration is currently failing due
>> to our use of the Ocata UCA on stable/pike leading to the following
>> issue with the libvirt 2.5.0 build it provides:
>>
>> libvirt 2.5.0-3ubuntu5.6~cloud0 appears to be compiled without gnutls
>> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1744758
>>
> Hey Lee,
> 
> We have a new version of libvirt in ocata-proposed now that should fix your
> issue and is ready for testing. Thanks for your work on this and for
> opening the bug.
 
Thanks Corey, as reported in the bug this WORKSFORME. Thanks for the
quick turn around with this, it's really appreciated!
 
>> I've cherry-picked the following devstack change back to stable/pike and
>> pulled it into the test change above for Nova, hopefully working around
>> these failures:
>>
>> Update to using pike cloud-archive
>> https://review.openstack.org/#/c/536798/

FWIW I still think we should enable the Pike UCA for our stable/pike
jobs. As noted in the stable review, testing the Ocata UCA with
stable/pike strikes me as pointless as no one will ever use that
combination of UCA and stable/pike bits in a real world deployment.

Cheers,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Native QEMU LUKS decryption review overview ahead of FF

[Corey Bryant]

On Tue, Jan 23, 2018 at 8:44 AM, Lee Yarwood  wrote:

> A breif progress update in-line below.
>
> On 22-01-18 14:22:12, Lee Yarwood wrote:
> > Hello,
> >
> > With M3 and FF rapidly approaching this week I wanted to post a brief
> > overview of the QEMU native LUKS series.
> >
> > The full series is available on the following topic, I'll go into more
> > detail on each of the changes below:
> >
> > https://review.openstack.org/#/q/topic:bp/libvirt-qemu-
> native-luks+status:open
> >
> > libvirt: Collocate encryptor and volume driver calls
> > https://review.openstack.org/#/c/460243/ (Missing final +2 and +W)
> >
> > This refactor of the Libvirt driver connect and disconnect volume code
> > has the added benefit of also correcting a number of bugs around the
> > attaching and detaching of os-brick encryptors. IMHO this would be
> > useful in Queens even if the rest of the series doesn't land.
> >
> > libvirt: Introduce disk encryption config classes
> > https://review.openstack.org/#/c/464008/ (Missing final +2 and +W)
> >
> > This is the most straight forward change of the series and simply
> > introduces the required config classes to wire up native LUKS decryption
> > within the domain XML of an instance. Hopefully nothing controversial.
>
> Both of these have landed, my thanks to jaypipes for his reviews!
>
> > libvirt: QEMU native LUKS decryption for encrypted volumes
> > https://review.openstack.org/#/c/523958/ (Missing both +2s and +W)
> >
> > This change carries the bulk of the implementation, wiring up encrypted
> > volumes during their initial attachment. The commit message has a
> > detailed run down of the various upgrade and LM corner cases we attempt
> > to handle here, such as LM from a P to Q compute, detaching a P attached
> > encrypted volume after upgrading to Q etc.
>
> Thanks to melwitt and mdbooth for your reviews! I've respun to address
> the various nits and typos pointed out in this change. Ready and waiting
> to respin again if any others crop up.
>
> > Upgrade and LM testing is enabled by the following changes:
> >
> > fixed_key: Use a single hardcoded key across devstack deployments
> > https://review.openstack.org/#/c/536343/
> >
> > compute: Introduce an encrypted volume LM test
> > https://review.openstack.org/#/c/536177/
> >
> > This is being tested by tempest-dsvm-multinode-live-migration and
> > grenade-dsvm-neutron-multinode-live-migration in the following DNM Nova
> > change, enabling volume backed LM tests:
> >
> > DNM: Test LM with encrypted volumes
> > https://review.openstack.org/#/c/536350/
> >
> > Hopefully that covers everything but please feel free to ping if you
> > would like more detail, background etc. Thanks in advance,
>
> grenade-dsvm-neutron-multinode-live-migration is currently failing due
> to our use of the Ocata UCA on stable/pike leading to the following
> issue with the libvirt 2.5.0 build it provides:
>
> libvirt 2.5.0-3ubuntu5.6~cloud0 appears to be compiled without gnutls
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1744758
>
>
Hey Lee,

We have a new version of libvirt in ocata-proposed now that should fix your
issue and is ready for testing. Thanks for your work on this and for
opening the bug.

Corey

I've cherry-picked the following devstack change back to stable/pike and
> pulled it into the test change above for Nova, hopefully working around
> these failures:
>
> Update to using pike cloud-archive
> https://review.openstack.org/#/c/536798/
>
> tempest-dsvm-multinode-live-migration is also failing but AFAICT they
> are unrelated to this overall series and appear to be more generic
> volume backed live migration failures.
>
> Thanks again!
>
> Lee
> --
> Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672
> 2D76
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Native QEMU LUKS decryption review overview ahead of FF

[Lee Yarwood]

A breif progress update in-line below.

On 22-01-18 14:22:12, Lee Yarwood wrote:
> Hello,
> 
> With M3 and FF rapidly approaching this week I wanted to post a brief
> overview of the QEMU native LUKS series.
> 
> The full series is available on the following topic, I'll go into more
> detail on each of the changes below:
> 
> https://review.openstack.org/#/q/topic:bp/libvirt-qemu-native-luks+status:open
> 
> libvirt: Collocate encryptor and volume driver calls
> https://review.openstack.org/#/c/460243/ (Missing final +2 and +W)
> 
> This refactor of the Libvirt driver connect and disconnect volume code
> has the added benefit of also correcting a number of bugs around the
> attaching and detaching of os-brick encryptors. IMHO this would be
> useful in Queens even if the rest of the series doesn't land.
> 
> libvirt: Introduce disk encryption config classes
> https://review.openstack.org/#/c/464008/ (Missing final +2 and +W)
> 
> This is the most straight forward change of the series and simply
> introduces the required config classes to wire up native LUKS decryption
> within the domain XML of an instance. Hopefully nothing controversial.

Both of these have landed, my thanks to jaypipes for his reviews!

> libvirt: QEMU native LUKS decryption for encrypted volumes
> https://review.openstack.org/#/c/523958/ (Missing both +2s and +W)
> 
> This change carries the bulk of the implementation, wiring up encrypted
> volumes during their initial attachment. The commit message has a
> detailed run down of the various upgrade and LM corner cases we attempt
> to handle here, such as LM from a P to Q compute, detaching a P attached
> encrypted volume after upgrading to Q etc.

Thanks to melwitt and mdbooth for your reviews! I've respun to address
the various nits and typos pointed out in this change. Ready and waiting
to respin again if any others crop up.
 
> Upgrade and LM testing is enabled by the following changes:
> 
> fixed_key: Use a single hardcoded key across devstack deployments
> https://review.openstack.org/#/c/536343/
> 
> compute: Introduce an encrypted volume LM test
> https://review.openstack.org/#/c/536177/
> 
> This is being tested by tempest-dsvm-multinode-live-migration and
> grenade-dsvm-neutron-multinode-live-migration in the following DNM Nova
> change, enabling volume backed LM tests:
> 
> DNM: Test LM with encrypted volumes
> https://review.openstack.org/#/c/536350/
> 
> Hopefully that covers everything but please feel free to ping if you
> would like more detail, background etc. Thanks in advance,

grenade-dsvm-neutron-multinode-live-migration is currently failing due
to our use of the Ocata UCA on stable/pike leading to the following
issue with the libvirt 2.5.0 build it provides:

libvirt 2.5.0-3ubuntu5.6~cloud0 appears to be compiled without gnutls
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1744758

I've cherry-picked the following devstack change back to stable/pike and
pulled it into the test change above for Nova, hopefully working around
these failures:

Update to using pike cloud-archive
https://review.openstack.org/#/c/536798/

tempest-dsvm-multinode-live-migration is also failing but AFAICT they
are unrelated to this overall series and appear to be more generic
volume backed live migration failures.

Thanks again!

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Native QEMU LUKS decryption review overview ahead of FF

[Lee Yarwood]

Hello,

With M3 and FF rapidly approaching this week I wanted to post a brief
overview of the QEMU native LUKS series.

The full series is available on the following topic, I'll go into more
detail on each of the changes below:

https://review.openstack.org/#/q/topic:bp/libvirt-qemu-native-luks+status:open

libvirt: Collocate encryptor and volume driver calls
https://review.openstack.org/#/c/460243/ (Missing final +2 and +W)

This refactor of the Libvirt driver connect and disconnect volume code
has the added benefit of also correcting a number of bugs around the
attaching and detaching of os-brick encryptors. IMHO this would be
useful in Queens even if the rest of the series doesn't land.

libvirt: Introduce disk encryption config classes
https://review.openstack.org/#/c/464008/ (Missing final +2 and +W)

This is the most straight forward change of the series and simply
introduces the required config classes to wire up native LUKS decryption
within the domain XML of an instance. Hopefully nothing controversial.

libvirt: QEMU native LUKS decryption for encrypted volumes
https://review.openstack.org/#/c/523958/ (Missing both +2s and +W)

This change carries the bulk of the implementation, wiring up encrypted
volumes during their initial attachment. The commit message has a
detailed run down of the various upgrade and LM corner cases we attempt
to handle here, such as LM from a P to Q compute, detaching a P attached
encrypted volume after upgrading to Q etc.

Upgrade and LM testing is enabled by the following changes:

fixed_key: Use a single hardcoded key across devstack deployments
https://review.openstack.org/#/c/536343/

compute: Introduce an encrypted volume LM test
https://review.openstack.org/#/c/536177/

This is being tested by tempest-dsvm-multinode-live-migration and
grenade-dsvm-neutron-multinode-live-migration in the following DNM Nova
change, enabling volume backed LM tests:

DNM: Test LM with encrypted volumes
https://review.openstack.org/#/c/536350/

Hopefully that covers everything but please feel free to ping if you
would like more detail, background etc. Thanks in advance,

Lee
-- 
Lee Yarwood A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev