Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
Malini -Original Message- From: Poulos, Brianna L. [mailto:brianna.pou...@jhuapl.edu] Sent: Wednesday, September 09, 2015 9:54 AM To: OpenStack Development Mailing List (not for usage questions) Cc: stuart.mcla...@hp.com Subject: Re: [openstack-dev] [glance] [nova] Verification of glance images before boot Stuart is right about what will currently happen in Nova when an image is downloaded, which protects against unintentional modifications to the image data. What is currently being worked on is adding the ability to verify a signature of the checksum. The flow of this is as follows: 1. The user creates a signature of the "checksum hash" (currently MD5) of the image data offline. 2. The user uploads a public key certificate, which can be used to verify the signature to a key manager (currently Barbican). 3. The user creates an image in glance, with signature metadata properties. 4. The user uploads the image data to glance. 5. If the signature metadata properties exist, glance verifies the signature of the "checksum hash", including retrieving the certificate >from the key manager. 6. If the signature verification fails, glance moves the image to a killed state, and returns an error message to the user. 7. If the signature verification succeeds, a log message indicates that it succeeded, and the image upload finishes successfully. 8. Nova requests the image from glance, along with the image properties, in order to boot it. 9. Nova uses the signature metadata properties to verify the signature (if a configuration option is set). 10. If the signature verification fails, nova does not boot the image, but errors out. 11. If the signature verification succeeds, nova boots the image, and a log message notes that the verification succeeded. Regarding what is currently in Liberty, the blueprint mentioned [1] has merged, and code [2] has also been merged in glance, which handles steps 1-7 of the flow above. For steps 7-11, there is currently a nova blueprint [3], along with code [4], which are proposed for Mitaka. Note that we are in the process of adding official documentation, with examples of creating the signature as well as the properties that need to be added for the image before upload. In the meantime, there's an etherpad that describes how to test the signature verification functionality in Glance [5]. Also note that this is the initial approach, and there are some limitations. For example, ideally the signature would be based on a cryptographically secure (i.e. not MD5) hash of the image. There is a spec in glance to allow this hash to be configurable [6]. [1] https://blueprints.launchpad.net/glance/+spec/ image-signing-and-verificati o n-support [2] https://github.com/openstack/glance/commit/ 484ef1b40b738c87adb203bba6107dd b 4b04ff6e [3] https://review.openstack.org/#/c/188874/ [4] https://review.openstack.org/#/c/189843/ [5] https://etherpad.openstack.org/p/ liberty-glance-image-signing-instructions [6] https://review.openstack.org/#/c/191542/ Thanks, ~Brianna On 9/9/15, 12:16 , "Nikhil Komawar" wrote: That's correct. The size and the checksum are to be verified outside of Glance, in this case Nova. However, you may want to note that it's not necessary that all Nova virt drivers would use py-glanceclient so you would want to check the download specific code in the virt driver your Nova deployment is using.
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
On 9/11/15 10:40 AM, Flavio Percoco wrote: > On 11/09/15 07:58 -0400, Nikhil Komawar wrote: >> You are right in the sense that's the ideal scenario. >> >> (Impl-wise) However, even today we do not guarantee that behavior. If >> someone >> were to propose a new driver or a change driver capability or any >> thing of such >> order, images in status killed won't be guaranteed to have removed >> the garbage >> data. The driver may not choose to be resilient enough or would not >> take the >> responsibility of data removal synchronously on failures. > > I think it's glance's responsibility to make sure the driver deletes > the image data. If the API is not strong enough to guarantee this, > then we should change that. > Sounds like a good direction but I still think we cannot commit to it. It really depends on the driver and introduction of new or upgrades to the existing drivers (if not their capabilities) is a not a good idea. This will result in backward incompatibility in many cases and there can be enough corner cases when we think about v1, v2 and tasks to address them at Glance API level. I personally prefer the image-data delete to be asynchronous and fault tolerant (on DELETE of image record). However, even that cannot match the performance of a scrubber like service. So, in certain cases we need to advice running the scrubber if the data has the tendency to be left behind. >> Taking that fact in account, I have thought of Brianna's patch to be >> okay. > > Oh sure, I'm not trying to say it was a wrong choice. Sorry if it > sounded like that. I was replying to the thought of extending scrubber > (unless there's a patch that does this that I might have missed). > > Cheers, > Flavio > >> >> On 9/11/15 4:42 AM, Flavio Percoco wrote: >> >>On 10/09/15 15:36 -0400, Nikhil Komawar wrote: >> >>The solution to this problem is to improve the scrubber to >> clean up the >>garbage data left behind in the backend store during such failed >>uploads. >> >>Currently, scrubber cleans up images in pending_delete and >> extending >>that to images in killed status would avoid such a situation. >> >> >>While the above would certainly help, I think it's not the right >>solution. Images in status "killed" should not have data to begin >>with. >> >>I'd rather find a way to clean that data as soon as the image is >>moved to a "killed" state instead of extending the scrubber. >> >>Cheers, >>Flavio >> >> >>On 9/10/15 3:28 PM, Poulos, Brianna L. wrote: >> >>Malini, >> >>Thank you for bringing up the ³killed² state as it >> relates to >>quota. We >>opted to move the image to a killed state since that is >> what occurs >>when >>an upload fails, and the signature verification failure >> would occur >>during >>an upload. But we should keep in mind the potential to >> take up >>space and >>yet not take up quota when signature verification fails. >> >>Regarding the MD5 hash, there is currently a glance spec >> [1] to >>allow the >>hash method used for the checksum to be >> configurable‹currently it >>is >>hardcoded in glance. After making it configurable, the >> default >>would >>transition from MD5 to something more secure (like SHA-256). >> >>[1] https://review.openstack.org/#/c/191542/ >> >>Thanks, >>~Brianna >> >> >> >> >>On 9/10/15, 5:10 , "Bhandaru, Malini K" >> >>wrote: >> >> >>Brianna, I can imagine a denial of service attack by >> uploading >>images >>whose signature is invalid if we allow them to reside >> in Glance >>In a "killed" state. This would be less of an issue >> "killed" >> images still >> consume storage quota until actually deleted. >>Also given MD-5 less secure, why not have the default >> hash be >>SHA-1 or 2? >>Regards >>Malini >> >>-Original Message- >>
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
On 11/09/15 07:58 -0400, Nikhil Komawar wrote:
You are right in the sense that's the ideal scenario.
(Impl-wise) However, even today we do not guarantee that behavior. If someone
were to propose a new driver or a change driver capability or any thing of such
order, images in status killed won't be guaranteed to have removed the garbage
data. The driver may not choose to be resilient enough or would not take the
responsibility of data removal synchronously on failures.
I think it's glance's responsibility to make sure the driver deletes
the image data. If the API is not strong enough to guarantee this,
then we should change that.
Taking that fact in account, I have thought of Brianna's patch to be okay.
Oh sure, I'm not trying to say it was a wrong choice. Sorry if it
sounded like that. I was replying to the thought of extending scrubber
(unless there's a patch that does this that I might have missed).
Cheers,
Flavio
On 9/11/15 4:42 AM, Flavio Percoco wrote:
On 10/09/15 15:36 -0400, Nikhil Komawar wrote:
The solution to this problem is to improve the scrubber to clean up the
garbage data left behind in the backend store during such failed
uploads.
Currently, scrubber cleans up images in pending_delete and extending
that to images in killed status would avoid such a situation.
While the above would certainly help, I think it's not the right
solution. Images in status "killed" should not have data to begin
with.
I'd rather find a way to clean that data as soon as the image is
moved to a "killed" state instead of extending the scrubber.
Cheers,
Flavio
On 9/10/15 3:28 PM, Poulos, Brianna L. wrote:
Malini,
Thank you for bringing up the ³killed² state as it relates to
quota. We
opted to move the image to a killed state since that is what occurs
when
an upload fails, and the signature verification failure would occur
during
an upload. But we should keep in mind the potential to take up
space and
yet not take up quota when signature verification fails.
Regarding the MD5 hash, there is currently a glance spec [1] to
allow the
hash method used for the checksum to be configurable‹currently it
is
hardcoded in glance. After making it configurable, the default
would
transition from MD5 to something more secure (like SHA-256).
[1] https://review.openstack.org/#/c/191542/
Thanks,
~Brianna
On 9/10/15, 5:10 , "Bhandaru, Malini K"
wrote:
Brianna, I can imagine a denial of service attack by uploading
images
whose signature is invalid if we allow them to reside in Glance
In a "killed" state. This would be less of an issue "killed"
images still
consume storage quota until actually deleted.
Also given MD-5 less secure, why not have the default hash be
SHA-1 or 2?
Regards
Malini
-Original Message-
From: Poulos, Brianna L. [mailto:brianna.pou...@jhuapl.edu]
Sent: Wednesday, September 09, 2015 9:54 AM
To: OpenStack Development Mailing List (not for usage
questions)
Cc: stuart.mcla...@hp.com
Subject: Re: [openstack-dev] [glance] [nova] Verification of
glance
images before boot
Stuart is right about what will currently happen in Nova when
an image is
downloaded, which protects against unintentional modifications
to the
image data.
What is currently being worked on is adding the ability to
verify a
signature of the checksum. The flow of this is as follows:
1. The user creates a signature of the "checksum hash"
(currently MD5) of
the image data offline.
2. The user uploads a public key certificate, which can be used
to verify
the signature to a key manager (currently Barbican).
3. The user creates an image in glance, with signature metadata
properties.
4. The user uploads the image data to glance.
5. If the signature metadata properties exist, glance verifies
the
signature of the "checksum hash", including retrieving the
certificate
>from the key manager.
6. If the signature verification fails, glance moves the image
to a
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
You are right in the sense that's the ideal scenario. (Impl-wise) However, even today we do not guarantee that behavior. If someone were to propose a new driver or a change driver capability or any thing of such order, images in status killed won't be guaranteed to have removed the garbage data. The driver may not choose to be resilient enough or would not take the responsibility of data removal synchronously on failures. Taking that fact in account, I have thought of Brianna's patch to be okay. On 9/11/15 4:42 AM, Flavio Percoco wrote: > On 10/09/15 15:36 -0400, Nikhil Komawar wrote: >> The solution to this problem is to improve the scrubber to clean up the >> garbage data left behind in the backend store during such failed >> uploads. >> >> Currently, scrubber cleans up images in pending_delete and extending >> that to images in killed status would avoid such a situation. > > While the above would certainly help, I think it's not the right > solution. Images in status "killed" should not have data to begin > with. > > I'd rather find a way to clean that data as soon as the image is > moved to a "killed" state instead of extending the scrubber. > > Cheers, > Flavio > >> On 9/10/15 3:28 PM, Poulos, Brianna L. wrote: >>> Malini, >>> >>> Thank you for bringing up the ³killed² state as it relates to >>> quota. We >>> opted to move the image to a killed state since that is what occurs >>> when >>> an upload fails, and the signature verification failure would occur >>> during >>> an upload. But we should keep in mind the potential to take up >>> space and >>> yet not take up quota when signature verification fails. >>> >>> Regarding the MD5 hash, there is currently a glance spec [1] to >>> allow the >>> hash method used for the checksum to be configurable‹currently it is >>> hardcoded in glance. After making it configurable, the default would >>> transition from MD5 to something more secure (like SHA-256). >>> >>> [1] https://review.openstack.org/#/c/191542/ >>> >>> Thanks, >>> ~Brianna >>> >>> >>> >>> >>> On 9/10/15, 5:10 , "Bhandaru, Malini K" >>> wrote: >>> >>>> Brianna, I can imagine a denial of service attack by uploading images >>>> whose signature is invalid if we allow them to reside in Glance >>>> In a "killed" state. This would be less of an issue "killed" images >>>> still >>>> consume storage quota until actually deleted. >>>> Also given MD-5 less secure, why not have the default hash be SHA-1 >>>> or 2? >>>> Regards >>>> Malini >>>> >>>> -Original Message- >>>> From: Poulos, Brianna L. [mailto:brianna.pou...@jhuapl.edu] >>>> Sent: Wednesday, September 09, 2015 9:54 AM >>>> To: OpenStack Development Mailing List (not for usage questions) >>>> Cc: stuart.mcla...@hp.com >>>> Subject: Re: [openstack-dev] [glance] [nova] Verification of glance >>>> images before boot >>>> >>>> Stuart is right about what will currently happen in Nova when an >>>> image is >>>> downloaded, which protects against unintentional modifications to the >>>> image data. >>>> >>>> What is currently being worked on is adding the ability to verify a >>>> signature of the checksum. The flow of this is as follows: >>>> 1. The user creates a signature of the "checksum hash" (currently >>>> MD5) of >>>> the image data offline. >>>> 2. The user uploads a public key certificate, which can be used to >>>> verify >>>> the signature to a key manager (currently Barbican). >>>> 3. The user creates an image in glance, with signature metadata >>>> properties. >>>> 4. The user uploads the image data to glance. >>>> 5. If the signature metadata properties exist, glance verifies the >>>> signature of the "checksum hash", including retrieving the certificate >>> >from the key manager. >>>> 6. If the signature verification fails, glance moves the image to a >>>> killed state, and returns an error message to the user. >>>> 7. If the signature verification succeeds, a log message indicates >>>> that >>>> it succeeded, and the image upload finishes successfully. >>>> >>>> 8. Nova requests the
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
On 10/09/15 15:36 -0400, Nikhil Komawar wrote: The solution to this problem is to improve the scrubber to clean up the garbage data left behind in the backend store during such failed uploads. Currently, scrubber cleans up images in pending_delete and extending that to images in killed status would avoid such a situation. While the above would certainly help, I think it's not the right solution. Images in status "killed" should not have data to begin with. I'd rather find a way to clean that data as soon as the image is moved to a "killed" state instead of extending the scrubber. Cheers, Flavio On 9/10/15 3:28 PM, Poulos, Brianna L. wrote: Malini, Thank you for bringing up the ³killed² state as it relates to quota. We opted to move the image to a killed state since that is what occurs when an upload fails, and the signature verification failure would occur during an upload. But we should keep in mind the potential to take up space and yet not take up quota when signature verification fails. Regarding the MD5 hash, there is currently a glance spec [1] to allow the hash method used for the checksum to be configurable‹currently it is hardcoded in glance. After making it configurable, the default would transition from MD5 to something more secure (like SHA-256). [1] https://review.openstack.org/#/c/191542/ Thanks, ~Brianna On 9/10/15, 5:10 , "Bhandaru, Malini K" wrote: Brianna, I can imagine a denial of service attack by uploading images whose signature is invalid if we allow them to reside in Glance In a "killed" state. This would be less of an issue "killed" images still consume storage quota until actually deleted. Also given MD-5 less secure, why not have the default hash be SHA-1 or 2? Regards Malini -Original Message- From: Poulos, Brianna L. [mailto:brianna.pou...@jhuapl.edu] Sent: Wednesday, September 09, 2015 9:54 AM To: OpenStack Development Mailing List (not for usage questions) Cc: stuart.mcla...@hp.com Subject: Re: [openstack-dev] [glance] [nova] Verification of glance images before boot Stuart is right about what will currently happen in Nova when an image is downloaded, which protects against unintentional modifications to the image data. What is currently being worked on is adding the ability to verify a signature of the checksum. The flow of this is as follows: 1. The user creates a signature of the "checksum hash" (currently MD5) of the image data offline. 2. The user uploads a public key certificate, which can be used to verify the signature to a key manager (currently Barbican). 3. The user creates an image in glance, with signature metadata properties. 4. The user uploads the image data to glance. 5. If the signature metadata properties exist, glance verifies the signature of the "checksum hash", including retrieving the certificate >from the key manager. 6. If the signature verification fails, glance moves the image to a killed state, and returns an error message to the user. 7. If the signature verification succeeds, a log message indicates that it succeeded, and the image upload finishes successfully. 8. Nova requests the image from glance, along with the image properties, in order to boot it. 9. Nova uses the signature metadata properties to verify the signature (if a configuration option is set). 10. If the signature verification fails, nova does not boot the image, but errors out. 11. If the signature verification succeeds, nova boots the image, and a log message notes that the verification succeeded. Regarding what is currently in Liberty, the blueprint mentioned [1] has merged, and code [2] has also been merged in glance, which handles steps 1-7 of the flow above. For steps 7-11, there is currently a nova blueprint [3], along with code [4], which are proposed for Mitaka. Note that we are in the process of adding official documentation, with examples of creating the signature as well as the properties that need to be added for the image before upload. In the meantime, there's an etherpad that describes how to test the signature verification functionality in Glance [5]. Also note that this is the initial approach, and there are some limitations. For example, ideally the signature would be based on a cryptographically secure (i.e. not MD5) hash of the image. There is a spec in glance to allow this hash to be configurable [6]. [1] https://blueprints.launchpad.net/glance/+spec/image-signing-and-verificati o n-support [2] https://github.com/openstack/glance/commit/484ef1b40b738c87adb203bba6107dd b 4b04ff6e [3] https://review.openstack.org/#/c/188874/ [4] https://review.openstack.org/#/c/189843/ [5] https://etherpad.openstack.org/p/liberty-glance-image-signing-instructions [6] https://review.openstack.org/#/c/191542/ Thanks, ~Brianna On 9/9/15, 12:16 , "Nikhil Komawar" wrote: That's correct. The size and the checksum are to be ve
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
The solution to this problem is to improve the scrubber to clean up the garbage data left behind in the backend store during such failed uploads. Currently, scrubber cleans up images in pending_delete and extending that to images in killed status would avoid such a situation. On 9/10/15 3:28 PM, Poulos, Brianna L. wrote: > Malini, > > Thank you for bringing up the ³killed² state as it relates to quota. We > opted to move the image to a killed state since that is what occurs when > an upload fails, and the signature verification failure would occur during > an upload. But we should keep in mind the potential to take up space and > yet not take up quota when signature verification fails. > > Regarding the MD5 hash, there is currently a glance spec [1] to allow the > hash method used for the checksum to be configurable‹currently it is > hardcoded in glance. After making it configurable, the default would > transition from MD5 to something more secure (like SHA-256). > > [1] https://review.openstack.org/#/c/191542/ > > Thanks, > ~Brianna > > > > > On 9/10/15, 5:10 , "Bhandaru, Malini K" > wrote: > >> Brianna, I can imagine a denial of service attack by uploading images >> whose signature is invalid if we allow them to reside in Glance >> In a "killed" state. This would be less of an issue "killed" images still >> consume storage quota until actually deleted. >> Also given MD-5 less secure, why not have the default hash be SHA-1 or 2? >> Regards >> Malini >> >> -Original Message- >> From: Poulos, Brianna L. [mailto:brianna.pou...@jhuapl.edu] >> Sent: Wednesday, September 09, 2015 9:54 AM >> To: OpenStack Development Mailing List (not for usage questions) >> Cc: stuart.mcla...@hp.com >> Subject: Re: [openstack-dev] [glance] [nova] Verification of glance >> images before boot >> >> Stuart is right about what will currently happen in Nova when an image is >> downloaded, which protects against unintentional modifications to the >> image data. >> >> What is currently being worked on is adding the ability to verify a >> signature of the checksum. The flow of this is as follows: >> 1. The user creates a signature of the "checksum hash" (currently MD5) of >> the image data offline. >> 2. The user uploads a public key certificate, which can be used to verify >> the signature to a key manager (currently Barbican). >> 3. The user creates an image in glance, with signature metadata >> properties. >> 4. The user uploads the image data to glance. >> 5. If the signature metadata properties exist, glance verifies the >> signature of the "checksum hash", including retrieving the certificate > >from the key manager. >> 6. If the signature verification fails, glance moves the image to a >> killed state, and returns an error message to the user. >> 7. If the signature verification succeeds, a log message indicates that >> it succeeded, and the image upload finishes successfully. >> >> 8. Nova requests the image from glance, along with the image properties, >> in order to boot it. >> 9. Nova uses the signature metadata properties to verify the signature >> (if a configuration option is set). >> 10. If the signature verification fails, nova does not boot the image, >> but errors out. >> 11. If the signature verification succeeds, nova boots the image, and a >> log message notes that the verification succeeded. >> >> Regarding what is currently in Liberty, the blueprint mentioned [1] has >> merged, and code [2] has also been merged in glance, which handles steps >> 1-7 of the flow above. >> >> For steps 7-11, there is currently a nova blueprint [3], along with code >> [4], which are proposed for Mitaka. >> >> Note that we are in the process of adding official documentation, with >> examples of creating the signature as well as the properties that need to >> be added for the image before upload. In the meantime, there's an >> etherpad that describes how to test the signature verification >> functionality in Glance [5]. >> >> Also note that this is the initial approach, and there are some >> limitations. For example, ideally the signature would be based on a >> cryptographically secure (i.e. not MD5) hash of the image. There is a >> spec in glance to allow this hash to be configurable [6]. >> >> [1] >> https://blueprints.launchpad.net/glance/+spec/image-signing-and-verificati >> o >> n-support >> [2] >> https://github.com/openstack/glance/commit/484ef1b40b738c87adb203bba6107dd &g
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
Malini, Thank you for bringing up the ³killed² state as it relates to quota. We opted to move the image to a killed state since that is what occurs when an upload fails, and the signature verification failure would occur during an upload. But we should keep in mind the potential to take up space and yet not take up quota when signature verification fails. Regarding the MD5 hash, there is currently a glance spec [1] to allow the hash method used for the checksum to be configurable‹currently it is hardcoded in glance. After making it configurable, the default would transition from MD5 to something more secure (like SHA-256). [1] https://review.openstack.org/#/c/191542/ Thanks, ~Brianna On 9/10/15, 5:10 , "Bhandaru, Malini K" wrote: >Brianna, I can imagine a denial of service attack by uploading images >whose signature is invalid if we allow them to reside in Glance >In a "killed" state. This would be less of an issue "killed" images still >consume storage quota until actually deleted. >Also given MD-5 less secure, why not have the default hash be SHA-1 or 2? >Regards >Malini > >-Original Message- >From: Poulos, Brianna L. [mailto:brianna.pou...@jhuapl.edu] >Sent: Wednesday, September 09, 2015 9:54 AM >To: OpenStack Development Mailing List (not for usage questions) >Cc: stuart.mcla...@hp.com >Subject: Re: [openstack-dev] [glance] [nova] Verification of glance >images before boot > >Stuart is right about what will currently happen in Nova when an image is >downloaded, which protects against unintentional modifications to the >image data. > >What is currently being worked on is adding the ability to verify a >signature of the checksum. The flow of this is as follows: >1. The user creates a signature of the "checksum hash" (currently MD5) of >the image data offline. >2. The user uploads a public key certificate, which can be used to verify >the signature to a key manager (currently Barbican). >3. The user creates an image in glance, with signature metadata >properties. >4. The user uploads the image data to glance. >5. If the signature metadata properties exist, glance verifies the >signature of the "checksum hash", including retrieving the certificate >from the key manager. >6. If the signature verification fails, glance moves the image to a >killed state, and returns an error message to the user. >7. If the signature verification succeeds, a log message indicates that >it succeeded, and the image upload finishes successfully. > >8. Nova requests the image from glance, along with the image properties, >in order to boot it. >9. Nova uses the signature metadata properties to verify the signature >(if a configuration option is set). >10. If the signature verification fails, nova does not boot the image, >but errors out. >11. If the signature verification succeeds, nova boots the image, and a >log message notes that the verification succeeded. > >Regarding what is currently in Liberty, the blueprint mentioned [1] has >merged, and code [2] has also been merged in glance, which handles steps >1-7 of the flow above. > >For steps 7-11, there is currently a nova blueprint [3], along with code >[4], which are proposed for Mitaka. > >Note that we are in the process of adding official documentation, with >examples of creating the signature as well as the properties that need to >be added for the image before upload. In the meantime, there's an >etherpad that describes how to test the signature verification >functionality in Glance [5]. > >Also note that this is the initial approach, and there are some >limitations. For example, ideally the signature would be based on a >cryptographically secure (i.e. not MD5) hash of the image. There is a >spec in glance to allow this hash to be configurable [6]. > >[1] >https://blueprints.launchpad.net/glance/+spec/image-signing-and-verificati >o >n-support >[2] >https://github.com/openstack/glance/commit/484ef1b40b738c87adb203bba6107dd >b >4b04ff6e >[3] https://review.openstack.org/#/c/188874/ >[4] https://review.openstack.org/#/c/189843/ >[5] >https://etherpad.openstack.org/p/liberty-glance-image-signing-instructions >[6] https://review.openstack.org/#/c/191542/ > > >Thanks, >~Brianna > > > > >On 9/9/15, 12:16 , "Nikhil Komawar" wrote: > >>That's correct. >> >>The size and the checksum are to be verified outside of Glance, in this >>case Nova. However, you may want to note that it's not necessary that >>all Nova virt drivers would use py-glanceclient so you would want to >>check the download specific code in the virt driver your Nova >>deployment is using. >> >>Having said tha
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
Brianna, I can imagine a denial of service attack by uploading images whose signature is invalid if we allow them to reside in Glance In a "killed" state. This would be less of an issue "killed" images still consume storage quota until actually deleted. Also given MD-5 less secure, why not have the default hash be SHA-1 or 2? Regards Malini -Original Message- From: Poulos, Brianna L. [mailto:brianna.pou...@jhuapl.edu] Sent: Wednesday, September 09, 2015 9:54 AM To: OpenStack Development Mailing List (not for usage questions) Cc: stuart.mcla...@hp.com Subject: Re: [openstack-dev] [glance] [nova] Verification of glance images before boot Stuart is right about what will currently happen in Nova when an image is downloaded, which protects against unintentional modifications to the image data. What is currently being worked on is adding the ability to verify a signature of the checksum. The flow of this is as follows: 1. The user creates a signature of the "checksum hash" (currently MD5) of the image data offline. 2. The user uploads a public key certificate, which can be used to verify the signature to a key manager (currently Barbican). 3. The user creates an image in glance, with signature metadata properties. 4. The user uploads the image data to glance. 5. If the signature metadata properties exist, glance verifies the signature of the "checksum hash", including retrieving the certificate from the key manager. 6. If the signature verification fails, glance moves the image to a killed state, and returns an error message to the user. 7. If the signature verification succeeds, a log message indicates that it succeeded, and the image upload finishes successfully. 8. Nova requests the image from glance, along with the image properties, in order to boot it. 9. Nova uses the signature metadata properties to verify the signature (if a configuration option is set). 10. If the signature verification fails, nova does not boot the image, but errors out. 11. If the signature verification succeeds, nova boots the image, and a log message notes that the verification succeeded. Regarding what is currently in Liberty, the blueprint mentioned [1] has merged, and code [2] has also been merged in glance, which handles steps 1-7 of the flow above. For steps 7-11, there is currently a nova blueprint [3], along with code [4], which are proposed for Mitaka. Note that we are in the process of adding official documentation, with examples of creating the signature as well as the properties that need to be added for the image before upload. In the meantime, there's an etherpad that describes how to test the signature verification functionality in Glance [5]. Also note that this is the initial approach, and there are some limitations. For example, ideally the signature would be based on a cryptographically secure (i.e. not MD5) hash of the image. There is a spec in glance to allow this hash to be configurable [6]. [1] https://blueprints.launchpad.net/glance/+spec/image-signing-and-verificatio n-support [2] https://github.com/openstack/glance/commit/484ef1b40b738c87adb203bba6107ddb 4b04ff6e [3] https://review.openstack.org/#/c/188874/ [4] https://review.openstack.org/#/c/189843/ [5] https://etherpad.openstack.org/p/liberty-glance-image-signing-instructions [6] https://review.openstack.org/#/c/191542/ Thanks, ~Brianna On 9/9/15, 12:16 , "Nikhil Komawar" wrote: >That's correct. > >The size and the checksum are to be verified outside of Glance, in this >case Nova. However, you may want to note that it's not necessary that >all Nova virt drivers would use py-glanceclient so you would want to >check the download specific code in the virt driver your Nova >deployment is using. > >Having said that, essentially the flow seems appropriate. Error must be >raise on mismatch. > >The signing BP was to help prevent the compromised Glance from changing >the checksum and image blob at the same time. Using a digital >signature, you can prevent download of compromised data. However, the >feature has just been implemented in Glance; Glance users may take time to >adopt. > > > >On 9/9/15 11:15 AM, stuart.mcla...@hp.com wrote: >> >> The glance client (running 'inside' the Nova server) will >> re-calculate the checksum as it downloads the image and then compare >> it against the expected value. If they don't match an error will be raised. >> >>> How can I know that the image that a new instance is spawned from - >>> is actually the image that was originally registered in glance - and >>> has not been maliciously tampered with in some way? >>> >>> Is there some kind of verification that is performed against the >>> md5sum of the registered image in glance before a new i
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
On 09/09/2015 10:53 AM, Poulos, Brianna L. wrote: Stuart is right about what will currently happen in Nova when an image is downloaded, which protects against unintentional modifications to the image data. What is currently being worked on is adding the ability to verify a signature of the checksum. It should be noted that this does not protect against a compromised compute node. For an end-user that cares about this case, I think you'd pretty much need self-checking within the guest to ensure that its running system matches a downloaded manifest (or something like that). Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
Stuart is right about what will currently happen in Nova when an image is downloaded, which protects against unintentional modifications to the image data. What is currently being worked on is adding the ability to verify a signature of the checksum. The flow of this is as follows: 1. The user creates a signature of the "checksum hash" (currently MD5) of the image data offline. 2. The user uploads a public key certificate, which can be used to verify the signature to a key manager (currently Barbican). 3. The user creates an image in glance, with signature metadata properties. 4. The user uploads the image data to glance. 5. If the signature metadata properties exist, glance verifies the signature of the "checksum hash", including retrieving the certificate from the key manager. 6. If the signature verification fails, glance moves the image to a killed state, and returns an error message to the user. 7. If the signature verification succeeds, a log message indicates that it succeeded, and the image upload finishes successfully. 8. Nova requests the image from glance, along with the image properties, in order to boot it. 9. Nova uses the signature metadata properties to verify the signature (if a configuration option is set). 10. If the signature verification fails, nova does not boot the image, but errors out. 11. If the signature verification succeeds, nova boots the image, and a log message notes that the verification succeeded. Regarding what is currently in Liberty, the blueprint mentioned [1] has merged, and code [2] has also been merged in glance, which handles steps 1-7 of the flow above. For steps 7-11, there is currently a nova blueprint [3], along with code [4], which are proposed for Mitaka. Note that we are in the process of adding official documentation, with examples of creating the signature as well as the properties that need to be added for the image before upload. In the meantime, there's an etherpad that describes how to test the signature verification functionality in Glance [5]. Also note that this is the initial approach, and there are some limitations. For example, ideally the signature would be based on a cryptographically secure (i.e. not MD5) hash of the image. There is a spec in glance to allow this hash to be configurable [6]. [1] https://blueprints.launchpad.net/glance/+spec/image-signing-and-verificatio n-support [2] https://github.com/openstack/glance/commit/484ef1b40b738c87adb203bba6107ddb 4b04ff6e [3] https://review.openstack.org/#/c/188874/ [4] https://review.openstack.org/#/c/189843/ [5] https://etherpad.openstack.org/p/liberty-glance-image-signing-instructions [6] https://review.openstack.org/#/c/191542/ Thanks, ~Brianna On 9/9/15, 12:16 , "Nikhil Komawar" wrote: >That's correct. > >The size and the checksum are to be verified outside of Glance, in this >case Nova. However, you may want to note that it's not necessary that >all Nova virt drivers would use py-glanceclient so you would want to >check the download specific code in the virt driver your Nova deployment >is using. > >Having said that, essentially the flow seems appropriate. Error must be >raise on mismatch. > >The signing BP was to help prevent the compromised Glance from changing >the checksum and image blob at the same time. Using a digital signature, >you can prevent download of compromised data. However, the feature has >just been implemented in Glance; Glance users may take time to adopt. > > > >On 9/9/15 11:15 AM, stuart.mcla...@hp.com wrote: >> >> The glance client (running 'inside' the Nova server) will re-calculate >> the checksum as it downloads the image and then compare it against the >> expected value. If they don't match an error will be raised. >> >>> How can I know that the image that a new instance is spawned from - is >>> actually the image that was originally registered in glance - and has >>> not been maliciously tampered with in some way? >>> >>> Is there some kind of verification that is performed against the md5sum >>> of the registered image in glance before a new instance is spawned? >>> >>> Is that done by Nova? >>> Glance? >>> Both? Neither? >>> >>> The reason I ask is some 'paranoid' security (that is their job I >>> suppose) people have raised these questions. >>> >>> I know there is a glance BP already merged for L [1] - but I would like >>> to understand the actual flow in a bit more detail. >>> >>> Thanks. >>> >>> [1] >>> >>>https://blueprints.launchpad.net/glance/+spec/image-signing-and-verifica >>>tion-support >>> >>> >>> -- >>> Best Regards, >>> Maish Saidel-Keesing >>> >>> >>> >>> -- >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> End of OpenStack-dev Digest, Vol 41, Issue 22 >>> * >>> >> >> >>__
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
That's correct. The size and the checksum are to be verified outside of Glance, in this case Nova. However, you may want to note that it's not necessary that all Nova virt drivers would use py-glanceclient so you would want to check the download specific code in the virt driver your Nova deployment is using. Having said that, essentially the flow seems appropriate. Error must be raise on mismatch. The signing BP was to help prevent the compromised Glance from changing the checksum and image blob at the same time. Using a digital signature, you can prevent download of compromised data. However, the feature has just been implemented in Glance; Glance users may take time to adopt. On 9/9/15 11:15 AM, stuart.mcla...@hp.com wrote: > > The glance client (running 'inside' the Nova server) will re-calculate > the checksum as it downloads the image and then compare it against the > expected value. If they don't match an error will be raised. > >> How can I know that the image that a new instance is spawned from - is >> actually the image that was originally registered in glance - and has >> not been maliciously tampered with in some way? >> >> Is there some kind of verification that is performed against the md5sum >> of the registered image in glance before a new instance is spawned? >> >> Is that done by Nova? >> Glance? >> Both? Neither? >> >> The reason I ask is some 'paranoid' security (that is their job I >> suppose) people have raised these questions. >> >> I know there is a glance BP already merged for L [1] - but I would like >> to understand the actual flow in a bit more detail. >> >> Thanks. >> >> [1] >> https://blueprints.launchpad.net/glance/+spec/image-signing-and-verification-support >> >> >> -- >> Best Regards, >> Maish Saidel-Keesing >> >> >> >> -- >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> End of OpenStack-dev Digest, Vol 41, Issue 22 >> * >> > > __ > > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Nikhil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] [nova] Verification of glance images before boot
The glance client (running 'inside' the Nova server) will re-calculate the checksum as it downloads the image and then compare it against the expected value. If they don't match an error will be raised. How can I know that the image that a new instance is spawned from - is actually the image that was originally registered in glance - and has not been maliciously tampered with in some way? Is there some kind of verification that is performed against the md5sum of the registered image in glance before a new instance is spawned? Is that done by Nova? Glance? Both? Neither? The reason I ask is some 'paranoid' security (that is their job I suppose) people have raised these questions. I know there is a glance BP already merged for L [1] - but I would like to understand the actual flow in a bit more detail. Thanks. [1] https://blueprints.launchpad.net/glance/+spec/image-signing-and-verification-support -- Best Regards, Maish Saidel-Keesing -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev End of OpenStack-dev Digest, Vol 41, Issue 22 * __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [glance] [nova] Verification of glance images before boot
How can I know that the image that a new instance is spawned from - is actually the image that was originally registered in glance - and has not been maliciously tampered with in some way? Is there some kind of verification that is performed against the md5sum of the registered image in glance before a new instance is spawned? Is that done by Nova? Glance? Both? Neither? The reason I ask is some 'paranoid' security (that is their job I suppose) people have raised these questions. I know there is a glance BP already merged for L [1] - but I would like to understand the actual flow in a bit more detail. Thanks. [1] https://blueprints.launchpad.net/glance/+spec/image-signing-and-verification-support -- Best Regards, Maish Saidel-Keesing __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
