Re: [openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
Thanks Matt, a very correct and precise assessment on the usage in Nova. I, however, wanted to point out one thing for the audience -- the v2 Image schemas are discoverable and assert the purpose of setting operator defined and in most cases operator used protected properties. This is when the api v2 was designed was considered, having a flat namespace with schematic validation on the properties that makes it richer experience for advanced usage. The downside, of course, if what one sees when having relatively straightforward calls that have similar assumptions with those of the v1 api. http://developer.openstack.org/api-ref-image-v2.html#image-schemas-v2 On 7/27/16 4:44 PM, Matt Riedemann wrote: > On 7/20/2016 7:01 PM, Nikhil Komawar wrote: > >> Thanks Matt. The bug looked very descriptive so, I commented my thoughts >> >> there https://bugs.launchpad.net/python-glanceclient/+bug/1596602 >> >> >> >> On 7/20/16 4:18 PM, Matt Riedemann wrote: >> >>> On 7/18/2016 4:41 AM, Erno Kuvaja wrote: >>> On Mon, Jul 18, 2016 at 5:19 AM, Nikhil Komawarwrote: > Thanks Matt. I've scheduled for a release of the client this week. > > > > On 7/16/16 4:09 AM, Matt Riedemann wrote: > >> This is more of a heads up than anything. >> >> >> >> Our internal CI is running Tempest with images that don't have >> >> kernel_id or ramdisk_id properties set. >> >> >> >> We're running from master so nova defaults to use_glance_v1=False. >> >> >> >> Because of this: >> >> >> >> https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 >> >> >> >> >> >> >> >> and this: >> >> >> >> https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 >> >> >> >> >> >> >> >> The snapshot image properties get kernel_id and ramdisk_id set to >> None >> >> since that's what the glance v2 schema requires. >> >> >> >> However, python-glanceclient has it's own outdated copy of the schema >> >> which doesn't allow null values for those properties, see bug: >> >> >> >> https://bugs.launchpad.net/python-glanceclient/+bug/1596602 >> >> >> >> We don't hit this in the community CI because the image that Tempest >> >> uses from devstack has the kernel_id and ramdisk_id properties set: >> >> >> >> http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 >> >> >> >> >> >> >> >> But for anyone else upgrading to Newton that has images without those >> >> properties set and doesn't have use_glance_v1=True in nova.conf is >> >> going to be broken. >> >> >> >> Since we really want to get people off glance v1 and move to >> >> deprecation in Ocata, we need to get this merged and released: >> >> >> >> https://review.openstack.org/#/c/335152/ >> >> >> >> And bump the minimum required python-glanceclient in >> >> global-requirements for Newton. >> >> >> >> I'm not really sure why python-glanceclient even has it's own copy of >> >> the image schema, that seems redundant and error prone given the >> >> glance API already validates that, but it's kind of beside the point >> >> right now. >> >> >> > > > -- > > > > Thanks, > > Nikhil > > > > > > __ > > > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Hi Matt, Something has broken on the way if that shipped schema actually affect your operation. That should be used _only_ when the client is called without connection to the Glance API (for example in a case you have no env set and you call `glance help`), this is btw the reason why we have that schema shipped with the client. As soon as you execute a call that actually interacts with Glance API we should be pulling the schema from there. So if this is not the case, we've broken somewhere on the way. I'll try to find time to have a look. - Erno __ OpenStack Development Mailing List (not for usage questions)
Re: [openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
On 7/20/2016 7:01 PM, Nikhil Komawar wrote: Thanks Matt. The bug looked very descriptive so, I commented my thoughts there https://bugs.launchpad.net/python-glanceclient/+bug/1596602 On 7/20/16 4:18 PM, Matt Riedemann wrote: On 7/18/2016 4:41 AM, Erno Kuvaja wrote: On Mon, Jul 18, 2016 at 5:19 AM, Nikhil Komawarwrote: Thanks Matt. I've scheduled for a release of the client this week. On 7/16/16 4:09 AM, Matt Riedemann wrote: This is more of a heads up than anything. Our internal CI is running Tempest with images that don't have kernel_id or ramdisk_id properties set. We're running from master so nova defaults to use_glance_v1=False. Because of this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 and this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 The snapshot image properties get kernel_id and ramdisk_id set to None since that's what the glance v2 schema requires. However, python-glanceclient has it's own outdated copy of the schema which doesn't allow null values for those properties, see bug: https://bugs.launchpad.net/python-glanceclient/+bug/1596602 We don't hit this in the community CI because the image that Tempest uses from devstack has the kernel_id and ramdisk_id properties set: http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 But for anyone else upgrading to Newton that has images without those properties set and doesn't have use_glance_v1=True in nova.conf is going to be broken. Since we really want to get people off glance v1 and move to deprecation in Ocata, we need to get this merged and released: https://review.openstack.org/#/c/335152/ And bump the minimum required python-glanceclient in global-requirements for Newton. I'm not really sure why python-glanceclient even has it's own copy of the image schema, that seems redundant and error prone given the glance API already validates that, but it's kind of beside the point right now. -- Thanks, Nikhil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Matt, Something has broken on the way if that shipped schema actually affect your operation. That should be used _only_ when the client is called without connection to the Glance API (for example in a case you have no env set and you call `glance help`), this is btw the reason why we have that schema shipped with the client. As soon as you execute a call that actually interacts with Glance API we should be pulling the schema from there. So if this is not the case, we've broken somewhere on the way. I'll try to find time to have a look. - Erno __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Yeah I guess the schema fix in glanceclient didn't resolve the issue, but I haven't had the time to dig into the bug since we're doing the midcycle this week. For anyone that cares, we figured out the problem. In our case, the issue was we didn't have /etc/glance/schema-image.json in the deployment. We're also using full disk images so no kernel_id or ramdisk_id set on the image (or the snapshot of the instance). devstack masks all of this because it sets up /etc/glance/schema-image.json and it uses a multipart uec cirros image by default, so kernel_id and ramdisk_id are set. The amount of schema configuration that is possible scares me, given the glance config option for allow_additional_image_properties (defaults to true, but if false, you can't even set kernel_id/ramdisk_id and snapshots would be broken with the libvirt driver and a uec image). And /etc/glance/schema-image.json is configuration, so it might not be there, or it might be different. Nova was passing on assumptions made with the defaults, and how devstack / our CI system works, but Nova as a client of glance isn't actually checking the image schema available when creating images, so we could potentially hit other issues in the future. Mike Fedosin has the simple fix for the existing bug though: https://review.openstack.org/#/c/347971 And it looks like the nova API schema prevents being able to create a snapshot with metadata that uses null values, so there is some additional protection in that case. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions)
Re: [openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
Thanks Matt. The bug looked very descriptive so, I commented my thoughts there https://bugs.launchpad.net/python-glanceclient/+bug/1596602 On 7/20/16 4:18 PM, Matt Riedemann wrote: > On 7/18/2016 4:41 AM, Erno Kuvaja wrote: >> On Mon, Jul 18, 2016 at 5:19 AM, Nikhil Komawar >>wrote: >>> Thanks Matt. I've scheduled for a release of the client this week. >>> >>> On 7/16/16 4:09 AM, Matt Riedemann wrote: This is more of a heads up than anything. Our internal CI is running Tempest with images that don't have kernel_id or ramdisk_id properties set. We're running from master so nova defaults to use_glance_v1=False. Because of this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 and this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 The snapshot image properties get kernel_id and ramdisk_id set to None since that's what the glance v2 schema requires. However, python-glanceclient has it's own outdated copy of the schema which doesn't allow null values for those properties, see bug: https://bugs.launchpad.net/python-glanceclient/+bug/1596602 We don't hit this in the community CI because the image that Tempest uses from devstack has the kernel_id and ramdisk_id properties set: http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 But for anyone else upgrading to Newton that has images without those properties set and doesn't have use_glance_v1=True in nova.conf is going to be broken. Since we really want to get people off glance v1 and move to deprecation in Ocata, we need to get this merged and released: https://review.openstack.org/#/c/335152/ And bump the minimum required python-glanceclient in global-requirements for Newton. I'm not really sure why python-glanceclient even has it's own copy of the image schema, that seems redundant and error prone given the glance API already validates that, but it's kind of beside the point right now. >>> >>> -- >>> >>> Thanks, >>> Nikhil >>> >>> >>> __ >>> >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> Hi Matt, >> >> Something has broken on the way if that shipped schema actually affect >> your operation. That should be used _only_ when the client is called >> without connection to the Glance API (for example in a case you have >> no env set and you call `glance help`), this is btw the reason why we >> have that schema shipped with the client. >> >> As soon as you execute a call that actually interacts with Glance API >> we should be pulling the schema from there. So if this is not the >> case, we've broken somewhere on the way. I'll try to find time to have >> a look. >> >> - Erno >> >> __ >> >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > Yeah I guess the schema fix in glanceclient didn't resolve the issue, > but I haven't had the time to dig into the bug since we're doing the > midcycle this week. > -- Thanks, Nikhil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
On 7/18/2016 4:41 AM, Erno Kuvaja wrote: On Mon, Jul 18, 2016 at 5:19 AM, Nikhil Komawarwrote: Thanks Matt. I've scheduled for a release of the client this week. On 7/16/16 4:09 AM, Matt Riedemann wrote: This is more of a heads up than anything. Our internal CI is running Tempest with images that don't have kernel_id or ramdisk_id properties set. We're running from master so nova defaults to use_glance_v1=False. Because of this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 and this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 The snapshot image properties get kernel_id and ramdisk_id set to None since that's what the glance v2 schema requires. However, python-glanceclient has it's own outdated copy of the schema which doesn't allow null values for those properties, see bug: https://bugs.launchpad.net/python-glanceclient/+bug/1596602 We don't hit this in the community CI because the image that Tempest uses from devstack has the kernel_id and ramdisk_id properties set: http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 But for anyone else upgrading to Newton that has images without those properties set and doesn't have use_glance_v1=True in nova.conf is going to be broken. Since we really want to get people off glance v1 and move to deprecation in Ocata, we need to get this merged and released: https://review.openstack.org/#/c/335152/ And bump the minimum required python-glanceclient in global-requirements for Newton. I'm not really sure why python-glanceclient even has it's own copy of the image schema, that seems redundant and error prone given the glance API already validates that, but it's kind of beside the point right now. -- Thanks, Nikhil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Matt, Something has broken on the way if that shipped schema actually affect your operation. That should be used _only_ when the client is called without connection to the Glance API (for example in a case you have no env set and you call `glance help`), this is btw the reason why we have that schema shipped with the client. As soon as you execute a call that actually interacts with Glance API we should be pulling the schema from there. So if this is not the case, we've broken somewhere on the way. I'll try to find time to have a look. - Erno __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Yeah I guess the schema fix in glanceclient didn't resolve the issue, but I haven't had the time to dig into the bug since we're doing the midcycle this week. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
For those needing this release soon, watch out for the announcement. Ref: https://review.openstack.org/#/c/343708 On 7/18/16 12:19 AM, Nikhil Komawar wrote: > Thanks Matt. I've scheduled for a release of the client this week. > > On 7/16/16 4:09 AM, Matt Riedemann wrote: >> This is more of a heads up than anything. >> >> Our internal CI is running Tempest with images that don't have >> kernel_id or ramdisk_id properties set. >> >> We're running from master so nova defaults to use_glance_v1=False. >> >> Because of this: >> >> https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 >> >> >> and this: >> >> https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 >> >> >> The snapshot image properties get kernel_id and ramdisk_id set to None >> since that's what the glance v2 schema requires. >> >> However, python-glanceclient has it's own outdated copy of the schema >> which doesn't allow null values for those properties, see bug: >> >> https://bugs.launchpad.net/python-glanceclient/+bug/1596602 >> >> We don't hit this in the community CI because the image that Tempest >> uses from devstack has the kernel_id and ramdisk_id properties set: >> >> http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 >> >> >> But for anyone else upgrading to Newton that has images without those >> properties set and doesn't have use_glance_v1=True in nova.conf is >> going to be broken. >> >> Since we really want to get people off glance v1 and move to >> deprecation in Ocata, we need to get this merged and released: >> >> https://review.openstack.org/#/c/335152/ >> >> And bump the minimum required python-glanceclient in >> global-requirements for Newton. >> >> I'm not really sure why python-glanceclient even has it's own copy of >> the image schema, that seems redundant and error prone given the >> glance API already validates that, but it's kind of beside the point >> right now. >> -- Thanks, Nikhil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
On Mon, Jul 18, 2016 at 5:19 AM, Nikhil Komawarwrote: > Thanks Matt. I've scheduled for a release of the client this week. > > On 7/16/16 4:09 AM, Matt Riedemann wrote: >> This is more of a heads up than anything. >> >> Our internal CI is running Tempest with images that don't have >> kernel_id or ramdisk_id properties set. >> >> We're running from master so nova defaults to use_glance_v1=False. >> >> Because of this: >> >> https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 >> >> >> and this: >> >> https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 >> >> >> The snapshot image properties get kernel_id and ramdisk_id set to None >> since that's what the glance v2 schema requires. >> >> However, python-glanceclient has it's own outdated copy of the schema >> which doesn't allow null values for those properties, see bug: >> >> https://bugs.launchpad.net/python-glanceclient/+bug/1596602 >> >> We don't hit this in the community CI because the image that Tempest >> uses from devstack has the kernel_id and ramdisk_id properties set: >> >> http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 >> >> >> But for anyone else upgrading to Newton that has images without those >> properties set and doesn't have use_glance_v1=True in nova.conf is >> going to be broken. >> >> Since we really want to get people off glance v1 and move to >> deprecation in Ocata, we need to get this merged and released: >> >> https://review.openstack.org/#/c/335152/ >> >> And bump the minimum required python-glanceclient in >> global-requirements for Newton. >> >> I'm not really sure why python-glanceclient even has it's own copy of >> the image schema, that seems redundant and error prone given the >> glance API already validates that, but it's kind of beside the point >> right now. >> > > -- > > Thanks, > Nikhil > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Matt, Something has broken on the way if that shipped schema actually affect your operation. That should be used _only_ when the client is called without connection to the Glance API (for example in a case you have no env set and you call `glance help`), this is btw the reason why we have that schema shipped with the client. As soon as you execute a call that actually interacts with Glance API we should be pulling the schema from there. So if this is not the case, we've broken somewhere on the way. I'll try to find time to have a look. - Erno __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
Thanks Matt. I've scheduled for a release of the client this week. On 7/16/16 4:09 AM, Matt Riedemann wrote: > This is more of a heads up than anything. > > Our internal CI is running Tempest with images that don't have > kernel_id or ramdisk_id properties set. > > We're running from master so nova defaults to use_glance_v1=False. > > Because of this: > > https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 > > > and this: > > https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 > > > The snapshot image properties get kernel_id and ramdisk_id set to None > since that's what the glance v2 schema requires. > > However, python-glanceclient has it's own outdated copy of the schema > which doesn't allow null values for those properties, see bug: > > https://bugs.launchpad.net/python-glanceclient/+bug/1596602 > > We don't hit this in the community CI because the image that Tempest > uses from devstack has the kernel_id and ramdisk_id properties set: > > http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 > > > But for anyone else upgrading to Newton that has images without those > properties set and doesn't have use_glance_v1=True in nova.conf is > going to be broken. > > Since we really want to get people off glance v1 and move to > deprecation in Ocata, we need to get this merged and released: > > https://review.openstack.org/#/c/335152/ > > And bump the minimum required python-glanceclient in > global-requirements for Newton. > > I'm not really sure why python-glanceclient even has it's own copy of > the image schema, that seems redundant and error prone given the > glance API already validates that, but it's kind of beside the point > right now. > -- Thanks, Nikhil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][glance] snapshots are broken by default with newton and glance v2
This is more of a heads up than anything. Our internal CI is running Tempest with images that don't have kernel_id or ramdisk_id properties set. We're running from master so nova defaults to use_glance_v1=False. Because of this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/compute/api.py#L867-L868 and this: https://github.com/openstack/nova/blob/47358449d359a287d21426b4e1f18479a4d1fd36/nova/image/glance.py#L835 The snapshot image properties get kernel_id and ramdisk_id set to None since that's what the glance v2 schema requires. However, python-glanceclient has it's own outdated copy of the schema which doesn't allow null values for those properties, see bug: https://bugs.launchpad.net/python-glanceclient/+bug/1596602 We don't hit this in the community CI because the image that Tempest uses from devstack has the kernel_id and ramdisk_id properties set: http://logs.openstack.org/52/335152/1/check/gate-tempest-dsvm-neutron-src-python-glanceclient/d393db9/logs/devstacklog.txt.gz#_2016-06-28_18_40_12_429 But for anyone else upgrading to Newton that has images without those properties set and doesn't have use_glance_v1=True in nova.conf is going to be broken. Since we really want to get people off glance v1 and move to deprecation in Ocata, we need to get this merged and released: https://review.openstack.org/#/c/335152/ And bump the minimum required python-glanceclient in global-requirements for Newton. I'm not really sure why python-glanceclient even has it's own copy of the image schema, that seems redundant and error prone given the glance API already validates that, but it's kind of beside the point right now. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev