Re: [openstack-dev] [glance] proposed priorities for Mitaka

[John Garbutt]

On 15 September 2015 at 18:22, Monty Taylor  wrote:
> On 09/15/2015 07:09 PM, stuart.mcla...@hp.com wrote:

 I've been looking into the existing task-based-upload that Doug
 mentions:
 can anyone clarify the following?

 On a default devstack install you can do this 'task' call:

 http://paste.openstack.org/show/462919
>>>
>>>
>>> Yup. That's the one.
>>>
 as an alternative to the traditional image upload (the bytes are
 streamed
 from the URL).

 It's not clear to me if this is just an interesting example of the kind
 of operator specific thing you can configure tasks to do, or a real
 attempt to define an alternative way to upload images.

 The change which added it [1] calls it a 'sample'.

 Is it just an example, or is it a second 'official' upload path?
>>>
>>>
>>> It's how you have to upload images on Rackspace.
>>
>>
>> Ok, so Rackspace have a task called image_import. But it seems to take
>> different json input than the devstack version. (A Swift container/object
>> rather than a URL.)
>>
>> That seems to suggest that tasks really are operator specific, that there
>> is no standard task based upload ... and it should be ok to try
>> again with a clean slate.
>
>
> Yes - as long as we don't use the payload as a defacto undefined API to
> avoid having specific things implemented in the API I think we're fine.
>
> Like, if it was:
>
> glance import-image
>
> and that presented an interface that had a status field ... I mean, that's a
> known OpenStack pattern - it's how nova boot works.
>
> Amongst the badness with this is:
>
> a) It's only implemented in one cloud and at that cloud with special code
> b) The interface is "send some JSON to this endpoint, and we'll infer a
> special sub-API from the JSON, which is not a published nor versioned API"

+1 for moving forward with a "works everywhere" upload API.

My memory of the issue here, is the want to validate images, before
allowing users to create instances from that image. If we can get that
working with the regular HTTP upload method, I think we will have
sorted the main issue.

Its more about failing as early as possible and defence in depth,
rather than a specific threat thats not already protected against,
AFAIK.

Forcing copy from swift is handy in terms of reducing glance API load,
but that shouldn't be done at the cost of interoperability.

Thanks,
johnthetubaguy

>>> If you want to see the
>>> full fun:
>>>
>>>
>>> https://github.com/openstack-infra/shade/blob/master/shade/__init__.py#L1335-L1510
>>>
>>>
>>> Which is "I want to upload an image to an OpenStack Cloud"
>>>
>>> I've listed it on this slide in CLI format too:
>>>
>>> http://inaugust.com/talks/product-management/index.html#/27
>>>
>>> It should be noted that once you create the task, you need to poll the
>>> task with task-show, and then the image id will be in the completed
>>> task-show output.
>>>
>>> Monty
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[John Garbutt]

On 14 September 2015 at 16:27, Thierry Carrez  wrote:
> Doug Hellmann wrote:
>> [...]
>> 1. Resolve the situation preventing the DefCore committee from
>>including image upload capabilities in the tests used for trademark
>>and interoperability validation.
>>
>> 2. Follow through on the original commitment of the project to
>>provide an image API by completing the integration work with
>>nova and cinder to ensure V2 API adoption.
>> [...]
>
> Thanks Doug for taking the time to dive into Glance and to write this
> email. I agree with your top two priorities as being a good summary of
> what the "rest of the community" expects the Glance leadership to focus
> on in the very short term.

Sorry for going back in time, but...

+1

This is a great summary of things, and I agree with the large strokes
of what is being suggested here.

Thank for helping ignite a very worthwhile debate here.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[stuart . mclaren]

I've been looking into the existing task-based-upload that Doug mentions:
can anyone clarify the following?

On a default devstack install you can do this 'task' call:

http://paste.openstack.org/show/462919


Yup. That's the one.


as an alternative to the traditional image upload (the bytes are streamed
from the URL).

It's not clear to me if this is just an interesting example of the kind
of operator specific thing you can configure tasks to do, or a real
attempt to define an alternative way to upload images.

The change which added it [1] calls it a 'sample'.

Is it just an example, or is it a second 'official' upload path?


It's how you have to upload images on Rackspace.


Ok, so Rackspace have a task called image_import. But it seems to take
different json input than the devstack version. (A Swift container/object
rather than a URL.)

That seems to suggest that tasks really are operator specific, that there
is no standard task based upload ... and it should be ok to try
again with a clean slate.


If you want to see the
full fun:

https://github.com/openstack-infra/shade/blob/master/shade/__init__.py#L1335-L1510

Which is "I want to upload an image to an OpenStack Cloud"

I've listed it on this slide in CLI format too:

http://inaugust.com/talks/product-management/index.html#/27

It should be noted that once you create the task, you need to poll the
task with task-show, and then the image id will be in the completed
task-show output.

Monty


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Flavio Percoco]

On 14/09/15 15:51 -0400, Doug Hellmann wrote:

Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:

On 14/09/15 08:10 -0400, Doug Hellmann wrote:

[snip]


The task upload process you're referring to is the one that uses the
`import` task, which allows you to download an image from an external
source, asynchronously, and import it in Glance. This is the old
`copy-from` behavior that was moved into a task.

The "fun" thing about this - and I'm sure other folks in the Glance
community will disagree - is that I don't consider tasks to be a
public API. That is to say, I would expect tasks to be an internal API
used by cloud admins to perform some actions (bsaed on its current
implementation). Eventually, some of these tasks could be triggered
from the external API but as background operations that are triggered
by the well-known public ones and not through the task API.


Does that mean it's more of an "admin" API?


As it is right now, yes. I don't think it's suitable for public use
and the current supported features are more useful for admins than
end-users.

Could it be improved to be a public API? Sure.

[snip]


This is definitely unfortunate. I believe a good step forward for this
discussion would be to create a list of issues related to uploading
images and see how those issues can be addressed. The result from that
work might be that it's not recommended to make that endpoint public
but again, without going through the issues, it'll be hard to
understand how we can improve this situation. I expect most of this
issues to have a security impact.


A report like that would be good to have. Can someone on the Glance team
volunteer to put it together?


Here's an attempt from someone that uses clouds but doesn't run any:

- Image authenticity (we recently landed code that allows for having
 signed images)
- Quota management: Glance's quota management is very basic and it
 allows for setting quota in a per-user level[1]
- Bandwidth requirements to upload images
- (add more here)

[0] 
http://specs.openstack.org/openstack/glance-specs/specs/liberty/image-signing-and-verification-support.html
[1] 
http://docs.openstack.org/developer/glance/configuring.html#configuring-glance-user-storage-quota

[snip]

This is, indeed, an interesting interpretation of what tasks are for.
I'd probably just blame us (Glance team) for not communicating
properly what tasks are meant to be. I don't believe tasks are a way
to extend the *public* API and I'd be curious to know if others see it
that way. I fully agree that just breaks interoperability and as I've
mentioned a couple of times in this reply already, I don't even think
tasks should be part of the public API.


Whether they are intended to be an extension mechanism, they
effectively are right now, as far as I can tell.


Sorry, I probably didn't express myself correctly. What I meant to say
is that I don't see them as a way to extend the *public* API but
rather as a way to add functionality to glance that is useful for
admins.


The mistake here could be that the library should've been refactored
*before* adopting it in Glance.


The fact that there is disagreement over the intent of the library makes
me think the plan for creating it wasn't sufficiently circulated or
detailed.


There wasn't much disagreement when it was created. Some folks think
the use-cases for the library don't exist anymore and some folks that
participated in this effort are not part of OpenStack anymore.

[snip]

Flavio

--
@flaper87
Flavio Percoco


pgpVhzArbV3YP.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Flavio Percoco]

On 14/09/15 16:46 -0400, Doug Hellmann wrote:

Excerpts from Clint Byrum's message of 2015-09-14 13:25:43 -0700:

Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:
> Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > >
> > >After having some conversations with folks at the Ops Midcycle a
> > >few weeks ago, and observing some of the more recent email threads
> > >related to glance, glance-store, the client, and the API, I spent
> > >last week contacting a few of you individually to learn more about
> > >some of the issues confronting the Glance team. I had some very
> > >frank, but I think constructive, conversations with all of you about
> > >the issues as you see them. As promised, this is the public email
> > >thread to discuss what I found, and to see if we can agree on what
> > >the Glance team should be focusing on going into the Mitaka summit
> > >and development cycle and how the rest of the community can support
> > >you in those efforts.
> > >
> > >I apologize for the length of this email, but there's a lot to go
> > >over. I've identified 2 high priority items that I think are critical
> > >for the team to be focusing on starting right away in order to use
> > >the upcoming summit time effectively. I will also describe several
> > >other issues that need to be addressed but that are less immediately
> > >critical. First the high priority items:
> > >
> > >1. Resolve the situation preventing the DefCore committee from
> > >   including image upload capabilities in the tests used for trademark
> > >   and interoperability validation.
> > >
> > >2. Follow through on the original commitment of the project to
> > >   provide an image API by completing the integration work with
> > >   nova and cinder to ensure V2 API adoption.
> >
> > Hi Doug,
> >
> > First and foremost, I'd like to thank you for taking the time to dig
> > into these issues, and for reaching out to the community seeking for
> > information and a better understanding of what the real issues are. I
> > can imagine how much time you had to dedicate on this and I'm glad you
> > did.
> >
> > Now, to your email, I very much agree with the priorities you
> > mentioned above and I'd like for, whomever will win Glance's PTL
> > election, to bring focus back on that.
> >
> > Please, find some comments in-line for each point:
> >
> > >
> > >I. DefCore
> > >
> > >The primary issue that attracted my attention was the fact that
> > >DefCore cannot currently include an image upload API in its
> > >interoperability test suite, and therefore we do not have a way to
> > >ensure interoperability between clouds for users or for trademark
> > >use. The DefCore process has been long, and at times confusing,
> > >even to those of us following it sort of closely. It's not entirely
> > >surprising that some projects haven't been following the whole time,
> > >or aren't aware of exactly what the whole thing means. I have
> > >proposed a cross-project summit session for the Mitaka summit to
> > >address this need for communication more broadly, but I'll try to
> > >summarize a bit here.
> >
> > +1
> >
> > I think it's quite sad that some projects, especially those considered
> > to be part of the `starter-kit:compute`[0], don't follow closely
> > what's going on in DefCore. I personally consider this a task PTLs
> > should incorporate in their role duties. I'm glad you proposed such
> > session, I hope it'll help raising awareness of this effort and it'll
> > help moving things forward on that front.
>
> Until fairly recently a lot of the discussion was around process
> and priorities for the DefCore committee. Now that those things are
> settled, and we have some approved policies, it's time to engage
> more fully.  I'll be working during Mitaka to improve the two-way
> communication.
>
> >
> > >
> > >DefCore is using automated tests, combined with business policies,
> > >to build a set of criteria for allowing trademark use. One of the
> > >goals of that process is to ensure that all OpenStack deployments
> > >are interoperable, so that users who write programs that talk to
> > >one cloud can use the same program with another cloud easily. This
> > >is a *REST API* level of compatibility. We cannot insert cloud-specific
> > >behavior into our client libraries, because not all cloud consumers
> > >will use those libraries to talk to the services. Similarly, we
> > >can't put the logic in the test suite, because that defeats the
> > >entire purpose of making the APIs interoperable. For this level of
> > >compatibility to work, we need well-defined APIs, with a long support
> > >period, that work the same no matter how the cloud is deployed. We
> > >need the entire community to support this effort. From what I can
> > >tell, that is going to require some changes to the current Glance
> > >API to meet the requirements. I'll list those requirements, and I
> > >hope we can 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Kuvaja, Erno]

> -Original Message-
> From: Doug Hellmann [mailto:d...@doughellmann.com]
> Sent: Monday, September 14, 2015 5:40 PM
> To: openstack-dev
> Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> 
> Excerpts from Kuvaja, Erno's message of 2015-09-14 15:02:59 +:
> > > -Original Message-
> > > From: Flavio Percoco [mailto:fla...@redhat.com]
> > > Sent: Monday, September 14, 2015 1:41 PM
> > > To: OpenStack Development Mailing List (not for usage questions)
> > > Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> > >
> > > On 14/09/15 08:10 -0400, Doug Hellmann wrote:

> > > >
> > > >I. DefCore
> > > >
> > > >The primary issue that attracted my attention was the fact that
> > > >DefCore cannot currently include an image upload API in its
> > > >interoperability test suite, and therefore we do not have a way to
> > > >ensure interoperability between clouds for users or for trademark
> > > >use. The DefCore process has been long, and at times confusing,
> > > >even to those of us following it sort of closely. It's not entirely
> > > >surprising that some projects haven't been following the whole
> > > >time, or aren't aware of exactly what the whole thing means. I have
> > > >proposed a cross-project summit session for the Mitaka summit to
> > > >address this need for communication more broadly, but I'll try to
> summarize a bit here.
> > >
> >
> > Looking how different OpenStack based public clouds limits or fully
> prevents their users to upload images to their deployments, I'm not
> convinced the Image Upload should be included to this definition.
> 
> The problem with that approach is that it means end consumers of those
> clouds cannot write common tools that include image uploads, which is a
> frequently used/desired feature. What makes that feature so special that we
> don't care about it for interoperability?
> 

I'm not sure it really is so special API or technical wise, it's just the one 
that was lifted to the pedestal in this discussion.

> >
> > > +1
> > >
> > > I think it's quite sad that some projects, especially those
> > > considered to be part of the `starter-kit:compute`[0], don't follow
> > > closely what's going on in DefCore. I personally consider this a
> > > task PTLs should incorporate in their role duties. I'm glad you
> > > proposed such session, I hope it'll help raising awareness of this effort
> and it'll help moving things forward on that front.
> > >
> > >
> > > >
> > > >DefCore is using automated tests, combined with business policies,
> > > >to build a set of criteria for allowing trademark use. One of the
> > > >goals of that process is to ensure that all OpenStack deployments
> > > >are interoperable, so that users who write programs that talk to
> > > >one cloud can use the same program with another cloud easily. This
> > > >is a *REST
> > > >API* level of compatibility. We cannot insert cloud-specific
> > > >behavior into our client libraries, because not all cloud consumers
> > > >will use those libraries to talk to the services. Similarly, we
> > > >can't put the logic in the test suite, because that defeats the
> > > >entire purpose of making the APIs interoperable. For this level of
> > > >compatibility to work, we need well-defined APIs, with a long
> > > >support period, that work the same no matter how the cloud is
> > > >deployed. We need the entire community to support this effort. From
> > > >what I can tell, that is going to require some changes to the
> > > >current Glance API to meet the requirements. I'll list those
> > > >requirements, and I hope we can discuss them to a degree that
> > > >ensures everyone understands them. I don't want this email thread
> > > >to get bogged down in implementation details or API designs,
> > > >though, so let's try to keep the discussion at a somewhat high
> > > >level, and leave the details for specs and summit discussions. I do
> > > >hope you will correct any misunderstandings or misconceptions,
> > > >because unwinding this as an outside observer has been quite a
> challenge and it's likely I have some details wrong.
> >
> > This just reinforces my doubt above. By including upload to the defcore
> requirements probably just closes out lots of the public clouds out there. Is
> that the intentio

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Flavio Percoco]

On 15/09/15 02:46 +0200, Monty Taylor wrote:

On 09/15/2015 02:06 AM, Clint Byrum wrote:

Excerpts from Doug Hellmann's message of 2015-09-14 13:46:16 -0700:

Excerpts from Clint Byrum's message of 2015-09-14 13:25:43 -0700:

Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:

Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:

On 14/09/15 08:10 -0400, Doug Hellmann wrote:


After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.

I apologize for the length of this email, but there's a lot to go
over. I've identified 2 high priority items that I think are critical
for the team to be focusing on starting right away in order to use
the upcoming summit time effectively. I will also describe several
other issues that need to be addressed but that are less immediately
critical. First the high priority items:

1. Resolve the situation preventing the DefCore committee from
  including image upload capabilities in the tests used for trademark
  and interoperability validation.

2. Follow through on the original commitment of the project to
  provide an image API by completing the integration work with
  nova and cinder to ensure V2 API adoption.


Hi Doug,

First and foremost, I'd like to thank you for taking the time to dig
into these issues, and for reaching out to the community seeking for
information and a better understanding of what the real issues are. I
can imagine how much time you had to dedicate on this and I'm glad you
did.

Now, to your email, I very much agree with the priorities you
mentioned above and I'd like for, whomever will win Glance's PTL
election, to bring focus back on that.

Please, find some comments in-line for each point:



I. DefCore

The primary issue that attracted my attention was the fact that
DefCore cannot currently include an image upload API in its
interoperability test suite, and therefore we do not have a way to
ensure interoperability between clouds for users or for trademark
use. The DefCore process has been long, and at times confusing,
even to those of us following it sort of closely. It's not entirely
surprising that some projects haven't been following the whole time,
or aren't aware of exactly what the whole thing means. I have
proposed a cross-project summit session for the Mitaka summit to
address this need for communication more broadly, but I'll try to
summarize a bit here.


+1

I think it's quite sad that some projects, especially those considered
to be part of the `starter-kit:compute`[0], don't follow closely
what's going on in DefCore. I personally consider this a task PTLs
should incorporate in their role duties. I'm glad you proposed such
session, I hope it'll help raising awareness of this effort and it'll
help moving things forward on that front.


Until fairly recently a lot of the discussion was around process
and priorities for the DefCore committee. Now that those things are
settled, and we have some approved policies, it's time to engage
more fully.  I'll be working during Mitaka to improve the two-way
communication.





DefCore is using automated tests, combined with business policies,
to build a set of criteria for allowing trademark use. One of the
goals of that process is to ensure that all OpenStack deployments
are interoperable, so that users who write programs that talk to
one cloud can use the same program with another cloud easily. This
is a *REST API* level of compatibility. We cannot insert cloud-specific
behavior into our client libraries, because not all cloud consumers
will use those libraries to talk to the services. Similarly, we
can't put the logic in the test suite, because that defeats the
entire purpose of making the APIs interoperable. For this level of
compatibility to work, we need well-defined APIs, with a long support
period, that work the same no matter how the cloud is deployed. We
need the entire community to support this effort. From what I can
tell, that is going to require some changes to the current Glance
API to meet the requirements. I'll list those requirements, and I
hope we can discuss them to a degree that ensures everyone understands
them. I don't want this email thread to get bogged down in
implementation details or API designs, though, so let's try to keep
the discussion at a somewhat high level, and leave the details for
specs and summit discussions. 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Flavio Percoco]

On 15/09/15 08:30 -0400, Doug Hellmann wrote:

Excerpts from Kuvaja, Erno's message of 2015-09-15 09:43:26 +:

[snip]

I'm not sure it really is so special API or technical wise, it's just the one 
that was lifted to the pedestal in this discussion.


OK. I'm concerned that my message of "we need an interoperable image
upload API" is sometimes being met with various versions of "that's not
possible." I think that's wrong, and we should fix it. I also think it's
possible to make the API consistent and still support background tasks,
image scanning, and other things deployers want.


Yes, this is a discussion that started in this cycle as part of
this[0] proposed spec. The discussion was put on hold until Mitaka.
One of the concerns raised was whether it's ok to make tasks part of
the upload process or not since that changes some of the existing
behavior.

For example, right now, when an image is uploaded, it can be used
right away. If we make async tasks part of the upload workflow, then
images won't be available until all tasks are executed.

Personally, I think the above is fine and it'd give the user a better
experience in comparison w/ the current task API. There are other
issues related to this that require a lenghtier discussion and are not
strictly related to the API.

[0] https://review.openstack.org/#/c/188388/

Flavio

--
@flaper87
Flavio Percoco


pgpbN1hj_vICL.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Monty Taylor]

On 09/15/2015 03:13 PM, stuart.mcla...@hp.com wrote:


After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.


Doug, thanks for reaching out here.

I've been looking into the existing task-based-upload that Doug mentions:
can anyone clarify the following?

On a default devstack install you can do this 'task' call:

http://paste.openstack.org/show/462919


Yup. That's the one.


as an alternative to the traditional image upload (the bytes are streamed
from the URL).

It's not clear to me if this is just an interesting example of the kind
of operator specific thing you can configure tasks to do, or a real
attempt to define an alternative way to upload images.

The change which added it [1] calls it a 'sample'.

Is it just an example, or is it a second 'official' upload path?


It's how you have to upload images on Rackspace. If you want to see the 
full fun:


https://github.com/openstack-infra/shade/blob/master/shade/__init__.py#L1335-L1510

Which is "I want to upload an image to an OpenStack Cloud"

I've listed it on this slide in CLI format too:

http://inaugust.com/talks/product-management/index.html#/27

It should be noted that once you create the task, you need to poll the 
task with task-show, and then the image id will be in the completed 
task-show output.


Monty

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Monty Taylor]

On 09/15/2015 07:09 PM, stuart.mcla...@hp.com wrote:

I've been looking into the existing task-based-upload that Doug
mentions:
can anyone clarify the following?

On a default devstack install you can do this 'task' call:

http://paste.openstack.org/show/462919


Yup. That's the one.


as an alternative to the traditional image upload (the bytes are
streamed
from the URL).

It's not clear to me if this is just an interesting example of the kind
of operator specific thing you can configure tasks to do, or a real
attempt to define an alternative way to upload images.

The change which added it [1] calls it a 'sample'.

Is it just an example, or is it a second 'official' upload path?


It's how you have to upload images on Rackspace.


Ok, so Rackspace have a task called image_import. But it seems to take
different json input than the devstack version. (A Swift container/object
rather than a URL.)

That seems to suggest that tasks really are operator specific, that there
is no standard task based upload ... and it should be ok to try
again with a clean slate.


Yes - as long as we don't use the payload as a defacto undefined API to 
avoid having specific things implemented in the API I think we're fine.


Like, if it was:

glance import-image

and that presented an interface that had a status field ... I mean, 
that's a known OpenStack pattern - it's how nova boot works.


Amongst the badness with this is:

a) It's only implemented in one cloud and at that cloud with special code
b) The interface is "send some JSON to this endpoint, and we'll infer a 
special sub-API from the JSON, which is not a published nor versioned API"



If you want to see the
full fun:

https://github.com/openstack-infra/shade/blob/master/shade/__init__.py#L1335-L1510


Which is "I want to upload an image to an OpenStack Cloud"

I've listed it on this slide in CLI format too:

http://inaugust.com/talks/product-management/index.html#/27

It should be noted that once you create the task, you need to poll the
task with task-show, and then the image id will be in the completed
task-show output.

Monty


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Doug Hellmann]

Excerpts from Kuvaja, Erno's message of 2015-09-15 09:43:26 +:
> > -Original Message-
> > From: Doug Hellmann [mailto:d...@doughellmann.com]
> > Sent: Monday, September 14, 2015 5:40 PM
> > To: openstack-dev
> > Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> > 
> > Excerpts from Kuvaja, Erno's message of 2015-09-14 15:02:59 +:
> > > > -Original Message-
> > > > From: Flavio Percoco [mailto:fla...@redhat.com]
> > > > Sent: Monday, September 14, 2015 1:41 PM
> > > > To: OpenStack Development Mailing List (not for usage questions)
> > > > Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> > > >
> > > > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > > > >
> > > > >I. DefCore
> > > > >
> > > > >The primary issue that attracted my attention was the fact that
> > > > >DefCore cannot currently include an image upload API in its
> > > > >interoperability test suite, and therefore we do not have a way to
> > > > >ensure interoperability between clouds for users or for trademark
> > > > >use. The DefCore process has been long, and at times confusing,
> > > > >even to those of us following it sort of closely. It's not entirely
> > > > >surprising that some projects haven't been following the whole
> > > > >time, or aren't aware of exactly what the whole thing means. I have
> > > > >proposed a cross-project summit session for the Mitaka summit to
> > > > >address this need for communication more broadly, but I'll try to
> > summarize a bit here.
> > > >
> > >
> > > Looking how different OpenStack based public clouds limits or fully
> > prevents their users to upload images to their deployments, I'm not
> > convinced the Image Upload should be included to this definition.
> > 
> > The problem with that approach is that it means end consumers of those
> > clouds cannot write common tools that include image uploads, which is a
> > frequently used/desired feature. What makes that feature so special that we
> > don't care about it for interoperability?
> > 
> 
> I'm not sure it really is so special API or technical wise, it's just the one 
> that was lifted to the pedestal in this discussion.

OK. I'm concerned that my message of "we need an interoperable image
upload API" is sometimes being met with various versions of "that's not
possible." I think that's wrong, and we should fix it. I also think it's
possible to make the API consistent and still support background tasks,
image scanning, and other things deployers want.

> 
> > > >
> > > > The task upload process you're referring to is the one that uses the
> > > > `import` task, which allows you to download an image from an
> > > > external source, asynchronously, and import it in Glance. This is
> > > > the old `copy-from` behavior that was moved into a task.
> > > >
> > > > The "fun" thing about this - and I'm sure other folks in the Glance
> > > > community will disagree - is that I don't consider tasks to be a
> > > > public API. That is to say, I would expect tasks to be an internal
> > > > API used by cloud admins to perform some actions (bsaed on its
> > > > current implementation). Eventually, some of these tasks could be
> > > > triggered from the external API but as background operations that
> > > > are triggered by the well-known public ones and not through the task
> > API.
> > > >
> > > > Ultimately, I believe end-users of the cloud simply shouldn't care
> > > > about what tasks are or aren't and more importantly, as you
> > > > mentioned later in the email, tasks make clouds not interoperable.
> > > > I'd be pissed if my public image service would ask me to learn about 
> > > > tasks
> > to be able to use the service.
> > >
> > > I'd like to bring another argument here. I think our Public Images API 
> > > should
> > behave consistently regardless if there is tasks enabled in the deployment 
> > or
> > not and with what plugins. This meaning that _if_ we expect glance upload
> > work over the POST API and that endpoint is available in the deployment I
> > would expect a) my image hash to match with the one the cloud returns b)
> > I'd assume all or none of the clouds rejecting my image if it gets flagged 
> > by
> > Vendor X virus def

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Doug Hellmann]

Excerpts from Clint Byrum's message of 2015-09-14 17:06:44 -0700:
> Excerpts from Doug Hellmann's message of 2015-09-14 13:46:16 -0700:
> > Excerpts from Clint Byrum's message of 2015-09-14 13:25:43 -0700:
> > > Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:
> > > > Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> > > > > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > > > > >
> > > > > >After having some conversations with folks at the Ops Midcycle a
> > > > > >few weeks ago, and observing some of the more recent email threads
> > > > > >related to glance, glance-store, the client, and the API, I spent
> > > > > >last week contacting a few of you individually to learn more about
> > > > > >some of the issues confronting the Glance team. I had some very
> > > > > >frank, but I think constructive, conversations with all of you about
> > > > > >the issues as you see them. As promised, this is the public email
> > > > > >thread to discuss what I found, and to see if we can agree on what
> > > > > >the Glance team should be focusing on going into the Mitaka summit
> > > > > >and development cycle and how the rest of the community can support
> > > > > >you in those efforts.
> > > > > >
> > > > > >I apologize for the length of this email, but there's a lot to go
> > > > > >over. I've identified 2 high priority items that I think are critical
> > > > > >for the team to be focusing on starting right away in order to use
> > > > > >the upcoming summit time effectively. I will also describe several
> > > > > >other issues that need to be addressed but that are less immediately
> > > > > >critical. First the high priority items:
> > > > > >
> > > > > >1. Resolve the situation preventing the DefCore committee from
> > > > > >   including image upload capabilities in the tests used for 
> > > > > > trademark
> > > > > >   and interoperability validation.
> > > > > >
> > > > > >2. Follow through on the original commitment of the project to
> > > > > >   provide an image API by completing the integration work with
> > > > > >   nova and cinder to ensure V2 API adoption.
> > > > > 
> > > > > Hi Doug,
> > > > > 
> > > > > First and foremost, I'd like to thank you for taking the time to dig
> > > > > into these issues, and for reaching out to the community seeking for
> > > > > information and a better understanding of what the real issues are. I
> > > > > can imagine how much time you had to dedicate on this and I'm glad you
> > > > > did.
> > > > > 
> > > > > Now, to your email, I very much agree with the priorities you
> > > > > mentioned above and I'd like for, whomever will win Glance's PTL
> > > > > election, to bring focus back on that.
> > > > > 
> > > > > Please, find some comments in-line for each point:
> > > > > 
> > > > > >
> > > > > >I. DefCore
> > > > > >
> > > > > >The primary issue that attracted my attention was the fact that
> > > > > >DefCore cannot currently include an image upload API in its
> > > > > >interoperability test suite, and therefore we do not have a way to
> > > > > >ensure interoperability between clouds for users or for trademark
> > > > > >use. The DefCore process has been long, and at times confusing,
> > > > > >even to those of us following it sort of closely. It's not entirely
> > > > > >surprising that some projects haven't been following the whole time,
> > > > > >or aren't aware of exactly what the whole thing means. I have
> > > > > >proposed a cross-project summit session for the Mitaka summit to
> > > > > >address this need for communication more broadly, but I'll try to
> > > > > >summarize a bit here.
> > > > > 
> > > > > +1
> > > > > 
> > > > > I think it's quite sad that some projects, especially those considered
> > > > > to be part of the `starter-kit:compute`[0], don't follow closely
> > > > > what's going on in DefCore. I personally consider this a task PTLs
> > > > > should incorporate in their role duties. I'm glad you proposed such
> > > > > session, I hope it'll help raising awareness of this effort and it'll
> > > > > help moving things forward on that front.
> > > > 
> > > > Until fairly recently a lot of the discussion was around process
> > > > and priorities for the DefCore committee. Now that those things are
> > > > settled, and we have some approved policies, it's time to engage
> > > > more fully.  I'll be working during Mitaka to improve the two-way
> > > > communication.
> > > > 
> > > > > 
> > > > > >
> > > > > >DefCore is using automated tests, combined with business policies,
> > > > > >to build a set of criteria for allowing trademark use. One of the
> > > > > >goals of that process is to ensure that all OpenStack deployments
> > > > > >are interoperable, so that users who write programs that talk to
> > > > > >one cloud can use the same program with another cloud easily. This
> > > > > >is a *REST API* level of compatibility. We cannot insert 
> > > > > >cloud-specific
> > > > > >behavior into our client 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[stuart . mclaren]

After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.


Doug, thanks for reaching out here.

I've been looking into the existing task-based-upload that Doug mentions:
can anyone clarify the following?

On a default devstack install you can do this 'task' call:

http://paste.openstack.org/show/462919

as an alternative to the traditional image upload (the bytes are streamed
from the URL).

It's not clear to me if this is just an interesting example of the kind
of operator specific thing you can configure tasks to do, or a real
attempt to define an alternative way to upload images.

The change which added it [1] calls it a 'sample'.

Is it just an example, or is it a second 'official' upload path?

-Stuart

[1] https://review.openstack.org/#/c/44355

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2015-09-15 10:54:04 +0200:
> On 14/09/15 15:51 -0400, Doug Hellmann wrote:
> >Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> >> On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> 
> >> This is definitely unfortunate. I believe a good step forward for this
> >> discussion would be to create a list of issues related to uploading
> >> images and see how those issues can be addressed. The result from that
> >> work might be that it's not recommended to make that endpoint public
> >> but again, without going through the issues, it'll be hard to
> >> understand how we can improve this situation. I expect most of this
> >> issues to have a security impact.
> >
> >A report like that would be good to have. Can someone on the Glance team
> >volunteer to put it together?
> 
> Here's an attempt from someone that uses clouds but doesn't run any:
> 
> - Image authenticity (we recently landed code that allows for having
>   signed images)
> - Quota management: Glance's quota management is very basic and it
>   allows for setting quota in a per-user level[1]
> - Bandwidth requirements to upload images
> - (add more here)

That seems like a good start. You could add a desire to optionally
take advantage of advanced object store services like Swift and
Ceph.

> >> The mistake here could be that the library should've been refactored
> >> *before* adopting it in Glance.
> >
> >The fact that there is disagreement over the intent of the library makes
> >me think the plan for creating it wasn't sufficiently circulated or
> >detailed.
> 
> There wasn't much disagreement when it was created. Some folks think
> the use-cases for the library don't exist anymore and some folks that
> participated in this effort are not part of OpenStack anymore.

OK. There is definite desire outside of the Glance team to have
*some* library for talking to the image store directly. The evidence
for that is the specs in nova related to a "seam" library, and the
requests by some Cinder driver authors to have something similar.
>From what I can tell, everyone else thought that's what glance-store
was going to be, but it's not quite what is needed.  It seems like
the use cases need to be revisited so the requirements can be
documented properly and then we can figure out what steps to take
with the existing code.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] proposed priorities for Mitaka

[Doug Hellmann]

After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.

I apologize for the length of this email, but there's a lot to go
over. I've identified 2 high priority items that I think are critical
for the team to be focusing on starting right away in order to use
the upcoming summit time effectively. I will also describe several
other issues that need to be addressed but that are less immediately
critical. First the high priority items:

1. Resolve the situation preventing the DefCore committee from
   including image upload capabilities in the tests used for trademark
   and interoperability validation.

2. Follow through on the original commitment of the project to
   provide an image API by completing the integration work with
   nova and cinder to ensure V2 API adoption.

I. DefCore

The primary issue that attracted my attention was the fact that
DefCore cannot currently include an image upload API in its
interoperability test suite, and therefore we do not have a way to
ensure interoperability between clouds for users or for trademark
use. The DefCore process has been long, and at times confusing,
even to those of us following it sort of closely. It's not entirely
surprising that some projects haven't been following the whole time,
or aren't aware of exactly what the whole thing means. I have
proposed a cross-project summit session for the Mitaka summit to
address this need for communication more broadly, but I'll try to
summarize a bit here.

DefCore is using automated tests, combined with business policies,
to build a set of criteria for allowing trademark use. One of the
goals of that process is to ensure that all OpenStack deployments
are interoperable, so that users who write programs that talk to
one cloud can use the same program with another cloud easily. This
is a *REST API* level of compatibility. We cannot insert cloud-specific
behavior into our client libraries, because not all cloud consumers
will use those libraries to talk to the services. Similarly, we
can't put the logic in the test suite, because that defeats the
entire purpose of making the APIs interoperable. For this level of
compatibility to work, we need well-defined APIs, with a long support
period, that work the same no matter how the cloud is deployed. We
need the entire community to support this effort. From what I can
tell, that is going to require some changes to the current Glance
API to meet the requirements. I'll list those requirements, and I
hope we can discuss them to a degree that ensures everyone understands
them. I don't want this email thread to get bogged down in
implementation details or API designs, though, so let's try to keep
the discussion at a somewhat high level, and leave the details for
specs and summit discussions. I do hope you will correct any
misunderstandings or misconceptions, because unwinding this as an
outside observer has been quite a challenge and it's likely I have
some details wrong.

As I understand it, there are basically two ways to upload an image
to glance using the V2 API today. The "POST" API pushes the image's
bits through the Glance API server, and the "task" API instructs
Glance to download the image separately in the background. At one
point apparently there was a bug that caused the results of the two
different paths to be incompatible, but I believe that is now fixed.
However, the two separate APIs each have different issues that make
them unsuitable for DefCore.

The DefCore process relies on several factors when designating APIs
for compliance. One factor is the technical direction, as communicated
by the contributor community -- that's where we tell them things
like "we plan to deprecate the Glance V1 API". In addition to the
technical direction, DefCore looks at the deployment history of an
API. They do not want to require deploying an API if it is not seen
as widely usable, and they look for some level of existing adoption
by cloud providers and distributors as an indication of that the
API is desired and can be successfully used. Because we have multiple
upload APIs, the message we're sending on technical direction is
weak right now, and so they have focused on deployment considerations
to resolve the question.

The POST API is enabled in many public clouds, but not consistently.
In some clouds like HP, a tenant requires special permission to use
the 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Flavio Percoco]

On 14/09/15 08:10 -0400, Doug Hellmann wrote:


After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.

I apologize for the length of this email, but there's a lot to go
over. I've identified 2 high priority items that I think are critical
for the team to be focusing on starting right away in order to use
the upcoming summit time effectively. I will also describe several
other issues that need to be addressed but that are less immediately
critical. First the high priority items:

1. Resolve the situation preventing the DefCore committee from
  including image upload capabilities in the tests used for trademark
  and interoperability validation.

2. Follow through on the original commitment of the project to
  provide an image API by completing the integration work with
  nova and cinder to ensure V2 API adoption.


Hi Doug,

First and foremost, I'd like to thank you for taking the time to dig
into these issues, and for reaching out to the community seeking for
information and a better understanding of what the real issues are. I
can imagine how much time you had to dedicate on this and I'm glad you
did.

Now, to your email, I very much agree with the priorities you
mentioned above and I'd like for, whomever will win Glance's PTL
election, to bring focus back on that.

Please, find some comments in-line for each point:




I. DefCore

The primary issue that attracted my attention was the fact that
DefCore cannot currently include an image upload API in its
interoperability test suite, and therefore we do not have a way to
ensure interoperability between clouds for users or for trademark
use. The DefCore process has been long, and at times confusing,
even to those of us following it sort of closely. It's not entirely
surprising that some projects haven't been following the whole time,
or aren't aware of exactly what the whole thing means. I have
proposed a cross-project summit session for the Mitaka summit to
address this need for communication more broadly, but I'll try to
summarize a bit here.


+1

I think it's quite sad that some projects, especially those considered
to be part of the `starter-kit:compute`[0], don't follow closely
what's going on in DefCore. I personally consider this a task PTLs
should incorporate in their role duties. I'm glad you proposed such
session, I hope it'll help raising awareness of this effort and it'll
help moving things forward on that front.




DefCore is using automated tests, combined with business policies,
to build a set of criteria for allowing trademark use. One of the
goals of that process is to ensure that all OpenStack deployments
are interoperable, so that users who write programs that talk to
one cloud can use the same program with another cloud easily. This
is a *REST API* level of compatibility. We cannot insert cloud-specific
behavior into our client libraries, because not all cloud consumers
will use those libraries to talk to the services. Similarly, we
can't put the logic in the test suite, because that defeats the
entire purpose of making the APIs interoperable. For this level of
compatibility to work, we need well-defined APIs, with a long support
period, that work the same no matter how the cloud is deployed. We
need the entire community to support this effort. From what I can
tell, that is going to require some changes to the current Glance
API to meet the requirements. I'll list those requirements, and I
hope we can discuss them to a degree that ensures everyone understands
them. I don't want this email thread to get bogged down in
implementation details or API designs, though, so let's try to keep
the discussion at a somewhat high level, and leave the details for
specs and summit discussions. I do hope you will correct any
misunderstandings or misconceptions, because unwinding this as an
outside observer has been quite a challenge and it's likely I have
some details wrong.

As I understand it, there are basically two ways to upload an image
to glance using the V2 API today. The "POST" API pushes the image's
bits through the Glance API server, and the "task" API instructs
Glance to download the image separately in the background. At one
point apparently there was a bug that caused the results of the two
different paths to be incompatible, but I believe that is now fixed.
However, the two separate APIs each have different issues 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Thierry Carrez]

Doug Hellmann wrote:
> [...]
> 1. Resolve the situation preventing the DefCore committee from
>including image upload capabilities in the tests used for trademark
>and interoperability validation.
> 
> 2. Follow through on the original commitment of the project to
>provide an image API by completing the integration work with
>nova and cinder to ensure V2 API adoption.
> [...]

Thanks Doug for taking the time to dive into Glance and to write this
email. I agree with your top two priorities as being a good summary of
what the "rest of the community" expects the Glance leadership to focus
on in the very short term.

Cheers,

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Kuvaja, Erno]

> -Original Message-
> From: Flavio Percoco [mailto:fla...@redhat.com]
> Sent: Monday, September 14, 2015 1:41 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> 
> On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> >
> >After having some conversations with folks at the Ops Midcycle a few
> >weeks ago, and observing some of the more recent email threads related
> >to glance, glance-store, the client, and the API, I spent last week
> >contacting a few of you individually to learn more about some of the
> >issues confronting the Glance team. I had some very frank, but I think
> >constructive, conversations with all of you about the issues as you see
> >them. As promised, this is the public email thread to discuss what I
> >found, and to see if we can agree on what the Glance team should be
> >focusing on going into the Mitaka summit and development cycle and how
> >the rest of the community can support you in those efforts.
> >
> >I apologize for the length of this email, but there's a lot to go over.
> >I've identified 2 high priority items that I think are critical for the
> >team to be focusing on starting right away in order to use the upcoming
> >summit time effectively. I will also describe several other issues that
> >need to be addressed but that are less immediately critical. First the
> >high priority items:
> >
> >1. Resolve the situation preventing the DefCore committee from
> >   including image upload capabilities in the tests used for trademark
> >   and interoperability validation.
> >
> >2. Follow through on the original commitment of the project to
> >   provide an image API by completing the integration work with
> >   nova and cinder to ensure V2 API adoption.
> 
> Hi Doug,
> 
> First and foremost, I'd like to thank you for taking the time to dig into 
> these
> issues, and for reaching out to the community seeking for information and a
> better understanding of what the real issues are. I can imagine how much
> time you had to dedicate on this and I'm glad you did.

++ Really thanks for taking the time for this.
> 
> Now, to your email, I very much agree with the priorities you mentioned
> above and I'd like for, whomever will win Glance's PTL election, to bring 
> focus
> back on that.
> 
> Please, find some comments in-line for each point:
> 
> 
> >
> >I. DefCore
> >
> >The primary issue that attracted my attention was the fact that DefCore
> >cannot currently include an image upload API in its interoperability
> >test suite, and therefore we do not have a way to ensure
> >interoperability between clouds for users or for trademark use. The
> >DefCore process has been long, and at times confusing, even to those of
> >us following it sort of closely. It's not entirely surprising that some
> >projects haven't been following the whole time, or aren't aware of
> >exactly what the whole thing means. I have proposed a cross-project
> >summit session for the Mitaka summit to address this need for
> >communication more broadly, but I'll try to summarize a bit here.
> 

Looking how different OpenStack based public clouds limits or fully prevents 
their users to upload images to their deployments, I'm not convinced the Image 
Upload should be included to this definition.
 
> +1
> 
> I think it's quite sad that some projects, especially those considered to be
> part of the `starter-kit:compute`[0], don't follow closely what's going on in
> DefCore. I personally consider this a task PTLs should incorporate in their 
> role
> duties. I'm glad you proposed such session, I hope it'll help raising 
> awareness
> of this effort and it'll help moving things forward on that front.
> 
> 
> >
> >DefCore is using automated tests, combined with business policies, to
> >build a set of criteria for allowing trademark use. One of the goals of
> >that process is to ensure that all OpenStack deployments are
> >interoperable, so that users who write programs that talk to one cloud
> >can use the same program with another cloud easily. This is a *REST
> >API* level of compatibility. We cannot insert cloud-specific behavior
> >into our client libraries, because not all cloud consumers will use
> >those libraries to talk to the services. Similarly, we can't put the
> >logic in the test suite, because that defeats the entire purpose of
> >making the APIs interoperable. For this level of compatibility to work,
> >we need well-defined APIs, with a long support period, that work the
> >same no matter how the clo

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Doug Hellmann]

Excerpts from Kuvaja, Erno's message of 2015-09-14 15:02:59 +:
> > -Original Message-
> > From: Flavio Percoco [mailto:fla...@redhat.com]
> > Sent: Monday, September 14, 2015 1:41 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> > 
> > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > >
> > >After having some conversations with folks at the Ops Midcycle a few
> > >weeks ago, and observing some of the more recent email threads related
> > >to glance, glance-store, the client, and the API, I spent last week
> > >contacting a few of you individually to learn more about some of the
> > >issues confronting the Glance team. I had some very frank, but I think
> > >constructive, conversations with all of you about the issues as you see
> > >them. As promised, this is the public email thread to discuss what I
> > >found, and to see if we can agree on what the Glance team should be
> > >focusing on going into the Mitaka summit and development cycle and how
> > >the rest of the community can support you in those efforts.
> > >
> > >I apologize for the length of this email, but there's a lot to go over.
> > >I've identified 2 high priority items that I think are critical for the
> > >team to be focusing on starting right away in order to use the upcoming
> > >summit time effectively. I will also describe several other issues that
> > >need to be addressed but that are less immediately critical. First the
> > >high priority items:
> > >
> > >1. Resolve the situation preventing the DefCore committee from
> > >   including image upload capabilities in the tests used for trademark
> > >   and interoperability validation.
> > >
> > >2. Follow through on the original commitment of the project to
> > >   provide an image API by completing the integration work with
> > >   nova and cinder to ensure V2 API adoption.
> > 
> > Hi Doug,
> > 
> > First and foremost, I'd like to thank you for taking the time to dig into 
> > these
> > issues, and for reaching out to the community seeking for information and a
> > better understanding of what the real issues are. I can imagine how much
> > time you had to dedicate on this and I'm glad you did.
> 
> ++ Really thanks for taking the time for this.
> > 
> > Now, to your email, I very much agree with the priorities you mentioned
> > above and I'd like for, whomever will win Glance's PTL election, to bring 
> > focus
> > back on that.
> > 
> > Please, find some comments in-line for each point:
> > 
> > 
> > >
> > >I. DefCore
> > >
> > >The primary issue that attracted my attention was the fact that DefCore
> > >cannot currently include an image upload API in its interoperability
> > >test suite, and therefore we do not have a way to ensure
> > >interoperability between clouds for users or for trademark use. The
> > >DefCore process has been long, and at times confusing, even to those of
> > >us following it sort of closely. It's not entirely surprising that some
> > >projects haven't been following the whole time, or aren't aware of
> > >exactly what the whole thing means. I have proposed a cross-project
> > >summit session for the Mitaka summit to address this need for
> > >communication more broadly, but I'll try to summarize a bit here.
> > 
> 
> Looking how different OpenStack based public clouds limits or fully prevents 
> their users to upload images to their deployments, I'm not convinced the 
> Image Upload should be included to this definition.

The problem with that approach is that it means end consumers of
those clouds cannot write common tools that include image uploads,
which is a frequently used/desired feature. What makes that feature
so special that we don't care about it for interoperability?

>  
> > +1
> > 
> > I think it's quite sad that some projects, especially those considered to be
> > part of the `starter-kit:compute`[0], don't follow closely what's going on 
> > in
> > DefCore. I personally consider this a task PTLs should incorporate in their 
> > role
> > duties. I'm glad you proposed such session, I hope it'll help raising 
> > awareness
> > of this effort and it'll help moving things forward on that front.
> > 
> > 
> > >
> > >DefCore is using automated tests, combined with business policies, to
> > >build a set of criteria for allowing

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Russell Bryant]

On 09/14/2015 11:27 AM, Thierry Carrez wrote:
> Doug Hellmann wrote:
>> [...]
>> 1. Resolve the situation preventing the DefCore committee from
>>including image upload capabilities in the tests used for trademark
>>and interoperability validation.
>>
>> 2. Follow through on the original commitment of the project to
>>provide an image API by completing the integration work with
>>nova and cinder to ensure V2 API adoption.
>> [...]
> 
> Thanks Doug for taking the time to dive into Glance and to write this
> email. I agree with your top two priorities as being a good summary of
> what the "rest of the community" expects the Glance leadership to focus
> on in the very short term.

+1

Thanks, Doug!  and agreed with Thierry's response here.

-- 
Russell Bryant

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Mike Perez]

On Mon, Sep 14, 2015 at 5:10 AM, Doug Hellmann  wrote:
>
> After having some conversations with folks at the Ops Midcycle a
> few weeks ago, and observing some of the more recent email threads
> related to glance, glance-store, the client, and the API, I spent
> last week contacting a few of you individually to learn more about
> some of the issues confronting the Glance team. I had some very
> frank, but I think constructive, conversations with all of you about
> the issues as you see them. As promised, this is the public email
> thread to discuss what I found, and to see if we can agree on what
> the Glance team should be focusing on going into the Mitaka summit
> and development cycle and how the rest of the community can support
> you in those efforts.

Hi Doug,

Thanks for all your work with investigating this. Just last month I
raised some problems in other projects like Cinder and Nova with
working with v2 Glance [1][2] that I felt were lost due to focuses in
other areas and missing the core reason for Glance [3].

I'd really appreciate the potential PTL's of Glance in the Mitaka
cycle to read Doug's email, and take in the support behind these
priorities.

The Glance team regardless should know that as a community, we're all
in this together and support their efforts in making a great image
service for OpenStack.


[1] - http://lists.openstack.org/pipermail/openstack-dev/2015-July/070714.html
[2] - 
http://eavesdrop.openstack.org/meetings/crossproject/2015/crossproject.2015-07-28-21.03.log.html#l-239
[3] - http://lists.openstack.org/pipermail/openstack-dev/2015-August/071993.html

--
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Monty Taylor]

On 09/14/2015 02:41 PM, Flavio Percoco wrote:

On 14/09/15 08:10 -0400, Doug Hellmann wrote:


After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.

I apologize for the length of this email, but there's a lot to go
over. I've identified 2 high priority items that I think are critical
for the team to be focusing on starting right away in order to use
the upcoming summit time effectively. I will also describe several
other issues that need to be addressed but that are less immediately
critical. First the high priority items:

1. Resolve the situation preventing the DefCore committee from
  including image upload capabilities in the tests used for trademark
  and interoperability validation.

2. Follow through on the original commitment of the project to
  provide an image API by completing the integration work with
  nova and cinder to ensure V2 API adoption.


Hi Doug,

First and foremost, I'd like to thank you for taking the time to dig
into these issues, and for reaching out to the community seeking for
information and a better understanding of what the real issues are. I
can imagine how much time you had to dedicate on this and I'm glad you
did.


Ditto. Thanks so much for the work Doug!


Now, to your email, I very much agree with the priorities you
mentioned above and I'd like for, whomever will win Glance's PTL
election, to bring focus back on that.

Please, find some comments in-line for each point:




I. DefCore

The primary issue that attracted my attention was the fact that
DefCore cannot currently include an image upload API in its
interoperability test suite, and therefore we do not have a way to
ensure interoperability between clouds for users or for trademark
use. The DefCore process has been long, and at times confusing,
even to those of us following it sort of closely. It's not entirely
surprising that some projects haven't been following the whole time,
or aren't aware of exactly what the whole thing means. I have
proposed a cross-project summit session for the Mitaka summit to
address this need for communication more broadly, but I'll try to
summarize a bit here.


+1

I think it's quite sad that some projects, especially those considered
to be part of the `starter-kit:compute`[0], don't follow closely
what's going on in DefCore. I personally consider this a task PTLs
should incorporate in their role duties. I'm glad you proposed such
session, I hope it'll help raising awareness of this effort and it'll
help moving things forward on that front.




DefCore is using automated tests, combined with business policies,
to build a set of criteria for allowing trademark use. One of the
goals of that process is to ensure that all OpenStack deployments
are interoperable, so that users who write programs that talk to
one cloud can use the same program with another cloud easily. This
is a *REST API* level of compatibility. We cannot insert cloud-specific
behavior into our client libraries, because not all cloud consumers
will use those libraries to talk to the services. Similarly, we
can't put the logic in the test suite, because that defeats the
entire purpose of making the APIs interoperable. For this level of
compatibility to work, we need well-defined APIs, with a long support
period, that work the same no matter how the cloud is deployed. We
need the entire community to support this effort. From what I can
tell, that is going to require some changes to the current Glance
API to meet the requirements. I'll list those requirements, and I
hope we can discuss them to a degree that ensures everyone understands
them. I don't want this email thread to get bogged down in
implementation details or API designs, though, so let's try to keep
the discussion at a somewhat high level, and leave the details for
specs and summit discussions. I do hope you will correct any
misunderstandings or misconceptions, because unwinding this as an
outside observer has been quite a challenge and it's likely I have
some details wrong.

As I understand it, there are basically two ways to upload an image
to glance using the V2 API today. The "POST" API pushes the image's
bits through the Glance API server, and the "task" API instructs
Glance to download the image separately in the background. At one
point apparently there was a bug that caused the results of the two
different paths to be incompatible, 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:
> Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > >
> > >After having some conversations with folks at the Ops Midcycle a
> > >few weeks ago, and observing some of the more recent email threads
> > >related to glance, glance-store, the client, and the API, I spent
> > >last week contacting a few of you individually to learn more about
> > >some of the issues confronting the Glance team. I had some very
> > >frank, but I think constructive, conversations with all of you about
> > >the issues as you see them. As promised, this is the public email
> > >thread to discuss what I found, and to see if we can agree on what
> > >the Glance team should be focusing on going into the Mitaka summit
> > >and development cycle and how the rest of the community can support
> > >you in those efforts.
> > >
> > >I apologize for the length of this email, but there's a lot to go
> > >over. I've identified 2 high priority items that I think are critical
> > >for the team to be focusing on starting right away in order to use
> > >the upcoming summit time effectively. I will also describe several
> > >other issues that need to be addressed but that are less immediately
> > >critical. First the high priority items:
> > >
> > >1. Resolve the situation preventing the DefCore committee from
> > >   including image upload capabilities in the tests used for trademark
> > >   and interoperability validation.
> > >
> > >2. Follow through on the original commitment of the project to
> > >   provide an image API by completing the integration work with
> > >   nova and cinder to ensure V2 API adoption.
> > 
> > Hi Doug,
> > 
> > First and foremost, I'd like to thank you for taking the time to dig
> > into these issues, and for reaching out to the community seeking for
> > information and a better understanding of what the real issues are. I
> > can imagine how much time you had to dedicate on this and I'm glad you
> > did.
> > 
> > Now, to your email, I very much agree with the priorities you
> > mentioned above and I'd like for, whomever will win Glance's PTL
> > election, to bring focus back on that.
> > 
> > Please, find some comments in-line for each point:
> > 
> > >
> > >I. DefCore
> > >
> > >The primary issue that attracted my attention was the fact that
> > >DefCore cannot currently include an image upload API in its
> > >interoperability test suite, and therefore we do not have a way to
> > >ensure interoperability between clouds for users or for trademark
> > >use. The DefCore process has been long, and at times confusing,
> > >even to those of us following it sort of closely. It's not entirely
> > >surprising that some projects haven't been following the whole time,
> > >or aren't aware of exactly what the whole thing means. I have
> > >proposed a cross-project summit session for the Mitaka summit to
> > >address this need for communication more broadly, but I'll try to
> > >summarize a bit here.
> > 
> > +1
> > 
> > I think it's quite sad that some projects, especially those considered
> > to be part of the `starter-kit:compute`[0], don't follow closely
> > what's going on in DefCore. I personally consider this a task PTLs
> > should incorporate in their role duties. I'm glad you proposed such
> > session, I hope it'll help raising awareness of this effort and it'll
> > help moving things forward on that front.
> 
> Until fairly recently a lot of the discussion was around process
> and priorities for the DefCore committee. Now that those things are
> settled, and we have some approved policies, it's time to engage
> more fully.  I'll be working during Mitaka to improve the two-way
> communication.
> 
> > 
> > >
> > >DefCore is using automated tests, combined with business policies,
> > >to build a set of criteria for allowing trademark use. One of the
> > >goals of that process is to ensure that all OpenStack deployments
> > >are interoperable, so that users who write programs that talk to
> > >one cloud can use the same program with another cloud easily. This
> > >is a *REST API* level of compatibility. We cannot insert cloud-specific
> > >behavior into our client libraries, because not all cloud consumers
> > >will use those libraries to talk to the services. Similarly, we
> > >can't put the logic in the test suite, because that defeats the
> > >entire purpose of making the APIs interoperable. For this level of
> > >compatibility to work, we need well-defined APIs, with a long support
> > >period, that work the same no matter how the cloud is deployed. We
> > >need the entire community to support this effort. From what I can
> > >tell, that is going to require some changes to the current Glance
> > >API to meet the requirements. I'll list those requirements, and I
> > >hope we can discuss them to a degree that ensures everyone understands
> > >them. I don't want this email thread to 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Doug Hellmann]

Excerpts from Clint Byrum's message of 2015-09-14 13:25:43 -0700:
> Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:
> > Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> > > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > > >
> > > >After having some conversations with folks at the Ops Midcycle a
> > > >few weeks ago, and observing some of the more recent email threads
> > > >related to glance, glance-store, the client, and the API, I spent
> > > >last week contacting a few of you individually to learn more about
> > > >some of the issues confronting the Glance team. I had some very
> > > >frank, but I think constructive, conversations with all of you about
> > > >the issues as you see them. As promised, this is the public email
> > > >thread to discuss what I found, and to see if we can agree on what
> > > >the Glance team should be focusing on going into the Mitaka summit
> > > >and development cycle and how the rest of the community can support
> > > >you in those efforts.
> > > >
> > > >I apologize for the length of this email, but there's a lot to go
> > > >over. I've identified 2 high priority items that I think are critical
> > > >for the team to be focusing on starting right away in order to use
> > > >the upcoming summit time effectively. I will also describe several
> > > >other issues that need to be addressed but that are less immediately
> > > >critical. First the high priority items:
> > > >
> > > >1. Resolve the situation preventing the DefCore committee from
> > > >   including image upload capabilities in the tests used for trademark
> > > >   and interoperability validation.
> > > >
> > > >2. Follow through on the original commitment of the project to
> > > >   provide an image API by completing the integration work with
> > > >   nova and cinder to ensure V2 API adoption.
> > > 
> > > Hi Doug,
> > > 
> > > First and foremost, I'd like to thank you for taking the time to dig
> > > into these issues, and for reaching out to the community seeking for
> > > information and a better understanding of what the real issues are. I
> > > can imagine how much time you had to dedicate on this and I'm glad you
> > > did.
> > > 
> > > Now, to your email, I very much agree with the priorities you
> > > mentioned above and I'd like for, whomever will win Glance's PTL
> > > election, to bring focus back on that.
> > > 
> > > Please, find some comments in-line for each point:
> > > 
> > > >
> > > >I. DefCore
> > > >
> > > >The primary issue that attracted my attention was the fact that
> > > >DefCore cannot currently include an image upload API in its
> > > >interoperability test suite, and therefore we do not have a way to
> > > >ensure interoperability between clouds for users or for trademark
> > > >use. The DefCore process has been long, and at times confusing,
> > > >even to those of us following it sort of closely. It's not entirely
> > > >surprising that some projects haven't been following the whole time,
> > > >or aren't aware of exactly what the whole thing means. I have
> > > >proposed a cross-project summit session for the Mitaka summit to
> > > >address this need for communication more broadly, but I'll try to
> > > >summarize a bit here.
> > > 
> > > +1
> > > 
> > > I think it's quite sad that some projects, especially those considered
> > > to be part of the `starter-kit:compute`[0], don't follow closely
> > > what's going on in DefCore. I personally consider this a task PTLs
> > > should incorporate in their role duties. I'm glad you proposed such
> > > session, I hope it'll help raising awareness of this effort and it'll
> > > help moving things forward on that front.
> > 
> > Until fairly recently a lot of the discussion was around process
> > and priorities for the DefCore committee. Now that those things are
> > settled, and we have some approved policies, it's time to engage
> > more fully.  I'll be working during Mitaka to improve the two-way
> > communication.
> > 
> > > 
> > > >
> > > >DefCore is using automated tests, combined with business policies,
> > > >to build a set of criteria for allowing trademark use. One of the
> > > >goals of that process is to ensure that all OpenStack deployments
> > > >are interoperable, so that users who write programs that talk to
> > > >one cloud can use the same program with another cloud easily. This
> > > >is a *REST API* level of compatibility. We cannot insert cloud-specific
> > > >behavior into our client libraries, because not all cloud consumers
> > > >will use those libraries to talk to the services. Similarly, we
> > > >can't put the logic in the test suite, because that defeats the
> > > >entire purpose of making the APIs interoperable. For this level of
> > > >compatibility to work, we need well-defined APIs, with a long support
> > > >period, that work the same no matter how the cloud is deployed. We
> > > >need the entire community to support this effort. From what I can
> > > >tell, that 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2015-09-14 13:46:16 -0700:
> Excerpts from Clint Byrum's message of 2015-09-14 13:25:43 -0700:
> > Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:
> > > Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> > > > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > > > >
> > > > >After having some conversations with folks at the Ops Midcycle a
> > > > >few weeks ago, and observing some of the more recent email threads
> > > > >related to glance, glance-store, the client, and the API, I spent
> > > > >last week contacting a few of you individually to learn more about
> > > > >some of the issues confronting the Glance team. I had some very
> > > > >frank, but I think constructive, conversations with all of you about
> > > > >the issues as you see them. As promised, this is the public email
> > > > >thread to discuss what I found, and to see if we can agree on what
> > > > >the Glance team should be focusing on going into the Mitaka summit
> > > > >and development cycle and how the rest of the community can support
> > > > >you in those efforts.
> > > > >
> > > > >I apologize for the length of this email, but there's a lot to go
> > > > >over. I've identified 2 high priority items that I think are critical
> > > > >for the team to be focusing on starting right away in order to use
> > > > >the upcoming summit time effectively. I will also describe several
> > > > >other issues that need to be addressed but that are less immediately
> > > > >critical. First the high priority items:
> > > > >
> > > > >1. Resolve the situation preventing the DefCore committee from
> > > > >   including image upload capabilities in the tests used for trademark
> > > > >   and interoperability validation.
> > > > >
> > > > >2. Follow through on the original commitment of the project to
> > > > >   provide an image API by completing the integration work with
> > > > >   nova and cinder to ensure V2 API adoption.
> > > > 
> > > > Hi Doug,
> > > > 
> > > > First and foremost, I'd like to thank you for taking the time to dig
> > > > into these issues, and for reaching out to the community seeking for
> > > > information and a better understanding of what the real issues are. I
> > > > can imagine how much time you had to dedicate on this and I'm glad you
> > > > did.
> > > > 
> > > > Now, to your email, I very much agree with the priorities you
> > > > mentioned above and I'd like for, whomever will win Glance's PTL
> > > > election, to bring focus back on that.
> > > > 
> > > > Please, find some comments in-line for each point:
> > > > 
> > > > >
> > > > >I. DefCore
> > > > >
> > > > >The primary issue that attracted my attention was the fact that
> > > > >DefCore cannot currently include an image upload API in its
> > > > >interoperability test suite, and therefore we do not have a way to
> > > > >ensure interoperability between clouds for users or for trademark
> > > > >use. The DefCore process has been long, and at times confusing,
> > > > >even to those of us following it sort of closely. It's not entirely
> > > > >surprising that some projects haven't been following the whole time,
> > > > >or aren't aware of exactly what the whole thing means. I have
> > > > >proposed a cross-project summit session for the Mitaka summit to
> > > > >address this need for communication more broadly, but I'll try to
> > > > >summarize a bit here.
> > > > 
> > > > +1
> > > > 
> > > > I think it's quite sad that some projects, especially those considered
> > > > to be part of the `starter-kit:compute`[0], don't follow closely
> > > > what's going on in DefCore. I personally consider this a task PTLs
> > > > should incorporate in their role duties. I'm glad you proposed such
> > > > session, I hope it'll help raising awareness of this effort and it'll
> > > > help moving things forward on that front.
> > > 
> > > Until fairly recently a lot of the discussion was around process
> > > and priorities for the DefCore committee. Now that those things are
> > > settled, and we have some approved policies, it's time to engage
> > > more fully.  I'll be working during Mitaka to improve the two-way
> > > communication.
> > > 
> > > > 
> > > > >
> > > > >DefCore is using automated tests, combined with business policies,
> > > > >to build a set of criteria for allowing trademark use. One of the
> > > > >goals of that process is to ensure that all OpenStack deployments
> > > > >are interoperable, so that users who write programs that talk to
> > > > >one cloud can use the same program with another cloud easily. This
> > > > >is a *REST API* level of compatibility. We cannot insert cloud-specific
> > > > >behavior into our client libraries, because not all cloud consumers
> > > > >will use those libraries to talk to the services. Similarly, we
> > > > >can't put the logic in the test suite, because that defeats the
> > > > >entire purpose of making the APIs interoperable. For this level 

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Monty Taylor]

On 09/15/2015 02:06 AM, Clint Byrum wrote:

Excerpts from Doug Hellmann's message of 2015-09-14 13:46:16 -0700:

Excerpts from Clint Byrum's message of 2015-09-14 13:25:43 -0700:

Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:

Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:

On 14/09/15 08:10 -0400, Doug Hellmann wrote:


After having some conversations with folks at the Ops Midcycle a
few weeks ago, and observing some of the more recent email threads
related to glance, glance-store, the client, and the API, I spent
last week contacting a few of you individually to learn more about
some of the issues confronting the Glance team. I had some very
frank, but I think constructive, conversations with all of you about
the issues as you see them. As promised, this is the public email
thread to discuss what I found, and to see if we can agree on what
the Glance team should be focusing on going into the Mitaka summit
and development cycle and how the rest of the community can support
you in those efforts.

I apologize for the length of this email, but there's a lot to go
over. I've identified 2 high priority items that I think are critical
for the team to be focusing on starting right away in order to use
the upcoming summit time effectively. I will also describe several
other issues that need to be addressed but that are less immediately
critical. First the high priority items:

1. Resolve the situation preventing the DefCore committee from
   including image upload capabilities in the tests used for trademark
   and interoperability validation.

2. Follow through on the original commitment of the project to
   provide an image API by completing the integration work with
   nova and cinder to ensure V2 API adoption.


Hi Doug,

First and foremost, I'd like to thank you for taking the time to dig
into these issues, and for reaching out to the community seeking for
information and a better understanding of what the real issues are. I
can imagine how much time you had to dedicate on this and I'm glad you
did.

Now, to your email, I very much agree with the priorities you
mentioned above and I'd like for, whomever will win Glance's PTL
election, to bring focus back on that.

Please, find some comments in-line for each point:



I. DefCore

The primary issue that attracted my attention was the fact that
DefCore cannot currently include an image upload API in its
interoperability test suite, and therefore we do not have a way to
ensure interoperability between clouds for users or for trademark
use. The DefCore process has been long, and at times confusing,
even to those of us following it sort of closely. It's not entirely
surprising that some projects haven't been following the whole time,
or aren't aware of exactly what the whole thing means. I have
proposed a cross-project summit session for the Mitaka summit to
address this need for communication more broadly, but I'll try to
summarize a bit here.


+1

I think it's quite sad that some projects, especially those considered
to be part of the `starter-kit:compute`[0], don't follow closely
what's going on in DefCore. I personally consider this a task PTLs
should incorporate in their role duties. I'm glad you proposed such
session, I hope it'll help raising awareness of this effort and it'll
help moving things forward on that front.


Until fairly recently a lot of the discussion was around process
and priorities for the DefCore committee. Now that those things are
settled, and we have some approved policies, it's time to engage
more fully.  I'll be working during Mitaka to improve the two-way
communication.





DefCore is using automated tests, combined with business policies,
to build a set of criteria for allowing trademark use. One of the
goals of that process is to ensure that all OpenStack deployments
are interoperable, so that users who write programs that talk to
one cloud can use the same program with another cloud easily. This
is a *REST API* level of compatibility. We cannot insert cloud-specific
behavior into our client libraries, because not all cloud consumers
will use those libraries to talk to the services. Similarly, we
can't put the logic in the test suite, because that defeats the
entire purpose of making the APIs interoperable. For this level of
compatibility to work, we need well-defined APIs, with a long support
period, that work the same no matter how the cloud is deployed. We
need the entire community to support this effort. From what I can
tell, that is going to require some changes to the current Glance
API to meet the requirements. I'll list those requirements, and I
hope we can discuss them to a degree that ensures everyone understands
them. I don't want this email thread to get bogged down in
implementation details or API designs, though, so let's try to keep
the discussion at a somewhat high level, and leave the details for
specs and summit discussions. I do hope you will correct any

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Robert Collins]

On 15 September 2015 at 00:10, Doug Hellmann  wrote:
>
> After having some conversations with folks at the Ops Midcycle a
> few weeks ago, and observing some of the more recent email threads
> related to glance, glance-store, the client, and the API, I spent
> last week contacting a few of you individually to learn more about
> some of the issues confronting the Glance team. I had some very
> frank, but I think constructive, conversations with all of you about
> the issues as you see them. As promised, this is the public email
> thread to discuss what I found, and to see if we can agree on what
> the Glance team should be focusing on going into the Mitaka summit
> and development cycle and how the rest of the community can support
> you in those efforts.

Thanks for getting the ball rolling here. I won't go down the design
rathole here - but I will say that I think that the goals you've
identified are well within our [all OpenStack contributors] power to
address during the next cycle, and I'm going to help facilitate that
however I can.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Flavio Percoco]

On 14/09/15 19:06 +0200, Monty Taylor wrote:

On 09/14/2015 02:41 PM, Flavio Percoco wrote:

On 14/09/15 08:10 -0400, Doug Hellmann wrote:

[snip]

The task API is also not widely deployed, so its adoption for DefCore
is problematic. If we provide a clear technical direction that this
API is preferred, that may overcome the lack of adoption, but the
current task API seems to have technical issues that make it
fundamentally unsuitable for DefCore consideration. While the task
API addresses the problem of a denial of service, and includes
useful features such as processing of the image during import, it
is not strongly enough defined in its current form to be interoperable.
Because it's a generic API, the caller must know how to fully
construct each task, and know what task types are supported in the
first place. There is only one "import" task type supported in the
Glance code repository right now, but it is not clear that "import"
always uses the same arguments, or interprets them in the same way.
For example, the upstream documentation [1] describes a task that
appears to use a URL as source, while the Rackspace documentation [2]
describes a task that appears to take a swift storage location.
I wasn't able to find JSONSchema validation for the "input" blob
portion of the task in the code [3], though that may happen down
inside the task implementation itself somewhere.



The above sounds pretty accurate as there's currently just 1 flow that
can be triggered (the import flow) and that accepts an input, which is
a json. As I mentioned above, I don't believe tasks should be part of
the public API and this is yet another reason why I think so. The
tasks API is not well defined as there's, currently, not good way to
define the expected input in a backwards compatible way and to provide
all the required validation.

I like having tasks in Glance, despite my comments above - but I like
them for cloud usage and not public usage.


I like them much more if they're not public facing. They're not BAD - 
they just don't have an end-user semantic.


++


As far as Rackspace's docs/endpoint goes, I'd assume this is an error
in their documetation since Glance currently doesn't allow[0] for
swift URLs to be imported (not even in juno[1]).

[0]
http://git.openstack.org/cgit/openstack/glance/tree/glance/common/scripts/utils.py#n84

[1]
http://git.openstack.org/cgit/openstack/glance/tree/glance/common/scripts/utils.py?h=stable/juno#n83


Nope. You MUST upload the image to swift and then provide a swift 
location. (Infra does this in production, I promise it's the only 
thing that works)


mmh. That's not vanilla Glance as you can see from the link pointing
to the code in juno (unless it's a previous version w/ different code
that I don't know off). This[0] is the commit where the first version of
the import task was added - it didn't use taskflow back then. I'll
leave someone from Rackspace to chime in here since I don't have any
other context.

Regardless, the above is the exact example of why I believe the task
API should not be considered the end-user, supported, way to upload
images. Requiring users to have an external (or in cloud), public (or
accessible from the cloud) source, were images are going to be
downloaded from, to create an image is far from ideal, usable and,
AFAICR, it was never intended as the recommended upload workflow but
an additional one, which should, arguably, be used only by admins.

[0] 
https://git.openstack.org/cgit/openstack/glance/commit/?id=186991bb9d2b8c568f3e9a0a89744fd6001ec74a

[snip]

Flavio

--
@flaper87
Flavio Percoco


pgpoxHoRoxMHE.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[James E. Blair]

Thierry Carrez  writes:

> Doug Hellmann wrote:
>> [...]
>> 1. Resolve the situation preventing the DefCore committee from
>>including image upload capabilities in the tests used for trademark
>>and interoperability validation.
>> 
>> 2. Follow through on the original commitment of the project to
>>provide an image API by completing the integration work with
>>nova and cinder to ensure V2 API adoption.
>> [...]
>
> Thanks Doug for taking the time to dive into Glance and to write this
> email. I agree with your top two priorities as being a good summary of
> what the "rest of the community" expects the Glance leadership to focus
> on in the very short term.

Agreed and thanks.  I'm also excited by the conversation this has
prompted and am optimistic that we will have agreement at the summit on
a way forward.

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] proposed priorities for Mitaka

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> >
> >After having some conversations with folks at the Ops Midcycle a
> >few weeks ago, and observing some of the more recent email threads
> >related to glance, glance-store, the client, and the API, I spent
> >last week contacting a few of you individually to learn more about
> >some of the issues confronting the Glance team. I had some very
> >frank, but I think constructive, conversations with all of you about
> >the issues as you see them. As promised, this is the public email
> >thread to discuss what I found, and to see if we can agree on what
> >the Glance team should be focusing on going into the Mitaka summit
> >and development cycle and how the rest of the community can support
> >you in those efforts.
> >
> >I apologize for the length of this email, but there's a lot to go
> >over. I've identified 2 high priority items that I think are critical
> >for the team to be focusing on starting right away in order to use
> >the upcoming summit time effectively. I will also describe several
> >other issues that need to be addressed but that are less immediately
> >critical. First the high priority items:
> >
> >1. Resolve the situation preventing the DefCore committee from
> >   including image upload capabilities in the tests used for trademark
> >   and interoperability validation.
> >
> >2. Follow through on the original commitment of the project to
> >   provide an image API by completing the integration work with
> >   nova and cinder to ensure V2 API adoption.
> 
> Hi Doug,
> 
> First and foremost, I'd like to thank you for taking the time to dig
> into these issues, and for reaching out to the community seeking for
> information and a better understanding of what the real issues are. I
> can imagine how much time you had to dedicate on this and I'm glad you
> did.
> 
> Now, to your email, I very much agree with the priorities you
> mentioned above and I'd like for, whomever will win Glance's PTL
> election, to bring focus back on that.
> 
> Please, find some comments in-line for each point:
> 
> >
> >I. DefCore
> >
> >The primary issue that attracted my attention was the fact that
> >DefCore cannot currently include an image upload API in its
> >interoperability test suite, and therefore we do not have a way to
> >ensure interoperability between clouds for users or for trademark
> >use. The DefCore process has been long, and at times confusing,
> >even to those of us following it sort of closely. It's not entirely
> >surprising that some projects haven't been following the whole time,
> >or aren't aware of exactly what the whole thing means. I have
> >proposed a cross-project summit session for the Mitaka summit to
> >address this need for communication more broadly, but I'll try to
> >summarize a bit here.
> 
> +1
> 
> I think it's quite sad that some projects, especially those considered
> to be part of the `starter-kit:compute`[0], don't follow closely
> what's going on in DefCore. I personally consider this a task PTLs
> should incorporate in their role duties. I'm glad you proposed such
> session, I hope it'll help raising awareness of this effort and it'll
> help moving things forward on that front.

Until fairly recently a lot of the discussion was around process
and priorities for the DefCore committee. Now that those things are
settled, and we have some approved policies, it's time to engage
more fully.  I'll be working during Mitaka to improve the two-way
communication.

> 
> >
> >DefCore is using automated tests, combined with business policies,
> >to build a set of criteria for allowing trademark use. One of the
> >goals of that process is to ensure that all OpenStack deployments
> >are interoperable, so that users who write programs that talk to
> >one cloud can use the same program with another cloud easily. This
> >is a *REST API* level of compatibility. We cannot insert cloud-specific
> >behavior into our client libraries, because not all cloud consumers
> >will use those libraries to talk to the services. Similarly, we
> >can't put the logic in the test suite, because that defeats the
> >entire purpose of making the APIs interoperable. For this level of
> >compatibility to work, we need well-defined APIs, with a long support
> >period, that work the same no matter how the cloud is deployed. We
> >need the entire community to support this effort. From what I can
> >tell, that is going to require some changes to the current Glance
> >API to meet the requirements. I'll list those requirements, and I
> >hope we can discuss them to a degree that ensures everyone understands
> >them. I don't want this email thread to get bogged down in
> >implementation details or API designs, though, so let's try to keep
> >the discussion at a somewhat high level, and leave the details for
> >specs and summit discussions. I do hope you will correct any
> >misunderstandings or