Re: [openstack-dev] [heat] [glance] Heater Proposal

[Georgy Okrokvertskhov]

Hi,

I think BP is a right way to organize this. I will submit BP for metadata
service from our side too.

Thanks
Georgy


On Wed, Dec 11, 2013 at 3:53 PM, Randall Burt wrote:

> On Dec 11, 2013, at 5:44 PM, Georgy Okrokvertskhov <
> gokrokvertsk...@mirantis.com>
>  wrote:
>
> > Hi,
> >
> > To keep this thread alive I would like to share the small screencast
> I've recorded for Murano Metadata repository. I would like to share with
> you what we have in Murano and start a conversation about metadata
> repository development in OpenStack. Here is a link to screencast
> http://www.youtube.com/watch?v=Yi4gC4ZhvPg Here is a link  to a detailed
> specification of PoC for metadata repository currently implemented in
> Murano.
> >
> > There is an etherpad (here) for new MetadataRepository design we started
> to write after lesson learn phase of PoC. This is a future version of
> repository we want to have. This proposal can be used as an initial basis
> for metadata repository design conversation.
> >
> > It will be great if we start conversation with Glance team to understand
> how this work can be organized. As it was revealed in this thread, the most
> probable candidate for metadata repository service implementation is Glance
> program.
> >
> > Thanks,
> > Georgy
>
> Thanks for the link and info. I think the general consensus is this
> belongs in Glance, however I think details are being deferred until the
> mid-summit meet up in Washington D.C. (I could be totally wrong about
> this). In any case, I think I'll also start converting the existing HeatR
> blueprints to Glance ones. Perhaps it would be a good idea at this point to
> propose specific blueprints and have further ML discussions focused on
> specific changes?
>
> > On Mon, Dec 9, 2013 at 3:24 AM, Thierry Carrez 
> wrote:
> > Vishvananda Ishaya wrote:
> > > On Dec 6, 2013, at 10:07 AM, Georgy Okrokvertskhov
> > > mailto:gokrokvertsk...@mirantis.com>>
> wrote:
> > >
> > >> I am really inspired by this thread. Frankly saying, Glance for Murano
> > >> was a kind of sacred entity, as it is a service with a long history in
> > >> OpenStack.  We even did not think in the direction of changing Glance.
> > >> Spending a night with these ideas, I am kind of having a dream about
> > >> unified catalog where the full range of different entities are
> > >> presented. Just imagine that we have everything as  first class
> > >> citizens of catalog treated equally: single VM (image), Heat template
> > >> (fixed number of VMs\ autoscaling groups), Murano Application
> > >> (generated Heat templates), Solum assemblies
> > >>
> > >> Projects like Solum will highly benefit from this catalog as it can
> > >> use all varieties of VM configurations talking with one service.
> > >> This catalog will be able not just list all possible deployable
> > >> entities but can be also a registry for already deployed
> > >> configurations. This is perfectly aligned with the goal for catalog to
> > >> be a kind of market place which provides billing information too.
> > >>
> > >> OpenStack users also will benefit from this as they will have the
> > >> unified approach for manage deployments and deployable entities.
> > >>
> > >> I doubt that it could be done by a single team. But if all teams join
> > >> this effort we can do this. From my perspective, this could be a part
> > >> of Glance program and it is not necessary to add a new program for
> > >> that. As it was mentioned earlier in this thread an idea of market
> > >> place for images in Glance was here for some time. I think we can
> > >> extend it to the idea of creating a marketplace for a deployable
> > >> entity regardless of the way of deployment. As Glance is a core
> > >> project which means it always exist in OpenStack deployment it makes
> > >> sense to as a central catalog for everything.
> > >
> > > +1
> >
> > +1 too.
> >
> > I don't think that Glance is collapsing under its current complexity
> > yet, so extending Glance to a general catalog service that can serve
> > more than just reference VM images makes sense IMHO.
> >
> > --
> > Thierry Carrez (ttx)
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> > --
> > Georgy Okrokvertskhov
> > Technical Program Manager,
> > Cloud and Infrastructure Services,
> > Mirantis
> > http://www.mirantis.com
> > Tel. +1 650 963 9828
> > Mob. +1 650 996 3284
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Georgy Okrokvertskhov
Technical Program Manager,
Cloud and Infrastructure Servi

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 11, 2013, at 5:44 PM, Georgy Okrokvertskhov 

 wrote:

> Hi,
> 
> To keep this thread alive I would like to share the small screencast I've 
> recorded for Murano Metadata repository. I would like to share with you what 
> we have in Murano and start a conversation about metadata repository 
> development in OpenStack. Here is a link to screencast 
> http://www.youtube.com/watch?v=Yi4gC4ZhvPg Here is a link  to a detailed 
> specification of PoC for metadata repository currently implemented in Murano.
> 
> There is an etherpad (here) for new MetadataRepository design we started to 
> write after lesson learn phase of PoC. This is a future version of repository 
> we want to have. This proposal can be used as an initial basis for metadata 
> repository design conversation.
> 
> It will be great if we start conversation with Glance team to understand how 
> this work can be organized. As it was revealed in this thread, the most 
> probable candidate for metadata repository service implementation is Glance 
> program. 
> 
> Thanks,
> Georgy

Thanks for the link and info. I think the general consensus is this belongs in 
Glance, however I think details are being deferred until the mid-summit meet up 
in Washington D.C. (I could be totally wrong about this). In any case, I think 
I'll also start converting the existing HeatR blueprints to Glance ones. 
Perhaps it would be a good idea at this point to propose specific blueprints 
and have further ML discussions focused on specific changes?

> On Mon, Dec 9, 2013 at 3:24 AM, Thierry Carrez  wrote:
> Vishvananda Ishaya wrote:
> > On Dec 6, 2013, at 10:07 AM, Georgy Okrokvertskhov
> > mailto:gokrokvertsk...@mirantis.com>> wrote:
> >
> >> I am really inspired by this thread. Frankly saying, Glance for Murano
> >> was a kind of sacred entity, as it is a service with a long history in
> >> OpenStack.  We even did not think in the direction of changing Glance.
> >> Spending a night with these ideas, I am kind of having a dream about
> >> unified catalog where the full range of different entities are
> >> presented. Just imagine that we have everything as  first class
> >> citizens of catalog treated equally: single VM (image), Heat template
> >> (fixed number of VMs\ autoscaling groups), Murano Application
> >> (generated Heat templates), Solum assemblies
> >>
> >> Projects like Solum will highly benefit from this catalog as it can
> >> use all varieties of VM configurations talking with one service.
> >> This catalog will be able not just list all possible deployable
> >> entities but can be also a registry for already deployed
> >> configurations. This is perfectly aligned with the goal for catalog to
> >> be a kind of market place which provides billing information too.
> >>
> >> OpenStack users also will benefit from this as they will have the
> >> unified approach for manage deployments and deployable entities.
> >>
> >> I doubt that it could be done by a single team. But if all teams join
> >> this effort we can do this. From my perspective, this could be a part
> >> of Glance program and it is not necessary to add a new program for
> >> that. As it was mentioned earlier in this thread an idea of market
> >> place for images in Glance was here for some time. I think we can
> >> extend it to the idea of creating a marketplace for a deployable
> >> entity regardless of the way of deployment. As Glance is a core
> >> project which means it always exist in OpenStack deployment it makes
> >> sense to as a central catalog for everything.
> >
> > +1
> 
> +1 too.
> 
> I don't think that Glance is collapsing under its current complexity
> yet, so extending Glance to a general catalog service that can serve
> more than just reference VM images makes sense IMHO.
> 
> --
> Thierry Carrez (ttx)
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> -- 
> Georgy Okrokvertskhov
> Technical Program Manager,
> Cloud and Infrastructure Services,
> Mirantis
> http://www.mirantis.com
> Tel. +1 650 963 9828
> Mob. +1 650 996 3284
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Georgy Okrokvertskhov]

Hi,

To keep this thread alive I would like to share the small screencast I've
recorded for Murano Metadata repository. I would like to share with you
what we have in Murano and start a conversation about metadata repository
development in OpenStack. Here is a link to screencast
http://www.youtube.com/watch?v=Yi4gC4ZhvPg Here is a
link
to a detailed specification of PoC for metadata repository currently
implemented in Murano.

There is an etherpad (here )
for new MetadataRepository design we started to write after lesson learn
phase of PoC. This is a future version of repository we want to have. This
proposal can be used as an initial basis for metadata repository design
conversation.

It will be great if we start conversation with Glance team to understand
how this work can be organized. As it was revealed in this thread, the most
probable candidate for metadata repository service implementation is Glance
program.

Thanks,
Georgy


On Mon, Dec 9, 2013 at 3:24 AM, Thierry Carrez wrote:

> Vishvananda Ishaya wrote:
> > On Dec 6, 2013, at 10:07 AM, Georgy Okrokvertskhov
> > mailto:gokrokvertsk...@mirantis.com>>
> wrote:
> >
> >> I am really inspired by this thread. Frankly saying, Glance for Murano
> >> was a kind of sacred entity, as it is a service with a long history in
> >> OpenStack.  We even did not think in the direction of changing Glance.
> >> Spending a night with these ideas, I am kind of having a dream about
> >> unified catalog where the full range of different entities are
> >> presented. Just imagine that we have everything as  first class
> >> citizens of catalog treated equally: single VM (image), Heat template
> >> (fixed number of VMs\ autoscaling groups), Murano Application
> >> (generated Heat templates), Solum assemblies
> >>
> >> Projects like Solum will highly benefit from this catalog as it can
> >> use all varieties of VM configurations talking with one service.
> >> This catalog will be able not just list all possible deployable
> >> entities but can be also a registry for already deployed
> >> configurations. This is perfectly aligned with the goal for catalog to
> >> be a kind of market place which provides billing information too.
> >>
> >> OpenStack users also will benefit from this as they will have the
> >> unified approach for manage deployments and deployable entities.
> >>
> >> I doubt that it could be done by a single team. But if all teams join
> >> this effort we can do this. From my perspective, this could be a part
> >> of Glance program and it is not necessary to add a new program for
> >> that. As it was mentioned earlier in this thread an idea of market
> >> place for images in Glance was here for some time. I think we can
> >> extend it to the idea of creating a marketplace for a deployable
> >> entity regardless of the way of deployment. As Glance is a core
> >> project which means it always exist in OpenStack deployment it makes
> >> sense to as a central catalog for everything.
> >
> > +1
>
> +1 too.
>
> I don't think that Glance is collapsing under its current complexity
> yet, so extending Glance to a general catalog service that can serve
> more than just reference VM images makes sense IMHO.
>
> --
> Thierry Carrez (ttx)
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Georgy Okrokvertskhov
Technical Program Manager,
Cloud and Infrastructure Services,
Mirantis
http://www.mirantis.com
Tel. +1 650 963 9828
Mob. +1 650 996 3284
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Thierry Carrez]

Vishvananda Ishaya wrote:
> On Dec 6, 2013, at 10:07 AM, Georgy Okrokvertskhov
> mailto:gokrokvertsk...@mirantis.com>> wrote:
> 
>> I am really inspired by this thread. Frankly saying, Glance for Murano
>> was a kind of sacred entity, as it is a service with a long history in
>> OpenStack.  We even did not think in the direction of changing Glance.
>> Spending a night with these ideas, I am kind of having a dream about
>> unified catalog where the full range of different entities are
>> presented. Just imagine that we have everything as  first class
>> citizens of catalog treated equally: single VM (image), Heat template
>> (fixed number of VMs\ autoscaling groups), Murano Application
>> (generated Heat templates), Solum assemblies
>>
>> Projects like Solum will highly benefit from this catalog as it can
>> use all varieties of VM configurations talking with one service.
>> This catalog will be able not just list all possible deployable
>> entities but can be also a registry for already deployed
>> configurations. This is perfectly aligned with the goal for catalog to
>> be a kind of market place which provides billing information too.
>>
>> OpenStack users also will benefit from this as they will have the
>> unified approach for manage deployments and deployable entities.
>>
>> I doubt that it could be done by a single team. But if all teams join
>> this effort we can do this. From my perspective, this could be a part
>> of Glance program and it is not necessary to add a new program for
>> that. As it was mentioned earlier in this thread an idea of market
>> place for images in Glance was here for some time. I think we can
>> extend it to the idea of creating a marketplace for a deployable
>> entity regardless of the way of deployment. As Glance is a core
>> project which means it always exist in OpenStack deployment it makes
>> sense to as a central catalog for everything.
> 
> +1 

+1 too.

I don't think that Glance is collapsing under its current complexity
yet, so extending Glance to a general catalog service that can serve
more than just reference VM images makes sense IMHO.

-- 
Thierry Carrez (ttx)



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Jay Pipes]

On 12/06/2013 08:55 PM, Clint Byrum wrote:

Excerpts from Jay Pipes's message of 2013-12-06 16:46:24 -0800:

On 12/06/2013 01:38 PM, Clint Byrum wrote:

Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:

On 12/05/2013 04:25 PM, Clint Byrum wrote:

Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49
-0800:

Excerpts from Randall Burt's message of 2013-12-05 09:05:44
-0800:

On Dec 5, 2013, at 10:10 AM, Clint Byrum  wrote:


Excerpts from Monty Taylor's message of 2013-12-04
17:54:45 -0800:

Why not just use glance?



I've asked that question a few times, and I think I can
collate the responses I've received below. I think
enhancing glance to do these things is on the table:

1. Glance is for big blobs of data not tiny templates. 2.
Versioning of a single resource is desired. 3.
Tagging/classifying/listing/sorting 4. Glance is designed
to expose the uploaded blobs to nova, not users

My responses:

1: Irrelevant. Smaller things will fit in it just fine.


Fitting is one thing, optimizations around particular
assumptions about the size of data and the frequency of
reads/writes might be an issue, but I admit to ignorance
about those details in Glance.



Optimizations can be improved for various use cases. The
design, however, has no assumptions that I know about that
would invalidate storing blobs of yaml/json vs. blobs of
kernel/qcow2/raw image.


I think we are getting out into the weeds a little bit here. It
is important to think about these apis in terms of what they
actually do, before the decision of combining them or not can
be made.

I think of HeatR as a template storage service, it provides
extra data and operations on templates. HeatR should not care
about how those templates are stored. Glance is an image
storage service, it provides extra data and operations on
images (not blobs), and it happens to use swift as a backend.

If HeatR and Glance were combined, it would result in taking
two very different types of data (template metadata vs image
metadata) and mashing them into one service. How would adding
the complexity of HeatR benefit Glance, when they are dealing
with conceptually two very different types of data? For
instance, should a template ever care about the field "minRam"
that is stored with an image? Combining them adds a huge
development complexity with a very small operations payoff, and
so Openstack is already so operationally complex that HeatR as
a separate service would be knowledgeable. Only clients of Heat
will ever care about data and operations on templates, so I
move that HeatR becomes it's own service, or becomes part of
Heat.



I spoke at length via G+ with Randall and Tim about this earlier
today. I think I understand the impetus for all of this a little
better now.

Basically what I'm suggesting is that Glance is only narrow in
scope because that was the only object that OpenStack needed a
catalog for before now.

However, the overlap between a catalog of images and a catalog
of templates is quite comprehensive. The individual fields that
matter to images are different than the ones that matter to
templates, but that is a really minor detail isn't it?

I would suggest that Glance be slightly expanded in scope to be
an object catalog. Each object type can have its own set of
fields that matter to it.

This doesn't have to be a minor change to glance to still have
many advantages over writing something from scratch and asking
people to deploy another service that is 99% the same as Glance.


My suggestion for long-term architecture would be to use Murano
for catalog/metadata information (for images/templates/whatever)
and move the block-streaming drivers into Cinder, and get rid of
the Glance project entirely. Murano would then become the
catalog/registry of objects in the OpenStack world, Cinder would be
the thing that manages and streams blocks of data or block devices,
and Glance could go away. Imagine it... OpenStack actually
*reducing* the number of projects instead of expanding! :)


Have we not learned our lesson with Nova-Net/Neutron yet? Rewrites
of existing functionality are painful.


I said *long-term* above, Clint.


The Murano-concerned people have already stated they are starting
over on that catalog.

I suggest they start over by expanding Glance's catalog. If the
block streaming bits of Glance need to move somewhere else, that
sounds like a completely separate concern that distracts from this
point.


Yes, the block streaming bits of Glance are separate -- sorry for
distracting from your point. However, Glance's architecture actually
went from having the registry and streaming pieces separated to having
the two pieces more closely interwoven in the last two release cycles.
It's harder now, IMO, to separate out the "registry" aspects of Glance
from the block management pieces. Which is why I suggested using Murano
as the catalog since there is already a separate catalog sub-project in
Murano and the Murano devs are familiar with this

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Jay Pipes's message of 2013-12-06 16:46:24 -0800:
> On 12/06/2013 01:38 PM, Clint Byrum wrote:
> > Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
> >> On 12/05/2013 04:25 PM, Clint Byrum wrote:
> >>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49
> >>> -0800:
> > Excerpts from Randall Burt's message of 2013-12-05 09:05:44
> > -0800:
> >> On Dec 5, 2013, at 10:10 AM, Clint Byrum  >> fewbar.com> wrote:
> >>
> >>> Excerpts from Monty Taylor's message of 2013-12-04
> >>> 17:54:45 -0800:
>  Why not just use glance?
> 
> >>>
> >>> I've asked that question a few times, and I think I can
> >>> collate the responses I've received below. I think
> >>> enhancing glance to do these things is on the table:
> >>>
> >>> 1. Glance is for big blobs of data not tiny templates. 2.
> >>> Versioning of a single resource is desired. 3.
> >>> Tagging/classifying/listing/sorting 4. Glance is designed
> >>> to expose the uploaded blobs to nova, not users
> >>>
> >>> My responses:
> >>>
> >>> 1: Irrelevant. Smaller things will fit in it just fine.
> >>
> >> Fitting is one thing, optimizations around particular
> >> assumptions about the size of data and the frequency of
> >> reads/writes might be an issue, but I admit to ignorance
> >> about those details in Glance.
> >>
> >
> > Optimizations can be improved for various use cases. The
> > design, however, has no assumptions that I know about that
> > would invalidate storing blobs of yaml/json vs. blobs of
> > kernel/qcow2/raw image.
> 
>  I think we are getting out into the weeds a little bit here. It
>  is important to think about these apis in terms of what they
>  actually do, before the decision of combining them or not can
>  be made.
> 
>  I think of HeatR as a template storage service, it provides
>  extra data and operations on templates. HeatR should not care
>  about how those templates are stored. Glance is an image
>  storage service, it provides extra data and operations on
>  images (not blobs), and it happens to use swift as a backend.
> 
>  If HeatR and Glance were combined, it would result in taking
>  two very different types of data (template metadata vs image
>  metadata) and mashing them into one service. How would adding
>  the complexity of HeatR benefit Glance, when they are dealing
>  with conceptually two very different types of data? For
>  instance, should a template ever care about the field "minRam"
>  that is stored with an image? Combining them adds a huge
>  development complexity with a very small operations payoff, and
>  so Openstack is already so operationally complex that HeatR as
>  a separate service would be knowledgeable. Only clients of Heat
>  will ever care about data and operations on templates, so I
>  move that HeatR becomes it's own service, or becomes part of
>  Heat.
> 
> >>>
> >>> I spoke at length via G+ with Randall and Tim about this earlier
> >>> today. I think I understand the impetus for all of this a little
> >>> better now.
> >>>
> >>> Basically what I'm suggesting is that Glance is only narrow in
> >>> scope because that was the only object that OpenStack needed a
> >>> catalog for before now.
> >>>
> >>> However, the overlap between a catalog of images and a catalog
> >>> of templates is quite comprehensive. The individual fields that
> >>> matter to images are different than the ones that matter to
> >>> templates, but that is a really minor detail isn't it?
> >>>
> >>> I would suggest that Glance be slightly expanded in scope to be
> >>> an object catalog. Each object type can have its own set of
> >>> fields that matter to it.
> >>>
> >>> This doesn't have to be a minor change to glance to still have
> >>> many advantages over writing something from scratch and asking
> >>> people to deploy another service that is 99% the same as Glance.
> >>
> >> My suggestion for long-term architecture would be to use Murano
> >> for catalog/metadata information (for images/templates/whatever)
> >> and move the block-streaming drivers into Cinder, and get rid of
> >> the Glance project entirely. Murano would then become the
> >> catalog/registry of objects in the OpenStack world, Cinder would be
> >> the thing that manages and streams blocks of data or block devices,
> >> and Glance could go away. Imagine it... OpenStack actually
> >> *reducing* the number of projects instead of expanding! :)
> >
> > Have we not learned our lesson with Nova-Net/Neutron yet? Rewrites
> > of existing functionality are painful.
> 
> I said *long-term* above, Clint.
> 
> > The Murano-concerned people have already stated they are starting
> > over on that catalog.
> >
> > I suggest they start over by expanding Glance's catalog. If the
> > block streaming bits of Glance n

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Jay Pipes]

On 12/06/2013 01:38 PM, Clint Byrum wrote:

Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:

On 12/05/2013 04:25 PM, Clint Byrum wrote:

Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49
-0800:

Excerpts from Randall Burt's message of 2013-12-05 09:05:44
-0800:

On Dec 5, 2013, at 10:10 AM, Clint Byrum  wrote:


Excerpts from Monty Taylor's message of 2013-12-04
17:54:45 -0800:

Why not just use glance?



I've asked that question a few times, and I think I can
collate the responses I've received below. I think
enhancing glance to do these things is on the table:

1. Glance is for big blobs of data not tiny templates. 2.
Versioning of a single resource is desired. 3.
Tagging/classifying/listing/sorting 4. Glance is designed
to expose the uploaded blobs to nova, not users

My responses:

1: Irrelevant. Smaller things will fit in it just fine.


Fitting is one thing, optimizations around particular
assumptions about the size of data and the frequency of
reads/writes might be an issue, but I admit to ignorance
about those details in Glance.



Optimizations can be improved for various use cases. The
design, however, has no assumptions that I know about that
would invalidate storing blobs of yaml/json vs. blobs of
kernel/qcow2/raw image.


I think we are getting out into the weeds a little bit here. It
is important to think about these apis in terms of what they
actually do, before the decision of combining them or not can
be made.

I think of HeatR as a template storage service, it provides
extra data and operations on templates. HeatR should not care
about how those templates are stored. Glance is an image
storage service, it provides extra data and operations on
images (not blobs), and it happens to use swift as a backend.

If HeatR and Glance were combined, it would result in taking
two very different types of data (template metadata vs image
metadata) and mashing them into one service. How would adding
the complexity of HeatR benefit Glance, when they are dealing
with conceptually two very different types of data? For
instance, should a template ever care about the field "minRam"
that is stored with an image? Combining them adds a huge
development complexity with a very small operations payoff, and
so Openstack is already so operationally complex that HeatR as
a separate service would be knowledgeable. Only clients of Heat
will ever care about data and operations on templates, so I
move that HeatR becomes it's own service, or becomes part of
Heat.



I spoke at length via G+ with Randall and Tim about this earlier
today. I think I understand the impetus for all of this a little
better now.

Basically what I'm suggesting is that Glance is only narrow in
scope because that was the only object that OpenStack needed a
catalog for before now.

However, the overlap between a catalog of images and a catalog
of templates is quite comprehensive. The individual fields that
matter to images are different than the ones that matter to
templates, but that is a really minor detail isn't it?

I would suggest that Glance be slightly expanded in scope to be
an object catalog. Each object type can have its own set of
fields that matter to it.

This doesn't have to be a minor change to glance to still have
many advantages over writing something from scratch and asking
people to deploy another service that is 99% the same as Glance.


My suggestion for long-term architecture would be to use Murano
for catalog/metadata information (for images/templates/whatever)
and move the block-streaming drivers into Cinder, and get rid of
the Glance project entirely. Murano would then become the
catalog/registry of objects in the OpenStack world, Cinder would be
the thing that manages and streams blocks of data or block devices,
and Glance could go away. Imagine it... OpenStack actually
*reducing* the number of projects instead of expanding! :)


Have we not learned our lesson with Nova-Net/Neutron yet? Rewrites
of existing functionality are painful.


I said *long-term* above, Clint.


The Murano-concerned people have already stated they are starting
over on that catalog.

I suggest they start over by expanding Glance's catalog. If the
block streaming bits of Glance need to move somewhere else, that
sounds like a completely separate concern that distracts from this
point.


Yes, the block streaming bits of Glance are separate -- sorry for
distracting from your point. However, Glance's architecture actually
went from having the registry and streaming pieces separated to having
the two pieces more closely interwoven in the last two release cycles.
It's harder now, IMO, to separate out the "registry" aspects of Glance 
from the block management pieces. Which is why I suggested using Murano 
as the catalog since there is already a separate catalog sub-project in 
Murano and the Murano devs are familiar with this space.


Please try not to shoot the messenger here -- I'm only bringing up my 
thoughts on where the var

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Georgy Okrokvertskhov]

That is great. How this work will be coordinated? I just want to be sure
that all assets are covered.

Thanks
Georgy


On Fri, Dec 6, 2013 at 3:15 PM, Randall Burt wrote:

> On Dec 6, 2013, at 5:04 PM, Clint Byrum 
>  wrote:
>
> > Excerpts from Randall Burt's message of 2013-12-06 14:43:05 -0800:
> >> I too have warmed to this idea but wonder about the actual
> implementation around it. While I like where Edmund is going with this, I
> wonder if it wouldn't be valuable in the short-to-mid-term (I/J) to just
> add /templates to Glance (/assemblies, /applications, etc) along side
> /images.  Initially, we could have separate endpoints and data structures
> for these different asset types, refactoring the easy bits along the way
> and leveraging the existing data storage and caching bits, but leaving more
> disruptive changes alone. That can get the functionality going, prove some
> concepts, and allow all of the interested parties to better plan a more
> general v3 api.
> >>
> >
> > +1 on bolting the different views for things on as new v2 pieces instead
> > of trying to solve the API genericism immediately.
> >
> > I would strive to make this a facade, and start immediately on making
> > Glance more generic under the hood.  Otherwise these will just end up
> > as silos inside Glance instead of silos inside OpenStack.
>
> Totally agreed. Where it makes sense to refactor we should do that rather
> than implementing essentially different services underneath.
>
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Georgy Okrokvertskhov
Technical Program Manager,
Cloud and Infrastructure Services,
Mirantis
http://www.mirantis.com
Tel. +1 650 963 9828
Mob. +1 650 996 3284
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Mark Washenberger]

On Fri, Dec 6, 2013 at 2:43 PM, Randall Burt wrote:

>  I too have warmed to this idea but wonder about the actual implementation
> around it. While I like where Edmund is going with this, I wonder if it
> wouldn't be valuable in the short-to-mid-term (I/J) to just add /templates
> to Glance (/assemblies, /applications, etc) along side /images.  Initially,
> we could have separate endpoints and data structures for these different
> asset types, refactoring the easy bits along the way and leveraging the
> existing data storage and caching bits, but leaving more disruptive changes
> alone. That can get the functionality going, prove some concepts, and allow
> all of the interested parties to better plan a more general v3 api.
>

I think this trajectory makes a lot of sense as an initial plan. We should
definitely see how much overlap there is through a detailed proposal. If
there are some extremely low-hanging fruit on the side of generalization,
maybe we can revise such a proposal before we get going too far.

It also occurs to me that this is a very big shift in focus for the Glance
team, however, so perhaps it would make sense to try to discuss this at the
midcycle meetup [1]? I know some of the discussion there is going to
revolve around finding a better solution to the image sharing / image
marketplace problem.


[1]
http://lists.openstack.org/pipermail/openstack-dev/2013-November/019230.html


>
>  On Dec 6, 2013, at 4:23 PM, Edmund Troche 
>  wrote:
>
>  I agree with what seems to also be the general consensus, that Glance
> can "become" Heater+Glance (the service that manages images in OS today).
> Clearly, if someone looks at the Glance DB schema, APIs and service type
> (as returned by keystone service-list), all of the terminology is about
> images, so we would need to more formally define what are the
> characteristics or "image", "template", maybe "assembly", "components" etc
> and find what is a good generalization. When looking at the attributes for
> "image" (image table), I can see where there are a few that would be
> generic enough to apply to "image", "template" etc, so those could be taken
> to be the base set of attributes, and then based on the "type" (image,
> template, etc) we could then have attributes that are type-specific (maybe
> by leveraging what is today "image_properties").
>
> As I read through the discussion, the one thing that came to mind is
> "asset management". I can see where if someone bothers to create an image,
> or a template, then it is for a good reason, and that perhaps you'd like to
> maintain it as an IT asset. Along those lines, it occurred to me that maybe
> what we need is to make Glance some sort of asset management service that
> can be leveraged by Service Catalogs, Nova, etc. Instead of storing
> "images" and "templates"  we store assets of one kind or another, with
> artifacts (like files, image content, etc), and associated metadata. There
> is some work we could borrow from, conceptually at least, from OSLC's Asset
> Management specification:
> http://open-services.net/wiki/asset-management/OSLC-Asset-Management-2.0-Specification/.
> Looking at this spec, it probably has more than we need, but there's plenty
> we could borrow from it.
>
>
> Edmund Troche
>
>
> Georgy Okrokvertskhov ---12/06/2013 01:34:13 PM---As a
> Murano team we will be happy to contribute to Glance. Our Murano metadata
> repository is a stand
>
>
> From: Georgy Okrokvertskhov 
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>,
> Date: 12/06/2013 01:34 PM
> Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal
>
> --
>
>
>
> As a Murano team we will be happy to contribute to Glance. Our Murano
> metadata repository is a standalone component (with its own git
> repository)which is not tightly coupled with Murano itself. We can easily
> add our functionality to Glance as a new component\subproject.
>
> Thanks
> Georgy
>
>
> On Fri, Dec 6, 2013 at 11:11 AM, Vishvananda Ishaya <
> *vishvana...@gmail.com* > wrote:
>
>
>On Dec 6, 2013, at 10:38 AM, Clint Byrum 
> <*cl...@fewbar.com*>
>wrote:
>
>> Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
>>> On 12/05/2013 04:25 PM, Clint Byrum wrote:
>>>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
>>>>>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44
>-0800:
>>>>>>> On Dec 5, 2013, at 

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 6, 2013, at 5:04 PM, Clint Byrum 
 wrote:

> Excerpts from Randall Burt's message of 2013-12-06 14:43:05 -0800:
>> I too have warmed to this idea but wonder about the actual implementation 
>> around it. While I like where Edmund is going with this, I wonder if it 
>> wouldn't be valuable in the short-to-mid-term (I/J) to just add /templates 
>> to Glance (/assemblies, /applications, etc) along side /images.  Initially, 
>> we could have separate endpoints and data structures for these different 
>> asset types, refactoring the easy bits along the way and leveraging the 
>> existing data storage and caching bits, but leaving more disruptive changes 
>> alone. That can get the functionality going, prove some concepts, and allow 
>> all of the interested parties to better plan a more general v3 api.
>> 
> 
> +1 on bolting the different views for things on as new v2 pieces instead
> of trying to solve the API genericism immediately.
> 
> I would strive to make this a facade, and start immediately on making
> Glance more generic under the hood.  Otherwise these will just end up
> as silos inside Glance instead of silos inside OpenStack.

Totally agreed. Where it makes sense to refactor we should do that rather than 
implementing essentially different services underneath.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Edmund Troche]

I thought about that, i.e. first step in implementation just adding
templates, but like you said, you might end up duplicating 5 of the 7
tables in the Glance database, for every new "asset type" (image, template,
etc). Then you would do a similar thing for the endpoints. So, I'm not sure
what's a better way to approach this. For all I know, doing a
"s/image/asset/g" for *.py,, adding attribute images.type, and a little
more refactoring, might get us 80% of the asset management functionality
that we would need initially ;-) Not knowing the Glance code base I'm only
going by the surface footprint, so I'll leave it to the experts to comment
on what would be a good approach to take Glance to the next level.


Edmund Troche



From:   Randall Burt 
To: "OpenStack Development Mailing List (not for usage questions)"
    ,
Date:   12/06/2013 04:47 PM
Subject:    Re: [openstack-dev] [heat] [glance] Heater Proposal



I too have warmed to this idea but wonder about the actual implementation
around it. While I like where Edmund is going with this, I wonder if it
wouldn't be valuable in the short-to-mid-term (I/J) to just add /templates
to Glance (/assemblies, /applications, etc) along side /images.  Initially,
we could have separate endpoints and data structures for these different
asset types, refactoring the easy bits along the way and leveraging the
existing data storage and caching bits, but leaving more disruptive changes
alone. That can get the functionality going, prove some concepts, and allow
all of the interested parties to better plan a more general v3 api.

On Dec 6, 2013, at 4:23 PM, Edmund Troche 
 wrote:



  I agree with what seems to also be the general consensus, that Glance
  can "become" Heater+Glance (the service that manages images in OS
  today). Clearly, if someone looks at the Glance DB schema, APIs and
  service type (as returned by keystone service-list), all of the
  terminology is about images, so we would need to more formally define
  what are the characteristics or "image", "template", maybe
  "assembly", "components" etc and find what is a good generalization.
  When looking at the attributes for "image" (image table), I can see
  where there are a few that would be generic enough to apply to
  "image", "template" etc, so those could be taken to be the base set
  of attributes, and then based on the "type" (image, template, etc) we
  could then have attributes that are type-specific (maybe by
  leveraging what is today "image_properties").

  As I read through the discussion, the one thing that came to mind is
  "asset management". I can see where if someone bothers to create an
  image, or a template, then it is for a good reason, and that perhaps
  you'd like to maintain it as an IT asset. Along those lines, it
  occurred to me that maybe what we need is to make Glance some sort of
  asset management service that can be leveraged by Service Catalogs,
  Nova, etc. Instead of storing "images" and "templates"  we store
  assets of one kind or another, with artifacts (like files, image
  content, etc), and associated metadata. There is some work we could
  borrow from, conceptually at least, from OSLC's Asset Management
  specification:
  
http://open-services.net/wiki/asset-management/OSLC-Asset-Management-2.0-Specification/
  . Looking at this spec, it probably has more than we need, but
  there's plenty we could borrow from it.


  Edmund Troche


  Georgy Okrokvertskhov ---12/06/2013 01:34:13 PM---As a
  Murano team we will be happy to contribute to Glance. Our Murano
  metadata repository is a stand

      From: Georgy Okrokvertskhov 
  To: "OpenStack Development Mailing List (not for usage questions)" <
  openstack-dev@lists.openstack.org>,
  Date: 12/06/2013 01:34 PM
  Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal





  As a Murano team we will be happy to contribute to Glance. Our Murano
  metadata repository is a standalone component (with its own git
  repository)which is not tightly coupled with Murano itself. We can
  easily add our functionality to Glance as a new component\subproject.

  Thanks
  Georgy


  On Fri, Dec 6, 2013 at 11:11 AM, Vishvananda Ishaya <
  vishvana...@gmail.com> wrote:

On Dec 6, 2013, at 10:38 AM, Clint Byrum 
wrote:

> Excerpts from Jay Pipes's message of 2013-12-05 21:32:54
-0800:
>> On 12/05/2013 04:25 PM, Clint Byrum wrote:
>>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49
-0800:
>>&

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Randall Burt's message of 2013-12-06 14:43:05 -0800:
> I too have warmed to this idea but wonder about the actual implementation 
> around it. While I like where Edmund is going with this, I wonder if it 
> wouldn't be valuable in the short-to-mid-term (I/J) to just add /templates to 
> Glance (/assemblies, /applications, etc) along side /images.  Initially, we 
> could have separate endpoints and data structures for these different asset 
> types, refactoring the easy bits along the way and leveraging the existing 
> data storage and caching bits, but leaving more disruptive changes alone. 
> That can get the functionality going, prove some concepts, and allow all of 
> the interested parties to better plan a more general v3 api.
> 

+1 on bolting the different views for things on as new v2 pieces instead
of trying to solve the API genericism immediately.

I would strive to make this a facade, and start immediately on making
Glance more generic under the hood.  Otherwise these will just end up
as silos inside Glance instead of silos inside OpenStack.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

I too have warmed to this idea but wonder about the actual implementation 
around it. While I like where Edmund is going with this, I wonder if it 
wouldn't be valuable in the short-to-mid-term (I/J) to just add /templates to 
Glance (/assemblies, /applications, etc) along side /images.  Initially, we 
could have separate endpoints and data structures for these different asset 
types, refactoring the easy bits along the way and leveraging the existing data 
storage and caching bits, but leaving more disruptive changes alone. That can 
get the functionality going, prove some concepts, and allow all of the 
interested parties to better plan a more general v3 api.

On Dec 6, 2013, at 4:23 PM, Edmund Troche 
mailto:edmund.tro...@us.ibm.com>>
 wrote:


I agree with what seems to also be the general consensus, that Glance can 
"become" Heater+Glance (the service that manages images in OS today). Clearly, 
if someone looks at the Glance DB schema, APIs and service type (as returned by 
keystone service-list), all of the terminology is about images, so we would 
need to more formally define what are the characteristics or "image", 
"template", maybe "assembly", "components" etc and find what is a good 
generalization. When looking at the attributes for "image" (image table), I can 
see where there are a few that would be generic enough to apply to "image", 
"template" etc, so those could be taken to be the base set of attributes, and 
then based on the "type" (image, template, etc) we could then have attributes 
that are type-specific (maybe by leveraging what is today "image_properties").

As I read through the discussion, the one thing that came to mind is "asset 
management". I can see where if someone bothers to create an image, or a 
template, then it is for a good reason, and that perhaps you'd like to maintain 
it as an IT asset. Along those lines, it occurred to me that maybe what we need 
is to make Glance some sort of asset management service that can be leveraged 
by Service Catalogs, Nova, etc. Instead of storing "images" and "templates"  we 
store assets of one kind or another, with artifacts (like files, image content, 
etc), and associated metadata. There is some work we could borrow from, 
conceptually at least, from OSLC's Asset Management specification: 
http://open-services.net/wiki/asset-management/OSLC-Asset-Management-2.0-Specification/.
 Looking at this spec, it probably has more than we need, but there's plenty we 
could borrow from it.


Edmund Troche


Georgy Okrokvertskhov ---12/06/2013 01:34:13 PM---As a Murano team 
we will be happy to contribute to Glance. Our Murano metadata repository is a 
stand

From: Georgy Okrokvertskhov 
mailto:gokrokvertsk...@mirantis.com>>
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>,
Date: 12/06/2013 01:34 PM
Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal





As a Murano team we will be happy to contribute to Glance. Our Murano metadata 
repository is a standalone component (with its own git repository)which is not 
tightly coupled with Murano itself. We can easily add our functionality to 
Glance as a new component\subproject.

Thanks
Georgy


On Fri, Dec 6, 2013 at 11:11 AM, Vishvananda Ishaya 
mailto:vishvana...@gmail.com>> wrote:

On Dec 6, 2013, at 10:38 AM, Clint Byrum 
mailto:cl...@fewbar.com>> wrote:

> Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
>> On 12/05/2013 04:25 PM, Clint Byrum wrote:
>>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
>>>>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>>>>>> On Dec 5, 2013, at 10:10 AM, Clint Byrum >>>>> fewbar.com<http://fewbar.com/>>
>>>>>>  wrote:
>>>>>>
>>>>>>> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>>>>>>>> Why not just use glance?
>>>>>>>>
>>>>>>>
>>>>>>> I've asked that question a few times, and I think I can collate the
>>>>>>> responses I've received below. I think enhancing glance to do these
>>>>>>> things is on the table:
>>>>>>>
>>>>>>> 1. Glance is for big blobs of data not tiny templates.
>>>>>>> 2. Versioning of a single resource is desired.
>>>>>>> 3. Tagging/classifying/listing/sorting
>>>>>>> 4. Glance is designed to expose the uploaded blobs to nova, not users
>>>>>>>
>>>>

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Edmund Troche]

I agree with what seems to also be the general consensus, that Glance can
"become" Heater+Glance (the service that manages images in OS today).
Clearly, if someone looks at the Glance DB schema, APIs and service type
(as returned by keystone service-list), all of the terminology is about
images, so we would need to more formally define what are the
characteristics or "image", "template", maybe "assembly", "components" etc
and find what is a good generalization. When looking at the attributes for
"image" (image table), I can see where there are a few that would be
generic enough to apply to "image", "template" etc, so those could be taken
to be the base set of attributes, and then based on the "type" (image,
template, etc) we could then have attributes that are type-specific (maybe
by leveraging what is today "image_properties").

As I read through the discussion, the one thing that came to mind is "asset
management". I can see where if someone bothers to create an image, or a
template, then it is for a good reason, and that perhaps you'd like to
maintain it as an IT asset. Along those lines, it occurred to me that maybe
what we need is to make Glance some sort of asset management service that
can be leveraged by Service Catalogs, Nova, etc. Instead of storing
"images" and "templates"  we store assets of one kind or another, with
artifacts (like files, image content, etc), and associated metadata. There
is some work we could borrow from, conceptually at least, from OSLC's Asset
Management specification:
http://open-services.net/wiki/asset-management/OSLC-Asset-Management-2.0-Specification/.
 Looking at this spec, it probably has more than we need, but there's
plenty we could borrow from it.


Edmund Troche




From:   Georgy Okrokvertskhov 
To:     "OpenStack Development Mailing List (not for usage questions)"
,
Date:   12/06/2013 01:34 PM
Subject:Re: [openstack-dev] [heat] [glance] Heater Proposal



As a Murano team we will be happy to contribute to Glance. Our Murano
metadata repository is a standalone component (with its own git
repository)which is not tightly coupled with Murano itself. We can easily
add our functionality to Glance as a new component\subproject.

Thanks
Georgy


On Fri, Dec 6, 2013 at 11:11 AM, Vishvananda Ishaya 
wrote:

  On Dec 6, 2013, at 10:38 AM, Clint Byrum  wrote:

  > Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
  >> On 12/05/2013 04:25 PM, Clint Byrum wrote:
  >>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
  >>>>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
  >>>>>> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
  >>>>>>  wrote:
  >>>>>>
  >>>>>>> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45
  -0800:
  >>>>>>>> Why not just use glance?
  >>>>>>>>
  >>>>>>>
  >>>>>>> I've asked that question a few times, and I think I can collate
  the
  >>>>>>> responses I've received below. I think enhancing glance to do
  these
  >>>>>>> things is on the table:
  >>>>>>>
  >>>>>>> 1. Glance is for big blobs of data not tiny templates.
  >>>>>>> 2. Versioning of a single resource is desired.
  >>>>>>> 3. Tagging/classifying/listing/sorting
  >>>>>>> 4. Glance is designed to expose the uploaded blobs to nova, not
  users
  >>>>>>>
  >>>>>>> My responses:
  >>>>>>>
  >>>>>>> 1: Irrelevant. Smaller things will fit in it just fine.
  >>>>>>
  >>>>>> Fitting is one thing, optimizations around particular assumptions
  about the size of data and the frequency of reads/writes might be an
  issue, but I admit to ignorance about those details in Glance.
  >>>>>>
  >>>>>
  >>>>> Optimizations can be improved for various use cases. The design,
  however,
  >>>>> has no assumptions that I know about that would invalidate storing
  blobs
  >>>>> of yaml/json vs. blobs of kernel/qcow2/raw image.
  >>>>
  >>>> I think we are getting out into the weeds a little bit here. It is
  important to think about these apis in terms of what they actually do,
  before the decision of combining them or not can be made.
  >>>>
  >>>> I think of HeatR as a template storage service, it provides extra
  data and operations on templates. HeatR should not care a

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Georgy Okrokvertskhov]

As a Murano team we will be happy to contribute to Glance. Our Murano
metadata repository is a standalone component (with its own git
repository)which is not tightly coupled with Murano itself. We can easily
add our functionality to Glance as a new component\subproject.

Thanks
Georgy


On Fri, Dec 6, 2013 at 11:11 AM, Vishvananda Ishaya
wrote:

>
> On Dec 6, 2013, at 10:38 AM, Clint Byrum  wrote:
>
> > Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
> >> On 12/05/2013 04:25 PM, Clint Byrum wrote:
> >>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
> > Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
> >> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
> >>  wrote:
> >>
> >>> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>  Why not just use glance?
> 
> >>>
> >>> I've asked that question a few times, and I think I can collate the
> >>> responses I've received below. I think enhancing glance to do these
> >>> things is on the table:
> >>>
> >>> 1. Glance is for big blobs of data not tiny templates.
> >>> 2. Versioning of a single resource is desired.
> >>> 3. Tagging/classifying/listing/sorting
> >>> 4. Glance is designed to expose the uploaded blobs to nova, not
> users
> >>>
> >>> My responses:
> >>>
> >>> 1: Irrelevant. Smaller things will fit in it just fine.
> >>
> >> Fitting is one thing, optimizations around particular assumptions
> about the size of data and the frequency of reads/writes might be an issue,
> but I admit to ignorance about those details in Glance.
> >>
> >
> > Optimizations can be improved for various use cases. The design,
> however,
> > has no assumptions that I know about that would invalidate storing
> blobs
> > of yaml/json vs. blobs of kernel/qcow2/raw image.
> 
>  I think we are getting out into the weeds a little bit here. It is
> important to think about these apis in terms of what they actually do,
> before the decision of combining them or not can be made.
> 
>  I think of HeatR as a template storage service, it provides extra
> data and operations on templates. HeatR should not care about how those
> templates are stored.
>  Glance is an image storage service, it provides extra data and
> operations on images (not blobs), and it happens to use swift as a backend.
> 
>  If HeatR and Glance were combined, it would result in taking two very
> different types of data (template metadata vs image metadata) and mashing
> them into one service. How would adding the complexity of HeatR benefit
> Glance, when they are dealing with conceptually two very different types of
> data? For instance, should a template ever care about the field "minRam"
> that is stored with an image? Combining them adds a huge development
> complexity with a very small operations payoff, and so Openstack is already
> so operationally complex that HeatR as a separate service would be
> knowledgeable. Only clients of Heat will ever care about data and
> operations on templates, so I move that HeatR becomes it's own service, or
> becomes part of Heat.
> 
> >>>
> >>> I spoke at length via G+ with Randall and Tim about this earlier today.
> >>> I think I understand the impetus for all of this a little better now.
> >>>
> >>> Basically what I'm suggesting is that Glance is only narrow in scope
> >>> because that was the only object that OpenStack needed a catalog for
> >>> before now.
> >>>
> >>> However, the overlap between a catalog of images and a catalog of
> >>> templates is quite comprehensive. The individual fields that matter to
> >>> images are different than the ones that matter to templates, but that
> >>> is a really minor detail isn't it?
> >>>
> >>> I would suggest that Glance be slightly expanded in scope to be an
> >>> object catalog. Each object type can have its own set of fields that
> >>> matter to it.
> >>>
> >>> This doesn't have to be a minor change to glance to still have many
> >>> advantages over writing something from scratch and asking people to
> >>> deploy another service that is 99% the same as Glance.
> >>
> >> My suggestion for long-term architecture would be to use Murano for
> >> catalog/metadata information (for images/templates/whatever) and move
> >> the block-streaming drivers into Cinder, and get rid of the Glance
> >> project entirely. Murano would then become the catalog/registry of
> >> objects in the OpenStack world, Cinder would be the thing that manages
> >> and streams blocks of data or block devices, and Glance could go away.
> >> Imagine it... OpenStack actually *reducing* the number of projects
> >> instead of expanding! :)
> >>
> >
> > Have we not learned our lesson with Nova-Net/Neutron yet? Rewrites of
> > existing functionality are painful.
> >
> > The Murano-concerned people have already stated they are starting over
> > on that catalog.
> >
> 

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Vishvananda Ishaya]

On Dec 6, 2013, at 10:38 AM, Clint Byrum  wrote:

> Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
>> On 12/05/2013 04:25 PM, Clint Byrum wrote:
>>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>>  wrote:
>> 
>>> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
 Why not just use glance?
 
>>> 
>>> I've asked that question a few times, and I think I can collate the
>>> responses I've received below. I think enhancing glance to do these
>>> things is on the table:
>>> 
>>> 1. Glance is for big blobs of data not tiny templates.
>>> 2. Versioning of a single resource is desired.
>>> 3. Tagging/classifying/listing/sorting
>>> 4. Glance is designed to expose the uploaded blobs to nova, not users
>>> 
>>> My responses:
>>> 
>>> 1: Irrelevant. Smaller things will fit in it just fine.
>> 
>> Fitting is one thing, optimizations around particular assumptions about 
>> the size of data and the frequency of reads/writes might be an issue, 
>> but I admit to ignorance about those details in Glance.
>> 
> 
> Optimizations can be improved for various use cases. The design, however,
> has no assumptions that I know about that would invalidate storing blobs
> of yaml/json vs. blobs of kernel/qcow2/raw image.
 
 I think we are getting out into the weeds a little bit here. It is 
 important to think about these apis in terms of what they actually do, 
 before the decision of combining them or not can be made.
 
 I think of HeatR as a template storage service, it provides extra data and 
 operations on templates. HeatR should not care about how those templates 
 are stored.
 Glance is an image storage service, it provides extra data and operations 
 on images (not blobs), and it happens to use swift as a backend.
 
 If HeatR and Glance were combined, it would result in taking two very 
 different types of data (template metadata vs image metadata) and mashing 
 them into one service. How would adding the complexity of HeatR benefit 
 Glance, when they are dealing with conceptually two very different types 
 of data? For instance, should a template ever care about the field 
 "minRam" that is stored with an image? Combining them adds a huge 
 development complexity with a very small operations payoff, and so 
 Openstack is already so operationally complex that HeatR as a separate 
 service would be knowledgeable. Only clients of Heat will ever care about 
 data and operations on templates, so I move that HeatR becomes it's own 
 service, or becomes part of Heat.
 
>>> 
>>> I spoke at length via G+ with Randall and Tim about this earlier today.
>>> I think I understand the impetus for all of this a little better now.
>>> 
>>> Basically what I'm suggesting is that Glance is only narrow in scope
>>> because that was the only object that OpenStack needed a catalog for
>>> before now.
>>> 
>>> However, the overlap between a catalog of images and a catalog of
>>> templates is quite comprehensive. The individual fields that matter to
>>> images are different than the ones that matter to templates, but that
>>> is a really minor detail isn't it?
>>> 
>>> I would suggest that Glance be slightly expanded in scope to be an
>>> object catalog. Each object type can have its own set of fields that
>>> matter to it.
>>> 
>>> This doesn't have to be a minor change to glance to still have many
>>> advantages over writing something from scratch and asking people to
>>> deploy another service that is 99% the same as Glance.
>> 
>> My suggestion for long-term architecture would be to use Murano for 
>> catalog/metadata information (for images/templates/whatever) and move 
>> the block-streaming drivers into Cinder, and get rid of the Glance 
>> project entirely. Murano would then become the catalog/registry of 
>> objects in the OpenStack world, Cinder would be the thing that manages 
>> and streams blocks of data or block devices, and Glance could go away. 
>> Imagine it... OpenStack actually *reducing* the number of projects 
>> instead of expanding! :)
>> 
> 
> Have we not learned our lesson with Nova-Net/Neutron yet? Rewrites of
> existing functionality are painful.
> 
> The Murano-concerned people have already stated they are starting over
> on that catalog.
> 
> I suggest they start over by expanding Glance's catalog. If the block
> streaming bits of Glance need to move somewhere else, that sounds like a
> completely separate concern that distracts from this point.
> 
> And to be clear, (I think I will just stop talking as I think I've
> made this point), my point is, we have a catalog, let's make it better.

+1

Vish

> 
> _

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Vishvananda Ishaya]

On Dec 6, 2013, at 10:07 AM, Georgy Okrokvertskhov 
 wrote:

> Hi,
> 
> I am really inspired by this thread. Frankly saying, Glance for Murano was a 
> kind of sacred entity, as it is a service with a long history in OpenStack.  
> We even did not think in the direction of changing Glance. Spending a night 
> with these ideas, I am kind of having a dream about unified catalog where the 
> full range of different entities are presented. Just imagine that we have 
> everything as  first class citizens of catalog treated equally: single VM 
> (image), Heat template (fixed number of VMs\ autoscaling groups), Murano 
> Application (generated Heat templates), Solum assemblies
> 
> Projects like Solum will highly benefit from this catalog as it can use all 
> varieties of VM configurations talking with one service.
> This catalog will be able not just list all possible deployable entities but 
> can be also a registry for already deployed configurations. This is perfectly 
> aligned with the goal for catalog to be a kind of market place which provides 
> billing information too.
> 
> OpenStack users also will benefit from this as they will have the unified 
> approach for manage deployments and deployable entities.
> 
> I doubt that it could be done by a single team. But if all teams join this 
> effort we can do this. From my perspective, this could be a part of Glance 
> program and it is not necessary to add a new program for that. As it was 
> mentioned earlier in this thread an idea of market place for images in Glance 
> was here for some time. I think we can extend it to the idea of creating a 
> marketplace for a deployable entity regardless of the way of deployment. As 
> Glance is a core project which means it always exist in OpenStack deployment 
> it makes sense to as a central catalog for everything.

+1 

Vish

> 
> Thanks
> Georgy
> 
> 
> On Fri, Dec 6, 2013 at 8:57 AM, Mark Washenberger 
>  wrote:
> 
> 
> 
> On Thu, Dec 5, 2013 at 9:32 PM, Jay Pipes  wrote:
> On 12/05/2013 04:25 PM, Clint Byrum wrote:
> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>   wrote:
> 
> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> Why not just use glance?
> 
> 
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:
> 
> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
> 
> My responses:
> 
> 1: Irrelevant. Smaller things will fit in it just fine.
> 
> Fitting is one thing, optimizations around particular assumptions about the 
> size of data and the frequency of reads/writes might be an issue, but I admit 
> to ignorance about those details in Glance.
> 
> 
> Optimizations can be improved for various use cases. The design, however,
> has no assumptions that I know about that would invalidate storing blobs
> of yaml/json vs. blobs of kernel/qcow2/raw image.
> 
> I think we are getting out into the weeds a little bit here. It is important 
> to think about these apis in terms of what they actually do, before the 
> decision of combining them or not can be made.
> 
> I think of HeatR as a template storage service, it provides extra data and 
> operations on templates. HeatR should not care about how those templates are 
> stored.
> Glance is an image storage service, it provides extra data and operations on 
> images (not blobs), and it happens to use swift as a backend.
> 
> If HeatR and Glance were combined, it would result in taking two very 
> different types of data (template metadata vs image metadata) and mashing 
> them into one service. How would adding the complexity of HeatR benefit 
> Glance, when they are dealing with conceptually two very different types of 
> data? For instance, should a template ever care about the field "minRam" that 
> is stored with an image? Combining them adds a huge development complexity 
> with a very small operations payoff, and so Openstack is already so 
> operationally complex that HeatR as a separate service would be 
> knowledgeable. Only clients of Heat will ever care about data and operations 
> on templates, so I move that HeatR becomes it's own service, or becomes part 
> of Heat.
> 
> 
> I spoke at length via G+ with Randall and Tim about this earlier today.
> I think I understand the impetus for all of this a little better now.
> 
> Basically what I'm suggesting is that Glance is only narrow in scope
> because that was the only object that OpenStack needed a catalog for
> before now.
> 
> However, the overlap between a catalog of images and a catalog of
> templates is quite comprehensive. The individual fields that matter to
> images are dif

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
> On 12/05/2013 04:25 PM, Clint Byrum wrote:
> > Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
> >>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>  On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>    wrote:
> 
> > Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> >> Why not just use glance?
> >>
> >
> > I've asked that question a few times, and I think I can collate the
> > responses I've received below. I think enhancing glance to do these
> > things is on the table:
> >
> > 1. Glance is for big blobs of data not tiny templates.
> > 2. Versioning of a single resource is desired.
> > 3. Tagging/classifying/listing/sorting
> > 4. Glance is designed to expose the uploaded blobs to nova, not users
> >
> > My responses:
> >
> > 1: Irrelevant. Smaller things will fit in it just fine.
> 
>  Fitting is one thing, optimizations around particular assumptions about 
>  the size of data and the frequency of reads/writes might be an issue, 
>  but I admit to ignorance about those details in Glance.
> 
> >>>
> >>> Optimizations can be improved for various use cases. The design, however,
> >>> has no assumptions that I know about that would invalidate storing blobs
> >>> of yaml/json vs. blobs of kernel/qcow2/raw image.
> >>
> >> I think we are getting out into the weeds a little bit here. It is 
> >> important to think about these apis in terms of what they actually do, 
> >> before the decision of combining them or not can be made.
> >>
> >> I think of HeatR as a template storage service, it provides extra data and 
> >> operations on templates. HeatR should not care about how those templates 
> >> are stored.
> >> Glance is an image storage service, it provides extra data and operations 
> >> on images (not blobs), and it happens to use swift as a backend.
> >>
> >> If HeatR and Glance were combined, it would result in taking two very 
> >> different types of data (template metadata vs image metadata) and mashing 
> >> them into one service. How would adding the complexity of HeatR benefit 
> >> Glance, when they are dealing with conceptually two very different types 
> >> of data? For instance, should a template ever care about the field 
> >> "minRam" that is stored with an image? Combining them adds a huge 
> >> development complexity with a very small operations payoff, and so 
> >> Openstack is already so operationally complex that HeatR as a separate 
> >> service would be knowledgeable. Only clients of Heat will ever care about 
> >> data and operations on templates, so I move that HeatR becomes it's own 
> >> service, or becomes part of Heat.
> >>
> >
> > I spoke at length via G+ with Randall and Tim about this earlier today.
> > I think I understand the impetus for all of this a little better now.
> >
> > Basically what I'm suggesting is that Glance is only narrow in scope
> > because that was the only object that OpenStack needed a catalog for
> > before now.
> >
> > However, the overlap between a catalog of images and a catalog of
> > templates is quite comprehensive. The individual fields that matter to
> > images are different than the ones that matter to templates, but that
> > is a really minor detail isn't it?
> >
> > I would suggest that Glance be slightly expanded in scope to be an
> > object catalog. Each object type can have its own set of fields that
> > matter to it.
> >
> > This doesn't have to be a minor change to glance to still have many
> > advantages over writing something from scratch and asking people to
> > deploy another service that is 99% the same as Glance.
> 
> My suggestion for long-term architecture would be to use Murano for 
> catalog/metadata information (for images/templates/whatever) and move 
> the block-streaming drivers into Cinder, and get rid of the Glance 
> project entirely. Murano would then become the catalog/registry of 
> objects in the OpenStack world, Cinder would be the thing that manages 
> and streams blocks of data or block devices, and Glance could go away. 
> Imagine it... OpenStack actually *reducing* the number of projects 
> instead of expanding! :)
> 

Have we not learned our lesson with Nova-Net/Neutron yet? Rewrites of
existing functionality are painful.

The Murano-concerned people have already stated they are starting over
on that catalog.

I suggest they start over by expanding Glance's catalog. If the block
streaming bits of Glance need to move somewhere else, that sounds like a
completely separate concern that distracts from this point.

And to be clear, (I think I will just stop talking as I think I've
made this point), my point is, we have a catalog, let's make it better.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinf

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Georgy Okrokvertskhov]

Hi,

I am really inspired by this thread. Frankly saying, Glance for Murano was
a kind of sacred entity, as it is a service with a long history in
OpenStack.  We even did not think in the direction of changing Glance.
Spending a night with these ideas, I am kind of having a dream about
unified catalog where the full range of different entities are presented.
Just imagine that we have everything as  first class citizens of catalog
treated equally: single VM (image), Heat template (fixed number of VMs\
autoscaling groups), Murano Application (generated Heat templates), Solum
assemblies

Projects like Solum will highly benefit from this catalog as it can use all
varieties of VM configurations talking with one service.

This catalog will be able not just list all possible deployable entities
but can be also a registry for already deployed configurations. This is
perfectly aligned with the goal for catalog to be a kind of market place
which provides billing information too.

OpenStack users also will benefit from this as they will have the unified
approach for manage deployments and deployable entities.

I doubt that it could be done by a single team. But if all teams join this
effort we can do this. From my perspective, this could be a part of Glance
program and it is not necessary to add a new program for that. As it was
mentioned earlier in this thread an idea of market place for images in
Glance was here for some time. I think we can extend it to the idea of
creating a marketplace for a deployable entity regardless of the way of
deployment. As Glance is a core project which means it always exist in
OpenStack deployment it makes sense to as a central catalog for everything.

Thanks
Georgy


On Fri, Dec 6, 2013 at 8:57 AM, Mark Washenberger <
mark.washenber...@markwash.net> wrote:

>
>
>
> On Thu, Dec 5, 2013 at 9:32 PM, Jay Pipes  wrote:
>
>> On 12/05/2013 04:25 PM, Clint Byrum wrote:
>>
>>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
>>>
 Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>
>> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>>   wrote:
>>
>>  Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>>>
 Why not just use glance?


>>> I've asked that question a few times, and I think I can collate the
>>> responses I've received below. I think enhancing glance to do these
>>> things is on the table:
>>>
>>> 1. Glance is for big blobs of data not tiny templates.
>>> 2. Versioning of a single resource is desired.
>>> 3. Tagging/classifying/listing/sorting
>>> 4. Glance is designed to expose the uploaded blobs to nova, not users
>>>
>>> My responses:
>>>
>>> 1: Irrelevant. Smaller things will fit in it just fine.
>>>
>>
>> Fitting is one thing, optimizations around particular assumptions
>> about the size of data and the frequency of reads/writes might be an 
>> issue,
>> but I admit to ignorance about those details in Glance.
>>
>>
> Optimizations can be improved for various use cases. The design,
> however,
> has no assumptions that I know about that would invalidate storing
> blobs
> of yaml/json vs. blobs of kernel/qcow2/raw image.
>

 I think we are getting out into the weeds a little bit here. It is
 important to think about these apis in terms of what they actually do,
 before the decision of combining them or not can be made.

 I think of HeatR as a template storage service, it provides extra data
 and operations on templates. HeatR should not care about how those
 templates are stored.
 Glance is an image storage service, it provides extra data and
 operations on images (not blobs), and it happens to use swift as a backend.

 If HeatR and Glance were combined, it would result in taking two very
 different types of data (template metadata vs image metadata) and mashing
 them into one service. How would adding the complexity of HeatR benefit
 Glance, when they are dealing with conceptually two very different types of
 data? For instance, should a template ever care about the field "minRam"
 that is stored with an image? Combining them adds a huge development
 complexity with a very small operations payoff, and so Openstack is already
 so operationally complex that HeatR as a separate service would be
 knowledgeable. Only clients of Heat will ever care about data and
 operations on templates, so I move that HeatR becomes it's own service, or
 becomes part of Heat.


>>> I spoke at length via G+ with Randall and Tim about this earlier today.
>>> I think I understand the impetus for all of this a little better now.
>>>
>>> Basically what I'm suggesting is that Glance is only narrow in scope
>>> because that was the only object that OpenStack needed a catalog for
>>> before now.
>>>
>>> However, the ov

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Mark Washenberger]

On Thu, Dec 5, 2013 at 9:32 PM, Jay Pipes  wrote:

> On 12/05/2013 04:25 PM, Clint Byrum wrote:
>
>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
>>
>>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:

> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>   wrote:
>
>  Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>>
>>> Why not just use glance?
>>>
>>>
>> I've asked that question a few times, and I think I can collate the
>> responses I've received below. I think enhancing glance to do these
>> things is on the table:
>>
>> 1. Glance is for big blobs of data not tiny templates.
>> 2. Versioning of a single resource is desired.
>> 3. Tagging/classifying/listing/sorting
>> 4. Glance is designed to expose the uploaded blobs to nova, not users
>>
>> My responses:
>>
>> 1: Irrelevant. Smaller things will fit in it just fine.
>>
>
> Fitting is one thing, optimizations around particular assumptions
> about the size of data and the frequency of reads/writes might be an 
> issue,
> but I admit to ignorance about those details in Glance.
>
>
 Optimizations can be improved for various use cases. The design,
 however,
 has no assumptions that I know about that would invalidate storing blobs
 of yaml/json vs. blobs of kernel/qcow2/raw image.

>>>
>>> I think we are getting out into the weeds a little bit here. It is
>>> important to think about these apis in terms of what they actually do,
>>> before the decision of combining them or not can be made.
>>>
>>> I think of HeatR as a template storage service, it provides extra data
>>> and operations on templates. HeatR should not care about how those
>>> templates are stored.
>>> Glance is an image storage service, it provides extra data and
>>> operations on images (not blobs), and it happens to use swift as a backend.
>>>
>>> If HeatR and Glance were combined, it would result in taking two very
>>> different types of data (template metadata vs image metadata) and mashing
>>> them into one service. How would adding the complexity of HeatR benefit
>>> Glance, when they are dealing with conceptually two very different types of
>>> data? For instance, should a template ever care about the field "minRam"
>>> that is stored with an image? Combining them adds a huge development
>>> complexity with a very small operations payoff, and so Openstack is already
>>> so operationally complex that HeatR as a separate service would be
>>> knowledgeable. Only clients of Heat will ever care about data and
>>> operations on templates, so I move that HeatR becomes it's own service, or
>>> becomes part of Heat.
>>>
>>>
>> I spoke at length via G+ with Randall and Tim about this earlier today.
>> I think I understand the impetus for all of this a little better now.
>>
>> Basically what I'm suggesting is that Glance is only narrow in scope
>> because that was the only object that OpenStack needed a catalog for
>> before now.
>>
>> However, the overlap between a catalog of images and a catalog of
>> templates is quite comprehensive. The individual fields that matter to
>> images are different than the ones that matter to templates, but that
>> is a really minor detail isn't it?
>>
>> I would suggest that Glance be slightly expanded in scope to be an
>> object catalog. Each object type can have its own set of fields that
>> matter to it.
>>
>> This doesn't have to be a minor change to glance to still have many
>> advantages over writing something from scratch and asking people to
>> deploy another service that is 99% the same as Glance.
>>
>
> My suggestion for long-term architecture would be to use Murano for
> catalog/metadata information (for images/templates/whatever) and move the
> block-streaming drivers into Cinder, and get rid of the Glance project
> entirely. Murano would then become the catalog/registry of objects in the
> OpenStack world, Cinder would be the thing that manages and streams blocks
> of data or block devices, and Glance could go away. Imagine it... OpenStack
> actually *reducing* the number of projects instead of expanding! :)
>

I think it is good to mention the idea of shrinking the overall OpenStack
code base. The fact that the best code offers a lot of features without a
hugely expanded codebase often seems forgotten--perhaps because it is
somewhat incompatible with our low-barrier-to-entry model of development.

However, as a mild defense of Glance's place in the OpenStack ecosystem,
I'm not sure yet that a general catalog/metadata service would be a proper
replacement. There are two key distinctions between Glance and a
catalog/metadata service. One is that Glance *owns* the reference to the
underlying data--meaning Glance can control the consistency of its
references. I.e. you should not be able to delete the image data out from
underneath Glance while the Image entry exists,

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Jay Pipes]

On 12/06/2013 01:53 AM, Keith Bray wrote:

Jay, I don't see reduction.  I count -Glance and +Murano in your email,
which is net zero addition of projects I think.


Murano is already in the OpenStack world and has been for some time now...

> Did I miss something?

Template catalog functionality could go into Heat in the short term with
no new project additions. It could be built in a way that it would be
easy to break it out and move it elsewhere in the future, similar to how
scaling resources is being incubated in Heat, but written in a way that
it could run standalone or be broken out if needed.


The point is that Murano already has this. I'd say make Murano better by 
making it (only slightly more) flexible, move the block drivers out of 
Glance and into Cinder, and say a farewell to Glance. I just don't see 
the need for a Glance project if there is a flexible catalog service. 
Cinder is for block devices/streaming.


Best,
-jay


-Keith

On Dec 5, 2013 11:35 PM, Jay Pipes  wrote:
On 12/05/2013 04:25 PM, Clint Byrum wrote:

Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:

Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:

On Dec 5, 2013, at 10:10 AM, Clint Byrum 
  wrote:


Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:

Why not just use glance?



I've asked that question a few times, and I think I can collate the
responses I've received below. I think enhancing glance to do these
things is on the table:

1. Glance is for big blobs of data not tiny templates.
2. Versioning of a single resource is desired.
3. Tagging/classifying/listing/sorting
4. Glance is designed to expose the uploaded blobs to nova, not users

My responses:

1: Irrelevant. Smaller things will fit in it just fine.


Fitting is one thing, optimizations around particular assumptions about the 
size of data and the frequency of reads/writes might be an issue, but I admit 
to ignorance about those details in Glance.



Optimizations can be improved for various use cases. The design, however,
has no assumptions that I know about that would invalidate storing blobs
of yaml/json vs. blobs of kernel/qcow2/raw image.


I think we are getting out into the weeds a little bit here. It is important to 
think about these apis in terms of what they actually do, before the decision 
of combining them or not can be made.

I think of HeatR as a template storage service, it provides extra data and 
operations on templates. HeatR should not care about how those templates are 
stored.
Glance is an image storage service, it provides extra data and operations on 
images (not blobs), and it happens to use swift as a backend.

If HeatR and Glance were combined, it would result in taking two very different 
types of data (template metadata vs image metadata) and mashing them into one 
service. How would adding the complexity of HeatR benefit Glance, when they are 
dealing with conceptually  two very different types of data? For instance, 
should a template ever

care about the field "minRam" that is stored with an image? Combining
them adds a huge development complexity with a very small operations
payoff, and so Openstack is already so operationally complex that HeatR
as a separate service would be knowledgeable. Only clients of Heat will
ever care about data and operations on templates, so I move that HeatR
becomes it's own service, or becomes part of Heat.




I spoke at length via G+ with Randall and Tim about this earlier today.
I think I understand the impetus for all of this a little better now.

Basically what I'm suggesting is that Glance is only narrow in scope
because that was the only object that OpenStack needed a catalog for
before now.

However, the overlap between a catalog of images and a catalog of
templates is quite comprehensive. The individual fields that matter to
images are different than the ones that matter to templates, but that
is a really minor detail isn't it?

I would suggest that Glance be slightly expanded in scope to be an
object catalog. Each object type can have its own set of fields that
matter to it.

This doesn't have to be a minor change to glance to still have many
advantages over writing something from scratch and asking people to
deploy another service that is 99% the same as Glance.


My suggestion for long-term architecture would be to use Murano for
catalog/metadata information (for images/templates/whatever) and move
the block-streaming drivers into Cinder, and get rid of the Glance
project entirely. Murano would then become the catalog/registry of
objects in the OpenStack world, Cinder would be the thing that manages
and streams blocks of data or block devices, and Glance could go away.
Imagine it... OpenStack actually *reducing* the number of projects
instead of expanding! :)

Best,
-jay

Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/li

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Flavio Percoco]

On 05/12/13 15:36 -0800, Mark Washenberger wrote:

On Thu, Dec 5, 2013 at 3:11 PM, Randall Burt 
wrote:
   On Dec 5, 2013, at 4:45 PM, Steve Baker 
wrote:
   On 12/06/2013 10:46 AM, Mark Washenberger wrote:
   On Thu, Dec 5, 2013 at 1:05 PM, Vishvananda Ishaya <
   vishvana...@gmail.com> wrote:


[snip]


   This is not completely correct. Glance already supports
   something akin to templates. You can create an "image" with
   metadata properties that specifies a complex block device
   mapping which would allow for multiple volumes and images to
   connected to the vm at boot time. This is functionally a
   template for a single vm.

   Glance is pretty useless if is just an "image storage" service,
   we already have other places that can store bits (swift,
   cinder). It is much more valuable as a searchable repository of
   bootable templates. I don't see any reason why this idea
   couldn't be extended to include more complex templates that
   could include more than one vm.


   FWIW I agree with all of this. I think Glance's real role in
   OpenStack is as a helper and optionally as a gatekeeper for the
   category of "stuff Nova can boot". So any parameter that affects
   what Nova is going to boot should in my view be something Glance
   can be aware of. This list of parameters *could* grow to include
   multiple device images, attached volumes, and other things that
   currently live in the realm of flavors such as extra hardware
   requirements and networking aspects.

   Just so things don't go too crazy, I'll add that since Nova is
   generally focused on provisioning individual VMs, anything above
   the level of an individual VM should be out of scope for Glance.

   I think Glance should alter its approach to be less generally
   agnostic about the contents of the objects it hosts. Right now, we
   are just starting to do this with images, as we slowly advance on
   offering server side format conversion. We could find similar use
   cases for single vm templates.

   The average heat template would provision more than one VM, plus any
   number of other cloud resources.

   An image is required to provision a single nova server;
   a template is required to provision a single heat stack.

   Hopefully the above "single vm" policy could be reworded to be agnostic
   to the service which consumes the object that glance is storing.

   To add to this, is it that Glance wants to be *more* integrated and geared
   towards vm or container images or that Glance wishes to have more intimate
   knowledge of the things its cataloging *regardless of what those things
   actually might be*? The reason I ask is that Glance supporting only "single
   vm templates" when Heat orchestrates the entire (or almost entire) spectrum
   of core and integrated projects means that its suitability as a candidate
   for a template repository plummets quite a bit.



Yes, I missed the boat a little bit there. I agree Glance could operate as a
repo for these kinds of templates. I don't know about expanding much further
beyond the Nova / Heat stack. But within that stack, I think the use cases are
largely the same.

It seems like heat templates likely have built-in relationships with vm
templates / images that would be really nice track closely in the Glance data
model--for example if you wanted something like a notification when deleting an
image would invalidate a template you've created. Another advantage is the
sharing model--Glance is still aiming to become something of an image
marketplace, and that kind of sharing is something that I see being very useful
for Heat as well.

Does this response sound more in line? Sorry I'm still catching up on the
thread from before it was tagged with [Glance].


FWIW, during last week's meeting, we discussed a bit about image
templates and what they should do. The discussion was oriented to
having support for things like OVF. This sounds like a great
opportunity to expand that concept to something that won't be useful
just for nova but for Heat as well. As Mark mentioned, it's more about
making glance aware about the difference between a template and an
image and their types.

None of this is written on stone. The discussion came out at the
summit and we brought it up at one of our meetings. So, lets make sure
we can cover the required needs.

There's still no blueprint but I created this[0] etherpad where we
could start writing the needs for Heat and other templates.

I agree with Mark. I don't think Glance should expand much further
beyond the Nova / Heat stack and akin.

[0] https://etherpad.openstack.org/p/glance-templates

Cheers,
FF

--
@flaper87
Flavio Per

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Keith Bray]

Jay, I don't see reduction.  I count -Glance and +Murano in your email, which 
is net zero addition of projects I think. Did I miss something?  Template 
catalog functionality could go into Heat in the short term with no new project 
additions. It could be built in a way that it would be easy to break it out and 
move it elsewhere in the future, similar to how scaling resources is being 
incubated in Heat, but written in a way that it could run standalone or be 
broken out if needed.

-Keith

On Dec 5, 2013 11:35 PM, Jay Pipes  wrote:
On 12/05/2013 04:25 PM, Clint Byrum wrote:
> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
>>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
 On Dec 5, 2013, at 10:10 AM, Clint Byrum 
   wrote:

> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> Why not just use glance?
>>
>
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:
>
> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
>
> My responses:
>
> 1: Irrelevant. Smaller things will fit in it just fine.

 Fitting is one thing, optimizations around particular assumptions about 
 the size of data and the frequency of reads/writes might be an issue, but 
 I admit to ignorance about those details in Glance.

>>>
>>> Optimizations can be improved for various use cases. The design, however,
>>> has no assumptions that I know about that would invalidate storing blobs
>>> of yaml/json vs. blobs of kernel/qcow2/raw image.
>>
>> I think we are getting out into the weeds a little bit here. It is important 
>> to think about these apis in terms of what they actually do, before the 
>> decision of combining them or not can be made.
>>
>> I think of HeatR as a template storage service, it provides extra data and 
>> operations on templates. HeatR should not care about how those templates are 
>> stored.
>> Glance is an image storage service, it provides extra data and operations on 
>> images (not blobs), and it happens to use swift as a backend.
>>
>> If HeatR and Glance were combined, it would result in taking two very 
>> different types of data (template metadata vs image metadata) and mashing 
>> them into one service. How would adding the complexity of HeatR benefit 
>> Glance, when they are dealing with conceptually two very different types of 
>> data? For instance, should a template ever care about the field "minRam" 
>> that is stored with an image? Combining them adds a huge development 
>> complexity with a very small operations payoff, and so Openstack is already 
>> so operationally complex that HeatR as a separate service would be 
>> knowledgeable. Only clients of Heat will ever care about data and operations 
>> on templates, so I move that HeatR becomes it's own service, or becomes part 
>> of Heat.
>>
>
> I spoke at length via G+ with Randall and Tim about this earlier today.
> I think I understand the impetus for all of this a little better now.
>
> Basically what I'm suggesting is that Glance is only narrow in scope
> because that was the only object that OpenStack needed a catalog for
> before now.
>
> However, the overlap between a catalog of images and a catalog of
> templates is quite comprehensive. The individual fields that matter to
> images are different than the ones that matter to templates, but that
> is a really minor detail isn't it?
>
> I would suggest that Glance be slightly expanded in scope to be an
> object catalog. Each object type can have its own set of fields that
> matter to it.
>
> This doesn't have to be a minor change to glance to still have many
> advantages over writing something from scratch and asking people to
> deploy another service that is 99% the same as Glance.

My suggestion for long-term architecture would be to use Murano for
catalog/metadata information (for images/templates/whatever) and move
the block-streaming drivers into Cinder, and get rid of the Glance
project entirely. Murano would then become the catalog/registry of
objects in the OpenStack world, Cinder would be the thing that manages
and streams blocks of data or block devices, and Glance could go away.
Imagine it... OpenStack actually *reducing* the number of projects
instead of expanding! :)

Best,
-jay

Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Jay Pipes]

On 12/05/2013 04:25 PM, Clint Byrum wrote:

Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:

Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:

On Dec 5, 2013, at 10:10 AM, Clint Byrum 
  wrote:


Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:

Why not just use glance?



I've asked that question a few times, and I think I can collate the
responses I've received below. I think enhancing glance to do these
things is on the table:

1. Glance is for big blobs of data not tiny templates.
2. Versioning of a single resource is desired.
3. Tagging/classifying/listing/sorting
4. Glance is designed to expose the uploaded blobs to nova, not users

My responses:

1: Irrelevant. Smaller things will fit in it just fine.


Fitting is one thing, optimizations around particular assumptions about the 
size of data and the frequency of reads/writes might be an issue, but I admit 
to ignorance about those details in Glance.



Optimizations can be improved for various use cases. The design, however,
has no assumptions that I know about that would invalidate storing blobs
of yaml/json vs. blobs of kernel/qcow2/raw image.


I think we are getting out into the weeds a little bit here. It is important to 
think about these apis in terms of what they actually do, before the decision 
of combining them or not can be made.

I think of HeatR as a template storage service, it provides extra data and 
operations on templates. HeatR should not care about how those templates are 
stored.
Glance is an image storage service, it provides extra data and operations on 
images (not blobs), and it happens to use swift as a backend.

If HeatR and Glance were combined, it would result in taking two very different types of 
data (template metadata vs image metadata) and mashing them into one service. How would 
adding the complexity of HeatR benefit Glance, when they are dealing with conceptually 
two very different types of data? For instance, should a template ever care about the 
field "minRam" that is stored with an image? Combining them adds a huge 
development complexity with a very small operations payoff, and so Openstack is already 
so operationally complex that HeatR as a separate service would be knowledgeable. Only 
clients of Heat will ever care about data and operations on templates, so I move that 
HeatR becomes it's own service, or becomes part of Heat.



I spoke at length via G+ with Randall and Tim about this earlier today.
I think I understand the impetus for all of this a little better now.

Basically what I'm suggesting is that Glance is only narrow in scope
because that was the only object that OpenStack needed a catalog for
before now.

However, the overlap between a catalog of images and a catalog of
templates is quite comprehensive. The individual fields that matter to
images are different than the ones that matter to templates, but that
is a really minor detail isn't it?

I would suggest that Glance be slightly expanded in scope to be an
object catalog. Each object type can have its own set of fields that
matter to it.

This doesn't have to be a minor change to glance to still have many
advantages over writing something from scratch and asking people to
deploy another service that is 99% the same as Glance.


My suggestion for long-term architecture would be to use Murano for 
catalog/metadata information (for images/templates/whatever) and move 
the block-streaming drivers into Cinder, and get rid of the Glance 
project entirely. Murano would then become the catalog/registry of 
objects in the OpenStack world, Cinder would be the thing that manages 
and streams blocks of data or block devices, and Glance could go away. 
Imagine it... OpenStack actually *reducing* the number of projects 
instead of expanding! :)


Best,
-jay

Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Angus Salkeld]

On 05/12/13 11:35 -0500, James Slagle wrote:

On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum  wrote:

Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:

Why not just use glance?



I've asked that question a few times, and I think I can collate the
responses I've received below. I think enhancing glance to do these
things is on the table:


I'm actually interested in the use cases laid out by Heater from both
a template perspective and image perspective.  For the templates, as
Robert mentioned, Tuskar needs a solution for this requirement, since
it's deploying using templates.  For the images, we have the concept
of a "golden" image in TripleO and are heavily focused on image based
deployments.  Therefore, it seems to make sense that TripleO also
needs a way to version/tag known good images.

Given that, I think it makes sense  to do this in a way so that it's
consumable for things other than just templates.  In fact, you can
almost s/template/image/g on the Heater wiki page, and it pretty well
lays out what I'd like to see for images as well.


1. Glance is for big blobs of data not tiny templates.
2. Versioning of a single resource is desired.
3. Tagging/classifying/listing/sorting
4. Glance is designed to expose the uploaded blobs to nova, not users

My responses:

1: Irrelevant. Smaller things will fit in it just fine.

2: The swift API supports versions. We could also have git as a
backend.


I would definitely like to see a git backend for versioning.  No
reason to reimplement a different solution for what already works
well.  I'm not sure we'd want to put a whole image into git though.
Perhaps just it's manifest (installed components, software versions,
etc) in json format would go into git, and that would be associated
back to the binary image via uuid.  That would even make it easy to
diff changes between versions, etc.


This feels like something we can add as an optional feature
without exploding Glance's scope and I imagine it would actually be a
welcome feature for image authors as well. Think about Ubuntu maintaining
official images. If they can keep the ID the same and just add a version
(allowing users to lock down to a version if updated images cause issue)
that seems like a really cool feature for images _and_ templates.

3: I'm sure glance image users would love to have those too.

4: Irrelevant. Heat will need to download templates just like nova, and
making images publicly downloadable is also a thing in glance.

It strikes me that this might be a silo problem instead of an
actual design problem. Folk should not be worried about jumping into
Glance and adding features. Unless of course the Glance folk have
reservations? (adding glance tag to the subject)


I'm +1 for adding these types of features to glance, or at least
something common, instead of making it specific to Heat templates.


+1 from me too, I don't see a reason for Heater at all, just add
the features to glance. Especially since Tuskar/TripelO have
overlapping needs. Just seems crazy not to all benefit from combining
efforts. (assuming glance are happy with the extension).

And there is a huge overhead to a new project.

-Angus




--
-- James Slagle
--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Mark Washenberger]

On Thu, Dec 5, 2013 at 3:11 PM, Randall Burt wrote:

>  On Dec 5, 2013, at 4:45 PM, Steve Baker 
>  wrote:
>
>  On 12/06/2013 10:46 AM, Mark Washenberger wrote:
>
>
>
>
> On Thu, Dec 5, 2013 at 1:05 PM, Vishvananda Ishaya 
> wrote:
>
>>
>> On Dec 5, 2013, at 12:42 PM, Andrew Plunk 
>> wrote:
>>
>> >> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>> >>> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>> >>> wrote:
>> >>>
>>  Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> > Why not just use glance?
>> >
>> 
>>  I've asked that question a few times, and I think I can collate the
>>  responses I've received below. I think enhancing glance to do these
>>  things is on the table:
>> 
>>  1. Glance is for big blobs of data not tiny templates.
>>  2. Versioning of a single resource is desired.
>>  3. Tagging/classifying/listing/sorting
>>  4. Glance is designed to expose the uploaded blobs to nova, not users
>> 
>>  My responses:
>> 
>>  1: Irrelevant. Smaller things will fit in it just fine.
>> >>>
>> >>> Fitting is one thing, optimizations around particular assumptions
>> about the size of data and the frequency of reads/writes might be an issue,
>> but I admit to ignorance about those details in Glance.
>> >>>
>> >>
>> >> Optimizations can be improved for various use cases. The design,
>> however,
>> >> has no assumptions that I know about that would invalidate storing
>> blobs
>> >> of yaml/json vs. blobs of kernel/qcow2/raw image.
>> >
>> > I think we are getting out into the weeds a little bit here. It is
>> important to think about these apis in terms of what they actually do,
>> before the decision of combining them or not can be made.
>> >
>> > I think of HeatR as a template storage service, it provides extra data
>> and operations on templates. HeatR should not care about how those
>> templates are stored.
>> > Glance is an image storage service, it provides extra data and
>> operations on images (not blobs), and it happens to use swift as a backend.
>>
>>  This is not completely correct. Glance already supports something akin
>> to templates. You can create an "image" with metadata properties that
>> specifies a complex block device mapping which would allow for multiple
>> volumes and images to connected to the vm at boot time. This is
>> functionally a template for a single vm.
>>
>> Glance is pretty useless if is just an "image storage" service, we
>> already have other places that can store bits (swift, cinder). It is much
>> more valuable as a searchable repository of bootable templates. I don't see
>> any reason why this idea couldn't be extended to include more complex
>> templates that could include more than one vm.
>>
>
>  FWIW I agree with all of this. I think Glance's real role in OpenStack
> is as a helper and optionally as a gatekeeper for the category of "stuff
> Nova can boot". So any parameter that affects what Nova is going to boot
> should in my view be something Glance can be aware of. This list of
> parameters *could* grow to include multiple device images, attached
> volumes, and other things that currently live in the realm of flavors such
> as extra hardware requirements and networking aspects.
>
>  Just so things don't go too crazy, I'll add that since Nova is generally
> focused on provisioning individual VMs, anything above the level of an
> individual VM should be out of scope for Glance.
>
>  I think Glance should alter its approach to be less generally agnostic
> about the contents of the objects it hosts. Right now, we are just starting
> to do this with images, as we slowly advance on offering server side format
> conversion. We could find similar use cases for single vm templates.
>
>
> The average heat template would provision more than one VM, plus any
> number of other cloud resources.
>
> An image is required to provision a single nova server;
> a template is required to provision a single heat stack.
>
> Hopefully the above "single vm" policy could be reworded to be agnostic to
> the service which consumes the object that glance is storing.
>
>
>  To add to this, is it that Glance wants to be *more* integrated and
> geared towards vm or container images or that Glance wishes to have more
> intimate knowledge of the things its cataloging *regardless of what those
> things actually might be*? The reason I ask is that Glance supporting only
> "single vm templates" when Heat orchestrates the entire (or almost entire)
> spectrum of core and integrated projects means that its suitability as a
> candidate for a template repository plummets quite a bit.
>
>
Yes, I missed the boat a little bit there. I agree Glance could operate as
a repo for these kinds of templates. I don't know about expanding much
further beyond the Nova / Heat stack. But within that stack, I think the
use cases are largely the same.

It seems like heat templates likely have built-in relationships with v

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 5, 2013, at 4:45 PM, Steve Baker 
mailto:sba...@redhat.com>>
 wrote:

On 12/06/2013 10:46 AM, Mark Washenberger wrote:



On Thu, Dec 5, 2013 at 1:05 PM, Vishvananda Ishaya 
mailto:vishvana...@gmail.com>> wrote:

On Dec 5, 2013, at 12:42 PM, Andrew Plunk 
mailto:andrew.pl...@rackspace.com>> wrote:

>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>>> On Dec 5, 2013, at 10:10 AM, Clint Byrum >> fewbar.com>
>>> wrote:
>>>
 Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> Why not just use glance?
>

 I've asked that question a few times, and I think I can collate the
 responses I've received below. I think enhancing glance to do these
 things is on the table:

 1. Glance is for big blobs of data not tiny templates.
 2. Versioning of a single resource is desired.
 3. Tagging/classifying/listing/sorting
 4. Glance is designed to expose the uploaded blobs to nova, not users

 My responses:

 1: Irrelevant. Smaller things will fit in it just fine.
>>>
>>> Fitting is one thing, optimizations around particular assumptions about the 
>>> size of data and the frequency of reads/writes might be an issue, but I 
>>> admit to ignorance about those details in Glance.
>>>
>>
>> Optimizations can be improved for various use cases. The design, however,
>> has no assumptions that I know about that would invalidate storing blobs
>> of yaml/json vs. blobs of kernel/qcow2/raw image.
>
> I think we are getting out into the weeds a little bit here. It is important 
> to think about these apis in terms of what they actually do, before the 
> decision of combining them or not can be made.
>
> I think of HeatR as a template storage service, it provides extra data and 
> operations on templates. HeatR should not care about how those templates are 
> stored.
> Glance is an image storage service, it provides extra data and operations on 
> images (not blobs), and it happens to use swift as a backend.

This is not completely correct. Glance already supports something akin to 
templates. You can create an "image" with metadata properties that specifies a 
complex block device mapping which would allow for multiple volumes and images 
to connected to the vm at boot time. This is functionally a template for a 
single vm.

Glance is pretty useless if is just an "image storage" service, we already have 
other places that can store bits (swift, cinder). It is much more valuable as a 
searchable repository of bootable templates. I don't see any reason why this 
idea couldn't be extended to include more complex templates that could include 
more than one vm.

FWIW I agree with all of this. I think Glance's real role in OpenStack is as a 
helper and optionally as a gatekeeper for the category of "stuff Nova can 
boot". So any parameter that affects what Nova is going to boot should in my 
view be something Glance can be aware of. This list of parameters *could* grow 
to include multiple device images, attached volumes, and other things that 
currently live in the realm of flavors such as extra hardware requirements and 
networking aspects.

Just so things don't go too crazy, I'll add that since Nova is generally 
focused on provisioning individual VMs, anything above the level of an 
individual VM should be out of scope for Glance.

I think Glance should alter its approach to be less generally agnostic about 
the contents of the objects it hosts. Right now, we are just starting to do 
this with images, as we slowly advance on offering server side format 
conversion. We could find similar use cases for single vm templates.


The average heat template would provision more than one VM, plus any number of 
other cloud resources.

An image is required to provision a single nova server;
a template is required to provision a single heat stack.

Hopefully the above "single vm" policy could be reworded to be agnostic to the 
service which consumes the object that glance is storing.

To add to this, is it that Glance wants to be *more* integrated and geared 
towards vm or container images or that Glance wishes to have more intimate 
knowledge of the things its cataloging *regardless of what those things 
actually might be*? The reason I ask is that Glance supporting only "single vm 
templates" when Heat orchestrates the entire (or almost entire) spectrum of 
core and integrated projects means that its suitability as a candidate for a 
template repository plummets quite a bit.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Steve Baker]

On 12/06/2013 10:46 AM, Mark Washenberger wrote:
>
>
>
> On Thu, Dec 5, 2013 at 1:05 PM, Vishvananda Ishaya
> mailto:vishvana...@gmail.com>> wrote:
>
>
> On Dec 5, 2013, at 12:42 PM, Andrew Plunk
> mailto:andrew.pl...@rackspace.com>>
> wrote:
>
> >> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
> >>> On Dec 5, 2013, at 10:10 AM, Clint Byrum  >
> >>> wrote:
> >>>
>  Excerpts from Monty Taylor's message of 2013-12-04 17:54:45
> -0800:
> > Why not just use glance?
> >
> 
>  I've asked that question a few times, and I think I can
> collate the
>  responses I've received below. I think enhancing glance to do
> these
>  things is on the table:
> 
>  1. Glance is for big blobs of data not tiny templates.
>  2. Versioning of a single resource is desired.
>  3. Tagging/classifying/listing/sorting
>  4. Glance is designed to expose the uploaded blobs to nova,
> not users
> 
>  My responses:
> 
>  1: Irrelevant. Smaller things will fit in it just fine.
> >>>
> >>> Fitting is one thing, optimizations around particular
> assumptions about the size of data and the frequency of
> reads/writes might be an issue, but I admit to ignorance about
> those details in Glance.
> >>>
> >>
> >> Optimizations can be improved for various use cases. The
> design, however,
> >> has no assumptions that I know about that would invalidate
> storing blobs
> >> of yaml/json vs. blobs of kernel/qcow2/raw image.
> >
> > I think we are getting out into the weeds a little bit here. It
> is important to think about these apis in terms of what they
> actually do, before the decision of combining them or not can be made.
> >
> > I think of HeatR as a template storage service, it provides
> extra data and operations on templates. HeatR should not care
> about how those templates are stored.
> > Glance is an image storage service, it provides extra data and
> operations on images (not blobs), and it happens to use swift as a
> backend.
>
> This is not completely correct. Glance already supports something
> akin to templates. You can create an "image" with metadata
> properties that specifies a complex block device mapping which
> would allow for multiple volumes and images to connected to the vm
> at boot time. This is functionally a template for a single vm.
>
> Glance is pretty useless if is just an "image storage" service, we
> already have other places that can store bits (swift, cinder). It
> is much more valuable as a searchable repository of bootable
> templates. I don't see any reason why this idea couldn't be
> extended to include more complex templates that could include more
> than one vm.
>
>
> FWIW I agree with all of this. I think Glance's real role in OpenStack
> is as a helper and optionally as a gatekeeper for the category of
> "stuff Nova can boot". So any parameter that affects what Nova is
> going to boot should in my view be something Glance can be aware of.
> This list of parameters *could* grow to include multiple device
> images, attached volumes, and other things that currently live in the
> realm of flavors such as extra hardware requirements and networking
> aspects.
>
> Just so things don't go too crazy, I'll add that since Nova is
> generally focused on provisioning individual VMs, anything above the
> level of an individual VM should be out of scope for Glance.
>
> I think Glance should alter its approach to be less generally agnostic
> about the contents of the objects it hosts. Right now, we are just
> starting to do this with images, as we slowly advance on offering
> server side format conversion. We could find similar use cases for
> single vm templates.
>

The average heat template would provision more than one VM, plus any
number of other cloud resources.

An image is required to provision a single nova server;
a template is required to provision a single heat stack.

Hopefully the above "single vm" policy could be reworded to be agnostic
to the service which consumes the object that glance is storing.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Tim Schnell's message of 2013-12-05 13:07:17 -0800:
> On 12/5/13 12:17 PM, "Clint Byrum"  wrote:
> 
> >Excerpts from Tim Schnell's message of 2013-12-05 09:49:03 -0800:
> >> 
> >> On 12/5/13 11:33 AM, "Randall Burt"  wrote:
> >> 
> >> >On Dec 5, 2013, at 11:10 AM, Clint Byrum 
> >> > wrote:
> >I would love to hear about "future requirements" for Heater that won't
> >fit into Glance. But thus far, I am struggling to see where this could
> >go that wouldn't be a place Glance also wants to go.
> 
> I know we covered this a bit in the hangout but one possibility that I
> think I failed to articulate is the fact that the template is a parseable
> data representation while an image is a binary object. In my original
> email I mentioned having a discussion about moving some of the "template"
> metadata like application information and documentation into the HOT
> specification.
> 

I've gone on record before as saying that having things in the template
schema that should be in the catalog is a mistake. I stand by that
assertion. However that is just for HOT, I think for a full blown
packaging format, that may not be the case and so I agree there may be
a case for parsing the contents of the uploaded object.

> This has important ramifications for the user experience as well as
> whether or not abstracting Glance to "one-size-fits-all" is actually
> feasible at an implementation level. Ideally the Heat template would serve
> as the source of truth for information like "application-version" and
> "template documentation", in the original Heater proposal this is
> important because this data would have to be extracted from the template
> and stored in a metadata field if it is relevant to the way that Heater
> allows you to search or index different templates.
> 

Given a separate service, how wold you design things differently than
glance to make this parsing work. As I recall, when you create an
image for glance, it just makes a record in the registry and a place to
upload it to in the storage. Would you do this differently? Does that
invalidate parsing?

Presumably after uploading your template to storage, you would need a
process which inspects the data and populates the registry. How would
glance's design preclude this given the more generic object schema that
we discussed.

> If we had to pass a duplicate version of this metadata in the HTTP Request
> we could avoid grabbing this information from the template but we open up
> data integrity issues. What happens when the end-user updates the metadata
> but not the template? At the same time, I would reiterate that we can't
> just make this data a "Heater/Template Catalog" metadata issue because we
> want the experience of the template to be consistent regardless of how the
> end-user obtains the template. This is why I would consider this
> information to belong as part of the HOT specification.
> 

Don't take this the wrong way, but that would be absurd. Either you
maintain it as the source of truth, or it is derived from the source
of truth. "What if" I upload lolcatz instead of a template? I can
has meta-data where there is none? Let's stick to the main question:
Can Glance's scope be expanded to have data derived from the objects
themselves. I think the answer is a resounding yes.

> This is relevant to the Glance conversation because, just by the template
> existing in a different format than an image, we have already introduced
> logic in Glance that cannot be abstracted. We could most likely build an
> abstraction layer that covers some large percentage of the overlap in CRUD
> operations between images and templates but I do not think that we can
> expect to do this for every use case with Heat templates.
> 

Who said that there would not be logic per object type? The point was to
have a generic object registry, but not an abstract object registry. You
still instantiate them by uploading content and in doing so the objects
take on all of the attributes of their class. So images are opaque.
Things that allow parsing are not.

> I know that I cannot provide a future use case that would exemplify this
> problem but I can imagine that if we choose this design path going
> forward, then we will continue to increase the scope of Glance with every
> new type of object (maybe Solum assemblies?).
> 

If Heater were its own service, and Solum assemblies were added,
requiring some new things like multi-file objects, would you support
Solum developers wanting to write a new registry because their use-case
is slightly different?

> I do agree that in general, we should continue to attempt to drive
> consistency and congruency across Openstack projects when possible. I am
> just concerned that this design will take us down rabbit holes both now
> and in the future when it comes to the implementation details. In the
> bigger picture I am attempting to weigh the value of building this
> complexity into Glance against putting Heater into its own project or

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Mark Washenberger]

On Thu, Dec 5, 2013 at 1:05 PM, Vishvananda Ishaya wrote:

>
> On Dec 5, 2013, at 12:42 PM, Andrew Plunk 
> wrote:
>
> >> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
> >>> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
> >>> wrote:
> >>>
>  Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> > Why not just use glance?
> >
> 
>  I've asked that question a few times, and I think I can collate the
>  responses I've received below. I think enhancing glance to do these
>  things is on the table:
> 
>  1. Glance is for big blobs of data not tiny templates.
>  2. Versioning of a single resource is desired.
>  3. Tagging/classifying/listing/sorting
>  4. Glance is designed to expose the uploaded blobs to nova, not users
> 
>  My responses:
> 
>  1: Irrelevant. Smaller things will fit in it just fine.
> >>>
> >>> Fitting is one thing, optimizations around particular assumptions
> about the size of data and the frequency of reads/writes might be an issue,
> but I admit to ignorance about those details in Glance.
> >>>
> >>
> >> Optimizations can be improved for various use cases. The design,
> however,
> >> has no assumptions that I know about that would invalidate storing blobs
> >> of yaml/json vs. blobs of kernel/qcow2/raw image.
> >
> > I think we are getting out into the weeds a little bit here. It is
> important to think about these apis in terms of what they actually do,
> before the decision of combining them or not can be made.
> >
> > I think of HeatR as a template storage service, it provides extra data
> and operations on templates. HeatR should not care about how those
> templates are stored.
> > Glance is an image storage service, it provides extra data and
> operations on images (not blobs), and it happens to use swift as a backend.
>
> This is not completely correct. Glance already supports something akin to
> templates. You can create an "image" with metadata properties that
> specifies a complex block device mapping which would allow for multiple
> volumes and images to connected to the vm at boot time. This is
> functionally a template for a single vm.
>
> Glance is pretty useless if is just an "image storage" service, we already
> have other places that can store bits (swift, cinder). It is much more
> valuable as a searchable repository of bootable templates. I don't see any
> reason why this idea couldn't be extended to include more complex templates
> that could include more than one vm.
>

FWIW I agree with all of this. I think Glance's real role in OpenStack is
as a helper and optionally as a gatekeeper for the category of "stuff Nova
can boot". So any parameter that affects what Nova is going to boot should
in my view be something Glance can be aware of. This list of parameters
*could* grow to include multiple device images, attached volumes, and other
things that currently live in the realm of flavors such as extra hardware
requirements and networking aspects.

Just so things don't go too crazy, I'll add that since Nova is generally
focused on provisioning individual VMs, anything above the level of an
individual VM should be out of scope for Glance.

I think Glance should alter its approach to be less generally agnostic
about the contents of the objects it hosts. Right now, we are just starting
to do this with images, as we slowly advance on offering server side format
conversion. We could find similar use cases for single vm templates.

It would be fantastic if we could figure out how to turn this idea into
some actionable work in late I/early J. It could be a fun thing to work on
at the midcycle meetup.


>
> We have discussed the future of glance a number of times, and if it is
> really just there to serve up blobs of data + metadata about images, it
> should go away. Glance should offer something more like the AWS image
> search console. And this could clearly support more than just images, you
> should be able to search for and launch more complicated templates as well.
>
> >
> > If HeatR and Glance were combined, it would result in taking two very
> different types of data (template metadata vs image metadata) and mashing
> them into one service. How would adding the complexity of HeatR benefit
> Glance, when they are dealing with conceptually two very different types of
> data? For instance, should a template ever care about the field "minRam"
> that is stored with an image?
>
>
> I don't see these as significantly different types of metadata. Metadata
> for heat templates might be a bit more broad (minFlavor?) I would think
> that a template would care about constraints like this, especially when you
> consider that a user might want to give a command to launch a template but
> then override certain characteristics.
>
> Vish
>
> > Combining them adds a huge development complexity with a very small
> operations payoff, and so Openstack is already so operationally complex
> that HeatR 

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Tim Schnell]

The original intention was to propose the Heater functionality as part of Heat. 
I would just like to clarify that having Heater as a separate project is not 
because of the impact of the gerrit reviews. The proposal for a separate Core 
team or sub-project team was to solve the impact on reviews. Heater is now 
being proposed as a separate project from Heat because previous conversations 
with the Heat Core team has lead us to believe that Heat is not in the business 
of managing templates.

Thanks,
Tim

From: Tim Bell mailto:tim.b...@cern.ch>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Thursday, December 5, 2013 2:13 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal

Completely agree with Brad… a new project for this is not what is needed.

>From an operator’s point of view, it is a REAL, REAL, REAL pain to be 
>configuring yet another project, yet another set of Puppet/Chef recipes, 
>additional monitoring, service nodes, new databases, more documentation, 
>further packages, pre-requisites and tough divergent roles follow…

While I appreciate that there is an impact from having reviews, this should not 
be the cause for creating more complexity for those who wish to use the 
functionality.

I’d also suggest the Heater guys talk to the Murano guys since my ideal 
scenario is for a simple user kiosk where I can ask for an app, be asked for 
some configuration details and then deploy. The two projects appear to be 
heading towards a converged solution.

Tim

From: Brad Topol [mailto:bto...@us.ibm.com]
Sent: 05 December 2013 20:06
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal

Lots of good discussion on this topic.   One thing I would like to point out is 
that we get feedback that OpenStack has too many projects as it is and 
customers get confused on how much of OpenStack they need to install.  So in 
the spirit of trying to help insure OpenStack does not continue to reinforce 
this perception, I am hoping that Heater functionality finds a home in either 
Glance or Heat.  I don't have a preference of which.   Either of these is 
superior to starting a new project if it can be avoided.

Thanks,

Brad

Brad Topol, Ph.D.
IBM Distinguished Engineer
OpenStack
(919) 543-0646
Internet:  bto...@us.ibm.com<mailto:bto...@us.ibm.com>
Assistant: Kendra Witherspoon (919) 254-0680



From:Randall Burt 
mailto:randall.b...@rackspace.com>>
To:"OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date:    12/05/2013 12:09 PM
Subject:Re: [openstack-dev] [heat] [glance] Heater Proposal




On Dec 5, 2013, at 10:10 AM, Clint Byrum 
mailto:cl...@fewbar.com>>
wrote:

> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> Why not just use glance?
>>
>
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:
>
> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
>
> My responses:
>
> 1: Irrelevant. Smaller things will fit in it just fine.

Fitting is one thing, optimizations around particular assumptions about the 
size of data and the frequency of reads/writes might be an issue, but I admit 
to ignorance about those details in Glance.

> 2: The swift API supports versions. We could also have git as a
> backend. This feels like something we can add as an optional feature
> without exploding Glance's scope and I imagine it would actually be a
> welcome feature for image authors as well. Think about Ubuntu maintaining
> official images. If they can keep the ID the same and just add a version
> (allowing users to lock down to a version if updated images cause issue)
> that seems like a really cool feature for images _and_ templates.

Agreed, though one could argue that using image names and looking up ID's or 
just using ID's as appropriate "sort of" handle this use case, but I agree that 
having image versioning seems a reasonable feature for Glance to have as well.

> 3: I'm sure glance image users would love to have those too.

And image metadata is already there so we don't have to go through those 
discussions all over again ;).

> 4: Irrelevant. Heat will need to download templates just like nova, and
> making images publicly

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
> >Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
> >> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
> >>  wrote:
> >>
> >> > Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> >> >> Why not just use glance?
> >> >>
> >> >
> >> > I've asked that question a few times, and I think I can collate the
> >> > responses I've received below. I think enhancing glance to do these
> >> > things is on the table:
> >> >
> >> > 1. Glance is for big blobs of data not tiny templates.
> >> > 2. Versioning of a single resource is desired.
> >> > 3. Tagging/classifying/listing/sorting
> >> > 4. Glance is designed to expose the uploaded blobs to nova, not users
> >> >
> >> > My responses:
> >> >
> >> > 1: Irrelevant. Smaller things will fit in it just fine.
> >>
> >> Fitting is one thing, optimizations around particular assumptions about 
> >> the size of data and the frequency of reads/writes might be an issue, but 
> >> I admit to ignorance about those details in Glance.
> >>
> >
> >Optimizations can be improved for various use cases. The design, however,
> >has no assumptions that I know about that would invalidate storing blobs
> >of yaml/json vs. blobs of kernel/qcow2/raw image.
> 
> I think we are getting out into the weeds a little bit here. It is important 
> to think about these apis in terms of what they actually do, before the 
> decision of combining them or not can be made.
> 
> I think of HeatR as a template storage service, it provides extra data and 
> operations on templates. HeatR should not care about how those templates are 
> stored.
> Glance is an image storage service, it provides extra data and operations on 
> images (not blobs), and it happens to use swift as a backend.
> 
> If HeatR and Glance were combined, it would result in taking two very 
> different types of data (template metadata vs image metadata) and mashing 
> them into one service. How would adding the complexity of HeatR benefit 
> Glance, when they are dealing with conceptually two very different types of 
> data? For instance, should a template ever care about the field "minRam" that 
> is stored with an image? Combining them adds a huge development complexity 
> with a very small operations payoff, and so Openstack is already so 
> operationally complex that HeatR as a separate service would be 
> knowledgeable. Only clients of Heat will ever care about data and operations 
> on templates, so I move that HeatR becomes it's own service, or becomes part 
> of Heat.
> 

I spoke at length via G+ with Randall and Tim about this earlier today.
I think I understand the impetus for all of this a little better now.

Basically what I'm suggesting is that Glance is only narrow in scope
because that was the only object that OpenStack needed a catalog for
before now.

However, the overlap between a catalog of images and a catalog of
templates is quite comprehensive. The individual fields that matter to
images are different than the ones that matter to templates, but that
is a really minor detail isn't it?

I would suggest that Glance be slightly expanded in scope to be an
object catalog. Each object type can have its own set of fields that
matter to it.

This doesn't have to be a minor change to glance to still have many
advantages over writing something from scratch and asking people to
deploy another service that is 99% the same as Glance.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Tim Schnell]

On 12/5/13 12:17 PM, "Clint Byrum"  wrote:


>Excerpts from Tim Schnell's message of 2013-12-05 09:49:03 -0800:
>> 
>> On 12/5/13 11:33 AM, "Randall Burt"  wrote:
>> 
>> >On Dec 5, 2013, at 11:10 AM, Clint Byrum 
>> > wrote:
>> >
>> >> Excerpts from James Slagle's message of 2013-12-05 08:35:12 -0800:
>> >>> On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum 
>>wrote:
>>  Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> > Why not just use glance?
>> > 
>>  
>>  I've asked that question a few times, and I think I can collate the
>>  responses I've received below. I think enhancing glance to do these
>>  things is on the table:
>> >>> 
>> >>> I'm actually interested in the use cases laid out by Heater from
>>both
>> >>> a template perspective and image perspective.  For the templates, as
>> >>> Robert mentioned, Tuskar needs a solution for this requirement,
>>since
>> >>> it's deploying using templates.  For the images, we have the concept
>> >>> of a "golden" image in TripleO and are heavily focused on image
>>based
>> >>> deployments.  Therefore, it seems to make sense that TripleO also
>> >>> needs a way to version/tag known good images.
>> >>> 
>> >>> Given that, I think it makes sense  to do this in a way so that it's
>> >>> consumable for things other than just templates.  In fact, you can
>> >>> almost s/template/image/g on the Heater wiki page, and it pretty
>>well
>> >>> lays out what I'd like to see for images as well.
>> >>> 
>>  1. Glance is for big blobs of data not tiny templates.
>>  2. Versioning of a single resource is desired.
>>  3. Tagging/classifying/listing/sorting
>>  4. Glance is designed to expose the uploaded blobs to nova, not
>>users
>>  
>>  My responses:
>>  
>>  1: Irrelevant. Smaller things will fit in it just fine.
>>  
>>  2: The swift API supports versions. We could also have git as a
>>  backend.
>> >>> 
>> >>> I would definitely like to see a git backend for versioning.  No
>> >>> reason to reimplement a different solution for what already works
>> >>> well.  I'm not sure we'd want to put a whole image into git though.
>> >>> Perhaps just it's manifest (installed components, software versions,
>> >>> etc) in json format would go into git, and that would be associated
>> >>> back to the binary image via uuid.  That would even make it easy to
>> >>> diff changes between versions, etc.
>> >>> 
>> >> 
>> >> Right, git for a big 'ol image makes little sense.
>> >> 
>> >> I'm suggesting that one might want to have two glances, one for
>>images
>> >> which just uses swift versions and would just expose a list of
>>versions,
>> >> and one for templates which would use git and thus expose more
>>features
>> >> like a git remote for the repo. I'm not sure if glance has embraced
>>the
>> >> extension paradigm yet, but this would fall nicely into it.
>> >
>> >Alternatively, Glance could have configurable backends for each image
>> >type allowing for optimization without the (often times messy)
>>extension
>> >mechanism? This is assuming it doesn't do this already - I really need
>>to
>> >start digging here. In the spirit of general OpenStack architectural
>> >paradigms, as long as the service exposes a consistent interface for
>> >templates that includes versioning support, the back-end store and
>> >(possibly) the versioning "engine" should certainly be configurable.
>> >Swift probably makes a decent first/default implementation.
>> 
>> I'm not sure why we are attempting to fit a round peg in a square hole
>> here. Glance's entire design is built around serving images and there
>> absolutely is a difference between serving BLOBs versus relatively small
>> templates. Just look at the existing implementations, Heat already
>>stores
>> the template directly in the database while Glance stores the blob in an
>> external file system.
>> 
>
>What aspects of circles and squares do you find match glance and the
>Heater problem space? :-P
>
>If this were as simple as geometric shape matching, my 4-year old could
>do it, right? :) Let's use analogies, I _love_ analogies, but perhaps
>lets be more careful about whether they actually aid the discussion.
>
>The blobs in the heat database are going to be a problem actually. I've
>dealt with a very similar relatively large system, storing millions
>of resumes in a single table for a job candidate database. It performs
>horribly even if you shard. These weren't 50MB resumes, these were 30 -
>50 kB resumes. That is because every time you need to pull all the rows
>you have to pull giant amounts of data and generally just blow out all
>of the caches, buffers, etc. Keystone's token table for the sql token
>backend has the same problem. RDBMS's are good at storing and retrieving
>rows of relatively predictably sized data. They suck for documents.
>
>Heat would do well to just store template references and fetch them in
>the rare cases where the raw templates are need

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Vishvananda Ishaya]

On Dec 5, 2013, at 12:42 PM, Andrew Plunk  wrote:

>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>>> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>>> wrote:
>>> 
 Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> Why not just use glance?
> 
 
 I've asked that question a few times, and I think I can collate the
 responses I've received below. I think enhancing glance to do these
 things is on the table:
 
 1. Glance is for big blobs of data not tiny templates.
 2. Versioning of a single resource is desired.
 3. Tagging/classifying/listing/sorting
 4. Glance is designed to expose the uploaded blobs to nova, not users
 
 My responses:
 
 1: Irrelevant. Smaller things will fit in it just fine.
>>> 
>>> Fitting is one thing, optimizations around particular assumptions about the 
>>> size of data and the frequency of reads/writes might be an issue, but I 
>>> admit to ignorance about those details in Glance.
>>> 
>> 
>> Optimizations can be improved for various use cases. The design, however,
>> has no assumptions that I know about that would invalidate storing blobs
>> of yaml/json vs. blobs of kernel/qcow2/raw image.
> 
> I think we are getting out into the weeds a little bit here. It is important 
> to think about these apis in terms of what they actually do, before the 
> decision of combining them or not can be made.
> 
> I think of HeatR as a template storage service, it provides extra data and 
> operations on templates. HeatR should not care about how those templates are 
> stored.
> Glance is an image storage service, it provides extra data and operations on 
> images (not blobs), and it happens to use swift as a backend.

This is not completely correct. Glance already supports something akin to 
templates. You can create an "image" with metadata properties that specifies a 
complex block device mapping which would allow for multiple volumes and images 
to connected to the vm at boot time. This is functionally a template for a 
single vm.

Glance is pretty useless if is just an "image storage" service, we already have 
other places that can store bits (swift, cinder). It is much more valuable as a 
searchable repository of bootable templates. I don't see any reason why this 
idea couldn't be extended to include more complex templates that could include 
more than one vm.

We have discussed the future of glance a number of times, and if it is really 
just there to serve up blobs of data + metadata about images, it should go 
away. Glance should offer something more like the AWS image search console. And 
this could clearly support more than just images, you should be able to search 
for and launch more complicated templates as well.

> 
> If HeatR and Glance were combined, it would result in taking two very 
> different types of data (template metadata vs image metadata) and mashing 
> them into one service. How would adding the complexity of HeatR benefit 
> Glance, when they are dealing with conceptually two very different types of 
> data? For instance, should a template ever care about the field "minRam" that 
> is stored with an image?


I don't see these as significantly different types of metadata. Metadata for 
heat templates might be a bit more broad (minFlavor?) I would think that a 
template would care about constraints like this, especially when you consider 
that a user might want to give a command to launch a template but then override 
certain characteristics.

Vish

> Combining them adds a huge development complexity with a very small 
> operations payoff, and so Openstack is already so operationally complex that 
> HeatR as a separate service would be knowledgeable. Only clients of Heat will 
> ever care about data and operations on templates, so I move that HeatR 
> becomes it's own service, or becomes part of Heat.
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Tim Bell]

Completely agree with Brad... a new project for this is not what is needed.

>From an operator's point of view, it is a REAL, REAL, REAL pain to be 
>configuring yet another project, yet another set of Puppet/Chef recipes, 
>additional monitoring, service nodes, new databases, more documentation, 
>further packages, pre-requisites and tough divergent roles follow...

While I appreciate that there is an impact from having reviews, this should not 
be the cause for creating more complexity for those who wish to use the 
functionality.

I'd also suggest the Heater guys talk to the Murano guys since my ideal 
scenario is for a simple user kiosk where I can ask for an app, be asked for 
some configuration details and then deploy. The two projects appear to be 
heading towards a converged solution.

Tim

From: Brad Topol [mailto:bto...@us.ibm.com]
Sent: 05 December 2013 20:06
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal

Lots of good discussion on this topic.   One thing I would like to point out is 
that we get feedback that OpenStack has too many projects as it is and 
customers get confused on how much of OpenStack they need to install.  So in 
the spirit of trying to help insure OpenStack does not continue to reinforce 
this perception, I am hoping that Heater functionality finds a home in either 
Glance or Heat.  I don't have a preference of which.   Either of these is 
superior to starting a new project if it can be avoided.

Thanks,

Brad

Brad Topol, Ph.D.
IBM Distinguished Engineer
OpenStack
(919) 543-0646
Internet:  bto...@us.ibm.com<mailto:bto...@us.ibm.com>
Assistant: Kendra Witherspoon (919) 254-0680



From:Randall Burt 
mailto:randall.b...@rackspace.com>>
To:"OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date:        12/05/2013 12:09 PM
Subject:Re: [openstack-dev] [heat] [glance] Heater Proposal




On Dec 5, 2013, at 10:10 AM, Clint Byrum 
mailto:cl...@fewbar.com>>
wrote:

> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> Why not just use glance?
>>
>
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:
>
> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
>
> My responses:
>
> 1: Irrelevant. Smaller things will fit in it just fine.

Fitting is one thing, optimizations around particular assumptions about the 
size of data and the frequency of reads/writes might be an issue, but I admit 
to ignorance about those details in Glance.

> 2: The swift API supports versions. We could also have git as a
> backend. This feels like something we can add as an optional feature
> without exploding Glance's scope and I imagine it would actually be a
> welcome feature for image authors as well. Think about Ubuntu maintaining
> official images. If they can keep the ID the same and just add a version
> (allowing users to lock down to a version if updated images cause issue)
> that seems like a really cool feature for images _and_ templates.

Agreed, though one could argue that using image names and looking up ID's or 
just using ID's as appropriate "sort of" handle this use case, but I agree that 
having image versioning seems a reasonable feature for Glance to have as well.

> 3: I'm sure glance image users would love to have those too.

And image metadata is already there so we don't have to go through those 
discussions all over again ;).

> 4: Irrelevant. Heat will need to download templates just like nova, and
> making images publicly downloadable is also a thing in glance.

Yeah, this was the kicker for me. I'd been thinking of adding the 
tenancy/public/private templates use case to the HeatR spec and realized that 
this was a good argument for Glance since it already has this feature.

> It strikes me that this might be a silo problem instead of an
> actual design problem. Folk should not be worried about jumping into
> Glance and adding features. Unless of course the Glance folk have
> reservations? (adding glance tag to the subject)

Perhaps, and if these use cases make sense for the Glance users in general, I 
wouldn't want to re-invent all those wheels either. I admit there's some appeal 
to being able to pass a template ID to stack-create or as the type of a 
provider resource and have an actual API to call that's already got a known, 
tested client that's already part

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Brad Topol]

Lots of good discussion on this topic.   One thing I would like to point 
out is that we get feedback that OpenStack has too many projects as it is 
and customers get confused on how much of OpenStack they need to install. 
So in the spirit of trying to help insure OpenStack does not continue to 
reinforce this perception, I am hoping that Heater functionality finds a 
home in either Glance or Heat.  I don't have a preference of which. Either 
of these is superior to starting a new project if it can be avoided.

Thanks,

Brad

Brad Topol, Ph.D.
IBM Distinguished Engineer
OpenStack
(919) 543-0646
Internet:  bto...@us.ibm.com
Assistant: Kendra Witherspoon (919) 254-0680



From:   Randall Burt 
To: "OpenStack Development Mailing List (not for usage questions)" 

Date:   12/05/2013 12:09 PM
Subject:    Re: [openstack-dev] [heat] [glance] Heater Proposal



On Dec 5, 2013, at 10:10 AM, Clint Byrum 
 wrote:

> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> Why not just use glance?
>> 
> 
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:
> 
> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
> 
> My responses:
> 
> 1: Irrelevant. Smaller things will fit in it just fine.

Fitting is one thing, optimizations around particular assumptions about 
the size of data and the frequency of reads/writes might be an issue, but 
I admit to ignorance about those details in Glance.

> 2: The swift API supports versions. We could also have git as a
> backend. This feels like something we can add as an optional feature
> without exploding Glance's scope and I imagine it would actually be a
> welcome feature for image authors as well. Think about Ubuntu 
maintaining
> official images. If they can keep the ID the same and just add a version
> (allowing users to lock down to a version if updated images cause issue)
> that seems like a really cool feature for images _and_ templates.

Agreed, though one could argue that using image names and looking up ID's 
or just using ID's as appropriate "sort of" handle this use case, but I 
agree that having image versioning seems a reasonable feature for Glance 
to have as well.

> 3: I'm sure glance image users would love to have those too.

And image metadata is already there so we don't have to go through those 
discussions all over again ;).

> 4: Irrelevant. Heat will need to download templates just like nova, and
> making images publicly downloadable is also a thing in glance.

Yeah, this was the kicker for me. I'd been thinking of adding the 
tenancy/public/private templates use case to the HeatR spec and realized 
that this was a good argument for Glance since it already has this 
feature.

> It strikes me that this might be a silo problem instead of an
> actual design problem. Folk should not be worried about jumping into
> Glance and adding features. Unless of course the Glance folk have
> reservations? (adding glance tag to the subject)

Perhaps, and if these use cases make sense for the Glance users in 
general, I wouldn't want to re-invent all those wheels either. I admit 
there's some appeal to being able to pass a template ID to stack-create or 
as the type of a provider resource and have an actual API to call that's 
already got a known, tested client that's already part of the OpenStack 
ecosystem

In the end, though, even if some and not all of our use cases make sense 
for the Glance folks, we still have the option of creating the HeatR 
service and having Glance as a possible back-end data store.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
> On Dec 5, 2013, at 10:10 AM, Clint Byrum 
>  wrote:
> 
> > Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> >> Why not just use glance?
> >> 
> > 
> > I've asked that question a few times, and I think I can collate the
> > responses I've received below. I think enhancing glance to do these
> > things is on the table:
> > 
> > 1. Glance is for big blobs of data not tiny templates.
> > 2. Versioning of a single resource is desired.
> > 3. Tagging/classifying/listing/sorting
> > 4. Glance is designed to expose the uploaded blobs to nova, not users
> > 
> > My responses:
> > 
> > 1: Irrelevant. Smaller things will fit in it just fine.
> 
> Fitting is one thing, optimizations around particular assumptions about the 
> size of data and the frequency of reads/writes might be an issue, but I admit 
> to ignorance about those details in Glance.
> 

Optimizations can be improved for various use cases. The design, however,
has no assumptions that I know about that would invalidate storing blobs
of yaml/json vs. blobs of kernel/qcow2/raw image.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Tim Schnell's message of 2013-12-05 09:49:03 -0800:
> 
> On 12/5/13 11:33 AM, "Randall Burt"  wrote:
> 
> >On Dec 5, 2013, at 11:10 AM, Clint Byrum 
> > wrote:
> >
> >> Excerpts from James Slagle's message of 2013-12-05 08:35:12 -0800:
> >>> On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum  wrote:
>  Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> > Why not just use glance?
> > 
>  
>  I've asked that question a few times, and I think I can collate the
>  responses I've received below. I think enhancing glance to do these
>  things is on the table:
> >>> 
> >>> I'm actually interested in the use cases laid out by Heater from both
> >>> a template perspective and image perspective.  For the templates, as
> >>> Robert mentioned, Tuskar needs a solution for this requirement, since
> >>> it's deploying using templates.  For the images, we have the concept
> >>> of a "golden" image in TripleO and are heavily focused on image based
> >>> deployments.  Therefore, it seems to make sense that TripleO also
> >>> needs a way to version/tag known good images.
> >>> 
> >>> Given that, I think it makes sense  to do this in a way so that it's
> >>> consumable for things other than just templates.  In fact, you can
> >>> almost s/template/image/g on the Heater wiki page, and it pretty well
> >>> lays out what I'd like to see for images as well.
> >>> 
>  1. Glance is for big blobs of data not tiny templates.
>  2. Versioning of a single resource is desired.
>  3. Tagging/classifying/listing/sorting
>  4. Glance is designed to expose the uploaded blobs to nova, not users
>  
>  My responses:
>  
>  1: Irrelevant. Smaller things will fit in it just fine.
>  
>  2: The swift API supports versions. We could also have git as a
>  backend.
> >>> 
> >>> I would definitely like to see a git backend for versioning.  No
> >>> reason to reimplement a different solution for what already works
> >>> well.  I'm not sure we'd want to put a whole image into git though.
> >>> Perhaps just it's manifest (installed components, software versions,
> >>> etc) in json format would go into git, and that would be associated
> >>> back to the binary image via uuid.  That would even make it easy to
> >>> diff changes between versions, etc.
> >>> 
> >> 
> >> Right, git for a big 'ol image makes little sense.
> >> 
> >> I'm suggesting that one might want to have two glances, one for images
> >> which just uses swift versions and would just expose a list of versions,
> >> and one for templates which would use git and thus expose more features
> >> like a git remote for the repo. I'm not sure if glance has embraced the
> >> extension paradigm yet, but this would fall nicely into it.
> >
> >Alternatively, Glance could have configurable backends for each image
> >type allowing for optimization without the (often times messy) extension
> >mechanism? This is assuming it doesn't do this already - I really need to
> >start digging here. In the spirit of general OpenStack architectural
> >paradigms, as long as the service exposes a consistent interface for
> >templates that includes versioning support, the back-end store and
> >(possibly) the versioning "engine" should certainly be configurable.
> >Swift probably makes a decent first/default implementation.
> 
> I'm not sure why we are attempting to fit a round peg in a square hole
> here. Glance's entire design is built around serving images and there
> absolutely is a difference between serving BLOBs versus relatively small
> templates. Just look at the existing implementations, Heat already stores
> the template directly in the database while Glance stores the blob in an
> external file system.
> 

What aspects of circles and squares do you find match glance and the
Heater problem space? :-P

If this were as simple as geometric shape matching, my 4-year old could
do it, right? :) Let's use analogies, I _love_ analogies, but perhaps
lets be more careful about whether they actually aid the discussion.

The blobs in the heat database are going to be a problem actually. I've
dealt with a very similar relatively large system, storing millions
of resumes in a single table for a job candidate database. It performs
horribly even if you shard. These weren't 50MB resumes, these were 30 -
50 kB resumes. That is because every time you need to pull all the rows
you have to pull giant amounts of data and generally just blow out all
of the caches, buffers, etc. Keystone's token table for the sql token
backend has the same problem. RDBMS's are good at storing and retrieving
rows of relatively predictably sized data. They suck for documents.

Heat would do well to just store template references and fetch them in
the rare cases where the raw templates are needed (updates, user asks to
see it, etc). That could so easily be a glance reference.  And if Glance
was suddenly much smarter and more capable of listing/searching/etc

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Fox, Kevin M]

My 2 cents. Glance currently deals with single file images.

A user is going to want the heat repo to operate at a "stack" level. IE, I want 
to launch stack "foo".

For all but the most trivial cases, a stack is made up of more then one 
template. These templates should be versioned as set (stack), not separately.

So, glance would probably need to change to support "images" with multiple 
files. This could be done, but may be a bigger architectural change then 
originally considered.

Thanks,
Kevin

From: Clint Byrum [cl...@fewbar.com]
Sent: Thursday, December 05, 2013 8:10 AM
To: openstack-dev
Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal

Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> Why not just use glance?
>

I've asked that question a few times, and I think I can collate the
responses I've received below. I think enhancing glance to do these
things is on the table:

1. Glance is for big blobs of data not tiny templates.
2. Versioning of a single resource is desired.
3. Tagging/classifying/listing/sorting
4. Glance is designed to expose the uploaded blobs to nova, not users

My responses:

1: Irrelevant. Smaller things will fit in it just fine.

2: The swift API supports versions. We could also have git as a
backend. This feels like something we can add as an optional feature
without exploding Glance's scope and I imagine it would actually be a
welcome feature for image authors as well. Think about Ubuntu maintaining
official images. If they can keep the ID the same and just add a version
(allowing users to lock down to a version if updated images cause issue)
that seems like a really cool feature for images _and_ templates.

3: I'm sure glance image users would love to have those too.

4: Irrelevant. Heat will need to download templates just like nova, and
making images publicly downloadable is also a thing in glance.

It strikes me that this might be a silo problem instead of an
actual design problem. Folk should not be worried about jumping into
Glance and adding features. Unless of course the Glance folk have
reservations? (adding glance tag to the subject)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Tim Schnell]

On 12/5/13 11:33 AM, "Randall Burt"  wrote:

>On Dec 5, 2013, at 11:10 AM, Clint Byrum 
> wrote:
>
>> Excerpts from James Slagle's message of 2013-12-05 08:35:12 -0800:
>>> On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum  wrote:
 Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> Why not just use glance?
> 
 
 I've asked that question a few times, and I think I can collate the
 responses I've received below. I think enhancing glance to do these
 things is on the table:
>>> 
>>> I'm actually interested in the use cases laid out by Heater from both
>>> a template perspective and image perspective.  For the templates, as
>>> Robert mentioned, Tuskar needs a solution for this requirement, since
>>> it's deploying using templates.  For the images, we have the concept
>>> of a "golden" image in TripleO and are heavily focused on image based
>>> deployments.  Therefore, it seems to make sense that TripleO also
>>> needs a way to version/tag known good images.
>>> 
>>> Given that, I think it makes sense  to do this in a way so that it's
>>> consumable for things other than just templates.  In fact, you can
>>> almost s/template/image/g on the Heater wiki page, and it pretty well
>>> lays out what I'd like to see for images as well.
>>> 
 1. Glance is for big blobs of data not tiny templates.
 2. Versioning of a single resource is desired.
 3. Tagging/classifying/listing/sorting
 4. Glance is designed to expose the uploaded blobs to nova, not users
 
 My responses:
 
 1: Irrelevant. Smaller things will fit in it just fine.
 
 2: The swift API supports versions. We could also have git as a
 backend.
>>> 
>>> I would definitely like to see a git backend for versioning.  No
>>> reason to reimplement a different solution for what already works
>>> well.  I'm not sure we'd want to put a whole image into git though.
>>> Perhaps just it's manifest (installed components, software versions,
>>> etc) in json format would go into git, and that would be associated
>>> back to the binary image via uuid.  That would even make it easy to
>>> diff changes between versions, etc.
>>> 
>> 
>> Right, git for a big 'ol image makes little sense.
>> 
>> I'm suggesting that one might want to have two glances, one for images
>> which just uses swift versions and would just expose a list of versions,
>> and one for templates which would use git and thus expose more features
>> like a git remote for the repo. I'm not sure if glance has embraced the
>> extension paradigm yet, but this would fall nicely into it.
>
>Alternatively, Glance could have configurable backends for each image
>type allowing for optimization without the (often times messy) extension
>mechanism? This is assuming it doesn't do this already - I really need to
>start digging here. In the spirit of general OpenStack architectural
>paradigms, as long as the service exposes a consistent interface for
>templates that includes versioning support, the back-end store and
>(possibly) the versioning "engine" should certainly be configurable.
>Swift probably makes a decent first/default implementation.

I'm not sure why we are attempting to fit a round peg in a square hole
here. Glance's entire design is built around serving images and there
absolutely is a difference between serving BLOBs versus relatively small
templates. Just look at the existing implementations, Heat already stores
the template directly in the database while Glance stores the blob in an
external file system.

I'm not opposed to having Glance as an optional configurable backend for
Heater but if you look at the existing database models for Glance, an
entirely separate schema would have to be created to support templates. I
also think that having Heater as a layer above Glance or whatever other
backend would allow for the flexibility for Heater's requirements to
diverge in the future from requirements that might make sense for images
but not templates.

-Tim

>
 This feels like something we can add as an optional feature
 without exploding Glance's scope and I imagine it would actually be a
 welcome feature for image authors as well. Think about Ubuntu
maintaining
 official images. If they can keep the ID the same and just add a
version
 (allowing users to lock down to a version if updated images cause
issue)
 that seems like a really cool feature for images _and_ templates.
 
 3: I'm sure glance image users would love to have those too.
 
 4: Irrelevant. Heat will need to download templates just like nova,
and
 making images publicly downloadable is also a thing in glance.
 
 It strikes me that this might be a silo problem instead of an
 actual design problem. Folk should not be worried about jumping into
 Glance and adding features. Unless of course the Glance folk have
 reservations? (adding glance tag to the subject)
>>> 
>>> I'm +1 for adding th

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 5, 2013, at 11:10 AM, Clint Byrum 
 wrote:

> Excerpts from James Slagle's message of 2013-12-05 08:35:12 -0800:
>> On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum  wrote:
>>> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
 Why not just use glance?
 
>>> 
>>> I've asked that question a few times, and I think I can collate the
>>> responses I've received below. I think enhancing glance to do these
>>> things is on the table:
>> 
>> I'm actually interested in the use cases laid out by Heater from both
>> a template perspective and image perspective.  For the templates, as
>> Robert mentioned, Tuskar needs a solution for this requirement, since
>> it's deploying using templates.  For the images, we have the concept
>> of a "golden" image in TripleO and are heavily focused on image based
>> deployments.  Therefore, it seems to make sense that TripleO also
>> needs a way to version/tag known good images.
>> 
>> Given that, I think it makes sense  to do this in a way so that it's
>> consumable for things other than just templates.  In fact, you can
>> almost s/template/image/g on the Heater wiki page, and it pretty well
>> lays out what I'd like to see for images as well.
>> 
>>> 1. Glance is for big blobs of data not tiny templates.
>>> 2. Versioning of a single resource is desired.
>>> 3. Tagging/classifying/listing/sorting
>>> 4. Glance is designed to expose the uploaded blobs to nova, not users
>>> 
>>> My responses:
>>> 
>>> 1: Irrelevant. Smaller things will fit in it just fine.
>>> 
>>> 2: The swift API supports versions. We could also have git as a
>>> backend.
>> 
>> I would definitely like to see a git backend for versioning.  No
>> reason to reimplement a different solution for what already works
>> well.  I'm not sure we'd want to put a whole image into git though.
>> Perhaps just it's manifest (installed components, software versions,
>> etc) in json format would go into git, and that would be associated
>> back to the binary image via uuid.  That would even make it easy to
>> diff changes between versions, etc.
>> 
> 
> Right, git for a big 'ol image makes little sense.
> 
> I'm suggesting that one might want to have two glances, one for images
> which just uses swift versions and would just expose a list of versions,
> and one for templates which would use git and thus expose more features
> like a git remote for the repo. I'm not sure if glance has embraced the
> extension paradigm yet, but this would fall nicely into it.

Alternatively, Glance could have configurable backends for each image type 
allowing for optimization without the (often times messy) extension mechanism? 
This is assuming it doesn't do this already - I really need to start digging 
here. In the spirit of general OpenStack architectural paradigms, as long as 
the service exposes a consistent interface for templates that includes 
versioning support, the back-end store and (possibly) the versioning "engine" 
should certainly be configurable. Swift probably makes a decent first/default 
implementation.

>>> This feels like something we can add as an optional feature
>>> without exploding Glance's scope and I imagine it would actually be a
>>> welcome feature for image authors as well. Think about Ubuntu maintaining
>>> official images. If they can keep the ID the same and just add a version
>>> (allowing users to lock down to a version if updated images cause issue)
>>> that seems like a really cool feature for images _and_ templates.
>>> 
>>> 3: I'm sure glance image users would love to have those too.
>>> 
>>> 4: Irrelevant. Heat will need to download templates just like nova, and
>>> making images publicly downloadable is also a thing in glance.
>>> 
>>> It strikes me that this might be a silo problem instead of an
>>> actual design problem. Folk should not be worried about jumping into
>>> Glance and adding features. Unless of course the Glance folk have
>>> reservations? (adding glance tag to the subject)
>> 
>> I'm +1 for adding these types of features to glance, or at least
>> something common, instead of making it specific to Heat templates.
>> 
> 
> Right, it may be that glance is too limited, but from what I've seen,
> it is not and it already has the base concepts that "HeaTeR" wants to
> have available.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from James Slagle's message of 2013-12-05 08:35:12 -0800:
> On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum  wrote:
> > Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> >> Why not just use glance?
> >>
> >
> > I've asked that question a few times, and I think I can collate the
> > responses I've received below. I think enhancing glance to do these
> > things is on the table:
> 
> I'm actually interested in the use cases laid out by Heater from both
> a template perspective and image perspective.  For the templates, as
> Robert mentioned, Tuskar needs a solution for this requirement, since
> it's deploying using templates.  For the images, we have the concept
> of a "golden" image in TripleO and are heavily focused on image based
> deployments.  Therefore, it seems to make sense that TripleO also
> needs a way to version/tag known good images.
> 
> Given that, I think it makes sense  to do this in a way so that it's
> consumable for things other than just templates.  In fact, you can
> almost s/template/image/g on the Heater wiki page, and it pretty well
> lays out what I'd like to see for images as well.
> 
> > 1. Glance is for big blobs of data not tiny templates.
> > 2. Versioning of a single resource is desired.
> > 3. Tagging/classifying/listing/sorting
> > 4. Glance is designed to expose the uploaded blobs to nova, not users
> >
> > My responses:
> >
> > 1: Irrelevant. Smaller things will fit in it just fine.
> >
> > 2: The swift API supports versions. We could also have git as a
> > backend.
> 
> I would definitely like to see a git backend for versioning.  No
> reason to reimplement a different solution for what already works
> well.  I'm not sure we'd want to put a whole image into git though.
> Perhaps just it's manifest (installed components, software versions,
> etc) in json format would go into git, and that would be associated
> back to the binary image via uuid.  That would even make it easy to
> diff changes between versions, etc.
> 

Right, git for a big 'ol image makes little sense.

I'm suggesting that one might want to have two glances, one for images
which just uses swift versions and would just expose a list of versions,
and one for templates which would use git and thus expose more features
like a git remote for the repo. I'm not sure if glance has embraced the
extension paradigm yet, but this would fall nicely into it.

> > This feels like something we can add as an optional feature
> > without exploding Glance's scope and I imagine it would actually be a
> > welcome feature for image authors as well. Think about Ubuntu maintaining
> > official images. If they can keep the ID the same and just add a version
> > (allowing users to lock down to a version if updated images cause issue)
> > that seems like a really cool feature for images _and_ templates.
> >
> > 3: I'm sure glance image users would love to have those too.
> >
> > 4: Irrelevant. Heat will need to download templates just like nova, and
> > making images publicly downloadable is also a thing in glance.
> >
> > It strikes me that this might be a silo problem instead of an
> > actual design problem. Folk should not be worried about jumping into
> > Glance and adding features. Unless of course the Glance folk have
> > reservations? (adding glance tag to the subject)
> 
> I'm +1 for adding these types of features to glance, or at least
> something common, instead of making it specific to Heat templates.
> 

Right, it may be that glance is too limited, but from what I've seen,
it is not and it already has the base concepts that "HeaTeR" wants to
have available.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 5, 2013, at 10:10 AM, Clint Byrum 
 wrote:

> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> Why not just use glance?
>> 
> 
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:
> 
> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
> 
> My responses:
> 
> 1: Irrelevant. Smaller things will fit in it just fine.

Fitting is one thing, optimizations around particular assumptions about the 
size of data and the frequency of reads/writes might be an issue, but I admit 
to ignorance about those details in Glance.

> 2: The swift API supports versions. We could also have git as a
> backend. This feels like something we can add as an optional feature
> without exploding Glance's scope and I imagine it would actually be a
> welcome feature for image authors as well. Think about Ubuntu maintaining
> official images. If they can keep the ID the same and just add a version
> (allowing users to lock down to a version if updated images cause issue)
> that seems like a really cool feature for images _and_ templates.

Agreed, though one could argue that using image names and looking up ID's or 
just using ID's as appropriate "sort of" handle this use case, but I agree that 
having image versioning seems a reasonable feature for Glance to have as well.

> 3: I'm sure glance image users would love to have those too.

And image metadata is already there so we don't have to go through those 
discussions all over again ;).

> 4: Irrelevant. Heat will need to download templates just like nova, and
> making images publicly downloadable is also a thing in glance.

Yeah, this was the kicker for me. I'd been thinking of adding the 
tenancy/public/private templates use case to the HeatR spec and realized that 
this was a good argument for Glance since it already has this feature.

> It strikes me that this might be a silo problem instead of an
> actual design problem. Folk should not be worried about jumping into
> Glance and adding features. Unless of course the Glance folk have
> reservations? (adding glance tag to the subject)

Perhaps, and if these use cases make sense for the Glance users in general, I 
wouldn't want to re-invent all those wheels either. I admit there's some appeal 
to being able to pass a template ID to stack-create or as the type of a 
provider resource and have an actual API to call that's already got a known, 
tested client that's already part of the OpenStack ecosystem

In the end, though, even if some and not all of our use cases make sense for 
the Glance folks, we still have the option of creating the HeatR service and 
having Glance as a possible back-end data store.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[James Slagle]

On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum  wrote:
> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> Why not just use glance?
>>
>
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:

I'm actually interested in the use cases laid out by Heater from both
a template perspective and image perspective.  For the templates, as
Robert mentioned, Tuskar needs a solution for this requirement, since
it's deploying using templates.  For the images, we have the concept
of a "golden" image in TripleO and are heavily focused on image based
deployments.  Therefore, it seems to make sense that TripleO also
needs a way to version/tag known good images.

Given that, I think it makes sense  to do this in a way so that it's
consumable for things other than just templates.  In fact, you can
almost s/template/image/g on the Heater wiki page, and it pretty well
lays out what I'd like to see for images as well.

> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
>
> My responses:
>
> 1: Irrelevant. Smaller things will fit in it just fine.
>
> 2: The swift API supports versions. We could also have git as a
> backend.

I would definitely like to see a git backend for versioning.  No
reason to reimplement a different solution for what already works
well.  I'm not sure we'd want to put a whole image into git though.
Perhaps just it's manifest (installed components, software versions,
etc) in json format would go into git, and that would be associated
back to the binary image via uuid.  That would even make it easy to
diff changes between versions, etc.

> This feels like something we can add as an optional feature
> without exploding Glance's scope and I imagine it would actually be a
> welcome feature for image authors as well. Think about Ubuntu maintaining
> official images. If they can keep the ID the same and just add a version
> (allowing users to lock down to a version if updated images cause issue)
> that seems like a really cool feature for images _and_ templates.
>
> 3: I'm sure glance image users would love to have those too.
>
> 4: Irrelevant. Heat will need to download templates just like nova, and
> making images publicly downloadable is also a thing in glance.
>
> It strikes me that this might be a silo problem instead of an
> actual design problem. Folk should not be worried about jumping into
> Glance and adding features. Unless of course the Glance folk have
> reservations? (adding glance tag to the subject)

I'm +1 for adding these types of features to glance, or at least
something common, instead of making it specific to Heat templates.


-- 
-- James Slagle
--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Clint Byrum]

Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> Why not just use glance?
> 

I've asked that question a few times, and I think I can collate the
responses I've received below. I think enhancing glance to do these
things is on the table:

1. Glance is for big blobs of data not tiny templates.
2. Versioning of a single resource is desired.
3. Tagging/classifying/listing/sorting
4. Glance is designed to expose the uploaded blobs to nova, not users

My responses:

1: Irrelevant. Smaller things will fit in it just fine.

2: The swift API supports versions. We could also have git as a
backend. This feels like something we can add as an optional feature
without exploding Glance's scope and I imagine it would actually be a
welcome feature for image authors as well. Think about Ubuntu maintaining
official images. If they can keep the ID the same and just add a version
(allowing users to lock down to a version if updated images cause issue)
that seems like a really cool feature for images _and_ templates.

3: I'm sure glance image users would love to have those too.

4: Irrelevant. Heat will need to download templates just like nova, and
making images publicly downloadable is also a thing in glance.

It strikes me that this might be a silo problem instead of an
actual design problem. Folk should not be worried about jumping into
Glance and adding features. Unless of course the Glance folk have
reservations? (adding glance tag to the subject)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev