Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
On 31/10/15 18:41 +, Ian Cordasco wrote: Does that mean that https://review.openstack.org/177397 will be updated to fully address the API-WG and Defcore feedback? How soon can we expect the spec to reflect that feedback and the code will be updated for it? Yes, All the updates will be posted there and the feedback will be addressed. I don't think any of this will happen in the next couple of weeks so, I'd guess end of month. I'll let the folks from the Artifacts team to deny/confirm the ETA above. Flavio On 10/30/15, 18:54, "Alexander Tivelkov" wrote: Hi folks, Thanks for the fruitful discussion on the summit: seems like we are in agreement and have a good plan to proceed. Here is the video recording of the PoC I've made to serve the current AppCatalog's data using Glance Artifacts. As the demo was recorded before the summit, it's called Glance V3 everywhere. Now we've agreed that this will be a Glance Artifacts Repository (aka Glare) API v1 - but everything else about its functionality is still valid :) https://youtu.be/5IxKqJiD2xw Please feel free to ask any questions you may have on the demo and Glare functionality in general. Thanks again On Wed, Oct 28, 2015 at 11:38 AM Flavio Percoco wrote: On 23/10/15 19:25 +, Fox, Kevin M wrote: The "Glance: Artefacts API" session was scheduled right on top of the "Nova: Cross Service Issues: Service Lock Server, Service Tokens, Instance Users (TBC) " session. The latter having been really hard to schedule since it involves so many different projects and something I must attend. So I won't be able to attend the glance session. The Murano folks have kindly offered to use one of their sessions: Murano contributors meetup (9:00am-12:30pm) http://mitakadesignsummit.sched.org/event/5e244fb7f342854dc5c112e76e77c93 0 to discuss Glance Artefacts/Murano integration/Community App Catalog needs. Can folks from the glance team that are interested in the discussion please attend? Kevin, thanks for the heads up! I'll help spreading the word in case some folks missed this email. Flavio Thanks, Kevin ━ ━━ From: Alexander Tivelkov [ativel...@mirantis.com] Sent: Thursday, October 15, 2015 3:04 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog Hi folks, I’ve noticed that the Community Application Catalog has begun to implement its own API, and it seems to me that we are going to have some significant duplication of efforts with the code which has already been implemented in Glance as Artifact Repository initiative (also known as Glance V3). From the very beginning of the App Catalog project (and I’ve been involved in it since February) I’ve been proposing to use Glance as its backend, because from my point of view it covers like 90% of the needed functionality. But it looks like we have some kind of miscommunication here, as I am getting some confusing questions and assumptions, like the vision of Glance V3 being dedicated backend for Murano (which is definitely incorrect). So, I am writing the email to clarify my vision of what Glance V3 is and how its features may be used to provide the REST API for Community App Catalog. 1. Versioned schema First of all, Glance V3 operates on entities called “artifacts”, and these ones perfectly map to the Data Assets of community app catalog. The artifacts are strongly typed: there are artifact types for murano packages, glance images, heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin, defining the schema of objects’ data and metadata and - optionally - custom logic. So, this thing is extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly. 2. Generic properties Next, all the artifact types in Glance V3 have some generic metadata properties (i.e. part of the schema which is common for all the types), including the name, the version, description, authorship information and so on. This also corresponds to the data schema of community app catalog. The mapping is not 1:1, but we can work together on this to make sure that these generic properties match the expectations of the catalog. 3. Versioning Versions are very important for catalogs of objects, so Glance V3 was initially designed keeping versioning questions in mind: each artifact has a semver-based version assigned, so the artifacts having the same name but different versions are grouped into the proper sequences. API is able to query artifacts based on their version spec, e.g. it is possible to fetch the latest artifact with the name “
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
Does that mean that https://review.openstack.org/177397 will be updated to fully address the API-WG and Defcore feedback? How soon can we expect the spec to reflect that feedback and the code will be updated for it? On 10/30/15, 18:54, "Alexander Tivelkov" wrote: >Hi folks, > > >Thanks for the fruitful discussion on the summit: seems like we are in >agreement and have a good plan to proceed. > > >Here is the video recording of the PoC I've made to serve the current >AppCatalog's data using Glance Artifacts. As the demo was recorded before >the summit, it's called Glance V3 everywhere. Now we've agreed that this >will be a Glance Artifacts Repository > (aka Glare) API v1 - but everything else about its functionality is >still valid :) > > >https://youtu.be/5IxKqJiD2xw > > > >Please feel free to ask any questions you may have on the demo and Glare >functionality in general. > > >Thanks again > > > > >On Wed, Oct 28, 2015 at 11:38 AM Flavio Percoco wrote: > > >On 23/10/15 19:25 +, Fox, Kevin M wrote: >>The "Glance: Artefacts API" session was scheduled right on top of the >>"Nova: >>Cross Service Issues: Service Lock Server, Service Tokens, Instance >>Users (TBC) >>" session. The latter having been really hard to schedule since it >>involves so >>many different projects and something I must attend. So I won't be able >>to >>attend the glance session. >> >>The Murano folks have kindly offered to use one of their sessions: >>Murano contributors meetup (9:00am-12:30pm) >>http://mitakadesignsummit.sched.org/event/5e244fb7f342854dc5c112e76e77c93 >>0 >>to discuss Glance Artefacts/Murano integration/Community App Catalog >>needs. >> >>Can folks from the glance team that are interested in the discussion >>please >>attend? > >Kevin, thanks for the heads up! I'll help spreading the word in case >some folks missed this email. > >Flavio > >> >>Thanks, >>Kevin >> >>━━━━━━━━━━━━━ >>━━ >>From: Alexander Tivelkov [ativel...@mirantis.com] >>Sent: Thursday, October 15, 2015 3:04 AM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API >>for >>App Catalog >> >>Hi folks, >> >>I’ve noticed that the Community Application Catalog has begun to >>implement its >>own API, and it seems to me that we are going to have some significant >>duplication of efforts with the code which has already been implemented >>in >>Glance as Artifact Repository initiative (also known as Glance V3). >>From the very beginning of the App Catalog project (and I’ve been >>involved in >>it since February) I’ve been proposing to use Glance as its backend, >>because >>from my point of view it covers like 90% of the needed functionality. >>But it >>looks like we have some kind of miscommunication here, as I am getting >>some >>confusing questions and assumptions, like the vision of Glance V3 being >>dedicated backend for Murano (which is definitely incorrect). >>So, I am writing the email to clarify my vision of what Glance V3 is and >>how >>its features may be used to provide the REST API for Community App >>Catalog. >> >>1. Versioned schema >>First of all, Glance V3 operates on entities called “artifacts”, and >>these ones >>perfectly map to the Data Assets of community app catalog. The artifacts >>are >>strongly typed: there are artifact types for murano packages, glance >>images, >>heat templates - and there may be (and will be) more. Each artifact type >>is >>represented by a plugin, defining the schema of objects’ data and >>metadata and >>- optionally - custom logic. So, this thing is extensible: when a new >>type of >>asset needs to be added to a catalog it can be done really quickly by >>just >>defining the schema and putting that schema into a plugin. Also, these >>plugins >>are versioned, so the possible changes in the schema are handled >>properly. >> >>2. Generic properties >>Next, all the artifact types in Glance V3 have some generic metadata >>properties >>(i.e. part of the schema which is common for all the types), including >>the >>name, the version, description, authorship information and so on. This >>also >>corresponds to the data schema of community app catalog. The mapping is >>not >>1:1, but we can work
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
Hi folks, Thanks for the fruitful discussion on the summit: seems like we are in agreement and have a good plan to proceed. Here is the video recording of the PoC I've made to serve the current AppCatalog's data using Glance Artifacts. As the demo was recorded before the summit, it's called Glance V3 everywhere. Now we've agreed that this will be a Glance Artifacts Repository (aka Glare) API v1 - but everything else about its functionality is still valid :) https://youtu.be/5IxKqJiD2xw Please feel free to ask any questions you may have on the demo and Glare functionality in general. Thanks again On Wed, Oct 28, 2015 at 11:38 AM Flavio Percoco wrote: > On 23/10/15 19:25 +, Fox, Kevin M wrote: > >The "Glance: Artefacts API" session was scheduled right on top of the > "Nova: > >Cross Service Issues: Service Lock Server, Service Tokens, Instance Users > (TBC) > >" session. The latter having been really hard to schedule since it > involves so > >many different projects and something I must attend. So I won't be able to > >attend the glance session. > > > >The Murano folks have kindly offered to use one of their sessions: > >Murano contributors meetup (9:00am-12:30pm) > > > http://mitakadesignsummit.sched.org/event/5e244fb7f342854dc5c112e76e77c930 > >to discuss Glance Artefacts/Murano integration/Community App Catalog > needs. > > > >Can folks from the glance team that are interested in the discussion > please > >attend? > > Kevin, thanks for the heads up! I'll help spreading the word in case > some folks missed this email. > > Flavio > > > > >Thanks, > >Kevin > > > > >━━━━━━━━━━━━━━━ > >From: Alexander Tivelkov [ativel...@mirantis.com] > >Sent: Thursday, October 15, 2015 3:04 AM > >To: OpenStack Development Mailing List (not for usage questions) > >Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API > for > >App Catalog > > > >Hi folks, > > > >I’ve noticed that the Community Application Catalog has begun to > implement its > >own API, and it seems to me that we are going to have some significant > >duplication of efforts with the code which has already been implemented in > >Glance as Artifact Repository initiative (also known as Glance V3). > >From the very beginning of the App Catalog project (and I’ve been > involved in > >it since February) I’ve been proposing to use Glance as its backend, > because > >from my point of view it covers like 90% of the needed functionality. But > it > >looks like we have some kind of miscommunication here, as I am getting > some > >confusing questions and assumptions, like the vision of Glance V3 being > >dedicated backend for Murano (which is definitely incorrect). > >So, I am writing the email to clarify my vision of what Glance V3 is and > how > >its features may be used to provide the REST API for Community App > Catalog. > > > >1. Versioned schema > >First of all, Glance V3 operates on entities called “artifacts”, and > these ones > >perfectly map to the Data Assets of community app catalog. The artifacts > are > >strongly typed: there are artifact types for murano packages, glance > images, > >heat templates - and there may be (and will be) more. Each artifact type > is > >represented by a plugin, defining the schema of objects’ data and > metadata and > >- optionally - custom logic. So, this thing is extensible: when a new > type of > >asset needs to be added to a catalog it can be done really quickly by just > >defining the schema and putting that schema into a plugin. Also, these > plugins > >are versioned, so the possible changes in the schema are handled properly. > > > >2. Generic properties > >Next, all the artifact types in Glance V3 have some generic metadata > properties > >(i.e. part of the schema which is common for all the types), including the > >name, the version, description, authorship information and so on. This > also > >corresponds to the data schema of community app catalog. The mapping is > not > >1:1, but we can work together on this to make sure that these generic > >properties match the expectations of the catalog. > > > >3. Versioning > >Versions are very important for catalogs of objects, so Glance V3 was > initially > >designed keeping versioning questions in mind: each artifact has a > semver-based > >version assigned, so the artifacts having the same name but different > versions > >are grouped into the proper seque
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
On 23/10/15 19:25 +, Fox, Kevin M wrote: The "Glance: Artefacts API" session was scheduled right on top of the "Nova: Cross Service Issues: Service Lock Server, Service Tokens, Instance Users (TBC) " session. The latter having been really hard to schedule since it involves so many different projects and something I must attend. So I won't be able to attend the glance session. The Murano folks have kindly offered to use one of their sessions: Murano contributors meetup (9:00am-12:30pm) http://mitakadesignsummit.sched.org/event/5e244fb7f342854dc5c112e76e77c930 to discuss Glance Artefacts/Murano integration/Community App Catalog needs. Can folks from the glance team that are interested in the discussion please attend? Kevin, thanks for the heads up! I'll help spreading the word in case some folks missed this email. Flavio Thanks, Kevin ━━━ From: Alexander Tivelkov [ativel...@mirantis.com] Sent: Thursday, October 15, 2015 3:04 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog Hi folks, I’ve noticed that the Community Application Catalog has begun to implement its own API, and it seems to me that we are going to have some significant duplication of efforts with the code which has already been implemented in Glance as Artifact Repository initiative (also known as Glance V3). From the very beginning of the App Catalog project (and I’ve been involved in it since February) I’ve been proposing to use Glance as its backend, because from my point of view it covers like 90% of the needed functionality. But it looks like we have some kind of miscommunication here, as I am getting some confusing questions and assumptions, like the vision of Glance V3 being dedicated backend for Murano (which is definitely incorrect). So, I am writing the email to clarify my vision of what Glance V3 is and how its features may be used to provide the REST API for Community App Catalog. 1. Versioned schema First of all, Glance V3 operates on entities called “artifacts”, and these ones perfectly map to the Data Assets of community app catalog. The artifacts are strongly typed: there are artifact types for murano packages, glance images, heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin, defining the schema of objects’ data and metadata and - optionally - custom logic. So, this thing is extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly. 2. Generic properties Next, all the artifact types in Glance V3 have some generic metadata properties (i.e. part of the schema which is common for all the types), including the name, the version, description, authorship information and so on. This also corresponds to the data schema of community app catalog. The mapping is not 1:1, but we can work together on this to make sure that these generic properties match the expectations of the catalog. 3. Versioning Versions are very important for catalogs of objects, so Glance V3 was initially designed keeping versioning questions in mind: each artifact has a semver-based version assigned, so the artifacts having the same name but different versions are grouped into the proper sequences. API is able to query artifacts based on their version spec, e.g. it is possible to fetch the latest artifact with the name “foo” having the version greater than 2.1 and less than 3.5.7 - or any other version spec, similar to pip or any other similar tool. As far as I know, community app catalog does not have such capabilities right now - and I strongly believe that it is really a must have feature for a catalog to be successful. At least it is absolutely mandatory for Murano packages, which are the only “real apps” among the asset types right now. 4. Cross artifact dependencies Glance V3 also has the dependency relations from the very beginning, so they may be defined as part of artifact type schema. As a result, an artifact may “reference” any number of other artifacts with various semantic. For example, murano package may define a set of references to other murano packages and call it “requires” - and this will act similar to the requirements of a python package. Similar properties may be defined for heat templates and glance images - they may reference each other with various semantics. Of course, the definitions of such dependencies may be done internally inside the packages, so they may be resolved locally by the service which is going to use it, but letting the catalog know about them will allow us to do the import-export operations for any given artifacts and its dependencies automatic
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
The "Glance: Artefacts API" session was scheduled right on top of the "Nova: Cross Service Issues: Service Lock Server, Service Tokens, Instance Users (TBC)" session. The latter having been really hard to schedule since it involves so many different projects and something I must attend. So I won't be able to attend the glance session. The Murano folks have kindly offered to use one of their sessions: Murano contributors meetup (9:00am-12:30pm) http://mitakadesignsummit.sched.org/event/5e244fb7f342854dc5c112e76e77c930<http://mitakadesignsummit.sched.org/event/5e244fb7f342854dc5c112e76e77c930#.VipbDpQuHGc> to discuss Glance Artefacts/Murano integration/Community App Catalog needs. Can folks from the glance team that are interested in the discussion please attend? Thanks, Kevin From: Alexander Tivelkov [ativel...@mirantis.com] Sent: Thursday, October 15, 2015 3:04 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog Hi folks, I’ve noticed that the Community Application Catalog has begun to implement its own API, and it seems to me that we are going to have some significant duplication of efforts with the code which has already been implemented in Glance as Artifact Repository initiative (also known as Glance V3). >From the very beginning of the App Catalog project (and I’ve been involved in >it since February) I’ve been proposing to use Glance as its backend, because >from my point of view it covers like 90% of the needed functionality. But it >looks like we have some kind of miscommunication here, as I am getting some >confusing questions and assumptions, like the vision of Glance V3 being >dedicated backend for Murano (which is definitely incorrect). So, I am writing the email to clarify my vision of what Glance V3 is and how its features may be used to provide the REST API for Community App Catalog. 1. Versioned schema First of all, Glance V3 operates on entities called “artifacts”, and these ones perfectly map to the Data Assets of community app catalog. The artifacts are strongly typed: there are artifact types for murano packages, glance images, heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin, defining the schema of objects’ data and metadata and - optionally - custom logic. So, this thing is extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly. 2. Generic properties Next, all the artifact types in Glance V3 have some generic metadata properties (i.e. part of the schema which is common for all the types), including the name, the version, description, authorship information and so on. This also corresponds to the data schema of community app catalog. The mapping is not 1:1, but we can work together on this to make sure that these generic properties match the expectations of the catalog. 3. Versioning Versions are very important for catalogs of objects, so Glance V3 was initially designed keeping versioning questions in mind: each artifact has a semver-based version assigned, so the artifacts having the same name but different versions are grouped into the proper sequences. API is able to query artifacts based on their version spec, e.g. it is possible to fetch the latest artifact with the name “foo” having the version greater than 2.1 and less than 3.5.7 - or any other version spec, similar to pip or any other similar tool. As far as I know, community app catalog does not have such capabilities right now - and I strongly believe that it is really a must have feature for a catalog to be successful. At least it is absolutely mandatory for Murano packages, which are the only “real apps” among the asset types right now. 4. Cross artifact dependencies Glance V3 also has the dependency relations from the very beginning, so they may be defined as part of artifact type schema. As a result, an artifact may “reference” any number of other artifacts with various semantic. For example, murano package may define a set of references to other murano packages and call it “requires” - and this will act similar to the requirements of a python package. Similar properties may be defined for heat templates and glance images - they may reference each other with various semantics. Of course, the definitions of such dependencies may be done internally inside the packages, so they may be resolved locally by the service which is going to use it, but letting the catalog know about them will allow us to do the import-export operations for any given artifacts and its dependencies automatically, only by the means of the catalog itself. 5.
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
Alexander had been attending our meetings for a while, asking us to use Glance for the backend, and offering to provide a POC. We liked the idea and waited a while to see it. For example: http://eavesdrop.openstack.org/meetings/app_catalog/2015/app_catalog.2015-06-18-17.00.log.html http://eavesdrop.openstack.org/meetings/app_catalog/2015/app_catalog.2015-06-25-17.01.log.html But, it never materialized, he disappeared from our meetings, and we had to start making progress, so we took the existing stuff and tweaked it to be good enough for V1 so we could have something for Liberty. There's still plenty of opportunity to resolve it for V2, and I'd love to be able to share as much as we can. We could have tried harder to contact the rest of the Glance team to get a POC. We were kind of swamped trying to get something out the door for Liberty. I do blame myself for not trying harder there... I'm sorry. Please don't get me wrong. I'm not blaming Alexander, or anyone else at all. I know how busy everyone is. Different priorities and things taking longer then expected to develop. Stuff happens. Lets just focus on figuring out how we can make things better in the future. :) Thanks, Kevin From: Flavio Percoco [fla...@redhat.com] Sent: Sunday, October 18, 2015 10:51 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API forApp Catalog On 16/10/15 16:44 +, Fox, Kevin M wrote: >I agree that there’s a lot more room for cross project communication here at >very least. I was really hoping that we could get representatives from the >glance artifact subproject, the app-catalog, and the Murano teams in a room at >the summit and hash out some of this and other things. There is some overlap >and ambiguity that would be great to iron out if we could. > > > >I think there’s too much to discuss to do it via email, but I’ll take a stab at >a few things here. > > > >We’ve been waiting for 6 months for a POC for what you describe below and kind >of gave up on it. We needed to ship something and couldn’t wait any more. If >you think your close enough that you can show us at the summit, that would be >great. We’d be very happy to see it. I'm sorry to ask but I really don't know. Was this a silent wait? Or did you guys actively showed interest and asked for the POC? I'd like to understand how this has escalated and see how we can do better. Especially in the best interest of de-duplicating efforts. >At present, here’s my thinking about some of the things listed below: > > > >I think sharing the schema at very least would be great. If the versioning >stuff you have covers everything the app-catalog needs, we should just use it >rather then coming up with something ourselves. If its not set in stone right >now, it’s an ideal time to ensure all the features needed are there. > > > >The root of the tree, as you describe it has some problems that the rest of the >tree doesn’t. Its global and unauthenticated. It has to be Internet scale, not >just single cloud scale. Like you said, glance supports features for >authorization. But those may be anti-features for the root that may cause >excessive overhead when used as a root, unauthenticated source. The same is >true of Searchlight. Searchlight buys you elasticsearch+authorization/ >multitenancy, which the root catalog’s don’t need. They just need >eleasticsearch, so Searchlight may add complexity without any benefit. Don’t >get me wrong. I think Searchlight is awesome, just maybe not for https:// >apps.openstack.org. > > > >Not saying any of that is true, just possible reasons why we may not want to >use glance’s implementation at the root. We have to carefully consider things. > > > >If that code base can’t be shared, what may make more sense (I’m not sure. Just >laying options out), is to share a glance v3 artifact compatible api that >glance can point at for the root? Or a subset of the v3 api that makes sense to >both use cases and still supports the tree hierarchy. > > > >Like you said, we may need to extend the api specification to support some of >the features that are unique to the app-catalog use cases too. If you’re open >to it, I think we are too. If we go the route of sharing an api but not a >server implementation, that may not be necessary though. > > > >Anyway, thanks for starting the conversation. I think it’s a great thing to get >going. Can we get together at the summit and discuss further? ++ Thanks for chiming in, Kevin Flavio > > > >Thanks, > >Kevin > > > > > >From: Alexander Tivelkov [mailto:ativel...@mirantis.com] >Sent: Thursday, Octob
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
On 16/10/15 16:44 +, Fox, Kevin M wrote: I agree that there’s a lot more room for cross project communication here at very least. I was really hoping that we could get representatives from the glance artifact subproject, the app-catalog, and the Murano teams in a room at the summit and hash out some of this and other things. There is some overlap and ambiguity that would be great to iron out if we could. I think there’s too much to discuss to do it via email, but I’ll take a stab at a few things here. We’ve been waiting for 6 months for a POC for what you describe below and kind of gave up on it. We needed to ship something and couldn’t wait any more. If you think your close enough that you can show us at the summit, that would be great. We’d be very happy to see it. I'm sorry to ask but I really don't know. Was this a silent wait? Or did you guys actively showed interest and asked for the POC? I'd like to understand how this has escalated and see how we can do better. Especially in the best interest of de-duplicating efforts. At present, here’s my thinking about some of the things listed below: I think sharing the schema at very least would be great. If the versioning stuff you have covers everything the app-catalog needs, we should just use it rather then coming up with something ourselves. If its not set in stone right now, it’s an ideal time to ensure all the features needed are there. The root of the tree, as you describe it has some problems that the rest of the tree doesn’t. Its global and unauthenticated. It has to be Internet scale, not just single cloud scale. Like you said, glance supports features for authorization. But those may be anti-features for the root that may cause excessive overhead when used as a root, unauthenticated source. The same is true of Searchlight. Searchlight buys you elasticsearch+authorization/ multitenancy, which the root catalog’s don’t need. They just need eleasticsearch, so Searchlight may add complexity without any benefit. Don’t get me wrong. I think Searchlight is awesome, just maybe not for https:// apps.openstack.org. Not saying any of that is true, just possible reasons why we may not want to use glance’s implementation at the root. We have to carefully consider things. If that code base can’t be shared, what may make more sense (I’m not sure. Just laying options out), is to share a glance v3 artifact compatible api that glance can point at for the root? Or a subset of the v3 api that makes sense to both use cases and still supports the tree hierarchy. Like you said, we may need to extend the api specification to support some of the features that are unique to the app-catalog use cases too. If you’re open to it, I think we are too. If we go the route of sharing an api but not a server implementation, that may not be necessary though. Anyway, thanks for starting the conversation. I think it’s a great thing to get going. Can we get together at the summit and discuss further? ++ Thanks for chiming in, Kevin Flavio Thanks, Kevin From: Alexander Tivelkov [mailto:ativel...@mirantis.com] Sent: Thursday, October 15, 2015 3:05 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog Hi folks, I’ve noticed that the Community Application Catalog has begun to implement its own API, and it seems to me that we are going to have some significant duplication of efforts with the code which has already been implemented in Glance as Artifact Repository initiative (also known as Glance V3). From the very beginning of the App Catalog project (and I’ve been involved in it since February) I’ve been proposing to use Glance as its backend, because from my point of view it covers like 90% of the needed functionality. But it looks like we have some kind of miscommunication here, as I am getting some confusing questions and assumptions, like the vision of Glance V3 being dedicated backend for Murano (which is definitely incorrect). So, I am writing the email to clarify my vision of what Glance V3 is and how its features may be used to provide the REST API for Community App Catalog. 1. Versioned schema First of all, Glance V3 operates on entities called “artifacts”, and these ones perfectly map to the Data Assets of community app catalog. The artifacts are strongly typed: there are artifact types for murano packages, glance images, heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin, defining the schema of objects’ data and metadata and - optionally - custom logic. So, this thing is extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly. 2. Generic properties
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
On 16/10/15 12:07 -0700, Christopher Aedo wrote: On Thu, Oct 15, 2015 at 3:04 AM, Alexander Tivelkov wrote: Hi folks, I’ve noticed that the Community Application Catalog has begun to implement its own API, and it seems to me that we are going to have some significant duplication of efforts with the code which has already been implemented in Glance as Artifact Repository initiative (also known as Glance V3). From the very beginning of the App Catalog project (and I’ve been involved in it since February) I’ve been proposing to use Glance as its backend, because from my point of view it covers like 90% of the needed functionality. But it looks like we have some kind of miscommunication here, as I am getting some confusing questions and assumptions, like the vision of Glance V3 being dedicated backend for Murano (which is definitely incorrect). So, I am writing the email to clarify my vision of what Glance V3 is and how its features may be used to provide the REST API for Community App Catalog. 1. Versioned schema First of all, Glance V3 operates on entities called “artifacts”, and these ones perfectly map to the Data Assets of community app catalog. The artifacts are strongly typed: there are artifact types for murano packages, glance images, heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin, defining the schema of objects’ data and metadata and - optionally - custom logic. So, this thing is extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly. 2. Generic properties Next, all the artifact types in Glance V3 have some generic metadata properties (i.e. part of the schema which is common for all the types), including the name, the version, description, authorship information and so on. This also corresponds to the data schema of community app catalog. The mapping is not 1:1, but we can work together on this to make sure that these generic properties match the expectations of the catalog. 3. Versioning Versions are very important for catalogs of objects, so Glance V3 was initially designed keeping versioning questions in mind: each artifact has a semver-based version assigned, so the artifacts having the same name but different versions are grouped into the proper sequences. API is able to query artifacts based on their version spec, e.g. it is possible to fetch the latest artifact with the name “foo” having the version greater than 2.1 and less than 3.5.7 - or any other version spec, similar to pip or any other similar tool. As far as I know, community app catalog does not have such capabilities right now - and I strongly believe that it is really a must have feature for a catalog to be successful. At least it is absolutely mandatory for Murano packages, which are the only “real apps” among the asset types right now. 4. Cross artifact dependencies Glance V3 also has the dependency relations from the very beginning, so they may be defined as part of artifact type schema. As a result, an artifact may “reference” any number of other artifacts with various semantic. For example, murano package may define a set of references to other murano packages and call it “requires” - and this will act similar to the requirements of a python package. Similar properties may be defined for heat templates and glance images - they may reference each other with various semantics. Of course, the definitions of such dependencies may be done internally inside the packages, so they may be resolved locally by the service which is going to use it, but letting the catalog know about them will allow us to do the import-export operations for any given artifacts and its dependencies automatically, only by the means of the catalog itself. 5. Search and filtering API Right now Glance V3 API is in experimental state (we plan to stabilize it during the Mitaka cycle), but it already provides quite good capabilities to discover things. It can search artifacts by their type, name and (optionally) aforementioned version specs, by tag or even by arbitrary set of metadata properties. We have plans to integrate Glance V3 with the Searchlight project to have even more index and search capabilities using its elastic search engine. 6. Data storage As you probably know, Glance does not own the binary data of its images. Instead, it provides an abstraction of the backend storage, which may be swift, ceph, s3 or something else. The same approach is used in Glance V3 for artifacts data, but with more per-type control: particular artifact types may be configured independently to store their blobs in different backends. This may be of use for Community App Catalog which operates on different storages for its assets. 7. Sharing and access control. Glance V3 inherits the same access mechanics present
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
Hi folks! Thanks all for the feedback. Yeah, this definitely needs more discussions, and the Summit is good place to have them. Thanks to Serg for offering a spot in the schedule. I hope to have some PoC by that time, so we can have something which can be demoed. I'll try to attend the AppCatalog sessions as well. I didn't finalize my summit schedule yet, but I'll do my best to attend as much as I can. Thanks! -- Regards, Alexander Tivelkov On Fri, Oct 16, 2015 at 10:07 PM, Christopher Aedo wrote: > On Thu, Oct 15, 2015 at 3:04 AM, Alexander Tivelkov > wrote: > > Hi folks, > > > > I’ve noticed that the Community Application Catalog has begun to > implement > > its own API, and it seems to me that we are going to have some > significant > > duplication of efforts with the code which has already been implemented > in > > Glance as Artifact Repository initiative (also known as Glance V3). > > From the very beginning of the App Catalog project (and I’ve been > involved > > in it since February) I’ve been proposing to use Glance as its backend, > > because from my point of view it covers like 90% of the needed > > functionality. But it looks like we have some kind of miscommunication > here, > > as I am getting some confusing questions and assumptions, like the > vision of > > Glance V3 being dedicated backend for Murano (which is definitely > > incorrect). > > So, I am writing the email to clarify my vision of what Glance V3 is and > how > > its features may be used to provide the REST API for Community App > Catalog. > > > > 1. Versioned schema > > First of all, Glance V3 operates on entities called “artifacts”, and > these > > ones perfectly map to the Data Assets of community app catalog. The > > artifacts are strongly typed: there are artifact types for murano > packages, > > glance images, heat templates - and there may be (and will be) more. Each > > artifact type is represented by a plugin, defining the schema of objects’ > > data and metadata and - optionally - custom logic. So, this thing is > > extensible: when a new type of asset needs to be added to a catalog it > can > > be done really quickly by just defining the schema and putting that > schema > > into a plugin. Also, these plugins are versioned, so the possible > changes in > > the schema are handled properly. > > > > 2. Generic properties > > Next, all the artifact types in Glance V3 have some generic metadata > > properties (i.e. part of the schema which is common for all the types), > > including the name, the version, description, authorship information and > so > > on. This also corresponds to the data schema of community app catalog. > The > > mapping is not 1:1, but we can work together on this to make sure that > these > > generic properties match the expectations of the catalog. > > > > 3. Versioning > > Versions are very important for catalogs of objects, so Glance V3 was > > initially designed keeping versioning questions in mind: each artifact > has a > > semver-based version assigned, so the artifacts having the same name but > > different versions are grouped into the proper sequences. API is able to > > query artifacts based on their version spec, e.g. it is possible to fetch > > the latest artifact with the name “foo” having the version greater than > 2.1 > > and less than 3.5.7 - or any other version spec, similar to pip or any > other > > similar tool. As far as I know, community app catalog does not have such > > capabilities right now - and I strongly believe that it is really a must > > have feature for a catalog to be successful. At least it is absolutely > > mandatory for Murano packages, which are the only “real apps” among the > > asset types right now. > > > > 4. Cross artifact dependencies > > Glance V3 also has the dependency relations from the very beginning, so > they > > may be defined as part of artifact type schema. As a result, an artifact > may > > “reference” any number of other artifacts with various semantic. For > > example, murano package may define a set of references to other murano > > packages and call it “requires” - and this will act similar to the > > requirements of a python package. Similar properties may be defined for > heat > > templates and glance images - they may reference each other with various > > semantics. > > Of course, the definitions of such dependencies may be done internally > > inside the packages, so they may be resolved locally by the service > which is > > going to use it, but letting the catalog know about them will allow us > to do > > the import-export operations for any given artifacts and its dependencies > > automatically, only by the means of the catalog itself. > > > > 5. Search and filtering API > > Right now Glance V3 API is in experimental state (we plan to stabilize it > > during the Mitaka cycle), but it already provides quite good > capabilities to > > discover things. It can search artifacts by their type, name and > > (optionally) aforementioned vers
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
On Thu, Oct 15, 2015 at 3:04 AM, Alexander Tivelkov wrote: > Hi folks, > > I’ve noticed that the Community Application Catalog has begun to implement > its own API, and it seems to me that we are going to have some significant > duplication of efforts with the code which has already been implemented in > Glance as Artifact Repository initiative (also known as Glance V3). > From the very beginning of the App Catalog project (and I’ve been involved > in it since February) I’ve been proposing to use Glance as its backend, > because from my point of view it covers like 90% of the needed > functionality. But it looks like we have some kind of miscommunication here, > as I am getting some confusing questions and assumptions, like the vision of > Glance V3 being dedicated backend for Murano (which is definitely > incorrect). > So, I am writing the email to clarify my vision of what Glance V3 is and how > its features may be used to provide the REST API for Community App Catalog. > > 1. Versioned schema > First of all, Glance V3 operates on entities called “artifacts”, and these > ones perfectly map to the Data Assets of community app catalog. The > artifacts are strongly typed: there are artifact types for murano packages, > glance images, heat templates - and there may be (and will be) more. Each > artifact type is represented by a plugin, defining the schema of objects’ > data and metadata and - optionally - custom logic. So, this thing is > extensible: when a new type of asset needs to be added to a catalog it can > be done really quickly by just defining the schema and putting that schema > into a plugin. Also, these plugins are versioned, so the possible changes in > the schema are handled properly. > > 2. Generic properties > Next, all the artifact types in Glance V3 have some generic metadata > properties (i.e. part of the schema which is common for all the types), > including the name, the version, description, authorship information and so > on. This also corresponds to the data schema of community app catalog. The > mapping is not 1:1, but we can work together on this to make sure that these > generic properties match the expectations of the catalog. > > 3. Versioning > Versions are very important for catalogs of objects, so Glance V3 was > initially designed keeping versioning questions in mind: each artifact has a > semver-based version assigned, so the artifacts having the same name but > different versions are grouped into the proper sequences. API is able to > query artifacts based on their version spec, e.g. it is possible to fetch > the latest artifact with the name “foo” having the version greater than 2.1 > and less than 3.5.7 - or any other version spec, similar to pip or any other > similar tool. As far as I know, community app catalog does not have such > capabilities right now - and I strongly believe that it is really a must > have feature for a catalog to be successful. At least it is absolutely > mandatory for Murano packages, which are the only “real apps” among the > asset types right now. > > 4. Cross artifact dependencies > Glance V3 also has the dependency relations from the very beginning, so they > may be defined as part of artifact type schema. As a result, an artifact may > “reference” any number of other artifacts with various semantic. For > example, murano package may define a set of references to other murano > packages and call it “requires” - and this will act similar to the > requirements of a python package. Similar properties may be defined for heat > templates and glance images - they may reference each other with various > semantics. > Of course, the definitions of such dependencies may be done internally > inside the packages, so they may be resolved locally by the service which is > going to use it, but letting the catalog know about them will allow us to do > the import-export operations for any given artifacts and its dependencies > automatically, only by the means of the catalog itself. > > 5. Search and filtering API > Right now Glance V3 API is in experimental state (we plan to stabilize it > during the Mitaka cycle), but it already provides quite good capabilities to > discover things. It can search artifacts by their type, name and > (optionally) aforementioned version specs, by tag or even by arbitrary set > of metadata properties. We have plans to integrate Glance V3 with the > Searchlight project to have even more index and search capabilities using > its elastic search engine. > > 6. Data storage > As you probably know, Glance does not own the binary data of its images. > Instead, it provides an abstraction of the backend storage, which may be > swift, ceph, s3 or something else. The same approach is used in Glance V3 > for artifacts data, but with more per-type control: particular artifact > types may be configured independently to store their blobs in different > backends. This may be of use for Community App Catalog which operates on > different stor
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
Yeah. That sounds great. Thanks. :) Alexander, would you be able to make it? Thanks, Kevin From: Serg Melikyan [smelik...@mirantis.com] Sent: Friday, October 16, 2015 10:39 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog Hi Kevin, >Can we get together at the summit and discuss further? I would be happy to add this meeting to the Murano summit schedule, we have table on contributors meetup for the whole Friday. Will it be convenient? This is very important for Murano, it would be great to seat together and think what actual requirements we have and how to satisfy them. On Fri, Oct 16, 2015 at 7:44 PM, Fox, Kevin M wrote: > I agree that there’s a lot more room for cross project communication here at > very least. I was really hoping that we could get representatives from the > glance artifact subproject, the app-catalog, and the Murano teams in a room > at the summit and hash out some of this and other things. There is some > overlap and ambiguity that would be great to iron out if we could. > > > > I think there’s too much to discuss to do it via email, but I’ll take a stab > at a few things here. > > > > We’ve been waiting for 6 months for a POC for what you describe below and > kind of gave up on it. We needed to ship something and couldn’t wait any > more. If you think your close enough that you can show us at the summit, > that would be great. We’d be very happy to see it. > > > > At present, here’s my thinking about some of the things listed below: > > > > I think sharing the schema at very least would be great. If the versioning > stuff you have covers everything the app-catalog needs, we should just use > it rather then coming up with something ourselves. If its not set in stone > right now, it’s an ideal time to ensure all the features needed are there. > > > > The root of the tree, as you describe it has some problems that the rest of > the tree doesn’t. Its global and unauthenticated. It has to be Internet > scale, not just single cloud scale. Like you said, glance supports features > for authorization. But those may be anti-features for the root that may > cause excessive overhead when used as a root, unauthenticated source. The > same is true of Searchlight. Searchlight buys you > elasticsearch+authorization/multitenancy, which the root catalog’s don’t > need. They just need eleasticsearch, so Searchlight may add complexity > without any benefit. Don’t get me wrong. I think Searchlight is awesome, > just maybe not for https://apps.openstack.org. > > > > Not saying any of that is true, just possible reasons why we may not want to > use glance’s implementation at the root. We have to carefully consider > things. > > > > If that code base can’t be shared, what may make more sense (I’m not sure. > Just laying options out), is to share a glance v3 artifact compatible api > that glance can point at for the root? Or a subset of the v3 api that makes > sense to both use cases and still supports the tree hierarchy. > > > > Like you said, we may need to extend the api specification to support some > of the features that are unique to the app-catalog use cases too. If you’re > open to it, I think we are too. If we go the route of sharing an api but not > a server implementation, that may not be necessary though. > > > > Anyway, thanks for starting the conversation. I think it’s a great thing to > get going. Can we get together at the summit and discuss further? > > > > Thanks, > > Kevin > > > > > > From: Alexander Tivelkov [mailto:ativel...@mirantis.com] > Sent: Thursday, October 15, 2015 3:05 AM > To: OpenStack Development Mailing List (not for usage questions) > Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for > App Catalog > > > > Hi folks, > > > > I’ve noticed that the Community Application Catalog has begun to implement > its own API, and it seems to me that we are going to have some significant > duplication of efforts with the code which has already been implemented in > Glance as Artifact Repository initiative (also known as Glance V3). > > From the very beginning of the App Catalog project (and I’ve been involved > in it since February) I’ve been proposing to use Glance as its backend, > because from my point of view it covers like 90% of the needed > functionality. But it looks like we have some kind of miscommunication here, > as I am getting some confusing questions and assumptions, like the vision of > Glance V3 being dedicated backend for Murano (which is definitely > incorrect). > > So, I am writin
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
On Fri, Oct 16, 2015 at 10:39 AM, Serg Melikyan wrote: > Hi Kevin, > >>Can we get together at the summit and discuss further? > I would be happy to add this meeting to the Murano summit schedule, we > have table on contributors meetup for the whole Friday. Will it be > convenient? > > This is very important for Murano, it would be great to seat together > and think what actual requirements we have and how to satisfy them. Serg, please feel free to add something to the agenda to discuss cross-project collaboration, I think it would be great. We also have two sessions specifically for the App Catalog which I encourage folks from Murano, Glance, Horizon and other projects to attend. I'll make an effort to be involved in their respective work sessions as well. The two sessions I urge you and others to join are: Status, progress and plans http://mitakadesignsummit.sched.org/event/27bf7f9a29094cf9e96026d682db1609 Thursday in the Kotobuki room, from 1:50 to 2:30 Work session http://mitakadesignsummit.sched.org/event/7754b46437c14cd4fdb51debebe89fb0 Thursday in the Tachibana room, from 2:40 to 3:20 -Christopher __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
Hi Kevin, >Can we get together at the summit and discuss further? I would be happy to add this meeting to the Murano summit schedule, we have table on contributors meetup for the whole Friday. Will it be convenient? This is very important for Murano, it would be great to seat together and think what actual requirements we have and how to satisfy them. On Fri, Oct 16, 2015 at 7:44 PM, Fox, Kevin M wrote: > I agree that there’s a lot more room for cross project communication here at > very least. I was really hoping that we could get representatives from the > glance artifact subproject, the app-catalog, and the Murano teams in a room > at the summit and hash out some of this and other things. There is some > overlap and ambiguity that would be great to iron out if we could. > > > > I think there’s too much to discuss to do it via email, but I’ll take a stab > at a few things here. > > > > We’ve been waiting for 6 months for a POC for what you describe below and > kind of gave up on it. We needed to ship something and couldn’t wait any > more. If you think your close enough that you can show us at the summit, > that would be great. We’d be very happy to see it. > > > > At present, here’s my thinking about some of the things listed below: > > > > I think sharing the schema at very least would be great. If the versioning > stuff you have covers everything the app-catalog needs, we should just use > it rather then coming up with something ourselves. If its not set in stone > right now, it’s an ideal time to ensure all the features needed are there. > > > > The root of the tree, as you describe it has some problems that the rest of > the tree doesn’t. Its global and unauthenticated. It has to be Internet > scale, not just single cloud scale. Like you said, glance supports features > for authorization. But those may be anti-features for the root that may > cause excessive overhead when used as a root, unauthenticated source. The > same is true of Searchlight. Searchlight buys you > elasticsearch+authorization/multitenancy, which the root catalog’s don’t > need. They just need eleasticsearch, so Searchlight may add complexity > without any benefit. Don’t get me wrong. I think Searchlight is awesome, > just maybe not for https://apps.openstack.org. > > > > Not saying any of that is true, just possible reasons why we may not want to > use glance’s implementation at the root. We have to carefully consider > things. > > > > If that code base can’t be shared, what may make more sense (I’m not sure. > Just laying options out), is to share a glance v3 artifact compatible api > that glance can point at for the root? Or a subset of the v3 api that makes > sense to both use cases and still supports the tree hierarchy. > > > > Like you said, we may need to extend the api specification to support some > of the features that are unique to the app-catalog use cases too. If you’re > open to it, I think we are too. If we go the route of sharing an api but not > a server implementation, that may not be necessary though. > > > > Anyway, thanks for starting the conversation. I think it’s a great thing to > get going. Can we get together at the summit and discuss further? > > > > Thanks, > > Kevin > > > > > > From: Alexander Tivelkov [mailto:ativel...@mirantis.com] > Sent: Thursday, October 15, 2015 3:05 AM > To: OpenStack Development Mailing List (not for usage questions) > Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for > App Catalog > > > > Hi folks, > > > > I’ve noticed that the Community Application Catalog has begun to implement > its own API, and it seems to me that we are going to have some significant > duplication of efforts with the code which has already been implemented in > Glance as Artifact Repository initiative (also known as Glance V3). > > From the very beginning of the App Catalog project (and I’ve been involved > in it since February) I’ve been proposing to use Glance as its backend, > because from my point of view it covers like 90% of the needed > functionality. But it looks like we have some kind of miscommunication here, > as I am getting some confusing questions and assumptions, like the vision of > Glance V3 being dedicated backend for Murano (which is definitely > incorrect). > > So, I am writing the email to clarify my vision of what Glance V3 is and how > its features may be used to provide the REST API for Community App Catalog. > > > > 1. Versioned schema > > First of all, Glance V3 operates on entities called “artifacts”, and these > ones perfectly map to the Data Assets of community app catalog. The > artifacts are strongly typed: th
Re: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
I agree that there’s a lot more room for cross project communication here at very least. I was really hoping that we could get representatives from the glance artifact subproject, the app-catalog, and the Murano teams in a room at the summit and hash out some of this and other things. There is some overlap and ambiguity that would be great to iron out if we could. I think there’s too much to discuss to do it via email, but I’ll take a stab at a few things here. We’ve been waiting for 6 months for a POC for what you describe below and kind of gave up on it. We needed to ship something and couldn’t wait any more. If you think your close enough that you can show us at the summit, that would be great. We’d be very happy to see it. At present, here’s my thinking about some of the things listed below: I think sharing the schema at very least would be great. If the versioning stuff you have covers everything the app-catalog needs, we should just use it rather then coming up with something ourselves. If its not set in stone right now, it’s an ideal time to ensure all the features needed are there. The root of the tree, as you describe it has some problems that the rest of the tree doesn’t. Its global and unauthenticated. It has to be Internet scale, not just single cloud scale. Like you said, glance supports features for authorization. But those may be anti-features for the root that may cause excessive overhead when used as a root, unauthenticated source. The same is true of Searchlight. Searchlight buys you elasticsearch+authorization/multitenancy, which the root catalog’s don’t need. They just need eleasticsearch, so Searchlight may add complexity without any benefit. Don’t get me wrong. I think Searchlight is awesome, just maybe not for https://apps.openstack.org. Not saying any of that is true, just possible reasons why we may not want to use glance’s implementation at the root. We have to carefully consider things. If that code base can’t be shared, what may make more sense (I’m not sure. Just laying options out), is to share a glance v3 artifact compatible api that glance can point at for the root? Or a subset of the v3 api that makes sense to both use cases and still supports the tree hierarchy. Like you said, we may need to extend the api specification to support some of the features that are unique to the app-catalog use cases too. If you’re open to it, I think we are too. If we go the route of sharing an api but not a server implementation, that may not be necessary though. Anyway, thanks for starting the conversation. I think it’s a great thing to get going. Can we get together at the summit and discuss further? Thanks, Kevin From: Alexander Tivelkov [mailto:ativel...@mirantis.com] Sent: Thursday, October 15, 2015 3:05 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog Hi folks, I’ve noticed that the Community Application Catalog has begun to implement its own API, and it seems to me that we are going to have some significant duplication of efforts with the code which has already been implemented in Glance as Artifact Repository initiative (also known as Glance V3). From the very beginning of the App Catalog project (and I’ve been involved in it since February) I’ve been proposing to use Glance as its backend, because from my point of view it covers like 90% of the needed functionality. But it looks like we have some kind of miscommunication here, as I am getting some confusing questions and assumptions, like the vision of Glance V3 being dedicated backend for Murano (which is definitely incorrect). So, I am writing the email to clarify my vision of what Glance V3 is and how its features may be used to provide the REST API for Community App Catalog. 1. Versioned schema First of all, Glance V3 operates on entities called “artifacts”, and these ones perfectly map to the Data Assets of community app catalog. The artifacts are strongly typed: there are artifact types for murano packages, glance images, heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin, defining the schema of objects’ data and metadata and - optionally - custom logic. So, this thing is extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly. 2. Generic properties Next, all the artifact types in Glance V3 have some generic metadata properties (i.e. part of the schema which is common for all the types), including the name, the version, description, authorship information and so on. This also corresponds to the data schema of community app catalog. The mapping is not 1:1, but we can work together on this to make sure that
[openstack-dev] [app-catalog] [glance] [murano] Data Assets API for App Catalog
Hi folks, I’ve noticed that the Community Application Catalog has begun to implement its own API, and it seems to me that we are going to have some significant duplication of efforts with the code which has already been implemented in Glance as Artifact Repository initiative (also known as Glance V3). >From the very beginning of the App Catalog project (and I’ve been involved in it since February) I’ve been proposing to use Glance as its backend, because from my point of view it covers like 90% of the needed functionality. But it looks like we have some kind of miscommunication here, as I am getting some confusing questions and assumptions, like the vision of Glance V3 being dedicated backend for Murano (which is definitely incorrect). So, I am writing the email to clarify my vision of what Glance V3 is and how its features may be used to provide the REST API for Community App Catalog. 1. Versioned schema First of all, Glance V3 operates on entities called “artifacts”, and these ones perfectly map to the Data Assets of community app catalog. The artifacts are strongly typed: there are artifact types for murano packages, glance images, heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin, defining the schema of objects’ data and metadata and - optionally - custom logic. So, this thing is extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly. 2. Generic properties Next, all the artifact types in Glance V3 have some generic metadata properties (i.e. part of the schema which is common for all the types), including the name, the version, description, authorship information and so on. This also corresponds to the data schema of community app catalog. The mapping is not 1:1, but we can work together on this to make sure that these generic properties match the expectations of the catalog. 3. Versioning Versions are very important for catalogs of objects, so Glance V3 was initially designed keeping versioning questions in mind: each artifact has a semver-based version assigned, so the artifacts having the same name but different versions are grouped into the proper sequences. API is able to query artifacts based on their version spec, e.g. it is possible to fetch the latest artifact with the name “foo” having the version greater than 2.1 and less than 3.5.7 - or any other version spec, similar to pip or any other similar tool. As far as I know, community app catalog does not have such capabilities right now - and I strongly believe that it is really a must have feature for a catalog to be successful. At least it is absolutely mandatory for Murano packages, which are the only “real apps” among the asset types right now. 4. Cross artifact dependencies Glance V3 also has the dependency relations from the very beginning, so they may be defined as part of artifact type schema. As a result, an artifact may “reference” any number of other artifacts with various semantic. For example, murano package may define a set of references to other murano packages and call it “requires” - and this will act similar to the requirements of a python package. Similar properties may be defined for heat templates and glance images - they may reference each other with various semantics. Of course, the definitions of such dependencies may be done internally inside the packages, so they may be resolved locally by the service which is going to use it, but letting the catalog know about them will allow us to do the import-export operations for any given artifacts and its dependencies automatically, only by the means of the catalog itself. 5. Search and filtering API Right now Glance V3 API is in experimental state (we plan to stabilize it during the Mitaka cycle), but it already provides quite good capabilities to discover things. It can search artifacts by their type, name and (optionally) aforementioned version specs, by tag or even by arbitrary set of metadata properties. We have plans to integrate Glance V3 with the Searchlight project to have even more index and search capabilities using its elastic search engine. 6. Data storage As you probably know, Glance does not own the binary data of its images. Instead, it provides an abstraction of the backend storage, which may be swift, ceph, s3 or something else. The same approach is used in Glance V3 for artifacts data, but with more per-type control: particular artifact types may be configured independently to store their blobs in different backends. This may be of use for Community App Catalog which operates on different storages for its assets. 7. Sharing and access control. Glance V3 inherits the same access mechanics present in Glance V2: an artifact may be visible to its owner tenant only, be public (i.e. visible to all the tenants)
