Hi Kevin,

>Can we get together at the summit and discuss further?
I would be happy to add this meeting to the Murano summit schedule, we
have table on contributors meetup for the whole Friday. Will it be
convenient?

This is very important for Murano, it would be great to seat together
and think what actual requirements we have and how to satisfy them.

On Fri, Oct 16, 2015 at 7:44 PM, Fox, Kevin M <kevin....@pnnl.gov> wrote:
> I agree that there’s a lot more room for cross project communication here at
> very least. I was really hoping that we could get representatives from the
> glance artifact subproject, the app-catalog, and the Murano teams in a room
> at the summit and hash out some of this and other things. There is some
> overlap and ambiguity that would be great to iron out if we could.
>
>
>
> I think there’s too much to discuss to do it via email, but I’ll take a stab
> at a few things here.
>
>
>
> We’ve been waiting for 6 months for a POC for what you describe below and
> kind of gave up on it. We needed to ship something and couldn’t wait any
> more. If you think your close enough that you can show us at the summit,
> that would be great. We’d be very happy to see it.
>
>
>
> At present, here’s my thinking about some of the things listed below:
>
>
>
> I think sharing the schema at very least would be great. If the versioning
> stuff you have covers everything the app-catalog needs, we should just use
> it rather then coming up with something ourselves. If its not set in stone
> right now, it’s an ideal time to ensure all the features needed are there.
>
>
>
> The root of the tree, as you describe it has some problems that the rest of
> the tree doesn’t. Its global and unauthenticated. It has to be Internet
> scale, not just single cloud scale. Like you said, glance supports features
> for authorization. But those may be anti-features for the root that may
> cause excessive overhead when used as a root, unauthenticated source. The
> same is true of Searchlight. Searchlight buys you
> elasticsearch+authorization/multitenancy, which the root catalog’s don’t
> need. They just need eleasticsearch, so Searchlight may add complexity
> without any benefit. Don’t get me wrong. I think Searchlight is awesome,
> just maybe not for https://apps.openstack.org.
>
>
>
> Not saying any of that is true, just possible reasons why we may not want to
> use glance’s implementation at the root. We have to carefully consider
> things.
>
>
>
> If that code base can’t be shared, what may make more sense (I’m not sure.
> Just laying options out), is to share a glance v3 artifact compatible api
> that glance can point at for the root? Or a subset of the v3 api that makes
> sense to both use cases and still supports the tree hierarchy.
>
>
>
> Like you said, we may need to extend the api specification to support some
> of the features that are unique to the app-catalog use cases too. If you’re
> open to it, I think we are too. If we go the route of sharing an api but not
> a server implementation, that may not be necessary though.
>
>
>
> Anyway, thanks for starting the conversation. I think it’s a great thing to
> get going. Can we get together at the summit and discuss further?
>
>
>
> Thanks,
>
> Kevin
>
>
>
>
>
> From: Alexander Tivelkov [mailto:ativel...@mirantis.com]
> Sent: Thursday, October 15, 2015 3:05 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: [openstack-dev] [app-catalog] [glance] [murano] Data Assets API for
> App Catalog
>
>
>
> Hi folks,
>
>
>
> I’ve noticed that the Community Application Catalog has begun to implement
> its own API, and it seems to me that we are going to have some significant
> duplication of efforts with the code which has already been implemented in
> Glance as Artifact Repository initiative (also known as Glance V3).
>
> From the very beginning of the App Catalog project (and I’ve been involved
> in it since February) I’ve been proposing to use Glance as its backend,
> because from my point of view it covers like 90% of the needed
> functionality. But it looks like we have some kind of miscommunication here,
> as I am getting some confusing questions and assumptions, like the vision of
> Glance V3 being dedicated backend for Murano (which is definitely
> incorrect).
>
> So, I am writing the email to clarify my vision of what Glance V3 is and how
> its features may be used to provide the REST API for Community App Catalog.
>
>
>
> 1.  Versioned schema
>
> First of all, Glance V3 operates on entities called “artifacts”, and these
> ones perfectly map to the Data Assets of community app catalog. The
> artifacts are strongly typed: there are artifact types for murano packages,
> glance images, heat templates - and there may be (and will be) more. Each
> artifact type is represented by a plugin, defining the schema of objects’
> data and metadata and - optionally - custom logic. So, this thing is
> extensible: when a new type of asset needs to be added to a catalog it can
> be done really quickly by just defining the schema and putting that schema
> into a plugin. Also, these plugins are versioned, so the possible changes in
> the schema are handled properly.
>
>
>
> 2. Generic properties
>
> Next, all the artifact types in Glance V3 have some generic metadata
> properties (i.e. part of the schema which is common for all the types),
> including the name, the version, description, authorship information and so
> on. This also corresponds to the data schema of community app catalog. The
> mapping is not 1:1, but we can work together on this to make sure that these
> generic properties match the expectations of the catalog.
>
>
>
> 3. Versioning
>
> Versions are very important for catalogs of objects, so Glance V3 was
> initially designed keeping versioning questions in mind: each artifact has a
> semver-based version assigned, so the artifacts having the same name but
> different versions are grouped into the proper sequences. API is able to
> query artifacts based on their version spec, e.g. it is possible to fetch
> the latest artifact with the name “foo” having the version greater than 2.1
> and less than 3.5.7 - or any other version spec, similar to pip or any other
> similar tool. As far as I know, community app catalog does not have such
> capabilities right now - and I strongly believe that it is really a must
> have feature for a catalog to be successful. At least it is absolutely
> mandatory for Murano packages, which are the only “real apps” among the
> asset types right now.
>
>
>
> 4. Cross artifact dependencies
>
> Glance V3 also has the dependency relations from the very beginning, so they
> may be defined as part of artifact type schema. As a result, an artifact may
> “reference” any number of other artifacts with various semantic. For
> example, murano package may define a set of references to other murano
> packages and call it “requires” - and this will act similar to the
> requirements of a python package. Similar properties may be defined for heat
> templates and glance images - they may reference each other with various
> semantics.
>
> Of course, the definitions of such dependencies may be done internally
> inside the packages, so they may be resolved locally by the service which is
> going to use it, but letting the catalog know about them will allow us to do
> the import-export operations for any given artifacts and its dependencies
> automatically, only by the means of the catalog itself.
>
>
>
> 5. Search and filtering API
>
> Right now Glance V3 API is in experimental state (we plan to stabilize it
> during the Mitaka cycle), but it already provides quite good capabilities to
> discover things. It can search artifacts by their type, name and
> (optionally) aforementioned version specs, by tag or even by arbitrary set
> of metadata properties. We have plans to integrate Glance V3 with the
> Searchlight project to have even more index and search capabilities using
> its elastic search engine.
>
>
>
> 6. Data storage
>
> As you probably know, Glance does not own the binary data of its images.
> Instead, it provides an abstraction of the backend storage, which may be
> swift, ceph, s3 or something else. The same approach is used in Glance V3
> for artifacts data, but with more per-type control: particular artifact
> types may be configured independently to store their blobs in different
> backends. This may be of use for Community App Catalog which operates on
> different storages for its assets.
>
>
>
> 7. Sharing and access control.
>
> Glance V3 inherits the same access mechanics present in Glance V2: an
> artifact may be visible to its owner tenant only, be public (i.e. visible to
> all the tenants) or directly shared by the owner to a specific tenant. Also,
> Glance can act in the anonymous mode (i.e. without an access token), thus
> providing access to public artifacts even to unauthenticated users.
>
> This can be easily applied to a public web service, such as community app
> catalog: regular unauthenticated users use anonymous mode to access only the
> public assets (this is the current behavior of apps.o.o), while registered
> users will have their own private spaces (“tenants”) with various access
> restrictions.
>
>
>
> 8. The federation.
>
> The ultimate goal for Glance Artifact Repository is ability to build trees
> of artifact repos in different clouds. The top node of that tree is some
> Global Application Catalog (and the apps.openstack.org may be this global
> catalog - if it is glance-based or at least supports glance v3 federation),
> then there are repositories of particular openstack vendors or distributors,
> then - the catalogs of enterprises operating different openstack clouds. The
> particular glance deployments in that clouds are the leafs of that tree,
> being able to search for data assets in all the upstream repositories,
> download them from there or - if permitted - submit their local assets back
> upstream. This will be the ultimate network for application delivery and
> exchange in openstack world - and this is one of the main reasons we’ve
> began the Artifacts initiative in Glance.
>
> Unlike other aforementioned features this one is not implemented yet, but we
> are planning to add it as soon as we are done with API stabilization goal.
>
>
>
>
>
> There are many other features which are present in V3’s roadmap and may be
> useful for the app catalog, such as ability to sign artifacts with their
> developers’ keys and verify that keys on usage to ensure the authenticity of
> the artifact.
>
>
>
> What we don’t have right now is the ability to associate ratings (“stars”)
> and comments to the artifact, as well as aggregating different usage and
> download statistics: such features are really needed only for the public
> website such as apps.o.o but are not required for Glance’s in particular
> clouds. But we may find some way to solve this, either by wrapping glance
> API with additional middleware which would add appropriate info from a
> different data source, or by having custom plugins which are able to do
> that, or in some other way: I am sure we may find a solution for this.
>
>
>
> So, this was just a brief description of what Glance v3 has to offer as a
> backend for App Catalog API.
>
> It also worths to mention that this API is in “EXPERIMENTAL” state right
> now, which means that it is not fixed and we may modify it significantly if
> there is a need to. So we may work closer together to adopt it for the needs
> of Community App Catalog.
>
>
>
> I would really prefer to not create any overlaps between Glance v3 and the
> community app catalog: if the app catalog builds its own incompatible
> implementation of assets discovery and distribution API then we’ll have a
> huge duplication of efforts for developers and lots of confusion to the
> end-users who will get two entirely different ways to do the same task.
>
>
>
> So, I’d propose to discuss these potential overlaps, look at the features
> need by App Catalog and see how Glance V3 may be of use here. I’ll be more
> than happy to help with that. We can dive deeper into the details here in
> the mailing list or meet in person in Tokyo. I'll try to have a
> demonstratable prototype by that time.
>
>
>
> --
>
> Regards,
> Alexander Tivelkov
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Serg Melikyan, Senior Software Engineer at Mirantis, Inc.
http://mirantis.com | smelik...@mirantis.com

+7 (495) 640-4904, 0261
+7 (903) 156-0836

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to