Hi Dave, This is a timely topic, since we have been prototyping and mocking up something similar at Mesosphere. We created a new public GitHub repository for it about three weeks ago called "universe" (http://github.com/mesosphere/universe).
Although we have added some informal specs, it's very malleable at this point. We're very much interested in making our "universe" compatible with, or the same as, the registry you're proposing. Without delving into implementation details, some of the goals we have in mind are outlined below. Data Source: The package repository should be easily consumable by third-party command-line and other programs. There should be a condensed “index” representation of the package repository available. Packages within the repository should be versioned. The package repository format itself should be versioned. Decentralization and Composability: The package metadata should be hosted in a public place (we like GitHub) so that additional packages can be added by the community by simply submitting pull requests. We have added some rudimentary commit hooks and automated validation to protect the repo against breaking changes. It’s important that no single entity “owns the keys” to the universe, and that the spec and implementation remain public. It should be easy and free for organizations to maintain a private package repository. A corollary is that it should be easy for consumers to pull from a hierarchy of upstream repositories. One setup we have in mind is that an organization might have staging and production repositories running internally. Packages are pushed to staging where integration testing can run before “deployment” to production. If a package isn’t in the local repository it might be looked up and installed from upstream. Repositories should be able to be proxied and cached in this way. Organizations should be able to isolate their datacenter but also selectively add external packages for experimentation. The system should be sufficiently portable and extensible to accomodate these and similar use cases. Meta-Framework Descriptors: Our conception of the package repository is a bit more expansive than just Mesos frameworks; it includes descriptions of how to install any piece of server software on a Mesos cluster. Frameworks and non-frameworks alike may be installed using some other meta-framework that’s responsible for starting all other cluster services. Likely candidates for this role are the long-lived frameworks: Aurora, Marathon, Singularity, and eventually Kubernetes. In any case, the repository spec should not be prescriptive with respect to this choice. The package repository metadata should make it easy for Mesos framework authors (and authors of non-Mesos-aware programs) to describe how to install their software on a Mesos cluster. To this end, our prototype package spec allows for Meta-framework descriptor files for each package in the repository. For example for a given package we might see a `marathon.json` file as well as a `my-app.aurora` file. An obvious concern is how to specify site-specific arguments upon installation. Here packages should describe data that must be marshalled from the environment (e.g. by prompting a user) and combined with the raw meta-framework descriptor to launch the app. These configuration parameters should be agnostic of the supported meta-frameworks. More concretely, in our prototype we describe configuration data in terms of a JSON-Schema. CLI Integration: Part of our proposed package format is an optional descriptor for how to fetch and install the command-line tools for interacting with the application. For now, we only have one implementation of this, which is to fetch a python egg from PyPI. Governance: All in all, we think that making this effort more community driven is a healthy way to proceed. Any input is very welcome. For example, if others think that what we have is a good starting point we could transfer ownership of the repository to the mesos organization on GitHub. Cheers, -- Connor Doyle http://mesosphere.com > On Nov 30, 2014, at 17:32, Dave Lester <[email protected]> wrote: > > As the number of Mesos frameworks grows (and now, a module system), I think > it's time to create a community-maintained registry with the goal of making > frameworks and modules easier to discover, contribute to, and install. > > There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted > with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod > Kone, Timothy St Clair, and Joe Stein). I'd like to advance the conversation > by offering a proposal on the public mailing list. > > I imagine two initiatives to achieve this: > > 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with how > Jenkins maintains their plugins on GitHub [1], but they allow individual > plugins to have their own repo within their GH org. Plugins are repos with a > specific naming convention (in their case, they've appended "-plugin" to the > repo name but this isn't the same approach we'd need to take). Having a > shared GH org creates greater visibility to what people are doing, and > encourages community participation and ownership. > > In the case of Mesos, not all frameworks will jump at the chance to have > community hosting which is fine. But particularly for smaller frameworks, I > think this is a good option given the success Jenkins has had. We could > potentially use the existing /mesos github org, or create a new org > /mesos-extensions or something of the like. It seems like the only potential > snag here is that we'll want to be explicit that these aren't Apache-blessed > plugins and instead maintained by the larger ecosystem, but I believe we can > achieve that by offering a notice in the GH org description. > > 2) A registry that allows framework writers to publish new versions of their > frameworks to a central repository that could be programmatically accessed > and rendered online. The v1 could be incredibly simple, but I think this is a > foundational piece that can grow with the project in the future. Since this > is a little bit more-involved, I've created a separate Google Doc [2] to > begin drafting the scope for this: > https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing, > and have intentionally punted on some of the implementation details until > the scope is finalized and I gauge interest from users and potential > collaborators. > > What do you think? If folks are on board I will assign the JIRA issue to > myself and get to work; I'd also welcome collaborators to help get this off > the ground since I think it will be a boost for the community. Feedback is > welcome! > > Thanks, > Dave > > [1] https://github.com/jenkinsci/ > [2] > https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
