Hi Dave, Your rephrasing is accurate, and you do bring up some interesting points around management. Sorry for the response delay; I and the rest of the Mesosphere folks will be back in SF following our Q4 offsite by mid-week.
-- Connor On Fri, Dec 12, 2014 at 12:39 PM, Dave Lester <[email protected]> wrote: > Hey Connor, > > It would be great to hear back about this and get the ball rolling on a > shared registry. Let me know what you think! > > Thanks, > Dave > > On Wed, Dec 3, 2014 at 1:11 PM, Dave Lester <[email protected]> wrote: >> >> Hi Connor, >> >> Thanks for sharing the work you've been doing with Universe. Sounds >> complimentary and pretty exciting! I thought I'd take a stab at making some >> clarifications. >> >> To rephrase your proposal by using a comparison, you're proposing the >> "homebrew model" of package management here in terms of how the registry is >> maintained, correct? A registry of metadata that's hosted via a single >> GitHub repository, that community members submit pull requests to when they >> want to have a release included in the registry. >> >> In this case of Homebrew, they manage "formulas"; in our case it would be >> package.json files (which is how NodeJS handles package info). >> >> Architecture >> The only way Universe diverges from my original thinking is that you're >> advocating decentralization of package metadata, whereas my original thought >> was to have it much more centralized. There's no reason it couldn't be >> brought together through a single interface (command-line, or searchable web >> interface) so I don't think it's incompatible technically, although I do >> think it creates some potential social/governance overhead (more below). >> >> Metadata >> The package.json file is minimal and similar to what you see in most >> package management systems. Where you're taking it a step further is a JSON >> manifest for how to actually run the program, which is outside of the scope >> of what I was hoping to accomplish but it looks awesome. >> >> Managing The Registry >> One part of your proposal that's unclear to me is when information is >> pushed to the registry repository, and how often (for each release, or just >> when the package is first added). Additionally, what information is pushed >> here -- just a pointer to the git repo? It's my current understanding that >> the package.json and framework.json files are both within the actual >> package's repo, not the registry. I would love to hear how you're thinking >> about this. >> >> If it's just a pointer to another repo, it would make it possible for any >> package already vetted to become part of the registry to do their own >> updates without requiring someone to accept a PR. I think this is ideal. If >> it's the latter, it brings up some interesting questions related to who >> would manage this and under what guidelines they would do this vetting; we'd >> essentially get a PR for every release within the community at that point. >> >> The Registry "Home" >> I feel strongly that this should be a community-driven effort, and I see >> your proposal along with my own as very compatible. Would love to work >> together, and if this works as I hope it will be a huge driver of activity >> in the ecosystem. >> >> Dave >> >> On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong <[email protected]> wrote: >>> >>> I like the idea of having a single UI for installing frameworks. One Web >>> UI endpoint for managing setting and view state of multiple installed >>> frameworks. >>> >>> >>> On 2 Dec 2014, at 09:26, Billy Bones <[email protected]> wrote: >>> >>> Wooo mesosphere is making such a good work pushing forward Mesos >>> technologies! >>> >>> I deeply think that Mesos should implement registry as a core feature. >>> >>> Let me explain that, Installing manually frameworks / modules / other >>> third party adds-on is quite cool when you're beginner and want to learn how >>> the things works, but once you're in production, you've to be fast and react >>> quickly. >>> >>> Let said that I'm a system engineer that have customers requests like >>> "Hum... Could you had this " insert Framework/module/other name here" on our >>> cluster region?", I'll have to add it as soon and quick as possible on the >>> targeted environment, I'll be OK to do it twice, but I wont make my job a >>> request nightmare and I'll probably search for a way to provide to my >>> customers a restrictive access (WebUI) to the cluster manager and allow them >>> to deploy those new add-on by themself. >>> >>> With this study case in mind, I'll be happy to have a registry which >>> allow a filtering mechanism to hide or show frameworks/modules/etc by status >>> (Official Apache Content / Public Content / Private Content). >>> >>> Here is my thoughts about the registry articulation: >>> >>> Mesos Registry is an integrated part of Mesos master services. >>> Mesos Registry is an endpoint available to WebUI and CLI. >>> Mesos Registry is nothing but a metadata registry. >>> Mesos Registry save its configs and metadata on a key/value store >>> (zookeeper?). >>> Mesos Registry is empty at the first launch. >>> Mesos Registry as three views: Official Apache Content | Publicly >>> Maintained Content | Private Maintained Content >>> Mesos Registry views content is: >>> >>> Official Apache Content == Link to existing Apache add-ons hosted on >>> Github/Git repository + metadata (Like those proposed by Dave and Connor). >>> Publicly Maintained Content == Links to existing repositories (Github / >>> Git / other) + metadata (Like those proposed by Dave and Connor). >>> Private Content == Links to existing private (GIT/GITHUB/Other >>> repositories) + metadata (Like those proposed by Dave and Connor), this >>> repository is kinda special as it is hosted and created by the cluster >>> operators and could be a mixed content of locally maintained repository (GIT >>> repos on a HDFS or TAR on HDFS) and public content repository cloned from >>> the public content URL/Metadata. >>> >>> I don't know if I'm really clear, so if I'm not, let me know it, I'll do >>> some sketches :D >>> >>> >>> >>> 2014-12-02 1:53 GMT+01:00 Connor Doyle <[email protected]>: >>>> >>>> Hi Dave, >>>> >>>> This is a timely topic, since we have been prototyping and mocking up >>>> something similar at Mesosphere. We created a new public GitHub repository >>>> for it about three weeks ago called "universe" >>>> (http://github.com/mesosphere/universe). >>>> >>>> Although we have added some informal specs, it's very malleable at this >>>> point. We're very much interested in making our "universe" compatible >>>> with, >>>> or the same as, the registry you're proposing. Without delving into >>>> implementation details, some of the goals we have in mind are outlined >>>> below. >>>> >>>> Data Source: >>>> >>>> The package repository should be easily consumable by third-party >>>> command-line and other programs. There should be a condensed “index” >>>> representation of the package repository available. >>>> >>>> Packages within the repository should be versioned. >>>> >>>> The package repository format itself should be versioned. >>>> >>>> Decentralization and Composability: >>>> >>>> The package metadata should be hosted in a public place (we like GitHub) >>>> so that additional packages can be added by the community by simply >>>> submitting pull requests. We have added some rudimentary commit hooks and >>>> automated validation to protect the repo against breaking changes. >>>> >>>> It’s important that no single entity “owns the keys” to the universe, >>>> and that the spec and implementation remain public. It should be easy and >>>> free for organizations to maintain a private package repository. >>>> >>>> A corollary is that it should be easy for consumers to pull from a >>>> hierarchy of upstream repositories. One setup we have in mind is that an >>>> organization might have staging and production repositories running >>>> internally. Packages are pushed to staging where integration testing can >>>> run before “deployment” to production. If a package isn’t in the local >>>> repository it might be looked up and installed from upstream. >>>> >>>> <hierarchy.png> >>>> >>>> >>>> Repositories should be able to be proxied and cached in this way. >>>> Organizations should be able to isolate their datacenter but also >>>> selectively add external packages for experimentation. The system should be >>>> sufficiently portable and extensible to accomodate these and similar use >>>> cases. >>>> >>>> Meta-Framework Descriptors: >>>> >>>> Our conception of the package repository is a bit more expansive than >>>> just Mesos frameworks; it includes descriptions of how to install any piece >>>> of server software on a Mesos cluster. Frameworks and non-frameworks alike >>>> may be installed using some other meta-framework that’s responsible for >>>> starting all other cluster services. Likely candidates for this role are >>>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually >>>> Kubernetes. In any case, the repository spec should not be prescriptive >>>> with respect to this choice. >>>> >>>> The package repository metadata should make it easy for Mesos framework >>>> authors (and authors of non-Mesos-aware programs) to describe how to >>>> install >>>> their software on a Mesos cluster. To this end, our prototype package spec >>>> allows for Meta-framework descriptor files for each package in the >>>> repository. For example for a given package we might see a `marathon.json` >>>> file as well as a `my-app.aurora` file. >>>> >>>> An obvious concern is how to specify site-specific arguments upon >>>> installation. Here packages should describe data that must be marshalled >>>> from the environment (e.g. by prompting a user) and combined with the raw >>>> meta-framework descriptor to launch the app. These configuration >>>> parameters >>>> should be agnostic of the supported meta-frameworks. More concretely, in >>>> our prototype we describe configuration data in terms of a JSON-Schema. >>>> >>>> CLI Integration: >>>> >>>> Part of our proposed package format is an optional descriptor for how to >>>> fetch and install the command-line tools for interacting with the >>>> application. For now, we only have one implementation of this, which is to >>>> fetch a python egg from PyPI. >>>> >>>> Governance: >>>> >>>> All in all, we think that making this effort more community driven is a >>>> healthy way to proceed. Any input is very welcome. For example, if others >>>> think that what we have is a good starting point we could transfer >>>> ownership >>>> of the repository to the mesos organization on GitHub. >>>> >>>> Cheers, >>>> -- >>>> Connor Doyle >>>> http://mesosphere.com >>>> >>>> >>>> >>>> >>>> >>>> On Nov 30, 2014, at 17:32, Dave Lester <[email protected]> wrote: >>>> >>>> As the number of Mesos frameworks grows (and now, a module system), I >>>> think it's time to create a community-maintained registry with the goal of >>>> making frameworks and modules easier to discover, contribute to, and >>>> install. >>>> >>>> There's already a JIRA ticket tracking this (MESOS-1759) and I've >>>> chatted with several folks (thanks in particular Victor Vieux, Tom Arnfeld, >>>> Vinod Kone, Timothy St Clair, and Joe Stein). I'd like to advance the >>>> conversation by offering a proposal on the public mailing list. >>>> >>>> I imagine two initiatives to achieve this: >>>> >>>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with >>>> how Jenkins maintains their plugins on GitHub [1], but they allow >>>> individual >>>> plugins to have their own repo within their GH org. Plugins are repos with >>>> a >>>> specific naming convention (in their case, they've appended "-plugin" to >>>> the >>>> repo name but this isn't the same approach we'd need to take). Having a >>>> shared GH org creates greater visibility to what people are doing, and >>>> encourages community participation and ownership. >>>> >>>> In the case of Mesos, not all frameworks will jump at the chance to have >>>> community hosting which is fine. But particularly for smaller frameworks, I >>>> think this is a good option given the success Jenkins has had. We could >>>> potentially use the existing /mesos github org, or create a new org >>>> /mesos-extensions or something of the like. It seems like the only >>>> potential >>>> snag here is that we'll want to be explicit that these aren't >>>> Apache-blessed >>>> plugins and instead maintained by the larger ecosystem, but I believe we >>>> can >>>> achieve that by offering a notice in the GH org description. >>>> >>>> 2) A registry that allows framework writers to publish new versions of >>>> their frameworks to a central repository that could be programmatically >>>> accessed and rendered online. The v1 could be incredibly simple, but I >>>> think >>>> this is a foundational piece that can grow with the project in the future. >>>> Since this is a little bit more-involved, I've created a separate Google >>>> Doc >>>> [2] to begin drafting the scope for this: >>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing, >>>> and have intentionally punted on some of the implementation details until >>>> the scope is finalized and I gauge interest from users and potential >>>> collaborators. >>>> >>>> What do you think? If folks are on board I will assign the JIRA issue to >>>> myself and get to work; I'd also welcome collaborators to help get this off >>>> the ground since I think it will be a boost for the community. Feedback is >>>> welcome! >>>> >>>> Thanks, >>>> Dave >>>> >>>> [1] https://github.com/jenkinsci/ >>>> [2] >>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing >>>> >>>> >>> >> > -- connor
