Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
-Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: 26 September 2014 22:51 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model Heh, I just got off the phone with Monty talking about this :) Comments inline... On 09/22/2014 03:11 PM, Tim Bell wrote: The quality designation is really important for the operator community who are trying to work out what we can give to our end users. So, I think it's important to point out here that there are three different kinds of operators/deployers: * Ones who use a distribution of OpenStack (RDO, UCA, MOS, Nebula, Piston, etc) * Ones who use Triple-O * Ones who go it alone and install (via source, a mixture of source and packages, via config management like Chef or Puppet, etc) In reality, you are referring to the last group, since operators in the first group are saying we are relying on a distribution to make informed choices about what is ready for prime time because we tested these things together. Operators in the second group are really only HP right now, AFAICT, and Triple- O's opinion on the production readiness of the things it deploys in the undercloud are roughly equal to all of the integrated release that the TC defines. I personally think that if an operator is choosing to be in the third group, then they are taking on the responsibility of testing what they deploy in a staging/test environment in order to validate that it meets their own requirements. In other words, the onus of determining whether something is production ready is on their own shoulders. If the operator chose to deploy OpenStack on top of a vanilla/custom Linux distribution instead of Red Hat, Ubuntu, OpenSUSE, or whatever, we would not complain to the Linux kernel community that they have not made quality designations available for all the pieces we decided to put in our custom Linux distribution. Likewise, I think that is the role of OpenStack distributions: the choices and options that the distributor makes are a result of testing certain things together and the distributor's opinion of quality. If a deployer of OpenStack chooses to go it alone and not use an OpenStack distribution, that's totally cool, but I don't believe it should be the OpenStack developer community's responsibility to vouch for the production readiness of each component of OpenStack. Now, Monty has a big problem with this idea. I know, because he just told me he does :) Monty thinks this attitude of relying on OpenStack distributions to make choices about production quality of OpenStack components leads inevitably to open core opportunities, with some companies choosing to label a few upstream components as production quality and others (the ones the company developed internally) as their production quality choices. I happen to doubt that relying on OpenStack distributions to make these choices will lead to open core stuff. I think there is a hybrid model in addition to your 3 listed of those who take a distribution for the core functions and are then cherry picking the other components according to the cloud they wish to offer. Thus, a distribution would define a base code set complying with the standard APIs and then deployers can select from a stackforge like repository for additional function not in the distribution. An example would be if you used RDO to deploy your cloud and then the Murano package from stackforge to give additional function not in the distribution. Puppetlabs recently announced 'approved' modules (http://www.infoq.com/news/2014/09/puppet-approved-modules) which are intended to help select modules which are not part of the core set but have had a good track record in production. Clearly, there is a difference in governance between the projects which would need a different mechanism for gathering the overall state such as some crowd-sourcing approach like Drupal does. Tim Best, -jay Offering early helps to establish the real-life experience and give good feedback on the designs. However, the operator then risks leaving their users orphaned if the project does not get a sustainable following or significant disruption if the APIs change. The packaging teams are key here as well. When do Ubuntu and Red Hat work out the chain of pre-reqs etc. to produce installable packages, packstack/juju tool support ? We do need to have some way to show that an layer #2 package is ready for prime time production and associated criteria (packages available, docs available, 1 company communities, models for HA and scale, ...) Tim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/26/2014 03:42 AM, Vishvananda Ishaya wrote: On Sep 25, 2014, at 4:01 PM, Robert Collins robe...@robertcollins.net wrote: So I guess I'm saying: Lets decouple 'what is openstack' from 'what we test together on every commit'. It seems that this discussion has actually illustrated shortcomings in our answers to 3 separate questions, and people have been throwing out ideas that attempt to solve all 3. Perhaps we need to address each one individually. The three questions are: 1. Which projects are “part of openstack”? 2. Which projects are released as a single unit? 3. Which projects are tested together The current answers are: 1. Three levels incubation, integration, core 2. Things that reach the integration level 3. Things that reach the integration level. Some proposed answers: 1. Lightweight incubation a la apache 2. Monty’s layer1 3. Direct dependencies and close collaborators Discussing the propased answers(in reverse order): I think we have rough consensus around 3: that we should move towards functional testing for direct dependencies and let the projects decide when they want to co-gate. The functional co-gating should ideally be based on important use-cases. 2 is a bit murkier. In the interest of staying true to our roots the best we can probably do is to allow projects to opt-out of the coordinated release and for thierry to specifically select which projects he is willing to coordinate. Any other project could co-release with the integrated release but wouldn’t be centrally managed by thierry. There is also a decision about what the TCs role is in these projects. Assuming there would be enough volunteers, would it make sense to have a release manager per layer? And have those release managers working together on an integrated release? or even have someone in each project working on the release with the release manager *but* still have an integrated release? I'm afraid that not having an integrated release will end up in a whole mess. When should packages be released? How will stable maintenance work? When will distros build the packages? 1 Has some unanswerd questions, like is there another level “graduation” where the tc has some kind of technical oversight? What is the criteria for it? etc. As mentioned in other emails[0], I think we should reduce this levels to the minimum and work on a guided grow path for projects that will take them to the point where they'll be considered production-ready/stable/you-name-it. Moreover, this process should encourage collaboration (certainly not as much as we've pushed it so far) over competition but still welcome the later. [0] http://lists.openstack.org/pipermail/openstack-dev/2014-September/047114.html Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Vishvananda Ishaya wrote: The three questions are: 1. Which projects are “part of openstack”? 2. Which projects are released as a single unit? 3. Which projects are tested together That's a good summary, yes. Currently we have a number of horizontal teams, which must support equally an always-increasing number of integrated projects. That means 1=2=3. But in the case of (3) it just might not make sense, and in the case of (2) it doesn't scale infinitely. That said, singling out the test infrastructure (3) and the release management (2) is a bit unfair to other horizontal efforts, like Documentation, Translations, or general QA, which also suffer from a scale issue. The Docs team, in particular, couldn't really scale either and already implemented two tiers within the integrated release -- the part they directly support, and the part they help / provide tools for. The current answers are: 1. Three levels incubation, integration, core 2. Things that reach the integration level 3. Things that reach the integration level. Some proposed answers: 1. Lightweight incubation a la apache 2. Monty’s layer1 3. Direct dependencies and close collaborators Discussing the propased answers(in reverse order): I think we have rough consensus around 3: that we should move towards functional testing for direct dependencies and let the projects decide when they want to co-gate. The functional co-gating should ideally be based on important use-cases. 2 is a bit murkier. In the interest of staying true to our roots the best we can probably do is to allow projects to opt-out of the coordinated release and for thierry to specifically select which projects he is willing to coordinate. Any other project could co-release with the integrated release but wouldn’t be centrally managed by thierry. There is also a decision about what the TCs role is in these projects. Like I said above, that seems to single out release management, while other horizontal efforts are facing the same challenge. I fear if we let each horizontal program choose which set of projects it supports, it will result in a bit of a mess, where no expectation is clearly set (project A is on the common release but Anne doesn't touch its docs directly, project B releases when it wants to and has Anne's team working on it directly...) If the horizontal teams all agree to directly support the same non-expanding set (for the sake of the exercise, let's call it layer #1), why not have it ? It's definitely a set that test infra, QA, Release Management and Docs would feel comfortable supporting, AFAIK. All the other projects would be able to get help and tools from those horizontal teams, but would just not be directly taken care of. This is how Docs currently work (the two tiers within integrated release), this is how Release management currently works (integrated vs. incubated), this is how the test infra would like to be able to work... Making sure we at least support the layer #1 is just a matter of setting the same basic expectations for the same basic set of projects. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/25/2014 08:42 PM, Vishvananda Ishaya wrote: It seems that this discussion has actually illustrated shortcomings in our answers to 3 separate questions, and people have been throwing out ideas that attempt to solve all 3. Perhaps we need to address each one individually. The three questions are: 1. Which projects are “part of openstack”? 2. Which projects are released as a single unit? 3. Which projects are tested together The current answers are: 1. Three levels incubation, integration, core 2. Things that reach the integration level 3. Things that reach the integration level. There is a fourth question, although it's somewhat related to the first: do we continue the concept of 'incubated' and 'integrated'? Do these processes and designations improve OpenStack, or do they impede progress? Does designating a development effort as in the club discourage other, potentially better solutions? Or would keeping things wide open simply dilute the efforts of people working on these solutions? Assuming that we can achieve consensus as to what constitutes Ring0/Layer1/core, the question then becomes how does everything else get handled? IOW, are they all treated as independent additions? Or do some receive a special designation (e.g., certified good enough for the Cern LHC), and others are simply your results may vary? The issues with gating everything together aren't the cause for making changes to how OpenStack is organized; they are the symptom that we can't continue along the current path indefinitely. Breaking out the testing of the core from everything else is a good start, and will help alleviate many of the gating issues, but I think it's also important to realize that the growing pains are not limited to testing. - -- Ed Leafe -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) iQIcBAEBAgAGBQJUJY6MAAoJEKMgtcocwZqLuxwP/1QSoHhhegZETtxVyrxBisOs sjJfh+wzLqjfEqkwUOZ0SX2aEFHuimNE5lWmPPB7hzffc6Z4aNxITjP/fCry77Ax IQsGTbTGG0gsXiyO/+/x6c6Yfkcmu6fui8L/hde0u7klN/8tATkp/5xwW4qvGXY7 1WJwBsHDx8MniqoJ2echP5uUytFMxGgK9tHA0bnWUEBNVCVt8tmczNTrRJzHY49P ZbdSDvwypvClfSLgu1/z8yb0Wv54dy0eafi6Gf+kYL0e3i5//Pv3YkSKrHTqm3Ww R1V6Acr/BOT6MC8Ln1TF2gNgSo8iCAOfIakkwmCLJCCuvOmKA3hbch16Ggqhm1Ua mDyuUcOICKs+NX07cIKnVkI5WpIGqCQ0M/HS1jiix3tOPHe3sie85Jqser9PGnis Hwtva100HqnBARp3vips7inXughfiiQ3UHW41l4aLuSf0faba1v2CiFrQfE6Nvm3 ownAMdp3qCtBgXkl7KcgFyyWeIZVUEDbxdMw+A5GR3PIsUDD07a2YB/mY03pVHc2 lacZFlw2aFoeVjnRhU988h+LnQvwM4+4mqpcnJvDsVGalV5CALI/I0CzJMjKLfZV t02jtLZz7CcUDPxGwmqF82rn+BYzJX/glpMYYHVQVzKL7MJr/H5eg8M1oi0huNoA lSfQ2U3/c4A1eURsLybm =qAHC -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 26, 2014, at 1:25 AM, Thierry Carrez thie...@openstack.org wrote: That said, singling out the test infrastructure (3) and the release management (2) is a bit unfair to other horizontal efforts, like Documentation, Translations, or general QA, which also suffer from a scale issue. The Docs team, in particular, couldn't really scale either and already implemented two tiers within the integrated release -- the part they directly support, and the part they help / provide tools for. You are correct I left out some very important cross project teams (Sorry Anne!). We call the projects that tap these shared resources today “integrated”. It seems like if we keep going down this path, we need to either: a) make “integrated” == layer1 This would be a tricky proposition because of the way it might look in the community, but it may be necessary if we are already too overloaded to handle the cross-project concerns at our current scale. b) clearly dilineate “integrated” vs layer1 in some other way This would likely involve changing the meaning of integrated to mean a bit less than it does today: just best effort for projects outside of layer1. All the other projects would be able to get help and tools from those horizontal teams, but would just not be directly taken care of. This is how Docs currently work (the two tiers within integrated release), this is how Release management currently works (integrated vs. incubated), this is how the test infra would like to be able to work... Making sure we at least support the layer #1 is just a matter of setting the same basic expectations for the same basic set of projects. It sounds like you are suggesting that b) has already occurred to some degree so that we should just continue along those lines? Vish signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/18/2014 02:53 PM, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Enjoy. I enjoyed your post (though I don't agree with everything, of course) :) I typed up some of my thoughts here about these questions we're struggling with and some proposals on possible ways forward. http://www.joinfu.com/2014/09/answering-the-existential-question-in-openstack/ Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Heh, I just got off the phone with Monty talking about this :) Comments inline... On 09/22/2014 03:11 PM, Tim Bell wrote: The quality designation is really important for the operator community who are trying to work out what we can give to our end users. So, I think it's important to point out here that there are three different kinds of operators/deployers: * Ones who use a distribution of OpenStack (RDO, UCA, MOS, Nebula, Piston, etc) * Ones who use Triple-O * Ones who go it alone and install (via source, a mixture of source and packages, via config management like Chef or Puppet, etc) In reality, you are referring to the last group, since operators in the first group are saying we are relying on a distribution to make informed choices about what is ready for prime time because we tested these things together. Operators in the second group are really only HP right now, AFAICT, and Triple-O's opinion on the production readiness of the things it deploys in the undercloud are roughly equal to all of the integrated release that the TC defines. I personally think that if an operator is choosing to be in the third group, then they are taking on the responsibility of testing what they deploy in a staging/test environment in order to validate that it meets their own requirements. In other words, the onus of determining whether something is production ready is on their own shoulders. If the operator chose to deploy OpenStack on top of a vanilla/custom Linux distribution instead of Red Hat, Ubuntu, OpenSUSE, or whatever, we would not complain to the Linux kernel community that they have not made quality designations available for all the pieces we decided to put in our custom Linux distribution. Likewise, I think that is the role of OpenStack distributions: the choices and options that the distributor makes are a result of testing certain things together and the distributor's opinion of quality. If a deployer of OpenStack chooses to go it alone and not use an OpenStack distribution, that's totally cool, but I don't believe it should be the OpenStack developer community's responsibility to vouch for the production readiness of each component of OpenStack. Now, Monty has a big problem with this idea. I know, because he just told me he does :) Monty thinks this attitude of relying on OpenStack distributions to make choices about production quality of OpenStack components leads inevitably to open core opportunities, with some companies choosing to label a few upstream components as production quality and others (the ones the company developed internally) as their production quality choices. I happen to doubt that relying on OpenStack distributions to make these choices will lead to open core stuff. Best, -jay Offering early helps to establish the real-life experience and give good feedback on the designs. However, the operator then risks leaving their users orphaned if the project does not get a sustainable following or significant disruption if the APIs change. The packaging teams are key here as well. When do Ubuntu and Red Hat work out the chain of pre-reqs etc. to produce installable packages, packstack/juju tool support ? We do need to have some way to show that an layer #2 package is ready for prime time production and associated criteria (packages available, docs available, 1 company communities, models for HA and scale, …) Tim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Fri, Sep 26, 2014 at 4:50 PM, Jay Pipes jaypi...@gmail.com wrote: Heh, I just got off the phone with Monty talking about this :) Comments inline... On 09/22/2014 03:11 PM, Tim Bell wrote: The quality designation is really important for the operator community who are trying to work out what we can give to our end users. So, I think it's important to point out here that there are three different kinds of operators/deployers: * Ones who use a distribution of OpenStack (RDO, UCA, MOS, Nebula, Piston, etc) * Ones who use Triple-O * Ones who go it alone and install (via source, a mixture of source and packages, via config management like Chef or Puppet, etc) I'm not sure TripleO fits in this list. It is not just a collection of prescriptive OpenStack bits used to do a deployment. TripleO is tooling to build OpenStack to deploy OpenStack. You can use whatever source (packages, distribution, released tarballs, straight from git) you want to build that OpenStack. TripleO could deploy your first or third bullet item. In reality, you are referring to the last group, since operators in the first group are saying we are relying on a distribution to make informed choices about what is ready for prime time because we tested these things together. Operators in the second group are really only HP right now, AFAICT, and Triple-O's opinion on the production readiness of the things it deploys in the undercloud are roughly equal to all of the integrated release that the TC defines. FWIW, TripleO offers deploying using distributions, by installing from packages from the RDO repositories. There's nothing RDO specific about it though, any packaged OpenStack distribution could be installed with the TripleO tooling. RDO is just likely the most well tested. Even when not installing via a distribution, and either directly from trunk or the integrated release tarballs, I don't know that any TripleO opinion enters into it. TripleO uses the integrated projects of OpenStack to deploy an overcloud. In an overcloud, you may see support for some incubated projects, depending on if there's interest from the community for that support. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Hi James, thanks for the corrections/explanations. A comment inline (and a further question) :) On 09/26/2014 05:35 PM, James Slagle wrote: On Fri, Sep 26, 2014 at 4:50 PM, Jay Pipes jaypi...@gmail.com wrote: Heh, I just got off the phone with Monty talking about this :) Comments inline... On 09/22/2014 03:11 PM, Tim Bell wrote: The quality designation is really important for the operator community who are trying to work out what we can give to our end users. So, I think it's important to point out here that there are three different kinds of operators/deployers: * Ones who use a distribution of OpenStack (RDO, UCA, MOS, Nebula, Piston, etc) * Ones who use Triple-O * Ones who go it alone and install (via source, a mixture of source and packages, via config management like Chef or Puppet, etc) I'm not sure TripleO fits in this list. It is not just a collection of prescriptive OpenStack bits used to do a deployment. TripleO is tooling to build OpenStack to deploy OpenStack. You can use whatever source (packages, distribution, released tarballs, straight from git) you want to build that OpenStack. TripleO could deploy your first or third bullet item. OK, fair point, thanks for that added bit of description. In reality, you are referring to the last group, since operators in the first group are saying we are relying on a distribution to make informed choices about what is ready for prime time because we tested these things together. Operators in the second group are really only HP right now, AFAICT, and Triple-O's opinion on the production readiness of the things it deploys in the undercloud are roughly equal to all of the integrated release that the TC defines. FWIW, TripleO offers deploying using distributions, by installing from packages from the RDO repositories. There's nothing RDO specific about it though, any packaged OpenStack distribution could be installed with the TripleO tooling. RDO is just likely the most well tested. Oh, good to know. Sorry, my information about Triple-O's undercloud setup is clearly outdated. I thought that the undercloud was build from source repositories devstack-style. Thanks for hitting me with a cluestick. :) Even when not installing via a distribution, and either directly from trunk or the integrated release tarballs, I don't know that any TripleO opinion enters into it. TripleO uses the integrated projects of OpenStack to deploy an overcloud. In an overcloud, you may see support for some incubated projects, depending on if there's interest from the community for that support. OK, interesting. So, in summary, Triple-O really doesn't offer much of a this is production-ready stamp to anything based on whether it deploys a project or not. So, operators who deploy with Triple-O would be in the you're on your own camp from the bulleted list above. Would that be a fair statement? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 27 September 2014 09:43, Jay Pipes jaypi...@gmail.com wrote: Hi James, thanks for the corrections/explanations. A comment inline (and a further question) :) Oh, good to know. Sorry, my information about Triple-O's undercloud setup is clearly outdated. I thought that the undercloud was build from source repositories devstack-style. Thanks for hitting me with a cluestick. :) Thats one installation strategy :). Even when not installing via a distribution, and either directly from trunk or the integrated release tarballs, I don't know that any TripleO opinion enters into it. TripleO uses the integrated projects of OpenStack to deploy an overcloud. In an overcloud, you may see support for some incubated projects, depending on if there's interest from the community for that support. OK, interesting. So, in summary, Triple-O really doesn't offer much of a this is production-ready stamp to anything based on whether it deploys a project or not. So, operators who deploy with Triple-O would be in the you're on your own camp from the bulleted list above. Would that be a fair statement? TripleO upstream doesn't offer a production-ready stamp for the workload clouds; for deploy clouds we do - simply because you can't use non-production-ready services to do your deploy... some of our stamps have substantial caveats today (e.g. Heat) - but they are being worked on. But then Nova upstream doesn't offer production-ready stamps either. Are cells production ready? Instance groups? Or generally any new feature? *distributions* of TripleO offer production ready stamps. RDO offers one HP Helion offers one. In exactly the same way that distributions offer production stamps about Nova, distributions that use TripleO offer production stamps about Nova :). And I think this is James's point. Your category 2 above saying that TripleO is different is just confused: TripleO is a deployment architecture [evolving into a set of such], *not* a distribution channel. 1 and 3 are distribution channels. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Excerpts from Jay Pipes's message of 2014-09-26 14:43:40 -0700: Hi James, thanks for the corrections/explanations. A comment inline (and a further question) :) On 09/26/2014 05:35 PM, James Slagle wrote: On Fri, Sep 26, 2014 at 4:50 PM, Jay Pipes jaypi...@gmail.com wrote: Heh, I just got off the phone with Monty talking about this :) Comments inline... On 09/22/2014 03:11 PM, Tim Bell wrote: The quality designation is really important for the operator community who are trying to work out what we can give to our end users. So, I think it's important to point out here that there are three different kinds of operators/deployers: * Ones who use a distribution of OpenStack (RDO, UCA, MOS, Nebula, Piston, etc) * Ones who use Triple-O * Ones who go it alone and install (via source, a mixture of source and packages, via config management like Chef or Puppet, etc) I'm not sure TripleO fits in this list. It is not just a collection of prescriptive OpenStack bits used to do a deployment. TripleO is tooling to build OpenStack to deploy OpenStack. You can use whatever source (packages, distribution, released tarballs, straight from git) you want to build that OpenStack. TripleO could deploy your first or third bullet item. OK, fair point, thanks for that added bit of description. In reality, you are referring to the last group, since operators in the first group are saying we are relying on a distribution to make informed choices about what is ready for prime time because we tested these things together. Operators in the second group are really only HP right now, AFAICT, and Triple-O's opinion on the production readiness of the things it deploys in the undercloud are roughly equal to all of the integrated release that the TC defines. FWIW, TripleO offers deploying using distributions, by installing from packages from the RDO repositories. There's nothing RDO specific about it though, any packaged OpenStack distribution could be installed with the TripleO tooling. RDO is just likely the most well tested. Oh, good to know. Sorry, my information about Triple-O's undercloud setup is clearly outdated. I thought that the undercloud was build from source repositories devstack-style. Thanks for hitting me with a cluestick. :) Even when not installing via a distribution, and either directly from trunk or the integrated release tarballs, I don't know that any TripleO opinion enters into it. TripleO uses the integrated projects of OpenStack to deploy an overcloud. In an overcloud, you may see support for some incubated projects, depending on if there's interest from the community for that support. OK, interesting. So, in summary, Triple-O really doesn't offer much of a this is production-ready stamp to anything based on whether it deploys a project or not. So, operators who deploy with Triple-O would be in the you're on your own camp from the bulleted list above. Would that be a fair statement? The one thing that builds an overcloud in a prescriptive manner is called 'tripleo-incubator' precisely because we don't believe it is ready to be called a release. It does focus on testing the integrated release only though, and so is closer to your original #2. It's just important to note that this is just our own distribution, and does not fully represent the whole output of the program. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Robert Collins on Friday, September 26, 2014 3:33 PM wrote: On 27 September 2014 09:43, Jay Pipes jaypi...@gmail.com wrote: Hi James, thanks for the corrections/explanations. A comment inline (and a further question) :) Oh, good to know. Sorry, my information about Triple-O's undercloud setup is clearly outdated. I thought that the undercloud was build from source repositories devstack-style. Thanks for hitting me with a cluestick. :) Thats one installation strategy :). Even when not installing via a distribution, and either directly from trunk or the integrated release tarballs, I don't know that any TripleO opinion enters into it. TripleO uses the integrated projects of OpenStack to deploy an overcloud. In an overcloud, you may see support for some incubated projects, depending on if there's interest from the community for that support. OK, interesting. So, in summary, Triple-O really doesn't offer much of a this is production-ready stamp to anything based on whether it deploys a project or not. So, operators who deploy with Triple-O would be in the you're on your own camp from the bulleted list above. Would that be a fair statement? TripleO upstream doesn't offer a production-ready stamp for the workload clouds; for deploy clouds we do - simply because you can't use non-production-ready services to do your deploy... some of our stamps have substantial caveats today (e.g. Heat) - but they are being worked on. But then Nova upstream doesn't offer production-ready stamps either. Are cells production ready? Instance groups? Or generally any new feature? *distributions* of TripleO offer production ready stamps. RDO offers one HP Helion offers one. In exactly the same way that distributions offer production stamps about Nova, distributions that use TripleO offer production stamps about Nova :). And I think this is James's point. Your category 2 above saying that TripleO is different is just confused: TripleO is a deployment architecture [evolving into a set of such], *not* a distribution channel. 1 and 3 are distribution channels. [Rocky] I'd like to make a couple of points; first is how many commercial deployers/operators would consider any of OpenStack production ready and if they do, what would that subset actually be? ( a little snarky, but not really) Second, I'd like to point out that Defcore is attempting to provide guidance along these lines, but may be considered a bit more strict than a Production Ready label. Then again, it may be less strict, depending on test coverage;-) Check out the scoring criteria here: https://wiki.openstack.org/wiki/Governance/CoreCriteria In principle, OpenStack functionality has to have been production tested with a fairly large distribution base, along with a number of other criteria. As defined, it would definitely be a subset of production ready but it would be a reasonably safe subset that could then be built upon. Beyond the DefCore criteria, RefStack is heading towards the point where any operator could run all of Tempest or any selection of Tempest tests against his/her installed cloud and view all the results. They could even save them to do trend analysis. But, this still begs the question of what is production ready especially for Open Source code. Perhaps we need the release notes to be very specific on which APIs, features and/or projects are new/radically altered in a specific release. This could allow operators to install Juno release/Icehouse functionality which would theoretically be much better than Icehouse Release which is what a lot of shops would do, reasoning that Icehouse has been out six months, so the gotchas are known and patches have been released to fix the worst issues. Whereas, I think that most people in QA and/or deep in the various projects would say that the functionality that was released in Icehouse that also is in Juno would be more performant and less buggy than the Icehouse release. So really, the question might really be: how do you get deployers who want greater stability to actually get it by deploying the current release with past release's functionality subset? And how do you communicate that some of the past release's functionality is still not production ready? Hard problem. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/24/2014 07:55 PM, Zane Bitter wrote: On 18/09/14 14:53, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Thanks Monty, I think there are some very interesting ideas in here. I'm particularly glad to see the 'big tent' camp reasserting itself, because I have no sympathy with anyone who wants to join the OpenStack community and then bolt the door behind them. Anyone who contributes to a project that is related to OpenStack's goals, is willing to do things the OpenStack way, and submits itself to the scrutiny of the TC deserves to be treated as a member of our community with voting rights, entry to the Design Summit and so on. I'm curious how you're suggesting we decide which projects satisfy those criteria though. Up until now, we've done it through the incubation process (or technically, the new program approval process... but in practice we've never added a project that was targeted for eventual inclusion in the integrated release to a program without incubating it). Would the TC continue to judge whether a project is doing things the OpenStack way prior to inclusion, or would we let projects self-certify? What does it mean for a project to submit itself to TC scrutiny if it knows that realistically the TC will never have time to actually scrutinise it? Or are you not suggesting a change to the current incubation process, just a willingness to incubate multiple projects in the same problem space? I feel like I need to play devil's advocate here, because overall I'm just not sure I understand the purpose of arbitrarily - and it *is* arbitrary - declaring Layer #1 to be anything required to run Wordpress. To anyone whose goal is not to run Wordpress, how is that relevant? Speaking of arbitrary, I had to laugh a little at this bit: Also, please someone notice that the above is too many steps and should be: openstack boot gentoo on-a 2G-VM with-a publicIP with-a 10G-volume call-it blog.inaugust.com That's kinda sorta exactly what Heat does ;) Minus the part about assuming there is only one kind of application, obviously. I think there are a number of unjustified assumptions behind this arrangement of things. I'm going to list some here, but I don't want anyone to interpret this as a personal criticism of Monty. The point is that we all suffer from biases - not for any questionable reasons but purely as a result of our own experiences, who we spend our time talking to and what we spend our time thinking about - and therefore we should all be extremely circumspect about trying to bake our own mental models of what OpenStack should be into the organisational structure of the project itself. * Assumption #1: The purpose of OpenStack is to provide a Compute cloud This assumption is front-and-centre throughout everything Monty wrote. Yet this wasn't how the OpenStack project started. In fact there are now at least three services - Swift, Nova, Zaqar - that could each make sense as the core of a standalone product. Yes, it's true that Nova effectively depends on Glance and Neutron (and everything depends on Keystone). We should definitely document that somewhere. But why does it make Nova special? * Assumption #2: Yawnoc's Law Don't bother Googling that, I just made it up. It's the reverse of Conway's Law: Infra engineers who design governance structures for OpenStack are constrained to produce designs that are copies of the structure of Tempest. I just don't understand why that needs to be the case. Currently, for understandable historic reasons, every project gates against every other project. That makes no sense any more, completely independently of the project governance structure. We should just change it! There is no organisational obstacle to changing how gating works. Even this proposal doesn't entirely make sense on this front - e.g. Designate requires only Neutron and Keystone... why should Nova, Glance and every other project in Layer 1 gate against it, and vice-versa? I suggested in another thread[1] a model where each project would publish a set of tests, each project would decide which sets of tests to pull in and gate on, and Tempest would just be a shell for setting up the environment and running the selected tests. Maybe that idea is crazy or at least needs more work (it certainly met with only crickets and tumbleweeds on the mailing list), but implementing it wouldn't require TC intervention and certainly not by-laws changes. It just requires... implementing it. Perhaps the idea here is that by designating Layer 1 the TC is indicating to projects which other projects they should accept gate test jobs from (a function previously fulfilled by Incubation). I'd argue that this is a very bad way to do it, because
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 24, 2014, at 10:55 AM, Zane Bitter zbit...@redhat.com wrote: On 18/09/14 14:53, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 I think there are a number of unjustified assumptions behind this arrangement of things. I'm going to list some here, but I don't want anyone to interpret this as a personal criticism of Monty. The point is that we all suffer from biases - not for any questionable reasons but purely as a result of our own experiences, who we spend our time talking to and what we spend our time thinking about - and therefore we should all be extremely circumspect about trying to bake our own mental models of what OpenStack should be into the organisational structure of the project itself. I think there were some assumptions that lead to the Layer1 model. Perhaps a little insight into the in-person debate[1] at OpenStack-SV might help explain where monty was coming from. The initial thought was a radical idea (pioneered by Jay) to completely dismantle the integrated release and have all projects release independently and functionally test against their real dependencies. This gained support from various people and I still think it is a great long-term goal. The worry that Monty (and others) had are two-fold: 1. When we had no co-gating in the past, we ended up with a lot of cross-project breakage. If we jump right into this we could end up in the wild west were different projects expect different keystone versions and there is no way to deploy a functional cloud. 2. We have set expectations in our community (and especially with distributions), that we release a set of things that all work together. It is not acceptable for us to just pull the rug out from under them. These concerns show that we must (in the short term) provide some kind of integrated testing and release. I see the layer1 model as a stepping stone towards the long term goal of having the projects release independently and depend on stable interfaces. We aren’t going to get there immediately, so having a smaller, integrated set of services representing our most common use case seems like a good first step. As our interfaces get more stable and our testing gets better it could move to a (once every X months) release that just packages the current version of the layer1 projects or even be completely managed by distributions. We need a way to move forward, but I’m hoping we can do it without a concept of “specialness” around layer1 projects. I actually see it as a limitation of these projects that we have to take this stepping stone and cannot disaggregate completely. Instead it should be seen as a necessary evil so that we don’t break our users. In addition, we should encourage other shared use cases in openstack both for testing (functional tests against groups of services) and for releases (shared releases of related projects). [1] Note this wasn’t a planned debate, but a spontaneous discussion that included (at various points) Monty Taylor, Jay Pipes, Joe Gordon, John Dickenson, Myself, and (undoubtedly) one or two people I”m forgetting. signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 25/09/14 15:12, Vishvananda Ishaya wrote: On Sep 24, 2014, at 10:55 AM, Zane Bitter zbit...@redhat.com wrote: On 18/09/14 14:53, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 I think there are a number of unjustified assumptions behind this arrangement of things. I'm going to list some here, but I don't want anyone to interpret this as a personal criticism of Monty. The point is that we all suffer from biases - not for any questionable reasons but purely as a result of our own experiences, who we spend our time talking to and what we spend our time thinking about - and therefore we should all be extremely circumspect about trying to bake our own mental models of what OpenStack should be into the organisational structure of the project itself. I think there were some assumptions that lead to the Layer1 model. Perhaps a little insight into the in-person debate[1] at OpenStack-SV might help explain where monty was coming from. Thanks Vish, that is indeed useful background. Apparently I need to get out more ;) The initial thought was a radical idea (pioneered by Jay) to completely dismantle the integrated release and have all projects release independently and functionally test against their real dependencies. This gained support from various people and I still think it is a great long-term goal. So it goes without saying that I support the latter part (functionally test against their real dependencies). I'm not convinced by the idea of not having an integrated release though. Time-based releases seem to be pretty popular lately - the Linux kernel, most distributions and the two major open-source web browsers use them, for example - and as a developer I don't feel especially qualified to second-guess what the release schedule for a particular component should actually be. The current cycle works decently well, is not obviously worse than any particular alternative, and aligns semi-nicely with the design summits (FWIW I think I would actually prefer the design summit take place _before_ the release, but that is a whole other discussion). We actually discussed with Monty at this week's Heat meeting his proposal to move the UI projects to a continuous release schedule. (For the moment Heat is actually blocked on severe limitations in the usefulness of standalone mode, but we expect those to shake out over the near term anyway.) I think there was general agreement that there would be some big upsides - it really sucks telling the 15th user that we already redesigned that thing to solve your issue like 4 months ago, but since you're stuck on Icehouse we can't help. On the other hand, there's big downsides too. We're still making major changes, and it's really nice to be able to let them bed in as part of a release cycle. (Since I started this discussion talking about bias, it's worth calling out a *huge* one here: my team has to figure out a way to package and distribute this stuff.) That said, if we can get to a situation where we *choose* to do a co-ordinated release for the convenience of developers, distributors and (hopefully) users - rather than being forced into it through sheer terror that everything will fall over in a flaming heap if we don't - that would obviously be a win :) The worry that Monty (and others) had are two-fold: 1. When we had no co-gating in the past, we ended up with a lot of cross-project breakage. If we jump right into this we could end up in the wild west were different projects expect different keystone versions and there is no way to deploy a functional cloud. 2. We have set expectations in our community (and especially with distributions), that we release a set of things that all work together. It is not acceptable for us to just pull the rug out from under them. These concerns show that we must (in the short term) provide some kind of integrated testing and release. I see the layer1 model as a stepping stone towards the long term goal of having the projects release independently and depend on stable interfaces. We aren’t going to get there immediately, so having a smaller, integrated set of services representing our most common use case seems like a good first step. As our interfaces get more stable and our testing gets better it could move to a (once every X months) release that just packages the current version of the layer1 projects or even be completely managed by distributions. We need a way to move forward, but I’m hoping we can do it without a concept of “specialness” around layer1 projects. I actually see it as a limitation of these projects that we have to take this stepping stone and cannot disaggregate completely. Instead it should be seen as a necessary evil so that we don’t break our users. Right, if we _have_ to have it
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 26 September 2014 10:28, Zane Bitter zbit...@redhat.com wrote: So it goes without saying that I support the latter part (functionally test against their real dependencies). I'm not convinced by the idea of not having an integrated release though. Time-based releases seem to be pretty popular lately - the Linux kernel, most distributions and the two major open-source web browsers use them, for example - and as a developer I don't feel especially qualified to second-guess what the release schedule for a particular component should actually be. The current cycle works decently well, is not obviously worse than any particular alternative, and aligns semi-nicely with the design summits (FWIW I think I would actually prefer the design summit take place _before_ the release, but that is a whole other discussion). We actually discussed with Monty at this week's Heat meeting his proposal to move the UI projects to a continuous release schedule. (For the moment Heat is actually blocked on severe limitations in the usefulness of standalone mode, but we expect those to shake out over the near term anyway.) I think there was general agreement that there would be some big upsides - it really sucks telling the 15th user that we already redesigned that thing to solve your issue like 4 months ago, but since you're stuck on Icehouse we can't help. On the other hand, there's big downsides too. We're still making major changes, and it's really nice to be able to let them bed in as part of a release cycle. I think bedding them in is great, but sometimes thats more than a cycle. We benefit if we can decouple 'make big change' from 'big change is bedded in and ready for users to use'. (Since I started this discussion talking about bias, it's worth calling out a *huge* one here: my team has to figure out a way to package and distribute this stuff.) :). That said, if we can get to a situation where we *choose* to do a co-ordinated release for the convenience of developers, distributors and (hopefully) users - rather than being forced into it through sheer terror that everything will fall over in a flaming heap if we don't - that would obviously be a win :) +1. Right, if we _have_ to have it let's not name it something aspirational like Layer 1 or ring 0. Let's call it Cluster Foxtrot and make sure that projects are queueing up to get _out_. We can start with Designate, which afaik has done nothing to bring upon themselves such a fate ;) I'm not convinced that it has to be a formal, named thing though. In terms of the gating thing, that would be an organisational solution to a purely technical problem: up to now there's been no middle ground between a cluster of projects that all gate against each other and just not gating against each other at all. I'm totally confident that the QA and Infra teams can fix that. Those folks are superb at what they do. And on the release side, I think we're running before we can walk. It's not clear that most, or even many, projects would want to abandon a co-ordinated release anyway. (I'd actually have no problem with letting projects opt-out if they have some reason to - TripleO already effectively did.) External stakeholders (including the board) would _inevitably_ treat a shrinking of the co-ordinated release as an endorsement of which projects we actually care about, and by extension which projects they should care about. Hmm, this isn't really representative of TripleO's position. We *want*, *desperately* to be in the integrated gate, and we're nearly there in terms of donated capacity - we're focusing on on reliability at the moment. We have no integrated API servers [yet], but Tuskar's got a very clear plan taking it into the integrated release. Projects != Programs :). The non-API-server components of OpenStack as a whole are mostly not part of the integrated release. The plan with Tuskar was to get it stable enough to meet the incubation requirements and then apply. Of course if incubation goes away, thats different :). If we ever reach a point where interfaces are stable enough that we don't need a co-ordinated release, let's consider _then_ whether to tear the whole thing down - in one fell swoop. Taking a half-measure sends exactly the wrong signal. So as soon as you said that the Board would care about what is in the integrated release, that re-instates the winners-and-losers thing that a lot of this discussion is about. And Swift is already on a separate schedule, but its one of our most popular projects... I think there's something fundamentally mixed up here :0. Finally, I would add that unco-ordinated releases, where distributions have to select a set of components that synced to the global requirements at different times, are not going to be truly feasible until we have a container-based deployment system. I expect it to not be an issue in the future, but that work is only just getting underway. I don't
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 25, 2014, at 4:01 PM, Robert Collins robe...@robertcollins.net wrote: So I guess I'm saying: Lets decouple 'what is openstack' from 'what we test together on every commit'. It seems that this discussion has actually illustrated shortcomings in our answers to 3 separate questions, and people have been throwing out ideas that attempt to solve all 3. Perhaps we need to address each one individually. The three questions are: 1. Which projects are “part of openstack”? 2. Which projects are released as a single unit? 3. Which projects are tested together The current answers are: 1. Three levels incubation, integration, core 2. Things that reach the integration level 3. Things that reach the integration level. Some proposed answers: 1. Lightweight incubation a la apache 2. Monty’s layer1 3. Direct dependencies and close collaborators Discussing the propased answers(in reverse order): I think we have rough consensus around 3: that we should move towards functional testing for direct dependencies and let the projects decide when they want to co-gate. The functional co-gating should ideally be based on important use-cases. 2 is a bit murkier. In the interest of staying true to our roots the best we can probably do is to allow projects to opt-out of the coordinated release and for thierry to specifically select which projects he is willing to coordinate. Any other project could co-release with the integrated release but wouldn’t be centrally managed by thierry. There is also a decision about what the TCs role is in these projects. 1 Has some unanswerd questions, like is there another level “graduation” where the tc has some kind of technical oversight? What is the criteria for it? etc. Maybe addressing these things separately will allow us to make progress. Vish signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
I think Joe's idea pretty sums it up, ASF model is definitely worth following (Mesos is awesome). Non layer #1 projects will still be shepherded but not that closely coupled to make OpenStack over-bloated. Incubation projects can't be just dropped. On Wed, Sep 24, 2014 at 2:46 AM, Joe Gordon joe.gord...@gmail.com wrote: On Tue, Sep 23, 2014 at 9:50 AM, Vishvananda Ishaya vishvana...@gmail.com wrote: On Sep 23, 2014, at 8:40 AM, Doug Hellmann d...@doughellmann.com wrote: If we are no longer incubating *programs*, which are the teams of people who we would like to ensure are involved in OpenStack governance, then how do we make that decision? From a practical standpoint, how do we make a list of eligible voters for a TC election? Today we pull a list of committers from the git history from the projects associated with “official programs, but if we are dropping “official programs” we need some other way to build the list. Joe Gordon mentioned an interesting idea to address this (which I am probably totally butchering), which is that we make incubation more similar to the ASF Incubator. In other words make it more lightweight with no promise of governance or infrastructure support. you only slightly butchered it :). From what I gather the Apache Software Foundation primary goals are to: * provide a foundation for open, collaborative software development projects by supplying hardware, communication, and business infrastructure * create an independent legal entity to which companies and individuals can donate resources and be assured that those resources will be used for the public benefit * provide a means for individual volunteers to be sheltered from legal suits directed at the Foundation's projects * protect the 'Apache' brand, as applied to its software products, from being abused by other organizations [0] This roughly translates into: JIRA, SVN, Bugzilla and Confluence etc. for infrastructure resources. So ASF provides infrastructure, legal support, a trademark and some basic oversight. The [Apache] incubator is responsible for: * filtering the proposals about the creation of a new project or sub-project * help the creation of the project and the infrastructure that it needs to operate * supervise and mentor the incubated community in order for them to reach an open meritocratic environment * evaluate the maturity of the incubated project, either promoting it to official project/ sub-project status or by retiring it, in case of failure. It must be noted that the incubator (just like the board) does not perform filtering on the basis of technical issues. This is because the foundation respects and suggests variety of technical approaches. It doesn't fear innovation or even internal confrontation between projects which overlap in functionality. [1] So my idea, which is very similar to Monty's, is to make move all the non-layer 1 projects into something closer to an ASF model where there is still incubation and graduation. But the only things a project receives out of this process is: * Legal support * A trademark * Mentorship * Infrastructure to use * Basic oversight via the incubation/graduation process with respect to the health of the community. They do not get: * Required co-gating or integration with any other projects * People to right there docs for them, etc. * Technical review/oversight * Technical requirements * Evaluation on how the project fits into a bigger picture * Language requirements * etc. Note: this is just an idea, not a fully formed proposal [0] http://www.apache.org/foundation/how-it-works.html#what [1] http://www.apache.org/foundation/how-it-works.html#incubator It is also interesting to consider that we may not need much governance for things outside of layer1. Of course, this may be dancing around the actual problem to some extent, because there are a bunch of projects that are not layer1 that are already a part of the community, and we need a solution that includes them somehow. Vish ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Zhipeng Huang Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipe...@uci.edu Office: Calit2 Building Room 2402 OpenStack, OpenDaylight, OpenCompute affcienado ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/18/2014 02:53 PM, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 When I first read Monty's post, my basic reaction was yes, please. I think there are plenty of devils in the details, but all of which we can work through, we're mostly all reasonable people. A couple of follow ups for parts of the thread: Concerning the summit: while I understand ttx's concerns at - http://lists.openstack.org/pipermail/openstack-dev/2014-September/046504.html - my experience with the summits is the in project alignment isn't being well served by current format. The absolutely most valuable parts of the last summit for me were the Operator meetup sessions, and some of the cross project sessions. I think there is an interesting question of what does the TC govern. Honestly, I'm more in the camp that the TC focus should be on the Foundational Infrastructure (hey, new words, not sure if they are any better than layer 1 or ring 0), and have the ecosystem largely outside TC governance per Joe / Vish's ASF model - http://lists.openstack.org/pipermail/openstack-dev/2014-September/046877.html. There are pragmatic reasons for that, which is a TC that's based around that Foundation will tend to have more shared context about how we make that better and move forward. I'm not sure I see the point of a TC who's main job is ranking 100s of ecosystem projects on their production readiness... when most of them don't run production clouds. I really like markmc's point about Production Ready being something the User Committee should probably have more of a hand in - http://lists.openstack.org/pipermail/openstack-dev/2014-September/046653.html. I actually think some kind of self certification by projects to make them easy to evaluate by potential consumers would be really handy. This template might be a good thing to co-evolve between the User Committee and the TC. I'm completely happy getting rid of incubation, given that we're talking about a basically static foundation. I think the process of raising TC expectations on projects this past year exposed an interesting fact that there were things some of us felt were core values of OpenStack, that a lot of projects weren't doing. Our approach was they are doing it wrong and to put them on an improvement plan. But I think Monty's slicing up of things brings out an interesting point. Maybe they were doing it fine, they just weren't part of the particular shared culture needed to build foundational infrastructure. Maybe that was ok, because they weren't actually part of that. So, honestly, I'd say full speed ahead on Monty's plan. Is it perfect, probably not. But I think it's a demonstrable move towards a more sustainable base, a more inclusive ecosystem, and a better consumption experience by our users. So how do we put a big stamp on it and make this the direction we are headed in? -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 18/09/14 14:53, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Thanks Monty, I think there are some very interesting ideas in here. I'm particularly glad to see the 'big tent' camp reasserting itself, because I have no sympathy with anyone who wants to join the OpenStack community and then bolt the door behind them. Anyone who contributes to a project that is related to OpenStack's goals, is willing to do things the OpenStack way, and submits itself to the scrutiny of the TC deserves to be treated as a member of our community with voting rights, entry to the Design Summit and so on. I'm curious how you're suggesting we decide which projects satisfy those criteria though. Up until now, we've done it through the incubation process (or technically, the new program approval process... but in practice we've never added a project that was targeted for eventual inclusion in the integrated release to a program without incubating it). Would the TC continue to judge whether a project is doing things the OpenStack way prior to inclusion, or would we let projects self-certify? What does it mean for a project to submit itself to TC scrutiny if it knows that realistically the TC will never have time to actually scrutinise it? Or are you not suggesting a change to the current incubation process, just a willingness to incubate multiple projects in the same problem space? I feel like I need to play devil's advocate here, because overall I'm just not sure I understand the purpose of arbitrarily - and it *is* arbitrary - declaring Layer #1 to be anything required to run Wordpress. To anyone whose goal is not to run Wordpress, how is that relevant? Speaking of arbitrary, I had to laugh a little at this bit: Also, please someone notice that the above is too many steps and should be: openstack boot gentoo on-a 2G-VM with-a publicIP with-a 10G-volume call-it blog.inaugust.com That's kinda sorta exactly what Heat does ;) Minus the part about assuming there is only one kind of application, obviously. I think there are a number of unjustified assumptions behind this arrangement of things. I'm going to list some here, but I don't want anyone to interpret this as a personal criticism of Monty. The point is that we all suffer from biases - not for any questionable reasons but purely as a result of our own experiences, who we spend our time talking to and what we spend our time thinking about - and therefore we should all be extremely circumspect about trying to bake our own mental models of what OpenStack should be into the organisational structure of the project itself. * Assumption #1: The purpose of OpenStack is to provide a Compute cloud This assumption is front-and-centre throughout everything Monty wrote. Yet this wasn't how the OpenStack project started. In fact there are now at least three services - Swift, Nova, Zaqar - that could each make sense as the core of a standalone product. Yes, it's true that Nova effectively depends on Glance and Neutron (and everything depends on Keystone). We should definitely document that somewhere. But why does it make Nova special? * Assumption #2: Yawnoc's Law Don't bother Googling that, I just made it up. It's the reverse of Conway's Law: Infra engineers who design governance structures for OpenStack are constrained to produce designs that are copies of the structure of Tempest. I just don't understand why that needs to be the case. Currently, for understandable historic reasons, every project gates against every other project. That makes no sense any more, completely independently of the project governance structure. We should just change it! There is no organisational obstacle to changing how gating works. Even this proposal doesn't entirely make sense on this front - e.g. Designate requires only Neutron and Keystone... why should Nova, Glance and every other project in Layer 1 gate against it, and vice-versa? I suggested in another thread[1] a model where each project would publish a set of tests, each project would decide which sets of tests to pull in and gate on, and Tempest would just be a shell for setting up the environment and running the selected tests. Maybe that idea is crazy or at least needs more work (it certainly met with only crickets and tumbleweeds on the mailing list), but implementing it wouldn't require TC intervention and certainly not by-laws changes. It just requires... implementing it. Perhaps the idea here is that by designating Layer 1 the TC is indicating to projects which other projects they should accept gate test jobs from (a function previously fulfilled by Incubation). I'd argue that this is a very bad way to do it, because (a) it says nothing to projects outside of Layer 1 how they should
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Excerpts from Robert Collins's message of 2014-09-23 21:14:47 -0700: No one helped me edit this :) http://rbtcollins.wordpress.com/2014/09/24/what-poles-for-the-tent/ I hope I haven't zoned out and just channelled someone else here ;) This sounds like API's are what matters. You did spend some time working with Simon Wardley, didn't you? ;) I think it's a sound argument, but I'd like to banish the term reference implementation from any discussions around what OpenStack, as a project, delivers. It has too many negative feelings wrapped up in it. I also want to call attention to how what you describe feels an awful lot like POSIX to me. Basically offering guarantees of API compatibility, but then letting vendors run wild around and behind it. I'm not sure if that is a good thing, or a bad thing. I do, however, think if we can avoid a massive vendor battle that involves multiple vendors pushing multiple implementations, we will save our companies a lot of money, and our users will get what they need sooner. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 19/09/14 22:37, Monty Taylor wrote: I think we can do what you're saying and generalize a little bit. What if we declared programs, as needed, when we think there is a need to pick a winner. (I think we can all agree that early winner picking is an unintended but very real side effect of the current system) Ooh, a challenge. I'll bite ;) Here's a question: how many instances are there of two Stackforge projects working on the same problem domain? I can think of one example: Gnocchi and StackTach - though those are (respectively) a branch of and a sort-of competitor to an integrated project (where we supposedly picked the winner already). So we have evidence of competitors surviving the picking of a winner, but not a lot of evidence of competition in general. (To be fair, you could make an argument that people are being put off starting competing projects by the prospect of eventually having to submit to winner-picking by the TC.) Don't get me wrong, I think it's clear that our implicit hope for the current model - that picking a winner would mean everyone getting on board with that project and making it better - has failed to materialise. But there's also every reason to think that a model where we rely on competition between projects in the marketplace to determine a winner owes just as much to wishful thinking. I suspect that part of the problem is that there are so many different things any given person could be working on to make users' lives better that when they see a project whose approach they don't agree with, they're more likely to just go work on something else than start a competitor or jump in to try and change the direction. (In fact, the people most qualified to do so are almost by definition the most busy already.) Maybe just throw a few rocks right before the graduation review, that sort of thing. If you were hoping this email would end with some kind of proposed solution, prepare for disappointment. Early winner-picking obviously sucks for the developer who realises that they need to make major changes and has to deal with the extra challenge of maintaining API compatibility and upgradability while doing it. On the other hand, it sucks precisely because that's great for users and operators. A competition model, even assuming that the competition actually arises, just means that the portion of operators who chose the 'wrong' side and their users get hosed when an eventual winner emerges. Potential consequences include delayed interoperability, delayed adoption of new features altogether to avoid the risk, or even permanent lack of interoperability with proprietary solutions being used instead. The only suggestion I can make is one I mentioned in another thread[1]: establish a design principle that the parts of the design which are hard to change (e.g. APIs) must be a simple as possible in order to provide the maximum flexibility of implementation, until such time as both the implementation and the need for more complexity have been validated in the real world. cheers, Zane. [1] http://lists.openstack.org/pipermail/openstack-dev/2014-September/046563.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/24/2014 02:48 PM, Clint Byrum wrote: Excerpts from Robert Collins's message of 2014-09-23 21:14:47 -0700: No one helped me edit this :) http://rbtcollins.wordpress.com/2014/09/24/what-poles-for-the-tent/ I hope I haven't zoned out and just channelled someone else here ;) This sounds like API's are what matters. You did spend some time working with Simon Wardley, didn't you? ;) I think it's a sound argument, but I'd like to banish the term reference implementation from any discussions around what OpenStack, as a project, delivers. It has too many negative feelings wrapped up in it. I also want to call attention to how what you describe feels an awful lot like POSIX to me. Basically offering guarantees of API compatibility, but then letting vendors run wild around and behind it. I'm not sure if that is a good thing, or a bad thing. I do, however, think if we can avoid a massive vendor battle that involves multiple vendors pushing multiple implementations, we will save our companies a lot of money, and our users will get what they need sooner. I like what Rob had to say here, and have expressed similar views. Having competition between implementations is good for every one (except for the losers) if that competition takes place in a way that shields users and the ecosystem from the aftermath of such competition. That is what standards, defined apis, whetever we want to call it, is all about. By analogy, competition by electronics companies around who can make the best performing blu-ray player with the most features is a good thing for users and that ecosystem. Competition about whether the ecosystem should use blu-ray or HD DVD, not so much: http://en.wikipedia.org/wiki/High_definition_optical_disc_format_war. This is what I see as the main virtue of the TC blessing things as the one OpenStack way to do X. There is also the potential of efficiency if more people contribute to the same project that is doing X as compared to multiple projects doing X. But as we have seen, that efficiency is only realized if X turns out to be the right thing. There is no particular reason to think the TC will be great at picking winners. Blessing apis, though difficult, would have huge benefit and provide more room for leeway and experimentation. Blessing code, before it has been proven in the real world, is the worst of all worlds when it turns out to be wrong. I believe our scale problems can be addressed by thoughtful decentralization and I hope we move in that direction, and in terms of how many pieces of the run a real cloud we have in our tent, we may have shot too high. But some of the recent proposals to move to an extreme in the other direction would be a mistake IMO. To be important, and be competitive with non-OpenStack cloud solutions, we need to provide a critical mass so that most other interesting things can glom on and form a larger ecosystem. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 2014-09-24 13:55:57 -0400 (-0400), Zane Bitter wrote: [...] * Assumption #2: Yawnoc's Law Don't bother Googling that, I just made it up. It's the reverse of Conway's Law: Infra engineers who design governance structures for OpenStack are constrained to produce designs that are copies of the structure of Tempest. I just don't understand why that needs to be the case. Currently, for understandable historic reasons, every project gates against every other project. That makes no sense any more, completely independently of the project governance structure. We should just change it! There is no organisational obstacle to changing how gating works. [...] Note that to a great extent the current proliferation of integration testing was driven from the developers of many projects. The Infra and QA teams didn't just wake up one morning and decide to chuck all the projects in. Rewind a year or two and remember that we had massive amounts of asymmetry in our testing. Project A would implement some new change and Project B developers would get their jobs insta-broken, then come complaining that clearly this meant we should be gating Project A on whether or not Project B works. For a time we had sufficient (human and server) resources to do that, and so it was comparatively cheap to just keep expanding the list of projects which shared a common set of jobs we hoped exercised enough of everything to act as a canary in the coal mine. We're now running into pretty clear scalability limits on the number of development teams whose changes you can effectively test against each other given the realities of nondeterministic failures, breakdowns in cross-group communication, growth in size of integrated releases, tragedy of the commons effects on horizontal efforts/shared resources, external factors like dependency changes, et cetera. As a community we should explore solutions (and clearly are) to these underlying problems, but also need to reconsider some old habits that need changing such as tight coupling to the internal APIs and implementation details of other projects... especially if doing so lets us scale back our integration testing, rather than leaning on it harder so that these undesirable development patterns can continue unabated. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Thu, Sep 25, 2014 at 3:55 AM, Zane Bitter zbit...@redhat.com wrote: On 18/09/14 14:53, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Thanks Monty, I think there are some very interesting ideas in here. I'm particularly glad to see the 'big tent' camp reasserting itself, because I have no sympathy with anyone who wants to join the OpenStack community and then bolt the door behind them. Anyone who contributes to a project that is related to OpenStack's goals, is willing to do things the OpenStack way, and submits itself to the scrutiny of the TC deserves to be treated as a member of our community with voting rights, entry to the Design Summit and so on. I'm curious how you're suggesting we decide which projects satisfy those criteria though. Up until now, we've done it through the incubation process (or technically, the new program approval process... but in practice we've never added a project that was targeted for eventual inclusion in the integrated release to a program without incubating it). Would the TC continue to judge whether a project is doing things the OpenStack way prior to inclusion, or would we let projects self-certify? What does it mean for a project to submit itself to TC scrutiny if it knows that realistically the TC will never have time to actually scrutinise it? Or are you not suggesting a change to the current incubation process, just a willingness to incubate multiple projects in the same problem space? I feel like I need to play devil's advocate here, because overall I'm just not sure I understand the purpose of arbitrarily - and it *is* arbitrary - declaring Layer #1 to be anything required to run Wordpress. To anyone whose goal is not to run Wordpress, how is that relevant? Speaking of arbitrary, I had to laugh a little at this bit: Also, please someone notice that the above is too many steps and should be: openstack boot gentoo on-a 2G-VM with-a publicIP with-a 10G-volume call-it blog.inaugust.com That's kinda sorta exactly what Heat does ;) Minus the part about assuming there is only one kind of application, obviously. :-) I think there are a number of unjustified assumptions behind this arrangement of things. I'm going to list some here, but I don't want anyone to interpret this as a personal criticism of Monty. The point is that we all suffer from biases - not for any questionable reasons but purely as a result of our own experiences, who we spend our time talking to and what we spend our time thinking about - and therefore we should all be extremely circumspect about trying to bake our own mental models of what OpenStack should be into the organisational structure of the project itself. * Assumption #1: The purpose of OpenStack is to provide a Compute cloud This assumption is front-and-centre throughout everything Monty wrote. Yet this wasn't how the OpenStack project started. In fact there are now at least three services - Swift, Nova, Zaqar - that could each make sense as the core of a standalone product. Agree. Yes, it's true that Nova effectively depends on Glance and Neutron (and everything depends on Keystone). We should definitely document that somewhere. But why does it make Nova special? * Assumption #2: Yawnoc's Law Don't bother Googling that, I just made it up. It's the reverse of Conway's Law: Infra engineers who design governance structures for OpenStack are constrained to produce designs that are copies of the structure of Tempest. I just don't understand why that needs to be the case. Currently, for understandable historic reasons, every project gates against every other project. That makes no sense any more, completely independently of the project governance structure. We should just change it! There is no organisational obstacle to changing how gating works. Even this proposal doesn't entirely make sense on this front - e.g. Designate requires only Neutron and Keystone... why should Nova, Glance and every other project in Layer 1 gate against it, and vice-versa? I suggested in another thread[1] a model where each project would publish a set of tests, each project would decide which sets of tests to pull in and gate on, and Tempest would just be a shell for setting up the environment and running the selected tests. Maybe that idea is crazy or at least needs more work (it certainly met with only crickets and tumbleweeds on the mailing list), but implementing it wouldn't require TC intervention and certainly not by-laws changes. It just requires... implementing it. Perhaps the idea here is that by designating Layer 1 the TC is indicating to projects which other projects they should accept gate test jobs from (a function previously fulfilled by Incubation). I'd argue that this is a very
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Devananda van der Veen wrote: On Mon, Sep 22, 2014 at 2:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 5:10 PM, Devananda van der Veen devananda@gmail.com wrote: One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. We have plenty of things which are clearly part of OpenStack, and yet which are not part of the Integrated Release. Oslo. Devstack. Zuul... As far as I can tell, the only time when integrated release equals the thing we say is OpenStack is when we're talking about the trademark. The main goal of incubation, as we did it in the past cycles, is a learning period where the new project aligns enough with the existing ones so that it integrates with them (Horizon shows Sahara dashboard) and won't break them around release time (stability, co-gate, respect of release deadlines). If we have a strict set of projects in layer #1, I don't see the point of keeping incubation. We wouldn't add new projects to layer #1 (only project splits which do not really require incubations), and additions to the big tent are considered on social alignment only (are you vaguely about cloud and do you follow the OpenStack way). If there is nothing to graduate to, there is no need for incubation. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. Project identity and governance are presently addressed by the creation of Programs and a fully-elected TC. Integration is not addressing these things at all, as far as I can tell, though I agree that it was initially intended to. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? I think a good basis for this is simply whether the developers of the project are part of our community, doing things in the way that we do things, and want this to happen. Voting and ATC status is already decoupled [0] from the integrated gate and the integrated release -- it's based on the accepted list of Programs [1], which actually has nothing to do with incubation or integration [2]. In Monty's proposal, ATC status would be linked to contributions to the big tent. Projects apply to become part of it, subject themselves to the oversight of the Technical Committee, and get the right to elect TC members in return. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Robert Collins wrote: On 19 September 2014 22:29, Thierry Carrez thie...@openstack.org wrote: ... current Heat team is not really interested in maintaining them. What's the point of being under the same program then ? And TripleO is not the only way to deploy OpenStack, but its mere existence (and name) prevented other flowers to bloom in our community. So I've a small issue here - *after* TripleO became an official program Tuskar was started - and we embraced and shuffled things to collaborate. There's a tripleo puppet repository, an ansible one coming soon I believe, and I'm hearing rumours of a spike using another thing altogether (whose thunder I don't want to steal so I'm going to be vague :)). We've collaborated to a degree with Fuel and Crowbar: I am not at all sure we've prevented other flowers blooming - and I hate the idea that we have done that. I agree that you went out of your way to be inclusive, but having a single PTL for a group of competing alternatives is a bit weird. As far as preventing blooming, that's something we've heard from the Board of Directors (as part of an objection to calling the TripleO program OpenStack Deployment). ... ## The release and the development cycle You touch briefly on the consequences of your model for the common release and our development cycle. Obviously we would release the ring 0 projects in an integrated manner on a time-based schedule. I'm not at all sure we need to do that - I've long been suspicious of the coordinated release. I see benefits to users in being able to grab a new set of projects all at once, but they can do that irrespective of our release process, as long as: *) we do releases *) we do at least one per project per 6 month period Tying all our things together makes for hugely destablising waves of changes and rushes: if we aimed at really smooth velocity and frequent independent releases I think we could do a lot better: contracts and interfaces are *useful* things for large scale architectures, and we need to stop kidding ourselves - OpenStack is not a lean little beastie anymore: its a large scale distributed system. Swift is doing exactly the right thing today - I'd like to see more of that. Note that I'm only advocating that Monty's layer #1 things would be released in an integrated manner on a time-based schedule. That's not all our things. All the other things would be released as-needed. That lets us focus on cleaning up interfaces between layer #1 and other things first. Swift is doing the right thing today because its loosely-integrated with layer #1 (and has a clean interface for doing so). So Swift is doing the right thing for an non-layer#1 project. Once our layer#1-internal interfaces are simple/stable enough that we can safely compose something that works at any time by picking the latest version of everything, we can discuss the benefits and the drawbacks of the common release model. I still think there are huge community benefits to sharing the same cycle, and gathering around a common release that can be advertised and consumed downstream... as far as layer #1 is concerned. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 23, 2014, at 5:18 AM, Thierry Carrez thie...@openstack.org wrote: Devananda van der Veen wrote: On Mon, Sep 22, 2014 at 2:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 5:10 PM, Devananda van der Veen devananda@gmail.com wrote: One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. We have plenty of things which are clearly part of OpenStack, and yet which are not part of the Integrated Release. Oslo. Devstack. Zuul... As far as I can tell, the only time when integrated release equals the thing we say is OpenStack is when we're talking about the trademark. The main goal of incubation, as we did it in the past cycles, is a learning period where the new project aligns enough with the existing ones so that it integrates with them (Horizon shows Sahara dashboard) and won't break them around release time (stability, co-gate, respect of release deadlines). If we have a strict set of projects in layer #1, I don't see the point of keeping incubation. We wouldn't add new projects to layer #1 (only project splits which do not really require incubations), and additions to the big tent are considered on social alignment only (are you vaguely about cloud and do you follow the OpenStack way). If there is nothing to graduate to, there is no need for incubation. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. Project identity and governance are presently addressed by the creation of Programs and a fully-elected TC. Integration is not addressing these things at all, as far as I can tell, though I agree that it was initially intended to. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? I think a good basis for this is simply whether the developers of the project are part of our community, doing things in the way that we do things, and want this to happen. Voting and ATC status is already decoupled [0] from the integrated gate and the integrated release -- it's based on the accepted list of Programs [1], which actually has nothing to do with incubation or integration [2]. In Monty's proposal, ATC status would be linked to contributions to the big tent. Projects apply to become part of it, subject themselves to the oversight of the Technical Committee, and get the right to elect TC members in return. That’s the part that wasn’t clear. If we’re not “incubating” those projects, then what criteria do we use to make decisions about the applications? Is a declaration of intent enough? Doug -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 22, 2014, at 8:05 PM, Devananda van der Veen devananda@gmail.com wrote: On Mon, Sep 22, 2014 at 2:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 5:10 PM, Devananda van der Veen devananda@gmail.com wrote: One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. We have plenty of things which are clearly part of OpenStack, and yet which are not part of the Integrated Release. Oslo. Devstack. Zuul... As far as I can tell, the only time when integrated release equals the thing we say is OpenStack is when we're talking about the trademark. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. Project identity and governance are presently addressed by the creation of Programs and a fully-elected TC. Integration is not addressing these things at all, as far as I can tell, though I agree that it was initially intended to. Good point: I’m mixing terms here. Programs and projects have tended to be incubated at the same time. We’ve insisted that it doesn’t make sense to have a program if there is nothing being produced, and that a project can’t be incubated if the program isn’t also incubated. The fact that we’ve also had 1:1 coupling between programs and projects is unfortunate, but orthogonal to the fact that we have been evaluating the teams as well as the code. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? I think a good basis for this is simply whether the developers of the project are part of our community, doing things in the way that we do things, and want this to happen. Voting and ATC status is already decoupled [0] from the integrated gate and the integrated release -- it's based on the accepted list of Programs [1], which actually has nothing to do with incubation or integration [2]. I’m concerned that we’re combining changes to the way we decide what we include in the gate with changes to the way we decide which groups of people have a say in how the overall OpenStack project is run. One is a technical discussion that has nothing at all to do with governance. The other is entirely about governance. If we are no longer incubating *programs*, which are the teams of people who we would like to ensure are involved in OpenStack governance, then how do we make that decision? From a practical standpoint, how do we make a list of eligible voters for a TC election? Today we pull a list of committers from the git history from the projects associated with “official programs, but if we are dropping “official programs” we need some other way to build the list. Doug -Devananda [0] http://git.openstack.org/cgit/openstack/governance/tree/reference/charter.rst#n132 [1] http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml [2] http://git.openstack.org/cgit/openstack/governance/tree/reference/new-programs-requirements.rst ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 22 September 2014 23:14, Robert Collins robe...@robertcollins.net wrote: I am not at all sure we've prevented other flowers blooming - and I hate the idea that we have done that. I've certainly sat around at discussions which shut down hard with somebody making the statement that 'that is TripleO's field and they don't like X', it is a genuine if entirely unintentional problem. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Tue, Sep 23, 2014 at 8:40 AM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 8:05 PM, Devananda van der Veen devananda@gmail.com wrote: On Mon, Sep 22, 2014 at 2:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 5:10 PM, Devananda van der Veen devananda@gmail.com wrote: One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. We have plenty of things which are clearly part of OpenStack, and yet which are not part of the Integrated Release. Oslo. Devstack. Zuul... As far as I can tell, the only time when integrated release equals the thing we say is OpenStack is when we're talking about the trademark. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. Project identity and governance are presently addressed by the creation of Programs and a fully-elected TC. Integration is not addressing these things at all, as far as I can tell, though I agree that it was initially intended to. Good point: I’m mixing terms here. Programs and projects have tended to be incubated at the same time. We’ve insisted that it doesn’t make sense to have a program if there is nothing being produced, and that a project can’t be incubated if the program isn’t also incubated. The fact that we’ve also had 1:1 coupling between programs and projects is unfortunate, but orthogonal to the fact that we have been evaluating the teams as well as the code. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? I think a good basis for this is simply whether the developers of the project are part of our community, doing things in the way that we do things, and want this to happen. Voting and ATC status is already decoupled [0] from the integrated gate and the integrated release -- it's based on the accepted list of Programs [1], which actually has nothing to do with incubation or integration [2]. I’m concerned that we’re combining changes to the way we decide what we include in the gate with changes to the way we decide which groups of people have a say in how the overall OpenStack project is run. These things already weren't related. One is a technical discussion that has nothing at all to do with governance. The other is entirely about governance. If we are no longer incubating *programs*, which are the teams of people who we would like to ensure are involved in OpenStack governance, then how do we make that decision? From a practical standpoint, how do we make a list of eligible voters for a TC election? Today we pull a list of committers from the git history from the projects associated with “official programs, but if we are dropping “official programs” we need some other way to build the list. I don't think incubation ever applied to programs. Any program listed in http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml is official and gets voting rights starting in the election after it was added to that file. I also don't think that Monty's proposal suggests that we drop programs. I think it's suggesting the opposite -- we allow *more* programs (and the projects associated with them) into the openstack/* fold without requiring them to join the integrated gating of the layer #1 projects. -Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Doug Hellmann wrote: On Sep 23, 2014, at 5:18 AM, Thierry Carrez thie...@openstack.org wrote: Devananda van der Veen wrote: On Mon, Sep 22, 2014 at 2:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 5:10 PM, Devananda van der Veen devananda@gmail.com wrote: One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. We have plenty of things which are clearly part of OpenStack, and yet which are not part of the Integrated Release. Oslo. Devstack. Zuul... As far as I can tell, the only time when integrated release equals the thing we say is OpenStack is when we're talking about the trademark. The main goal of incubation, as we did it in the past cycles, is a learning period where the new project aligns enough with the existing ones so that it integrates with them (Horizon shows Sahara dashboard) and won't break them around release time (stability, co-gate, respect of release deadlines). If we have a strict set of projects in layer #1, I don't see the point of keeping incubation. We wouldn't add new projects to layer #1 (only project splits which do not really require incubations), and additions to the big tent are considered on social alignment only (are you vaguely about cloud and do you follow the OpenStack way). If there is nothing to graduate to, there is no need for incubation. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. Project identity and governance are presently addressed by the creation of Programs and a fully-elected TC. Integration is not addressing these things at all, as far as I can tell, though I agree that it was initially intended to. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? I think a good basis for this is simply whether the developers of the project are part of our community, doing things in the way that we do things, and want this to happen. Voting and ATC status is already decoupled [0] from the integrated gate and the integrated release -- it's based on the accepted list of Programs [1], which actually has nothing to do with incubation or integration [2]. In Monty's proposal, ATC status would be linked to contributions to the big tent. Projects apply to become part of it, subject themselves to the oversight of the Technical Committee, and get the right to elect TC members in return. That’s the part that wasn’t clear. If we’re not “incubating” those projects, then what criteria do we use to make decisions about the applications? Is a declaration of intent enough? In Monty's proposal, the big tent is pretty welcoming. The bar is are you one of us: Some examples of community that we care about: being on stackforge rather than github; having a PTL who you elect rather than a BDFL; having meetings on IRC. Do any of the people who hack on the project also hack on any other existing OpenStack projects, or are the people completely unconnected? is a potential social touchstone as well. All in all, meeting the requirements for being one of us is not particularly hard, nor should it be. It's a community values assessment... I don't see what incubating would give us there, apart from preserving red tape. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 23, 2014, at 8:40 AM, Doug Hellmann d...@doughellmann.com wrote: If we are no longer incubating *programs*, which are the teams of people who we would like to ensure are involved in OpenStack governance, then how do we make that decision? From a practical standpoint, how do we make a list of eligible voters for a TC election? Today we pull a list of committers from the git history from the projects associated with “official programs, but if we are dropping “official programs” we need some other way to build the list. Joe Gordon mentioned an interesting idea to address this (which I am probably totally butchering), which is that we make incubation more similar to the ASF Incubator. In other words make it more lightweight with no promise of governance or infrastructure support. It is also interesting to consider that we may not need much governance for things outside of layer1. Of course, this may be dancing around the actual problem to some extent, because there are a bunch of projects that are not layer1 that are already a part of the community, and we need a solution that includes them somehow. Vish signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Devananda van der Veen wrote: On Tue, Sep 23, 2014 at 8:40 AM, Doug Hellmann d...@doughellmann.com wrote: One is a technical discussion that has nothing at all to do with governance. The other is entirely about governance. If we are no longer incubating *programs*, which are the teams of people who we would like to ensure are involved in OpenStack governance, then how do we make that decision? From a practical standpoint, how do we make a list of eligible voters for a TC election? Today we pull a list of committers from the git history from the projects associated with “official programs, but if we are dropping “official programs” we need some other way to build the list. I don't think incubation ever applied to programs. Any program listed in http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml is official and gets voting rights starting in the election after it was added to that file. I confirm there never was incubation for programs. The only thing that goes through incubation are projects that want to become part of the integrated release. I also don't think that Monty's proposal suggests that we drop programs. I think it's suggesting the opposite -- we allow *more* programs (and the projects associated with them) into the openstack/* fold without requiring them to join the integrated gating of the layer #1 projects. Although the proposal might make them so cheap we wouldn't need a formal word for them. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Tue, Sep 23, 2014 at 9:50 AM, Vishvananda Ishaya vishvana...@gmail.com wrote: On Sep 23, 2014, at 8:40 AM, Doug Hellmann d...@doughellmann.com wrote: If we are no longer incubating *programs*, which are the teams of people who we would like to ensure are involved in OpenStack governance, then how do we make that decision? From a practical standpoint, how do we make a list of eligible voters for a TC election? Today we pull a list of committers from the git history from the projects associated with “official programs, but if we are dropping “official programs” we need some other way to build the list. Joe Gordon mentioned an interesting idea to address this (which I am probably totally butchering), which is that we make incubation more similar to the ASF Incubator. In other words make it more lightweight with no promise of governance or infrastructure support. you only slightly butchered it :). From what I gather the Apache Software Foundation primary goals are to: * provide a foundation for open, collaborative software development projects by supplying hardware, communication, and business infrastructure * create an independent legal entity to which companies and individuals can donate resources and be assured that those resources will be used for the public benefit * provide a means for individual volunteers to be sheltered from legal suits directed at the Foundation's projects * protect the 'Apache' brand, as applied to its software products, from being abused by other organizations [0] This roughly translates into: JIRA, SVN, Bugzilla and Confluence etc. for infrastructure resources. So ASF provides infrastructure, legal support, a trademark and some basic oversight. The [Apache] incubator is responsible for: * filtering the proposals about the creation of a new project or sub-project * help the creation of the project and the infrastructure that it needs to operate * supervise and mentor the incubated community in order for them to reach an open meritocratic environment * evaluate the maturity of the incubated project, either promoting it to official project/ sub-project status or by retiring it, in case of failure. It must be noted that the incubator (just like the board) does not perform filtering on the basis of technical issues. This is because the foundation respects and suggests variety of technical approaches. It doesn't fear innovation or even internal confrontation between projects which overlap in functionality. [1] So my idea, which is very similar to Monty's, is to make move all the non-layer 1 projects into something closer to an ASF model where there is still incubation and graduation. But the only things a project receives out of this process is: * Legal support * A trademark * Mentorship * Infrastructure to use * Basic oversight via the incubation/graduation process with respect to the health of the community. They do not get: * Required co-gating or integration with any other projects * People to right there docs for them, etc. * Technical review/oversight * Technical requirements * Evaluation on how the project fits into a bigger picture * Language requirements * etc. Note: this is just an idea, not a fully formed proposal [0] http://www.apache.org/foundation/how-it-works.html#what [1] http://www.apache.org/foundation/how-it-works.html#incubator It is also interesting to consider that we may not need much governance for things outside of layer1. Of course, this may be dancing around the actual problem to some extent, because there are a bunch of projects that are not layer1 that are already a part of the community, and we need a solution that includes them somehow. Vish ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/23/2014 02:18 AM, Thierry Carrez wrote: The main goal of incubation, as we did it in the past cycles, is a learning period where the new project aligns enough with the existing ones so that it integrates with them (Horizon shows Sahara dashboard) and won't break them around release time (stability, co-gate, respect of release deadlines). If we have a strict set of projects in layer #1, I don't see the point of keeping incubation. We wouldn't add new projects to layer #1 (only project splits which do not really require incubations), and additions to the big tent are considered on social alignment only (are you vaguely about cloud and do you follow the OpenStack way). If there is nothing to graduate to, there is no need for incubation. There's no need for incubation, as such, but it's worth taking the time to think about the technical and social functions that incubation and integration served (sometimes ineffectively, or only as side-effects), and what will replace them. You've identified a few there: - learning period for new projects - alignment with existing projects - stability (in which gating served as a weak crutch, and the real answer will likely lie in more extensive cross-project communication, also carefully filtered to avoid information overload) - respect of release deadlines (which doesn't necessarily mean releasing all at the same time, just being cognizant of network-effects of releases, and the cadence of other projects in an up-or-down dependency relationship with yours) In Monty's proposal, ATC status would be linked to contributions to the big tent. Projects apply to become part of it, subject themselves to the oversight of the Technical Committee, and get the right to elect TC members in return. And, a few more here: - transitioning from island to part of the big tent - accepting oversight of TC - accepting responsibility to participate in TC election Allison ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 23, 2014, at 12:52 PM, Thierry Carrez thie...@openstack.org wrote: Devananda van der Veen wrote: On Tue, Sep 23, 2014 at 8:40 AM, Doug Hellmann d...@doughellmann.com wrote: One is a technical discussion that has nothing at all to do with governance. The other is entirely about governance. If we are no longer incubating *programs*, which are the teams of people who we would like to ensure are involved in OpenStack governance, then how do we make that decision? From a practical standpoint, how do we make a list of eligible voters for a TC election? Today we pull a list of committers from the git history from the projects associated with “official programs, but if we are dropping “official programs” we need some other way to build the list. I don't think incubation ever applied to programs. Any program listed in http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml is official and gets voting rights starting in the election after it was added to that file. I confirm there never was incubation for programs. The only thing that goes through incubation are projects that want to become part of the integrated release. I’m having some trouble making my question clear. We’re getting hung up on terminology, and part of that is because I think I’m using old terms for describing things and their states of being under the proposed changed system. Let me try rephrasing my concern, but first let me frame the question by saying that whatever else we say it is, OpenStack is made of people. The proposal includes a number of relatively small changes that affect how those people work and what they work on, combined with one big change to how we decide who those people are. I’m asking for more details about that big change. While the proposal has many concrete components, it also is purposefully more vague on some points as a way to lower the barrier for defining new groups of contributors and creating new projects. I agree with many of those goals, but I think that some of the points on which the proposal is vague are important for us to spell out clearly before they can be implemented. One such point is, how do new people — not their code, but the contributors themselves — obtain a status within the overall OpenStack project granting them the option of participating in our governance? In the past, we have had formal votes by members of the Technical Committee to decide whether or not to accept new groups (at different times called projects and programs) as being official parts of the overall OpenStack project. There are guidelines for what standards those groups need to follow once they are official [1] and other guidelines for the software created by those groups at designated points in its lifecycle [2]. While I’m not finding a specific procedure documented the TC has historically voted on the official status of groups either in IRC meetings or using gerrit based on changes to files in the governance repository [3]. The request for a vote from the TC is usually initiated by a leader in the group asking for official status. The official status of the group, and the git repositories it manages, are used to build our community voter list, and that is why it is important to understand how the proposal we’re discussing is different from what we do now. The “Who we are and what we all work on” section touches on this, but does not actually describe a process to be followed. The guidelines listed there seem easy to meet, but who decides if they actually are met for a given person or group of people? What, if any, expectations are placed on the group after they are given official status? What, if any, benefits come with the designation? The proposal includes process changes, testing configuration changes, and feature proposals. Most of the items it covers are not related to governance, and evaluating them all together masks some of the more important aspects of the governance changes. We could just go change the way we define the integrated gate without changing anything else we do, for example. It would be easier to evaluate the proposed governance changes if they were a patch on the existing repository, leaving out all of the unrelated suggestions to be evaluated separately. Doug [1] http://git.openstack.org/cgit/openstack/governance/tree/reference/new-programs-requirements.rst [2] http://git.openstack.org/cgit/openstack/governance/tree/reference/incubation-integration-requirements.rst [3] http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml I also don't think that Monty's proposal suggests that we drop programs. I think it's suggesting the opposite -- we allow *more* programs (and the projects associated with them) into the openstack/* fold without requiring them to join the integrated gating of the layer #1 projects. Although the proposal might make them so cheap we wouldn't
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
No one helped me edit this :) http://rbtcollins.wordpress.com/2014/09/24/what-poles-for-the-tent/ I hope I haven't zoned out and just channelled someone else here ;) -Rob On 19 September 2014 06:53, Monty Taylor mord...@inaugust.com wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Enjoy. Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
John Dickinson wrote: I propose that we can get the benefits of Monty's proposal and implement all of his concrete suggestions (which are fantastic) by slightly adjusting our usage of the program/project concepts. I had originally hoped that the program concept would have been a little higher-level instead of effectively spelling project as program. I'd love to see a hierarchy of openstack-program-project/team-repos. Right now, we have added the program layer but have effectively mapped it 1:1 to the project. For example, we used to have a few repos in the Swift project managed by the same group of people, and now we have a few repos in the object storage program, all managed by the same group of people. And every time something is added to OpenStack, its added as a new program, effectively putting us exactly where we were before we called it a program with the same governance and management scaling problems. Today, I'd group existing OpenStack projects into programs as follows: Compute: nova, sahara, ironic Storage: swift, cinder, glance, trove Network: neutron, designate, zaqar Deployment/management: heat, triple-o, horizon, ceilometer Identity: keystone, barbican Support (not user facing): infra, docs, tempest, devstack, oslo (potentially even) stackforge: lots I'm not clear on what those $THINGS would actually represent. Would they have a single PTL ? (I guess not, otherwise we are back at the team/theme duality I called out in my answer to Monty). Who decides which $THINGS may exist ? (I suppose anybody can declare one, otherwise we are back with TC blessing fields). Can $THINGS contain alternative competing solutions ? (I suppose yes, otherwise we are back to preventing innovation). So what are they ? A tag you can apply to your project ? A category you can file yourself under ? I like Monty's end-user approach to defining layer #1. The use case being, you want a functioning compute instance. Your taxonomy seems more artificial. Swift, Cinder, Glance and Trove (and Manila) all represent very different end-user use cases. Yes they all store stuff, but that's about the only thing they have in common. And no user in their sane mind would use all of them at the same time (if they do, they are probably doing it wrong). I guess we could recognize more basic use cases. I.e. Object storage is a end-user use case all by itself, and it only requires Swift + Keystone. So we could have a Layer #1bis that is Swift + Keystone. But then we are back to blessing stuff as essential/official, and next thing we know, Sahara will knock on the TC door to get Map/Reduce recognized as essential layer-1-like end-user use case. I can see how a give-me-a-damn-instance layer-1 definition is restrictive, but that's the beauty and the predictability of Monty's proposal. It limits integration where it really matters, unleashes competition where it will help, and removes almost all of the badge-granting role of the TC. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Hey On Thu, 2014-09-18 at 11:53 -0700, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Lots of great stuff here, but too much to respond to everything in detail. I love the way you've framed this in terms of the needs of developers, distributors, deployers and end users. I'd like to make sure we're focused on tackling those places where we're failing these groups, so: - Developers I think we're catering pretty well to developers with the big tent concept of Programs. There's been some good discussion about how Programs could be better at embracing projects in their related area, and that would be great to pursue. But the general concept - of endorsing and empowering teams of people collaborating in the OpenStack way - has a lot of legs, I think. I also think our release cadence does a pretty good job of serving developers. We've talked many times about the benefit of it, and I'd like to see it applied to more than just the server projects. OTOH, the integrated gate is straining, and a source of frustration for everyone. You raise the question of whether everything currently in the integrated release needs to be co-gated, and I totally agree that needs re-visiting. - Distributors We may be doing a better job of catering to distributors than any other group. For example, things like the release cadence, stable branch and common set of dependencies works pretty well. . The concept of an integrated release (with an incubation process) is great, because it nicely defines a set of stuff that distributors should ship. Certainly, life would be more difficult for distributors if there was a smaller set of projects in the release and a whole bunch of other projects which are interesting to distro users, but with an ambiguous level of commitment from our community. Right now, our integration process has a huge amount of influence over what gets shipped by distros, and that in turn serves distro users by ensuring a greater level of commonality between distros. - Operators I think the feedback we've been getting over the past few cycles suggests we are failing this group the most. Operators want to offer a compelling set of services to their users, but they want those services to be stable, performant, and perhaps most importantly, cost-effective. No operator wants to have to invest a huge amount of time in getting a new service running. You suggest a Production Ready tag. Absolutely - our graduation of projects has been interpreted as meaning production ready, when it's actually more useful as a signal to distros rather than operators. Graduation does not necessarily imply that a service is ready for production, no matter how you define production. I'd like to think we could give more nuanced advice to operators than a simple tag, but perhaps the information gathering process that projects would need to go through to be awarded that tag would uncover the more detailed information for operators. You could question whether the TC is the right body for this process. How might it work if the User Committee owned this? There are many other ways we can and should help operators, obviously, but this setting expectations is the aspect most relevant to this discussion. - End users You're right that we don't pay sufficient attention to this group. For me, the highest priority challenge here is interoperability. Particularly interoperability between public clouds. The only real interop effort to date you can point to is the board-owned DefCore and RefStack efforts. The idea being that a trademark program with API testing requirements will focus minds on interoperability. I'd love us (as a community) to be making more rapid progress on interoperability, but at least there are no encouraging signs that we should make some definite progress soon. Your end-user focused concrete suggestions (#7-#10) are interesting, and I find myself thinking about how much of a positive effect on interop each of them would have. For example, making our tools multi-cloud aware would help encourage people to demand interop from their providers. I also agree that end-user tools should support older versions of our APIs, but don't think that necessarily implies rolling releases. So, if I was to pick the areas which I think would address our most pressing challenges: 1) Shrinking the integrated gate, and allowing per-project testing strategies other than shoving every integrated project into the gate. 2) Giving more direction to operators about the readiness of our projects for different use cases. A process around awarding
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Great conversations here. I'd like to echo Dean Troyer's comment on Suggestion 9, for multi-cloud span node pooling ,we need standards. It'll make life easier when user tools could be configured against a limit as well as standard set of rules, instead of numerous different rules by vendors. It is gonna be a difficult task, since this kind of thing is always hard to do, but some form of intra cloud standard has to be introduced and worked on so users could actually have a resource pool Best Regards Zhipeng -- Zhipeng Huang Research Assistant Mobile Ad-Hoc Network Lab, Calit2 University of California, Irvine Email: zhipe...@uci.edu Office: Calit2 Building Room 2402 OpenStack, OpenDaylight, OpenCompute affcienado ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
FWIW, I think this is a great approach to evolving our thinking of the projects and ecosystem around OpenStack. I’m far too removed these days from the details of the day-to-day running of the programs and projects to comment on details. But, I’ve long felt a need to go beyond the simple core + everyone else thinking. This is moving in the right direction IMHO. Troy On Sep 22, 2014, at 5:49 AM, Mark McLoughlin mar...@redhat.com wrote: Hey On Thu, 2014-09-18 at 11:53 -0700, Monty Taylor wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Lots of great stuff here, but too much to respond to everything in detail. I love the way you've framed this in terms of the needs of developers, distributors, deployers and end users. I'd like to make sure we're focused on tackling those places where we're failing these groups, so: - Developers I think we're catering pretty well to developers with the big tent concept of Programs. There's been some good discussion about how Programs could be better at embracing projects in their related area, and that would be great to pursue. But the general concept - of endorsing and empowering teams of people collaborating in the OpenStack way - has a lot of legs, I think. I also think our release cadence does a pretty good job of serving developers. We've talked many times about the benefit of it, and I'd like to see it applied to more than just the server projects. OTOH, the integrated gate is straining, and a source of frustration for everyone. You raise the question of whether everything currently in the integrated release needs to be co-gated, and I totally agree that needs re-visiting. - Distributors We may be doing a better job of catering to distributors than any other group. For example, things like the release cadence, stable branch and common set of dependencies works pretty well. . The concept of an integrated release (with an incubation process) is great, because it nicely defines a set of stuff that distributors should ship. Certainly, life would be more difficult for distributors if there was a smaller set of projects in the release and a whole bunch of other projects which are interesting to distro users, but with an ambiguous level of commitment from our community. Right now, our integration process has a huge amount of influence over what gets shipped by distros, and that in turn serves distro users by ensuring a greater level of commonality between distros. - Operators I think the feedback we've been getting over the past few cycles suggests we are failing this group the most. Operators want to offer a compelling set of services to their users, but they want those services to be stable, performant, and perhaps most importantly, cost-effective. No operator wants to have to invest a huge amount of time in getting a new service running. You suggest a Production Ready tag. Absolutely - our graduation of projects has been interpreted as meaning production ready, when it's actually more useful as a signal to distros rather than operators. Graduation does not necessarily imply that a service is ready for production, no matter how you define production. I'd like to think we could give more nuanced advice to operators than a simple tag, but perhaps the information gathering process that projects would need to go through to be awarded that tag would uncover the more detailed information for operators. You could question whether the TC is the right body for this process. How might it work if the User Committee owned this? There are many other ways we can and should help operators, obviously, but this setting expectations is the aspect most relevant to this discussion. - End users You're right that we don't pay sufficient attention to this group. For me, the highest priority challenge here is interoperability. Particularly interoperability between public clouds. The only real interop effort to date you can point to is the board-owned DefCore and RefStack efforts. The idea being that a trademark program with API testing requirements will focus minds on interoperability. I'd love us (as a community) to be making more rapid progress on interoperability, but at least there are no encouraging signs that we should make some definite progress soon. Your end-user focused concrete suggestions (#7-#10) are interesting, and I find myself thinking about how much of a positive effect on interop each of them would have. For example, making our tools multi-cloud aware would help encourage people to demand interop from their providers. I also agree that end-user tools should
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 19, 2014, at 10:37 PM, Monty Taylor mord...@inaugust.com wrote: On 09/19/2014 03:29 AM, Thierry Carrez wrote: Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster competition/innovation within our community, and generally address the problem of community scaling. There are a few details on the consequences though, and in those as always the devil lurks. ## The Technical Committee The Technical Committee is defined in the OpenStack bylaws, and is the representation of the contributors to the project. Teams work on code repositories, and at some point ask their work to be recognized as part of OpenStack. In doing so, they place their work under the oversight of the Technical Committee. In return, team members get to participate in electing the technical committee members (they become ATC). It's a balanced system, where both parties need to agree: the TC can't force itself as overseer of a random project, and a random project can't just decide by itself it is OpenStack. I don't see your proposal breaking that balanced system, but it changes its dynamics a bit. The big tent would contain a lot more members. And while the TC would arguably bring a significant share of its attention to Ring 0, its voters constituency would mostly consist of developers who do not participate in Ring 0 development. I don't really see it as changing dramatically the membership of the TC, but it's a consequence worth mentioning. Agree. I'm willing to bet it'll be better not worse to have a large constituency - but it's also problem that it's a giant disaster. I'm still on board with going for it. ## Programs Programs were created relatively recently as a way to describe which teams are in OpenStack vs. which ones aren't. They directly tie into the ATC system: if you contribute to code repositories under a blessed program, then you're an ATC, you vote in TC elections and the TC has some oversight over your code repositories. Previously, this was granted at a code repository level, but that failed to give flexibility for teams to organize their code in the most convenient manner for them. So we started to bless teams rather than specific code repositories. Now, that didn't work out so well. Some programs were a 'theme', like Infrastructure, or Docs. For those, competing efforts do not really make sense: there can only be one, and competition should happen inside those efforts rather than outside. Some programs were a 'team', like Orchestration/Heat or Deployment/TripleO. And that's where the model started to break: some new orchestration things need space, but the current Heat team is not really interested in maintaining them. What's the point of being under the same program then ? And TripleO is not the only way to deploy OpenStack, but its mere existence (and name) prevented other flowers to bloom in our community. You don't talk much about programs in your proposal. In particular, you only mention layer 1, Cloud Native applications, User Interface applications, and Operator applications. So I'm unsure of where, if anywhere, would Infrastructure or Docs repositories live. Here is how I see it could work. We could keep 'theme' programs (think Infra, Release cycle management, Docs, QA) with their current structure (collection of code repositories under a single team/PTL). We would get rid of 'team' programs completely, and just have a registry of OpenStack code repositories (openstack*/*, big tent). Each of those could have a specific PTL, or explicitely inherit its PTL from another code repository. Under that PTL, they could have separate or same core review team, whatever maps reality and how they want to work (so we could say that openstack/python-novaclient inherits its PTL from openstack/nova and doesn't need a specific one). That would allow us to map anything that may come in. Oslo falls a bit in between, could be considered a 'theme' program or several repos sharing PTL. I think we can do what you're saying and generalize a little bit. What if we declared programs, as needed, when we think there is a need to pick a winner. (I think we can all agree that early winner picking is an unintended but very real side effect of the current system) And when I say need to - I mean it in the same sense as Production Ready The themes you listed are excellent ones - it makes no sense to have two Infras, two QAs or two Docs teams. On
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 19, 2014, at 6:29 AM, Thierry Carrez thie...@openstack.org wrote: Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster I’m not sure I see this change reducing the number of incubated projects unless we no longer incubate and graduate projects at all. Would everything just live on stackforge and have a quality designation instead of an “officialness” designation? Or would we have both? ATC status seems to imply we need some sort of officialness designation, as you mention below. competition/innovation within our community, and generally address the problem of community scaling. There are a few details on the consequences though, and in those as always the devil lurks. ## The Technical Committee The Technical Committee is defined in the OpenStack bylaws, and is the representation of the contributors to the project. Teams work on code repositories, and at some point ask their work to be recognized as part of OpenStack. In doing so, they place their work under the oversight of the Technical Committee. In return, team members get to participate in electing the technical committee members (they become ATC). It's a balanced system, where both parties need to agree: the TC can't force itself as overseer of a random project, and a random project can't just decide by itself it is OpenStack. I don't see your proposal breaking that balanced system, but it changes its dynamics a bit. The big tent would contain a lot more members. And while the TC would arguably bring a significant share of its attention to Ring 0, its voters constituency would mostly consist of developers who do not participate in Ring 0 development. I don't really see it as changing dramatically the membership of the TC, but it's a consequence worth mentioning. ## Programs Programs were created relatively recently as a way to describe which teams are in OpenStack vs. which ones aren't. They directly tie into the ATC system: if you contribute to code repositories under a blessed program, then you're an ATC, you vote in TC elections and the TC has some oversight over your code repositories. Previously, this was granted at a code repository level, but that failed to give flexibility for teams to organize their code in the most convenient manner for them. So we started to bless teams rather than specific code repositories. Now, that didn't work out so well. Some programs were a 'theme', like Infrastructure, or Docs. For those, competing efforts do not really make sense: there can only be one, and competition should happen inside those efforts rather than outside. Some programs were a 'team', like Orchestration/Heat or Deployment/TripleO. And that's where the model started to break: some new orchestration things need space, but the current Heat team is not really interested in maintaining them. What's the point of being under the same program then ? And TripleO is not the only way to deploy OpenStack, but its mere existence (and name) prevented other flowers to bloom in our community. You don't talk much about programs in your proposal. In particular, you only mention layer 1, Cloud Native applications, User Interface applications, and Operator applications. So I'm unsure of where, if anywhere, would Infrastructure or Docs repositories live. Here is how I see it could work. We could keep 'theme' programs (think Infra, Release cycle management, Docs, QA) with their current structure (collection of code repositories under a single team/PTL). We would get rid of 'team' programs completely, and just have a registry of OpenStack code repositories (openstack*/*, big tent). Each of those could have a specific PTL, or explicitely inherit its PTL from another code repository. Under that PTL, they could have separate or same core review team, whatever maps reality and how they want to work (so we could say that openstack/python-novaclient inherits its PTL from openstack/nova and doesn't need a specific one). That would allow us to map anything that may come in. Oslo falls a bit in between, could be considered a 'theme' program or several repos sharing PTL. I like the idea of chartered programs as a way to tackling our cross-project issues. We need fewer of programs, but for the ones we do need we still need to integrate them with the rest of our governance. ## The release and the development cycle You touch briefly on the consequences of your model for the common
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 18, 2014, at 2:53 PM, Monty Taylor mord...@inaugust.com wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Enjoy. I’ve read through this a few times now, and I think I can support most of it. It doesn’t address some of the issues we have, but most of the concrete proposals seem like they take us to incrementally better places than where we are now. I definitely like the idea of making the integrated gate different for each project, based on the other projects it actually integrates with. I could see extending the two-project gate idea for projects outside of layer 1 to include more than 2 projects eventually. The assumption that all layer 1 projects can depend on the other members of layer 1 being present may have ramifications for the trademark “core” definition. I don’t think those cause problems, based on the mechanisms for defining capabilities and designated sections being worked out now, but it’s worth pointing out as something we’ll need to keep in mind. The self-organizing groupings that Vish, John, and others have mentioned may well lead us to create additional trademarks in a more naturally evolving way than the single big mark we’re trying to squeeze everything into now. Does the unified client SDK fit into layer 1 as one of the “common libraries … necessary for these”? Or do we anticipate the services still using their own individual libraries for talking to each other? Having a quality designation will help distros and deployers. I’m not sure we want Cern specifically to be our arbiter of that quality, but I do like the idea of having users be involved in the determination. Maybe it's something the User Committee could help with in a more general way. This proposal only addresses some of the challenges we have right now. If we maintain a big tent approach, and I think we should, we still need a way to implement cross-project policies and initiatives outside of the scope of any one of our existing programs. I agree with Vish that we need a different name for Layer 1. A name that doesn’t imply “leveling” or “layering” would be good, since some of the cloud-native services don’t build on those layer 1 services. Doug ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 22 Sep 2014, at 20:53, Doug Hellmann d...@doughellmann.com wrote: On Sep 19, 2014, at 6:29 AM, Thierry Carrez thie...@openstack.org wrote: Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster I’m not sure I see this change reducing the number of incubated projects unless we no longer incubate and graduate projects at all. Would everything just live on stackforge and have a quality designation instead of an “officialness” designation? Or would we have both? ATC status seems to imply we need some sort of officialness designation, as you mention below. The quality designation is really important for the operator community who are trying to work out what we can give to our end users. Offering early helps to establish the real-life experience and give good feedback on the designs. However, the operator then risks leaving their users orphaned if the project does not get a sustainable following or significant disruption if the APIs change. The packaging teams are key here as well. When do Ubuntu and Red Hat work out the chain of pre-reqs etc. to produce installable packages, packstack/juju tool support ? We do need to have some way to show that an layer #2 package is ready for prime time production and associated criteria (packages available, docs available, 1 company communities, models for HA and scale, …) Tim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 22, 2014, at 3:11 PM, Tim Bell tim.b...@cern.ch wrote: On 22 Sep 2014, at 20:53, Doug Hellmann d...@doughellmann.com wrote: On Sep 19, 2014, at 6:29 AM, Thierry Carrez thie...@openstack.org wrote: Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster I’m not sure I see this change reducing the number of incubated projects unless we no longer incubate and graduate projects at all. Would everything just live on stackforge and have a quality designation instead of an “officialness” designation? Or would we have both? ATC status seems to imply we need some sort of officialness designation, as you mention below. The quality designation is really important for the operator community who are trying to work out what we can give to our end users. Offering early helps to establish the real-life experience and give good feedback on the designs. However, the operator then risks leaving their users orphaned if the project does not get a sustainable following or significant disruption if the APIs change. The packaging teams are key here as well. When do Ubuntu and Red Hat work out the chain of pre-reqs etc. to produce installable packages, packstack/juju tool support ? We do need to have some way to show that an layer #2 package is ready for prime time production and associated criteria (packages available, docs available, 1 company communities, models for HA and scale, …) Right. I’m trying to understand if we are talking about doing that *instead* of our existing incubation/graduation process, or in addition to that process as a new thing. I like the idea of adding a quality designation. I’m not sure replacing our existing process with that designation is a good idea. Doug Tim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Mon, Sep 22, 2014 at 12:33 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 3:11 PM, Tim Bell tim.b...@cern.ch wrote: On 22 Sep 2014, at 20:53, Doug Hellmann d...@doughellmann.com wrote: On Sep 19, 2014, at 6:29 AM, Thierry Carrez thie...@openstack.org wrote: Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster I’m not sure I see this change reducing the number of incubated projects unless we no longer incubate and graduate projects at all. Would everything just live on stackforge and have a quality designation instead of an “officialness” designation? Or would we have both? ATC status seems to imply we need some sort of officialness designation, as you mention below. The quality designation is really important for the operator community who are trying to work out what we can give to our end users. Offering early helps to establish the real-life experience and give good feedback on the designs. However, the operator then risks leaving their users orphaned if the project does not get a sustainable following or significant disruption if the APIs change. The packaging teams are key here as well. When do Ubuntu and Red Hat work out the chain of pre-reqs etc. to produce installable packages, packstack/juju tool support ? We do need to have some way to show that an layer #2 package is ready for prime time production and associated criteria (packages available, docs available, 1 company communities, models for HA and scale, …) Right. I’m trying to understand if we are talking about doing that *instead* of our existing incubation/graduation process, or in addition to that process as a new thing. I like the idea of adding a quality designation. I’m not sure replacing our existing process with that designation is a good idea. A production ready tag wouldn't replace the incubation process because incubation isn't about that. Also, the incubation process would go away because the TC wouldn't be in the business of adding projects to the integrated gate any more. The content of integrated release would be based on a use-case-based scope definition, which is satisfied by the projects in the Layer #1 list. I think it's interesting to think about other use-cases, and those could well inform the trademark discussion in the long term, but I think we need to start with one use case, and I would like to think this is one we can all agree on as central to the mission of OpenStack. That doesn't invalidate other use cases. And it doesn't make things not in Layer #1 suddenly become not-OpenStack. (A triple negative? Yep.) One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC decision, and the decision to cogate with another project should be up to the projects themselves. If Nova feels that supporting the Ironic virt driver is important, Nova would agree to a two-project gate with Ironic. Other projects that Nova also gates with would not need to participate (eg, a change to Glance would also test Nova, but wouldn't necessarily run the nova-ironic test). I think this can be extended out to a big tent far better than our current gating system. Whether or not there is a process to apply for / get the production ready sticker would no longer be related to being part of the release process or gate tests. Of course, it's essential this includes feedback from users and operators. For what it's worth, I would want to see at least one production deployment of a project - and get feedback from the operators of said deployment - before I would feel comfortable calling it production-ready. -Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 22, 2014, at 5:10 PM, Devananda van der Veen devananda@gmail.com wrote: On Mon, Sep 22, 2014 at 12:33 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 3:11 PM, Tim Bell tim.b...@cern.ch wrote: On 22 Sep 2014, at 20:53, Doug Hellmann d...@doughellmann.com wrote: On Sep 19, 2014, at 6:29 AM, Thierry Carrez thie...@openstack.org wrote: Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster I’m not sure I see this change reducing the number of incubated projects unless we no longer incubate and graduate projects at all. Would everything just live on stackforge and have a quality designation instead of an “officialness” designation? Or would we have both? ATC status seems to imply we need some sort of officialness designation, as you mention below. The quality designation is really important for the operator community who are trying to work out what we can give to our end users. Offering early helps to establish the real-life experience and give good feedback on the designs. However, the operator then risks leaving their users orphaned if the project does not get a sustainable following or significant disruption if the APIs change. The packaging teams are key here as well. When do Ubuntu and Red Hat work out the chain of pre-reqs etc. to produce installable packages, packstack/juju tool support ? We do need to have some way to show that an layer #2 package is ready for prime time production and associated criteria (packages available, docs available, 1 company communities, models for HA and scale, …) Right. I’m trying to understand if we are talking about doing that *instead* of our existing incubation/graduation process, or in addition to that process as a new thing. I like the idea of adding a quality designation. I’m not sure replacing our existing process with that designation is a good idea. A production ready tag wouldn't replace the incubation process because incubation isn't about that. Also, the incubation process would go away because the TC wouldn't be in the business of adding projects to the integrated gate any more. The content of integrated release would be based on a use-case-based scope definition, which is satisfied by the projects in the Layer #1 list. I think it's interesting to think about other use-cases, and those could well inform the trademark discussion in the long term, but I think we need to start with one use case, and I would like to think this is one we can all agree on as central to the mission of OpenStack. That doesn't invalidate other use cases. And it doesn't make things not in Layer #1 suddenly become not-OpenStack. (A triple negative? Yep.) One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? Based on what you’ve said, I’m afraid that this change makes project governance murkier, and introduces challenges to accomplish some of the unifying cross-project work that we have recently discussed as being so important. decision, and the decision to cogate with another project should be up to the projects themselves. If Nova feels that supporting the Ironic virt driver is important, Nova would agree to a two-project gate with Ironic. Other projects that Nova also gates with would not need to participate (eg, a change to Glance would also test Nova, but wouldn't necessarily run the nova-ironic test). I think this can be extended out to a big tent far better than our current gating system. Whether or not there is a process to apply for / get the production ready sticker would no longer be related to being part of the release process or gate
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 19 September 2014 22:29, Thierry Carrez thie...@openstack.org wrote: ... current Heat team is not really interested in maintaining them. What's the point of being under the same program then ? And TripleO is not the only way to deploy OpenStack, but its mere existence (and name) prevented other flowers to bloom in our community. So I've a small issue here - *after* TripleO became an official program Tuskar was started - and we embraced and shuffled things to collaborate. There's a tripleo puppet repository, an ansible one coming soon I believe, and I'm hearing rumours of a spike using another thing altogether (whose thunder I don't want to steal so I'm going to be vague :)). We've collaborated to a degree with Fuel and Crowbar: I am not at all sure we've prevented other flowers blooming - and I hate the idea that we have done that. ... ## The release and the development cycle You touch briefly on the consequences of your model for the common release and our development cycle. Obviously we would release the ring 0 projects in an integrated manner on a time-based schedule. I'm not at all sure we need to do that - I've long been suspicious of the coordinated release. I see benefits to users in being able to grab a new set of projects all at once, but they can do that irrespective of our release process, as long as: *) we do releases *) we do at least one per project per 6 month period Tying all our things together makes for hugely destablising waves of changes and rushes: if we aimed at really smooth velocity and frequent independent releases I think we could do a lot better: contracts and interfaces are *useful* things for large scale architectures, and we need to stop kidding ourselves - OpenStack is not a lean little beastie anymore: its a large scale distributed system. Swift is doing exactly the right thing today - I'd like to see more of that. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
tl;dr: we're not broken, but under stress...changing (outside) expectations requires changing the expression of the model...while it's called a 'stack' maybe it's multiple tiered stacks. MultiStack! On Mon, Sep 22, 2014 at 4:27 PM, Doug Hellmann d...@doughellmann.com wrote: The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? This is one reason for multiple layers. The original 4 layer stack was meant as a technical dependency stack but has morphed into a social/project organizational stack. I don't think it is total coincidence that the technical hierarchy was interpreted as a social/governance hierarchy by some as there is a lot of similarity. The mapping between the two in my mind is fairly easy, but those details is not what is important. We love to look at the Apache Foundation for inspiration. In the current set of Apache projects most of them are not focused on adding value to a web server. They grew beyond that in multiple directions. I'm selfish and want to keep the layer nomenclature for the technical organization that helped re-structure DevStack and propose we think of these as *thing*[0] where we have Multiple *Things*: * IaaS thing: the stuff that builds excellent clouds * PaaS thing: the stuff that does amazing things that may or may not be built on top of excellent clouds * XaaS thing(s): more things I can not visualize through the fog of antihistamines * Non-aas developer things: what enables us s developers to make the above things (infra, qa, etc) * Non-aas consumer[1] things: what enables the rest of the world to enjoy the above things (docs, SDKs, clients, etc) This separates the technical hierarchical from the organizational relationships. All of the above things are still called OpenStack. But maybe it's a 'MultiStack'. - New projects wanting to join can apply and receive a 'provisional' attribute that tells the world we (OpenStack people) thing this looks promising and want to see if it matures to our standards. Similar to incubation but maybe the bar has some differences between the things. It _should_ require a higher bar to add to the foundation than a new deck on the side. - The integrated release still applies to a subset of the projects and/or things. The IaaS thing establishes the base release cycle other things apply common sense and follow along or release more often. - The test matrix is built using the technical layers and not the above organizational structure. I'm getting a bit hand-wavy here, but the point is to clearly state to the world that it isn't the organizational things that determine testing beyond having at least the 'provisional' attribute on a project. If the above feels very familiar it should. Some of it is terminology changes to help change expectations and some divisions based on use cases. What we have isn't totally broken, it is under growth related stress. Taking a different perspective on it will help expose where things can be improved. And what we are doing right. dt [0] TODO: cut-n-paste in actual allegorical noun [1] Yuck, better word here too! -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 23 September 2014 10:32, Dean Troyer dtro...@gmail.com wrote: tl;dr: we're not broken, but under stress...changing (outside) expectations requires changing the expression of the model...while it's called a 'stack' maybe it's multiple tiered stacks. MultiStack! ... This is one reason for multiple layers. The original 4 layer stack was meant as a technical dependency stack but has morphed into a social/project organizational stack. I don't think it is total coincidence that the technical hierarchy was interpreted as a social/governance hierarchy by some as there is a lot of similarity. The mapping between the two in my mind is fairly easy, but those details is not what is important. Are you familiar with Conway's law? http://en.wikipedia.org/wiki/Conway's_law - I think it works both ways. When we drive towards a particular structure be it technical or social, expect the other side to follow suit. Its very hard to to have a social structure decoupled from the technical structure. One of the reasons Launchpad has such poor cross-component (bugs/code hosting/translations) features compared to the in-component features is that for a long time there were separate teams for each thing - which resulted in great domain knowledge, but the vast amount of work being done in-component. If we want to be better at 'cross project' things, I think we need to factor Conway's law into our *social* design process. I don't doubt we need separate teams - we're way past Dunbar's number, for any estimate of it - but I think we need to seriously consider making our teams align with our user needs, not our codebases. Codebases are an implementation detail - an important one, but a detail. Our goals, our focus, *our structure* should be on meeting user needs [including those users that do deployments], rather than structured around our code. * IaaS thing: the stuff that builds excellent clouds * PaaS thing: the stuff that does amazing things that may or may not be built on top of excellent clouds * XaaS thing(s): more things I can not visualize through the fog of antihistamines * Non-aas developer things: what enables us s developers to make the above things (infra, qa, etc) * Non-aas consumer[1] things: what enables the rest of the world to enjoy the above things (docs, SDKs, clients, etc) This separates the technical hierarchical from the organizational relationships. All of the above things are still called OpenStack. But maybe it's a 'MultiStack'. I think this would result in excellent intra-layer features/quality etc, but perhaps cross-layer would become the poor stepchild. And - I think thats ok. I think it would help overall. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Mon, Sep 22, 2014 at 2:27 PM, Doug Hellmann d...@doughellmann.com wrote: On Sep 22, 2014, at 5:10 PM, Devananda van der Veen devananda@gmail.com wrote: One of the primary effects of integration, as far as the release process is concerned, is being allowed to co-gate with other integrated projects, and having those projects accept your changes (integrate back with the other project). That shouldn't be a TC The point of integration is to add the projects to the integrated *release*, not just the gate, because the release is the thing we have said is OpenStack. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. We have plenty of things which are clearly part of OpenStack, and yet which are not part of the Integrated Release. Oslo. Devstack. Zuul... As far as I can tell, the only time when integrated release equals the thing we say is OpenStack is when we're talking about the trademark. Integration was about our overall project identity and governance. The testing was a requirement to be accepted, not a goal. Project identity and governance are presently addressed by the creation of Programs and a fully-elected TC. Integration is not addressing these things at all, as far as I can tell, though I agree that it was initially intended to. If there is no incubation process, and only a fixed list of projects will be in that new layer 1 group, then do contributors to the other projects have ATC status and vote for the TC? What is the basis for the TC accepting any responsibility for the project, and for the project agreeing to the TC’s leadership? I think a good basis for this is simply whether the developers of the project are part of our community, doing things in the way that we do things, and want this to happen. Voting and ATC status is already decoupled [0] from the integrated gate and the integrated release -- it's based on the accepted list of Programs [1], which actually has nothing to do with incubation or integration [2]. -Devananda [0] http://git.openstack.org/cgit/openstack/governance/tree/reference/charter.rst#n132 [1] http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml [2] http://git.openstack.org/cgit/openstack/governance/tree/reference/new-programs-requirements.rst ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster competition/innovation within our community, and generally address the problem of community scaling. There are a few details on the consequences though, and in those as always the devil lurks. ## The Technical Committee The Technical Committee is defined in the OpenStack bylaws, and is the representation of the contributors to the project. Teams work on code repositories, and at some point ask their work to be recognized as part of OpenStack. In doing so, they place their work under the oversight of the Technical Committee. In return, team members get to participate in electing the technical committee members (they become ATC). It's a balanced system, where both parties need to agree: the TC can't force itself as overseer of a random project, and a random project can't just decide by itself it is OpenStack. I don't see your proposal breaking that balanced system, but it changes its dynamics a bit. The big tent would contain a lot more members. And while the TC would arguably bring a significant share of its attention to Ring 0, its voters constituency would mostly consist of developers who do not participate in Ring 0 development. I don't really see it as changing dramatically the membership of the TC, but it's a consequence worth mentioning. ## Programs Programs were created relatively recently as a way to describe which teams are in OpenStack vs. which ones aren't. They directly tie into the ATC system: if you contribute to code repositories under a blessed program, then you're an ATC, you vote in TC elections and the TC has some oversight over your code repositories. Previously, this was granted at a code repository level, but that failed to give flexibility for teams to organize their code in the most convenient manner for them. So we started to bless teams rather than specific code repositories. Now, that didn't work out so well. Some programs were a 'theme', like Infrastructure, or Docs. For those, competing efforts do not really make sense: there can only be one, and competition should happen inside those efforts rather than outside. Some programs were a 'team', like Orchestration/Heat or Deployment/TripleO. And that's where the model started to break: some new orchestration things need space, but the current Heat team is not really interested in maintaining them. What's the point of being under the same program then ? And TripleO is not the only way to deploy OpenStack, but its mere existence (and name) prevented other flowers to bloom in our community. You don't talk much about programs in your proposal. In particular, you only mention layer 1, Cloud Native applications, User Interface applications, and Operator applications. So I'm unsure of where, if anywhere, would Infrastructure or Docs repositories live. Here is how I see it could work. We could keep 'theme' programs (think Infra, Release cycle management, Docs, QA) with their current structure (collection of code repositories under a single team/PTL). We would get rid of 'team' programs completely, and just have a registry of OpenStack code repositories (openstack*/*, big tent). Each of those could have a specific PTL, or explicitely inherit its PTL from another code repository. Under that PTL, they could have separate or same core review team, whatever maps reality and how they want to work (so we could say that openstack/python-novaclient inherits its PTL from openstack/nova and doesn't need a specific one). That would allow us to map anything that may come in. Oslo falls a bit in between, could be considered a 'theme' program or several repos sharing PTL. ## The release and the development cycle You touch briefly on the consequences of your model for the common release and our development cycle. Obviously we would release the ring 0 projects in an integrated manner on a time-based schedule. For the other projects, we have a choice: - the ring 0 model (development milestones, coordinated final release) - the Swift model (release as-needed, coordinated final release) - the Oslo model (release as-needed, try to have one final one before end of cycle) - the Client libraries model (release as-needed) If possible, I would like to avoid the Swift model, which is the most costly from a release management standpoint. All projects following the ring 0 model are easy to keep track of, using common freezes etc. So it's easy to make sure they will be ready in time for the coordinated
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Fri, Sep 19, 2014 at 4:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Great stuff, mixed on point 2 raised by Vish but honestly I think that's something that could evolve over time, but I looked at that differently as in Cinder, SWIFT and some day Manilla live under a Storage Services umbrella, and ideally at some point there's some convergence there. Anyway, I don't want to start a rat-hole on that, it's kind of irrelevant right now. Bottom line is I think the direction and initial ideas in Monty's post are what a lot of us have been thinking about and looking for. I'm in!! ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Can someone do a small little Visio or other visual to explain what's being proposed here? My head sported a small crack at around the 5-6th page... ; ) But seriously, I couldn't understand the proposal. Maybe I'm not the audience which is fine, just saying, the words got in the way. Sounds like a song! *Adam Lawson* AQORN, Inc. 427 North Tatnall Street Ste. 58461 Wilmington, Delaware 19801-2230 Toll-free: (844) 4-AQORN-NOW ext. 101 International: +1 302-387-4660 Direct: +1 916-246-2072 On Fri, Sep 19, 2014 at 5:46 AM, John Griffith john.griff...@solidfire.com wrote: On Fri, Sep 19, 2014 at 4:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Great stuff, mixed on point 2 raised by Vish but honestly I think that's something that could evolve over time, but I looked at that differently as in Cinder, SWIFT and some day Manilla live under a Storage Services umbrella, and ideally at some point there's some convergence there. Anyway, I don't want to start a rat-hole on that, it's kind of irrelevant right now. Bottom line is I think the direction and initial ideas in Monty's post are what a lot of us have been thinking about and looking for. I'm in!! ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 19, 2014, at 5:46 AM, John Griffith john.griff...@solidfire.com wrote: On Fri, Sep 19, 2014 at 4:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Great stuff, mixed on point 2 raised by Vish but honestly I think that's something that could evolve over time, but I looked at that differently as in Cinder, SWIFT and some day Manilla live under a Storage Services umbrella, and ideally at some point there's some convergence there. Anyway, I don't want to start a rat-hole on that, it's kind of irrelevant right now. Bottom line is I think the direction and initial ideas in Monty's post are what a lot of us have been thinking about and looking for. I'm in!! I too am generally supportive of the concept, but I do want to think about the vishy/tts/jgriffith points above. It's interesting that the proposed layer #1 stuff is very very similar to what was originally in OpenStack at the very beginning as Nova. Over time, many of these pieces of functionality required for compute were split out (block, networking, image, etc), and I think that's why so many people look at these pieces and say (rightly), of course these are required all together and tightly coupled. That's how these projects started, and we still see evidence of their birth today. For that reason, I do agree with Vish that there should be similar collaborations for other things. While the layer #1 (or compute) use case is very common, we can all see that it's not the only one that people are solving with OpenStack parts. And this is reflected in the products build and sold by companies, too. Some sell one subset of openstack stuff as product X and maybe a different subset as product Y. (The most common example here is compute vs object storage.) This reality has led to a lot of the angst around definitions since there is effort to define openstack all as one thing (or worse, as a base thing that others are defined as built upon). I propose that we can get the benefits of Monty's proposal and implement all of his concrete suggestions (which are fantastic) by slightly adjusting our usage of the program/project concepts. I had originally hoped that the program concept would have been a little higher-level instead of effectively spelling project as program. I'd love to see a hierarchy of openstack-program-project/team-repos. Right now, we have added the program layer but have effectively mapped it 1:1 to the project. For example, we used to have a few repos in the Swift project managed by the same group of people, and now we have a few repos in the object storage program, all managed by the same group of people. And every time something is added to OpenStack, its added as a new program, effectively putting us exactly where we were before we called it a program with the same governance and management scaling problems. Today, I'd group existing OpenStack projects into programs as follows: Compute: nova, sahara, ironic Storage: swift, cinder, glance, trove Network: neutron, designate, zaqar Deployment/management: heat, triple-o, horizon, ceilometer Identity: keystone, barbican Support (not user facing): infra, docs, tempest, devstack, oslo (potentially even) stackforge: lots I like two things about this. First, it allows people to compose a solution. Second, it allows us as a community to thing more about the strategic/product things. For example, it lets us as a community say, We think storage is important. How are we solving it today? What gaps do we have in that? How can the various storage things we have work together better? Thierry makes the point that more themes will nullify the benefits of Monty's proposal. I agree, if we continue to allow the explosion of projects/programs/themes to continue. The benefit of what Monty is proposing is that it identifies and focusses on a particular use case (deploy a VM, add a volume, get an IP, configure a domain) so that we know we have solved it well. I think that focus is vital, and by grouping
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 19, 2014, at 3:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. I was assuming that these would be self-organized rather than managed by the TC. Vish signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Fri, Sep 19, 2014 at 12:02 PM, Adam Lawson alaw...@aqorn.com wrote: Can someone do a small little Visio or other visual to explain what's being Sean's blog post included a nice diagram that is Monty's starting point: https://dague.net/2014/08/26/openstack-as-layers/ AIUI Monty's Layer 1 is basically the original layers 1+2. I had separated them originally as layer 2 is optional from a purely technical perspective, but not really from a cloud user perspective. layers tl;dr: For my purposes layers 1, 2 and a hand-wavey 'everything else' is useful. Others find combing layers 1 and 2 more useful, especially for organizational and other purposes. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Sep 19, 2014, at 10:14 AM, John Dickinson m...@not.mn wrote: On Sep 19, 2014, at 5:46 AM, John Griffith john.griff...@solidfire.com wrote: On Fri, Sep 19, 2014 at 4:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Great stuff, mixed on point 2 raised by Vish but honestly I think that's something that could evolve over time, but I looked at that differently as in Cinder, SWIFT and some day Manilla live under a Storage Services umbrella, and ideally at some point there's some convergence there. Anyway, I don't want to start a rat-hole on that, it's kind of irrelevant right now. Bottom line is I think the direction and initial ideas in Monty's post are what a lot of us have been thinking about and looking for. I'm in!! I too am generally supportive of the concept, but I do want to think about the vishy/tts/jgriffith points above. Today, I'd group existing OpenStack projects into programs as follows: Compute: nova, sahara, ironic Storage: swift, cinder, glance, trove Network: neutron, designate, zaqar Deployment/management: heat, triple-o, horizon, ceilometer Identity: keystone, barbican Support (not user facing): infra, docs, tempest, devstack, oslo (potentially even) stackforge: lots There is a pretty different division of things in this breakdown than in what monty was proposing. This divides things up by conceptual similarity which I think is actually less useful than breaking things down by use case. I really like the grouping and testing of things which are tightly coupled. If we say launching a VM and using it is the primary use case of our community corrently then things group into monty’s layer #1. It seems fairly clear that a large section of our community is focused on this use case so this should be a primary focus of infrastructure resources. There are other use cases in our community, for example: Object Storage: Swift (depends on keystone) Orchestrating Multiple VMs: Heat (depends on layer1) DBaSS: Trove: (depends on heat) These are also important use cases for parts of our community, but swift has demostrated that it isn’t required to be a part of an integrated release schedule, so these could be managed by smaller groups in the community. Note that these are primarily individual projects today, but I could see a future where some of these projects decide to group and do an integrated release. In the future we might see (totally making this up): Public Cloud Application services: Trove, Zaqar Application Deployment services: Heat, Murano Operations services: Ceilometer, Congress As I mentioned previously though, I don’t think we need to define these groups in advance. These groups can organize as needed. Vish signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/19/2014 01:15 PM, Vishvananda Ishaya wrote: On Sep 19, 2014, at 3:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. I was assuming that these would be self-organized rather than managed by the TC. Vish Some groups are able to self-organize better than others, I have been hoping the third party theme would self-organize for the last 10 months and while some folks are trying, their numbers don't keep up with the demand. This group insists that someone else tell them what to do. It is not a scalable model, it also doesn't inspire anyone outside of the theme to want to help out with the answering of questions either. Just as a data point. Thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/19/2014 03:29 AM, Thierry Carrez wrote: Monty Taylor wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Hey Monty, As you can imagine, I read that post with great attention. I generally like the concept of a tightly integrated, limited-by-design layer #1 (I'd personally call it Ring 0) and a large collection of OpenStack things gravitating around it. That would at least solve the attraction of the integrated release, suppress the need for incubation, foster competition/innovation within our community, and generally address the problem of community scaling. There are a few details on the consequences though, and in those as always the devil lurks. ## The Technical Committee The Technical Committee is defined in the OpenStack bylaws, and is the representation of the contributors to the project. Teams work on code repositories, and at some point ask their work to be recognized as part of OpenStack. In doing so, they place their work under the oversight of the Technical Committee. In return, team members get to participate in electing the technical committee members (they become ATC). It's a balanced system, where both parties need to agree: the TC can't force itself as overseer of a random project, and a random project can't just decide by itself it is OpenStack. I don't see your proposal breaking that balanced system, but it changes its dynamics a bit. The big tent would contain a lot more members. And while the TC would arguably bring a significant share of its attention to Ring 0, its voters constituency would mostly consist of developers who do not participate in Ring 0 development. I don't really see it as changing dramatically the membership of the TC, but it's a consequence worth mentioning. Agree. I'm willing to bet it'll be better not worse to have a large constituency - but it's also problem that it's a giant disaster. I'm still on board with going for it. ## Programs Programs were created relatively recently as a way to describe which teams are in OpenStack vs. which ones aren't. They directly tie into the ATC system: if you contribute to code repositories under a blessed program, then you're an ATC, you vote in TC elections and the TC has some oversight over your code repositories. Previously, this was granted at a code repository level, but that failed to give flexibility for teams to organize their code in the most convenient manner for them. So we started to bless teams rather than specific code repositories. Now, that didn't work out so well. Some programs were a 'theme', like Infrastructure, or Docs. For those, competing efforts do not really make sense: there can only be one, and competition should happen inside those efforts rather than outside. Some programs were a 'team', like Orchestration/Heat or Deployment/TripleO. And that's where the model started to break: some new orchestration things need space, but the current Heat team is not really interested in maintaining them. What's the point of being under the same program then ? And TripleO is not the only way to deploy OpenStack, but its mere existence (and name) prevented other flowers to bloom in our community. You don't talk much about programs in your proposal. In particular, you only mention layer 1, Cloud Native applications, User Interface applications, and Operator applications. So I'm unsure of where, if anywhere, would Infrastructure or Docs repositories live. Here is how I see it could work. We could keep 'theme' programs (think Infra, Release cycle management, Docs, QA) with their current structure (collection of code repositories under a single team/PTL). We would get rid of 'team' programs completely, and just have a registry of OpenStack code repositories (openstack*/*, big tent). Each of those could have a specific PTL, or explicitely inherit its PTL from another code repository. Under that PTL, they could have separate or same core review team, whatever maps reality and how they want to work (so we could say that openstack/python-novaclient inherits its PTL from openstack/nova and doesn't need a specific one). That would allow us to map anything that may come in. Oslo falls a bit in between, could be considered a 'theme' program or several repos sharing PTL. I think we can do what you're saying and generalize a little bit. What if we declared programs, as needed, when we think there is a need to pick a winner. (I think we can all agree that early winner picking is an unintended but very real side effect of the current system) And when I say need to - I mean it in the same sense as Production Ready The themes you listed are excellent ones - it makes no sense to have two Infras, two QAs or two Docs teams. On the other hand, maybe someone else wants to take a stab at the general problem space that
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/19/2014 10:14 AM, John Dickinson wrote: On Sep 19, 2014, at 5:46 AM, John Griffith john.griff...@solidfire.com wrote: On Fri, Sep 19, 2014 at 4:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Great stuff, mixed on point 2 raised by Vish but honestly I think that's something that could evolve over time, but I looked at that differently as in Cinder, SWIFT and some day Manilla live under a Storage Services umbrella, and ideally at some point there's some convergence there. Anyway, I don't want to start a rat-hole on that, it's kind of irrelevant right now. Bottom line is I think the direction and initial ideas in Monty's post are what a lot of us have been thinking about and looking for. I'm in!! I too am generally supportive of the concept, but I do want to think about the vishy/tts/jgriffith points above. It's interesting that the proposed layer #1 stuff is very very similar to what was originally in OpenStack at the very beginning as Nova. Over time, many of these pieces of functionality required for compute were split out (block, networking, image, etc), and I think that's why so many people look at these pieces and say (rightly), of course these are required all together and tightly coupled. That's how these projects started, and we still see evidence of their birth today. For that reason, I do agree with Vish that there should be similar collaborations for other things. While the layer #1 (or compute) use case is very common, we can all see that it's not the only one that people are solving with OpenStack parts. And this is reflected in the products build and sold by companies, too. Some sell one subset of openstack stuff as product X and maybe a different subset as product Y. (The most common example here is compute vs object storage.) This reality has led to a lot of the angst around definitions since there is effort to define openstack all as one thing (or worse, as a base thing that others are defined as built upon). I propose that we can get the benefits of Monty's proposal and implement all of his concrete suggestions (which are fantastic) by slightly adjusting our usage of the program/project concepts. I had originally hoped that the program concept would have been a little higher-level instead of effectively spelling project as program. I'd love to see a hierarchy of openstack-program-project/team-repos. Right now, we have added the program layer but have effectively mapped it 1:1 to the project. For example, we used to have a few repos in the Swift project managed by the same group of people, and now we have a few repos in the object storage program, all managed by the same group of people. And every time something is added to OpenStack, its added as a new program, effectively putting us exactly where we were before we called it a program with the same governance and management scaling problems. Today, I'd group existing OpenStack projects into programs as follows: Compute: nova, sahara, ironic Storage: swift, cinder, glance, trove Network: neutron, designate, zaqar Deployment/management: heat, triple-o, horizon, ceilometer Identity: keystone, barbican Support (not user facing): infra, docs, tempest, devstack, oslo (potentially even) stackforge: lots I like two things about this. First, it allows people to compose a solution. Second, it allows us as a community to thing more about the strategic/product things. For example, it lets us as a community say, We think storage is important. How are we solving it today? What gaps do we have in that? How can the various storage things we have work together better? Thierry makes the point that more themes will nullify the benefits of Monty's proposal. I agree, if we continue to allow the explosion of projects/programs/themes to continue. The benefit of what Monty is proposing is that it identifies and focusses on a particular use case (deploy a VM, add a volume, get an IP, configure a domain) so that we know we
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On 09/19/2014 10:50 AM, Vishvananda Ishaya wrote: On Sep 19, 2014, at 10:14 AM, John Dickinson m...@not.mn wrote: On Sep 19, 2014, at 5:46 AM, John Griffith john.griff...@solidfire.com wrote: On Fri, Sep 19, 2014 at 4:33 AM, Thierry Carrez thie...@openstack.org wrote: Vishvananda Ishaya wrote: Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? I think that would nullify most of the benefits of Monty's proposal. If we keep on blessing themes or special groups, we'll soon be back at step 0, with projects banging on the TC door to become special, and companies not allocating resources to anything that's not special. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Great stuff, mixed on point 2 raised by Vish but honestly I think that's something that could evolve over time, but I looked at that differently as in Cinder, SWIFT and some day Manilla live under a Storage Services umbrella, and ideally at some point there's some convergence there. Anyway, I don't want to start a rat-hole on that, it's kind of irrelevant right now. Bottom line is I think the direction and initial ideas in Monty's post are what a lot of us have been thinking about and looking for. I'm in!! I too am generally supportive of the concept, but I do want to think about the vishy/tts/jgriffith points above. Today, I'd group existing OpenStack projects into programs as follows: Compute: nova, sahara, ironic Storage: swift, cinder, glance, trove Network: neutron, designate, zaqar Deployment/management: heat, triple-o, horizon, ceilometer Identity: keystone, barbican Support (not user facing): infra, docs, tempest, devstack, oslo (potentially even) stackforge: lots There is a pretty different division of things in this breakdown than in what monty was proposing. This divides things up by conceptual similarity which I think is actually less useful than breaking things down by use case. I really like the grouping and testing of things which are tightly coupled. If we say launching a VM and using it is the primary use case of our community corrently then things group into monty’s layer #1. It seems fairly clear that a large section of our community is focused on this use case so this should be a primary focus of infrastructure resources. There are other use cases in our community, for example: Object Storage: Swift (depends on keystone) Orchestrating Multiple VMs: Heat (depends on layer1) DBaSS: Trove: (depends on heat) These are also important use cases for parts of our community, but swift has demostrated that it isn’t required to be a part of an integrated release schedule, so these could be managed by smaller groups in the community. Note that these are primarily individual projects today, but I could see a future where some of these projects decide to group and do an integrated release. In the future we might see (totally making this up): Public Cloud Application services: Trove, Zaqar Application Deployment services: Heat, Murano Operations services: Ceilometer, Congress As I mentioned previously though, I don’t think we need to define these groups in advance. These groups can organize as needed. I'm kinda interested to see what self-organization happens... ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
On Thu, Sep 18, 2014 at 1:53 PM, Monty Taylor mord...@inaugust.com wrote: I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. Thanks for writing that Monty. Sean took a concept meant for organizing the relationships in DevStack and Grenade and made it relatable to OpenStack as a whole. This brings it out where we can actually use it as a model. I do think there is value in distinctions between the original layers 1,2 and 3 (you combined 1 and 2). But for non-technical purposes 1 and 2 are indeed the same. Suggestion 6: Isn't it funny how everything old is new again? The DevStack exercises started out as exactly this, then grew more functional over time until Tempest came along. How hipster is that? Suggestion 9: Wouldn't it be wonderful if a small set of cloud definition files could be used for the myriad of user tools out there? Standards, we don't have enough of them! In the long term I would like to see more of our base (toolsets and services) be pluggable so that the less-blessed projects can participate without requiring additions to the base repos. This should also be true for user-facing tools. OpenStackClient already picks up installed plugins with the proper entry points configured so everything doesn't need to be in the primary repo to play along. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Thoughts on OpenStack Layers and a Big Tent model
Great writeup. I think there are some great concrete suggestions here. A couple more: 1. I think we need a better name for Layer #1 that actually represents what the goal of it is: Infrastructure Services? 2. We need to be be open to having other Layer #1s within the community. We should allow for similar collaborations and group focus to grow up as well. Storage Services? Platform Services? Computation Services? Vish On Sep 18, 2014, at 11:53 AM, Monty Taylor mord...@inaugust.com wrote: Hey all, I've recently been thinking a lot about Sean's Layers stuff. So I wrote a blog post which Jim Blair and Devananda were kind enough to help me edit. http://inaugust.com/post/108 Enjoy. Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
