Re: [openstack-dev] Kilo Cycle Goals Exercise
As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread ... Here's my list of high-level cycle goals, for consideration ... 1. Address our usability debts With some justification, we've been saddled with the perception of not caring enough about the plight of users and operators. The frustrating thing is that much of this is very fixable, *if* we take time out from the headlong rush to add features. Achievable things like documentation completeness, API consistency, CLI intuitiveness, logging standardization, would all go a long way here. These things are of course all not beyond the wit of man, but we need to take the time out to actually do them. This may involve a milestone, or even longer, where we accept that the rate of feature addition will be deliberately slowed down. 2. Address the drags on our development velocity Despite the Trojan efforts of the QA team, the periodic brownouts in the gate are having a serious impact on our velocity. Over the past few cycles, we've seen the turnaround time for patch check/ verification spike up unacceptably long multiple times, mostly around the milestones. Whatever we can do to smoothen out these spikes, whether it be moving much of the Tempest coverage into the project trees, or switching focus onto post-merge verification as suggested by Sean on this thread, or even considering some more left-field approaches such as staggered milestones, we need to grasp this nettle as a matter of urgency. Further back in the pipeline, the effort required to actually get something shepherded through review is steadily growing. To the point that we need to consider some radical approaches that retain the best of our self-organizing model, while setting more reasonable reliable expectations for patch authors, and making it more likely that narrow domain expertise is available to review their contributions in timely way. For the larger projects, this is likely to mean something different (along the lines of splits or sub-domains) than it does for the smaller projects. 3. Address the long-running what's in and what's out questions The way some of the discussions about integration and incubation played out this cycle have made me sad. Not all of these discussions have been fully supported by the facts on the ground IMO. And not all of the issues that have been held up as justifications for whatever course of exclusion or inclusion would IMO actually be solved in that way. I think we need to move the discussion around a new concept of layering, or redefining what it means to be in the tent, to a more constructive and collaborative place than heretofore. 4. Address the fuzziness in cross-service interactions In a semi-organic way, we've gone and built ourselves a big ol' service-oriented architecture. But without necessarily always following the strong contracts, loose coupling, discoverability, and autonomy that a SOA approach implies. We need to take the time to go back and pay down some of the debt that has accreted over multiple cycles around these these cross-service interactions. The most pressing of these would include finally biting the bullet on the oft-proposed but never delivered-upon notion of stabilizing notifications behind a well-defined contract. Also, the more recently advocated notions of moving away from coarse-grained versioning of the inter-service APIs, and supporting better introspection and discovery of capabilities. by end of day Wednesday, September 10th. Oh, yeah, and impose fewer arbitrary deadlines ;) Cheers, Eoghan After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/11/2014 07:32 AM, Eoghan Glynn wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread ... Here's my list of high-level cycle goals, for consideration ... 1. Address our usability debts With some justification, we've been saddled with the perception of not caring enough about the plight of users and operators. The frustrating thing is that much of this is very fixable, *if* we take time out from the headlong rush to add features. Achievable things like documentation completeness, API consistency, CLI intuitiveness, logging standardization, would all go a long way here. These things are of course all not beyond the wit of man, but we need to take the time out to actually do them. This may involve a milestone, or even longer, where we accept that the rate of feature addition will be deliberately slowed down. 2. Address the drags on our development velocity Despite the Trojan efforts of the QA team, the periodic brownouts in the gate are having a serious impact on our velocity. Over the past few cycles, we've seen the turnaround time for patch check/ verification spike up unacceptably long multiple times, mostly around the milestones. Whatever we can do to smoothen out these spikes, whether it be moving much of the Tempest coverage into the project trees, or switching focus onto post-merge verification as suggested by Sean on this thread, or even considering some more left-field approaches such as staggered milestones, we need to grasp this nettle as a matter of urgency. Further back in the pipeline, the effort required to actually get something shepherded through review is steadily growing. To the point that we need to consider some radical approaches that retain the best of our self-organizing model, while setting more reasonable reliable expectations for patch authors, and making it more likely that narrow domain expertise is available to review their contributions in timely way. For the larger projects, this is likely to mean something different (along the lines of splits or sub-domains) than it does for the smaller projects. 3. Address the long-running what's in and what's out questions The way some of the discussions about integration and incubation played out this cycle have made me sad. Not all of these discussions have been fully supported by the facts on the ground IMO. And not all of the issues that have been held up as justifications for whatever course of exclusion or inclusion would IMO actually be solved in that way. I think we need to move the discussion around a new concept of layering, or redefining what it means to be in the tent, to a more constructive and collaborative place than heretofore. 4. Address the fuzziness in cross-service interactions In a semi-organic way, we've gone and built ourselves a big ol' service-oriented architecture. But without necessarily always following the strong contracts, loose coupling, discoverability, and autonomy that a SOA approach implies. We need to take the time to go back and pay down some of the debt that has accreted over multiple cycles around these these cross-service interactions. The most pressing of these would include finally biting the bullet on the oft-proposed but never delivered-upon notion of stabilizing notifications behind a well-defined contract. Also, the more recently advocated notions of moving away from coarse-grained versioning of the inter-service APIs, and supporting better introspection and discovery of capabilities. +1 IMO, almost all of the other ills discussed recently derive from this single failure. -David by end of day Wednesday, September 10th. Oh, yeah, and impose fewer arbitrary deadlines ;) Cheers, Eoghan After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 2014-09-11 01:27:23 -0400 (-0400), Russell Bryant wrote: [...] But seriously, we should probably put out a more official notice about this once Kilo opens up. It's probably worth carrying in the release notes for all Juno servers... This is the last release of OpenStack with official support for Python 2.6-based platforms. Of course we're still supporting it on the Juno stable branch for its lifetime (probably something like a year depending on what the stable branch managers feel they can provide), and in all involved clients and libraries until Juno reaches end of support. So don't get all excited that 2.6 is going away entirely in a couple months. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
-Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Tuesday, September 09, 2014 4:29 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Kilo Cycle Goals Exercise 3. Another long-term topic is standardizing our APIs so that we use consistent terminology and formatting (I think we have at least 3 forms of errors returned now?). I’m not sure we have anyone ready to drive this, yet, so I don’t think it’s something to consider for Kilo. +10 Frankly, I believe this should be our #1 priority from a cross-project perspective. The inconsistencies in the current APIs (even within the same project's APIs!) is just poor form and since our REST APIs are the very first impression that we give to the outside developer community, it really is incumbent on us to make sure they are explicit, free of side-effects, well-documented, consistent, easy-to-use, and hide implementation details thoroughly. +1 The REST API consistency is important for whole OpenStack projects, The list for it in my mind is that * API URL/Attribute naming * HTTP status code on success * Behavior against wrong input (HTTP status code, message in a response) It is difficult to apply them to all projects, but it would be worth for improving the quality from the viewpoint of the outside world. Thanks Ken'ichi Ohmichi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
Joe Gordon wrote: To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. Thanks again to Joe for starting this important discussion. My personal list of Kilo goals goes as follows: #1: Fix our growth pains We grew a lot, and our recipes were designed for a smaller group where trust happens more naturally. With our community growing to a Dunbar order of magnitude above, some of those recipes don't work so great, so we need to revisit them... That includes the binary integrated release (introduce layers?), cross-project functional testing (push it at project level?), the programs concept, encouraging PTL delegation (the czar/liaisons proposal ?), scaling core reviewing in larger projects (Nova driver split ?), etc. We already started the conversation on those important topics, but Kilo is the timeframe for us to implement those changes, because I don't see our community wait for more than one cycle to see the light at the end of the tunnel. #2: Fix the end user experience Monty expressed it better than I can. We need consistent and better-designed APIs, client SDKs that provide useful primitives and actually abstract differences in implementations, etc. #3: Fix what we have: bugfixes, consistency, scaling up, scaling down, upgrading Rather than adding more features for the mid-size private cloud, let's make sure that what we have works well, provides a consistent experience across projects, scales up beautifully, can be easily used at smaller scale as well (simplicity), and allows seamless upgrades. This is another way of looking at paying our technical debt that others have mentioned as goals -- generally polishing what we have rather than building out new things. Regards, -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
tl;dr I'm concerned we're conflating user concerns and contributor concerns. I'd like to see laser focus on two things that help both: 1) Define layers in our governance and integration efforts 2) Pay down technical debt with a focus on supporting users. more Great kick off Joe, thanks. I have been waiting to reply to this thread on purpose while I try to gather as much info as I can about what both users and contributors are interested in for our future. I have a pretty specific curiosity about how we're framing this request: I see a spilt between contributor concerns and user concerns. Example contributor concerns: scope definition incubation and integration requirements heavy-handed processes review pileups contributor license agreement concerns about preventing contributions and openness technical debt weighing a project down Example user concerns: how do I know what's tested; what's buggy how do I trust a part of OpenStack to put it into production and maintain over time why isn't there complete driver or hypervisor documentation for fill-in-the-blank logging security best practices high availability across OpenStack monitoring OpenStack monitoring apps on my OpenStack infrastructure my own users have needs; how can I get upstream to care about them These example concerns are not comprehensive but I worry about conflation causing us all to burnout and flail. I know we all work in the gray areas but there's a black-and-white problem due to our current governance. Since I write docs and work on technical committee concerns, I sit on a cross-project and cross-concern bridge all the time, advocating, prioritizing, standardizing, and weighing needs across many project teams. The user concerns are a higher priority for docs currently, because they're a support mechanism. I think the Kilo cycle should be dedicated to: - Fulfill the promise of layers for defining OpenStack integration gradients in governance and integration taxes such as consistency across projects, reviews on projects with many drivers/contributors, infra, docs, and testing - Pay down technical debt by sharply focusing on capabilities rather than drivers that fulfill them Two focus areas. Each of those will have a lot of work items. Both find more common ground for devs and users. Let's do this! Anne On Wed, Sep 3, 2014 at 10:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Wed, Sep 10, 2014 at 7:13 AM, Thierry Carrez thie...@openstack.org wrote: Joe Gordon wrote: To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. Thanks again to Joe for starting this important discussion. +100 I was going to propose my own list, but to be honest, Thierry sums it all up so eloquently below that I'd be hard pressed to come up with a different list. My personal list of Kilo goals goes as follows: #1: Fix our growth pains We grew a lot, and our recipes were designed for a smaller group where trust happens more naturally. With our community growing to a Dunbar order of magnitude above, some of those recipes don't work so great, so we need to revisit them... That includes the binary integrated release (introduce layers?), cross-project functional testing (push it at project level?), the programs concept, encouraging PTL delegation (the czar/liaisons proposal ?), scaling core reviewing in larger projects (Nova driver split ?), etc. We already started the conversation on those important topics, but Kilo is the timeframe for us to implement those changes, because I don't see our community wait for more than one cycle to see the light at the end of the tunnel. #2: Fix the end user experience Monty expressed it better than I can. We need consistent and better-designed APIs, client SDKs that provide useful primitives and actually abstract differences in implementations, etc. #3: Fix what we have: bugfixes, consistency, scaling up, scaling down, upgrading Rather than adding more features for the mid-size private cloud, let's make sure that what we have works well, provides a consistent experience across projects, scales up beautifully, can be easily used at smaller scale as well (simplicity), and allows seamless upgrades. This is another way of looking at paying our technical debt that others have mentioned as goals -- generally polishing what we have rather than building out new things. I agree with all of your goals here Thierry, and as I stated above, my list matches these one for one. I think we have to continue fixing the growing pains which are happening at a micro-level in each project, and at a macro-level overall. There are lots of solid ideas around this for sure, we just need to execute on the ones which we think will have the most benefit. To me, this is the biggest issue we have and the one we should tackle first. Thanks, Kyle Regards, -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/03/2014 11:37 AM, Joe Gordon wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. In OpenStack, we have no shortage of interest and enthusiasm on all fronts, including development contributors, deployers, and cloud end users. When looking at project wide-priorities, we need to make sure our tools, processes, and resulting technology facilitate turning all of that interest into real value. We need to identify which areas have the most pain, and work to improve them. A lot of this is certainly about Kilo, but it's longer term, too. It's the way we should always be thinking about this. 1) Dev community We clearly have a lot of growing pains here. What's quite encouraging is that we also have a lot of hard work going into several different proposals to figure out ways to help. The largest projects (Nova and Neutron) are overwhelmed and approaching breaking points. We have to find ways to relieve this pressure. This may involve aggressively pursing project splits or other code review workflow changes. I think the problems and solutions here are project-wide issues, as solutions put in place tend to rapidly spread to the rest of OpenStack. This is an area I'm especially concerned about and eager to help look for solutions. We should evaluate all potential improvements against how well they help us scale our teams and processes to remove bottlenecks to productivity in the dev communtiy. There are several other encouraging proposals related to easing pain in the dev community: - re-working how we do functional testing by making it more project focused - discussions like this one to discuss both priorities, but also how we turn priorities into real action (like the nova projects discussions around using priorities in development) - evolving project leadership (the PTL position) so that we can provide more guidance around delegation in a way that is reasonably consistent across projects - continued discussion about the contents of the integrated release and how we can continue to foster growth without sacrificing quality We are always going to have problems like this, and I hope we continue to think about, discuss, and improve the way we run our projects every release cycle to come. 2) End Users A few others have done a very nice job describing end user experience problems. Monty's description of getting an instance with an IP was painful and embarrassing to read. We've got to figure out ways to provide better focus on these sorts of usability issues. They're obviously not getting the attention they deserve. There have also been lots of good points about improving our API consistency. I totally agree. I'd love to see a group of people step up and emerge as leaders in this area across all projects. I feel like that's something we're sorely missing right now. 3) Deployers OpenStack is still painful to deploy, and even more painful to manage. I'm still quite pleased that we have a deployment program working on this space. I'd actually really like to see how we can facilitate better integration and discussion between TripleO and the rest of the project teams. I'm also very pleased with the progress we've made in Nova towards the initial support for live upgrades. We still have more work to do in Nova, but I'd also like to see more work done in other projects towards the same goal. For both deployers and the dev community, figuring out what went wrong when OpenStack breaks sucks. Others provided some good pointers to several areas we can improve that area (better logs, tooling, ...) and I hope we can make some real progress in this area in the coming cycle. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/03/2014 11:37 AM, Joe Gordon wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. Again, thanks for bringing up this important topic. Here's my take on the cross-project, larger problem domains that I believe we should focus as a community on in Kilo: 1) Apply Drano to the current blockage in our governance policies and program pipeline I'm finalizing a blog post about the community and governance policy issues that I see, but the Cliff's Notes version is that I think we should: a) get rid of the official incubated status b) adopt strictly the OpenStack Layers taxonomy for classification of projects (and get rid of Programs entirely) c) loosen the restrictions on projects living in the openstack/ code namespace d) replace the official integrated project status with finer-grained information that is actually useful to deployers 2) Apply LiquidPlumber to the current testing infrastructure gate I think it's clear that there is significant pain currently felt by contributors across the board, and a large amount of pain experience by folks setting up and maintaining external CI systems. We need to focus on how to make our gating systems as efficient as possible, with as few false failures and as few non-relevant test runs as possible. I believe Dan Berrange's proposal to split out the virt drivers in Nova (all of them) into separate repositories will be a huge win here in terms of reducing false positives and avoiding running external CI systems for drivers that are not related to a patch in another driver. I think Neutron and Cinder would be able to alleviate some gate pain by doing a similar effort. I think pulling functional tests into the individual projects will also help to give the gate a bit of unclogging, since functional tests can be removed from the lengthier devstack-gate-based Tempest integration tests. We need to continue making progress in documenting our upstream CI systems, how the gate works, how to diagnose and debug gate issues, how to bootstrap a developer environment, and how to set up external CI systems. Finally, I believe there are significant things that can be done to reduce both unit and functional test run time. Personally, I'm almost finished with a branch in Nova that replaces the functional tests in /nova/tests/integrated/api_samples/ with a separate runner that only does environment setup once instead of on each test method setUp(). The runtime for testing api_samples goes from 5 minutes to 30 seconds on my machines. Similar things can and should be looked at for other projects to give our gate something of an enema. 3) Apply Mr. Clean to the crufty interfaces that litter our projects Folks have discussed this at length, and I'm also finishing up another longish etherpad and ML discussion about refactoring in Nova's api-conductor-scheduler internal interfaces, but the bottom line is that there are a significant number of internal interfaces that need to be cleaned up, versioned, and objectified before splitting out any of the drivers or subsystems can be done easily. We need to pay down the technical debt that has built up, and refactoring crufty interfaces is one good way to do that. Thanks, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
So easy/obvious it probably isn't even worth mentioning: Drop support for python2.6 -- - Gus ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/11/2014 12:52 AM, Angus Lees wrote: So easy/obvious it probably isn't even worth mentioning: Drop support for python2.6 Yeah, that's been the plan. We discussed this at the Juno summit and representatives from most (all?) distributions carrying OpenStack were there. Dropping in Kilo seemed like a reasonable time frame at the time. https://etherpad.openstack.org/p/juno-cross-project-future-of-python And obviously tweeting about it makes it official, right? https://twitter.com/russellbryant/status/466241078472228864 But seriously, we should probably put out a more official notice about this once Kilo opens up. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
I haven't had a chance to read other people's posts, so I am sure there is duplication here. What would I have all of OpenStack working on if I was ruler of the universe? Let's see... 1. Fixing our flakey gate: we're all annoyed by our code failing tests with transient errors, but most people just recheck. Here's the kicker though -- those errors sometimes affect production deployments as well. I don't have a magical solution to incent people to work on fixing these bugs, but we need to fix these. Now. 2. Pay down our tech debt in general. The most obvious example of this is bugs. Nova has nearly 1,000 of these, and not enough people working on them compared with features. This is a horrible user experience for our users, and we should all be embarrassed by it. 3. Find a way to scale nova and neutron development. Our biggest projects are suffering, and we need to come up with a consistent way to solve that problem. Michael On Thu, Sep 4, 2014 at 1:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Wed, Sep 3, 2014 at 8:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. 1. Strengthen our north bound APIs * API micro-versioning * Improved CLI's and SDKs * Better capability discovery * Hide usability issues with client side logic * Improve reliability As others have said in this thread trying to use OpenStack as a user is a very frustrating experience. For a long time now we have focused on southbound APIs such as drivers, configuration options, supported architectures etc. But as a project we have not spent nearly enough time on the end user experience. If our northbound APIs aren't something developers want to use, our southbound API work doesn't matter. 2. 'Fix' our development process * openstack-specs. Currently we don't have any good way to work on big entire-project efforts, hopefully something like a openstack-specs repo (with liasons from each core-team reviewing it) will help make it possible for us to tackle these issues. I see us addressing the API micro-versioning and capability discovery issues here. * functional testing and post merge testing. As discussed elsewhere in this thread our current testing model isn't meeting our current requirements. 3. Pay down technical debt This is the one I am actually least sure about, as I can really only speak for nova on this one. In our constant push forward we have accumulated a lot of technical debt. The debt manifests itself as hard to maintain code, bugs (nova had over 1000 open bugs until yesterday), performance/scaling issues and missing basic features. I think its time for us to take inventory if our technical debt and fix some of the biggest issues. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
Deleting unnecessary code, introducing a stabilization cycle and/or making definite steps towards a unified SDK are definitely my votes. *Adam Lawson* AQORN, Inc. 427 North Tatnall Street Ste. 58461 Wilmington, Delaware 19801-2230 Toll-free: (844) 4-AQORN-NOW ext. 101 International: +1 302-387-4660 Direct: +1 916-246-2072 On Tue, Sep 9, 2014 at 5:09 PM, Joe Gordon joe.gord...@gmail.com wrote: On Wed, Sep 3, 2014 at 8:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. 1. Strengthen our north bound APIs * API micro-versioning * Improved CLI's and SDKs * Better capability discovery * Hide usability issues with client side logic * Improve reliability As others have said in this thread trying to use OpenStack as a user is a very frustrating experience. For a long time now we have focused on southbound APIs such as drivers, configuration options, supported architectures etc. But as a project we have not spent nearly enough time on the end user experience. If our northbound APIs aren't something developers want to use, our southbound API work doesn't matter. 2. 'Fix' our development process * openstack-specs. Currently we don't have any good way to work on big entire-project efforts, hopefully something like a openstack-specs repo (with liasons from each core-team reviewing it) will help make it possible for us to tackle these issues. I see us addressing the API micro-versioning and capability discovery issues here. * functional testing and post merge testing. As discussed elsewhere in this thread our current testing model isn't meeting our current requirements. 3. Pay down technical debt This is the one I am actually least sure about, as I can really only speak for nova on this one. In our constant push forward we have accumulated a lot of technical debt. The debt manifests itself as hard to maintain code, bugs (nova had over 1000 open bugs until yesterday), performance/scaling issues and missing basic features. I think its time for us to take inventory if our technical debt and fix some of the biggest issues. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Sun, 7 Sep 2014, Monty Taylor wrote: 1. Caring about end user experience at all 2. Less features, more win 3. Deleting things Yes. I'll give away all of my list for any one of these. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Sep 7, 2014, at 9:27 PM, Anita Kuno ante...@anteaya.info wrote: On 09/07/2014 09:12 PM, Angus Salkeld wrote: Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. I don't understand why you would encourage writers of blog posts you disagree with by sending them traffic. silencing users who have issues with your project is a really bad idea.If you want to create something great you absolutely need to be obsessed with your detractors and the weight of what they have to say. Because unless they are a competitor engaged in outright slander, there will be some truth in it. Ignore criticism at your peril.Someone who takes the time to write out an even somewhat well reasoned criticism is doing your project a service. I found the above blog post very interesting as I’d like to get more data on what the large, perceived issues are. Anita. 1) Consistent/easy upgrading. all projects should follow a consistent model to the way they approach upgrading. it should actually work. - REST versioning - RPC versioning - db (data) migrations - ordering of procedures and clear documentation of it. [this has been begged for by operators, but not sure how we have delivered] 2) HA - ability to continue operations after been restated - functional tests to prove the above? 3) Make it easier for small business to give OpenStack a go - produce standard docker images as part of ci with super simple instructions on running them. -Angus On Thu, Sep 4, 2014 at 1:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/08/2014 11:12 AM, Mike Bayer wrote: On Sep 7, 2014, at 9:27 PM, Anita Kuno ante...@anteaya.info wrote: On 09/07/2014 09:12 PM, Angus Salkeld wrote: Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. I don't understand why you would encourage writers of blog posts you disagree with by sending them traffic. silencing users who have issues with your project is a really bad idea.If you want to create something great you absolutely need to be obsessed with your detractors and the weight of what they have to say. Because unless they are a competitor engaged in outright slander, there will be some truth in it. Ignore criticism at your peril.Someone who takes the time to write out an even somewhat well reasoned criticism is doing your project a service. I found the above blog post very interesting as I’d like to get more data on what the large, perceived issues are. Wow, we are really taking liberties with my question today. What part of any of my actions current or previous have led you to believe that I want to now or ever have silenced anyone? I am curious what led you to believe that silencing users was the motivation for my question of Angus. I now see, through asking Angus for clarity which he did provide (not silencing him, you will notice), that Angus' motivation was prevention of poor user experience through better attention. I am highly aware, particularly in the area in which I work - the third party space- of the leading nature of behavioural training that takes place particularly of new contributors and contributors who don't have English as a first language of anything we ask or expect them to do. Many times what seems to be a reasonable comment or expectation can be taken completely out of context by folks who don't have English as a first language and don't have the cultural context and filters that English speakers have. Actually my question was motivated from a user experience point of view, the third party user, since I am only too aware of what kind of questions and confusion many comments cause because the commenter doesn't take the non-English speaker point of view into account. By clarifying Angus' motivation with Angus, hopefully his meaning - create better user experiences, and better relationships with users - has come through. And I agree with all of your points, which is why I take such pains to create clarity on the mailing lists and other communication. Thanks, Anita. Anita. 1) Consistent/easy upgrading. all projects should follow a consistent model to the way they approach upgrading. it should actually work. - REST versioning - RPC versioning - db (data) migrations - ordering of procedures and clear documentation of it. [this has been begged for by operators, but not sure how we have delivered] 2) HA - ability to continue operations after been restated - functional tests to prove the above? 3) Make it easier for small business to give OpenStack a go - produce standard docker images as part of ci with super simple instructions on running them. -Angus On Thu, Sep 4, 2014 at 1:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Sep 7, 2014, at 8:14 PM, Monty Taylor mord...@inaugust.com wrote: 2. Less features, more win In a perfect world, I'd argue that we should merge exactly zero new features in all of kilo, and instead focus on making the ones we have work well. Some of making the ones we have work well may wind up feeling just like writing features, as I imagine some of our features are probably only half features in the first place. 3. Deleting things We should delete a bunch of code. Deleting code is fun, and it makes your life better, because it means you have less code. So we should start doing it. In particular, we should look for places where we wrote something as part of OpenStack because the python community did not have a thing already, but now there is one. In those cases, we should delete ours and use theirs. Or we should contribute to theirs if it's not quite good enough yet. Or we should figure out how to make more of the oslo libraries things that can truly target non-OpenStack things. I have to agree that “Deleting things” is the best, best thing. Anytime you can refactor around things and delete more code, a weight is lifted, your code becomes easier to understand, maintain, and expand upon.Simpler code then gives way to refactorings that you couldn’t even see earlier, and sometimes you can even get a big performance boost once a bunch of supporting code now reveals itself to be superfluous. This is most critical for Openstack as Openstack is written in Python, and for as long as we have to stay on the cPython interpreter, number of function calls is directly proportional to how slow something is. Function calls are enormously expensive in Python. Something that helps greatly with the goal of “Deleting things” is to reduce dependencies between systems. In SQLAlchemy, the kind of change I’m usually striving for is one where we take a module that does one Main Thing, but then has a bunch of code spread throughout it to do some Other Thing, that is really much less important, but complicates the Main Thing. What we do is reorganize the crap out of it and get the Other Thing out of the core Main Thing, move it out to a totally optional “extension” module that bothers noone, and we essentially forget about it because nobody ever uses it (examples include http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_08.html#instrumentationmanager-and-alternate-class-instrumentation-is-now-an-extension, http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_08.html#mutabletype). When we make these kinds of changes, major performance enhancements come right in - the Main Thing no longer has to worry about those switches and left turns introduced by the Other Thing, and tons of superfluous logic can be thrown away.SQLAlchemy’s architecture gains from these kinds of changes with every major release and 1.0 is no exception. This is not quite the same as “Deleting things” but it has more or less the same effect; you isolate code that everyone uses from code that only some people occasionally use. In SQLAlchemy specifically, we have the issue of individual database dialects that are still packaged along; e.g. there is sqlalchemy.dialects.mysql, sqlalchemy.dialects.postgresql, etc. However, a few years back I went through a lot of effort to modernize the system by which users can provide their own database backends; while you can not only provide your own custom backend using setuptools entry points, I also made a major reorganization of SQLAlchemy’s test suite to produce the “dialect test suite”; so that when you write your custom dialect, you can actually run a large, pre-fabricated test suite out of SQLAlchemy’s core against your dialect, without the need for your dialect to be actually *in* SQLAlchemy. There were many wins from this system, including that it forced me to write lots of tests that were very well focused on testing specifically what a dialect needs to do, in isolation of anything SQLAlchemy itself needs to do. It allowed a whole batch of new third party dialects like that for Amazon Redshift, FoundationDB, MonetDB, and also was a huge boon to IBM’s DB2 driver who I helped to get onto the new system. And since then I’ve been able to go into SQLAlchemy and dump out lots of old dialects that are much better off being maintained separately, at a different level of velocity and hopefully by individual contributors who are interested in them, like MS Access, Informix, MaxDB, and Drizzle. Having all these dialects in one big codebase only served as a weight on the project, and theoretically it wouldn’t be a bad idea for SQLA to have *all* dialects as separate projects, but we’re not there yet. The only reason I’m rambling on about a SQLAlchemy’s Core/Dialect dichotomy is just that I was very much *reminded* of it by the thread regarding Nova and the various “virt” drivers. I know
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Sep 8, 2014, at 11:30 AM, Anita Kuno ante...@anteaya.info wrote: Wow, we are really taking liberties with my question today. What part of any of my actions current or previous have led you to believe that I want to now or ever have silenced anyone? I am curious what led you to believe that silencing users was the motivation for my question of Angus. I was only replying to your single message in isolation of the full conversation; the notion that one would not want to send traffic to a blog because they disagree with it, at face value seems like a bad idea. Apparently that isn’t the meaning you wished to convey, so I apologize for missing the larger context. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/08/2014 12:00 PM, Mike Bayer wrote: On Sep 8, 2014, at 11:30 AM, Anita Kuno ante...@anteaya.info wrote: Wow, we are really taking liberties with my question today. What part of any of my actions current or previous have led you to believe that I want to now or ever have silenced anyone? I am curious what led you to believe that silencing users was the motivation for my question of Angus. I was only replying to your single message in isolation of the full conversation; the notion that one would not want to send traffic to a blog because they disagree with it, at face value seems like a bad idea. Actually the word used was prevent, and if I personally want to prevent something I don't encourage it by giving it attention. Not understanding something due to disagreement with it is, I agree, a perspective which is limiting and which ultimately does the party at the heart of the discussion the most harm, it is self-defeating, yes. Apparently that isn’t the meaning you wished to convey, so I apologize for missing the larger context. I appreciate you taking the time to talk to me so that we might understand each other better. Thank you, Mike, I'm grateful for your time with this. Thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
Monty, +1!! I fully agree!! How can I help :-)? Can we dedicate some design summit sessions to this topic? Ideally, having some stakeholder driven sessions where we can hear about the user experiences issues causing the most pain would be a good start to get this to become a focus area. Thanks, Brad Brad Topol, Ph.D. IBM Distinguished Engineer OpenStack (919) 543-0646 Internet: bto...@us.ibm.com Assistant: Kendra Witherspoon (919) 254-0680 From: Monty Taylor mord...@inaugust.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org, Date: 09/07/2014 08:15 PM Subject:Re: [openstack-dev] Kilo Cycle Goals Exercise On 09/03/2014 08:37 AM, Joe Gordon wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. If I were king ... 1. Caring about end user experience at all It's pretty clear, if you want to do things with OpenStack that are not running your own cloud, that we collectively have not valued the class of user who is a person who wants to use the cloud. Examples of this are that the other day I had to read a section of the admin guide to find information about how to boot a nova instance with a cinder volume attached all in one go. Spoiler alert, it doesn't work. Another spoiler alert - even though the python client has an option for requesting that a volume that is to be attached on boot be formatted in a particular way, this does not work for cinder volumes, which means it does not work for an end user - EVEN THOUGH this is a very basic thing to want. Our client libraries are clearly not written with end users in mind, and this has been the case for quite some time. However, openstacksdk is not yet to the point of being usable for end users - although good work is going on there to get it to be a basis for an end user python library. We give deployers so much flexibility, that in order to write even a SIMPLE program that uses OpenStack, an end user has to know generally four of five pieces of information to check for that are different ways that a deployer may have decided to do things. Example: - As a user, I want a compute instance that has an IP address that can do things. WELL, now you're screwed, because there is no standard way to do that. You may first want to try booting your instance and then checking to see if nova returns a network labeled public. You may get no networks. This indicates that your provider decided to deploy neutron, but as part of your account creation did not create default networks. You now need to go create a router, network and port in neutron. Now you can try again. Or, you may get networks back, but neither of them are labeled public - instead, you may get a public and a private address back in the network labeled private. Or, you may only get a private network back. This indicates that you may be expected to create a thing called a floating-ip. First, you need to verify that your provider has installed the floating-ip's extension. If they have, then you can create a floating-ip and attach it to your host. NOW - once you have those things done, you need to connect to your host and verify that its outbound networking has not been blocked by a thing called security groups, which you also may not have been expecting to exist, but I'll stop there, because the above is long enough. Every. Single. One. Of. Those. Cases. is real and has occurred across only the two public openstack clouds that infra uses. That means that our provisioning code takes every single one of them in to account, and anyone who writes code that wants to get a machine to use must take them all in to account or else their code is buggy. That's RIDICULOUS. So we should fix it. I'd say we should fix it by removing 1000% of the choices we've given deployers in this case, but I won't win there. So how about let's make at least one client library that encodes all of the above logic behind some simple task oriented API calls? How about we make that library not something which is just a repackaging of requests that does not contain intelligent, but instead something that is fundamentally usable. How about we have
Re: [openstack-dev] Kilo Cycle Goals Exercise
The End User working group is being described at https://wiki.openstack.org/wiki/End_User_Working_Group. Chris Kemp is establishing the structure. This page covers how to get involved... Tim From: Brad Topol [mailto:bto...@us.ibm.com] Sent: 08 September 2014 19:50 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Kilo Cycle Goals Exercise Monty, +1!! I fully agree!! How can I help :-)? Can we dedicate some design summit sessions to this topic? Ideally, having some stakeholder driven sessions where we can hear about the user experiences issues causing the most pain would be a good start to get this to become a focus area. Thanks, Brad Brad Topol, Ph.D. IBM Distinguished Engineer OpenStack (919) 543-0646 Internet: bto...@us.ibm.commailto:bto...@us.ibm.com Assistant: Kendra Witherspoon (919) 254-0680 From:Monty Taylor mord...@inaugust.commailto:mord...@inaugust.com To:OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org, Date:09/07/2014 08:15 PM Subject:Re: [openstack-dev] Kilo Cycle Goals Exercise On 09/03/2014 08:37 AM, Joe Gordon wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. If I were king ... 1. Caring about end user experience at all It's pretty clear, if you want to do things with OpenStack that are not running your own cloud, that we collectively have not valued the class of user who is a person who wants to use the cloud. Examples of this are that the other day I had to read a section of the admin guide to find information about how to boot a nova instance with a cinder volume attached all in one go. Spoiler alert, it doesn't work. Another spoiler alert - even though the python client has an option for requesting that a volume that is to be attached on boot be formatted in a particular way, this does not work for cinder volumes, which means it does not work for an end user - EVEN THOUGH this is a very basic thing to want. Our client libraries are clearly not written with end users in mind, and this has been the case for quite some time. However, openstacksdk is not yet to the point of being usable for end users - although good work is going on there to get it to be a basis for an end user python library. We give deployers so much flexibility, that in order to write even a SIMPLE program that uses OpenStack, an end user has to know generally four of five pieces of information to check for that are different ways that a deployer may have decided to do things. Example: - As a user, I want a compute instance that has an IP address that can do things. WELL, now you're screwed, because there is no standard way to do that. You may first want to try booting your instance and then checking to see if nova returns a network labeled public. You may get no networks. This indicates that your provider decided to deploy neutron, but as part of your account creation did not create default networks. You now need to go create a router, network and port in neutron. Now you can try again. Or, you may get networks back, but neither of them are labeled public - instead, you may get a public and a private address back in the network labeled private. Or, you may only get a private network back. This indicates that you may be expected to create a thing called a floating-ip. First, you need to verify that your provider has installed the floating-ip's extension. If they have, then you can create a floating-ip and attach it to your host. NOW - once you have those things done, you need to connect to your host and verify that its outbound networking has not been blocked by a thing called security groups, which you also may not have been expecting to exist, but I'll stop there, because the above is long enough. Every. Single. One. Of. Those. Cases. is real and has occurred across only the two public openstack clouds that infra uses. That means that our provisioning code takes every single one of them in to account, and anyone who writes code that wants to get a machine to use must take them all in to account or else their code is buggy
Re: [openstack-dev] Kilo Cycle Goals Exercise
Amen to this! I've always felt bad that before yahoo tries to include a new feature in its openstack cloud/s we have to figure out how much the feature is a land-mine, how much of it works, how much of it doesn't and so-on. That type of investigation imho shouldn't really be needed and the fact that it is makes me want more and more a stability cycle or two (or three). More and more recently I've be thinking that we have spent way to much on drivers and features and not enough on our own 'infrastructure'. While of course there is a balance, it just seems like the balance currently isn't right (IMHO). Maybe we should start asking ourselves why it is so much easier to add a driver vs. do a cross-project functionality like gantt (or centralized quota management or other...) that removes some of those land-mines. When it becomes easier to land gantt vs a new driver then I think we might be in a better place. After all, our infrastructure is what makes the project a long-term success and not adding new drivers. Just my 2 cents, Josh On Sep 7, 2014, at 5:14 PM, Monty Taylor mord...@inaugust.com wrote: On 09/03/2014 08:37 AM, Joe Gordon wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. If I were king ... 1. Caring about end user experience at all It's pretty clear, if you want to do things with OpenStack that are not running your own cloud, that we collectively have not valued the class of user who is a person who wants to use the cloud. Examples of this are that the other day I had to read a section of the admin guide to find information about how to boot a nova instance with a cinder volume attached all in one go. Spoiler alert, it doesn't work. Another spoiler alert - even though the python client has an option for requesting that a volume that is to be attached on boot be formatted in a particular way, this does not work for cinder volumes, which means it does not work for an end user - EVEN THOUGH this is a very basic thing to want. Our client libraries are clearly not written with end users in mind, and this has been the case for quite some time. However, openstacksdk is not yet to the point of being usable for end users - although good work is going on there to get it to be a basis for an end user python library. We give deployers so much flexibility, that in order to write even a SIMPLE program that uses OpenStack, an end user has to know generally four of five pieces of information to check for that are different ways that a deployer may have decided to do things. Example: - As a user, I want a compute instance that has an IP address that can do things. WELL, now you're screwed, because there is no standard way to do that. You may first want to try booting your instance and then checking to see if nova returns a network labeled public. You may get no networks. This indicates that your provider decided to deploy neutron, but as part of your account creation did not create default networks. You now need to go create a router, network and port in neutron. Now you can try again. Or, you may get networks back, but neither of them are labeled public - instead, you may get a public and a private address back in the network labeled private. Or, you may only get a private network back. This indicates that you may be expected to create a thing called a floating-ip. First, you need to verify that your provider has installed the floating-ip's extension. If they have, then you can create a floating-ip and attach it to your host. NOW - once you have those things done, you need to connect to your host and verify that its outbound networkin g has not been blocked by a thing called security groups, which you also may not have been expecting to exist, but I'll stop there, because the above is long enough. Every. Single. One. Of. Those. Cases. is real and has occurred across only the two public openstack clouds that infra uses. That means that our provisioning code takes every single one of them in to account, and anyone who writes code that wants to get a machine to use must take them all in to account or else their code is buggy. That's
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/03/2014 12:16 PM, Doug Hellmann wrote: On Sep 3, 2014, at 11:37 AM, Joe Gordon joe.gord...@gmail.com mailto:joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. Thanks for starting this discussion, Joe. It’s important for us to start working on “OpenStack” as a whole, in addition to our individual projects. Agreed. Thank you, Joe. 1. Sean has done a lot of analysis and started a spec on standardizing logging guidelines where he is gathering input from developers, deployers, and operators [1]. Because it is far enough for us to see real progress, it’s a good place for us to start experimenting with how to drive cross-project initiatives involving code and policy changes from outside of a single project. We have a couple of potentially related specs in Oslo as part of the oslo.log graduation work [2] [3], but I think most of the work will be within the applications. No surprise, I'm a huge +1 on this. 2. A longer-term topic is standardizing our notification content and format. See the thread Treating notifications as a contract” for details. We could set a goal for Kilo of establishing the requirements and proposing a spec, with implementation to begin in L. +1. 3. Another long-term topic is standardizing our APIs so that we use consistent terminology and formatting (I think we have at least 3 forms of errors returned now?). I’m not sure we have anyone ready to drive this, yet, so I don’t think it’s something to consider for Kilo. +10 Frankly, I believe this should be our #1 priority from a cross-project perspective. The inconsistencies in the current APIs (even within the same project's APIs!) is just poor form and since our REST APIs are the very first impression that we give to the outside developer community, it really is incumbent on us to make sure they are explicit, free of side-effects, well-documented, consistent, easy-to-use, and hide implementation details thoroughly. 4. I would also like to see the unified SDK and command line client projects continued (or resumed, I haven’t been following the SDK work closely). Both of those projects will eventually make using OpenStack clouds easier. It would be nice to see some movement towards a “user tools” program to encompass both of these projects, perhaps with an eye on incubation at the end of Kilo. +1 5. And we should also be considering the Python 3 porting work. We’ve made some progress with the Oslo libraries, with oslo.messaging eventlet still our main blocker. We need to put together a more concrete plan to finish that work and for tackling applications, as well as a team willing to help projects through their transition. This is very long term, but does need attention, and I think it’s reasonable to ask for a plan by the end of Kilo. +0 (only because I don't consider it a priority compared with the other things you've documented here) From a practical standpoint, we do need to work out details like where we make decisions about the plans for these projects once the general idea is approved. We’ve done some of this in the Oslo project in the past (log translations, for example) but I don’t think that’s the right place for projects at this scale. A new openstack-specs repository would give us a place to work on them, but we need to answer the question of how to decide what is approved. An openstack-specs repo might indeed be a nice place to put this cross-project, OpenStack-wide type of stuff. Best, -jay Doug best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 2014-09-07 8:14 PM, Monty Taylor wrote: If I were king ... 1. Caring about end user experience at all If I don't do anything at all next cycle, I will see the above fixed. Because it's embarrassing. Seriously. Try to use OpenStack from python some time. I dare you. [...] Between 2 and 3, maybe we can make a kilo release that has a net negative SLOC count. But, honestly, screw 2 and 3 - let's all just work on 1. On 2014-09-08 5:07 PM, James E. Blair wrote: 3) A real SDK OpenStack is so nearly impossible to use, that we have a substantial amount of code in the infrastructure program to do things that, frankly, we are a bit surprised that the client libraries don't do. Just getting an instance with an IP address is an enormous challenge, and something that took us years to get right. We still have problems deleting instances. We need client libraries (an SDK if you will) and command line clients that are easy for users to understand and work with, and hide the gory details of how the sausage is made. I 100% agree with both of you. The user experience is a MAJOR concern for us. I'm not a good writer able to articulate my thoughts as good as I would like but both Monty and James managed to communicate and summarize them. As a technical person, I often don't see the level of complexity in tools I use, I like challenges. I will gladly learn new complex stuff if needed. But when I first tried to use OpenStack client libraries, it was one of those times where I told myself: wow, it sucks. Especially around lack of consistency. Or as Monty said, the number of hoops you have to go through just to get a pingable instance. And it was and still is the opinion shared by some of my coworkers. If we could improve this aspect, I would be so happy. -- Mathieu ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Wed, 3 Sep 2014, Joe Gordon wrote: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. I think this is a good idea, but the timing (right at the end of j-3) might be a problematic. I'll jump in, despite being a newb; perhaps that perspective is useful. I'm sure these represent the biases of my limited experience, so apply salt as required and please be aware that I'm not entirely ignorant of the fact that there are diverse forces of history that lead to the present. Things I'd like to help address in Kilo: * Notifications as a contract[1], better yet as events, with events taking primacy over projects. The main thrust of this topic has been the development of standards that allow endpoints to have some confidence that what is sent or received is the right thing. This is a good thing, but I think misses a larger issue with the notification environment. One of my first BPs was to make Ceilometer capable of hearing notifications from Ironic that contain metrics generated from IPMI readings. I was shocked to discover that _code_ was required to make this happen; my newbie naivety thought it ought to just be a configuration change: a dict on the wire transformed into a data store. I was further shocked to discover that the message bus was being modeled as RPC. I had assumed that at the scale OpenStack is expected to operate most activity on the bus would be modeled as events and swarms of semi-autonomous agents would process them. In both cases my surprise was driven by what I perceived to be a bad ordering of priority between project and events in the discussion of making things happen. In this specific case the idea was presented as _Ironic_ needs to send some information to _Ceilometer_. Would it not be better to say: there is hardware health information that happens and various things can process? With that prioritization lots of different tools can produce and access the information. * Testing is slow and insufficiently reliable. Despite everyone's valiant effort this is true, we see evidence all over this list of trouble at the level of integration testing and testing during the development processing. My own experience has been that the tests (that is the way they are written and run) are relatively okay at preventing regression but not great at enabling TDD nor at being a pathway to understanding the code. This is probably because I think OO unittests are wack so just haven't developed the skill to read them well, but still: Tests are hard and that makes it harder to make good code. We can and should make it better. Facile testing makes it a lot easier to do tech debt cleanup that everyone(?) says we need. I reckon the efforts to library-ize tempest and things like Monty's dox will be useful tools. * Containers are a good idea, let's have more of them. There's a few different ways in which this matters: * Skate to where the puck will be, not where it is or ZOMG VMs are like so last decade. * dox, as above * Containerization of OpenStack services for easy deployment and development. Perhaps `dock_it` instead of `screen_it` in devstack. * Focus on user experience. This one is the most important. The size and number of projects that assemble to become OpenStack inevitably leads to difficulty seeing the big picture when focusing on the individual features within each project. OpenStack is big, hard to deploy and manage, and challenging to understand and use effectively. I _really_ like Sean Dague's idea (sorry, I've lost the ref) that OpenStack needs to be usable and useful to small universities that want to run relatively small clouds. I think this needs to be true _without_ the value-adds that our corporate benefactors package around the core to ease deployment and management. Or to put all this another way: As we are evaluating what we want to do and how we want to do it we need to think less about the projects and technologies that are involved and more about the actions and results that our efforts hope to allow and enable. [1] http://lists.openstack.org/pipermail/openstack-dev/2014-September/044748.html -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
Comments in line (added my thoughts on a couple of the targets Sean outlined). On Thursday, September 4, 2014, Sean Dague s...@dague.net wrote: Here is my top 5 list: 1. Functional Testing in Integrated projects The justification for this is here - http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html. We need projects to take more ownership of their functional testing so that by the time we get to integration testing we're not exposing really fundamental bugs like being unable to handle 2 requests at the same time. For Kilo: I think we can and should be able to make progress on this on all integrated projects, as well as the python clients (which are basically untested and often very broken). Big +1 from me on this. 2. Consistency in southbound interfaces (Logging first) Logging and notifications are south bound interfaces from OpenStack providing information to people, or machines, about what is going on. There is also a 3rd proposed south bound with osprofiler. For Kilo: I think it's reasonable to complete the logging standards and implement them. I expect notifications (which haven't quite kicked off) are going to take 2 cycles. I'd honestly *really* love to see a unification path for all the the southbound parts, logging, osprofiler, notifications, because there is quite a bit of overlap in the instrumentation/annotation inside the main code for all of these. I agree here as well. we should prioritize logging and use that success as the template for the other southbound parts. If we get profiler, notifications, etc it is a win, but hitting logging hard and getting it right is a huge step in the right direction. 3. API micro version path forward We have Cinder v2, Glance v2, Keystone v3. We've had them for a long time. When we started Juno cycle Nova used *none* of them. And with good reason, as the path forward was actually pretty bumpy. Nova has been trying to create a v3 for 3 cycles, and that effort collapsed under it's own weight. I think major API revisions in OpenStack are not actually possible any more, as there is too much intertia on existing interfaces. How to sanely and gradually evolve the OpenStack API is tremendously important, especially as a bunch of new projects are popping up that implement parts of it. We have the beginnings of a plan here in Nova, which now just needs a bunch of heavy lifting. For Kilo: A working microversion stack in at least one OpenStack service. Nova is probably closest, though Mark McClain wants to also take a spin on this in Neutron. I think if we could come up with a model that worked in both of those projects, we'd pick up some steam in making this long term approach across all of OpenStack. I like the concept but I absolutely want a definition on what micro versioning should look like. That way we don't end up with 10 different implementations of micro versioning. I am very concerned that we will see nova do this in one way, neutron in a different way, and then other projects taking bits and peices and ending up with something highly inconsistent. I am unsure how to resolve this consistency issue if multiple projects are implementing during the same cycle since retrofitting a different implementation could break the API contract. Generally speaking the micro versioning will be much more maintainable than the current major API version methods. 4. Post merge testing As explained here - http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html we could probably get a lot more bang for our buck if we had a smaller # of integration configurations in the pre merge gate, and a much more expansive set of post merge jobs. For Kilo: I think this could be implemented, it probably needs more hands than it has right now. 5. Consistent OpenStack python SDK / clients I think the client projects being inside the server programs has not served us well, especially as the # of servers has expanded. We as a project need to figure out how to get the SDK / unified client effort moving forward faster. For Kilo: I'm not sure how close to done we could take this, but this needs to become a larger overall push for the project as a whole, as I think our use exposed interface here is inhibiting adoption. -Sean -- Sean Dague http://dague.net Cheers, --Morgan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/03/2014 08:37 AM, Joe Gordon wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. If I were king ... 1. Caring about end user experience at all It's pretty clear, if you want to do things with OpenStack that are not running your own cloud, that we collectively have not valued the class of user who is a person who wants to use the cloud. Examples of this are that the other day I had to read a section of the admin guide to find information about how to boot a nova instance with a cinder volume attached all in one go. Spoiler alert, it doesn't work. Another spoiler alert - even though the python client has an option for requesting that a volume that is to be attached on boot be formatted in a particular way, this does not work for cinder volumes, which means it does not work for an end user - EVEN THOUGH this is a very basic thing to want. Our client libraries are clearly not written with end users in mind, and this has been the case for quite some time. However, openstacksdk is not yet to the point of being usable for end users - although good work is going on there to get it to be a basis for an end user python library. We give deployers so much flexibility, that in order to write even a SIMPLE program that uses OpenStack, an end user has to know generally four of five pieces of information to check for that are different ways that a deployer may have decided to do things. Example: - As a user, I want a compute instance that has an IP address that can do things. WELL, now you're screwed, because there is no standard way to do that. You may first want to try booting your instance and then checking to see if nova returns a network labeled public. You may get no networks. This indicates that your provider decided to deploy neutron, but as part of your account creation did not create default networks. You now need to go create a router, network and port in neutron. Now you can try again. Or, you may get networks back, but neither of them are labeled public - instead, you may get a public and a private address back in the network labeled private. Or, you may only get a private network back. This indicates that you may be expected to create a thing called a floating-ip. First, you need to verify that your provider has installed the floating-ip's extension. If they have, then you can create a floating-ip and attach it to your host. NOW - once you have those things done, you need to connect to your host and verify that its outbound networking has not been blocked by a thing called security groups, which you also may not have been expecting to exist, but I'll stop there, because the above is long enough. Every. Single. One. Of. Those. Cases. is real and has occurred across only the two public openstack clouds that infra uses. That means that our provisioning code takes every single one of them in to account, and anyone who writes code that wants to get a machine to use must take them all in to account or else their code is buggy. That's RIDICULOUS. So we should fix it. I'd say we should fix it by removing 1000% of the choices we've given deployers in this case, but I won't win there. So how about let's make at least one client library that encodes all of the above logic behind some simple task oriented API calls? How about we make that library not something which is just a repackaging of requests that does not contain intelligent, but instead something that is fundamentally usable. How about we have synchronous versions of all calls that do the polling and error checking. (if you attach a cinder volume to a nova instance, apparently, you need to continually re-fetch the volume from cinder and check its attachments property to see when the attach actually happens, because even though there is a python library call to do it, it's an async operation and there is no status field field to check, nor is there any indication that the operation is async, so when the call returns, the volume may or may not be attached.) This client library should contain exactly ZERO admin functions, because hopefully the number of people running OpenStack clouds will be smaller than the number of people who are
Re: [openstack-dev] Kilo Cycle Goals Exercise
Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. 1) Consistent/easy upgrading. all projects should follow a consistent model to the way they approach upgrading. it should actually work. - REST versioning - RPC versioning - db (data) migrations - ordering of procedures and clear documentation of it. [this has been begged for by operators, but not sure how we have delivered] 2) HA - ability to continue operations after been restated - functional tests to prove the above? 3) Make it easier for small business to give OpenStack a go - produce standard docker images as part of ci with super simple instructions on running them. -Angus On Thu, Sep 4, 2014 at 1:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/07/2014 09:12 PM, Angus Salkeld wrote: Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. I don't understand why you would encourage writers of blog posts you disagree with by sending them traffic. Anita. 1) Consistent/easy upgrading. all projects should follow a consistent model to the way they approach upgrading. it should actually work. - REST versioning - RPC versioning - db (data) migrations - ordering of procedures and clear documentation of it. [this has been begged for by operators, but not sure how we have delivered] 2) HA - ability to continue operations after been restated - functional tests to prove the above? 3) Make it easier for small business to give OpenStack a go - produce standard docker images as part of ci with super simple instructions on running them. -Angus On Thu, Sep 4, 2014 at 1:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 8 September 2014 13:27, Anita Kuno ante...@anteaya.info wrote: On 09/07/2014 09:12 PM, Angus Salkeld wrote: Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. I don't understand why you would encourage writers of blog posts you disagree with by sending them traffic. Because a) the post is whats disagreed with not the person; b) without the post to read we'd have no context here. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Mon, Sep 8, 2014 at 11:27 AM, Anita Kuno ante...@anteaya.info wrote: On 09/07/2014 09:12 PM, Angus Salkeld wrote: Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. I don't understand why you would encourage writers of blog posts you disagree with by sending them traffic. I am not disagreeing with the blog, I am saying we need to respond better to user/operator requests so that they don't feel the need to complain. -Angus Anita. 1) Consistent/easy upgrading. all projects should follow a consistent model to the way they approach upgrading. it should actually work. - REST versioning - RPC versioning - db (data) migrations - ordering of procedures and clear documentation of it. [this has been begged for by operators, but not sure how we have delivered] 2) HA - ability to continue operations after been restated - functional tests to prove the above? 3) Make it easier for small business to give OpenStack a go - produce standard docker images as part of ci with super simple instructions on running them. -Angus On Thu, Sep 4, 2014 at 1:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/07/2014 09:37 PM, Angus Salkeld wrote: On Mon, Sep 8, 2014 at 11:27 AM, Anita Kuno ante...@anteaya.info wrote: On 09/07/2014 09:12 PM, Angus Salkeld wrote: Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. I don't understand why you would encourage writers of blog posts you disagree with by sending them traffic. I am not disagreeing with the blog, I am saying we need to respond better to user/operator requests so that they don't feel the need to complain. -Angus Oh I misunderstood your use of the word prevent. Thanks for clarifying. Anita. Anita. 1) Consistent/easy upgrading. all projects should follow a consistent model to the way they approach upgrading. it should actually work. - REST versioning - RPC versioning - db (data) migrations - ordering of procedures and clear documentation of it. [this has been begged for by operators, but not sure how we have delivered] 2) HA - ability to continue operations after been restated - functional tests to prove the above? 3) Make it easier for small business to give OpenStack a go - produce standard docker images as part of ci with super simple instructions on running them. -Angus On Thu, Sep 4, 2014 at 1:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On 09/03/2014 11:37 AM, Joe Gordon wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html Here is my top 5 list: 1. Functional Testing in Integrated projects The justification for this is here - http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html. We need projects to take more ownership of their functional testing so that by the time we get to integration testing we're not exposing really fundamental bugs like being unable to handle 2 requests at the same time. For Kilo: I think we can and should be able to make progress on this on all integrated projects, as well as the python clients (which are basically untested and often very broken). 2. Consistency in southbound interfaces (Logging first) Logging and notifications are south bound interfaces from OpenStack providing information to people, or machines, about what is going on. There is also a 3rd proposed south bound with osprofiler. For Kilo: I think it's reasonable to complete the logging standards and implement them. I expect notifications (which haven't quite kicked off) are going to take 2 cycles. I'd honestly *really* love to see a unification path for all the the southbound parts, logging, osprofiler, notifications, because there is quite a bit of overlap in the instrumentation/annotation inside the main code for all of these. 3. API micro version path forward We have Cinder v2, Glance v2, Keystone v3. We've had them for a long time. When we started Juno cycle Nova used *none* of them. And with good reason, as the path forward was actually pretty bumpy. Nova has been trying to create a v3 for 3 cycles, and that effort collapsed under it's own weight. I think major API revisions in OpenStack are not actually possible any more, as there is too much intertia on existing interfaces. How to sanely and gradually evolve the OpenStack API is tremendously important, especially as a bunch of new projects are popping up that implement parts of it. We have the beginnings of a plan here in Nova, which now just needs a bunch of heavy lifting. For Kilo: A working microversion stack in at least one OpenStack service. Nova is probably closest, though Mark McClain wants to also take a spin on this in Neutron. I think if we could come up with a model that worked in both of those projects, we'd pick up some steam in making this long term approach across all of OpenStack. 4. Post merge testing As explained here - http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html we could probably get a lot more bang for our buck if we had a smaller # of integration configurations in the pre merge gate, and a much more expansive set of post merge jobs. For Kilo: I think this could be implemented, it probably needs more hands than it has right now. 5. Consistent OpenStack python SDK / clients I think the client projects being inside the server programs has not served us well, especially as the # of servers has expanded. We as a project need to figure out how to get the SDK / unified client effort moving forward faster. For Kilo: I'm not sure how close to done we could take this, but this needs to become a larger overall push for the project as a whole, as I think our use exposed interface here is inhibiting adoption. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Kilo Cycle Goals Exercise
As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Kilo Cycle Goals Exercise
On Sep 3, 2014, at 11:37 AM, Joe Gordon joe.gord...@gmail.com wrote: As you all know, there has recently been several very active discussions around how to improve assorted aspects of our development process. One idea that was brought up is to come up with a list of cycle goals/project priorities for Kilo [0]. To that end, I would like to propose an exercise as discussed in the TC meeting yesterday [1]: Have anyone interested (especially TC members) come up with a list of what they think the project wide Kilo cycle goals should be and post them on this thread by end of day Wednesday, September 10th. After which time we can begin discussing the results. The goal of this exercise is to help us see if our individual world views align with the greater community, and to get the ball rolling on a larger discussion of where as a project we should be focusing more time. Thanks for starting this discussion, Joe. It’s important for us to start working on “OpenStack” as a whole, in addition to our individual projects. 1. Sean has done a lot of analysis and started a spec on standardizing logging guidelines where he is gathering input from developers, deployers, and operators [1]. Because it is far enough for us to see real progress, it’s a good place for us to start experimenting with how to drive cross-project initiatives involving code and policy changes from outside of a single project. We have a couple of potentially related specs in Oslo as part of the oslo.log graduation work [2] [3], but I think most of the work will be within the applications. [1] https://review.openstack.org/#/c/91446/ [2] https://blueprints.launchpad.net/oslo.log/+spec/app-agnostic-logging-parameters [3] https://blueprints.launchpad.net/oslo.log/+spec/remove-context-adapter 2. A longer-term topic is standardizing our notification content and format. See the thread Treating notifications as a contract” for details. We could set a goal for Kilo of establishing the requirements and proposing a spec, with implementation to begin in L. 3. Another long-term topic is standardizing our APIs so that we use consistent terminology and formatting (I think we have at least 3 forms of errors returned now?). I’m not sure we have anyone ready to drive this, yet, so I don’t think it’s something to consider for Kilo. 4. I would also like to see the unified SDK and command line client projects continued (or resumed, I haven’t been following the SDK work closely). Both of those projects will eventually make using OpenStack clouds easier. It would be nice to see some movement towards a “user tools” program to encompass both of these projects, perhaps with an eye on incubation at the end of Kilo. 5. And we should also be considering the Python 3 porting work. We’ve made some progress with the Oslo libraries, with oslo.messaging eventlet still our main blocker. We need to put together a more concrete plan to finish that work and for tackling applications, as well as a team willing to help projects through their transition. This is very long term, but does need attention, and I think it’s reasonable to ask for a plan by the end of Kilo. From a practical standpoint, we do need to work out details like where we make decisions about the plans for these projects once the general idea is approved. We’ve done some of this in the Oslo project in the past (log translations, for example) but I don’t think that’s the right place for projects at this scale. A new openstack-specs repository would give us a place to work on them, but we need to answer the question of how to decide what is approved. Doug best, Joe Gordon [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html [1] http://eavesdrop.openstack.org/meetings/tc/2014/tc.2014-09-02-20.04.log.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
