Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Dmitry Tantsur wrote: On 05/09/2017 07:59 PM, Joshua Harlow wrote: Matt Riedemann wrote: On 5/8/2017 1:10 PM, Octave J. Orgeron wrote: I do agree that scalability and high-availability are definitely issues for OpenStack when you dig deeper into the sub-components. There is a lot of re-inventing of the wheel when you look at how distributed services are implemented inside of OpenStack and deficiencies. For some services you have a scheduler that can scale-out, but the conductor or worker process doesn't. A good example is cinder, where cinder-volume doesn't scale-out in a distributed manner and doesn't have a good mechanism for recovering when an instance fails. All across the services you see different methods for coordinating requests and tasks such as rabbitmq, redis, memcached, tooz, mysql, etc. So for an operator, you have to sift through those choices and configure the per-requisite infrastructure. This is a good example of a problem that should be solved with a single architecturally sound solution that all services can standardize on. There was an architecture workgroup specifically designed to understand past architectural decisions in OpenStack, and what the differences are in the projects, and how to address some of those issues, but from lack of participation the group dissolved shortly after the Barcelona summit. This is, again, another example of if you want to make these kinds of massive changes, it's going to take massive involvement and leadership. I agree on the 'massive changes, it's going to take massive involvement and leadership.' though I am not sure how such changes and involvement actually happens; especially nowadays where companies which such leadership are moving on to something else (k8s, mesos, or other...) So knowing that what are the options to actually make some kind of change occur? IMHO it must be driven by PTLs (yes I know they are always busy, to bad, so sad, lol). I'd like all the PTLs to get together and restart the arch-wg and make it a *requirement* that PTLs actually show up (and participate) in that group/meeting vs it just being a bunch of senior(ish) folks, such as myself, that showed up. Then if PTLs do not show up, I would start to say that the next time around they are running for PTL said lack of participation in the wider openstack vision should be known and potentially cause them to get kicked out (voted out?) of being a PTL in the future. How we have whom to blame. Problem solved? Not likely just yet problem solved, but sometimes tough love (IMHO) is needed. I believe it is, you may disagree and that's cool, but then I might give you some tough love also, lol. The problem in a lot of those cases comes down to development being detached from the actual use cases customers and operators are going to use in the real world. Having a distributed control plane with multiple instances of the api, scheduler, coordinator, and other processes is typically not testable without a larger hardware setup. When you get to large scale deployments, you need an active/active setup for the control plane. It's definitely not something you could develop for or test against on a single laptop with devstack. Especially, if you want to use more than a handful of the OpenStack services. I've heard *crazy* things about actual use cases customers and operators are doing because of the scaling limits that projects have (ie nova has a limit of 300 compute nodes so ABC customer will then setup X * 300 clouds to reach Y compute nodes because of that limit). IMHO I'm not even sure I would want to target said use-cases in the first place, because they feel messed up in the first place (and it seems bad/dumb? to go down the rabbit hole of targeting use-cases that were deployed to band-aid over the initial problems that created those use-cases/deployments in the first place). I think we can all agree with this. Developers don't have a lab with 1000 nodes lying around to hack on. There was OSIC but that's gone. I've been requesting help in Nova from companies to do scale testing and help us out with knowing what the major issues are, and report those back in a form so we can work on those issues. People will report there are issues, but not do the profiling, or at least not report the results of profiling, upstream to help us out. So again, this is really up to companies that have the resources to do this kind of scale testing and report back and help fix the issues upstream in the community. That doesn't require OpenStack 2.0. So how do we close that gap? The only way I really know is by having people that can see the problems from the get-go, instead of having to discover it at some later point (when it falls over and ABC customer starts to start having Y clouds just to reach the target number of compute nodes they want to reach). Now maybe the skill level in openstack (especially in regards to distributed systems) is just to low and the only real way to ga
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/09/2017 07:59 PM, Joshua Harlow wrote: Matt Riedemann wrote: On 5/8/2017 1:10 PM, Octave J. Orgeron wrote: I do agree that scalability and high-availability are definitely issues for OpenStack when you dig deeper into the sub-components. There is a lot of re-inventing of the wheel when you look at how distributed services are implemented inside of OpenStack and deficiencies. For some services you have a scheduler that can scale-out, but the conductor or worker process doesn't. A good example is cinder, where cinder-volume doesn't scale-out in a distributed manner and doesn't have a good mechanism for recovering when an instance fails. All across the services you see different methods for coordinating requests and tasks such as rabbitmq, redis, memcached, tooz, mysql, etc. So for an operator, you have to sift through those choices and configure the per-requisite infrastructure. This is a good example of a problem that should be solved with a single architecturally sound solution that all services can standardize on. There was an architecture workgroup specifically designed to understand past architectural decisions in OpenStack, and what the differences are in the projects, and how to address some of those issues, but from lack of participation the group dissolved shortly after the Barcelona summit. This is, again, another example of if you want to make these kinds of massive changes, it's going to take massive involvement and leadership. I agree on the 'massive changes, it's going to take massive involvement and leadership.' though I am not sure how such changes and involvement actually happens; especially nowadays where companies which such leadership are moving on to something else (k8s, mesos, or other...) So knowing that what are the options to actually make some kind of change occur? IMHO it must be driven by PTLs (yes I know they are always busy, to bad, so sad, lol). I'd like all the PTLs to get together and restart the arch-wg and make it a *requirement* that PTLs actually show up (and participate) in that group/meeting vs it just being a bunch of senior(ish) folks, such as myself, that showed up. Then if PTLs do not show up, I would start to say that the next time around they are running for PTL said lack of participation in the wider openstack vision should be known and potentially cause them to get kicked out (voted out?) of being a PTL in the future. How we have whom to blame. Problem solved? The problem in a lot of those cases comes down to development being detached from the actual use cases customers and operators are going to use in the real world. Having a distributed control plane with multiple instances of the api, scheduler, coordinator, and other processes is typically not testable without a larger hardware setup. When you get to large scale deployments, you need an active/active setup for the control plane. It's definitely not something you could develop for or test against on a single laptop with devstack. Especially, if you want to use more than a handful of the OpenStack services. I've heard *crazy* things about actual use cases customers and operators are doing because of the scaling limits that projects have (ie nova has a limit of 300 compute nodes so ABC customer will then setup X * 300 clouds to reach Y compute nodes because of that limit). IMHO I'm not even sure I would want to target said use-cases in the first place, because they feel messed up in the first place (and it seems bad/dumb? to go down the rabbit hole of targeting use-cases that were deployed to band-aid over the initial problems that created those use-cases/deployments in the first place). I think we can all agree with this. Developers don't have a lab with 1000 nodes lying around to hack on. There was OSIC but that's gone. I've been requesting help in Nova from companies to do scale testing and help us out with knowing what the major issues are, and report those back in a form so we can work on those issues. People will report there are issues, but not do the profiling, or at least not report the results of profiling, upstream to help us out. So again, this is really up to companies that have the resources to do this kind of scale testing and report back and help fix the issues upstream in the community. That doesn't require OpenStack 2.0. So how do we close that gap? The only way I really know is by having people that can see the problems from the get-go, instead of having to discover it at some later point (when it falls over and ABC customer starts to start having Y clouds just to reach the target number of compute nodes they want to reach). Now maybe the skill level in openstack (especially in regards to distributed systems) is just to low and the only real way to gather data is by having companies do scale testing (ie some kind of architecting things to work after they are deployed); if so that's sad... -Josh
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Matt Riedemann wrote: On 5/8/2017 1:10 PM, Octave J. Orgeron wrote: I do agree that scalability and high-availability are definitely issues for OpenStack when you dig deeper into the sub-components. There is a lot of re-inventing of the wheel when you look at how distributed services are implemented inside of OpenStack and deficiencies. For some services you have a scheduler that can scale-out, but the conductor or worker process doesn't. A good example is cinder, where cinder-volume doesn't scale-out in a distributed manner and doesn't have a good mechanism for recovering when an instance fails. All across the services you see different methods for coordinating requests and tasks such as rabbitmq, redis, memcached, tooz, mysql, etc. So for an operator, you have to sift through those choices and configure the per-requisite infrastructure. This is a good example of a problem that should be solved with a single architecturally sound solution that all services can standardize on. There was an architecture workgroup specifically designed to understand past architectural decisions in OpenStack, and what the differences are in the projects, and how to address some of those issues, but from lack of participation the group dissolved shortly after the Barcelona summit. This is, again, another example of if you want to make these kinds of massive changes, it's going to take massive involvement and leadership. I agree on the 'massive changes, it's going to take massive involvement and leadership.' though I am not sure how such changes and involvement actually happens; especially nowadays where companies which such leadership are moving on to something else (k8s, mesos, or other...) So knowing that what are the options to actually make some kind of change occur? IMHO it must be driven by PTLs (yes I know they are always busy, to bad, so sad, lol). I'd like all the PTLs to get together and restart the arch-wg and make it a *requirement* that PTLs actually show up (and participate) in that group/meeting vs it just being a bunch of senior(ish) folks, such as myself, that showed up. Then if PTLs do not show up, I would start to say that the next time around they are running for PTL said lack of participation in the wider openstack vision should be known and potentially cause them to get kicked out (voted out?) of being a PTL in the future. The problem in a lot of those cases comes down to development being detached from the actual use cases customers and operators are going to use in the real world. Having a distributed control plane with multiple instances of the api, scheduler, coordinator, and other processes is typically not testable without a larger hardware setup. When you get to large scale deployments, you need an active/active setup for the control plane. It's definitely not something you could develop for or test against on a single laptop with devstack. Especially, if you want to use more than a handful of the OpenStack services. I've heard *crazy* things about actual use cases customers and operators are doing because of the scaling limits that projects have (ie nova has a limit of 300 compute nodes so ABC customer will then setup X * 300 clouds to reach Y compute nodes because of that limit). IMHO I'm not even sure I would want to target said use-cases in the first place, because they feel messed up in the first place (and it seems bad/dumb? to go down the rabbit hole of targeting use-cases that were deployed to band-aid over the initial problems that created those use-cases/deployments in the first place). I think we can all agree with this. Developers don't have a lab with 1000 nodes lying around to hack on. There was OSIC but that's gone. I've been requesting help in Nova from companies to do scale testing and help us out with knowing what the major issues are, and report those back in a form so we can work on those issues. People will report there are issues, but not do the profiling, or at least not report the results of profiling, upstream to help us out. So again, this is really up to companies that have the resources to do this kind of scale testing and report back and help fix the issues upstream in the community. That doesn't require OpenStack 2.0. So how do we close that gap? The only way I really know is by having people that can see the problems from the get-go, instead of having to discover it at some later point (when it falls over and ABC customer starts to start having Y clouds just to reach the target number of compute nodes they want to reach). Now maybe the skill level in openstack (especially in regards to distributed systems) is just to low and the only real way to gather data is by having companies do scale testing (ie some kind of architecting things to work after they are deployed); if so that's sad... -Josh __ OpenStack Development Mailing List (not for usage
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5 May 2017 at 23:45, Chris Friesen wrote: > On 05/05/2017 02:04 PM, John Griffith wrote: >> I'd love some detail on this. What falls over? > It's been a while since I looked at it, but the main issue was that with LIO > as the iSCSI server there is no automatic traffic shaping/QoS between > guests, or between guests and the host. (There's no iSCSI server process to > assign to a cgroup, for example.) > > The throttling in IOPS/Bps is better than nothing, but doesn't really help > when you don't necessarily know what your total IOPS/bandwidth actually is > or how many volumes could get created. > > So you have one or more guests that are hammering on the disk as fast as > they can, combined with disks on the cinder server that maybe aren't as fast > as they should be, and it ended up slowing down all the other guests. And > if the host is using the same physical disks for things like glance > downloads or image conversion, then a badly-behaved guest can cause > performance issues on the host as well due to IO congestion. And if they > fill up the host caches they can even affect writes to other unrelated > devices. > > So yes, it wasn't the ideal hardware for the purpose, and there are some > tuning knobs, but in an ideal world we'd be able to reserve some > amount/percentage of bandwidth/IOPs for the host and have the rest shared > equally between all active iSCSI sessions (or unequally via a share > allocation if desired). So that's a complaint that it can't do magic with underspecced, overloaded hardware, plus a request for fair-share I/O or network scheduling? The latter is maybe something cinder could look at, though we're limited by the available technologies - array vendors tend to keep such things proprietary. Note that it is trivial to overload many SAN too, both the data path and the control path. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5/8/2017 1:24 PM, Octave J. Orgeron wrote: Now for Oracle, we definitely need more 3rd party CI to make it easier to test our drivers, components, and patches against so that it's easier for the community to validate things. However, it takes time, resources, and money to make that happen. Hopefully that will get sorted out over time. But even if we make all of the investments in setting that up, we still need the upstream teams to come to the table and not shun us away just because we are Oracle :) I'd recommend talking with Drew Thorstensen and the IBM PowerVM team. They persistently worked with the nova team over a few cycles to finally get to the point where we agreed to bring their driver in tree, but not before we knew they already had open-sourced their nova driver code (the driver code, not the hypervisor code) and were running 3rd party CI against it successfully. But they are really the example now for anyone trying to get new virt drivers into Nova. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5/8/2017 1:10 PM, Octave J. Orgeron wrote: I do agree that scalability and high-availability are definitely issues for OpenStack when you dig deeper into the sub-components. There is a lot of re-inventing of the wheel when you look at how distributed services are implemented inside of OpenStack and deficiencies. For some services you have a scheduler that can scale-out, but the conductor or worker process doesn't. A good example is cinder, where cinder-volume doesn't scale-out in a distributed manner and doesn't have a good mechanism for recovering when an instance fails. All across the services you see different methods for coordinating requests and tasks such as rabbitmq, redis, memcached, tooz, mysql, etc. So for an operator, you have to sift through those choices and configure the per-requisite infrastructure. This is a good example of a problem that should be solved with a single architecturally sound solution that all services can standardize on. There was an architecture workgroup specifically designed to understand past architectural decisions in OpenStack, and what the differences are in the projects, and how to address some of those issues, but from lack of participation the group dissolved shortly after the Barcelona summit. This is, again, another example of if you want to make these kinds of massive changes, it's going to take massive involvement and leadership. The problem in a lot of those cases comes down to development being detached from the actual use cases customers and operators are going to use in the real world. Having a distributed control plane with multiple instances of the api, scheduler, coordinator, and other processes is typically not testable without a larger hardware setup. When you get to large scale deployments, you need an active/active setup for the control plane. It's definitely not something you could develop for or test against on a single laptop with devstack. Especially, if you want to use more than a handful of the OpenStack services. I think we can all agree with this. Developers don't have a lab with 1000 nodes lying around to hack on. There was OSIC but that's gone. I've been requesting help in Nova from companies to do scale testing and help us out with knowing what the major issues are, and report those back in a form so we can work on those issues. People will report there are issues, but not do the profiling, or at least not report the results of profiling, upstream to help us out. So again, this is really up to companies that have the resources to do this kind of scale testing and report back and help fix the issues upstream in the community. That doesn't require OpenStack 2.0. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Excerpts from Jeremy Stanley's message of 2017-05-08 20:18:35 +: > On 2017-05-08 11:24:00 -0600 (-0600), Octave J. Orgeron wrote: > [...] > > none of those products that those drivers are written for are open > > sourced and they meet less resistance to committing code upstream. > > So I have to call BS on your comment that the community can't work > > with us because Solaris isn't open sourced. > > Totally not what I said. > > My point was that constantly reminding management of one of the > primary sources of friction might help. Working with free software > communities becomes easier when you don't outright reject their > values by deciding to cancel your open version of the thing you want > them to help you support. > > > Now for Oracle, we definitely need more 3rd party CI to make it > > easier to test our drivers, components, and patches against so > > that it's easier for the community to validate things. However, it > > takes time, resources, and money to make that happen. Hopefully > > that will get sorted out over time. > > And _this_ was entirely the rest of my point, yes. Your needs seem > quite similar to to those of VMWare, XenServer and HyperV, so I > fully expect Nova's core reviewers will hold Solaris support patches > to the same validation requirements. We can't run Solaris in our > upstream testing for the same reasons we can't run those other > examples (they're not free software), so the onus is on the vendor > to satisfy this need for continuous testing and reporting instead. > > > But even if we make all of the investments in setting that up, we > > still need the upstream teams to come to the table and not shun us > > away just because we are Oracle :) > [...] > > Smiley or no, the assertion that our quality assurance choices are > based on personal preference for some particular company over > another is still mildly offensive. Yes, let's keep in mind that the answer to these questions about stable branches are and have been the same no matter who asked them. Early in this thread we pointed out that this topic comes up regularly, from different sources, and the answer remains the same: Start by contributing to the existing stable maintenance, and either improve the processes and tools to make it easier to do more and/or recruit more people to spread the work around. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 2017-05-08 11:24:00 -0600 (-0600), Octave J. Orgeron wrote: [...] > none of those products that those drivers are written for are open > sourced and they meet less resistance to committing code upstream. > So I have to call BS on your comment that the community can't work > with us because Solaris isn't open sourced. Totally not what I said. My point was that constantly reminding management of one of the primary sources of friction might help. Working with free software communities becomes easier when you don't outright reject their values by deciding to cancel your open version of the thing you want them to help you support. > Now for Oracle, we definitely need more 3rd party CI to make it > easier to test our drivers, components, and patches against so > that it's easier for the community to validate things. However, it > takes time, resources, and money to make that happen. Hopefully > that will get sorted out over time. And _this_ was entirely the rest of my point, yes. Your needs seem quite similar to to those of VMWare, XenServer and HyperV, so I fully expect Nova's core reviewers will hold Solaris support patches to the same validation requirements. We can't run Solaris in our upstream testing for the same reasons we can't run those other examples (they're not free software), so the onus is on the vendor to satisfy this need for continuous testing and reporting instead. > But even if we make all of the investments in setting that up, we > still need the upstream teams to come to the table and not shun us > away just because we are Oracle :) [...] Smiley or no, the assertion that our quality assurance choices are based on personal preference for some particular company over another is still mildly offensive. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Hi Jeremy, I'm sure everyone would love to see Solaris open sourced again. I know I do! But unfortunately, it's not something within my power or control. However, there is the reality that OpenStack wouldn't be as successful without commercial companies contributing to it. A good example is all of the excellent work done in Cinder and Neutron, where we have hundreds of drivers for networking gear, SDNs, NAS/SAN storage, etc. In those cases, none of those products that those drivers are written for are open sourced and they meet less resistance to committing code upstream. So I have to call BS on your comment that the community can't work with us because Solaris isn't open sourced. Now for Oracle, we definitely need more 3rd party CI to make it easier to test our drivers, components, and patches against so that it's easier for the community to validate things. However, it takes time, resources, and money to make that happen. Hopefully that will get sorted out over time. But even if we make all of the investments in setting that up, we still need the upstream teams to come to the table and not shun us away just because we are Oracle :) Octave On 5/6/2017 6:26 AM, Jeremy Stanley wrote: On 2017-05-05 15:35:16 -0600 (-0600), Octave J. Orgeron wrote: [...] If it's in support of Oracle specific technologies such as Solaris, [...] we are often shunned away because it's not Linux or "mainstream" enough. A great example is how our Nova drivers for Solaris Zones, Kernel Zones, and LDoms are turned away. So we have to spend extra cycles maintaining our patches because they are shunned away from getting into the gate. [...] Hopefully I'm not hitting a sore spot here, but bring back OpenSolaris and the answer becomes simpler. The Microsoft and VMWare devs have similar challenges in this regard because if you attempt to combine your proprietary software with our free software, then it's not something we're going to be able to test upstream (and the burden to prove that it's working and soundly maintained is substantially greater than if what you're integrating is also free software). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
just kube-scheduler. This means many fewer things to manage for ops, and allows for faster communication times (lower latency). In theory OpenStack could scale out much better with the finer grained services. but I'm not sure thats really ever shown true in practice. Layered/eating own dogfood: * OpenStack assumes the operator will install "all the things". * K8s uses k8s to deploy lots of itself. You use kubelet with the same yaml file format normally used to deploy stuff to deploy etcd/kube-apiserver/kube-scheduler/kube-controller-manager to get a working base system. You then use the base system to launch sdn, ingress, service-discovery, the ui, etc. This means the learning required is substantially less when it comes to debugging problems, performing upgrades, etc, because its the same for the most part for k8s as it is for any other app running on it. The learning costs is way lower. Versioning: * With OpenStack, Upgrades are hard, mismatched version servers/agents are hard/impossible. * K8s, they support the controllers being 2 versions ahead of the clients. Its hard to bolt this on after the fact, but its also harder when you have multiple communications channels to do it with. Having to do it in http, sql, and rabbit messages makes it so much harder. Having only one place talk to the single datastore (etcd) makes that easier, as is only having one place everything interacts with the servers kube-apiserver. Some amount of distribution: * OpenStack components are generally expected to come from distro's. * K8s, Core pieces like kube-apiserver are distributed prebuilt and ready to go in container images if you chose to use them. Minimal silo's: * the various OpenStack projects are very silo'ed. * Most of the k8s subsystems currently are all tightly integrated with each other and are managed by the same teams. This has lead to architectural decisions that take more of the bigger picture into account. Under OpenStack's current model, each project does their own thing without too much design into how it all comes together in the end. A lot of problems spring from this. I'm sure k8s will get more and more silo'ed as it grows and matures. But right now, the lack of silo's really are speeding its development. Anyway, I think I'm done rambling for now... I do hope we can reflect on some of the differences between the projects and see if we can figure out how to pull in some of the good ideas from k8s. Thanks, Kevin From: Michał Jastrzębski [inc...@gmail.com] Sent: Friday, May 05, 2017 3:20 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time You are talking about OpenStack being hard because it's complex and at the same time you're talking about using "non-linux-mainstream" tools around. It's either flexibility or ease guys... Prescriptive is easy, flexible is hard. When you want to learn about linux you're not starting from compiling gentoo, you're installing ubuntu, click "next" until it's finished and just trust it's working, after some time you grow skills in linux and customize it to your needs. We are talking about software that runs physics, hpc-like clusters, mobile phone communication and wordpresses across thousands of companies. It won't ever be simple and prescriptive. Best we can do is as you said, hide complexity of it and allow smoother entry until someone learns complexity. No tooling will ever replace experienced operator, tooling can make easier time to gain this experience. You mentioned Kubernetes as good example, Kubernetes is still relatively young project and doesn't support some of things that you yourself said you need. As it grows, as options becomes available, it too will become more and more complex. On 5 May 2017 at 14:52, Octave J. Orgeron wrote: +1 On 5/5/2017 3:46 PM, Alex Schultz wrote: Sooo... I always get a little triggered when I hear that OpenStack is hard to deploy. We've spent last few years fixing it and I think it's pretty well fixed now. Even as we speak I'm deploying 500+ vms on OpenStack cluster I deployed last week within one day. No, you've written a complex tool (that you understand) to do it. That's not the same someone who is not familiar with OpenStack trying to deploy OpenStack. I too could quickly deploy a decently scaled infrastructure with some of the tools (fuel/tripleo/puppet/etc), but the reality is that each one of these tools is inherently hiding the complexities of OpenStack. Each (including yours) has their own flavor of assumptions baked in to make it work. That is also confusing for the end user who tries to switch between them and only gets some of the flexibility of each but then runs
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 08.05.2017 16:06, Doug Hellmann wrote: >>> >>> option #3: Do not support or nurse gates for stable branches upstream. >>> Instead, only create and close them and attach 3rd party gating, if >>> asked by contributors willing to support LTS and nurse their gates. >>> Note, closing a branch should be an exceptional case, if only no one >>> willing to support and gate it for a long. >> >> As i mentioned before, folks can join the Stable Team and make things >> like this happen. Won't happen by an email to the mailing list. Good point. There are *several* action items to do first, by the topic discussion results (as I see it): * Propose changes for stable branch maintaining policy (which an option of #1/#2/#3 to pick?) * Make a TC vote, I suppose, accept the changes officially * Given the accepted change is #3, start implementation steps, like: * Stop all stable/* gating jobs and merge freeze them * Join the Stable Team and make things happen, but first: * Get hardware for 3rd party CI gating, from willing to contribute operators' associated enterprises. Yes, the option #3 assumes that as a prerequisite for stable branches to "unfreeze" and step onto its "LTS adoption" path. * Setting up 3rd party CI, joining and learning from openstack infra team, for anyone who wants to help and who is going to consume and submit stable/* patches upstream instead of downstream forks, from now on. * Unfreeze stable branches * End up nominating more Stable Team core members with +2+1 from *operators* world and allowing changes to get in fast. So yes, it is important to encourage people to join, but one who is willing to contribute may be not enough to bring with him required hardware and operators from enterprises willing to contribute as well. *First of all, those folks need to be interested to stop patching things downstream*. And if started upside down, like jumping in and starting contributing things w/o other required changes, this would bring all but results we'd really like to see: * More folks pushing hard for making backports and upstream gating them against 'vanilla' stable branches, which *no one* of operators (read enterprises) consume as is - everything is being done downstream, rebased on top and needed to be retested again. * Folks abandoning Stable Team in a few months as they see the value of work done for 'vanilla' backporting and gating is near to a zero. >> >> Thanks, >> Dims > > Right. We need to change the tone of this thread from "you should do X" > to "I want to do X, where should I start?" > > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Bogdan Dobrelya, Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Excerpts from Davanum Srinivas (dims)'s message of 2017-05-08 06:12:51 -0400: > On Mon, May 8, 2017 at 3:52 AM, Bogdan Dobrelya wrote: > > On 06.05.2017 23:06, Doug Hellmann wrote: > >> Excerpts from Thierry Carrez's message of 2017-05-04 16:14:07 +0200: > >>> Chris Dent wrote: > On Wed, 3 May 2017, Drew Fisher wrote: > > "Most large customers move slowly and thus are running older versions, > > which are EOL upstream sometimes before they even deploy them." > > Can someone with more of the history give more detail on where the > expectation arose that upstream ought to be responsible things like > long term support? I had always understood that such features were > part of the way in which the corporately avaialable products added > value? > >>> > >>> We started with no stable branches, we were just producing releases and > >>> ensuring that updates vaguely worked from N-1 to N. There were a lot of > >>> distributions, and they all maintained their own stable branches, > >>> handling backport of critical fixes. That is a pretty classic upstream / > >>> downstream model. > >>> > >>> Some of us (including me) spotted the obvious duplication of effort > >>> there, and encouraged distributions to share that stable branch > >>> maintenance work rather than duplicate it. Here the stable branches were > >>> born, mostly through a collaboration between Red Hat developers and > >>> Canonical developers. All was well. Nobody was saying LTS back then > >>> because OpenStack was barely usable so nobody wanted to stay on any > >>> given version for too long. > >>> > >>> Maintaining stable branches has a cost. Keeping the infrastructure that > >>> ensures that stable branches are actually working is a complex endeavor > >>> that requires people to constantly pay attention. As time passed, we saw > >>> the involvement of distro packagers become more limited. We therefore > >>> limited the number of stable branches (and the length of time we > >>> maintained them) to match the staffing of that team. Fast-forward to > >>> today: the stable team is mostly one person, who is now out of his job > >>> and seeking employment. > >>> > >>> In parallel, OpenStack became more stable, so the demand for longer-term > >>> maintenance is stronger. People still expect "upstream" to provide it, > >>> not realizing upstream is made of people employed by various > >>> organizations, and that apparently their interest in funding work in > >>> that area is pretty dead. > >>> > >>> I agree that our current stable branch model is inappropriate: > >>> maintaining stable branches for one year only is a bit useless. But I > >>> only see two outcomes: > >>> > >>> 1/ The OpenStack community still thinks there is a lot of value in doing > >>> this work upstream, in which case organizations should invest resources > >>> in making that happen (starting with giving the Stable branch > >>> maintenance PTL a job), and then, yes, we should definitely consider > >>> things like LTS or longer periods of support for stable branches, to > >>> match the evolving usage of OpenStack. > >>> > >>> 2/ The OpenStack community thinks this is better handled downstream, and > >>> we should just get rid of them completely. This is a valid approach, and > >>> a lot of other open source communities just do that. > >> > >> Dropping stable branches completely would mean no upstream bugfix > >> or security releases at all. I don't think we want that. > >> > > > > I'd like to bring this up once again: > > > > option #3: Do not support or nurse gates for stable branches upstream. > > Instead, only create and close them and attach 3rd party gating, if > > asked by contributors willing to support LTS and nurse their gates. > > Note, closing a branch should be an exceptional case, if only no one > > willing to support and gate it for a long. > > As i mentioned before, folks can join the Stable Team and make things > like this happen. Won't happen by an email to the mailing list. > > Thanks, > Dims Right. We need to change the tone of this thread from "you should do X" to "I want to do X, where should I start?" Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Mon, May 8, 2017 at 3:52 AM, Bogdan Dobrelya wrote: > On 06.05.2017 23:06, Doug Hellmann wrote: >> Excerpts from Thierry Carrez's message of 2017-05-04 16:14:07 +0200: >>> Chris Dent wrote: On Wed, 3 May 2017, Drew Fisher wrote: > "Most large customers move slowly and thus are running older versions, > which are EOL upstream sometimes before they even deploy them." Can someone with more of the history give more detail on where the expectation arose that upstream ought to be responsible things like long term support? I had always understood that such features were part of the way in which the corporately avaialable products added value? >>> >>> We started with no stable branches, we were just producing releases and >>> ensuring that updates vaguely worked from N-1 to N. There were a lot of >>> distributions, and they all maintained their own stable branches, >>> handling backport of critical fixes. That is a pretty classic upstream / >>> downstream model. >>> >>> Some of us (including me) spotted the obvious duplication of effort >>> there, and encouraged distributions to share that stable branch >>> maintenance work rather than duplicate it. Here the stable branches were >>> born, mostly through a collaboration between Red Hat developers and >>> Canonical developers. All was well. Nobody was saying LTS back then >>> because OpenStack was barely usable so nobody wanted to stay on any >>> given version for too long. >>> >>> Maintaining stable branches has a cost. Keeping the infrastructure that >>> ensures that stable branches are actually working is a complex endeavor >>> that requires people to constantly pay attention. As time passed, we saw >>> the involvement of distro packagers become more limited. We therefore >>> limited the number of stable branches (and the length of time we >>> maintained them) to match the staffing of that team. Fast-forward to >>> today: the stable team is mostly one person, who is now out of his job >>> and seeking employment. >>> >>> In parallel, OpenStack became more stable, so the demand for longer-term >>> maintenance is stronger. People still expect "upstream" to provide it, >>> not realizing upstream is made of people employed by various >>> organizations, and that apparently their interest in funding work in >>> that area is pretty dead. >>> >>> I agree that our current stable branch model is inappropriate: >>> maintaining stable branches for one year only is a bit useless. But I >>> only see two outcomes: >>> >>> 1/ The OpenStack community still thinks there is a lot of value in doing >>> this work upstream, in which case organizations should invest resources >>> in making that happen (starting with giving the Stable branch >>> maintenance PTL a job), and then, yes, we should definitely consider >>> things like LTS or longer periods of support for stable branches, to >>> match the evolving usage of OpenStack. >>> >>> 2/ The OpenStack community thinks this is better handled downstream, and >>> we should just get rid of them completely. This is a valid approach, and >>> a lot of other open source communities just do that. >> >> Dropping stable branches completely would mean no upstream bugfix >> or security releases at all. I don't think we want that. >> > > I'd like to bring this up once again: > > option #3: Do not support or nurse gates for stable branches upstream. > Instead, only create and close them and attach 3rd party gating, if > asked by contributors willing to support LTS and nurse their gates. > Note, closing a branch should be an exceptional case, if only no one > willing to support and gate it for a long. As i mentioned before, folks can join the Stable Team and make things like this happen. Won't happen by an email to the mailing list. Thanks, Dims >> Doug >> >>> >>> The current reality in terms of invested resources points to (2). I >>> personally would prefer (1), because that lets us address security >>> issues more efficiently and avoids duplicating effort downstream. But >>> unfortunately I don't control where development resources are posted. >>> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > -- > Best regards, > Bogdan Dobrelya, > Irc #bogdando > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 06.05.2017 23:06, Doug Hellmann wrote: > Excerpts from Thierry Carrez's message of 2017-05-04 16:14:07 +0200: >> Chris Dent wrote: >>> On Wed, 3 May 2017, Drew Fisher wrote: "Most large customers move slowly and thus are running older versions, which are EOL upstream sometimes before they even deploy them." >>> >>> Can someone with more of the history give more detail on where the >>> expectation arose that upstream ought to be responsible things like >>> long term support? I had always understood that such features were >>> part of the way in which the corporately avaialable products added >>> value? >> >> We started with no stable branches, we were just producing releases and >> ensuring that updates vaguely worked from N-1 to N. There were a lot of >> distributions, and they all maintained their own stable branches, >> handling backport of critical fixes. That is a pretty classic upstream / >> downstream model. >> >> Some of us (including me) spotted the obvious duplication of effort >> there, and encouraged distributions to share that stable branch >> maintenance work rather than duplicate it. Here the stable branches were >> born, mostly through a collaboration between Red Hat developers and >> Canonical developers. All was well. Nobody was saying LTS back then >> because OpenStack was barely usable so nobody wanted to stay on any >> given version for too long. >> >> Maintaining stable branches has a cost. Keeping the infrastructure that >> ensures that stable branches are actually working is a complex endeavor >> that requires people to constantly pay attention. As time passed, we saw >> the involvement of distro packagers become more limited. We therefore >> limited the number of stable branches (and the length of time we >> maintained them) to match the staffing of that team. Fast-forward to >> today: the stable team is mostly one person, who is now out of his job >> and seeking employment. >> >> In parallel, OpenStack became more stable, so the demand for longer-term >> maintenance is stronger. People still expect "upstream" to provide it, >> not realizing upstream is made of people employed by various >> organizations, and that apparently their interest in funding work in >> that area is pretty dead. >> >> I agree that our current stable branch model is inappropriate: >> maintaining stable branches for one year only is a bit useless. But I >> only see two outcomes: >> >> 1/ The OpenStack community still thinks there is a lot of value in doing >> this work upstream, in which case organizations should invest resources >> in making that happen (starting with giving the Stable branch >> maintenance PTL a job), and then, yes, we should definitely consider >> things like LTS or longer periods of support for stable branches, to >> match the evolving usage of OpenStack. >> >> 2/ The OpenStack community thinks this is better handled downstream, and >> we should just get rid of them completely. This is a valid approach, and >> a lot of other open source communities just do that. > > Dropping stable branches completely would mean no upstream bugfix > or security releases at all. I don't think we want that. > I'd like to bring this up once again: option #3: Do not support or nurse gates for stable branches upstream. Instead, only create and close them and attach 3rd party gating, if asked by contributors willing to support LTS and nurse their gates. Note, closing a branch should be an exceptional case, if only no one willing to support and gate it for a long. > Doug > >> >> The current reality in terms of invested resources points to (2). I >> personally would prefer (1), because that lets us address security >> issues more efficiently and avoids duplicating effort downstream. But >> unfortunately I don't control where development resources are posted. >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Bogdan Dobrelya, Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5/5/2017 6:44 PM, Fox, Kevin M wrote: Note, when I say OpenStack below, I'm talking about nova/glance/cinder/neutron/horizon/heat/octavia/designate. No offence to the other projects intended. just trying to constrain the conversation a bit... Those parts are fairly comparable to what k8s provides. I think part of your point is valid, that k8s isn't as feature rich in some ways, (networking for example), and will get more complex in time. But it has a huge amount of functionality for significantly less effort compared to an OpenStack deployment with similar functionality today. I think there are some major things different between the two projects that are really paying off for k8s over OpenStack right now. We can use those as learning opportunities moving forward or the gap will continue to widen, as will the user migrations away from OpenStack. These are mostly architectural things. Versions: * The real core of OpenStack is essentially version 1 + iterative changes. * k8s is essentially the third version of Borg. Plenty of room to ditch bad ideas/decisions. That means OpenStack's architecture has essentially grown organically rather then being as carefully thought out. The backwards compatibility has been a good goal, but its so hard to upgrade most places burn it down and stand up something new anyway so a lot of work with a lot less payoff then you would think. Maybe it is time to consider OpenStack version 2... I think OpenStack's greatest strength is its standardized api's. Thus far we've been changing the api's over time and keeping the implementation mostly the same... maybe we should consider keeping the api the same and switch some of the implementations out... It might take a while to get back to where we are now, but I suspect the overall solution would be much better now that we have so much experience with building the first one. k8s and OpenStack do largely the same thing. get in user request, schedule the resource onto some machines and allow management/lifecycle of the thing. Why then does k8s scalability goal target 5000 nodes and OpenStack really struggle with more then 300 nodes without a huge amount of extra work? I think its architecture. OpenStack really abuses rabbit, does a lot with relational databases that maybe are better done elsewhere, and forces isolation between projects that maybe is not the best solution. Part of it I think is combined services. They don't have separate services for cinder-api/nova-api,neutron-api/heat-api/etc. Just kube-apiserver. same with the *-schedulers, just kube-scheduler. This means many fewer things to manage for ops, and allows for faster communication times (lower latency). In theory OpenStack could scale out much better with the finer grained services. but I'm not sure thats really ever shown true in practice. Layered/eating own dogfood: * OpenStack assumes the operator will install "all the things". * K8s uses k8s to deploy lots of itself. You use kubelet with the same yaml file format normally used to deploy stuff to deploy etcd/kube-apiserver/kube-scheduler/kube-controller-manager to get a working base system. You then use the base system to launch sdn, ingress, service-discovery, the ui, etc. This means the learning required is substantially less when it comes to debugging problems, performing upgrades, etc, because its the same for the most part for k8s as it is for any other app running on it. The learning costs is way lower. Versioning: * With OpenStack, Upgrades are hard, mismatched version servers/agents are hard/impossible. * K8s, they support the controllers being 2 versions ahead of the clients. Its hard to bolt this on after the fact, but its also harder when you have multiple communications channels to do it with. Having to do it in http, sql, and rabbit messages makes it so much harder. Having only one place talk to the single datastore (etcd) makes that easier, as is only having one place everything interacts with the servers kube-apiserver. Some amount of distribution: * OpenStack components are generally expected to come from distro's. * K8s, Core pieces like kube-apiserver are distributed prebuilt and ready to go in container images if you chose to use them. Minimal silo's: * the various OpenStack projects are very silo'ed. * Most of the k8s subsystems currently are all tightly integrated with each other and are managed by the same teams. This has lead to architectural decisions that take more of the bigger picture into account. Under OpenStack's current model, each project does their own thing without too much design into how it all comes together in the end. A lot of problems spring from this. I'm sure k8s will get more and more silo'ed as it grows and matures. But right now, the lack of silo's really are speeding its development. Anyway, I think I'm done rambling for now... I do hope we can reflect on some of the differences between the project
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Excerpts from Thierry Carrez's message of 2017-05-04 16:14:07 +0200: > Chris Dent wrote: > > On Wed, 3 May 2017, Drew Fisher wrote: > >> "Most large customers move slowly and thus are running older versions, > >> which are EOL upstream sometimes before they even deploy them." > > > > Can someone with more of the history give more detail on where the > > expectation arose that upstream ought to be responsible things like > > long term support? I had always understood that such features were > > part of the way in which the corporately avaialable products added > > value? > > We started with no stable branches, we were just producing releases and > ensuring that updates vaguely worked from N-1 to N. There were a lot of > distributions, and they all maintained their own stable branches, > handling backport of critical fixes. That is a pretty classic upstream / > downstream model. > > Some of us (including me) spotted the obvious duplication of effort > there, and encouraged distributions to share that stable branch > maintenance work rather than duplicate it. Here the stable branches were > born, mostly through a collaboration between Red Hat developers and > Canonical developers. All was well. Nobody was saying LTS back then > because OpenStack was barely usable so nobody wanted to stay on any > given version for too long. > > Maintaining stable branches has a cost. Keeping the infrastructure that > ensures that stable branches are actually working is a complex endeavor > that requires people to constantly pay attention. As time passed, we saw > the involvement of distro packagers become more limited. We therefore > limited the number of stable branches (and the length of time we > maintained them) to match the staffing of that team. Fast-forward to > today: the stable team is mostly one person, who is now out of his job > and seeking employment. > > In parallel, OpenStack became more stable, so the demand for longer-term > maintenance is stronger. People still expect "upstream" to provide it, > not realizing upstream is made of people employed by various > organizations, and that apparently their interest in funding work in > that area is pretty dead. > > I agree that our current stable branch model is inappropriate: > maintaining stable branches for one year only is a bit useless. But I > only see two outcomes: > > 1/ The OpenStack community still thinks there is a lot of value in doing > this work upstream, in which case organizations should invest resources > in making that happen (starting with giving the Stable branch > maintenance PTL a job), and then, yes, we should definitely consider > things like LTS or longer periods of support for stable branches, to > match the evolving usage of OpenStack. > > 2/ The OpenStack community thinks this is better handled downstream, and > we should just get rid of them completely. This is a valid approach, and > a lot of other open source communities just do that. Dropping stable branches completely would mean no upstream bugfix or security releases at all. I don't think we want that. Doug > > The current reality in terms of invested resources points to (2). I > personally would prefer (1), because that lets us address security > issues more efficiently and avoids duplicating effort downstream. But > unfortunately I don't control where development resources are posted. > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Excerpts from Octave J. Orgeron's message of 2017-05-05 15:35:16 -0600: > Hi Matt, > > And this is actually part of the problem for vendors. Many Oracle > engineers, including myself, have tried to get features and fixes pushed > upstream. While that may sound easy, the reality is that it isn't! In > many cases, it takes months for us to get something in or we get shot > down altogether. Here are the big issues we run into: > > * If it's in support of Oracle specific technologies such as Solaris, > ZFS, MySQL Cluster, etc. we are often shunned away because it's not > Linux or "mainstream" enough. A great example is how our Nova > drivers for Solaris Zones, Kernel Zones, and LDoms are turned away. > So we have to spend extra cycles maintaining our patches because > they are shunned away from getting into the gate. > * If we release an OpenStack distribution and a year later, a major > CVE security bug comes along.. we will patch it. But is there a way > for us to push those changes back in? No, because the branch for > that release is EOL'd and burned. So we have to maintain our own > copy of the repos so we have something to work against. > * Taking a release and productizing it takes more than just pulling > the git repo and building packages. It requires integrated testing > on a given OS distribution, hardware, and infrastructure. We have to > test it against our own products and handle upgrades from the > previous product release. We have to make sure it works for > customers. Then we have to spin up our distribution, documentation, etc. > > Lastly, just throwing resources at this isn't going to solve the > cultural or logistics problems. Everyone has to work together and Oracle Can you expand on what you see as cultural and logistical problems? Doug > will continue to try and work with the community. If other vendors, > customers, and operators are willing to work together to build an LTS > branch and the governance around it, then Oracle will support that > effort. But to go it alone I think is risky for any single individual or > vendor. It's pretty obvious that over the past year, a lot of vendors > that were ponying up efforts have had to pull the plug on their > investments. A lot of the issues that I've out-lined effect the > bottom-line for OpenStack vendors. This is not about which vendor does > more or less or who has the bigger budget to spend. It's about making it > easier for vendors to support and for customers to consume. > > Octave > > On 5/5/2017 2:40 PM, Matt Riedemann wrote: > > > > If you're spending exorbitant amounts of time patching in your forks > > to keep up with the upstream code, then you're doing the wrong thing. > > Upstream your changes, or work against the APIs, or try to get the > > APIs you need upstream to build on for your downstream features. > > Otherwise this is all just burden you've put on yourself and I can't > > justify an LTS support model because it might make someone's > > downstream fork strategy easier to manage. As noted earlier, I don't > > see Oracle developers leading the way upstream. If you want to see > > major changes, then contribute those resources, get involved and make > > a lasting effect. > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 2017-05-05 15:35:16 -0600 (-0600), Octave J. Orgeron wrote: [...] > If it's in support of Oracle specific technologies such as Solaris, [...] > we are often shunned away because it's not Linux or "mainstream" > enough. A great example is how our Nova drivers for Solaris Zones, > Kernel Zones, and LDoms are turned away. So we have to spend extra > cycles maintaining our patches because they are shunned away from > getting into the gate. [...] Hopefully I'm not hitting a sore spot here, but bring back OpenSolaris and the answer becomes simpler. The Microsoft and VMWare devs have similar challenges in this regard because if you attempt to combine your proprietary software with our free software, then it's not something we're going to be able to test upstream (and the burden to prove that it's working and soundly maintained is substantially greater than if what you're integrating is also free software). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
t of the k8s subsystems currently are all tightly integrated with each other and are managed by the same teams. This has lead to architectural decisions that take more of the bigger picture into account. Under OpenStack's current model, each project does their own thing without too much design into how it all comes together in the end. A lot of problems spring from this. I'm sure k8s will get more and more silo'ed as it grows and matures. But right now, the lack of silo's really are speeding its development. No disagreement from me, I am though unsure how to correct the course since it feels like people are just trying to avoid themselves drowning vs. focusing on the architecture problems (and imho fixing those architecture problems needs to be done real quick, and that is in itself scary). Though I'm definitely open to ideas and I'd like to gain peoples trust back (though I do not really know if such a thing is possible in the software industry). Ideas? Anyway, I think I'm done rambling for now... I do hope we can reflect on some of the differences between the projects and see if we can figure out how to pull in some of the good ideas from k8s. Thanks, Kevin From: Michał Jastrzębski [inc...@gmail.com] Sent: Friday, May 05, 2017 3:20 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time You are talking about OpenStack being hard because it's complex and at the same time you're talking about using "non-linux-mainstream" tools around. It's either flexibility or ease guys... Prescriptive is easy, flexible is hard. When you want to learn about linux you're not starting from compiling gentoo, you're installing ubuntu, click "next" until it's finished and just trust it's working, after some time you grow skills in linux and customize it to your needs. We are talking about software that runs physics, hpc-like clusters, mobile phone communication and wordpresses across thousands of companies. It won't ever be simple and prescriptive. Best we can do is as you said, hide complexity of it and allow smoother entry until someone learns complexity. No tooling will ever replace experienced operator, tooling can make easier time to gain this experience. You mentioned Kubernetes as good example, Kubernetes is still relatively young project and doesn't support some of things that you yourself said you need. As it grows, as options becomes available, it too will become more and more complex. On 5 May 2017 at 14:52, Octave J. Orgeron wrote: +1 On 5/5/2017 3:46 PM, Alex Schultz wrote: Sooo... I always get a little triggered when I hear that OpenStack is hard to deploy. We've spent last few years fixing it and I think it's pretty well fixed now. Even as we speak I'm deploying 500+ vms on OpenStack cluster I deployed last week within one day. No, you've written a complex tool (that you understand) to do it. That's not the same someone who is not familiar with OpenStack trying to deploy OpenStack. I too could quickly deploy a decently scaled infrastructure with some of the tools (fuel/tripleo/puppet/etc), but the reality is that each one of these tools is inherently hiding the complexities of OpenStack. Each (including yours) has their own flavor of assumptions baked in to make it work. That is also confusing for the end user who tries to switch between them and only gets some of the flexibility of each but then runs face first into each tool's short comings. Rather than assuming a tool has solved it (which it hasn't or we'd all be using the same one by now), how about we take some time to understand why we've had to write these tools in the first place and see if there's something we improve on? Learning the tool to deploy OpenStack is not the same as deploying OpenStack, managing it, and turning it around for the true cloud end user to consume. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
an reflect on some of the differences between the projects and see if we can figure out how to pull in some of the good ideas from k8s. Thanks, Kevin From: Michał Jastrzębski [inc...@gmail.com] Sent: Friday, May 05, 2017 3:20 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time You are talking about OpenStack being hard because it's complex and at the same time you're talking about using "non-linux-mainstream" tools around. It's either flexibility or ease guys... Prescriptive is easy, flexible is hard. When you want to learn about linux you're not starting from compiling gentoo, you're installing ubuntu, click "next" until it's finished and just trust it's working, after some time you grow skills in linux and customize it to your needs. We are talking about software that runs physics, hpc-like clusters, mobile phone communication and wordpresses across thousands of companies. It won't ever be simple and prescriptive. Best we can do is as you said, hide complexity of it and allow smoother entry until someone learns complexity. No tooling will ever replace experienced operator, tooling can make easier time to gain this experience. You mentioned Kubernetes as good example, Kubernetes is still relatively young project and doesn't support some of things that you yourself said you need. As it grows, as options becomes available, it too will become more and more complex. On 5 May 2017 at 14:52, Octave J. Orgeron wrote: > +1 > > On 5/5/2017 3:46 PM, Alex Schultz wrote: >> >> >>> Sooo... I always get a little triggered when I hear that OpenStack is >>> hard to deploy. We've spent last few years fixing it and I think it's >>> pretty well fixed now. Even as we speak I'm deploying 500+ vms on >>> OpenStack cluster I deployed last week within one day. >>> >> No, you've written a complex tool (that you understand) to do it. >> That's not the same someone who is not familiar with OpenStack trying >> to deploy OpenStack. I too could quickly deploy a decently scaled >> infrastructure with some of the tools (fuel/tripleo/puppet/etc), but >> the reality is that each one of these tools is inherently hiding the >> complexities of OpenStack. Each (including yours) has their own >> flavor of assumptions baked in to make it work. That is also >> confusing for the end user who tries to switch between them and only >> gets some of the flexibility of each but then runs face first into >> each tool's short comings. Rather than assuming a tool has solved it >> (which it hasn't or we'd all be using the same one by now), how about >> we take some time to understand why we've had to write these tools in >> the first place and see if there's something we improve on? Learning >> the tool to deploy OpenStack is not the same as deploying OpenStack, >> managing it, and turning it around for the true cloud end user to >> consume. >> > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 02:34 PM, Matt Riedemann wrote: I don't have a nice way to wrap this up. I'm depressed and just want to go outside. I don't expect pleasant replies to my rant, so I probably won't reply again after this anyway since this is always a never-ending back and forth which just leaves everyone bitter. For what it's worth, I think that there are always going to be disagreements of this sort. The upstream developers want to see things get done upstream. Makes sense. The downstream teams that add stuff on top of vanilla OpenStack usually do it because customers want functionality, not just for fun or to cause vendor lock-in. And sometimes those things that get added aren't easily upstreamable, either because the use-case is too specific, or because the implementation doesn't match upstream standards, or because it's too tightly coupled to other code outside of OpenStack. Sometimes there just aren't the resources available to push everything upstream. This leads to downstream distros carrying private patches, which naturally makes them want less upstream churn. I don't think there's any magic answer to this other than to encourage downstream folks to continue to try to push things upstream, with the expectation that not everything will make it. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 02:04 PM, John Griffith wrote: On Fri, May 5, 2017 at 11:24 AM, Chris Friesen mailto:chris.frie...@windriver.com>> wrote: On 05/05/2017 10:48 AM, Chris Dent wrote: Would it be accurate to say, then, that from your perpsective the tendency of OpenStack to adopt new projects willy nilly contributes to the sense of features winning out over deployment, configuration and usability issues? Personally I don't care about the new projects...if I'm not using them I can ignore them, and if I am using them then I'll pay attention to them. But within existing established projects there are some odd gaps. Like nova hasn't implemented cold-migration or resize (or live-migration) of an instance with LVM local storage if you're using libvirt. Image properties get validated, but not flavor extra-specs or instance metadata. Cinder theoretically supports LVM/iSCSI, but if you actually try to use it for anything stressful it falls over. Oh really? I'd love some detail on this. What falls over? It's been a while since I looked at it, but the main issue was that with LIO as the iSCSI server there is no automatic traffic shaping/QoS between guests, or between guests and the host. (There's no iSCSI server process to assign to a cgroup, for example.) The throttling in IOPS/Bps is better than nothing, but doesn't really help when you don't necessarily know what your total IOPS/bandwidth actually is or how many volumes could get created. So you have one or more guests that are hammering on the disk as fast as they can, combined with disks on the cinder server that maybe aren't as fast as they should be, and it ended up slowing down all the other guests. And if the host is using the same physical disks for things like glance downloads or image conversion, then a badly-behaved guest can cause performance issues on the host as well due to IO congestion. And if they fill up the host caches they can even affect writes to other unrelated devices. So yes, it wasn't the ideal hardware for the purpose, and there are some tuning knobs, but in an ideal world we'd be able to reserve some amount/percentage of bandwidth/IOPs for the host and have the rest shared equally between all active iSCSI sessions (or unequally via a share allocation if desired). Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5/4/2017 10:08 AM, Alex Schultz wrote: On Thu, May 4, 2017 at 5:32 AM, Chris Dent wrote: I know you're not speaking as the voice of your employer when making this message, so this is not directed at you, but from what I can tell Oracle's presense upstream (both reviews and commits) in Ocata and thus far in Pike has not been huge. Probably because they are still on Kilo. I don't want to stray off topic, but it seems worth clarifying that Oracle OpenStack for Oracle Linux 3.0 is based on Mitaka. http://www.oracle.com/technetwork/server-storage/openstack/linux/downloads/index.html Our messaging tends to get confused because we (Oracle) publish separate OpenStack distributions for Solaris and Linux. They are distinct products built and supported by different teams on their own schedules. It would be less confusing to the community and to customers if those efforts were coordinated or even consolidated, but so far that has not been possible. -- Michael Glasgow __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
You are talking about OpenStack being hard because it's complex and at the same time you're talking about using "non-linux-mainstream" tools around. It's either flexibility or ease guys... Prescriptive is easy, flexible is hard. When you want to learn about linux you're not starting from compiling gentoo, you're installing ubuntu, click "next" until it's finished and just trust it's working, after some time you grow skills in linux and customize it to your needs. We are talking about software that runs physics, hpc-like clusters, mobile phone communication and wordpresses across thousands of companies. It won't ever be simple and prescriptive. Best we can do is as you said, hide complexity of it and allow smoother entry until someone learns complexity. No tooling will ever replace experienced operator, tooling can make easier time to gain this experience. You mentioned Kubernetes as good example, Kubernetes is still relatively young project and doesn't support some of things that you yourself said you need. As it grows, as options becomes available, it too will become more and more complex. On 5 May 2017 at 14:52, Octave J. Orgeron wrote: > +1 > > On 5/5/2017 3:46 PM, Alex Schultz wrote: >> >> >>> Sooo... I always get a little triggered when I hear that OpenStack is >>> hard to deploy. We've spent last few years fixing it and I think it's >>> pretty well fixed now. Even as we speak I'm deploying 500+ vms on >>> OpenStack cluster I deployed last week within one day. >>> >> No, you've written a complex tool (that you understand) to do it. >> That's not the same someone who is not familiar with OpenStack trying >> to deploy OpenStack. I too could quickly deploy a decently scaled >> infrastructure with some of the tools (fuel/tripleo/puppet/etc), but >> the reality is that each one of these tools is inherently hiding the >> complexities of OpenStack. Each (including yours) has their own >> flavor of assumptions baked in to make it work. That is also >> confusing for the end user who tries to switch between them and only >> gets some of the flexibility of each but then runs face first into >> each tool's short comings. Rather than assuming a tool has solved it >> (which it hasn't or we'd all be using the same one by now), how about >> we take some time to understand why we've had to write these tools in >> the first place and see if there's something we improve on? Learning >> the tool to deploy OpenStack is not the same as deploying OpenStack, >> managing it, and turning it around for the true cloud end user to >> consume. >> > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
+1 On 5/5/2017 3:46 PM, Alex Schultz wrote: Sooo... I always get a little triggered when I hear that OpenStack is hard to deploy. We've spent last few years fixing it and I think it's pretty well fixed now. Even as we speak I'm deploying 500+ vms on OpenStack cluster I deployed last week within one day. No, you've written a complex tool (that you understand) to do it. That's not the same someone who is not familiar with OpenStack trying to deploy OpenStack. I too could quickly deploy a decently scaled infrastructure with some of the tools (fuel/tripleo/puppet/etc), but the reality is that each one of these tools is inherently hiding the complexities of OpenStack. Each (including yours) has their own flavor of assumptions baked in to make it work. That is also confusing for the end user who tries to switch between them and only gets some of the flexibility of each but then runs face first into each tool's short comings. Rather than assuming a tool has solved it (which it hasn't or we'd all be using the same one by now), how about we take some time to understand why we've had to write these tools in the first place and see if there's something we improve on? Learning the tool to deploy OpenStack is not the same as deploying OpenStack, managing it, and turning it around for the true cloud end user to consume. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 1:02 PM, Michał Jastrzębski wrote: > On 5 May 2017 at 11:33, Alex Schultz wrote: >> On Fri, May 5, 2017 at 10:48 AM, Chris Dent wrote: >>> On Fri, 5 May 2017, Alex Schultz wrote: >>> You have to understand that as I'm mainly dealing with having to actually deploy/configure the software, when I see 'new project X' that does 'cool new things Y, Z' it makes me cringe. Because it's just added complexity for new features that who knows if they are actually going to be consumed by a majority of end users. I see a lot of new features for edge cases while the core functionally (insert the most used project configuration) still have awkward deployment, configuration and usability issues. But those aren't exciting so people don't want to work on them... >>> >>> >>> Would it be accurate to say, then, that from your perpsective the >>> tendency of OpenStack to adopt new projects willy nilly contributes >>> to the sense of features winning out over deployment, configuration >>> and usability issues? >>> >> >> It does not help. >> >>> I think a lot of project contributors may not really see it that way >>> because they think of their project and within that project there is >>> a constant effort to clean things up, address bugs and tech debt, >>> and try to slowly but surely evolve to some level of maturity. In >>> their eyes those new projects are something else separate from their >>> project. >>> >>> From the outside, however, it is all OpenStack and maybe it looks >>> like there's loads of diffuse attention. >>> >>> If that's the case, then a question is whether or not the people who >>> are spending time on those new projects would be spending time on >>> the older projects instead if the new projects didn't exist. I don't >>> know, but seems unlikely. >>> >> >> So there's a trade off and I don't think we can just restrict entry >> because some projects aren't user friendly. I see it as a common >> issue across all projects. Some are better than others, but what I >> would like to see is the bar for usability raised within the OpenStack >> community such that the end user (and deployer/operator) are all taken >> into consideration. For me the usability also goes with adoption. The >> easier it is to consume, the easier it would be to adopt something. >> If you take a look at what is required to configure OpenStack for a >> basic deployment, it is not easy to consume. If you were to compare >> the basic getting started/install guide for Kubernetes[0] vs >> OpenStack[1], you can see what I mean about complexity. I think just >> the install guide for neutron on a controller node[2] is about the >> same length as the kubernetes guide. And we think this is ok? We >> should keep adding additional installation/management complexity for >> each project? You could argue that OpenStack has more features or >> more flexible so it's apples to oranges but I don't think it has to be >> if we worked on better patterns for configuration/deployment/upgrades. >> It feels like OpenStack is the thing that you should pay professional >> services to deploy rather than I do it yourself. And I think that's a >> shame. > > Sooo... I always get a little triggered when I hear that OpenStack is > hard to deploy. We've spent last few years fixing it and I think it's > pretty well fixed now. Even as we speak I'm deploying 500+ vms on > OpenStack cluster I deployed last week within one day. > No, you've written a complex tool (that you understand) to do it. That's not the same someone who is not familiar with OpenStack trying to deploy OpenStack. I too could quickly deploy a decently scaled infrastructure with some of the tools (fuel/tripleo/puppet/etc), but the reality is that each one of these tools is inherently hiding the complexities of OpenStack. Each (including yours) has their own flavor of assumptions baked in to make it work. That is also confusing for the end user who tries to switch between them and only gets some of the flexibility of each but then runs face first into each tool's short comings. Rather than assuming a tool has solved it (which it hasn't or we'd all be using the same one by now), how about we take some time to understand why we've had to write these tools in the first place and see if there's something we improve on? Learning the tool to deploy OpenStack is not the same as deploying OpenStack, managing it, and turning it around for the true cloud end user to consume. > These problems aren't factor of OpenStack growing too fast, it's > tooling that people are using. Granted, it took some time for us to > build these tools, but we did build them. One of reasons why we could > build them is that OpenStack, after being turned into Big Tent allowed > us (Kolla) to quickly join "main herd" of OpenStack and innovate in > our own way. If we'd put lot's of barriers like incubation, we'd still > have same issue with deployment. Stability no
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
+1. From: Octave J. Orgeron [octave.orge...@oracle.com] Sent: Friday, May 05, 2017 1:23 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time Thank you Alex for the points you made below. These are some of the big issues that I see all OpenStack customers and operators struggling with. Regardless of the tools that the community or vendors put on the table, OpenStack is still a very complicated piece of technology to deploy and manage. Even if we look at trippleo, kolla, puppet, Fuel, JuJu, etc. they address very specific use-cases. Most will handle deploying a basic HA control plane and configure things that are suited for a dev/test setup, demo, or POC type of use-case. But do they handle a production ready deployment for a bank, telco, retailer, or government? Is everything configured to handle HA, scale, security, auditing, multi-tenancy, etc. with all of the knobs and options set the way the customer needs? Do we end up with ceilometer, aodh, gnocchi, ELK, etc. all configured optimally? How about networking and security? How about upgrades? Can you expect people to hit an upgrade button and not have anything break? Let's be realistic here.. these tools are all good starting points, but you're going to have to get your hands dirty at some level to configure OpenStack to fit your actual business and technical requirements. The life-cycle management of OpenStack is not easy and requires a lot of resources. Sure vendors can try to fill the void, but everything they build is on quicksand. This is why you see vendors and major consulting houses jumping all over this to fill the void with professional services. There are plenty of big shops that are now looking at ways to out-source the management of OpenStack because of how complex it is to manage and maintain. BTW, this is the biggest market sign that a product is too complicated. From a vendors perspective, it's incredibly difficult to keep up with the releases because once you get your automation tooling and any extra value-added components integrated with a release, it's more than likely already behind or EOL. Plus there won't be enough soak time with customers to adopt it! Not only that, but by the time you make something work for customers, there is a very high chance that the upstream version of those components will have changed enough that you'll have to either patch, re-architect, or slash and burn what you've already delivered to your customers. Not to mention it maybe impossible to upgrade your customers in a seamless or automated fashion. This is why customers will stick to an older release because the upgrade path is too painful. If you consider the realities of all of the above, what ends up happening? Enterprise customers will end up sticking to an older release or paying someone else to deal with the complexity. This places vendors at risk because there isn't a clear revenue model that can sustain the engineering overhead required for maintaining and developing their distribution. If customers aren't buy more or upgrading, then who's keeping the lights on? You can only charge so much for support. So they end up being bought or going under. Which leaves customers with fewer choices and more risk. So ultimately, the right way to fix this is to have an LTS branch and for there to be better governance around how new features are introduced. There needs to be more customer and vendor driven involvement to solidifying a core set of features that everyone can rely on working consistently across releases and upgrades. When new features or enhancements come along, there should be more emphasis on usability, sustainability, and upgradeability so that customers and vendors are not stuck. If we had an LTS branch and a solid governance model, both vendors and customers would benefit greatly. It would help buffer the chaos of the bleeding edge away from customers and allow vendors to deliver and support them properly. Octave On 5/5/2017 12:33 PM, Alex Schultz wrote: > So there's a trade off and I don't think we can just restrict entry > because some projects aren't user friendly. I see it as a common issue > across all projects. Some are better than others, but what I would > like to see is the bar for usability raised within the OpenStack > community such that the end user (and deployer/operator) are all taken > into consideration. For me the usability also goes with adoption. The > easier it is to consume, the easier it would be to adopt something. If > you take a look at what is required to configure OpenStack for a basic > deployment, it is not easy to consume. If you were to compare the > basic getting started/install guide for Kubernetes[0] vs OpenStack[1], > you can see what I mean about complexity. I think just th
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 05, 2017 at 04:17:29PM -0400, Jonathan Proulx wrote: > On Fri, May 05, 2017 at 02:04:43PM -0600, John Griffith wrote: > :On Fri, May 5, 2017 at 11:24 AM, Chris Friesen > > :> Cinder theoretically supports LVM/iSCSI, but if you actually try to use it > :> for anything stressful it falls over. > :> > : > :Oh really? > : > :I'd love some detail on this. What falls over? > > I'm a bit out of date on this personally, but we ditched all iSCSI a > few years ago becasue we found it generally flaky on Linux. We we were > motly using Equallogic SAN both for OpenStack and stand alone servers > but saw same issues with some other targets as well. > > So I wonder if this is a Cinder issue or just a Linux issue. > > What we saw fall over was any slight network bump would permanently > drop conenct to backing storage requiring reset. But as I say this > was decidedly not a Cinder issue. > > -Jon > Yeah, Cinder doesn't get in the data path, so it wouldn't be Cinder itself. LVM will configure the iSCSI target, but then that is all in the Linux and distro package realm. Same for iSCSI initiator. But since we have a lot of external storage, besides LVM, that uses the Linux iSCSI initiator, that makes me want to point fingers at it being an environment issue with the network or something interfering with the traffic. At any rate, if you do see issues with the way Cinder is configuring any of that, please do let us know. > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Hi Matt, And this is actually part of the problem for vendors. Many Oracle engineers, including myself, have tried to get features and fixes pushed upstream. While that may sound easy, the reality is that it isn't! In many cases, it takes months for us to get something in or we get shot down altogether. Here are the big issues we run into: * If it's in support of Oracle specific technologies such as Solaris, ZFS, MySQL Cluster, etc. we are often shunned away because it's not Linux or "mainstream" enough. A great example is how our Nova drivers for Solaris Zones, Kernel Zones, and LDoms are turned away. So we have to spend extra cycles maintaining our patches because they are shunned away from getting into the gate. * If we release an OpenStack distribution and a year later, a major CVE security bug comes along.. we will patch it. But is there a way for us to push those changes back in? No, because the branch for that release is EOL'd and burned. So we have to maintain our own copy of the repos so we have something to work against. * Taking a release and productizing it takes more than just pulling the git repo and building packages. It requires integrated testing on a given OS distribution, hardware, and infrastructure. We have to test it against our own products and handle upgrades from the previous product release. We have to make sure it works for customers. Then we have to spin up our distribution, documentation, etc. Lastly, just throwing resources at this isn't going to solve the cultural or logistics problems. Everyone has to work together and Oracle will continue to try and work with the community. If other vendors, customers, and operators are willing to work together to build an LTS branch and the governance around it, then Oracle will support that effort. But to go it alone I think is risky for any single individual or vendor. It's pretty obvious that over the past year, a lot of vendors that were ponying up efforts have had to pull the plug on their investments. A lot of the issues that I've out-lined effect the bottom-line for OpenStack vendors. This is not about which vendor does more or less or who has the bigger budget to spend. It's about making it easier for vendors to support and for customers to consume. Octave On 5/5/2017 2:40 PM, Matt Riedemann wrote: If you're spending exorbitant amounts of time patching in your forks to keep up with the upstream code, then you're doing the wrong thing. Upstream your changes, or work against the APIs, or try to get the APIs you need upstream to build on for your downstream features. Otherwise this is all just burden you've put on yourself and I can't justify an LTS support model because it might make someone's downstream fork strategy easier to manage. As noted earlier, I don't see Oracle developers leading the way upstream. If you want to see major changes, then contribute those resources, get involved and make a lasting effect. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 1:49 PM, Sean Dague wrote: > On 05/05/2017 12:36 PM, Alex Schultz wrote: >> On Fri, May 5, 2017 at 6:16 AM, Sean Dague wrote: >>> On 05/04/2017 11:08 AM, Alex Schultz wrote: >>> >>> The general statement of "people care more about features than >>> usability/stability" gets thrown around a lot. And gets lots of head >>> nodding. But rarely comes with specifics. >>> >>> Can we be specific about what feature work is outpacing the consumers >>> that don't help with usability/stability? >>> >> >> The cell v2 initial implementation was neither usable or stable (for >> my definition of stable). Yea you could say 'but it's a work and >> progress' and I would say, why is it required for the end user then? >> If I wanted to I could probably go back and go through every project >> and point out when a feature was added yet we still have a pile of >> outstanding issues. As Chris Friesen pointed out in his reply email, >> there are things out there are specifics if you go looking. You have >> to understand that as I'm mainly dealing with having to actually >> deploy/configure the software, when I see 'new project X' that does >> 'cool new things Y, Z' it makes me cringe. Because it's just added >> complexity for new features that who knows if they are actually going >> to be consumed by a majority of end users. I see a lot of new >> features for edge cases while the core functionally (insert the most >> used project configuration) still have awkward deployment, >> configuration and usability issues. But those aren't exciting so >> people don't want to work on them... > > Chris pointed out bugs to partially implemented features that have not > completed. Those are things that haven't gotten done. > > Calling cells v2 an unneeded feature seems kind of a stretch. There was > so much operator push to get cells v1 merged even though it was wildly > incomplete, and full of races and very weird unfixable bugs. It was > merged on user request because many large operators were patching it in, > out of tree. And cells v2 was exactly a usability and stability path out > of cells v1. > I didn't say it wasn't an unneeded feature. I said the "initial implementation as not usable or stable". This being due to missing commands for managing (operator needs) or stable because things had to change when people actually tried to use them (tooling/workflow changes). I understand why we need it (although storing credentials in a database table makes me cry), it's just that I wish it was more baked before made a requirement outside of devstack. > While there may have been bumps in the road getting there, calling it a > feature unrelated to stability and usability doesn't seem right to me. > That's more of a "I wish the following bits were done differently." > The whole point was that yes, I wish it was released differently such that it didn't end up being a multiple month affair of bug exposing and lack of implemented tooling when it was switched to be mandatory. That to me was the usability/stability problem. A feature was made mandatory without proper understanding of the implications on the end user. Thanks, -Alex > -Sean > > -- > Sean Dague > http://dague.net > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 3:39 PM, Davanum Srinivas wrote: > "If we had an LTS branch and a solid governance model" - The way to > make it happen is for everyone to show up in the Stable team, do the > work that is need to define/setup etc.. > > Who is up for it? Please show up in the next weekly meeting and get > active in the work needed to take care of the current stable work and > the work needed to setup a LTS. Thank you dims, this gets said every time we have this conversation, and at best new interest wanes after a month, at worst nothing changes. I'm going to adopt sdague's stance here and ask for specifics. Octave (quoted in dims' first line above) suggests LTS + governance is the solution. Make specific proposals that include actionable tasks. Include a clear problem statement so we know _which_ problem is being solved. The number one task for LTS is dims' suggestion above, show up next week. Number two is to sustain the level of interest that will be expressed next week for 6 months and prove that there are resources available and fully trained and operable before Newton is EOL. I can't judge if starting next week is soon enough to extend Netwon's EOL date, but starting in August most certainly is too late. (Newton is our first realistic opportunity based on the LTS status of the current testing configurations.) Oh, and the number zero task is to make sure our stable team PTL can stay for the party. dt -- Dean Troyer dtro...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5/5/2017 3:23 PM, Octave J. Orgeron wrote: From a vendors perspective, it's incredibly difficult to keep up with the releases because once you get your automation tooling and any extra value-added components integrated with a release, it's more than likely already behind or EOL. Plus there won't be enough soak time with customers to adopt it! Not only that, but by the time you make something work for customers, there is a very high chance that the upstream version of those components will have changed enough that you'll have to either patch, re-architect, or slash and burn what you've already delivered to your customers. Not to mention it maybe impossible to upgrade your customers in a seamless or automated fashion. This is why customers will stick to an older release because the upgrade path is too painful. If you're spending exorbitant amounts of time patching in your forks to keep up with the upstream code, then you're doing the wrong thing. Upstream your changes, or work against the APIs, or try to get the APIs you need upstream to build on for your downstream features. Otherwise this is all just burden you've put on yourself and I can't justify an LTS support model because it might make someone's downstream fork strategy easier to manage. As noted earlier, I don't see Oracle developers leading the way upstream. If you want to see major changes, then contribute those resources, get involved and make a lasting effect. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Octave, Folks, "If we had an LTS branch and a solid governance model" - The way to make it happen is for everyone to show up in the Stable team, do the work that is need to define/setup etc.. Who is up for it? Please show up in the next weekly meeting and get active in the work needed to take care of the current stable work and the work needed to setup a LTS. Thanks, Dims On Fri, May 5, 2017 at 4:23 PM, Octave J. Orgeron wrote: > Thank you Alex for the points you made below. These are some of the big > issues that I see all OpenStack customers and operators struggling with. > Regardless of the tools that the community or vendors put on the table, > OpenStack is still a very complicated piece of technology to deploy and > manage. Even if we look at trippleo, kolla, puppet, Fuel, JuJu, etc. they > address very specific use-cases. Most will handle deploying a basic HA > control plane and configure things that are suited for a dev/test setup, > demo, or POC type of use-case. But do they handle a production ready > deployment for a bank, telco, retailer, or government? Is everything > configured to handle HA, scale, security, auditing, multi-tenancy, etc. with > all of the knobs and options set the way the customer needs? Do we end up > with ceilometer, aodh, gnocchi, ELK, etc. all configured optimally? How > about networking and security? How about upgrades? Can you expect people to > hit an upgrade button and not have anything break? Let's be realistic here.. > these tools are all good starting points, but you're going to have to get > your hands dirty at some level to configure OpenStack to fit your actual > business and technical requirements. The life-cycle management of OpenStack > is not easy and requires a lot of resources. Sure vendors can try to fill > the void, but everything they build is on quicksand. > > This is why you see vendors and major consulting houses jumping all over > this to fill the void with professional services. There are plenty of big > shops that are now looking at ways to out-source the management of OpenStack > because of how complex it is to manage and maintain. BTW, this is the > biggest market sign that a product is too complicated. > > From a vendors perspective, it's incredibly difficult to keep up with the > releases because once you get your automation tooling and any extra > value-added components integrated with a release, it's more than likely > already behind or EOL. Plus there won't be enough soak time with customers > to adopt it! Not only that, but by the time you make something work for > customers, there is a very high chance that the upstream version of those > components will have changed enough that you'll have to either patch, > re-architect, or slash and burn what you've already delivered to your > customers. Not to mention it maybe impossible to upgrade your customers in a > seamless or automated fashion. This is why customers will stick to an older > release because the upgrade path is too painful. > > If you consider the realities of all of the above, what ends up happening? > Enterprise customers will end up sticking to an older release or paying > someone else to deal with the complexity. This places vendors at risk > because there isn't a clear revenue model that can sustain the engineering > overhead required for maintaining and developing their distribution. If > customers aren't buy more or upgrading, then who's keeping the lights on? > You can only charge so much for support. So they end up being bought or > going under. Which leaves customers with fewer choices and more risk. > > So ultimately, the right way to fix this is to have an LTS branch and for > there to be better governance around how new features are introduced. There > needs to be more customer and vendor driven involvement to solidifying a > core set of features that everyone can rely on working consistently across > releases and upgrades. When new features or enhancements come along, there > should be more emphasis on usability, sustainability, and upgradeability so > that customers and vendors are not stuck. > > If we had an LTS branch and a solid governance model, both vendors and > customers would benefit greatly. It would help buffer the chaos of the > bleeding edge away from customers and allow vendors to deliver and support > them properly. > > Octave > > On 5/5/2017 12:33 PM, Alex Schultz wrote: >> >> So there's a trade off and I don't think we can just restrict entry >> because some projects aren't user friendly. I see it as a common issue >> across all projects. Some are better than others, but what I would like to >> see is the bar for usability raised within the OpenStack community such that >> the end user (and deployer/operator) are all taken into consideration. For >> me the usability also goes with adoption. The easier it is to consume, the >> easier it would be to adopt something. If you take a look at what is >> required to configure OpenStack for a basic deplo
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5/5/2017 11:36 AM, Alex Schultz wrote: The cell v2 initial implementation was neither usable or stable (for my definition of stable). Yea you could say 'but it's a work and progress' and I would say, why is it required for the end user then? If I wanted to I could probably go back and go through every project and point out when a feature was added yet we still have a pile of outstanding issues. As Chris Friesen pointed out in his reply email, there are things out there are specifics if you go looking. You have to understand that as I'm mainly dealing with having to actually deploy/configure the software, when I see 'new project X' that does 'cool new things Y, Z' it makes me cringe. Because it's just added complexity for new features that who knows if they are actually going to be consumed by a majority of end users. I see a lot of new features for edge cases while the core functionally (insert the most used project configuration) still have awkward deployment, configuration and usability issues. But those aren't exciting so people don't want to work on them... We've talked about bug-fix only stability releases before and there is never enough support to do that. If I had a "go talk to x" every time I have to say no to someone's blueprint after we do spec freeze it would make my life much easier and less depressing. My first release as PTL in Newton I really wanted to focus in fixing bugs, technical debt, docs, broken features, testing gaps, etc. That's still a main focus of mine, but the constant grind of feature requests and pressure is going to make you change priorities sometimes. So I've accepted that Nova is going to approve somewhere around 70 blueprints per release (and merge less than that, but that's another problem), and we just have to be better about requiring the docs/tests at the time that the feature is added, before the code is merged, and follow up with people that said they'd deliver those things. We've also been pretty aggressive about deprecating a lot of old busted API and legacy code in Nova over the last couple of years. I'm sure there is a camp of people that would never want us to remove anything no matter how busted or not tested it is, or how only one virt driver supports that feature. If you want to get an idea about the horrible mess of resource tracking and complexity that goes into scheduling in Nova, attend Jay Pipes' talks on Placement at the summit (or watch the videos afterward). There are still gremlins in there that are news to me today. And yes, Ocata sucked. We lost our main cells v2 driver (alaski), we lost two core reviewers for the entire release basically (danpb was re-assigned off openstack, sdague was laid off from HPE and was looking for another job), and I personally was looking for a job and interviewing with companies for about 4 months, just trying to continue working upstream in some fashion. Several people put a lot of attention and care into the in-tree placement and cells v2 docs, but we dropped the ball on the install guide for those. I've admitted that before and I'll admit it again if it makes anyone feel any better. We didn't anticipate some of the issues that TripleO would have with deployment changes. I appreciate all of the work that Emilien put into working with us on raising those issues and being patient while we worked through them. The Kolla team was/is also a pleasure to work with, mostly because I don't hear from them. :) Big changes are hard to get right. We've also put a hell of a lot of effort into keeping rolling upgrades in mind when working on any of this stuff, but I don't know if anyone has ever acknowledged that, or appreciated it, maybe because if it's not something to complain about it's not worth mentioning. Anyway, you'll never hear me disagree that I'd love to put less of a focus on new features and instead focus on hardening the things we already have, but when push comes to shove it's not easy justifying the time you spend (if you're lucky enough to even pick what you want to work on at any given time) on stuff like that. For example, I don't hear anyone asking me every other day in IRC to fix the mess that is the shelve feature. It's also unfair to say that the rest of us who don't work in deployment projects think or care about users, be those end users of the cloud, or the operators. I'd say we put a lot of thought into how something we're working on is going to be consumed and try to make that as painless as possible. It doesn't always end up that way, or maybe isn't appreciated for the alternatives we didn't take. It's easy to throw stones from the outside. I don't have a nice way to wrap this up. I'm depressed and just want to go outside. I don't expect pleasant replies to my rant, so I probably won't reply again after this anyway since this is always a never-ending back and forth which just leaves everyone bitter. -- Thanks, Matt ___
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Thank you Alex for the points you made below. These are some of the big issues that I see all OpenStack customers and operators struggling with. Regardless of the tools that the community or vendors put on the table, OpenStack is still a very complicated piece of technology to deploy and manage. Even if we look at trippleo, kolla, puppet, Fuel, JuJu, etc. they address very specific use-cases. Most will handle deploying a basic HA control plane and configure things that are suited for a dev/test setup, demo, or POC type of use-case. But do they handle a production ready deployment for a bank, telco, retailer, or government? Is everything configured to handle HA, scale, security, auditing, multi-tenancy, etc. with all of the knobs and options set the way the customer needs? Do we end up with ceilometer, aodh, gnocchi, ELK, etc. all configured optimally? How about networking and security? How about upgrades? Can you expect people to hit an upgrade button and not have anything break? Let's be realistic here.. these tools are all good starting points, but you're going to have to get your hands dirty at some level to configure OpenStack to fit your actual business and technical requirements. The life-cycle management of OpenStack is not easy and requires a lot of resources. Sure vendors can try to fill the void, but everything they build is on quicksand. This is why you see vendors and major consulting houses jumping all over this to fill the void with professional services. There are plenty of big shops that are now looking at ways to out-source the management of OpenStack because of how complex it is to manage and maintain. BTW, this is the biggest market sign that a product is too complicated. From a vendors perspective, it's incredibly difficult to keep up with the releases because once you get your automation tooling and any extra value-added components integrated with a release, it's more than likely already behind or EOL. Plus there won't be enough soak time with customers to adopt it! Not only that, but by the time you make something work for customers, there is a very high chance that the upstream version of those components will have changed enough that you'll have to either patch, re-architect, or slash and burn what you've already delivered to your customers. Not to mention it maybe impossible to upgrade your customers in a seamless or automated fashion. This is why customers will stick to an older release because the upgrade path is too painful. If you consider the realities of all of the above, what ends up happening? Enterprise customers will end up sticking to an older release or paying someone else to deal with the complexity. This places vendors at risk because there isn't a clear revenue model that can sustain the engineering overhead required for maintaining and developing their distribution. If customers aren't buy more or upgrading, then who's keeping the lights on? You can only charge so much for support. So they end up being bought or going under. Which leaves customers with fewer choices and more risk. So ultimately, the right way to fix this is to have an LTS branch and for there to be better governance around how new features are introduced. There needs to be more customer and vendor driven involvement to solidifying a core set of features that everyone can rely on working consistently across releases and upgrades. When new features or enhancements come along, there should be more emphasis on usability, sustainability, and upgradeability so that customers and vendors are not stuck. If we had an LTS branch and a solid governance model, both vendors and customers would benefit greatly. It would help buffer the chaos of the bleeding edge away from customers and allow vendors to deliver and support them properly. Octave On 5/5/2017 12:33 PM, Alex Schultz wrote: So there's a trade off and I don't think we can just restrict entry because some projects aren't user friendly. I see it as a common issue across all projects. Some are better than others, but what I would like to see is the bar for usability raised within the OpenStack community such that the end user (and deployer/operator) are all taken into consideration. For me the usability also goes with adoption. The easier it is to consume, the easier it would be to adopt something. If you take a look at what is required to configure OpenStack for a basic deployment, it is not easy to consume. If you were to compare the basic getting started/install guide for Kubernetes[0] vs OpenStack[1], you can see what I mean about complexity. I think just the install guide for neutron on a controller node[2] is about the same length as the kubernetes guide. And we think this is ok? We should keep adding additional installation/management complexity for each project? You could argue that OpenStack has more features or more flexible so it's apples to oranges but I don't think
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 05, 2017 at 02:04:43PM -0600, John Griffith wrote: :On Fri, May 5, 2017 at 11:24 AM, Chris Friesen :> Cinder theoretically supports LVM/iSCSI, but if you actually try to use it :> for anything stressful it falls over. :> : :Oh really? : :I'd love some detail on this. What falls over? I'm a bit out of date on this personally, but we ditched all iSCSI a few years ago becasue we found it generally flaky on Linux. We we were motly using Equallogic SAN both for OpenStack and stand alone servers but saw same issues with some other targets as well. So I wonder if this is a Cinder issue or just a Linux issue. What we saw fall over was any slight network bump would permanently drop conenct to backing storage requiring reset. But as I say this was decidedly not a Cinder issue. -Jon __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
> +1. ocata's cell v2 stuff added a lot of extra required complexity > with no perceivable benefit to end users. If there was a long term > stable version, then putting it in the non lts release would have > been ok. In absence of lts, I would have recommended the cell v2 > stuff have been done in a branch instead and merged all together when > it provided something (pike I think) That's how cellsv1 was developed and that turned out spectacularly well. --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 11:24 AM, Chris Friesen wrote: > On 05/05/2017 10:48 AM, Chris Dent wrote: > > Would it be accurate to say, then, that from your perpsective the >> tendency of OpenStack to adopt new projects willy nilly contributes >> to the sense of features winning out over deployment, configuration >> and usability issues? >> > > Personally I don't care about the new projects...if I'm not using them I > can ignore them, and if I am using them then I'll pay attention to them. > > But within existing established projects there are some odd gaps. > > Like nova hasn't implemented cold-migration or resize (or live-migration) > of an instance with LVM local storage if you're using libvirt. > > Image properties get validated, but not flavor extra-specs or instance > metadata. > > Cinder theoretically supports LVM/iSCSI, but if you actually try to use it > for anything stressful it falls over. > Oh really? I'd love some detail on this. What falls over? > Some of the database pruning tools don't cover all the tables so the DB > gets bigger over time. > > I'm sure there are historical reasons for all of these, I'm just pointing > out some of the things that were surprising to me. > > Chris > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 10:09 AM, Chris Friesen wrote: > On 05/05/2017 06:16 AM, Sean Dague wrote: >> On 05/04/2017 11:08 AM, Alex Schultz wrote: > >>> Probably because they are still on Kilo. Not sure how much they could >>> be contributing to the current when their customers are demanding that >>> something is rock solid which by now looks nothing like the current >>> upstream. I think this is part of the problem as the upstream can >>> tend to outpace anyone else in terms of features or anything else. I >>> think the the bigger question could be what's the benefit of >>> continuing to press forward and add yet more features when consumers >>> cannot keep up to consume these? Personally I think usability (and >>> some stability) sometimes tends to take a backseat to features in the >>> upstream which is unfortunate because it makes these problems worse. >> >> The general statement of "people care more about features than >> usability/stability" gets thrown around a lot. And gets lots of head >> nodding. But rarely comes with specifics. >> >> Can we be specific about what feature work is outpacing the consumers >> that don't help with usability/stability? > > On the usability/stability front, in nova you still can't correctly > live-migrate if you have dedicated CPUs or hugepages, or a specific NUMA > topology. The commits for this have been under review since Kilo, but > never quite make it in. At the same time, there are no warnings or > errors to the user saying that it's not stable...it just migrates and > hopes that it doesn't collide with another instance. > > On the usability front, the new "openstack" client doesn't support > microversions, which limits its usefulness with nova. (I think some > folks are starting to look at this one.) Those are things that have not yet been done. Which there are lots. Would love for more of it to be done. But neither is actually an answer to what features that have no impact on usability/stability, are getting prioritized. I'm sorry about being pedantic here, but I really want anyone claiming that features without stability/usability improvements are trumping that work to identify the features in that category. (not just complain about things they wish also fit into the 5lb bag). -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 12:36 PM, Alex Schultz wrote: > On Fri, May 5, 2017 at 6:16 AM, Sean Dague wrote: >> On 05/04/2017 11:08 AM, Alex Schultz wrote: >> >> The general statement of "people care more about features than >> usability/stability" gets thrown around a lot. And gets lots of head >> nodding. But rarely comes with specifics. >> >> Can we be specific about what feature work is outpacing the consumers >> that don't help with usability/stability? >> > > The cell v2 initial implementation was neither usable or stable (for > my definition of stable). Yea you could say 'but it's a work and > progress' and I would say, why is it required for the end user then? > If I wanted to I could probably go back and go through every project > and point out when a feature was added yet we still have a pile of > outstanding issues. As Chris Friesen pointed out in his reply email, > there are things out there are specifics if you go looking. You have > to understand that as I'm mainly dealing with having to actually > deploy/configure the software, when I see 'new project X' that does > 'cool new things Y, Z' it makes me cringe. Because it's just added > complexity for new features that who knows if they are actually going > to be consumed by a majority of end users. I see a lot of new > features for edge cases while the core functionally (insert the most > used project configuration) still have awkward deployment, > configuration and usability issues. But those aren't exciting so > people don't want to work on them... Chris pointed out bugs to partially implemented features that have not completed. Those are things that haven't gotten done. Calling cells v2 an unneeded feature seems kind of a stretch. There was so much operator push to get cells v1 merged even though it was wildly incomplete, and full of races and very weird unfixable bugs. It was merged on user request because many large operators were patching it in, out of tree. And cells v2 was exactly a usability and stability path out of cells v1. While there may have been bumps in the road getting there, calling it a feature unrelated to stability and usability doesn't seem right to me. That's more of a "I wish the following bits were done differently." -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5/5/2017 9:30 AM, Thierry Carrez wrote: Davanum Srinivas wrote: I would encourage folks here to help the stable branches we have right now! Release/Requirements constantly wait on Stable team and Stable team is way short of hands. Please join #openstack-stable, throw your name in wiki etc (https://wiki.openstack.org/wiki/CrossProjectLiaisons#Stable_Branch) and get active. If we have trouble taking care of what we have now, how can we do more? Shameless plug: For those in Boston next week, you can join the following on-boarding session on Monday afternoon, to see what this work really means. It's not as hard or time-consuming as you'd think: https://www.openstack.org/summit/boston-2017/summit-schedule/events/18694/infraqarelease-mgmtregsstable-project-onboarding If you want to be extra lazy on a sunny Friday, watch a video: https://www.openstack.org/videos/video/openstack-stable-what-it-actually-means-to-maintain-stable-branches I also ran a cross-project session at the Newton summit about the concept of extending the life of the oldest stable branch: https://etherpad.openstack.org/p/stable-branch-eol-policy-newton I even made slides! https://docs.google.com/presentation/d/1k0mCHwRZ3_Z8zJw_WilsuTYYqnUDlY2PkgVJLz_xVQc/edit?usp=sharing In the end it was determined that the cost of doing this didn't justify the drain on, or lack of, resources to do this, particularly on the infra team to be supporting older distro versions once they were past LTS. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 5 May 2017 at 11:33, Alex Schultz wrote: > On Fri, May 5, 2017 at 10:48 AM, Chris Dent wrote: >> On Fri, 5 May 2017, Alex Schultz wrote: >> >>> You have to understand that as I'm mainly dealing with having to >>> actually deploy/configure the software, when I see 'new project X' >>> that does 'cool new things Y, Z' it makes me cringe. Because it's >>> just added complexity for new features that who knows if they are >>> actually going to be consumed by a majority of end users. I see a >>> lot of new features for edge cases while the core functionally >>> (insert the most used project configuration) still have awkward >>> deployment, configuration and usability issues. But those aren't >>> exciting so people don't want to work on them... >> >> >> Would it be accurate to say, then, that from your perpsective the >> tendency of OpenStack to adopt new projects willy nilly contributes >> to the sense of features winning out over deployment, configuration >> and usability issues? >> > > It does not help. > >> I think a lot of project contributors may not really see it that way >> because they think of their project and within that project there is >> a constant effort to clean things up, address bugs and tech debt, >> and try to slowly but surely evolve to some level of maturity. In >> their eyes those new projects are something else separate from their >> project. >> >> From the outside, however, it is all OpenStack and maybe it looks >> like there's loads of diffuse attention. >> >> If that's the case, then a question is whether or not the people who >> are spending time on those new projects would be spending time on >> the older projects instead if the new projects didn't exist. I don't >> know, but seems unlikely. >> > > So there's a trade off and I don't think we can just restrict entry > because some projects aren't user friendly. I see it as a common > issue across all projects. Some are better than others, but what I > would like to see is the bar for usability raised within the OpenStack > community such that the end user (and deployer/operator) are all taken > into consideration. For me the usability also goes with adoption. The > easier it is to consume, the easier it would be to adopt something. > If you take a look at what is required to configure OpenStack for a > basic deployment, it is not easy to consume. If you were to compare > the basic getting started/install guide for Kubernetes[0] vs > OpenStack[1], you can see what I mean about complexity. I think just > the install guide for neutron on a controller node[2] is about the > same length as the kubernetes guide. And we think this is ok? We > should keep adding additional installation/management complexity for > each project? You could argue that OpenStack has more features or > more flexible so it's apples to oranges but I don't think it has to be > if we worked on better patterns for configuration/deployment/upgrades. > It feels like OpenStack is the thing that you should pay professional > services to deploy rather than I do it yourself. And I think that's a > shame. Sooo... I always get a little triggered when I hear that OpenStack is hard to deploy. We've spent last few years fixing it and I think it's pretty well fixed now. Even as we speak I'm deploying 500+ vms on OpenStack cluster I deployed last week within one day. These problems aren't factor of OpenStack growing too fast, it's tooling that people are using. Granted, it took some time for us to build these tools, but we did build them. One of reasons why we could build them is that OpenStack, after being turned into Big Tent allowed us (Kolla) to quickly join "main herd" of OpenStack and innovate in our own way. If we'd put lot's of barriers like incubation, we'd still have same issue with deployment. Stability not always comes from age of project, sometimes change of methodology alltogether gives you better stability at the end. Big Tent is meant to allow this kind of innovation among other things. Setup I'm using now was deployed in similar manner as this short guide I wrote for Boston summit [1]. Deployment, upgrades and such are problems we're fixing as we go. Sometimes we make things harder (nova placement API caused a bit of a headache for deployment tools...), then we make it easier again with new feature. We might want to put some constrains in terms of significant, deployment-changing, features merge timelines, but that's logistics that we handle as community. I keep hearing that OpenStack lacks leadership, and that's true, but consider that "leadership" is always limiting for innovation. [1] https://github.com/inc0/kolla-ansible-workshop Cheers, Michal > Thanks, > -Alex > > [0] > https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/ > [1] https://docs.openstack.org/newton/install-guide-rdo/ > [2] > https://docs.openstack.org/newton/install-guide-rdo/neutron-controller-install.html > >> >> -- >> Chris Dent
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 10:48 AM, Chris Dent wrote: > On Fri, 5 May 2017, Alex Schultz wrote: > >> You have to understand that as I'm mainly dealing with having to >> actually deploy/configure the software, when I see 'new project X' >> that does 'cool new things Y, Z' it makes me cringe. Because it's >> just added complexity for new features that who knows if they are >> actually going to be consumed by a majority of end users. I see a >> lot of new features for edge cases while the core functionally >> (insert the most used project configuration) still have awkward >> deployment, configuration and usability issues. But those aren't >> exciting so people don't want to work on them... > > > Would it be accurate to say, then, that from your perpsective the > tendency of OpenStack to adopt new projects willy nilly contributes > to the sense of features winning out over deployment, configuration > and usability issues? > It does not help. > I think a lot of project contributors may not really see it that way > because they think of their project and within that project there is > a constant effort to clean things up, address bugs and tech debt, > and try to slowly but surely evolve to some level of maturity. In > their eyes those new projects are something else separate from their > project. > > From the outside, however, it is all OpenStack and maybe it looks > like there's loads of diffuse attention. > > If that's the case, then a question is whether or not the people who > are spending time on those new projects would be spending time on > the older projects instead if the new projects didn't exist. I don't > know, but seems unlikely. > So there's a trade off and I don't think we can just restrict entry because some projects aren't user friendly. I see it as a common issue across all projects. Some are better than others, but what I would like to see is the bar for usability raised within the OpenStack community such that the end user (and deployer/operator) are all taken into consideration. For me the usability also goes with adoption. The easier it is to consume, the easier it would be to adopt something. If you take a look at what is required to configure OpenStack for a basic deployment, it is not easy to consume. If you were to compare the basic getting started/install guide for Kubernetes[0] vs OpenStack[1], you can see what I mean about complexity. I think just the install guide for neutron on a controller node[2] is about the same length as the kubernetes guide. And we think this is ok? We should keep adding additional installation/management complexity for each project? You could argue that OpenStack has more features or more flexible so it's apples to oranges but I don't think it has to be if we worked on better patterns for configuration/deployment/upgrades. It feels like OpenStack is the thing that you should pay professional services to deploy rather than I do it yourself. And I think that's a shame. Thanks, -Alex [0] https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/ [1] https://docs.openstack.org/newton/install-guide-rdo/ [2] https://docs.openstack.org/newton/install-guide-rdo/neutron-controller-install.html > > -- > Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ > freenode: cdent tw: @anticdent > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 05, 2017 at 11:24:49AM -0600, Chris Friesen wrote: > [snip] > > Cinder theoretically supports LVM/iSCSI, but if you actually try to use it > for anything stressful it falls over. > A bit of a tangent, but I would love to hear more about this. We have a lot of folks using LVM successfully, at least as far as I've seen, so if there are issues out there that are viewed as Cinder "ignoring" over adding new features, I would really love to know what they are. And ideally have bug reports to give us something to track and work on. > Some of the database pruning tools don't cover all the tables so the DB gets > bigger over time. > > I'm sure there are historical reasons for all of these, I'm just pointing > out some of the things that were surprising to me. > > Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 10:52 AM, Dean Troyer wrote: > On Fri, May 5, 2017 at 11:36 AM, Alex Schultz wrote: >> configuration and usability issues. But those aren't exciting so >> people don't want to work on them... > > [Not picking on Alex here, but I've seen this repeated again in this > thread like every other one] > > s/people/corporate project managers/ > > Yes, there are individual preferences involved here, but those of us > who are in a position to CHOOSE which projects we invest in are > actually in the minority, and shrinking. Yes I do not want to discount the folks who get paid to do a thing. That being said, I believe there's a push for some of the initiatives (or at least specific implementations) that are outside of the 'I'm getting paid for this' and rather 'I have to do this because I'm paid to, so let me go do it with this $hotness'. We could point to the $language conversations as well where it's asked "is that really necessary? Is this better for the end user?". I know my view is a bit different because I personally like solving issues for the end user and I understand that's not everyone's thing. It's just something that I think the community could benefit from. > > The choices of what contributing companies invest in are largely > determined by their customers, or where they feel their business needs > to go. Infrastructure, SDKs, documentation, none of these are on any > of the public roadmaps that I have seen (I have not seen them all so > speak up where I am wrong here). > I know I like to push for upstream documentation to make sure when we write something we actually write the thing that we said we would and it works as expected. But yea, other items like infrastructure is probably not high on some companies list. > Those who are customers of these sponsor/contributor companies also > need to be bugging them for these things too, and not just "enterprise > ready" or "telco ready" or "GPU ready" or whatever their particular > shiny thing is this year. > > Make long term releases important on corporate bottom lines and it > WILL become a thing. > They already are, but just not in the public. So at that point, they become a reason why people continue to use older versions and still diverge from upstream because they don't need to because $vendor is the person they go to when they need something. Thanks, -Alex > dt > -- > > Dean Troyer > dtro...@gmail.com > > > Tragedy. Commons. Rinse. Repeat. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
+1. ocata's cell v2 stuff added a lot of extra required complexity with no perceivable benefit to end users. If there was a long term stable version, then putting it in the non lts release would have been ok. In absence of lts, I would have recommended the cell v2 stuff have been done in a branch instead and merged all together when it provided something (pike I think) OpenStack Operators already suffer a lot of pain and more pain without benefit is something we should be avoiding at all costs. Thanks, Kevin From: Alex Schultz [aschu...@redhat.com] Sent: Friday, May 05, 2017 9:36 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time On Fri, May 5, 2017 at 6:16 AM, Sean Dague wrote: > On 05/04/2017 11:08 AM, Alex Schultz wrote: >> On Thu, May 4, 2017 at 5:32 AM, Chris Dent wrote: >>> On Wed, 3 May 2017, Drew Fisher wrote: >>> >>>> This email is meant to be the ML discussion of a question I brought up >>>> during the TC meeting on April 25th.; [1] >>> >>> >>> Thanks for starting this Drew, I hope my mentioning it in my tc >>> report email wasn't too much of a nag. >>> >>> I've added [tc] and [all] tags to the subject in case people are >>> filtering. More within. >>> >>>> The TL;DR version is: >>>> >>>> Reading the user survey [2], I see the same issues time and time again. >>>> Pages 18-19 of the survey are especially common points. >>>> Things move too fast, no LTS release, upgrades are terrifying for >>>> anything that isn't N-1 -> N. >>>> These come up time and time again >>>> How is the TC working with the dev teams to address these critical issues? >>> >>> >>> As I recall the "OpenStack-wide Goals"[a] are supposed to help address >>> some of this sort of thing but it of course relies on people first >>> proposing and detailing goals and then there actually being people >>> to act on them. The first part was happening at[b] but it's not >>> clear if that's the current way. >>> >>> Having people is the hard part. Given the current contribution >>> model[c] that pretty much means enterprises ponying up the people do >>> the work. If they don't do that then the work won't get done, and >>> people won't buy the products they are supporting, I guess? Seems a >>> sad state of affairs. >>> >>> There's also an issue where we seem to have decided that it is only >>> appropriate to demand a very small number of goals per cycle >>> (because each project already has too much on their plate, or too big >>> a backlog, relative to resources). It might be that as the >>> _Technical_ Committe is could be legitimate to make a larger demand. >>> (Or it could be completely crazy.) >>> >>>> I asked this because on page 18 is this comment: >>>> >>>> "Most large customers move slowly and thus are running older versions, >>>> which are EOL upstream sometimes before they even deploy them." >>> >>> >>> Can someone with more of the history give more detail on where the >>> expectation arose that upstream ought to be responsible things like >>> long term support? I had always understood that such features were >>> part of the way in which the corporately avaialable products added >>> value? >>> >>>> This is exactly what we're seeing with some of our customers and I >>>> wanted to ask the TC about it. >>> >>> >>> I know you're not speaking as the voice of your employer when making >>> this message, so this is not directed at you, but from what I can >>> tell Oracle's presense upstream (both reviews and commits) in Ocata >>> and thus far in Pike has not been huge. Maybe that's something that >>> needs to change to keep the customers happy? Or at all. >>> >> >> Probably because they are still on Kilo. Not sure how much they could >> be contributing to the current when their customers are demanding that >> something is rock solid which by now looks nothing like the current >> upstream. I think this is part of the problem as the upstream can >> tend to outpace anyone else in terms of features or anything else. I >> think the the bigger question could be what's the benefit of >> continuing to press forward and
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 11:52 AM, Dean Troyer wrote: On Fri, May 5, 2017 at 11:36 AM, Alex Schultz wrote: configuration and usability issues. But those aren't exciting so people don't want to work on them... [Not picking on Alex here, but I've seen this repeated again in this thread like every other one] s/people/corporate project managers/ Yes, there are individual preferences involved here, but those of us who are in a position to CHOOSE which projects we invest in are actually in the minority, and shrinking. It's worth noting that with almost no exceptions I am aware of, those of us who are in a position to choose what we work on actually are the ones focusing on the "not exciting" things. As Dean says, there aren't many of us in that position - so there's only so much we can accomplish in a given unit of time. I bug companies all the time to get them to allocate resources to these things. It is rarely successful. The choices of what contributing companies invest in are largely determined by their customers, or where they feel their business needs to go. Infrastructure, SDKs, documentation, none of these are on any of the public roadmaps that I have seen (I have not seen them all so speak up where I am wrong here). Those who are customers of these sponsor/contributor companies also need to be bugging them for these things too, and not just "enterprise ready" or "telco ready" or "GPU ready" or whatever their particular shiny thing is this year. Make long term releases important on corporate bottom lines and it WILL become a thing. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 10:48 AM, Chris Dent wrote: Would it be accurate to say, then, that from your perpsective the tendency of OpenStack to adopt new projects willy nilly contributes to the sense of features winning out over deployment, configuration and usability issues? Personally I don't care about the new projects...if I'm not using them I can ignore them, and if I am using them then I'll pay attention to them. But within existing established projects there are some odd gaps. Like nova hasn't implemented cold-migration or resize (or live-migration) of an instance with LVM local storage if you're using libvirt. Image properties get validated, but not flavor extra-specs or instance metadata. Cinder theoretically supports LVM/iSCSI, but if you actually try to use it for anything stressful it falls over. Some of the database pruning tools don't cover all the tables so the DB gets bigger over time. I'm sure there are historical reasons for all of these, I'm just pointing out some of the things that were surprising to me. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 11:36 AM, Alex Schultz wrote: > configuration and usability issues. But those aren't exciting so > people don't want to work on them... [Not picking on Alex here, but I've seen this repeated again in this thread like every other one] s/people/corporate project managers/ Yes, there are individual preferences involved here, but those of us who are in a position to CHOOSE which projects we invest in are actually in the minority, and shrinking. The choices of what contributing companies invest in are largely determined by their customers, or where they feel their business needs to go. Infrastructure, SDKs, documentation, none of these are on any of the public roadmaps that I have seen (I have not seen them all so speak up where I am wrong here). Those who are customers of these sponsor/contributor companies also need to be bugging them for these things too, and not just "enterprise ready" or "telco ready" or "GPU ready" or whatever their particular shiny thing is this year. Make long term releases important on corporate bottom lines and it WILL become a thing. dt -- Dean Troyer dtro...@gmail.com Tragedy. Commons. Rinse. Repeat. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, 5 May 2017, Alex Schultz wrote: You have to understand that as I'm mainly dealing with having to actually deploy/configure the software, when I see 'new project X' that does 'cool new things Y, Z' it makes me cringe. Because it's just added complexity for new features that who knows if they are actually going to be consumed by a majority of end users. I see a lot of new features for edge cases while the core functionally (insert the most used project configuration) still have awkward deployment, configuration and usability issues. But those aren't exciting so people don't want to work on them... Would it be accurate to say, then, that from your perpsective the tendency of OpenStack to adopt new projects willy nilly contributes to the sense of features winning out over deployment, configuration and usability issues? I think a lot of project contributors may not really see it that way because they think of their project and within that project there is a constant effort to clean things up, address bugs and tech debt, and try to slowly but surely evolve to some level of maturity. In their eyes those new projects are something else separate from their project. From the outside, however, it is all OpenStack and maybe it looks like there's loads of diffuse attention. If that's the case, then a question is whether or not the people who are spending time on those new projects would be spending time on the older projects instead if the new projects didn't exist. I don't know, but seems unlikely. -- Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 6:16 AM, Sean Dague wrote: > On 05/04/2017 11:08 AM, Alex Schultz wrote: >> On Thu, May 4, 2017 at 5:32 AM, Chris Dent wrote: >>> On Wed, 3 May 2017, Drew Fisher wrote: >>> This email is meant to be the ML discussion of a question I brought up during the TC meeting on April 25th.; [1] >>> >>> >>> Thanks for starting this Drew, I hope my mentioning it in my tc >>> report email wasn't too much of a nag. >>> >>> I've added [tc] and [all] tags to the subject in case people are >>> filtering. More within. >>> The TL;DR version is: Reading the user survey [2], I see the same issues time and time again. Pages 18-19 of the survey are especially common points. Things move too fast, no LTS release, upgrades are terrifying for anything that isn't N-1 -> N. These come up time and time again How is the TC working with the dev teams to address these critical issues? >>> >>> >>> As I recall the "OpenStack-wide Goals"[a] are supposed to help address >>> some of this sort of thing but it of course relies on people first >>> proposing and detailing goals and then there actually being people >>> to act on them. The first part was happening at[b] but it's not >>> clear if that's the current way. >>> >>> Having people is the hard part. Given the current contribution >>> model[c] that pretty much means enterprises ponying up the people do >>> the work. If they don't do that then the work won't get done, and >>> people won't buy the products they are supporting, I guess? Seems a >>> sad state of affairs. >>> >>> There's also an issue where we seem to have decided that it is only >>> appropriate to demand a very small number of goals per cycle >>> (because each project already has too much on their plate, or too big >>> a backlog, relative to resources). It might be that as the >>> _Technical_ Committe is could be legitimate to make a larger demand. >>> (Or it could be completely crazy.) >>> I asked this because on page 18 is this comment: "Most large customers move slowly and thus are running older versions, which are EOL upstream sometimes before they even deploy them." >>> >>> >>> Can someone with more of the history give more detail on where the >>> expectation arose that upstream ought to be responsible things like >>> long term support? I had always understood that such features were >>> part of the way in which the corporately avaialable products added >>> value? >>> This is exactly what we're seeing with some of our customers and I wanted to ask the TC about it. >>> >>> >>> I know you're not speaking as the voice of your employer when making >>> this message, so this is not directed at you, but from what I can >>> tell Oracle's presense upstream (both reviews and commits) in Ocata >>> and thus far in Pike has not been huge. Maybe that's something that >>> needs to change to keep the customers happy? Or at all. >>> >> >> Probably because they are still on Kilo. Not sure how much they could >> be contributing to the current when their customers are demanding that >> something is rock solid which by now looks nothing like the current >> upstream. I think this is part of the problem as the upstream can >> tend to outpace anyone else in terms of features or anything else. I >> think the the bigger question could be what's the benefit of >> continuing to press forward and add yet more features when consumers >> cannot keep up to consume these? Personally I think usability (and >> some stability) sometimes tends to take a backseat to features in the >> upstream which is unfortunate because it makes these problems worse. > > The general statement of "people care more about features than > usability/stability" gets thrown around a lot. And gets lots of head > nodding. But rarely comes with specifics. > > Can we be specific about what feature work is outpacing the consumers > that don't help with usability/stability? > The cell v2 initial implementation was neither usable or stable (for my definition of stable). Yea you could say 'but it's a work and progress' and I would say, why is it required for the end user then? If I wanted to I could probably go back and go through every project and point out when a feature was added yet we still have a pile of outstanding issues. As Chris Friesen pointed out in his reply email, there are things out there are specifics if you go looking. You have to understand that as I'm mainly dealing with having to actually deploy/configure the software, when I see 'new project X' that does 'cool new things Y, Z' it makes me cringe. Because it's just added complexity for new features that who knows if they are actually going to be consumed by a majority of end users. I see a lot of new features for edge cases while the core functionally (insert the most used project configuration) still have awkward deployment, configuration and usability issues. But those aren't exciting so people don't want
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Davanum Srinivas wrote: > I would encourage folks here to help the stable branches we have right > now! Release/Requirements constantly wait on Stable team and Stable > team is way short of hands. > > Please join #openstack-stable, throw your name in wiki etc > (https://wiki.openstack.org/wiki/CrossProjectLiaisons#Stable_Branch) > and get active. > > If we have trouble taking care of what we have now, how can we do more? Shameless plug: For those in Boston next week, you can join the following on-boarding session on Monday afternoon, to see what this work really means. It's not as hard or time-consuming as you'd think: https://www.openstack.org/summit/boston-2017/summit-schedule/events/18694/infraqarelease-mgmtregsstable-project-onboarding -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 9:34 AM, Bogdan Dobrelya wrote: > On 05.05.2017 14:12, Sean Dague wrote: >> On 05/05/2017 07:16 AM, Bogdan Dobrelya wrote: >>> So perhaps there is a (naive?) option #3: Do not support or nurse gates >>> for stable branches upstream. Instead, only create and close them and >>> attach 3rd party gating, if asked by contributors willing to support and >>> nurse their gates. >> >> I think it's important to clarify the amount of infrastructure that goes >> into testing OpenStack. We build a whole cloud, from source, installing >> ~ 200 python packages, many from git trees, configure and boot 100+ VMs >> on it in different ways. And do that with a number of different default >> configs. >> >> Nothing prevents anyone from building a kilo branch in a public github >> and doing their own CI against it. But we've never seen anyone do that, >> because there is a lot of work in maintaining a CI system. A lot of >> expertise needed to debug when things go wrong. Anyone can captain a > > We need no to underscore complexity of stable branches maintaining, > indeed. Complexity and costs are huge, and it sounds even more > reasonable for operators to join efforts of isolated teams that have > been patching Kilo for years, multiple downstream - therefore isolated > as well - places, fighting same problems but different ways, wasting > engineering, management resources and hardware ever and ever again. It > should be obvious that it is much less expensive to cooperate here. I > can't get it what "prevents anyone from building a kilo branch in a > public github and doing their own CI against it", a riddle it is. Bogdan, Folks, I would encourage folks here to help the stable branches we have right now! Release/Requirements constantly wait on Stable team and Stable team is way short of hands. Please join #openstack-stable, throw your name in wiki etc (https://wiki.openstack.org/wiki/CrossProjectLiaisons#Stable_Branch) and get active. If we have trouble taking care of what we have now, how can we do more? Thanks, Dims >> ship when it's a sunny day. Only under failures do we see what is >> required to actually keep the ship afloat. >> >> You could always drop all the hard parts of CI, actually testing the >> trees build a full running cloud. But at that point, it becomes very odd >> to call it a stable branch, as it is far less rigorous in validation >> than master. >> >> At any rate, this basically comes up every year, and I don't think the >> fundamental equation has changed. >> >> -Sean >> > > > -- > Best regards, > Bogdan Dobrelya, > Irc #bogdando > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 06:16 AM, Sean Dague wrote: On 05/04/2017 11:08 AM, Alex Schultz wrote: Probably because they are still on Kilo. Not sure how much they could be contributing to the current when their customers are demanding that something is rock solid which by now looks nothing like the current upstream. I think this is part of the problem as the upstream can tend to outpace anyone else in terms of features or anything else. I think the the bigger question could be what's the benefit of continuing to press forward and add yet more features when consumers cannot keep up to consume these? Personally I think usability (and some stability) sometimes tends to take a backseat to features in the upstream which is unfortunate because it makes these problems worse. The general statement of "people care more about features than usability/stability" gets thrown around a lot. And gets lots of head nodding. But rarely comes with specifics. Can we be specific about what feature work is outpacing the consumers that don't help with usability/stability? On the usability/stability front, in nova you still can't correctly live-migrate if you have dedicated CPUs or hugepages, or a specific NUMA topology. The commits for this have been under review since Kilo, but never quite make it in. At the same time, there are no warnings or errors to the user saying that it's not stable...it just migrates and hopes that it doesn't collide with another instance. On the usability front, the new "openstack" client doesn't support microversions, which limits its usefulness with nova. (I think some folks are starting to look at this one.) Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 05:16 AM, Bogdan Dobrelya wrote: I may be wrong, but I have a strong perception that even when major upgrades run smooth and flawless, operators tend to operate legacy enterprise software for long, long, long. It would be an utopia to expect any changes here, IMO. One reason for this is that the end-users of telco or enterprise companies have high expectations for robustness and reliability. Telecom-level validation takes a *long* time. It could easily be six months to a year before new software actually makes it out to end-users. Because this effort is expensive, they want to amortize it over a longer period. Even when an upgrade is smooth and easy, it's still new code, with new unknown bugs. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05.05.2017 14:12, Sean Dague wrote: > On 05/05/2017 07:16 AM, Bogdan Dobrelya wrote: >> So perhaps there is a (naive?) option #3: Do not support or nurse gates >> for stable branches upstream. Instead, only create and close them and >> attach 3rd party gating, if asked by contributors willing to support and >> nurse their gates. > > I think it's important to clarify the amount of infrastructure that goes > into testing OpenStack. We build a whole cloud, from source, installing > ~ 200 python packages, many from git trees, configure and boot 100+ VMs > on it in different ways. And do that with a number of different default > configs. > > Nothing prevents anyone from building a kilo branch in a public github > and doing their own CI against it. But we've never seen anyone do that, > because there is a lot of work in maintaining a CI system. A lot of > expertise needed to debug when things go wrong. Anyone can captain a We need no to underscore complexity of stable branches maintaining, indeed. Complexity and costs are huge, and it sounds even more reasonable for operators to join efforts of isolated teams that have been patching Kilo for years, multiple downstream - therefore isolated as well - places, fighting same problems but different ways, wasting engineering, management resources and hardware ever and ever again. It should be obvious that it is much less expensive to cooperate here. I can't get it what "prevents anyone from building a kilo branch in a public github and doing their own CI against it", a riddle it is. > ship when it's a sunny day. Only under failures do we see what is > required to actually keep the ship afloat. > > You could always drop all the hard parts of CI, actually testing the > trees build a full running cloud. But at that point, it becomes very odd > to call it a stable branch, as it is far less rigorous in validation > than master. > > At any rate, this basically comes up every year, and I don't think the > fundamental equation has changed. > > -Sean > -- Best regards, Bogdan Dobrelya, Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Excerpts from Zane Bitter's message of 2017-05-04 20:09:35 -0400: > On 04/05/17 10:14, Thierry Carrez wrote: > > Chris Dent wrote: > >> On Wed, 3 May 2017, Drew Fisher wrote: > >>> "Most large customers move slowly and thus are running older versions, > >>> which are EOL upstream sometimes before they even deploy them." > >> > >> Can someone with more of the history give more detail on where the > >> expectation arose that upstream ought to be responsible things like > >> long term support? I had always understood that such features were > >> part of the way in which the corporately avaialable products added > >> value? > > > > We started with no stable branches, we were just producing releases and > > ensuring that updates vaguely worked from N-1 to N. There were a lot of > > distributions, and they all maintained their own stable branches, > > handling backport of critical fixes. That is a pretty classic upstream / > > downstream model. > > > > Some of us (including me) spotted the obvious duplication of effort > > there, and encouraged distributions to share that stable branch > > maintenance work rather than duplicate it. Here the stable branches were > > born, mostly through a collaboration between Red Hat developers and > > Canonical developers. All was well. Nobody was saying LTS back then > > because OpenStack was barely usable so nobody wanted to stay on any > > given version for too long. > > Heh, if you go back _that_ far then upgrades between versions basically > weren't feasible, so everybody stayed on a given version for too long. > It's true that nobody *wanted* to though :D > > > Maintaining stable branches has a cost. Keeping the infrastructure that > > ensures that stable branches are actually working is a complex endeavor > > that requires people to constantly pay attention. As time passed, we saw > > the involvement of distro packagers become more limited. We therefore > > limited the number of stable branches (and the length of time we > > maintained them) to match the staffing of that team. > > I wonder if this is one that needs revisiting. There was certainly a > time when closing a branch came with a strong sense of relief that you > could stop nursing the gate. I personally haven't felt that way in a > couple of years, thanks to a lot of *very* hard work done by the folks > looking after the gate to systematically solve a lot of those recurring > issues (e.g. by introducing upper constraints). We're still assuming > that stable branches are expensive, but what if they aren't any more? > > > Fast-forward to > > today: the stable team is mostly one person, who is now out of his job > > and seeking employment. > > > > In parallel, OpenStack became more stable, so the demand for longer-term > > maintenance is stronger. People still expect "upstream" to provide it, > > not realizing upstream is made of people employed by various > > organizations, and that apparently their interest in funding work in > > that area is pretty dead. > > > > I agree that our current stable branch model is inappropriate: > > maintaining stable branches for one year only is a bit useless. But I > > only see two outcomes: > > > > 1/ The OpenStack community still thinks there is a lot of value in doing > > this work upstream, in which case organizations should invest resources > > in making that happen (starting with giving the Stable branch > > maintenance PTL a job), and then, yes, we should definitely consider > > things like LTS or longer periods of support for stable branches, to > > match the evolving usage of OpenStack. > > Speaking as a downstream maintainer, it sucks that backports I'm still > doing to, say, Liberty don't benefit anybody but Red Hat customers, > because there's nowhere upstream that I can share them. I want everyone > in the community to benefit. Even if I could only upload patches to > Gerrit and not merge them, that would at least be something. > > (In a related bugbear, why must we delete the branch at EOL? This is > pure evil for consumers of the code. It breaks existing git checkouts > and thousands of web links in bug reports, review comments, IRC logs...) Among other things, closing the branch lets us avoid all of the discussions about why no one is reviewing patches there and why folks shouldn't bother submitting them. I would support having the stable maintenance team review the state of the gate and revise the policy, if it's warranted. But we've had that conversation at least once a year for the last 5 years, and we only came to a different conclusion one time that I remember. Even if branches are cheaper to maintain now, they aren't free. We need people to be around to do the work. > > 2/ The OpenStack community thinks this is better handled downstream, and > > we should just get rid of them completely. This is a valid approach, and > > a lot of other open source communities just do that. > > Maybe we need a 5th 'Open', because to me the idea tha
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 2017-05-04 20:09:35 -0400 (-0400), Zane Bitter wrote: > On 04/05/17 10:14, Thierry Carrez wrote: [...] > >Maintaining stable branches has a cost. Keeping the infrastructure that > >ensures that stable branches are actually working is a complex endeavor > >that requires people to constantly pay attention. As time passed, we saw > >the involvement of distro packagers become more limited. We therefore > >limited the number of stable branches (and the length of time we > >maintained them) to match the staffing of that team. > > I wonder if this is one that needs revisiting. There was certainly a time > when closing a branch came with a strong sense of relief that you could stop > nursing the gate. I personally haven't felt that way in a couple of years, > thanks to a lot of *very* hard work done by the folks looking after the gate > to systematically solve a lot of those recurring issues (e.g. by introducing > upper constraints). We're still assuming that stable branches are expensive, > but what if they aren't any more? [...] > Speaking as a downstream maintainer, it sucks that backports I'm still doing > to, say, Liberty don't benefit anybody but Red Hat customers, because > there's nowhere upstream that I can share them. I want everyone in the > community to benefit. Even if I could only upload patches to Gerrit and not > merge them, that would at least be something. > > (In a related bugbear, why must we delete the branch at EOL? This is > pure evil for consumers of the code. It breaks existing git checkouts and > thousands of web links in bug reports, review comments, IRC logs...) [...] The points above are all interrelated. We close upstream development of a branch at some point (by tagging and deleting it) to ensure contributors _don't_ post new changes to Gerrit targeting those branches _because_ we can't indefinitely maintain the contemporary infrastructure required to test them and confirm they work and don't introduce new regressions. OpenStack upstream development has taken the approach that everything we officially maintain should be continuously buildable and testable. Revisiting the other points means revisiting that decision as well. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 08:50 AM, Dean Troyer wrote: > On Fri, May 5, 2017 at 7:16 AM, Sean Dague wrote: >> The general statement of "people care more about features than >> usability/stability" gets thrown around a lot. And gets lots of head >> nodding. But rarely comes with specifics. > > I think a lot of this general sentiment comes from watching what gets > the investment from contributors, specifically from contributing > companies. HP may have been an aberration for a time given the > quantity of infra folk they once employed, but take a look at the > distribution between feature work and non-feature work that our major > contributing companies sponsor. Ok, I still don't see that as specific, unless you are stating that everything that's not Infra is a Feature? (which I honestly don't think you believe, but I'm going to keep pressing on all respondents to be very specific) -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Fri, May 5, 2017 at 7:16 AM, Sean Dague wrote: > The general statement of "people care more about features than > usability/stability" gets thrown around a lot. And gets lots of head > nodding. But rarely comes with specifics. I think a lot of this general sentiment comes from watching what gets the investment from contributors, specifically from contributing companies. HP may have been an aberration for a time given the quantity of infra folk they once employed, but take a look at the distribution between feature work and non-feature work that our major contributing companies sponsor. Maybe it is time to set up direct sponsorship for some of this work that keeps coming up as important but "nobody wants to pay for it". Like an endowed chair or just a GoFundMe that lets someone continue to be the Stable Branch Overlord. It is the sort of work that no single company wants to be on the hook for, nor is it a good idea from the community perspective to be single sourced in that regard (again, the infra situation last fall) so lets make it easy for downstream consumers to share the funding of the upstream work they all want to see done but do not want or can not afford the responsibility of taking on themselves. You might say we already do this, the Foundation employs people to work directly on these things, and yes they do to great effect. But this topic keeps coming up because it is a specific request for specific OpenStack consumers. Lets help them help themselves. dt -- Dean Troyer dtro...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/04/2017 11:08 AM, Alex Schultz wrote: > On Thu, May 4, 2017 at 5:32 AM, Chris Dent wrote: >> On Wed, 3 May 2017, Drew Fisher wrote: >> >>> This email is meant to be the ML discussion of a question I brought up >>> during the TC meeting on April 25th.; [1] >> >> >> Thanks for starting this Drew, I hope my mentioning it in my tc >> report email wasn't too much of a nag. >> >> I've added [tc] and [all] tags to the subject in case people are >> filtering. More within. >> >>> The TL;DR version is: >>> >>> Reading the user survey [2], I see the same issues time and time again. >>> Pages 18-19 of the survey are especially common points. >>> Things move too fast, no LTS release, upgrades are terrifying for >>> anything that isn't N-1 -> N. >>> These come up time and time again >>> How is the TC working with the dev teams to address these critical issues? >> >> >> As I recall the "OpenStack-wide Goals"[a] are supposed to help address >> some of this sort of thing but it of course relies on people first >> proposing and detailing goals and then there actually being people >> to act on them. The first part was happening at[b] but it's not >> clear if that's the current way. >> >> Having people is the hard part. Given the current contribution >> model[c] that pretty much means enterprises ponying up the people do >> the work. If they don't do that then the work won't get done, and >> people won't buy the products they are supporting, I guess? Seems a >> sad state of affairs. >> >> There's also an issue where we seem to have decided that it is only >> appropriate to demand a very small number of goals per cycle >> (because each project already has too much on their plate, or too big >> a backlog, relative to resources). It might be that as the >> _Technical_ Committe is could be legitimate to make a larger demand. >> (Or it could be completely crazy.) >> >>> I asked this because on page 18 is this comment: >>> >>> "Most large customers move slowly and thus are running older versions, >>> which are EOL upstream sometimes before they even deploy them." >> >> >> Can someone with more of the history give more detail on where the >> expectation arose that upstream ought to be responsible things like >> long term support? I had always understood that such features were >> part of the way in which the corporately avaialable products added >> value? >> >>> This is exactly what we're seeing with some of our customers and I >>> wanted to ask the TC about it. >> >> >> I know you're not speaking as the voice of your employer when making >> this message, so this is not directed at you, but from what I can >> tell Oracle's presense upstream (both reviews and commits) in Ocata >> and thus far in Pike has not been huge. Maybe that's something that >> needs to change to keep the customers happy? Or at all. >> > > Probably because they are still on Kilo. Not sure how much they could > be contributing to the current when their customers are demanding that > something is rock solid which by now looks nothing like the current > upstream. I think this is part of the problem as the upstream can > tend to outpace anyone else in terms of features or anything else. I > think the the bigger question could be what's the benefit of > continuing to press forward and add yet more features when consumers > cannot keep up to consume these? Personally I think usability (and > some stability) sometimes tends to take a backseat to features in the > upstream which is unfortunate because it makes these problems worse. The general statement of "people care more about features than usability/stability" gets thrown around a lot. And gets lots of head nodding. But rarely comes with specifics. Can we be specific about what feature work is outpacing the consumers that don't help with usability/stability? -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05/05/2017 07:16 AM, Bogdan Dobrelya wrote: > On 05.05.2017 2:09, Zane Bitter wrote: >> On 04/05/17 10:14, Thierry Carrez wrote: >>> We started with no stable branches, we were just producing releases and >>> ensuring that updates vaguely worked from N-1 to N. There were a lot of >>> distributions, and they all maintained their own stable branches, >>> handling backport of critical fixes. That is a pretty classic upstream / >>> downstream model. >>> >>> Some of us (including me) spotted the obvious duplication of effort >>> there, and encouraged distributions to share that stable branch >>> maintenance work rather than duplicate it. Here the stable branches were >>> born, mostly through a collaboration between Red Hat developers and >>> Canonical developers. All was well. Nobody was saying LTS back then >>> because OpenStack was barely usable so nobody wanted to stay on any >>> given version for too long. >> >> Heh, if you go back _that_ far then upgrades between versions basically >> weren't feasible, so everybody stayed on a given version for too long. >> It's true that nobody *wanted* to though :D > > I may be wrong, but I have a strong perception that even when major > upgrades run smooth and flawless, operators tend to operate legacy > enterprise software for long, long, long. It would be an utopia to > expect any changes here, IMO. Unless the major shift comes to > enterprises for the very software delivery paradigm (infrastructure as a > code, blue/green deployments done by CD pipeline promoting stable > changes from trunk commits, unicorns all around), which is an utopia > even more though. > >>> Maintaining stable branches has a cost. Keeping the infrastructure that >>> ensures that stable branches are actually working is a complex endeavor >>> that requires people to constantly pay attention. As time passed, we saw >>> the involvement of distro packagers become more limited. We therefore >>> limited the number of stable branches (and the length of time we >>> maintained them) to match the staffing of that team. >> >> I wonder if this is one that needs revisiting. There was certainly a >> time when closing a branch came with a strong sense of relief that you >> could stop nursing the gate. I personally haven't felt that way in a > > This. What we really need, IMO, that sense of relief, but inversed, > which is *operators* of enterprises world to feel it and start nursing > their 3rd party CI gates *upstream* once a stable branch created! And > hopefully never closed, for the time they do care of it. It sounds fair > to open source software. As Jay noted, one shall get that one had put, > which is a direct function of how much do one care and gives away. > > So perhaps there is a (naive?) option #3: Do not support or nurse gates > for stable branches upstream. Instead, only create and close them and > attach 3rd party gating, if asked by contributors willing to support and > nurse their gates. I think it's important to clarify the amount of infrastructure that goes into testing OpenStack. We build a whole cloud, from source, installing ~ 200 python packages, many from git trees, configure and boot 100+ VMs on it in different ways. And do that with a number of different default configs. Nothing prevents anyone from building a kilo branch in a public github and doing their own CI against it. But we've never seen anyone do that, because there is a lot of work in maintaining a CI system. A lot of expertise needed to debug when things go wrong. Anyone can captain a ship when it's a sunny day. Only under failures do we see what is required to actually keep the ship afloat. You could always drop all the hard parts of CI, actually testing the trees build a full running cloud. But at that point, it becomes very odd to call it a stable branch, as it is far less rigorous in validation than master. At any rate, this basically comes up every year, and I don't think the fundamental equation has changed. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 05.05.2017 2:09, Zane Bitter wrote: > On 04/05/17 10:14, Thierry Carrez wrote: >> We started with no stable branches, we were just producing releases and >> ensuring that updates vaguely worked from N-1 to N. There were a lot of >> distributions, and they all maintained their own stable branches, >> handling backport of critical fixes. That is a pretty classic upstream / >> downstream model. >> >> Some of us (including me) spotted the obvious duplication of effort >> there, and encouraged distributions to share that stable branch >> maintenance work rather than duplicate it. Here the stable branches were >> born, mostly through a collaboration between Red Hat developers and >> Canonical developers. All was well. Nobody was saying LTS back then >> because OpenStack was barely usable so nobody wanted to stay on any >> given version for too long. > > Heh, if you go back _that_ far then upgrades between versions basically > weren't feasible, so everybody stayed on a given version for too long. > It's true that nobody *wanted* to though :D I may be wrong, but I have a strong perception that even when major upgrades run smooth and flawless, operators tend to operate legacy enterprise software for long, long, long. It would be an utopia to expect any changes here, IMO. Unless the major shift comes to enterprises for the very software delivery paradigm (infrastructure as a code, blue/green deployments done by CD pipeline promoting stable changes from trunk commits, unicorns all around), which is an utopia even more though. >> Maintaining stable branches has a cost. Keeping the infrastructure that >> ensures that stable branches are actually working is a complex endeavor >> that requires people to constantly pay attention. As time passed, we saw >> the involvement of distro packagers become more limited. We therefore >> limited the number of stable branches (and the length of time we >> maintained them) to match the staffing of that team. > > I wonder if this is one that needs revisiting. There was certainly a > time when closing a branch came with a strong sense of relief that you > could stop nursing the gate. I personally haven't felt that way in a This. What we really need, IMO, that sense of relief, but inversed, which is *operators* of enterprises world to feel it and start nursing their 3rd party CI gates *upstream* once a stable branch created! And hopefully never closed, for the time they do care of it. It sounds fair to open source software. As Jay noted, one shall get that one had put, which is a direct function of how much do one care and gives away. So perhaps there is a (naive?) option #3: Do not support or nurse gates for stable branches upstream. Instead, only create and close them and attach 3rd party gating, if asked by contributors willing to support and nurse their gates. > couple of years, thanks to a lot of *very* hard work done by the folks > looking after the gate to systematically solve a lot of those recurring > issues (e.g. by introducing upper constraints). We're still assuming > that stable branches are expensive, but what if they aren't any more? > >> Fast-forward to >> today: the stable team is mostly one person, who is now out of his job >> and seeking employment. >> >> In parallel, OpenStack became more stable, so the demand for longer-term >> maintenance is stronger. People still expect "upstream" to provide it, >> not realizing upstream is made of people employed by various >> organizations, and that apparently their interest in funding work in >> that area is pretty dead. >> >> I agree that our current stable branch model is inappropriate: >> maintaining stable branches for one year only is a bit useless. But I >> only see two outcomes: >> >> 1/ The OpenStack community still thinks there is a lot of value in doing >> this work upstream, in which case organizations should invest resources >> in making that happen (starting with giving the Stable branch >> maintenance PTL a job), and then, yes, we should definitely consider >> things like LTS or longer periods of support for stable branches, to >> match the evolving usage of OpenStack. > > Speaking as a downstream maintainer, it sucks that backports I'm still > doing to, say, Liberty don't benefit anybody but Red Hat customers, > because there's nowhere upstream that I can share them. I want everyone > in the community to benefit. Even if I could only upload patches to > Gerrit and not merge them, that would at least be something. > > (In a related bugbear, why must we delete the branch at EOL? This is > pure evil for consumers of the code. It breaks existing git checkouts > and thousands of web links in bug reports, review comments, IRC logs...) > >> 2/ The OpenStack community thinks this is better handled downstream, and >> we should just get rid of them completely. This is a valid approach, and >> a lot of other open source communities just do that. > > Maybe we need a 5th 'Open', because to me the idea that t
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 04/05/17 10:14, Thierry Carrez wrote: Chris Dent wrote: On Wed, 3 May 2017, Drew Fisher wrote: "Most large customers move slowly and thus are running older versions, which are EOL upstream sometimes before they even deploy them." Can someone with more of the history give more detail on where the expectation arose that upstream ought to be responsible things like long term support? I had always understood that such features were part of the way in which the corporately avaialable products added value? We started with no stable branches, we were just producing releases and ensuring that updates vaguely worked from N-1 to N. There were a lot of distributions, and they all maintained their own stable branches, handling backport of critical fixes. That is a pretty classic upstream / downstream model. Some of us (including me) spotted the obvious duplication of effort there, and encouraged distributions to share that stable branch maintenance work rather than duplicate it. Here the stable branches were born, mostly through a collaboration between Red Hat developers and Canonical developers. All was well. Nobody was saying LTS back then because OpenStack was barely usable so nobody wanted to stay on any given version for too long. Heh, if you go back _that_ far then upgrades between versions basically weren't feasible, so everybody stayed on a given version for too long. It's true that nobody *wanted* to though :D Maintaining stable branches has a cost. Keeping the infrastructure that ensures that stable branches are actually working is a complex endeavor that requires people to constantly pay attention. As time passed, we saw the involvement of distro packagers become more limited. We therefore limited the number of stable branches (and the length of time we maintained them) to match the staffing of that team. I wonder if this is one that needs revisiting. There was certainly a time when closing a branch came with a strong sense of relief that you could stop nursing the gate. I personally haven't felt that way in a couple of years, thanks to a lot of *very* hard work done by the folks looking after the gate to systematically solve a lot of those recurring issues (e.g. by introducing upper constraints). We're still assuming that stable branches are expensive, but what if they aren't any more? Fast-forward to today: the stable team is mostly one person, who is now out of his job and seeking employment. In parallel, OpenStack became more stable, so the demand for longer-term maintenance is stronger. People still expect "upstream" to provide it, not realizing upstream is made of people employed by various organizations, and that apparently their interest in funding work in that area is pretty dead. I agree that our current stable branch model is inappropriate: maintaining stable branches for one year only is a bit useless. But I only see two outcomes: 1/ The OpenStack community still thinks there is a lot of value in doing this work upstream, in which case organizations should invest resources in making that happen (starting with giving the Stable branch maintenance PTL a job), and then, yes, we should definitely consider things like LTS or longer periods of support for stable branches, to match the evolving usage of OpenStack. Speaking as a downstream maintainer, it sucks that backports I'm still doing to, say, Liberty don't benefit anybody but Red Hat customers, because there's nowhere upstream that I can share them. I want everyone in the community to benefit. Even if I could only upload patches to Gerrit and not merge them, that would at least be something. (In a related bugbear, why must we delete the branch at EOL? This is pure evil for consumers of the code. It breaks existing git checkouts and thousands of web links in bug reports, review comments, IRC logs...) 2/ The OpenStack community thinks this is better handled downstream, and we should just get rid of them completely. This is a valid approach, and a lot of other open source communities just do that. Maybe we need a 5th 'Open', because to me the idea that the software isn't so much 'released' as 'abandoned' is problematic in many of the same ways that Open Core and code dumps are. cheers, Zane. The current reality in terms of invested resources points to (2). I personally would prefer (1), because that lets us address security issues more efficiently and avoids duplicating effort downstream. But unfortunately I don't control where development resources are posted. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Flavio Percoco wrote: > On 04/05/17 11:18 -0400, Jonathan Proulx wrote: >> On Thu, May 04, 2017 at 04:14:07PM +0200, Thierry Carrez wrote: >> :I agree that our current stable branch model is inappropriate: >> :maintaining stable branches for one year only is a bit useless. But I >> :only see two outcomes: >> : >> :1/ The OpenStack community still thinks there is a lot of value in doing >> :this work upstream, in which case organizations should invest resources >> :in making that happen (starting with giving the Stable branch >> :maintenance PTL a job), and then, yes, we should definitely consider >> :things like LTS or longer periods of support for stable branches, to >> :match the evolving usage of OpenStack. >> : >> :2/ The OpenStack community thinks this is better handled downstream, and >> :we should just get rid of them completely. This is a valid approach, and >> :a lot of other open source communities just do that. >> : >> :The current reality in terms of invested resources points to (2). I >> :personally would prefer (1), because that lets us address security >> :issues more efficiently and avoids duplicating effort downstream. But >> :unfortunately I don't control where development resources are posted. > > Have there been issues with downstream distros not addressing security > fixes properly? No, not at all -- but usually they package upstream vulnerability fixes, which are produced on stable branches. In mode #2 we would only patch master, forcing downstream to do backports for more branches. That is what I meant by "more efficiently". Sorry for being unclear. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On 04/05/17 11:18 -0400, Jonathan Proulx wrote: On Thu, May 04, 2017 at 04:14:07PM +0200, Thierry Carrez wrote: :I agree that our current stable branch model is inappropriate: :maintaining stable branches for one year only is a bit useless. But I :only see two outcomes: : :1/ The OpenStack community still thinks there is a lot of value in doing :this work upstream, in which case organizations should invest resources :in making that happen (starting with giving the Stable branch :maintenance PTL a job), and then, yes, we should definitely consider :things like LTS or longer periods of support for stable branches, to :match the evolving usage of OpenStack. : :2/ The OpenStack community thinks this is better handled downstream, and :we should just get rid of them completely. This is a valid approach, and :a lot of other open source communities just do that. : :The current reality in terms of invested resources points to (2). I :personally would prefer (1), because that lets us address security :issues more efficiently and avoids duplicating effort downstream. But :unfortunately I don't control where development resources are posted. Have there been issues with downstream distros not addressing security fixes properly? Yes it seems that way to me as well. just killing the stable branch model without some plan either internally or externally to provide a better stability story seems like it would send the wrong signal. So I'd much prefer the distro people to either back option 1) with significant resources so it can really work or make public commitments to handle option 2) in a reasonable way. I think downstream distros are already doing #2, unless I'm missing something. How public/vocal they are about it might be a different discussion. I'd prefer #1 too because I'd rather have everything upstream. However, with the current flux of people, the current roadmaps and the current status of the community, it's unrealistic for us to expect #1 to happen. So, I'd rather dedicate time documenting/communicating #2 properly. Now, one big problem with LTS releases of OpenStack (regardless they happen upstream or downstream) is the upgrade path, which is one of the problems Drew raised. -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Thu, May 04, 2017 at 04:14:07PM +0200, Thierry Carrez wrote: :Chris Dent wrote: :> On Wed, 3 May 2017, Drew Fisher wrote: :>> "Most large customers move slowly and thus are running older versions, :>> which are EOL upstream sometimes before they even deploy them." :> :> Can someone with more of the history give more detail on where the :> expectation arose that upstream ought to be responsible things like :> long term support? I had always understood that such features were :> part of the way in which the corporately avaialable products added :> value? :In parallel, OpenStack became more stable, so the demand for longer-term :maintenance is stronger. People still expect "upstream" to provide it, :not realizing upstream is made of people employed by various :organizations, and that apparently their interest in funding work in :that area is pretty dead. Wearing my Operator hat I don't really care if "LTS" comes from upstream or downstream. I think the upstream expectation has developed becuase there has been some upstream efforts and as far as I can see no recent downstream efforts in support of stable releases, though obviously I mostly pay attention to "my" distro so may be missing things in this space. Having watched this for some time I agree with everything Thierry has said. The increasing demand for "LTS" like releases is definitely a tribute to the overall maturity of core services. I used to be desperate for the next release and back porting patches into custom packages just to keep things working. Now if I belived Ubuntu (which my world OpenStack and otherwise happens to be built on) would provide a direct upgrade path from their 16.04 released OpenStack to what ever lands in their next LTS I'd probably sit rather happily on that. Which is a hugely positive shift. :I agree that our current stable branch model is inappropriate: :maintaining stable branches for one year only is a bit useless. But I :only see two outcomes: : :1/ The OpenStack community still thinks there is a lot of value in doing :this work upstream, in which case organizations should invest resources :in making that happen (starting with giving the Stable branch :maintenance PTL a job), and then, yes, we should definitely consider :things like LTS or longer periods of support for stable branches, to :match the evolving usage of OpenStack. : :2/ The OpenStack community thinks this is better handled downstream, and :we should just get rid of them completely. This is a valid approach, and :a lot of other open source communities just do that. : :The current reality in terms of invested resources points to (2). I :personally would prefer (1), because that lets us address security :issues more efficiently and avoids duplicating effort downstream. But :unfortunately I don't control where development resources are posted. Yes it seems that way to me as well. just killing the stable branch model without some plan either internally or externally to provide a better stability story seems like it would send the wrong signal. So I'd much prefer the distro people to either back option 1) with significant resources so it can really work or make public commitments to handle option 2) in a reasonable way. -Jon __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Thu, May 4, 2017 at 5:32 AM, Chris Dent wrote: > On Wed, 3 May 2017, Drew Fisher wrote: > >> This email is meant to be the ML discussion of a question I brought up >> during the TC meeting on April 25th.; [1] > > > Thanks for starting this Drew, I hope my mentioning it in my tc > report email wasn't too much of a nag. > > I've added [tc] and [all] tags to the subject in case people are > filtering. More within. > >> The TL;DR version is: >> >> Reading the user survey [2], I see the same issues time and time again. >> Pages 18-19 of the survey are especially common points. >> Things move too fast, no LTS release, upgrades are terrifying for >> anything that isn't N-1 -> N. >> These come up time and time again >> How is the TC working with the dev teams to address these critical issues? > > > As I recall the "OpenStack-wide Goals"[a] are supposed to help address > some of this sort of thing but it of course relies on people first > proposing and detailing goals and then there actually being people > to act on them. The first part was happening at[b] but it's not > clear if that's the current way. > > Having people is the hard part. Given the current contribution > model[c] that pretty much means enterprises ponying up the people do > the work. If they don't do that then the work won't get done, and > people won't buy the products they are supporting, I guess? Seems a > sad state of affairs. > > There's also an issue where we seem to have decided that it is only > appropriate to demand a very small number of goals per cycle > (because each project already has too much on their plate, or too big > a backlog, relative to resources). It might be that as the > _Technical_ Committe is could be legitimate to make a larger demand. > (Or it could be completely crazy.) > >> I asked this because on page 18 is this comment: >> >> "Most large customers move slowly and thus are running older versions, >> which are EOL upstream sometimes before they even deploy them." > > > Can someone with more of the history give more detail on where the > expectation arose that upstream ought to be responsible things like > long term support? I had always understood that such features were > part of the way in which the corporately avaialable products added > value? > >> This is exactly what we're seeing with some of our customers and I >> wanted to ask the TC about it. > > > I know you're not speaking as the voice of your employer when making > this message, so this is not directed at you, but from what I can > tell Oracle's presense upstream (both reviews and commits) in Ocata > and thus far in Pike has not been huge. Maybe that's something that > needs to change to keep the customers happy? Or at all. > Probably because they are still on Kilo. Not sure how much they could be contributing to the current when their customers are demanding that something is rock solid which by now looks nothing like the current upstream. I think this is part of the problem as the upstream can tend to outpace anyone else in terms of features or anything else. I think the the bigger question could be what's the benefit of continuing to press forward and add yet more features when consumers cannot keep up to consume these? Personally I think usability (and some stability) sometimes tends to take a backseat to features in the upstream which is unfortunate because it makes these problems worse. Thanks, -Alex > [a]: https://governance.openstack.org/tc/goals/index.html > [b]: https://etherpad.openstack.org/p/community-goals > [c]: There's talk that the current model will change from devs hired > to do OpenStack development being the main engine of contribution to > users of OpenStack, who happen to be devs, being the main engine. Do > we know the slope on that trend? > > >> Thanks, >> >> -Drew >> >> [1] >> >> http://eavesdrop.openstack.org/meetings/tc/2017/tc.2017-04-25-20.00.log.html#l-177 >> [2] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf > > > -- > Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ > freenode: cdent tw: @anticdent > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
Chris Dent wrote: > On Wed, 3 May 2017, Drew Fisher wrote: >> "Most large customers move slowly and thus are running older versions, >> which are EOL upstream sometimes before they even deploy them." > > Can someone with more of the history give more detail on where the > expectation arose that upstream ought to be responsible things like > long term support? I had always understood that such features were > part of the way in which the corporately avaialable products added > value? We started with no stable branches, we were just producing releases and ensuring that updates vaguely worked from N-1 to N. There were a lot of distributions, and they all maintained their own stable branches, handling backport of critical fixes. That is a pretty classic upstream / downstream model. Some of us (including me) spotted the obvious duplication of effort there, and encouraged distributions to share that stable branch maintenance work rather than duplicate it. Here the stable branches were born, mostly through a collaboration between Red Hat developers and Canonical developers. All was well. Nobody was saying LTS back then because OpenStack was barely usable so nobody wanted to stay on any given version for too long. Maintaining stable branches has a cost. Keeping the infrastructure that ensures that stable branches are actually working is a complex endeavor that requires people to constantly pay attention. As time passed, we saw the involvement of distro packagers become more limited. We therefore limited the number of stable branches (and the length of time we maintained them) to match the staffing of that team. Fast-forward to today: the stable team is mostly one person, who is now out of his job and seeking employment. In parallel, OpenStack became more stable, so the demand for longer-term maintenance is stronger. People still expect "upstream" to provide it, not realizing upstream is made of people employed by various organizations, and that apparently their interest in funding work in that area is pretty dead. I agree that our current stable branch model is inappropriate: maintaining stable branches for one year only is a bit useless. But I only see two outcomes: 1/ The OpenStack community still thinks there is a lot of value in doing this work upstream, in which case organizations should invest resources in making that happen (starting with giving the Stable branch maintenance PTL a job), and then, yes, we should definitely consider things like LTS or longer periods of support for stable branches, to match the evolving usage of OpenStack. 2/ The OpenStack community thinks this is better handled downstream, and we should just get rid of them completely. This is a valid approach, and a lot of other open source communities just do that. The current reality in terms of invested resources points to (2). I personally would prefer (1), because that lets us address security issues more efficiently and avoids duplicating effort downstream. But unfortunately I don't control where development resources are posted. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time
On Wed, 3 May 2017, Drew Fisher wrote: This email is meant to be the ML discussion of a question I brought up during the TC meeting on April 25th.; [1] Thanks for starting this Drew, I hope my mentioning it in my tc report email wasn't too much of a nag. I've added [tc] and [all] tags to the subject in case people are filtering. More within. The TL;DR version is: Reading the user survey [2], I see the same issues time and time again. Pages 18-19 of the survey are especially common points. Things move too fast, no LTS release, upgrades are terrifying for anything that isn't N-1 -> N. These come up time and time again How is the TC working with the dev teams to address these critical issues? As I recall the "OpenStack-wide Goals"[a] are supposed to help address some of this sort of thing but it of course relies on people first proposing and detailing goals and then there actually being people to act on them. The first part was happening at[b] but it's not clear if that's the current way. Having people is the hard part. Given the current contribution model[c] that pretty much means enterprises ponying up the people do the work. If they don't do that then the work won't get done, and people won't buy the products they are supporting, I guess? Seems a sad state of affairs. There's also an issue where we seem to have decided that it is only appropriate to demand a very small number of goals per cycle (because each project already has too much on their plate, or too big a backlog, relative to resources). It might be that as the _Technical_ Committe is could be legitimate to make a larger demand. (Or it could be completely crazy.) I asked this because on page 18 is this comment: "Most large customers move slowly and thus are running older versions, which are EOL upstream sometimes before they even deploy them." Can someone with more of the history give more detail on where the expectation arose that upstream ought to be responsible things like long term support? I had always understood that such features were part of the way in which the corporately avaialable products added value? This is exactly what we're seeing with some of our customers and I wanted to ask the TC about it. I know you're not speaking as the voice of your employer when making this message, so this is not directed at you, but from what I can tell Oracle's presense upstream (both reviews and commits) in Ocata and thus far in Pike has not been huge. Maybe that's something that needs to change to keep the customers happy? Or at all. [a]: https://governance.openstack.org/tc/goals/index.html [b]: https://etherpad.openstack.org/p/community-goals [c]: There's talk that the current model will change from devs hired to do OpenStack development being the main engine of contribution to users of OpenStack, who happen to be devs, being the main engine. Do we know the slope on that trend? Thanks, -Drew [1] http://eavesdrop.openstack.org/meetings/tc/2017/tc.2017-04-25-20.00.log.html#l-177 [2] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf -- Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
