It would be nice to record this discussion/summary on an etherpad or somewhere, because it's not just the problem for Designate, IMHO, many other big tent projects are facing the same issue. Thanks.
On 22/02/17 09:25, Hayes, Graham wrote: > For those that are interested, we will have an hour discussion about > how designate can move forward tomorrow (wed 22nd) at the PTG. > > We will be in Savannah 3 @ 11:00 for an hour. > > Thanks, > > Graham > > On 09/02/2017 14:22, Hayes, Graham wrote: >> The HTML version of this is here: >> http://graham.hayes.ie/posts/openstack-designate-where-we-are/ >> >> I have been asked a few times recently "What is the state of the Designate >> project?", "How is Designate getting on?", and by people who know what is >> happening "What are you going to do about Designate?". >> >> Needless to say, all of this is depressing to me, and the people that I have >> worked with for the last number of years to make Designate a truly useful, >> feature rich project. >> >> *TL;DR;* for this - Designate is not in a sustainable place. >> >> To start out - Designate has always been a small project. DNS does not have >> massive *cool* appeal - its not shiny, pretty, or something you see on the >> front page of HackerNews (unless it breaks - then oh boy do people >> become DNS >> experts). >> >> A line a previous PTL for the project used to use, and I have happily >> robbed is >> "DNS is like plumbing, no one cares about it until it breaks, and then >> you are >> standing knee deep in $expletive". (As an aside, that was the reason we >> chose >> the crocodile as our mascot - its basically a dinosaur, old as dirt, and >> when >> it bites it causes some serious complications). >> >> Unfortunately that comes over into the development of DNS products >> sometimes. >> DNSaaS is a check box on a tender response, an assumption. >> >> We were lucky in the beginning - we had 2 large(ish) public clouds that >> needed >> DNS services, and nothing currently existed in the eco-system, so we got >> funding for a team from a few sources. >> >> We got a ton done in that period - we moved from a v1 API which was >> synchronous to a new v2 async API, we massively increased the amount of DNS >> servers we supported, and added new features. >> >> Unfortunately, this didn't last. Internal priorities within companies >> sponsoring the development changed, and we started to shed contributors, >> which >> happens, however disappointing. Usually when this happens if a project is >> important enough the community will pick up where the previous group >> left off. >> >> We have yet to see many (meaningful) commits from the community though. We >> have some great deployers who will file bugs, and if they can put up patch >> sets - but they are (incredibly valuable and appreciated) tactical >> contributions. A project cannot survive on them, and we are no exception. >> >> So where does that leave us? Let have a look at how many actual commits we >> have had: >> >> Commits per cycle >> >> +--------------+---------+ >> | Havana | 172 | >> +--------------+---------+ >> | Icehouse | 165 | >> +--------------+---------+ >> | Juno | 254 | >> +--------------+---------+ >> | Kilo | 340 | >> +--------------+---------+ >> | Liberty | 327 | >> +--------------+---------+ >> | Mitaka | 246 | >> +--------------+---------+ >> | Newton | 299 | >> +--------------+---------+ >> | Ocata | 98 | >> +--------------+---------+ >> >> Next cycle, we are going to have 2 community goals: >> >> * Control Plane API endpoints deployment via WSGI >> * Python 3.5 functional testing >> >> We would have been actually OK for the tempest one - we were one of the >> first >> external repo based plug-ins with `designate-tempest-plugin`_ >> >> For WSGI based APIs, this will be a chunk of work - due to our internal code >> structure splitting out the API is going to be ... an issue. (and I think it >> will be harder than most people expect - anyone using olso.service has >> eventlet imported - I am not sure how that affects running in a WSGI server) >> >> Python 3.5 - I have no idea. We can't even run all our unit tests on python >> 3.5, so I suspect getting functional testing may be an issue. And, >> convincing >> management that re-factoring parts of the code base due to "community goals" >> or a future potential pay-off can be more difficult than it should. >> >> http://graham.hayes.ie/images/oct-2016-projects-prod.jpg >> >> We now have a situation where the largest "non-core" project [1]_ in the >> tent >> has a tiny number of developers working on it. 42% of deployers are >> evaluating >> Designate, so we should see this start to increase. >> >> How did this happen? >> ==================== >> >> Like most situations, there is no single cause. >> >> Certainly there may have been fault on the side of the Designate leadership. >> We had started out as a small team, and had built a huge amount of trust and >> respect based on in person interactions over a few years, which meant that >> there was a fair bit of "tribal knowledge" in the heads of a few people, and >> that new people had a hard time becoming part of the group. >> >> Also, due to volume of work done by this small group, a lot of users / >> distros >> were OK leaving us work - some of us were also running a production >> designate >> service during this time, so we knew what we needed to develop, and we had >> pretty quick feedback when we made a mistake, or caused a bug. All of this >> resulted in the major development cost being funded by two companies, which >> left us vulnerable to changes in direction from those companies. Then that >> shoe dropped. We are now one corporate change of direction from having no >> cores on the project being paid to work on the project. [2]_ >> >> Preceding this, the governance of OpenStack changed to the `Big Tent`_ >> While this change was a good thing for the OpenStack project as a >> whole it had quite a bad impact on us. >> >> Pre Big Tent, you got integrated. This was at least a cycle, where you moved >> docs to docs.openstack.org, integrated with QA testing tooling, got packaged >> by Linux distros, and build cross project features. >> >> When this was a selective thing, there was teams available to help with >> that, >> docs teams would help with content (and tooling - docs was a mass of XML >> back >> then), QA would help with tempest and devstack, horizon would help with >> panels. >> >> In Big Tent, there just wasn't resources to do this - the scope of the >> project >> expansion was huge. However the big tent happened (in my opinion - I have >> written about this before) before the horizontal / cross project teams were >> ready. They stuck to covering the "integrated" projects, which was all >> they could do at the time. >> >> This left us in a position of having to reimplement tooling, figure out >> what tooling we did have access to, and migrate everything we had on our >> own. And, as a project that (at our peak level of contribution) only >> ever had >> 5% of the number of contributors compared to a project like nova, this put >> quite a load on our developers. Things like grenade, tempest and horizon >> plug-ins, took weeks to figure out all of which took time from other vital >> things like docs, functional tests and getting designate into other tools. >> >> One of the companies who invested in designate had a QE engineer that >> used to >> contribute, and I can honestly say that the quality of our testing >> improved 10 >> fold during the time he worked with us. Not just from in repo tests, but >> from >> standing up full deployment stacks, and trying to break them - we >> learned a lot >> about how we could improve things from his expertise. >> >> Which is kind of the point I think. Nobody is amazing at everything. You >> need >> people with domain knowledge to work on these areas. If you asked me to do a >> multi-node grenade job, I would either start drinking, throw my laptop >> at you >> or do both. >> >> We still have some of these problems to this day - most of our docs are >> in a messy pile in https://docs.openstack.org/developer/designate >> while we still have a small amount of old functional tests that are not >> ported >> from our old non plug-in style. >> >> All of this adds up to make projects like Designate much less attractive >> to users - we just need to look at the `project navigator`_ to see what >> a bad >> image potential users get of us. [3]_ This is for a project that was ran >> as a >> full (non beta) service in a public cloud. [4]_ >> >> Where too now then? >> =================== >> >> Well, this is where I call out to people who actually use the project - >> don't >> jump ship and use something else because of the picture I have painted. >> We are >> a dedicated team, how cares about the project. We just need some help. >> >> I know there are large telcos who use Designate. I am sure there is tooling, >> or docs build up in these companies that could be very useful to the >> project. >> >> Nearly every commercial OpenStack distro has Designate. Some have had it >> since >> the beginning. Again, developers, docs, tooling, testers, anything and >> everything is welcome. We don't need a massive amount of resources - we >> are a >> small ish, stable, project. >> >> We need developers with upstream time allocated, and the budget to go to >> events >> like the PTG - for cross project work, and internal designate road map, >> these >> events form the core of how we work. >> >> We also need help from cross project teams - the work done by them is >> brilliant >> but it can be hard for smaller projects to consume. We have had a lot of >> progress since the `Leveller Playing Field`_ debate, but a lot of work is >> still optimised for the larger teams who get direct support, or well >> resourced >> teams who can dedicate people to the implementation of plugins / code. >> >> As someone I was talking to recently said - AWS is not winning public cloud >> because of commodity compute (that does help - a lot), but because of the >> added services that make using the cloud, well, cloud like. OpenStack >> needs to >> decide that either it is just compute, or if it wants the eco-system. [5]_ >> Designate is far from alone in this. >> >> I am happy to talk to anyone about helping to fill in the needed resources - >> Designate is a project that started in the very office I am writing this >> blog >> post in, and something I want to last. >> >> For a visual this is Designate team in Atlanta, just before we got >> incubated. >> >> http://graham.hayes.ie/images/ATL.jpg >> >> and this was our last mid cycle: >> >> http://graham.hayes.ie/images/mid-cycle.jpg >> >> and in Atlanta at the PTG, there will be two of us. >> >> >> [1] In the `Oct-2016`_ User Survey Designate was deployed in 23% of clouds >> [2] I have been lucky to have a management chain that is OK with me >> spending some time on Designate, and have not asked me to take time off >> for Summits or Gatherings, but my day job is working on a completely >> different project. >> [3] I do have other issues with the metrics - mainly that we existed >> before becoming leaving stackforge, and some of the other stats are set >> so high, that non "core" projects will probably never meet them. >> [4] I recently went to an internal training talk, where they were >> talking about new features in Newton. There was a whole slide about how >> projects and improved, or gotten worse on these scores. A whole slide. >> With tables of scores, and I think there may have even been a graph. >> [5] Now, I am slightly biased, but I would argue that DNS is needed in >> commodity compute, but again, that is my view. >> >> .. _Oct-2016: https://www.openstack.org/analytics >> .. _Big Tent: >> https://governance.openstack.org/tc/resolutions/20141202-project-structure-reform-spec.html >> .. _designate-tempest-plugin: >> https://github.com/openstack/designate-tempest-plugin >> .. _project navigator: >> https://www.openstack.org/software/releases/newton/components/designate >> .. _Leveller Playing Field: >> http://graham.hayes.ie/posts/openstack-a-leveler-playing-field/ >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: [email protected] Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington -------------------------------------------------------------------------- __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
