Re: [openstack-dev] Reasoning behind my vote on the Go topic
Excerpts from Michael Barton's message of 2016-06-09 15:59:24 -0500: > On Thu, Jun 9, 2016 at 2:49 PM, Clint Byrum wrote: > > > > Agreed it isn't done in uvloop. But it is done in libuv and the uvloop > > devs agree it should be done. So this is the kind of thing where the > > community can invest in python + C to help solve problems thought only > > solvable by other languages. > > > I mean, if someone wants to figure out a file server in python that can > compete in any way with a go version, I'm totally down for rewriting swift > to target some other python-based architecture. > > But personally, my desire to try to build a universe where such a thing is > possible is pretty low. Because I've been fighting with it for years, and > go already works great and there's nothing wrong with it. > Mike, the whole entire crux of this thread, and Monty's words, is that this sort of sentiment is hard to ignore, but it's even harder to ignore the massive amount of inertia and power there is in having a community that can all work on each others' code without investing a lot of time in learning a new language. That inertia is entirely the reason why other languages have surpassed Python in some areas like concurrency. It takes longer to turn a massive community going really hard in one direction than it does to just start off heading in that direction in the first place. But that turn is starting, and I for one think it's worth everyone's time to take a hard look at whether or not we can in fact get it done together. Nobody will force you to, but what I think Monty, and the rest of the TC members who have voted to stay the course, are asking us all to do, is to try to throw what we can at python solutions for these problems. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On Thu, Jun 9, 2016 at 2:49 PM, Clint Byrum wrote: > > Agreed it isn't done in uvloop. But it is done in libuv and the uvloop > devs agree it should be done. So this is the kind of thing where the > community can invest in python + C to help solve problems thought only > solvable by other languages. I mean, if someone wants to figure out a file server in python that can compete in any way with a go version, I'm totally down for rewriting swift to target some other python-based architecture. But personally, my desire to try to build a universe where such a thing is possible is pretty low. Because I've been fighting with it for years, and go already works great and there's nothing wrong with it. - Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
Excerpts from Michael Barton's message of 2016-06-09 14:01:11 -0500: > On Thu, Jun 9, 2016 at 9:58 AM, Ben Meyer wrote: > > > > > uvloop (first commit 2015-11-01) is newer than Swift's hummingbird > > (2015-04-20, based on > > > > https://github.com/openstack/swift/commit/a0e300df180f7f4ca64fc1eaf3601a1a73fc68cb > > and github network graph) so it would not have been part of the > > consideration. > > > > And it still wouldn't be, since it doesn't solve the problem. > Agreed it isn't done in uvloop. But it is done in libuv and the uvloop devs agree it should be done. So this is the kind of thing where the community can invest in python + C to help solve problems thought only solvable by other languages. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On Thu, Jun 9, 2016 at 9:58 AM, Ben Meyer wrote: > > uvloop (first commit 2015-11-01) is newer than Swift's hummingbird > (2015-04-20, based on > > https://github.com/openstack/swift/commit/a0e300df180f7f4ca64fc1eaf3601a1a73fc68cb > and github network graph) so it would not have been part of the > consideration. > And it still wouldn't be, since it doesn't solve the problem. - Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On 06/08/2016 11:05 PM, Chris Friesen wrote: > On 06/07/2016 04:26 PM, Ben Meyer wrote: >> On 06/07/2016 06:09 PM, Samuel Merritt wrote: >>> On 6/7/16 12:00 PM, Monty Taylor wrote: [snip] I'd rather see us focus energy on Python3, asyncio and its pluggable event loops. The work in: http://magic.io/blog/uvloop-blazing-fast-python-networking/ is a great indication in an actual apples-to-apples comparison of what can be accomplished in python doing IO-bound activities by using modern Python techniques. I think that comparing python2+eventlet to a fresh rewrite in Go isn't 100% of the story. A TON of work has gone in to Python that we're not taking advantage of because we're still supporting Python2. So what I've love to see in the realm of comparative experimentation is to see if the existing Python we already have can be leveraged as we adopt newer and more modern things. >>> >>> Asyncio, eventlet, and other similar libraries are all very good for >>> performing asynchronous IO on sockets and pipes. However, none of them >>> help for filesystem IO. That's why Swift needs a golang object server: >>> the go runtime will keep some goroutines running even though some >>> other goroutines are performing filesystem IO, whereas filesystem IO >>> in Python blocks the entire process, asyncio or no asyncio. >> >> That can be modified. gevent has a tool >> (http://www.gevent.org/gevent.fileobject.html) that enables the File IO >> to be async as well by putting the file into non-blocking mode. I've >> used it, and it works and scales well. > > Arguably non-blocking isn't really async when it comes to reads. I > suspect what we really want is full-async where you issue a request > and then get notified when it's done. So when it comes to Swift or Glance where you have to transfer large amounts of data between the HTTP client and HTTP serverunder WSGI, the only way to make it truly cooperative for eventlet, gevent, etc is to use non-blocking File I/O. These situations also reveal how uncooperative green threads are too since if you can keep the data pipeline full (e.g the read is continuous because the OS is able to quickly service it) then one greed thread will take over and block the others. Non-blocking I/O is only part of the solution, not the entire solution. Furthermore, the constraint of the web head having to provide the data transfer makes the full-async offload with a notification impossible for data transfer services like Swift. Tools like uvloop make great strides in providing better environments for doing cooperative tasks than what existed prior, and some of the articles comparing it to the various tools - both alone and in combination - make for some fascinating possibilities and solutions. Most likely the best solution here is going to be going to Py3's asyncio+uvloop+non-blocking I/O on the files in order to hit the throughput for Swift. uvloop (first commit 2015-11-01) is newer than Swift's hummingbird (2015-04-20, based on https://github.com/openstack/swift/commit/a0e300df180f7f4ca64fc1eaf3601a1a73fc68cb and github network graph) so it would not have been part of the consideration. $0.02 Ben __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On 06/07/2016 04:26 PM, Ben Meyer wrote: On 06/07/2016 06:09 PM, Samuel Merritt wrote: On 6/7/16 12:00 PM, Monty Taylor wrote: [snip] I'd rather see us focus energy on Python3, asyncio and its pluggable event loops. The work in: http://magic.io/blog/uvloop-blazing-fast-python-networking/ is a great indication in an actual apples-to-apples comparison of what can be accomplished in python doing IO-bound activities by using modern Python techniques. I think that comparing python2+eventlet to a fresh rewrite in Go isn't 100% of the story. A TON of work has gone in to Python that we're not taking advantage of because we're still supporting Python2. So what I've love to see in the realm of comparative experimentation is to see if the existing Python we already have can be leveraged as we adopt newer and more modern things. Asyncio, eventlet, and other similar libraries are all very good for performing asynchronous IO on sockets and pipes. However, none of them help for filesystem IO. That's why Swift needs a golang object server: the go runtime will keep some goroutines running even though some other goroutines are performing filesystem IO, whereas filesystem IO in Python blocks the entire process, asyncio or no asyncio. That can be modified. gevent has a tool (http://www.gevent.org/gevent.fileobject.html) that enables the File IO to be async as well by putting the file into non-blocking mode. I've used it, and it works and scales well. Arguably non-blocking isn't really async when it comes to reads. I suspect what we really want is full-async where you issue a request and then get notified when it's done. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On Wed, Jun 8, 2016, at 03:46 AM, Thierry Carrez wrote: > Another option (raised by dims) is to find a way to allow usage of > golang (or another language) in a more granular way: selectively allow > projects which really need another tool to use it. The benefit is that > it lets project teams make a case and use the best tool for the job, > while limiting the cross-project impact and without opening the > distraction floodgates of useless rewrites. The drawback is that > depending on how it's done, it may place the TC in the role of granting > "you're tall enough to use Go" badges, creating new endless discussions > and more "you're special" exceptions. That said, I'm still interested in > exploring that option, for one reason. I think that whenever a project > team considers adding a component or a rewrite in another language, they > are running into an interesting problem with Python, on which they > really could use advice from the rest of the OpenStack community. I'd > really like to see a cross-project team of Python performance experts to > look at the problem this specific team has that makes them want to use > another language. That seems like a great way to drive more practice > sharing and reduce fragmentation in "OpenStack" in general. We might > just need to put the bar pretty high so that we are not flooded by silly > rewrite requests. > ++. There's a lot of value in these issues getting bubbled up to the cross-project level: If we have identified a serious hurdle then this knowledge really shouldn't live inside of a single project. Otherwise, if we haven't identified such an issue, then the (the greater openstack community) can offer some alternative solutions which is also a huge win. I completely understand the fear that we might be creating an endless review stream for the TC by making them the review squad for getting approval to use a new language, and I agree that we need to make sure that doesn't happen. OTOH, I strongly believe that in almost all of the cases which would be proposed some alternative solutions could be found. I worry that if we just tell these folks 'the solution you thought of isn't allowed' rather than offer an outlet for seriously investigating the issue were likely to see teams try and find ways around that restriction when really we want to identify another solution to the problem. A perf team sounds like a great way to help both the tribal knowledge problem and support the type of problem solving we are asking for. Sign me up :). Cheers, Greg __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
Samuel Merritt wrote: On 6/7/16 12:00 PM, Monty Taylor wrote: [snip] > I'd rather see us focus energy on Python3, asyncio and its pluggable event loops. The work in: http://magic.io/blog/uvloop-blazing-fast-python-networking/ is a great indication in an actual apples-to-apples comparison of what can be accomplished in python doing IO-bound activities by using modern Python techniques. I think that comparing python2+eventlet to a fresh rewrite in Go isn't 100% of the story. A TON of work has gone in to Python that we're not taking advantage of because we're still supporting Python2. So what I've love to see in the realm of comparative experimentation is to see if the existing Python we already have can be leveraged as we adopt newer and more modern things. Asyncio, eventlet, and other similar libraries are all very good for performing asynchronous IO on sockets and pipes. However, none of them help for filesystem IO. That's why Swift needs a golang object server: the go runtime will keep some goroutines running even though some other goroutines are performing filesystem IO, whereas filesystem IO in Python blocks the entire process, asyncio or no asyncio. As you probably know by now, there was a majority of TC members to deny the addition of Golang as another generally available tool in our toolbelt. The resolution (which was made to follow the guidelines the TC itself set as the process to follow in that specific case) was therefore rejected. What made this decision pretty hard for everyone involved is that the Swift team built a compelling argument (see just above) on why they can't really solve a real user problem they have (scale and perform better) using the tools they are allowed to use in "OpenStack". There was probably also a majority of TC members agreeing that Swift needs to use another language to solve the specific filesystem IO problem they are running into. So we have been looking at various trade-off solutions to get the projects the tools they really need without making cross-project work significantly more complex, and without triggering the gigantic distraction of superfluous rewrites across OpenStack projects. None of those middle-ground solutions is optimal. But I've been asked to list them for clarity. One option is to develop generally-useful libraries or services outside of OpenStack that solve the specific issue you're running into. OpenStack relies on a lot of dependencies in the wider open source community, we don't have to develop *everything* as official "openstack projects" under TC governance. We even have a number of them that are using OpenStack infrastructure (git, gerrit, gate...). The benefit of this approach is that it lets you pick whatever language or framework is most appropriate, and forces clean contracts between the components, which makes replacing the external piece easier in the future. The drawback of this approach is that it creates local fragmentation at the project team level, as they have to artificially split the project into "OpenStack" and non-"OpenStack" bits. And it is also not optimal for Swift due to the way it's currently designed, you can't really slice it in this way. Another option (raised by dims) is to find a way to allow usage of golang (or another language) in a more granular way: selectively allow projects which really need another tool to use it. The benefit is that it lets project teams make a case and use the best tool for the job, while limiting the cross-project impact and without opening the distraction floodgates of useless rewrites. The drawback is that depending on how it's done, it may place the TC in the role of granting "you're tall enough to use Go" badges, creating new endless discussions and more "you're special" exceptions. That said, I'm still interested in exploring that option, for one reason. I think that whenever a project team considers adding a component or a rewrite in another language, they are running into an interesting problem with Python, on which they really could use advice from the rest of the OpenStack community. I'd really like to see a cross-project team of Python performance experts to look at the problem this specific team has that makes them want to use another language. That seems like a great way to drive more practice sharing and reduce fragmentation in "OpenStack" in general. We might just need to put the bar pretty high so that we are not flooded by silly rewrite requests. For completeness I'll also list the nuclear option: to reject the Technical Committee oversight altogether and decide that your project is better off as an autonomous project in the OpenStack ecosystem. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/c
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On Tue, Jun 7, 2016 at 4:26 PM, Ben Meyer wrote: > On 06/07/2016 06:09 PM, Samuel Merritt wrote: > > On 6/7/16 12:00 PM, Monty Taylor wrote: > >> [snip] > > > > >> I'd rather see us focus energy on Python3, asyncio and its pluggable > >> event loops. The work in: > >> > >> http://magic.io/blog/uvloop-blazing-fast-python-networking/ > >> > >> is a great indication in an actual apples-to-apples comparison of what > >> can be accomplished in python doing IO-bound activities by using modern > >> Python techniques. I think that comparing python2+eventlet to a fresh > >> rewrite in Go isn't 100% of the story. A TON of work has gone in to > >> Python that we're not taking advantage of because we're still supporting > >> Python2. So what I've love to see in the realm of comparative > >> experimentation is to see if the existing Python we already have can be > >> leveraged as we adopt newer and more modern things. > > > > Asyncio, eventlet, and other similar libraries are all very good for > > performing asynchronous IO on sockets and pipes. However, none of them > > help for filesystem IO. That's why Swift needs a golang object server: > > the go runtime will keep some goroutines running even though some > > other goroutines are performing filesystem IO, whereas filesystem IO > > in Python blocks the entire process, asyncio or no asyncio. > > That can be modified. gevent has a tool > (http://www.gevent.org/gevent.fileobject.html) that enables the File IO > to be async as well by putting the file into non-blocking mode. I've > used it, and it works and scales well. > > Sadly, Python doesn't offer this by default; perhaps OpenStack can get > that changed. > > $0.02 > > Ben > > The uvloop library is very new, but libuv itself uses a thread pool, so it should work just fine with files once this functionality is implemented; see https://github.com/MagicStack/uvloop/issues/1 and https://nikhilm.github.io/uvbook/filesystem.html I don't see any difference here with how Golang would implement such async support, in terms of mapping goroutines against its own thread pool. With respect to latency, there could even be better performance for certain workloads with uvloop than Go. This is because of CPython's refcounting, vs Go's GC - even with all the recent improvements in Go ( https://blog.golang.org/go15gc), Go still does stop the world. A key piece here is that uvloop does not hold the GIL in these ops - a big advantage that Python C extensions enjoy, including libraries they wrap, and Cython explicitly enables with nogil support. So in particular we have the following code in the the core uvloop function, which in turn is just calling into libuv with its own core function, uv_run: ``` # Although every UVHandle holds a reference to the loop, # we want to do everything to ensure that the loop will # never deallocate during the run -- so we do some # manual refs management. Py_INCREF(self) with nogil: err = uv.uv_run(self.uvloop, mode) Py_DECREF(self) ``` More here on nogil support in Cython: http://docs.cython.org/src/userguide/external_C_code.html#declaring-a-function-as-callable-without-the-gil - Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On 06/07/2016 06:09 PM, Samuel Merritt wrote: > On 6/7/16 12:00 PM, Monty Taylor wrote: >> [snip] > > >> I'd rather see us focus energy on Python3, asyncio and its pluggable >> event loops. The work in: >> >> http://magic.io/blog/uvloop-blazing-fast-python-networking/ >> >> is a great indication in an actual apples-to-apples comparison of what >> can be accomplished in python doing IO-bound activities by using modern >> Python techniques. I think that comparing python2+eventlet to a fresh >> rewrite in Go isn't 100% of the story. A TON of work has gone in to >> Python that we're not taking advantage of because we're still supporting >> Python2. So what I've love to see in the realm of comparative >> experimentation is to see if the existing Python we already have can be >> leveraged as we adopt newer and more modern things. > > Asyncio, eventlet, and other similar libraries are all very good for > performing asynchronous IO on sockets and pipes. However, none of them > help for filesystem IO. That's why Swift needs a golang object server: > the go runtime will keep some goroutines running even though some > other goroutines are performing filesystem IO, whereas filesystem IO > in Python blocks the entire process, asyncio or no asyncio. That can be modified. gevent has a tool (http://www.gevent.org/gevent.fileobject.html) that enables the File IO to be async as well by putting the file into non-blocking mode. I've used it, and it works and scales well. Sadly, Python doesn't offer this by default; perhaps OpenStack can get that changed. $0.02 Ben __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
On 6/7/16 12:00 PM, Monty Taylor wrote: [snip] > I'd rather see us focus energy on Python3, asyncio and its pluggable event loops. The work in: http://magic.io/blog/uvloop-blazing-fast-python-networking/ is a great indication in an actual apples-to-apples comparison of what can be accomplished in python doing IO-bound activities by using modern Python techniques. I think that comparing python2+eventlet to a fresh rewrite in Go isn't 100% of the story. A TON of work has gone in to Python that we're not taking advantage of because we're still supporting Python2. So what I've love to see in the realm of comparative experimentation is to see if the existing Python we already have can be leveraged as we adopt newer and more modern things. Asyncio, eventlet, and other similar libraries are all very good for performing asynchronous IO on sockets and pipes. However, none of them help for filesystem IO. That's why Swift needs a golang object server: the go runtime will keep some goroutines running even though some other goroutines are performing filesystem IO, whereas filesystem IO in Python blocks the entire process, asyncio or no asyncio. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Reasoning behind my vote on the Go topic
>rather than making progress on OpenStack, we'll spend the next 4 years bikeshedding broadly about which bits, if any, should be rewritten in Go. 100% agreed, and well said. On Tue, Jun 7, 2016 at 12:00 PM, Monty Taylor wrote: > This text is in my vote, but as I'm sure there are people who do not > read all of the gerrit comments for governance changes, I'm posting it > here so that my thoughts are clear. > > Please know that this has actually kept me up at night. I cast my vote > on this neither glibly or superficially. I have talked to everyone I can > possibly think of on the topic, and at the end, the only thing I can do > is use my judgment and vote to the best of my ability. I apologize from > the bottom of my heart to the people I find myself in disagreement with. > I have nothing but the utmost respect for you all. > > I vote against allowing Go as an official language of OpenStack. > > "The needs of the many outweigh the needs of the few, or the one" > > I'm super unhappy about both possible votes here. > > I think go is a wonderful language. I think hummingbird is a well > considered solution to a particular problem. I think that lack of > flexibility is broadly speaking not a problem we have in OpenStack > currently. I'm more worried about community cohesion in a post-Big Tent > world than I am about specific optimization. > > I do not think that adding Go as a language to OpenStack today is enough > of a win to justify the cost, so I don't like accepting it. > > I do not think that this should preclude serious thought about > OpenStack's technology underpinnings, so I don't like rejecting it. > > "only a great fool would reach for what he was given..." > > I think that one of OpenStack's biggest and most loudly spoken about > problems is too many per-project solutions and not enough holistic > solutions. Because of that, and the six years of experience we have > seeing where that gets us, I do not think that adding Go into the mix > and "seeing what happens" is going to cause anything other than chaos. > > If we want to add Go, or any other language, into the mix for server > projects, I think it should be done with the intent that we are going to > do it because it's a markedly better choice across the board, that we > are going to rewrite literally everything, and I believe that we should > consider the cost associated with retraining 2000 developers as part of > considering that. Before you think that that's me throwing the baby out > with the bathwater... > > In a previous comment, Deklan says: > > "If Go was accepted as an officially supported language in the OpenStack > community, I'd be the first to start to rewrite as much code as possible > in Go." > > That is, in fact, EXACTLY the concern. That rather than making progress > on OpenStack, we'll spend the next 4 years bikeshedding broadly about > which bits, if any, should be rewritten in Go. It took Juju YEARS to > rewrite from Python to Go and to hit feature parity. The size of that > codebase was much smaller and they even had a BDFL (which people keep > telling us makes things go quicker) > > It could be argued that we could exercise consideration about which > things get rewritten in Go so as to avoid that, but I'm pretty sure that > would just mean that the only conversation the TC would have for the > next two years would be "should X be in Go or Python" - and we'd have > strong proponents from each project on each side of the argument. > > David Goetz says "you aren’t doing the community any favors by deciding > for them how they do their jobs". I get that, and can respect that point > of view. However, for the most part, the negative feedback we get as > members of the TC is actually that we're too lax, not that we're too > strict. > > I know that it's a popular sentiment with some folks to say "let devs > use whatever tool they want to." However, that has never been our > approach with OpenStack. It has been suggested multiple times and > aligning on limited chosen tech has always been the thing we've chosen. > I tend to align in my personal thinking more with Dan McKinley in: > > http://mcfunley.com/choose-boring-technology > > I have effectively been arguing his point for as long as I've been > involved in OpenStack governance - although probably not as well as he > does. I don't see any reason to reverse myself now. > > I'd rather see us focus energy on Python3, asyncio and its pluggable > event loops. The work in: > > http://magic.io/blog/uvloop-blazing-fast-python-networking/ > > is a great indication in an actual apples-to-apples comparison of what > can be accomplished in python doing IO-bound activities by using modern > Python techniques. I think that comparing python2+eventlet to a fresh > rewrite in Go isn't 100% of the story. A TON of work has gone in to > Python that we're not taking advantage of because we're still supporting > Python2. So what I've love to see in the realm of comparative > experimentation i
Re: [openstack-dev] Reasoning behind my vote on the Go topic
Well put Monty. It is a tough choice and neither choice was inexpensive. -amrith > -Original Message- > From: Monty Taylor [mailto:mord...@inaugust.com] > Sent: Tuesday, June 07, 2016 3:00 PM > To: OpenStack Development Mailing List (not for usage questions) > > Subject: [openstack-dev] Reasoning behind my vote on the Go topic > > This text is in my vote, but as I'm sure there are people who do not > read all of the gerrit comments for governance changes, I'm posting it > here so that my thoughts are clear. > > Please know that this has actually kept me up at night. I cast my vote > on this neither glibly or superficially. I have talked to everyone I can > possibly think of on the topic, and at the end, the only thing I can do > is use my judgment and vote to the best of my ability. I apologize from > the bottom of my heart to the people I find myself in disagreement with. > I have nothing but the utmost respect for you all. > > I vote against allowing Go as an official language of OpenStack. > > "The needs of the many outweigh the needs of the few, or the one" > > I'm super unhappy about both possible votes here. > > I think go is a wonderful language. I think hummingbird is a well > considered solution to a particular problem. I think that lack of > flexibility is broadly speaking not a problem we have in OpenStack > currently. I'm more worried about community cohesion in a post-Big Tent > world than I am about specific optimization. > > I do not think that adding Go as a language to OpenStack today is enough > of a win to justify the cost, so I don't like accepting it. > > I do not think that this should preclude serious thought about > OpenStack's technology underpinnings, so I don't like rejecting it. > > "only a great fool would reach for what he was given..." > > I think that one of OpenStack's biggest and most loudly spoken about > problems is too many per-project solutions and not enough holistic > solutions. Because of that, and the six years of experience we have > seeing where that gets us, I do not think that adding Go into the mix > and "seeing what happens" is going to cause anything other than chaos. > > If we want to add Go, or any other language, into the mix for server > projects, I think it should be done with the intent that we are going to > do it because it's a markedly better choice across the board, that we > are going to rewrite literally everything, and I believe that we should > consider the cost associated with retraining 2000 developers as part of > considering that. Before you think that that's me throwing the baby out > with the bathwater... > > In a previous comment, Deklan says: > > "If Go was accepted as an officially supported language in the OpenStack > community, I'd be the first to start to rewrite as much code as possible > in Go." > > That is, in fact, EXACTLY the concern. That rather than making progress > on OpenStack, we'll spend the next 4 years bikeshedding broadly about > which bits, if any, should be rewritten in Go. It took Juju YEARS to > rewrite from Python to Go and to hit feature parity. The size of that > codebase was much smaller and they even had a BDFL (which people keep > telling us makes things go quicker) > > It could be argued that we could exercise consideration about which > things get rewritten in Go so as to avoid that, but I'm pretty sure that > would just mean that the only conversation the TC would have for the > next two years would be "should X be in Go or Python" - and we'd have > strong proponents from each project on each side of the argument. > > David Goetz says "you aren’t doing the community any favors by deciding > for them how they do their jobs". I get that, and can respect that point > of view. However, for the most part, the negative feedback we get as > members of the TC is actually that we're too lax, not that we're too > strict. > > I know that it's a popular sentiment with some folks to say "let devs > use whatever tool they want to." However, that has never been our > approach with OpenStack. It has been suggested multiple times and > aligning on limited chosen tech has always been the thing we've chosen. > I tend to align in my personal thinking more with Dan McKinley in: > > http://mcfunley.com/choose-boring-technology > > I have effectively been arguing his point for as long as I've been > involved in OpenStack governance - although probably not as well as he > does. I don't see any reason to reverse myself now. > > I'd rather see us focus energy o
Re: [openstack-dev] Reasoning behind my vote on the Go topic
Well said and reasoned, Monty. On 06/07/2016 03:00 PM, Monty Taylor wrote: This text is in my vote, but as I'm sure there are people who do not read all of the gerrit comments for governance changes, I'm posting it here so that my thoughts are clear. Please know that this has actually kept me up at night. I cast my vote on this neither glibly or superficially. I have talked to everyone I can possibly think of on the topic, and at the end, the only thing I can do is use my judgment and vote to the best of my ability. I apologize from the bottom of my heart to the people I find myself in disagreement with. I have nothing but the utmost respect for you all. I vote against allowing Go as an official language of OpenStack. "The needs of the many outweigh the needs of the few, or the one" I'm super unhappy about both possible votes here. I think go is a wonderful language. I think hummingbird is a well considered solution to a particular problem. I think that lack of flexibility is broadly speaking not a problem we have in OpenStack currently. I'm more worried about community cohesion in a post-Big Tent world than I am about specific optimization. I do not think that adding Go as a language to OpenStack today is enough of a win to justify the cost, so I don't like accepting it. I do not think that this should preclude serious thought about OpenStack's technology underpinnings, so I don't like rejecting it. "only a great fool would reach for what he was given..." I think that one of OpenStack's biggest and most loudly spoken about problems is too many per-project solutions and not enough holistic solutions. Because of that, and the six years of experience we have seeing where that gets us, I do not think that adding Go into the mix and "seeing what happens" is going to cause anything other than chaos. If we want to add Go, or any other language, into the mix for server projects, I think it should be done with the intent that we are going to do it because it's a markedly better choice across the board, that we are going to rewrite literally everything, and I believe that we should consider the cost associated with retraining 2000 developers as part of considering that. Before you think that that's me throwing the baby out with the bathwater... In a previous comment, Deklan says: "If Go was accepted as an officially supported language in the OpenStack community, I'd be the first to start to rewrite as much code as possible in Go." That is, in fact, EXACTLY the concern. That rather than making progress on OpenStack, we'll spend the next 4 years bikeshedding broadly about which bits, if any, should be rewritten in Go. It took Juju YEARS to rewrite from Python to Go and to hit feature parity. The size of that codebase was much smaller and they even had a BDFL (which people keep telling us makes things go quicker) It could be argued that we could exercise consideration about which things get rewritten in Go so as to avoid that, but I'm pretty sure that would just mean that the only conversation the TC would have for the next two years would be "should X be in Go or Python" - and we'd have strong proponents from each project on each side of the argument. David Goetz says "you aren’t doing the community any favors by deciding for them how they do their jobs". I get that, and can respect that point of view. However, for the most part, the negative feedback we get as members of the TC is actually that we're too lax, not that we're too strict. I know that it's a popular sentiment with some folks to say "let devs use whatever tool they want to." However, that has never been our approach with OpenStack. It has been suggested multiple times and aligning on limited chosen tech has always been the thing we've chosen. I tend to align in my personal thinking more with Dan McKinley in: http://mcfunley.com/choose-boring-technology I have effectively been arguing his point for as long as I've been involved in OpenStack governance - although probably not as well as he does. I don't see any reason to reverse myself now. I'd rather see us focus energy on Python3, asyncio and its pluggable event loops. The work in: http://magic.io/blog/uvloop-blazing-fast-python-networking/ is a great indication in an actual apples-to-apples comparison of what can be accomplished in python doing IO-bound activities by using modern Python techniques. I think that comparing python2+eventlet to a fresh rewrite in Go isn't 100% of the story. A TON of work has gone in to Python that we're not taking advantage of because we're still supporting Python2. So what I've love to see in the realm of comparative experimentation is to see if the existing Python we already have can be leveraged as we adopt newer and more modern things. In summary, while I think that Go is a lovely language and the people who work on it are lovely people, while I'm sure that hummingbird is beneficial to the Cloud Files team in real ways and while I'm sure that if
[openstack-dev] Reasoning behind my vote on the Go topic
This text is in my vote, but as I'm sure there are people who do not read all of the gerrit comments for governance changes, I'm posting it here so that my thoughts are clear. Please know that this has actually kept me up at night. I cast my vote on this neither glibly or superficially. I have talked to everyone I can possibly think of on the topic, and at the end, the only thing I can do is use my judgment and vote to the best of my ability. I apologize from the bottom of my heart to the people I find myself in disagreement with. I have nothing but the utmost respect for you all. I vote against allowing Go as an official language of OpenStack. "The needs of the many outweigh the needs of the few, or the one" I'm super unhappy about both possible votes here. I think go is a wonderful language. I think hummingbird is a well considered solution to a particular problem. I think that lack of flexibility is broadly speaking not a problem we have in OpenStack currently. I'm more worried about community cohesion in a post-Big Tent world than I am about specific optimization. I do not think that adding Go as a language to OpenStack today is enough of a win to justify the cost, so I don't like accepting it. I do not think that this should preclude serious thought about OpenStack's technology underpinnings, so I don't like rejecting it. "only a great fool would reach for what he was given..." I think that one of OpenStack's biggest and most loudly spoken about problems is too many per-project solutions and not enough holistic solutions. Because of that, and the six years of experience we have seeing where that gets us, I do not think that adding Go into the mix and "seeing what happens" is going to cause anything other than chaos. If we want to add Go, or any other language, into the mix for server projects, I think it should be done with the intent that we are going to do it because it's a markedly better choice across the board, that we are going to rewrite literally everything, and I believe that we should consider the cost associated with retraining 2000 developers as part of considering that. Before you think that that's me throwing the baby out with the bathwater... In a previous comment, Deklan says: "If Go was accepted as an officially supported language in the OpenStack community, I'd be the first to start to rewrite as much code as possible in Go." That is, in fact, EXACTLY the concern. That rather than making progress on OpenStack, we'll spend the next 4 years bikeshedding broadly about which bits, if any, should be rewritten in Go. It took Juju YEARS to rewrite from Python to Go and to hit feature parity. The size of that codebase was much smaller and they even had a BDFL (which people keep telling us makes things go quicker) It could be argued that we could exercise consideration about which things get rewritten in Go so as to avoid that, but I'm pretty sure that would just mean that the only conversation the TC would have for the next two years would be "should X be in Go or Python" - and we'd have strong proponents from each project on each side of the argument. David Goetz says "you aren’t doing the community any favors by deciding for them how they do their jobs". I get that, and can respect that point of view. However, for the most part, the negative feedback we get as members of the TC is actually that we're too lax, not that we're too strict. I know that it's a popular sentiment with some folks to say "let devs use whatever tool they want to." However, that has never been our approach with OpenStack. It has been suggested multiple times and aligning on limited chosen tech has always been the thing we've chosen. I tend to align in my personal thinking more with Dan McKinley in: http://mcfunley.com/choose-boring-technology I have effectively been arguing his point for as long as I've been involved in OpenStack governance - although probably not as well as he does. I don't see any reason to reverse myself now. I'd rather see us focus energy on Python3, asyncio and its pluggable event loops. The work in: http://magic.io/blog/uvloop-blazing-fast-python-networking/ is a great indication in an actual apples-to-apples comparison of what can be accomplished in python doing IO-bound activities by using modern Python techniques. I think that comparing python2+eventlet to a fresh rewrite in Go isn't 100% of the story. A TON of work has gone in to Python that we're not taking advantage of because we're still supporting Python2. So what I've love to see in the realm of comparative experimentation is to see if the existing Python we already have can be leveraged as we adopt newer and more modern things. In summary, while I think that Go is a lovely language and the people who work on it are lovely people, while I'm sure that hummingbird is beneficial to the Cloud Files team in real ways and while I'm sure that if we were starting OpenStack from scratch today the conversations about how to