Re: [openstack-dev] [oslo.db] PyMySQL review
On Mon Feb 02 2015 at 11:49:31 AM Julien Danjou jul...@danjou.info wrote: On Fri, Jan 30 2015, Yuriy Taraday wrote: That's a great research! Under its impression I've spent most of last evening reading PyMySQL sources. It looks like it not as much need C speedups currently as plain old Python optimizations. Protocol parsing code seems very inefficient (chained struct.unpack's interleaved with data copying and util method calls that do the same struct.unpack with unnecessary type check... wow...) That's a huge place for improvement. I think it worth spending time on coming vacation to fix these slowdowns. We'll see if they'll pay back those 10% slowdown people are talking about. With all my respect, you may be right, but I need to say it'd be better to profile and then optimize rather than spend time rewriting random parts of the code then hoping it's going to be faster. :-) Don't worry, I do profile. Currently I use mini-benchmark Mike provided an optimizing hottest methods. I'm already getting 25% more speed in this case and that's not a limit. I will be posting pull requests to pymysql soon. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Fri, Jan 30 2015, Yuriy Taraday wrote: That's a great research! Under its impression I've spent most of last evening reading PyMySQL sources. It looks like it not as much need C speedups currently as plain old Python optimizations. Protocol parsing code seems very inefficient (chained struct.unpack's interleaved with data copying and util method calls that do the same struct.unpack with unnecessary type check... wow...) That's a huge place for improvement. I think it worth spending time on coming vacation to fix these slowdowns. We'll see if they'll pay back those 10% slowdown people are talking about. With all my respect, you may be right, but I need to say it'd be better to profile and then optimize rather than spend time rewriting random parts of the code then hoping it's going to be faster. :-) -- Julien Danjou // Free Software hacker // http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Thu, Jan 29, 2015, at 07:03 PM, Angus Lees wrote: The other factor I haven't seen mentioned here is that our usual eventlet+mysqldb setup can deadlock rather easily(*), resulting in the entire python process sitting idle for 30s until the mysqldb deadlock timer goes off and raises an exception. At this point (in nova at least), the request is usually retried and almost certainly succeeds. In neutron (as another example), the requests weren't so easy to retry and code examples where this was noticed had to be explicitly restructured to defer work that might yield the greenlet. ie: it's a big cost on coding, and deployments, as well as just being plain incorrect code. (*) eg: send simultaneous requests that modify the same resource. Eventually you'll have two greenlets within the same api server executing conflicting database operations. Entire python process freezes. (Note this is not a regular db transaction block, neither greenlet will ever exit their transactions without the deadlock timer). Bug here: https://bugs.launchpad.net/oslo.db/+bug/1350149 Test that reproduces the issue here: https://review.openstack.org/#/c/104436/ I'm dismayed that the conversation always skews towards discussing performance when the current choice isn't even correct yet, and that as a community we seem to be unwilling/unable to get behind what should be quite an obvious technical issue with straightforward solutions. My impression of this thread is that we're reaching consensus that we should move to PyMySQL, but that we want to make that move carefully because we expect issues due to the changes in performance and context switching behaviors. We've seen in the past that even slight changes in some timings expose race conditions, so while raw performance isn't critical the timing can be as tests run in the gate. Spending some time on a migration test plan, and making sure that people are ready to help debug those sorts of issues, are good next steps. The issue of moving off of eventlet can be addressed separately, after the driver change. So, do we need a cross-project spec to write down those details? Doug If we absolutely can't switch to another mysql driver, another option that was suggested recently (and passes the above test) is using eventlet.monkey_patch(MySQLdb=True). I haven't done the investigation to find out why that isn't the default, or what the downsides are. This obviously doesn't help us with other factors, like python3-ness either. - Gus __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Fri, 30 Jan 2015, Doug Hellmann wrote: My impression of this thread is that we're reaching consensus that we should move to PyMySQL, but that we want to make that move carefully because we expect issues due to the changes in performance and context switching behaviors. We've seen in the past that even slight changes in some timings expose race conditions, so while raw performance isn't critical the timing can be as tests run in the gate. Spending some time on a migration test plan, and making sure that people are ready to help debug those sorts of issues, are good next steps. If changing drivers is going to expose bugs and then lead to fixing them, clearly we should be changing drivers frequently. (Or to be less indirect: fragility in the face of timing changes is a bug, let's look upon these things as positive opportunities to fix things what lurk below.) -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On 01/28/2015 06:33 PM, Johannes Erdfelt wrote: On Wed, Jan 28, 2015, Mike Bayer mba...@redhat.com wrote: I can envision turning this driver into a total monster, adding C-speedups where needed but without getting in the way of async patching, adding new APIs for explicit async, and everything else. However, I’ve no idea what the developers have an appetite for. This is great information. I appreciate the work on evaluating it. First response spawns a thread about eventlet ... I would like to register my support of using this MySQL library. Can I bring up the alternative of dropping eventlet and switching to native threads? We spend a lot of time working on the various incompatibilies between eventlet and other libraries we use. It also restricts us by making it difficult to use an entire class of python modules (that use C extensions for performance, etc). I personally have spent more time than I wish to admit fixing bugs in eventlet and troubleshooting problems we've had. And it's never been clear to me why we *need* to use eventlet or green threads in general. Our modern Nova appears to only be weakly tied to eventlet and greenlet. I think we would spend less time replacing eventlet with native threads than we'll spend in the future trying to fit our code and dependencies into the eventlet shaped hole we currently have. I'm not as familiar with the code in other OpenStack projects, but from what I have seen, they appear to be similar to Nova and are only weakly tied to eventlet/greenlet. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Thu Jan 29 2015 at 12:59:34 AM Mike Bayer mba...@redhat.com wrote: Hey list - Hey, Mike. While PyMySQL is lacking test coverage in some areas, has no external documentation, and has at least some areas where Python performance can be improved, the basic structure of the driver is perfectly fine and straightforward. I can envision turning this driver into a total monster, adding C-speedups where needed but without getting in the way of async patching, adding new APIs for explicit async, and everything else. However, I’ve no idea what the developers have an appetite for. Please review the document at https://wiki.openstack.org/ wiki/PyMySQL_evaluation. That's a great research! Under its impression I've spent most of last evening reading PyMySQL sources. It looks like it not as much need C speedups currently as plain old Python optimizations. Protocol parsing code seems very inefficient (chained struct.unpack's interleaved with data copying and util method calls that do the same struct.unpack with unnecessary type check... wow...) That's a huge place for improvement. I think it worth spending time on coming vacation to fix these slowdowns. We'll see if they'll pay back those 10% slowdown people are talking about. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On 2015-01-29 18:35:20 +0200 (+0200), Roman Podoliaka wrote: [...] Otherwise, PyMySQL would be much slower than MySQL-Python for the typical SQL queries we do (e.g. ask for *a lot* of data from the DB). [...] Is this assertion based on representative empirical testing (for example profiling devstack+tempest, or perhaps comparing rally benchmarks), or merely an assumption which still needs validating? -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Hi all, Mike, thanks for summarizing this in https://wiki.openstack.org/wiki/PyMySQL_evaluation ! On PyMySQL: this is something we need to enable testing of oslo.db on Python 3.x and PyPy. Though, I doubt we want to make PyMySQL the default DB API driver for OpenStack services for Python 2.x. At least, not until PyMySQL provides C-speedups for hot spots in the driver code (I assume this can be done in eventlet/PyPy friendly way using cffi). Otherwise, PyMySQL would be much slower than MySQL-Python for the typical SQL queries we do (e.g. ask for *a lot* of data from the DB). On native threads vs green threads: I very much like the Keystone approach, which allows to run the service using either eventlet or Apache. It would be great, if we could do that for other services as well. Thanks, Roman On Thu, Jan 29, 2015 at 5:54 PM, Ed Leafe e...@leafe.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/28/2015 06:57 PM, Johannes Erdfelt wrote: Not sure if it helps more than this explanation, but there was a blueprint and accompanying wiki page that explains the move from twisted to eventlet: Yeah, it was never threads vs. greenthreads. There was a lot of pushback to relying on Twisted, which many people found confusing to use, and more importantly, to follow when reading code. Whatever the performance difference may be, eventlet code is a lot easier to follow, as it more closely resembles single-threaded linear execution. - -- - -- Ed Leafe -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) iQIcBAEBAgAGBQJUyle8AAoJEKMgtcocwZqLRX4P/j3LhEhubBOJmWepfO6KrG9V KxltAshKRvZ9OAKlezprZh8N5XUcxRTDxLhxkYP6Qsaf1pxHacAIOhLRobnV5Y+A oF6E6eRORs13pUhLkL+EzZ07Kon+SjSmvDDZiIo+xe8OTbgfMpo5j7zMPeLJpcr0 RUSS+knJ/ewNCMaX4gTwTY3sDYFZTbVuGHFUtjSgeoJP0T5aP05UR73xeT7/AsbW O8dOL4tX+6GcxIHyX4XdFH9hng1P2vBZJ5l8yV6BxB6U8xsiQjlsCpwrb8orYJ0r f7+YW0D0FHOvY/TV4dzsLC/2NGc2AwMszWL3kB/AzbUuDyMMDEGpbAS/VHDcyhZg l7zFKEQy+9UybVzWjw764hpzcUT/ICPbTBiX/QuN4qY9YTUNTlCNrRAslgM+cr2y x0/nb6cd+Qq21RPIygJ9HavRqOm8npF6HpUrE55Dn+3/OvfAftlWNPcdlXAjtDOt 4WUFUoZjUTsNUjLlEiiTzgfJg7+eQqbR/HFubCpILFQgOlOAuZIDcr3g8a3yw7ED wt5UZz/89LDQqpF2TZX8lKFWxeKk1CnxWEWO208+E/700JS4xKHpnVi4tj18udsY AHnihUwGO4d9Q0i+TqbomnzyqOW6SM+gDUcahfJ92IJj9e13bqpbuoN/YMbpD/o8 evOz8G3OeC/KaOgG5F/1 =1M8U -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Roman Podoliaka rpodoly...@mirantis.com wrote: On native threads vs green threads: I very much like the Keystone approach, which allows to run the service using either eventlet or Apache. It would be great, if we could do that for other services as well. but why do we need two approaches to be at all explicit? Basically, if you write a WSGI application, you normally are writing non-threaded code with a shared nothing approach. Whether the WSGI app is used in a threaded apache container or a gevent style uWSGI container is a deployment option. This shouldn’t be exposed in the code. Thanks, Roman On Thu, Jan 29, 2015 at 5:54 PM, Ed Leafe e...@leafe.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/28/2015 06:57 PM, Johannes Erdfelt wrote: Not sure if it helps more than this explanation, but there was a blueprint and accompanying wiki page that explains the move from twisted to eventlet: Yeah, it was never threads vs. greenthreads. There was a lot of pushback to relying on Twisted, which many people found confusing to use, and more importantly, to follow when reading code. Whatever the performance difference may be, eventlet code is a lot easier to follow, as it more closely resembles single-threaded linear execution. - -- - -- Ed Leafe -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) iQIcBAEBAgAGBQJUyle8AAoJEKMgtcocwZqLRX4P/j3LhEhubBOJmWepfO6KrG9V KxltAshKRvZ9OAKlezprZh8N5XUcxRTDxLhxkYP6Qsaf1pxHacAIOhLRobnV5Y+A oF6E6eRORs13pUhLkL+EzZ07Kon+SjSmvDDZiIo+xe8OTbgfMpo5j7zMPeLJpcr0 RUSS+knJ/ewNCMaX4gTwTY3sDYFZTbVuGHFUtjSgeoJP0T5aP05UR73xeT7/AsbW O8dOL4tX+6GcxIHyX4XdFH9hng1P2vBZJ5l8yV6BxB6U8xsiQjlsCpwrb8orYJ0r f7+YW0D0FHOvY/TV4dzsLC/2NGc2AwMszWL3kB/AzbUuDyMMDEGpbAS/VHDcyhZg l7zFKEQy+9UybVzWjw764hpzcUT/ICPbTBiX/QuN4qY9YTUNTlCNrRAslgM+cr2y x0/nb6cd+Qq21RPIygJ9HavRqOm8npF6HpUrE55Dn+3/OvfAftlWNPcdlXAjtDOt 4WUFUoZjUTsNUjLlEiiTzgfJg7+eQqbR/HFubCpILFQgOlOAuZIDcr3g8a3yw7ED wt5UZz/89LDQqpF2TZX8lKFWxeKk1CnxWEWO208+E/700JS4xKHpnVi4tj18udsY AHnihUwGO4d9Q0i+TqbomnzyqOW6SM+gDUcahfJ92IJj9e13bqpbuoN/YMbpD/o8 evOz8G3OeC/KaOgG5F/1 =1M8U -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Jeremy, I don't have exact numbers, so yeah, it's just an assumption based on looking at the nova-api/scheduler logs with connection_debug set to 100. But that's a good point you are making here: it will be interesting to see what difference enabling of PyMySQL will make for tempest/rally workloads, rather than just running synthetic tests. I'm going to give it a try on my devstack installation. Thanks, Roman On Thu, Jan 29, 2015 at 6:42 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-01-29 18:35:20 +0200 (+0200), Roman Podoliaka wrote: [...] Otherwise, PyMySQL would be much slower than MySQL-Python for the typical SQL queries we do (e.g. ask for *a lot* of data from the DB). [...] Is this assertion based on representative empirical testing (for example profiling devstack+tempest, or perhaps comparing rally benchmarks), or merely an assumption which still needs validating? -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Thursday, January 29, 2015, Roman Podoliaka rpodoly...@mirantis.com wrote: Hi all, Mike, thanks for summarizing this in https://wiki.openstack.org/wiki/PyMySQL_evaluation ! On PyMySQL: this is something we need to enable testing of oslo.db on Python 3.x and PyPy. Though, I doubt we want to make PyMySQL the default DB API driver for OpenStack services for Python 2.x. At least, not until PyMySQL provides C-speedups for hot spots in the driver code (I assume this can be done in eventlet/PyPy friendly way using cffi). Otherwise, PyMySQL would be much slower than MySQL-Python for the typical SQL queries we do (e.g. ask for *a lot* of data from the DB). On native threads vs green threads: I very much like the Keystone approach, which allows to run the service using either eventlet or Apache. It would be great, if we could do that for other services as well. I have heard that a few of the very large deployments out there use Apache or nginx and mod_wsgi / equiv for nginx for all of the services. I personally think this deployment model is the way I'd like to see all OpenStack API services go. For cases like conductor or nova compute it may not be as good of a deployment option (not exactly traditional wsgi apps). --Morgan Thanks, Roman On Thu, Jan 29, 2015 at 5:54 PM, Ed Leafe e...@leafe.com javascript:; wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/28/2015 06:57 PM, Johannes Erdfelt wrote: Not sure if it helps more than this explanation, but there was a blueprint and accompanying wiki page that explains the move from twisted to eventlet: Yeah, it was never threads vs. greenthreads. There was a lot of pushback to relying on Twisted, which many people found confusing to use, and more importantly, to follow when reading code. Whatever the performance difference may be, eventlet code is a lot easier to follow, as it more closely resembles single-threaded linear execution. - -- - -- Ed Leafe -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) iQIcBAEBAgAGBQJUyle8AAoJEKMgtcocwZqLRX4P/j3LhEhubBOJmWepfO6KrG9V KxltAshKRvZ9OAKlezprZh8N5XUcxRTDxLhxkYP6Qsaf1pxHacAIOhLRobnV5Y+A oF6E6eRORs13pUhLkL+EzZ07Kon+SjSmvDDZiIo+xe8OTbgfMpo5j7zMPeLJpcr0 RUSS+knJ/ewNCMaX4gTwTY3sDYFZTbVuGHFUtjSgeoJP0T5aP05UR73xeT7/AsbW O8dOL4tX+6GcxIHyX4XdFH9hng1P2vBZJ5l8yV6BxB6U8xsiQjlsCpwrb8orYJ0r f7+YW0D0FHOvY/TV4dzsLC/2NGc2AwMszWL3kB/AzbUuDyMMDEGpbAS/VHDcyhZg l7zFKEQy+9UybVzWjw764hpzcUT/ICPbTBiX/QuN4qY9YTUNTlCNrRAslgM+cr2y x0/nb6cd+Qq21RPIygJ9HavRqOm8npF6HpUrE55Dn+3/OvfAftlWNPcdlXAjtDOt 4WUFUoZjUTsNUjLlEiiTzgfJg7+eQqbR/HFubCpILFQgOlOAuZIDcr3g8a3yw7ED wt5UZz/89LDQqpF2TZX8lKFWxeKk1CnxWEWO208+E/700JS4xKHpnVi4tj18udsY AHnihUwGO4d9Q0i+TqbomnzyqOW6SM+gDUcahfJ92IJj9e13bqpbuoN/YMbpD/o8 evOz8G3OeC/KaOgG5F/1 =1M8U -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Mike, I can't agree more: as far as we are concerned, every service is yet another WSGI app. And it should be left up to operator, how to deploy it. So 'green thread awareness' (i.e. patching of the world) should go to separate keystone|*-eventlet binary, while everyone else will still be able to use it as a general WSGI app. Thanks, Roman On Thu, Jan 29, 2015 at 6:55 PM, Mike Bayer mba...@redhat.com wrote: Roman Podoliaka rpodoly...@mirantis.com wrote: On native threads vs green threads: I very much like the Keystone approach, which allows to run the service using either eventlet or Apache. It would be great, if we could do that for other services as well. but why do we need two approaches to be at all explicit? Basically, if you write a WSGI application, you normally are writing non-threaded code with a shared nothing approach. Whether the WSGI app is used in a threaded apache container or a gevent style uWSGI container is a deployment option. This shouldn’t be exposed in the code. Thanks, Roman On Thu, Jan 29, 2015 at 5:54 PM, Ed Leafe e...@leafe.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/28/2015 06:57 PM, Johannes Erdfelt wrote: Not sure if it helps more than this explanation, but there was a blueprint and accompanying wiki page that explains the move from twisted to eventlet: Yeah, it was never threads vs. greenthreads. There was a lot of pushback to relying on Twisted, which many people found confusing to use, and more importantly, to follow when reading code. Whatever the performance difference may be, eventlet code is a lot easier to follow, as it more closely resembles single-threaded linear execution. - -- - -- Ed Leafe -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) iQIcBAEBAgAGBQJUyle8AAoJEKMgtcocwZqLRX4P/j3LhEhubBOJmWepfO6KrG9V KxltAshKRvZ9OAKlezprZh8N5XUcxRTDxLhxkYP6Qsaf1pxHacAIOhLRobnV5Y+A oF6E6eRORs13pUhLkL+EzZ07Kon+SjSmvDDZiIo+xe8OTbgfMpo5j7zMPeLJpcr0 RUSS+knJ/ewNCMaX4gTwTY3sDYFZTbVuGHFUtjSgeoJP0T5aP05UR73xeT7/AsbW O8dOL4tX+6GcxIHyX4XdFH9hng1P2vBZJ5l8yV6BxB6U8xsiQjlsCpwrb8orYJ0r f7+YW0D0FHOvY/TV4dzsLC/2NGc2AwMszWL3kB/AzbUuDyMMDEGpbAS/VHDcyhZg l7zFKEQy+9UybVzWjw764hpzcUT/ICPbTBiX/QuN4qY9YTUNTlCNrRAslgM+cr2y x0/nb6cd+Qq21RPIygJ9HavRqOm8npF6HpUrE55Dn+3/OvfAftlWNPcdlXAjtDOt 4WUFUoZjUTsNUjLlEiiTzgfJg7+eQqbR/HFubCpILFQgOlOAuZIDcr3g8a3yw7ED wt5UZz/89LDQqpF2TZX8lKFWxeKk1CnxWEWO208+E/700JS4xKHpnVi4tj18udsY AHnihUwGO4d9Q0i+TqbomnzyqOW6SM+gDUcahfJ92IJj9e13bqpbuoN/YMbpD/o8 evOz8G3OeC/KaOgG5F/1 =1M8U -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Wed, Jan 28, 2015, at 07:57 PM, Johannes Erdfelt wrote: On Wed, Jan 28, 2015, Vishvananda Ishaya vishvana...@gmail.com wrote: On Jan 28, 2015, at 4:03 PM, Doug Hellmann d...@doughellmann.com wrote: I hope someone who was around at the time will chime in with more detail about why green threads were deemed better than regular threads, and I look forward to seeing your analysis of a change. There is already a thread-based executor in oslo.messaging, which *should* be usable in the applications when you remove eventlet. Threading was never really considered. The initial version tried to get a working api server up as quickly as possible and it used tonado. This was quickly replaced with twisted since tornado was really new at the time and had bugs. We then switched to eventlet when swift joined the party so we didn’t have multiple concurrency stacks. By the time someone came up with the idea of using different concurrency models for the api server and the backend services, we were already pretty far down the greenthread path. Not sure if it helps more than this explanation, but there was a blueprint and accompanying wiki page that explains the move from twisted to eventlet: https://blueprints.launchpad.net/nova/+spec/unified-service-architecture https://wiki.openstack.org/wiki/UnifiedServiceArchitecture Thanks Vish Johannes, that's good background. There are a lot of factors to consider, and having a full picture of the history of how we got to where we are will help us make informed decisions about any proposed changes. Doug JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On 01/29/2015 05:57 PM, Roman Podoliaka wrote: Jeremy, I don't have exact numbers, so yeah, it's just an assumption based on looking at the nova-api/scheduler logs with connection_debug set to 100. But that's a good point you are making here: it will be interesting to see what difference enabling of PyMySQL will make for tempest/rally workloads, rather than just running synthetic tests. I'm going to give it a try on my devstack installation. Yeah, also realistic testing would mean a decent level of parallelism in requests. If you compare serial scenarios, then yes, of course mysqldb-python will be quicker. /Ihar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Jan 29, 2015, at 8:57 AM, Roman Podoliaka rpodoly...@mirantis.com wrote: Jeremy, I don't have exact numbers, so yeah, it's just an assumption based on looking at the nova-api/scheduler logs with connection_debug set to 100. But that's a good point you are making here: it will be interesting to see what difference enabling of PyMySQL will make for tempest/rally workloads, rather than just running synthetic tests. I'm going to give it a try on my devstack installation. FWIW I tested this a while ago on some perf tests on nova and cinder that we run internally and I found pymysql to be slower by about 10%. It appears that we were cpu bound in python more often than we were blocking talking to the db. I do recall someone doing a similar test in neutron saw some speedup, however. On our side we also exposed a few race conditions which made it less stable. We hit a few hard deadlocks in volume create IIRC. I don’t think switching is going to give us much benefit right away. We will need a few optimizations and bugfixes in other areas (particularly in our sqlalchemy usage) before we will derive any benefit from the switch. Vish Thanks, Roman On Thu, Jan 29, 2015 at 6:42 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-01-29 18:35:20 +0200 (+0200), Roman Podoliaka wrote: [...] Otherwise, PyMySQL would be much slower than MySQL-Python for the typical SQL queries we do (e.g. ask for *a lot* of data from the DB). [...] Is this assertion based on representative empirical testing (for example profiling devstack+tempest, or perhaps comparing rally benchmarks), or merely an assumption which still needs validating? -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Excerpts from Vishvananda Ishaya's message of 2015-01-29 10:21:58 -0800: On Jan 29, 2015, at 8:57 AM, Roman Podoliaka rpodoly...@mirantis.com wrote: Jeremy, I don't have exact numbers, so yeah, it's just an assumption based on looking at the nova-api/scheduler logs with connection_debug set to 100. But that's a good point you are making here: it will be interesting to see what difference enabling of PyMySQL will make for tempest/rally workloads, rather than just running synthetic tests. I'm going to give it a try on my devstack installation. FWIW I tested this a while ago on some perf tests on nova and cinder that we run internally and I found pymysql to be slower by about 10%. It appears that we were cpu bound in python more often than we were blocking talking to the db. I do recall someone doing a similar test in neutron saw some speedup, however. On our side we also exposed a few race conditions which made it less stable. We hit a few hard deadlocks in volume create IIRC. I don’t think switching is going to give us much benefit right away. We will need a few optimizations and bugfixes in other areas (particularly in our sqlalchemy usage) before we will derive any benefit from the switch. No magic bullets, right? I think we can all resolve this statement in our heads though: fast and never concurrent will eventually lose to concurrent and potentially fast with optimizations. The question is, how long does the hare (python-mysqldb) have to sleep before the tortoise (PyMySQL) wins? Right now it's still a close race, but if the tortoise even gains 10% speed, it likely becomes no contest at all. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/28/2015 06:57 PM, Johannes Erdfelt wrote: Not sure if it helps more than this explanation, but there was a blueprint and accompanying wiki page that explains the move from twisted to eventlet: Yeah, it was never threads vs. greenthreads. There was a lot of pushback to relying on Twisted, which many people found confusing to use, and more importantly, to follow when reading code. Whatever the performance difference may be, eventlet code is a lot easier to follow, as it more closely resembles single-threaded linear execution. - -- - -- Ed Leafe -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) iQIcBAEBAgAGBQJUyle8AAoJEKMgtcocwZqLRX4P/j3LhEhubBOJmWepfO6KrG9V KxltAshKRvZ9OAKlezprZh8N5XUcxRTDxLhxkYP6Qsaf1pxHacAIOhLRobnV5Y+A oF6E6eRORs13pUhLkL+EzZ07Kon+SjSmvDDZiIo+xe8OTbgfMpo5j7zMPeLJpcr0 RUSS+knJ/ewNCMaX4gTwTY3sDYFZTbVuGHFUtjSgeoJP0T5aP05UR73xeT7/AsbW O8dOL4tX+6GcxIHyX4XdFH9hng1P2vBZJ5l8yV6BxB6U8xsiQjlsCpwrb8orYJ0r f7+YW0D0FHOvY/TV4dzsLC/2NGc2AwMszWL3kB/AzbUuDyMMDEGpbAS/VHDcyhZg l7zFKEQy+9UybVzWjw764hpzcUT/ICPbTBiX/QuN4qY9YTUNTlCNrRAslgM+cr2y x0/nb6cd+Qq21RPIygJ9HavRqOm8npF6HpUrE55Dn+3/OvfAftlWNPcdlXAjtDOt 4WUFUoZjUTsNUjLlEiiTzgfJg7+eQqbR/HFubCpILFQgOlOAuZIDcr3g8a3yw7ED wt5UZz/89LDQqpF2TZX8lKFWxeKk1CnxWEWO208+E/700JS4xKHpnVi4tj18udsY AHnihUwGO4d9Q0i+TqbomnzyqOW6SM+gDUcahfJ92IJj9e13bqpbuoN/YMbpD/o8 evOz8G3OeC/KaOgG5F/1 =1M8U -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On 2015-01-29 10:21:58 -0800 (-0800), Vishvananda Ishaya wrote: [...] I don’t think switching is going to give us much benefit right away. We will need a few optimizations and bugfixes in other areas (particularly in our sqlalchemy usage) before we will derive any benefit from the switch. There are other benefits which we can potentially leverage right away though. Off the top of my head: 1. It removes yet another system library header dependency on our toolchain, simplifying testing and development environments. 2. It removes yet another blocker to running/testing under Python 3. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Thu, Jan 29, 2015 at 11:05 AM, Roman Podoliaka rpodoly...@mirantis.com wrote: Mike, I can't agree more: as far as we are concerned, every service is yet another WSGI app. And it should be left up to operator, how to deploy it. So 'green thread awareness' (i.e. patching of the world) should go to separate keystone|*-eventlet binary, This is keystone-all: http://git.openstack.org/cgit/openstack/keystone/tree/bin/keystone-all while everyone else will still be able to use it as a general WSGI app. This is httpd/keystone: http://git.openstack.org/cgit/openstack/keystone/tree/httpd/keystone.py Thanks, Roman - Brant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
The other factor I haven't seen mentioned here is that our usual eventlet+mysqldb setup can deadlock rather easily(*), resulting in the entire python process sitting idle for 30s until the mysqldb deadlock timer goes off and raises an exception. At this point (in nova at least), the request is usually retried and almost certainly succeeds. In neutron (as another example), the requests weren't so easy to retry and code examples where this was noticed had to be explicitly restructured to defer work that might yield the greenlet. ie: it's a big cost on coding, and deployments, as well as just being plain incorrect code. (*) eg: send simultaneous requests that modify the same resource. Eventually you'll have two greenlets within the same api server executing conflicting database operations. Entire python process freezes. (Note this is not a regular db transaction block, neither greenlet will ever exit their transactions without the deadlock timer). Bug here: https://bugs.launchpad.net/oslo.db/+bug/1350149 Test that reproduces the issue here: https://review.openstack.org/#/c/104436/ I'm dismayed that the conversation always skews towards discussing performance when the current choice isn't even correct yet, and that as a community we seem to be unwilling/unable to get behind what should be quite an obvious technical issue with straightforward solutions. If we absolutely can't switch to another mysql driver, another option that was suggested recently (and passes the above test) is using eventlet.monkey_patch(MySQLdb=True). I haven't done the investigation to find out why that isn't the default, or what the downsides are. This obviously doesn't help us with other factors, like python3-ness either. - Gus __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Angus Lees g...@inodes.org wrote: If we absolutely can't switch to another mysql driver, another option that was suggested recently (and passes the above test) is using eventlet.monkey_patch(MySQLdb=True). I haven't done the investigation to find out why that isn't the default, or what the downsides are. This obviously doesn't help us with other factors, like python3-ness either. it throws the MySQLdb driver into a thread pool, just like Twisted’s DBAPI adapter does: https://github.com/eventlet/eventlet/blob/master/eventlet/green/MySQLdb.py Which leads into “tpool”, which we can confirm is native threads right here: https://github.com/eventlet/eventlet/blob/master/eventlet/tpool.py#L33 So using MySQLdb patching with eventlet within database-centric processes, we gain: absolutely nothing. At the end of the day, we’re still using threads, or we’re doing it wrong.Our connection pools are configured (or more accurately, are left at SQLAlchemy’s default setting) to only allow 15 concurrent connections at a time in any case, beyond which the pool blocks on new connection attempts; we need this in any case because when running a few dozen apps/processes, each with their own pool of 5-15 connections, that adds up to many dozens against the database overall. It is not possible or even appropriate for us to get anywhere near the many dozens / hundreds of concurrent connections per-process that is really the only thing gevent/eventlet can do that regular threads cannot. The entire eventlet / async game is utterly absurd.It’s done absolutely nothing for Openstack except cause confusion, instability, and complexity. It has never been done correctly, is incorrect in all of our applications right now, and if it were done “correctly”, our code could only run slower if anything, since both the GIL and eventlet are not IO-bound but both serialize CPU through a single straw within a process. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Excerpts from Johannes Erdfelt's message of 2015-01-28 15:33:25 -0800: On Wed, Jan 28, 2015, Mike Bayer mba...@redhat.com wrote: I can envision turning this driver into a total monster, adding C-speedups where needed but without getting in the way of async patching, adding new APIs for explicit async, and everything else. However, I’ve no idea what the developers have an appetite for. This is great information. I appreciate the work on evaluating it. Can I bring up the alternative of dropping eventlet and switching to native threads? We spend a lot of time working on the various incompatibilies between eventlet and other libraries we use. It also restricts us by making it difficult to use an entire class of python modules (that use C extensions for performance, etc). I personally have spent more time than I wish to admit fixing bugs in eventlet and troubleshooting problems we've had. And it's never been clear to me why we *need* to use eventlet or green threads in general. Our modern Nova appears to only be weakly tied to eventlet and greenlet. I think we would spend less time replacing eventlet with native threads than we'll spend in the future trying to fit our code and dependencies into the eventlet shaped hole we currently have. I'm not as familiar with the code in other OpenStack projects, but from what I have seen, they appear to be similar to Nova and are only weakly tied to eventlet/greenlet. As is often the case with threading, a reason to avoid using it is that libraries often aren't able or willing to assert thread safety. That said, one way to fix that, is to fix those libraries that we do want to use, to be thread safe. :) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Wed, Jan 28, 2015, Clint Byrum cl...@fewbar.com wrote: As is often the case with threading, a reason to avoid using it is that libraries often aren't able or willing to assert thread safety. That said, one way to fix that, is to fix those libraries that we do want to use, to be thread safe. :) I floated this idea across some coworkers recently and they brought up a similar concern, which is concurrency in general, both within our code and dependencies. I can't find many places in Nova (at least) that are concurrent in the sense that one object will be used by multiple threads. nova-scheduler is likely one place. nova-compute would likely be easy to fix if there are any problems. That said, I think the only way to know for sure is to try it out and see. I'm going to hack up a proof of concept and see how difficult this will be. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Johannes Erdfelt johan...@erdfelt.com wrote: On Wed, Jan 28, 2015, Mike Bayer mba...@redhat.com wrote: I can envision turning this driver into a total monster, adding C-speedups where needed but without getting in the way of async patching, adding new APIs for explicit async, and everything else. However, I’ve no idea what the developers have an appetite for. This is great information. I appreciate the work on evaluating it. Can I bring up the alternative of dropping eventlet and switching to native threads? I'd be your best friend for life. But I don’t think you’re going to get much traction here on that. There’s three camps on this issue: the threads-are-fine camp, the “everything should be async! you should never trust threads / IO to run implicitly!” camp, and the “we need 1000 concurrent connections but don’t want to get into explicit async so let’s use eventlet/gevent”. Myself, I am in a fourth camp, use explicit async for those kinds of logic where it makes sense from a code organization point of view, and not for those where it does not. The use case of gevent/eventlet is itself in somewhat of a fringe camp, the one where you need many hundreds of concurrent database connections within one process for something (which you do not). The kinds of use cases that are very valid for explicit async are things like code that is heavy on web service requests and asynchronous message queues. However, code that deals with databases and transactions really gains nothing and loses everything by being muddied and complicated with this very awkward approach. Unfortunately, the “debate” as I continue to follow it is really not about these things. From my observations, it seems to revolve around the following points: A. async code is “faster” (it is not) B. threads are “incorrect” (they are not) C. code that performs IO without the programmer setting aside explicit directives, deferrals, and other whatnot to accommodate this event is “wrong” (it is not). As long as the debate is over these largely ephemeral assertions, which keep coming up over and over again, Openstack has stuck its feet in the mud with the eventlet thing as a kind of compromise (even though crowd C is not appeased by this). We spend a lot of time working on the various incompatibilies between eventlet and other libraries we use. It also restricts us by making it difficult to use an entire class of python modules (that use C extensions for performance, etc). I personally have spent more time than I wish to admit fixing bugs in eventlet and troubleshooting problems we've had. And it's never been clear to me why we *need* to use eventlet or green threads in general. Our modern Nova appears to only be weakly tied to eventlet and greenlet. I think we would spend less time replacing eventlet with native threads than we'll spend in the future trying to fit our code and dependencies into the eventlet shaped hole we currently have. I'm not as familiar with the code in other OpenStack projects, but from what I have seen, they appear to be similar to Nova and are only weakly tied to eventlet/greenlet. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Wed, Jan 28, 2015, Mike Bayer mba...@redhat.com wrote: I can envision turning this driver into a total monster, adding C-speedups where needed but without getting in the way of async patching, adding new APIs for explicit async, and everything else. However, I’ve no idea what the developers have an appetite for. This is great information. I appreciate the work on evaluating it. Can I bring up the alternative of dropping eventlet and switching to native threads? We spend a lot of time working on the various incompatibilies between eventlet and other libraries we use. It also restricts us by making it difficult to use an entire class of python modules (that use C extensions for performance, etc). I personally have spent more time than I wish to admit fixing bugs in eventlet and troubleshooting problems we've had. And it's never been clear to me why we *need* to use eventlet or green threads in general. Our modern Nova appears to only be weakly tied to eventlet and greenlet. I think we would spend less time replacing eventlet with native threads than we'll spend in the future trying to fit our code and dependencies into the eventlet shaped hole we currently have. I'm not as familiar with the code in other OpenStack projects, but from what I have seen, they appear to be similar to Nova and are only weakly tied to eventlet/greenlet. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Wed, Jan 28, 2015, at 06:50 PM, Johannes Erdfelt wrote: On Wed, Jan 28, 2015, Clint Byrum cl...@fewbar.com wrote: As is often the case with threading, a reason to avoid using it is that libraries often aren't able or willing to assert thread safety. That said, one way to fix that, is to fix those libraries that we do want to use, to be thread safe. :) I floated this idea across some coworkers recently and they brought up a similar concern, which is concurrency in general, both within our code and dependencies. I can't find many places in Nova (at least) that are concurrent in the sense that one object will be used by multiple threads. nova-scheduler is likely one place. nova-compute would likely be easy to fix if there are any problems. That said, I think the only way to know for sure is to try it out and see. I'm going to hack up a proof of concept and see how difficult this will be. I hope someone who was around at the time will chime in with more detail about why green threads were deemed better than regular threads, and I look forward to seeing your analysis of a change. There is already a thread-based executor in oslo.messaging, which *should* be usable in the applications when you remove eventlet. Doug JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
+1 Will be an interesting analysis; I've also never understood this desire to use eventlet (perhaps it's written somewhere?). It seems like most of the applications in openstack (maybe leaving out the WSGI entrypoints) can be scaled horizontally and mostly just do a large amount of work that blocks (and python afaik itself will swap in/out threads when this happens) on I/O, sockets, other... The highly concurrent situation seems only *slightly* applicable on the WSGI/webserver entrypoints (although this IMHO is 'meh', since we have multiple workers here anyway, and those should be scalable with new processes if needed) and maybe 'conductor' applications (although this should also be horizontally scalable using child-processes if it's really a bottleneck). -Josh Johannes Erdfelt wrote: On Wed, Jan 28, 2015, Clint Byrumcl...@fewbar.com wrote: As is often the case with threading, a reason to avoid using it is that libraries often aren't able or willing to assert thread safety. That said, one way to fix that, is to fix those libraries that we do want to use, to be thread safe. :) I floated this idea across some coworkers recently and they brought up a similar concern, which is concurrency in general, both within our code and dependencies. I can't find many places in Nova (at least) that are concurrent in the sense that one object will be used by multiple threads. nova-scheduler is likely one place. nova-compute would likely be easy to fix if there are any problems. That said, I think the only way to know for sure is to try it out and see. I'm going to hack up a proof of concept and see how difficult this will be. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Jan 28, 2015, at 4:03 PM, Doug Hellmann d...@doughellmann.com wrote: On Wed, Jan 28, 2015, at 06:50 PM, Johannes Erdfelt wrote: On Wed, Jan 28, 2015, Clint Byrum cl...@fewbar.com wrote: As is often the case with threading, a reason to avoid using it is that libraries often aren't able or willing to assert thread safety. That said, one way to fix that, is to fix those libraries that we do want to use, to be thread safe. :) I floated this idea across some coworkers recently and they brought up a similar concern, which is concurrency in general, both within our code and dependencies. I can't find many places in Nova (at least) that are concurrent in the sense that one object will be used by multiple threads. nova-scheduler is likely one place. nova-compute would likely be easy to fix if there are any problems. That said, I think the only way to know for sure is to try it out and see. I'm going to hack up a proof of concept and see how difficult this will be. I hope someone who was around at the time will chime in with more detail about why green threads were deemed better than regular threads, and I look forward to seeing your analysis of a change. There is already a thread-based executor in oslo.messaging, which *should* be usable in the applications when you remove eventlet. Threading was never really considered. The initial version tried to get a working api server up as quickly as possible and it used tonado. This was quickly replaced with twisted since tornado was really new at the time and had bugs. We then switched to eventlet when swift joined the party so we didn’t have multiple concurrency stacks. By the time someone came up with the idea of using different concurrency models for the api server and the backend services, we were already pretty far down the greenthread path. Vish Doug JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
On Wed, Jan 28, 2015, Vishvananda Ishaya vishvana...@gmail.com wrote: On Jan 28, 2015, at 4:03 PM, Doug Hellmann d...@doughellmann.com wrote: I hope someone who was around at the time will chime in with more detail about why green threads were deemed better than regular threads, and I look forward to seeing your analysis of a change. There is already a thread-based executor in oslo.messaging, which *should* be usable in the applications when you remove eventlet. Threading was never really considered. The initial version tried to get a working api server up as quickly as possible and it used tonado. This was quickly replaced with twisted since tornado was really new at the time and had bugs. We then switched to eventlet when swift joined the party so we didn’t have multiple concurrency stacks. By the time someone came up with the idea of using different concurrency models for the api server and the backend services, we were already pretty far down the greenthread path. Not sure if it helps more than this explanation, but there was a blueprint and accompanying wiki page that explains the move from twisted to eventlet: https://blueprints.launchpad.net/nova/+spec/unified-service-architecture https://wiki.openstack.org/wiki/UnifiedServiceArchitecture JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo.db] PyMySQL review
Hey list - As many are aware, we’ve been looking for the one MySQL driver to rule them all. As has been the case for some weeks now, that driver is PyMySQL, meeting all the critical requirements we have of: 1. pure python, so eventlet patchable, 2. Python 3 support 3. Is released on Pypi (which is what disqualifies MySQL-connector-Python). I have experience with PyMySQL and I was at first a little skeptical that it’s ready for openstack’s level of activity, so I offered to have oslo.db write up a full review of everything we know about all the MySQL drivers, including that I’d evaluate PyMySQL and try to put some actual clue behind my vague notions. I finally got around to writing up the code review portion, so that we at least have awareness of what we’re getting with PyMySQL. This is also a document that I’m very much hoping we can use to get the PyMySQL developers involved with. Basically PyMySQL is fine, it lacks some polish and test coverage that can certainly be added, and as far as performance, we’re going to be really disappointed with pure-Python MySQL drivers in general, though PyMySQL itself can still be improved within the realm of pure Python. It at least allows for eventlet monkey patching, so that we will for the first time be able to observe the benefits of some real database concurrency ever since Openstack decided not to use threads (if Openstack did in fact ever use normal threads…), which also means we will be able to observe the headaches of real database concurrency, especially the really scary ones I’ve already observed when working with eventlet/gevent style monkeypatching :). While PyMySQL is lacking test coverage in some areas, has no external documentation, and has at least some areas where Python performance can be improved, the basic structure of the driver is perfectly fine and straightforward. I can envision turning this driver into a total monster, adding C-speedups where needed but without getting in the way of async patching, adding new APIs for explicit async, and everything else. However, I’ve no idea what the developers have an appetite for. Please review the document at https://wiki.openstack.org/wiki/PyMySQL_evaluation. - mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] PyMySQL review
Mike Bayer mba...@redhat.com wrote: Hey list - As many are aware, we’ve been looking for the one MySQL driver to rule them all. As has been the case for some weeks now, that driver is PyMySQL, meeting all the critical requirements we have of: 1. pure python, so eventlet patchable, 2. Python 3 support 3. Is released on Pypi (which is what disqualifies MySQL-connector-Python). I have experience with PyMySQL and I was at first a little skeptical that it’s ready for openstack’s level of activity, so I offered to have oslo.db write up a full review of everything we know about all the MySQL drivers, including that I’d evaluate PyMySQL and try to put some actual clue behind my vague notions. I finally got around to writing up the code review portion, so that we at least have awareness of what we’re getting with PyMySQL. This is also a document that I’m very much hoping we can use to get the PyMySQL developers involved with. Basically PyMySQL is fine, it lacks some polish and test coverage that can certainly be added, and as far as performance, we’re going to be really disappointed with pure-Python MySQL drivers in general, though PyMySQL itself can still be improved within the realm of pure Python. It at least allows for eventlet monkey patching, so that we will for the first time be able to observe the benefits of some real database concurrency ever since Openstack decided not to use threads (if Openstack did in fact ever use normal threads…), which also means we will be able to observe the headaches of real database concurrency, especially the really scary ones I’ve already observed when working with eventlet/gevent style monkeypatching :). While PyMySQL is lacking test coverage in some areas, has no external documentation, and has at least some areas where Python performance can be improved, the basic structure of the driver is perfectly fine and straightforward. I can envision turning this driver into a total monster, adding C-speedups where needed but without getting in the way of async patching, adding new APIs for explicit async, and everything else. However, I’ve no idea what the developers have an appetite for. Please review the document at https://wiki.openstack.org/wiki/PyMySQL_evaluation. I’ve also landed onto planet PyMySQL, walked down the gangway to the planet’s surface, and offered greetings and souvenirs from planet Openstack: https://groups.google.com/forum/#!topic/pymysql-users/LfLBD1zcMpY. I now await the response of PyMySQL’s leaders, whether it be offers of peace or an armada of laser cannons (or just the so-typical crickets chirping), to our kind world’s offer of friendship. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
