-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 14/07/14 15:54, Clark Boylan wrote: > On Sun, Jul 13, 2014 at 9:20 AM, Ihar Hrachyshka > <ihrac...@redhat.com> wrote: On 11/07/14 19:20, Clark Boylan > wrote: >>>> Before we get too far ahead of ourselves mysql-connector is >>>> not hosted on pypi. Instead it is an external package link. >>>> We recently managed to remove all packages that are hosted as >>>> external package links from openstack and will not add new >>>> ones in. Before we can use mysql-connector in the gate oracle >>>> will need to publish mysql-connector on pypi properly. > > There is misunderstanding in our community on how we deploy db > client modules. No project actually depends on any of them. We > assume deployers will install the proper one and configure > 'connection' string to use it. In case of devstack, we install the > appropriate package from distribution packages, not pip. > >> Correct, but for all of the other test suites (unittests) we >> install the db clients via pip because tox runs them and >> virtualenvs allowing site packages cause too many problems. See >> https://git.openstack.org/cgit/openstack/nova/tree/test-requirements.txt#n8. >> >> So we do actually depend on these things being pip installable. >> Basically this allows devs to run `tox` and it works.
Roger that, and thanks for clarification. I'm trying to reach the author and the maintainer of mysqlconnector-python to see whether I'll be able to convince him to publish the packages on pypi.python.org. > >> I would argue that we should have devstack install via pip too >> for consistency, but that is a different issue (it is already >> installing all of the other python dependencies this way so why >> special case?). > > What we do is recommending a module for our users in our > documentation. > > That said, I assume the gate is a non-issue. Correct? > >>>> >>>> That said there is at least one other pure python >>>> alternative, PyMySQL. PyMySQL supports py3k and pypy. We >>>> should look at using PyMySQL instead if we want to start with >>>> a reasonable path to getting this in the gate. > > MySQL Connector supports py3k too (not sure about pypy though). > >>>> >>>> Clark >>>> >>>> On Fri, Jul 11, 2014 at 10:07 AM, Miguel Angel Ajo Pelayo >>>> <mangel...@redhat.com> wrote: >>>>> +1 here too, >>>>> >>>>> Amazed with the performance gains, x2.4 seems a lot, and >>>>> we'd get rid of deadlocks. >>>>> >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> +1 >>>>>> >>>>>> I'm pretty excited about the possibilities here. I've >>>>>> had this mysqldb/eventlet contention in the back of my >>>>>> mind for some time now. I'm glad to see some work being >>>>>> done in this area. >>>>>> >>>>>> Carl >>>>>> >>>>>> On Fri, Jul 11, 2014 at 7:04 AM, Ihar Hrachyshka >>>>>> <ihrac...@redhat.com> wrote: >>>> On 09/07/14 13:17, Ihar Hrachyshka wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Multiple projects are suffering from db lock >>>>>>>>> timeouts due to deadlocks deep in mysqldb library >>>>>>>>> that we use to interact with mysql servers. In >>>>>>>>> essence, the problem is due to missing eventlet >>>>>>>>> support in mysqldb module, meaning when a db lock >>>>>>>>> is encountered, the library does not yield to the >>>>>>>>> next green thread, allowing other threads to >>>>>>>>> eventually unlock the grabbed lock, and instead it >>>>>>>>> just blocks the main thread, that eventually raises >>>>>>>>> timeout exception (OperationalError). >>>>>>>>> >>>>>>>>> The failed operation is not retried, leaving >>>>>>>>> failing request not served. In Nova, there is a >>>>>>>>> special retry mechanism for deadlocks, though I >>>>>>>>> think it's more a hack than a proper fix. >>>>>>>>> >>>>>>>>> Neutron is one of the projects that suffer from >>>>>>>>> those timeout errors a lot. Partly it's due to lack >>>>>>>>> of discipline in how we do nested calls in l3_db >>>>>>>>> and ml2_plugin code, but that's not something to >>>>>>>>> change in foreseeable future, so we need to find >>>>>>>>> another solution that is applicable for Juno. >>>>>>>>> Ideally, the solution should be applicable for >>>>>>>>> Icehouse too to allow distributors to resolve >>>>>>>>> existing deadlocks without waiting for Juno. >>>>>>>>> >>>>>>>>> We've had several discussions and attempts to >>>>>>>>> introduce a solution to the problem. Thanks to >>>>>>>>> oslo.db guys, we now have more or less clear view >>>>>>>>> on the cause of the failures and how to easily fix >>>>>>>>> them. The solution is to switch mysqldb to >>>>>>>>> something eventlet aware. The best candidate is >>>>>>>>> probably MySQL Connector module that is an >>>>>>>>> official MySQL client for Python and that shows >>>>>>>>> some (preliminary) good results in terms of >>>>>>>>> performance. >>>> >>>> I've made additional testing, creating 2000 networks in >>>> parallel (10 thread workers) for both drivers and comparing >>>> results. >>>> >>>> With mysqldb: 215.81 sec With mysql-connector: 88.66 >>>> >>>> ~2.4 times performance boost, ok? ;) >>>> >>>> I think we should switch to that library *even* if we forget >>>> about all the nasty deadlocks we experience now. >>>> >>>>>>>>> >>>>>>>>> I've posted a Neutron spec for the switch to the >>>>>>>>> new client in Juno at [1]. Ideally, switch is just >>>>>>>>> a matter of several fixes to oslo.db that would >>>>>>>>> enable full support for the new driver already >>>>>>>>> supported by SQLAlchemy, plus 'connection' string >>>>>>>>> modified in service configuration files, plus >>>>>>>>> documentation updates to refer to the new official >>>>>>>>> way to configure services for MySQL. The database >>>>>>>>> code won't, ideally, require any major changes, >>>>>>>>> though some adaptation for the new client library >>>>>>>>> may be needed. That said, Neutron does not seem to >>>>>>>>> require any changes, though it was revealed that >>>>>>>>> there are some alembic migration rules in Keystone >>>>>>>>> or Glance that need (trivial) modifications. >>>>>>>>> >>>>>>>>> You can see how trivial the switch can be achieved >>>>>>>>> for a service based on example for Neutron [2]. >>>>>>>>> >>>>>>>>> While this is a Neutron specific proposal, there is >>>>>>>>> an obvious wish to switch to the new library >>>>>>>>> globally throughout all the projects, to reduce >>>>>>>>> devops burden, among other things. My vision is >>>>>>>>> that, ideally, we switch all projects to the new >>>>>>>>> library in Juno, though we still may leave several >>>>>>>>> projects for K in case any issues arise, similar to >>>>>>>>> the way projects switched to oslo.messaging during >>>>>>>>> two cycles instead of one. Though looking at how >>>>>>>>> easy Neutron can be switched to the new library, I >>>>>>>>> wouldn't expect any issues that would postpone the >>>>>>>>> switch till K. >>>>>>>>> >>>>>>>>> It was mentioned in comments to the spec proposal >>>>>>>>> that there were some discussions at the latest >>>>>>>>> summit around possible switch in context of Nova >>>>>>>>> that revealed some concerns, though they do not >>>>>>>>> seem to be documented anywhere. So if you know >>>>>>>>> anything about it, please comment. >>>>>>>>> >>>>>>>>> So, we'd like to hear from other projects what's >>>>>>>>> your take on that move, whether you see any issues >>>>>>>>> or have concerns about it. >>>>>>>>> >>>>>>>>> Thanks for your comments, /Ihar >>>>>>>>> >>>>>>>>> [1]: https://review.openstack.org/#/c/104905/ [2]: >>>>>>>>> https://review.openstack.org/#/c/105209/ >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> OpenStack-dev mailing list >>>>>>>>> OpenStack-dev@lists.openstack.org >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>> > >>>>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> OpenStack-dev@lists.openstack.org >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>>>> > >>>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> OpenStack-dev@lists.openstack.org >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>> >>>>> >>>>>> > >>>>>> _______________________________________________ >>>>> OpenStack-dev mailing list >>>>> OpenStack-dev@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>>> >>>> >>>>> _______________________________________________ OpenStack-dev >>>> mailing list OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >> >> >>>> _______________________________________________ >> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > _______________________________________________ OpenStack-dev > mailing list OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCgAGBQJTw/EmAAoJEC5aWaUY1u57dYoH/it9kTKsjYiI/8hWLwr08ezt Lk7xwm2mWvTjWSsT3eZypAupIS4BR1t6/jJyGQO2YD1dI2IwmEsV7MZnoE9Cl7Iz XtKVi+PGMMnVo6XvLej75BbFlXccIGo5AYBGqQ87JEiktzMxfFyubsARv8vu6MQp O1PinoCj9dCfTl8tKu+RzbcoSC9aFN07vXbRnh1ouzBVpK2Ps/oBlWzusawaUsgI 8mYaCa+hn5z10PLalxaDn5PaJbk0RsGLr2eLXABRzE6qGYzYyjfJ343AIUbUbz/4 TUTXwnVSZrJ8TUgpU+nJkrWORkwC3rIzfKM+tHlEeA2BaxyAsyaf6zYk8CN3QGs= =LGh/ -----END PGP SIGNATURE----- _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev