Hi Ihar, AFAIU, the switch is a matter of pip install + specifying the correct db URI in the config files. I'm not sure why you are filing a spec in Neutron project. IMHO, this has nothing to do with projects, but rather a purely deployment question. E.g. don't we have PostgreSQL+psycopg2 or MySQL+pymysql deployments of OpenStack right now?
I think what you really want is to change the defaults we test in the gate, which is a different problem. Thanks, Roman On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka <ihrac...@redhat.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi all, > > Multiple projects are suffering from db lock timeouts due to deadlocks > deep in mysqldb library that we use to interact with mysql servers. In > essence, the problem is due to missing eventlet support in mysqldb > module, meaning when a db lock is encountered, the library does not > yield to the next green thread, allowing other threads to eventually > unlock the grabbed lock, and instead it just blocks the main thread, > that eventually raises timeout exception (OperationalError). > > The failed operation is not retried, leaving failing request not > served. In Nova, there is a special retry mechanism for deadlocks, > though I think it's more a hack than a proper fix. > > Neutron is one of the projects that suffer from those timeout errors a > lot. Partly it's due to lack of discipline in how we do nested calls > in l3_db and ml2_plugin code, but that's not something to change in > foreseeable future, so we need to find another solution that is > applicable for Juno. Ideally, the solution should be applicable for > Icehouse too to allow distributors to resolve existing deadlocks > without waiting for Juno. > > We've had several discussions and attempts to introduce a solution to > the problem. Thanks to oslo.db guys, we now have more or less clear > view on the cause of the failures and how to easily fix them. The > solution is to switch mysqldb to something eventlet aware. The best > candidate is probably MySQL Connector module that is an official MySQL > client for Python and that shows some (preliminary) good results in > terms of performance. > > I've posted a Neutron spec for the switch to the new client in Juno at > [1]. Ideally, switch is just a matter of several fixes to oslo.db that > would enable full support for the new driver already supported by > SQLAlchemy, plus 'connection' string modified in service configuration > files, plus documentation updates to refer to the new official way to > configure services for MySQL. The database code won't, ideally, > require any major changes, though some adaptation for the new client > library may be needed. That said, Neutron does not seem to require any > changes, though it was revealed that there are some alembic migration > rules in Keystone or Glance that need (trivial) modifications. > > You can see how trivial the switch can be achieved for a service based > on example for Neutron [2]. > > While this is a Neutron specific proposal, there is an obvious wish to > switch to the new library globally throughout all the projects, to > reduce devops burden, among other things. My vision is that, ideally, > we switch all projects to the new library in Juno, though we still may > leave several projects for K in case any issues arise, similar to the > way projects switched to oslo.messaging during two cycles instead of > one. Though looking at how easy Neutron can be switched to the new > library, I wouldn't expect any issues that would postpone the switch > till K. > > It was mentioned in comments to the spec proposal that there were some > discussions at the latest summit around possible switch in context of > Nova that revealed some concerns, though they do not seem to be > documented anywhere. So if you know anything about it, please comment. > > So, we'd like to hear from other projects what's your take on that > move, whether you see any issues or have concerns about it. > > Thanks for your comments, > /Ihar > > [1]: https://review.openstack.org/#/c/104905/ > [2]: https://review.openstack.org/#/c/105209/ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBCgAGBQJTvSS+AAoJEC5aWaUY1u57uk0IAMNIW4e59fU8uiF7eg8KwIgU > 5vjzDP4GX454Oxm0h5q3Olc0nXIeB6zSBGDoomgLk9+4AS250ihGRA/V10iDEJTF > yubcvknep/ZfF+lKkmgBA3WNXJgTXffXeN2bimC6t5zJA+8Cmn3lUPu0djt0GWs7 > AktufkPbVj7ZauN6w9OpW4AnoZX1fARvynCilTuHYu+lb8nQ/Hatqu5dgqdeDyRp > jodLoN1ow3VYR7Cq5jocqhw719aiKLJdlUgWVHNL5A5oTR1Uxu0AdleeUzXVUvFm > +EQO0Xe+slMSBgzBsgPJAiX0Vkc6kfJdFHR571QUWCXaXF1nUEIkgra/7j+0Uqs= > =bgds > -----END PGP SIGNATURE----- > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev