On 7/9/14, 10:59 AM, Roman Podoliaka wrote: > Hi all, > > Not sure what issues you are talking about, but I just replaced > "mysql" with "mysql+mysqlconnector" in my db connection string in > neutron.conf and "neutron-db-manage upgrade head" worked like a charm > for an empty schema. > > Ihar, could please elaborate on what changes to oslo.db are needed? > (as an oslo.db developer I'm very interested in this part :) ) > > Thanks, > Roman > > On Wed, Jul 9, 2014 at 5:43 PM, Ihar Hrachyshka <ihrac...@redhat.com> wrote: > On 09/07/14 15:40, Sean Dague wrote: > >>> On 07/09/2014 09:00 AM, Roman Podoliaka wrote: > >>>> Hi Ihar, > >>>> > >>>> AFAIU, the switch is a matter of pip install + specifying the > >>>> correct db URI in the config files. I'm not sure why you are > >>>> filing a spec in Neutron project. IMHO, this has nothing to do > >>>> with projects, but rather a purely deployment question. E.g. > >>>> don't we have PostgreSQL+psycopg2 or MySQL+pymysql deployments of > >>>> OpenStack right now? > >>>> > >>>> I think what you really want is to change the defaults we test in > >>>> the gate, which is a different problem. > >>> > >>> Because this is really a *new* driver. As you can see by the > >>> attempted run, it doesn't work with alembic given the definitions > >>> that neutron has. So it's not like this is currently compatible > >>> with OpenStack code. > > Well, to fix that, you just need to specify raise_on_warnings=False > for connection (it's default for mysqldb but not mysql-connector). > I've done it in devstack patch for now, but probably it belongs to
this is also semi-my fault as mysqlconnector apparently defaults this to False now, but for some reason the SQLAlchemy mysqlconnector dialect is flipping it to True (this dialect was contributed by MySQL-connector's folks, so not sure why the inconsistency, perhaps they changed their minds) > oslo.db. > > >>> > >>>> > >>>> Thanks, Roman > >>>> > >>>> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka > >>>> <ihrac...@redhat.com> wrote: Hi all, > >>>> > >>>> Multiple projects are suffering from db lock timeouts due to > >>>> deadlocks deep in mysqldb library that we use to interact with > >>>> mysql servers. In essence, the problem is due to missing eventlet > >>>> support in mysqldb module, meaning when a db lock is encountered, > >>>> the library does not yield to the next green thread, allowing > >>>> other threads to eventually unlock the grabbed lock, and instead > >>>> it just blocks the main thread, that eventually raises timeout > >>>> exception (OperationalError). > >>>> > >>>> The failed operation is not retried, leaving failing request not > >>>> served. In Nova, there is a special retry mechanism for > >>>> deadlocks, though I think it's more a hack than a proper fix. > >>>> > >>>> Neutron is one of the projects that suffer from those timeout > >>>> errors a lot. Partly it's due to lack of discipline in how we do > >>>> nested calls in l3_db and ml2_plugin code, but that's not > >>>> something to change in foreseeable future, so we need to find > >>>> another solution that is applicable for Juno. Ideally, the > >>>> solution should be applicable for Icehouse too to allow > >>>> distributors to resolve existing deadlocks without waiting for > >>>> Juno. > >>>> > >>>> We've had several discussions and attempts to introduce a > >>>> solution to the problem. Thanks to oslo.db guys, we now have more > >>>> or less clear view on the cause of the failures and how to easily > >>>> fix them. The solution is to switch mysqldb to something eventlet > >>>> aware. The best candidate is probably MySQL Connector module that > >>>> is an official MySQL client for Python and that shows some > >>>> (preliminary) good results in terms of performance. > >>>> > >>>> I've posted a Neutron spec for the switch to the new client in > >>>> Juno at [1]. Ideally, switch is just a matter of several fixes to > >>>> oslo.db that would enable full support for the new driver already > >>>> supported by SQLAlchemy, plus 'connection' string modified in > >>>> service configuration files, plus documentation updates to refer > >>>> to the new official way to configure services for MySQL. The > >>>> database code won't, ideally, require any major changes, though > >>>> some adaptation for the new client library may be needed. That > >>>> said, Neutron does not seem to require any changes, though it was > >>>> revealed that there are some alembic migration rules in Keystone > >>>> or Glance that need (trivial) modifications. > >>>> > >>>> You can see how trivial the switch can be achieved for a service > >>>> based on example for Neutron [2]. > >>>> > >>>> While this is a Neutron specific proposal, there is an obvious > >>>> wish to switch to the new library globally throughout all the > >>>> projects, to reduce devops burden, among other things. My vision > >>>> is that, ideally, we switch all projects to the new library in > >>>> Juno, though we still may leave several projects for K in case > >>>> any issues arise, similar to the way projects switched to > >>>> oslo.messaging during two cycles instead of one. Though looking > >>>> at how easy Neutron can be switched to the new library, I > >>>> wouldn't expect any issues that would postpone the switch till > >>>> K. > >>>> > >>>> It was mentioned in comments to the spec proposal that there were > >>>> some discussions at the latest summit around possible switch in > >>>> context of Nova that revealed some concerns, though they do not > >>>> seem to be documented anywhere. So if you know anything about it, > >>>> please comment. > >>>> > >>>> So, we'd like to hear from other projects what's your take on > >>>> that move, whether you see any issues or have concerns about it. > >>>> > >>>> Thanks for your comments, /Ihar > >>>> > >>>> [1]: https://review.openstack.org/#/c/104905/ [2]: > >>>> https://review.openstack.org/#/c/105209/ > >>>>> > >>>>> _______________________________________________ OpenStack-dev > >>>>> mailing list OpenStack-dev@lists.openstack.org > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >>>> > >>>>> > _______________________________________________ > >>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >>> > >>>> > >>> > >>> > >>> _______________________________________________ OpenStack-dev > >>> mailing list OpenStack-dev@lists.openstack.org > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev