-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 12/07/14 02:49, Jay Pipes wrote: > On 07/11/2014 08:04 AM, Ihar Hrachyshka wrote: On 09/07/14 13:17, > Ihar Hrachyshka wrote: >>>> Hi all, >>>> >>>> Multiple projects are suffering from db lock timeouts due to >>>> deadlocks deep in mysqldb library that we use to interact >>>> with mysql servers. In essence, the problem is due to missing >>>> eventlet support in mysqldb module, meaning when a db lock is >>>> encountered, the library does not yield to the next green >>>> thread, allowing other threads to eventually unlock the >>>> grabbed lock, and instead it just blocks the main thread, >>>> that eventually raises timeout exception (OperationalError). >>>> >>>> The failed operation is not retried, leaving failing request >>>> not served. In Nova, there is a special retry mechanism for >>>> deadlocks, though I think it's more a hack than a proper >>>> fix. >>>> >>>> Neutron is one of the projects that suffer from those >>>> timeout errors a lot. Partly it's due to lack of discipline >>>> in how we do nested calls in l3_db and ml2_plugin code, but >>>> that's not something to change in foreseeable future, so we >>>> need to find another solution that is applicable for Juno. >>>> Ideally, the solution should be applicable for Icehouse too >>>> to allow distributors to resolve existing deadlocks without >>>> waiting for Juno. >>>> >>>> We've had several discussions and attempts to introduce a >>>> solution to the problem. Thanks to oslo.db guys, we now have >>>> more or less clear view on the cause of the failures and how >>>> to easily fix them. The solution is to switch mysqldb to >>>> something eventlet aware. The best candidate is probably >>>> MySQL Connector module that is an official MySQL client for >>>> Python and that shows some (preliminary) good results in >>>> terms of performance. > > I've made additional testing, creating 2000 networks in parallel > (10 thread workers) for both drivers and comparing results. > > With mysqldb: 215.81 sec With mysql-connector: 88.66 > > ~2.4 times performance boost, ok? ;) > >> That really doesn't tell me much. > >> Please remember that performance != scalability.
I actually agree. I need to spend more time preparing benchmarks (I'll spend a part of my next week on this). > >> If you showed the test/benchmark code, that would be great. You >> need to Here is a gist: https://gist.github.com/booxter/c4f3e743a2573ba7809f >> run your benchmarks at varying levels of concurrency and varying >> levels of read/write ratios for the workers. Otherwise it's like >> looking at a a single dot of paint on a painting. Without looking >> at the patterns of throughput (performance) and >> concurrency/locking (scalability) with various levels of workers >> and read/write ratios, you miss the whole picture. Good point. I surely need to check read operations, and play with multiple API/RPC workers. Before I go with implementing my own tests, do you have more formal requirements or suggestions on which data points we can be interested in? > >> Another thing to ensure is that you are using real *processes*, >> not threads, so that you actually simulate a real OpenStack >> service like Nova or Neutron, which are multi-plexed, not >> multi-threaded, and have a greenlet pool within each worker >> process. My test is an actual neutron client that issues full blown REST requests to a local neutron server, in multiple thread workers. I think this is ok, right? > >> Best -jay > > I think we should switch to that library *even* if we forget about > all the nasty deadlocks we experience now. > >>>> >>>> I've posted a Neutron spec for the switch to the new client >>>> in Juno at [1]. Ideally, switch is just a matter of several >>>> fixes to oslo.db that would enable full support for the new >>>> driver already supported by SQLAlchemy, plus 'connection' >>>> string modified in service configuration files, plus >>>> documentation updates to refer to the new official way to >>>> configure services for MySQL. The database code won't, >>>> ideally, require any major changes, though some adaptation >>>> for the new client library may be needed. That said, Neutron >>>> does not seem to require any changes, though it was revealed >>>> that there are some alembic migration rules in Keystone or >>>> Glance that need (trivial) modifications. >>>> >>>> You can see how trivial the switch can be achieved for a >>>> service based on example for Neutron [2]. >>>> >>>> While this is a Neutron specific proposal, there is an >>>> obvious wish to switch to the new library globally throughout >>>> all the projects, to reduce devops burden, among other >>>> things. My vision is that, ideally, we switch all projects to >>>> the new library in Juno, though we still may leave several >>>> projects for K in case any issues arise, similar to the way >>>> projects switched to oslo.messaging during two cycles instead >>>> of one. Though looking at how easy Neutron can be switched to >>>> the new library, I wouldn't expect any issues that would >>>> postpone the switch till K. >>>> >>>> It was mentioned in comments to the spec proposal that there >>>> were some discussions at the latest summit around possible >>>> switch in context of Nova that revealed some concerns, though >>>> they do not seem to be documented anywhere. So if you know >>>> anything about it, please comment. >>>> >>>> So, we'd like to hear from other projects what's your take on >>>> that move, whether you see any issues or have concerns about >>>> it. >>>> >>>> Thanks for your comments, /Ihar >>>> >>>> [1]: https://review.openstack.org/#/c/104905/ [2]: >>>> https://review.openstack.org/#/c/105209/ >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >> >> >>>> _______________________________________________ >> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > > _______________________________________________ OpenStack-dev > mailing list OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCgAGBQJTwrUrAAoJEC5aWaUY1u57CFsH/2Yz8VZreSK1gFTLz8UxCSSV 3HGx/o3QlEomOWmSOkX4IfnE7XBouyPvIFoTwXdgrjwzQBW/iril+G2TA+B/RcER +2ccZO9DkCKxCeAU4Tq/a7G92s3sLW+OgihxoIdpnEDdkMGa6XGDa/hLDrngaGyj ow/DJeHDJVXd0xflTckMQusm8SfjLES/3yxW9A5rEH+f+4dn+bOf4SSQMdL44kLC oTNQ4Fv6m0EOEg2JhliLbljFiJCfhDBXAHuoDT3zb+XGgIXG8zf4A8Q99pppM1Xd snhtn5E5Fny+GdApBUb9JXE8uBo6E0EXMvZB2qPrwbk//GsxNtOLJxuWrmeffzM= =Xprg -----END PGP SIGNATURE----- _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev