Because I'm just going to spam the list all afternoon... Out of curiousity, I ramped up the numbers in the script to get a most sustained attack on the db. With the old db code (pre pool) - running 10 concurrent copies of dbrepro.py gave me 10 db connections (as you'd expect) and a consistent sustained throughput of 45 qps for a while. When I pulled forward to trunk, well - the first time I hit out of connections, because I was configured with max_connections of 150. Once I upped that and restarted mysql, I got 185 concurrent connections to the database (why 185? I don't know) with bursts of 10 queries a second once every 5 seconds with a single burst on the front end of 101 queries in a second. (oh, and btw - with trunk and 10 concurrent copies I started getting that traceback again)
fwiw On 09/25/2011 05:59 PM, Monty Taylor wrote: > Hrm. It's not piling on with the locking like I originally thought - > from the db's perspective there's not a whole hell of a lot going on, > actually - even two copies of this script run concurrently cause them to > totally get in to some app-level spinning that ends up with at least one > script erroring. I'll keep poking, mainly just because it's interesting > - but it's certainly not happy. > > On 09/25/2011 01:53 PM, Vishvananda Ishaya wrote: >> Hey everyone, >> >> I'm a bit concerned with the connection pooling in the db. It seems that >> things are not getting cleaned up properly. I have a repro-case that causes >> failures that we have seen before. if I revert the >> nova/db/sqlalchemy/session.py to before the eventlet db pool was added I get >> no failures. If you want to see the issue, try the attached code. You will >> need to run from the nova directory or do python setup.py develop. You will >> also need to create a mysql database called test and edit the sql_connection >> string if you have a mysql password, etc. Please check this code. If we >> can't come up with a fix, I htink we need to revert back to no connection >> pooling. >> >> Run the attached script at least 3 times The code below runs fine the first >> couple of times, Then it starts to fail with the following error: >> >> 2011-09-24 12:51:02,799 INFO sqlalchemy.engine.base.Engine.0x...36d0 [-] >> ROLLBACK >> Traceback (most recent call last): >> File "/Library/Python/2.7/site-packages/eventlet/hubs/hub.py", line 336, in >> fire_timers >> timer() >> File "/Library/Python/2.7/site-packages/eventlet/hubs/timer.py", line 56, >> in __call__ >> cb(*args, **kw) >> File "/Library/Python/2.7/site-packages/eventlet/event.py", line 163, in >> _do_send >> waiter.switch(result) >> File "/Library/Python/2.7/site-packages/eventlet/greenthread.py", line 192, >> in main >> result = function(*args, **kwargs) >> File "dbrepro.py", line 44, in associate >> ip = db.fixed_ip_associate_pool(ctxt, 1, instance_id=val) >> File "/Users/vishvananda/os/nova/nova/db/api.py", line 352, in >> fixed_ip_associate_pool >> instance_id, host) >> File "/Users/vishvananda/os/nova/nova/db/sqlalchemy/api.py", line 102, in >> wrapper >> return f(*args, **kwargs) >> File "/Users/vishvananda/os/nova/nova/db/sqlalchemy/api.py", line 725, in >> fixed_ip_associate_pool >> filter_by(host=None).\ >> File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line >> 1496, in first >> ret = list(self[0:1]) >> File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line >> 1405, in __getitem__ >> return list(res) >> File "/Library/Python/2.7/site-packages/sqlalchemy/orm/query.py", line >> 1669, in instances >> fetch = cursor.fetchall() >> File "/Library/Python/2.7/site-packages/sqlalchemy/engine/base.py", line >> 2383, in fetchall >> l = self.process_rows(self._fetchall_impl()) >> File "/Library/Python/2.7/site-packages/sqlalchemy/engine/base.py", line >> 2366, in process_rows >> keymap = metadata._keymap >> AttributeError: 'NoneType' object has no attribute '_keymap' >> >> >> >> >> >> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~openstack >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : [email protected] > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
