Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Eugene Nikanorov
> > > *You are 100% correct that setting the transaction isolation level to READ > COMMITTED works in the retry loop*. > > I stand corrected, and humbled :) Please accept my apologies. > Thanks for letting me know :) > > One thing I did note, though, is that changing the isolation level of an > *

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Mike Bayer
regardless, I’d still very much prefer isolation being set here in an explicit fashion and on a per-transaction basis, rather than across the board, if a method is known to require it. The advantage of this is that the isolation level in general can be changed without any risk of the individua

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Jay Pipes
Eugene, I just spent about an hour playing around with an example scenario in both MySQL and PostgreSQL using READ COMMITTED and REPEATABLE READ isolation levels. The scenario I used had a single row being updated, with a loop and a check on rows affected. *You are 100% correct that setting th

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Eugene Nikanorov
Comments inline: On Thu, Nov 20, 2014 at 4:34 AM, Jay Pipes wrote: > >> > So while the SELECTs may return different data on successive calls when > you use the READ COMMITTED isolation level, the UPDATE statements will > continue to return 0 rows affected **if they attempt to change rows that >

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Jay Pipes
On 11/19/2014 04:27 PM, Eugene Nikanorov wrote: Wow, lots of feedback in a matter of hours. First of all, reading postgres docs I see that READ COMMITTED is the same as for mysql, so it should address the issue we're discussing: "/Read Committed/ is the default isolation level in PostgreSQL. Wh

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Eugene Nikanorov
> But the isolation mode change won’t really help here as pointed out by Jay; discrete transactions have to be used instead. I still think it will, per postgres documentation (which might look confusing, but still...) It actually helps for mysql, that was confirmed. For postgres it appears to be th

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
> On Nov 19, 2014, at 4:14 PM, Clint Byrum wrote: > > > One simply cannot rely on multi-statement transactions to always succeed. agree, but the thing you want is that the transaction either succeeds or explicitly fails, the latter hopefully in such a way that a retry can be added which has

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Eugene Nikanorov
Wow, lots of feedback in a matter of hours. First of all, reading postgres docs I see that READ COMMITTED is the same as for mysql, so it should address the issue we're discussing: "*Read Committed* is the default isolation level in PostgreSQL. When a transaction uses this isolation level, a SELE

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
> On Nov 19, 2014, at 3:47 PM, Ryan Moats wrote: > > > > BTW, I view your examples from oslo as helping make my argument for > me (and I don't think that was your intent :) ) > I disagree with that as IMHO the differences in producing MM in the app layer against arbitrary backends (Postgresq

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Clint Byrum
Excerpts from Mike Bayer's message of 2014-11-19 10:05:35 -0800: > > > On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov > > wrote: > > > > Hi neutron folks, > > > > There is an ongoing effort to refactor some neutron DB logic to be > > compatible with galera/mysql which doesn't support locking

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ryan Moats
Ian Wells wrote on 11/19/2014 02:33:40 PM: [snip] > When you have a plugin that's decided to be synchronous, then there > are cases where the DB lock is held for a technically indefinite > period of time.  This is basically broken. A big +1 to this statement Ryan Moats__

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ryan Moats
> > > Mike Bayer wrote on 11/19/2014 02:10:18 PM: > > > From: Mike Bayer > > To: "OpenStack Development Mailing List (not for usage questions)" > > > > Date: 11/19/2014 02:11 PM > > Subject: Re: [openstack-dev] [Neutron] DB: transaction i

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ian Wells
On 19 November 2014 11:58, Jay Pipes wrote: > Some code paths that used locking in the past were rewritten to retry > >> the operation if they detect that an object was modified concurrently. >> The problem here is that all DB operations (CRUD) are performed in the >> scope of some transaction th

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
> On Nov 19, 2014, at 2:58 PM, Jay Pipes wrote: > > >> In other words, the retry logic like the following will not work: >> >> def allocate_obj(): >> with session.begin(subtrans=True): >> for i in xrange(n_retries): >> obj = session.query(Model).filter_by(filters) >>

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
t ? > > Ryan Moats > > Mike Bayer wrote on 11/19/2014 12:05:35 PM: > > > From: Mike Bayer > > To: "OpenStack Development Mailing List (not for usage questions)" > > > > Date: 11/19/2014 12:05 PM > > Subject: Re: [openstack-dev] [Neutron] D

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Jay Pipes
Hi Eugene, please see comments inline. But, bottom line, is that setting the transaction isolation level to READ_COMMITTED should be avoided. On 11/18/2014 01:38 PM, Eugene Nikanorov wrote: Hi neutron folks, There is an ongoing effort to refactor some neutron DB logic to be compatible with gal

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ryan Moats
014 12:05 PM > Subject: Re: [openstack-dev] [Neutron] DB: transaction isolation and > related questions > > On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov wrote: > > Hi neutron folks, > > There is an ongoing effort to refactor some neutron DB logic to be > compatible wit

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
> On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov wrote: > > Hi neutron folks, > > There is an ongoing effort to refactor some neutron DB logic to be compatible > with galera/mysql which doesn't support locking (with_lockmode('update')). > > Some code paths that used locking in the past were re