Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Eugene Nikanorov
Comments inline: On Thu, Nov 20, 2014 at 4:34 AM, Jay Pipes jaypi...@gmail.com wrote: So while the SELECTs may return different data on successive calls when you use the READ COMMITTED isolation level, the UPDATE statements will continue to return 0 rows affected **if they attempt to change

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Jay Pipes
Eugene, I just spent about an hour playing around with an example scenario in both MySQL and PostgreSQL using READ COMMITTED and REPEATABLE READ isolation levels. The scenario I used had a single row being updated, with a loop and a check on rows affected. *You are 100% correct that setting

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Mike Bayer
regardless, I’d still very much prefer isolation being set here in an explicit fashion and on a per-transaction basis, rather than across the board, if a method is known to require it. The advantage of this is that the isolation level in general can be changed without any risk of the

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Eugene Nikanorov
*You are 100% correct that setting the transaction isolation level to READ COMMITTED works in the retry loop*. I stand corrected, and humbled :) Please accept my apologies. Thanks for letting me know :) One thing I did note, though, is that changing the isolation level of an

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov enikano...@mirantis.com wrote: Hi neutron folks, There is an ongoing effort to refactor some neutron DB logic to be compatible with galera/mysql which doesn't support locking (with_lockmode('update')). Some code paths that used locking in

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ryan Moats
@lists.openstack.org Date: 11/19/2014 12:05 PM Subject: Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov enikano...@mirantis.com wrote: Hi neutron folks, There is an ongoing effort to refactor some neutron DB logic to be compatible

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Jay Pipes
Hi Eugene, please see comments inline. But, bottom line, is that setting the transaction isolation level to READ_COMMITTED should be avoided. On 11/18/2014 01:38 PM, Eugene Nikanorov wrote: Hi neutron folks, There is an ongoing effort to refactor some neutron DB logic to be compatible with

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 11/19/2014 12:05 PM Subject: Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov enikano...@mirantis.com wrote: Hi neutron folks

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 2:58 PM, Jay Pipes jaypi...@gmail.com wrote: In other words, the retry logic like the following will not work: def allocate_obj(): with session.begin(subtrans=True): for i in xrange(n_retries): obj =

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ian Wells
On 19 November 2014 11:58, Jay Pipes jaypi...@gmail.com wrote: Some code paths that used locking in the past were rewritten to retry the operation if they detect that an object was modified concurrently. The problem here is that all DB operations (CRUD) are performed in the scope of some

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ryan Moats
Ian Wells ijw.ubu...@cack.org.uk wrote on 11/19/2014 02:33:40 PM: [snip] When you have a plugin that's decided to be synchronous, then there are cases where the DB lock is held for a technically indefinite period of time.  This is basically broken. A big +1 to this statement Ryan

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Ryan Moats
Mike Bayer mba...@redhat.com wrote on 11/19/2014 02:10:18 PM: From: Mike Bayer mba...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 11/19/2014 02:11 PM Subject: Re: [openstack-dev] [Neutron] DB: transaction

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Clint Byrum
Excerpts from Mike Bayer's message of 2014-11-19 10:05:35 -0800: On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov enikano...@mirantis.com wrote: Hi neutron folks, There is an ongoing effort to refactor some neutron DB logic to be compatible with galera/mysql which doesn't support

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 3:47 PM, Ryan Moats rmo...@us.ibm.com wrote: BTW, I view your examples from oslo as helping make my argument for me (and I don't think that was your intent :) ) I disagree with that as IMHO the differences in producing MM in the app layer against arbitrary backends

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Eugene Nikanorov
Wow, lots of feedback in a matter of hours. First of all, reading postgres docs I see that READ COMMITTED is the same as for mysql, so it should address the issue we're discussing: *Read Committed* is the default isolation level in PostgreSQL. When a transaction uses this isolation level, a

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 4:14 PM, Clint Byrum cl...@fewbar.com wrote: One simply cannot rely on multi-statement transactions to always succeed. agree, but the thing you want is that the transaction either succeeds or explicitly fails, the latter hopefully in such a way that a retry can be

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Eugene Nikanorov
But the isolation mode change won’t really help here as pointed out by Jay; discrete transactions have to be used instead. I still think it will, per postgres documentation (which might look confusing, but still...) It actually helps for mysql, that was confirmed. For postgres it appears to be

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Jay Pipes
On 11/19/2014 04:27 PM, Eugene Nikanorov wrote: Wow, lots of feedback in a matter of hours. First of all, reading postgres docs I see that READ COMMITTED is the same as for mysql, so it should address the issue we're discussing: /Read Committed/ is the default isolation level in PostgreSQL.

[openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-18 Thread Eugene Nikanorov
Hi neutron folks, There is an ongoing effort to refactor some neutron DB logic to be compatible with galera/mysql which doesn't support locking (with_lockmode('update')). Some code paths that used locking in the past were rewritten to retry the operation if they detect that an object was