Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-27 Thread Jay Pipes 

On 05/24/2016 01:54 AM, Gary Kotton wrote:

Hi,

We have used tooz to enable concurrency. Zookeeper and Redis worked
well. I think that it is certainly something that we need to consider.
The challenge becomes a deployment.


I'm not following. What does tooz, ZK or Redis have to do with 
concurrency of NeutronDbObject and oslo.versionedobject interfaces?


Best,
-jay


*From: *Damon Wang 
*Reply-To: *OpenStack List 
*Date: *Tuesday, May 24, 2016 at 5:58 AM
*To: *OpenStack List 
*Subject: *Re: [openstack-dev] [neutron][ovo] NeutronDbObject
concurrency issues

Hi,

I want to add an option which handle by another project Tooz.

https://github.com/openstack/tooz
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_tooz&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=BlFY_Ge8eQsKWmvzFjG8WDvC_476YtRBCancu5pAbt8&s=eX2MSKfmcWhBdoSfE7czdazEibklQWO9Z2bSdGtluOo&e=>

with redis or some other drivers, it seems pretty a good choice.

Any comments?

Wei Wang

2016-05-17 6:53 GMT+08:00 Ilya Chukhnakov mailto:ichukhna...@mirantis.com>>:

On 16 May 2016, at 20:01, Michał Dulko mailto:michal.du...@intel.com>> wrote:


It's not directly related, but this reminds me of tests done by
geguileo
[1] some time ago that were comparing different methods of
preventing DB
race conditions in concurrent environment. Maybe you'll also
find them
useful as you'll probably need to do something like conditional
update
to increment a revision number.

[1] https://github.com/Akrog/test-cinder-atomic-states

<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Akrog_test-2Dcinder-2Datomic-2Dstates&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=BlFY_Ge8eQsKWmvzFjG8WDvC_476YtRBCancu5pAbt8&s=NcW1-muP9ymke4gz9blqXQ1mbrfpmfLHZHjQspx6jMY&e=>

Thanks for the link. The SQLA revisions are similar to the
'solutions/update_with_where',

but they use the dedicated column for that [2]. And as long as it is
properly configured,

it happens 'automagically' (SQLA will take care of adding proper
'where' to 'update').

[2] http://docs.sqlalchemy.org/en/latest/orm/versioning.html

<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.sqlalchemy.org_en_latest_orm_versioning.html&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=BlFY_Ge8eQsKWmvzFjG8WDvC_476YtRBCancu5pAbt8&s=-WhOlX8NTukfXp5q3ud1TpLWsGMqRa4SlHYYYMGGTqs&e=>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-24 Thread John Schwarz 
The incorporation of tooz and Neutron is being discussed as part of
https://bugs.launchpad.net/neutron/+bug/1552680 as an RFE for Newton.
Hopefully I'll have some news to break in on this matter in the
upcoming days (and if I do I'll update on the launchpad to eliminate
duplicity).

On Tue, May 24, 2016 at 8:54 AM, Gary Kotton  wrote:
> Hi,
>
> We have used tooz to enable concurrency. Zookeeper and Redis worked well. I
> think that it is certainly something that we need to consider. The challenge
> becomes a deployment.
>
> Thanks
>
> Gary
>
>
>
> From: Damon Wang 
> Reply-To: OpenStack List 
> Date: Tuesday, May 24, 2016 at 5:58 AM
> To: OpenStack List 
> Subject: Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency
> issues
>
>
>
> Hi,
>
>
>
> I want to add an option which handle by another project Tooz.
>
>
>
> https://github.com/openstack/tooz
>
>
>
> with redis or some other drivers, it seems pretty a good choice.
>
>
>
> Any comments?
>
>
>
> Wei Wang
>
>
>
> 2016-05-17 6:53 GMT+08:00 Ilya Chukhnakov :
>
>
>
> On 16 May 2016, at 20:01, Michał Dulko  wrote:
>
>
> It's not directly related, but this reminds me of tests done by geguileo
> [1] some time ago that were comparing different methods of preventing DB
> race conditions in concurrent environment. Maybe you'll also find them
> useful as you'll probably need to do something like conditional update
> to increment a revision number.
>
> [1] https://github.com/Akrog/test-cinder-atomic-states
>
>
>
> Thanks for the link. The SQLA revisions are similar to the
> 'solutions/update_with_where',
>
> but they use the dedicated column for that [2]. And as long as it is
> properly configured,
>
> it happens 'automagically' (SQLA will take care of adding proper 'where' to
> 'update').
>
>
>
> [2] http://docs.sqlalchemy.org/en/latest/orm/versioning.html
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
John Schwarz,
Red Hat.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-23 Thread Gary Kotton 
Hi,
We have used tooz to enable concurrency. Zookeeper and Redis worked well. I 
think that it is certainly something that we need to consider. The challenge 
becomes a deployment.
Thanks
Gary

From: Damon Wang 
Reply-To: OpenStack List 
Date: Tuesday, May 24, 2016 at 5:58 AM
To: OpenStack List 
Subject: Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

Hi,

I want to add an option which handle by another project Tooz.

https://github.com/openstack/tooz<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_tooz&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=BlFY_Ge8eQsKWmvzFjG8WDvC_476YtRBCancu5pAbt8&s=eX2MSKfmcWhBdoSfE7czdazEibklQWO9Z2bSdGtluOo&e=>

with redis or some other drivers, it seems pretty a good choice.

Any comments?

Wei Wang

2016-05-17 6:53 GMT+08:00 Ilya Chukhnakov 
mailto:ichukhna...@mirantis.com>>:

On 16 May 2016, at 20:01, Michał Dulko 
mailto:michal.du...@intel.com>> wrote:

It's not directly related, but this reminds me of tests done by geguileo
[1] some time ago that were comparing different methods of preventing DB
race conditions in concurrent environment. Maybe you'll also find them
useful as you'll probably need to do something like conditional update
to increment a revision number.

[1] 
https://github.com/Akrog/test-cinder-atomic-states<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Akrog_test-2Dcinder-2Datomic-2Dstates&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=BlFY_Ge8eQsKWmvzFjG8WDvC_476YtRBCancu5pAbt8&s=NcW1-muP9ymke4gz9blqXQ1mbrfpmfLHZHjQspx6jMY&e=>

Thanks for the link. The SQLA revisions are similar to the 
'solutions/update_with_where',
but they use the dedicated column for that [2]. And as long as it is properly 
configured,
it happens 'automagically' (SQLA will take care of adding proper 'where' to 
'update').

[2] 
http://docs.sqlalchemy.org/en/latest/orm/versioning.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.sqlalchemy.org_en_latest_orm_versioning.html&d=CwMFaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=BlFY_Ge8eQsKWmvzFjG8WDvC_476YtRBCancu5pAbt8&s=-WhOlX8NTukfXp5q3ud1TpLWsGMqRa4SlHYYYMGGTqs&e=>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-23 Thread Damon Wang 
Hi,

I want to add an option which handle by another project Tooz.

https://github.com/openstack/tooz

with redis or some other drivers, it seems pretty a good choice.

Any comments?

Wei Wang

2016-05-17 6:53 GMT+08:00 Ilya Chukhnakov :

>
> On 16 May 2016, at 20:01, Michał Dulko  wrote:
>
> It's not directly related, but this reminds me of tests done by geguileo
> [1] some time ago that were comparing different methods of preventing DB
> race conditions in concurrent environment. Maybe you'll also find them
> useful as you'll probably need to do something like conditional update
> to increment a revision number.
>
> [1] https://github.com/Akrog/test-cinder-atomic-states
>
>
> Thanks for the link. The SQLA revisions are similar to the
> 'solutions/update_with_where',
> but they use the dedicated column for that [2]. And as long as it is
> properly configured,
> it happens 'automagically' (SQLA will take care of adding proper 'where'
> to 'update').
>
> [2] http://docs.sqlalchemy.org/en/latest/orm/versioning.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-16 Thread Ilya Chukhnakov 

> On 16 May 2016, at 20:01, Michał Dulko  wrote:
> 
> It's not directly related, but this reminds me of tests done by geguileo
> [1] some time ago that were comparing different methods of preventing DB
> race conditions in concurrent environment. Maybe you'll also find them
> useful as you'll probably need to do something like conditional update
> to increment a revision number.
> 
> [1] https://github.com/Akrog/test-cinder-atomic-states 
> 

Thanks for the link. The SQLA revisions are similar to the 
'solutions/update_with_where',
but they use the dedicated column for that [2]. And as long as it is properly 
configured,
it happens 'automagically' (SQLA will take care of adding proper 'where' to 
'update').

[2] http://docs.sqlalchemy.org/en/latest/orm/versioning.html__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-16 Thread Michał Dulko 
On 05/16/2016 05:49 PM, Ilya Chukhnakov wrote:
> On 16 May 2016, at 18:18, Michał Dulko  > wrote:
>> In Cinder we're handling similar problems related to race conditions
>> between status check and status change with conditional updates.
>> Basically we're doing "UPDATE table SET status='abc' WHERE id=1 AND
>> status='status_allowing_transition_to_foo';".
> Thanks for the info. I'll certainly look into it. But for now I'm
> planning to reuse
> revision numbers from [1].
>
>> In general the problem you're mentioning is related more to the
>> concurrent DB updates and o.vo never aimed for magically solving that
>> problem. I believe you've had same problem with raw SQLA objects.
> For SQLA we at least have an option to use with_for_update (I've found
> it is being
> used in some places). But with OVO we do not have that level of
> control yet.
>
> [1] https://review.openstack.org/#/c/303966
>

It's not directly related, but this reminds me of tests done by geguileo
[1] some time ago that were comparing different methods of preventing DB
race conditions in concurrent environment. Maybe you'll also find them
useful as you'll probably need to do something like conditional update
to increment a revision number.

[1] https://github.com/Akrog/test-cinder-atomic-states


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-16 Thread Ilya Chukhnakov 
On 16 May 2016, at 18:18, Michał Dulko  wrote:
> In Cinder we're handling similar problems related to race conditions
> between status check and status change with conditional updates.
> Basically we're doing "UPDATE table SET status='abc' WHERE id=1 AND
> status='status_allowing_transition_to_foo';".
Thanks for the info. I'll certainly look into it. But for now I'm planning to 
reuse
revision numbers from [1].

> In general the problem you're mentioning is related more to the
> concurrent DB updates and o.vo never aimed for magically solving that
> problem. I believe you've had same problem with raw SQLA objects.
For SQLA we at least have an option to use with_for_update (I've found it is 
being
used in some places). But with OVO we do not have that level of control yet.

[1] https://review.openstack.org/#/c/303966__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-16 Thread Michał Dulko 
On 05/12/2016 08:26 PM, Ilya Chukhnakov wrote:
> Hi everyone.
>
> I’ve recently found that straightforward use of NeutronDbObject is prone to
> concurrency-related problems.
>
> I’ve submitted a patch set [3] with some tests to show that without special
> treatment using NeutronDbObject could lead to unexpected results.
>
> Further patch sets will provide acquire_object/acquire_objects contextmanager
> methods to the NeutronDbObject class. These methods are to be used in place of
> get_object/get_objects whenever the user intends to make changes to the 
> object.
> These methods would start an autonested_transaction.
>
> There are (at least) two potential options for the implementation:
>
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).
>
>pros:
>  - the object is guaranteed to not be changed while within the context
>
>cons:
>  - prone to deadlocks ([1] and potentially when locking multiple objects)
>
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
>counter [2] or add our own. If conflicting changes are detected upon 
> exiting
>the context (i.e. version counter held differs from the one in the DB), 
> will
>raise OSLO RetryRequest exception.
>
>pros:
>  - does not require locking
>
>cons:
>  - require an additional field in the models
>
> While opt.2 only prevents the conflicting changes, but does not guarantee that
> the object does not change while within the context, opt.1 may seem
> preferential. But even with opt.1 the user should not expect that the changes
> made to the object while within the context will get to the database as the
> autonested_transaction could fail on flush/commit.
>
> So I’d like to hear others’ opinion on the problem and which of the two
> implementation options would be preferred? Or maybe someone has a better idea.
>
> [1] 
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html
>
> [3] https://review.openstack.org/#/c/315705/

In Cinder we're handling similar problems related to race conditions
between status check and status change with conditional updates.
Basically we're doing "UPDATE table SET status='abc' WHERE id=1 AND
status='status_allowing_transition_to_foo';". You can check out o.vo
layer of that stuff at [1].

You could provide fields you care not to be modified in the
expected_values and retry DB operations on failure.

In general the problem you're mentioning is related more to the
concurrent DB updates and o.vo never aimed for magically solving that
problem. I believe you've had same problem with raw SQLA objects.

[1]
https://github.com/openstack/cinder/blob/master/cinder/objects/base.py#L173-L289



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-13 Thread Ilya Chukhnakov 
> there is ongoing work to add the revision number
Thanks for pointing out. Then I’ll stick with opt.2.

> Then, the object should before update() fetch the current state and try to 
> merge the requested changes to the current state of the object
I don’t think merging states automatically could be possible without knowledge 
of business logic (the ab = a + b test case clearly shows that I believe). So I 
was thinking about throwing the Retry exception and let the user take care of 
that (e.g. with_db_retry).


> On 13 May 2016, at 11:22, Korzeniewski, Artur  
> wrote:
> 
> Hi Ilya,
> Thanks for investigating the concurrency issue. It is indeed a problem from 
> multithreaded neutron-server point of view.
> 
> Regarding the opt.2, there is ongoing work to add the revision number[1] to 
> main neutron resources (port, network, subnet...) This is connected to spec 
> [2] and Launchpad bug [3]: push all object changes as AMQP notifications. I 
> think [1] is implementing part of your idea about version number. If version 
> number will be added to standard attributes table, it will automagically 
> appear as a new field in object.
> 
> What is left to be done in opt.2 is to handle situation when object is trying 
> to update the state and the revision is not the latest. Then, the object 
> should before update() fetch the current state and try to merge the requested 
> changes to the current state of the object, and then apply it in the DB. Also 
> it would send the newest object state via AMQP notification, obsoleting the 
> previous configuration.
> 
> We can also check how nova is handling the concurrency issue in their OVO 
> usage model.
> 
> Regards,
> Artur
> IRC: korzen
> 
> [1] https://review.openstack.org/#/c/303966/11
> [2] https://review.openstack.org/#/c/225995
> [3] https://bugs.launchpad.net/neutron/+bug/1516195
> 
> -Original Message-
> From: Ilya Chukhnakov [mailto:ichukhna...@mirantis.com] 
> Sent: Thursday, May 12, 2016 8:26 PM
> To: OpenStack Development Mailing List 
> Subject: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues
> 
> Hi everyone.
> 
> I’ve recently found that straightforward use of NeutronDbObject is prone to 
> concurrency-related problems.
> 
> I’ve submitted a patch set [3] with some tests to show that without special 
> treatment using NeutronDbObject could lead to unexpected results.
> 
> Further patch sets will provide acquire_object/acquire_objects contextmanager 
> methods to the NeutronDbObject class. These methods are to be used in place 
> of get_object/get_objects whenever the user intends to make changes to the 
> object.
> These methods would start an autonested_transaction.
> 
> There are (at least) two potential options for the implementation:
> 
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).
> 
>   pros:
> - the object is guaranteed to not be changed while within the context
> 
>   cons:
> - prone to deadlocks ([1] and potentially when locking multiple objects)
> 
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
>   counter [2] or add our own. If conflicting changes are detected upon exiting
>   the context (i.e. version counter held differs from the one in the DB), will
>   raise OSLO RetryRequest exception.
> 
>   pros:
> - does not require locking
> 
>   cons:
> - require an additional field in the models
> 
> While opt.2 only prevents the conflicting changes, but does not guarantee 
> that the object does not change while within the context, opt.1 may seem 
> preferential. But even with opt.1 the user should not expect that the changes 
> made to the object while within the context will get to the database as the 
> autonested_transaction could fail on flush/commit.
> 
> So I’d like to hear others’ opinion on the problem and which of the two 
> implementation options would be preferred? Or maybe someone has a better idea.
> 
> [1] 
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html
> 
> [3] https://review.openstack.org/#/c/315705/
> 
> --
> Thanks,
> Ilya
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-13 Thread Ilya Chukhnakov 
>In a multi-writer galera cluster both SELECT FOR UPDATE and CAS won't fail 
>until commit time

Good point, I did not consider multi-writer. But anyway, my point was that 
opt.2 is no worse than opt.1.


> On 13 May 2016, at 12:04, Kevin Benton  wrote:
> 
> >While opt.2 only prevents the conflicting changes, but does not guarantee 
> >that the object does not change while within the context,
> 
> I'm not sure what you mean here. In a multi-writer galera cluster both SELECT 
> FOR UPDATE and CAS won't fail until commit time if someone writes to another 
> server.
> So the DB lock only provides the conflicting change guarantee in a 
> single-writer mysql setup.
> 
> On Thu, May 12, 2016 at 11:26 AM, Ilya Chukhnakov  > wrote:
> Hi everyone.
> 
> I’ve recently found that straightforward use of NeutronDbObject is prone to
> concurrency-related problems.
> 
> I’ve submitted a patch set [3] with some tests to show that without special
> treatment using NeutronDbObject could lead to unexpected results.
> 
> Further patch sets will provide acquire_object/acquire_objects contextmanager
> methods to the NeutronDbObject class. These methods are to be used in place of
> get_object/get_objects whenever the user intends to make changes to the 
> object.
> These methods would start an autonested_transaction.
> 
> There are (at least) two potential options for the implementation:
> 
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).
> 
>pros:
>  - the object is guaranteed to not be changed while within the context
> 
>cons:
>  - prone to deadlocks ([1] and potentially when locking multiple objects)
> 
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
>counter [2] or add our own. If conflicting changes are detected upon 
> exiting
>the context (i.e. version counter held differs from the one in the DB), 
> will
>raise OSLO RetryRequest exception.
> 
>pros:
>  - does not require locking
> 
>cons:
>  - require an additional field in the models
> 
> While opt.2 only prevents the conflicting changes, but does not guarantee that
> the object does not change while within the context, opt.1 may seem
> preferential. But even with opt.1 the user should not expect that the changes
> made to the object while within the context will get to the database as the
> autonested_transaction could fail on flush/commit.
> 
> So I’d like to hear others’ opinion on the problem and which of the two
> implementation options would be preferred? Or maybe someone has a better idea.
> 
> [1] 
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
>  
> 
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html 
> 
> 
> [3] https://review.openstack.org/#/c/315705/ 
> 
> 
> --
> Thanks,
> Ilya
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-13 Thread Kevin Benton 
>While opt.2 only prevents the conflicting changes, but does not guarantee
that the object does not change while within the context,

I'm not sure what you mean here. In a multi-writer galera cluster both
SELECT FOR UPDATE and CAS won't fail until commit time if someone writes to
another server.
So the DB lock only provides the conflicting change guarantee in a
single-writer mysql setup.

On Thu, May 12, 2016 at 11:26 AM, Ilya Chukhnakov 
wrote:

> Hi everyone.
>
> I’ve recently found that straightforward use of NeutronDbObject is prone to
> concurrency-related problems.
>
> I’ve submitted a patch set [3] with some tests to show that without special
> treatment using NeutronDbObject could lead to unexpected results.
>
> Further patch sets will provide acquire_object/acquire_objects
> contextmanager
> methods to the NeutronDbObject class. These methods are to be used in
> place of
> get_object/get_objects whenever the user intends to make changes to the
> object.
> These methods would start an autonested_transaction.
>
> There are (at least) two potential options for the implementation:
>
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy
> with_for_update).
>
>pros:
>  - the object is guaranteed to not be changed while within the context
>
>cons:
>  - prone to deadlocks ([1] and potentially when locking multiple
> objects)
>
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy
> version
>counter [2] or add our own. If conflicting changes are detected upon
> exiting
>the context (i.e. version counter held differs from the one in the DB),
> will
>raise OSLO RetryRequest exception.
>
>pros:
>  - does not require locking
>
>cons:
>  - require an additional field in the models
>
> While opt.2 only prevents the conflicting changes, but does not guarantee
> that
> the object does not change while within the context, opt.1 may seem
> preferential. But even with opt.1 the user should not expect that the
> changes
> made to the object while within the context will get to the database as the
> autonested_transaction could fail on flush/commit.
>
> So I’d like to hear others’ opinion on the problem and which of the two
> implementation options would be preferred? Or maybe someone has a better
> idea.
>
> [1]
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html
>
> [3] https://review.openstack.org/#/c/315705/
>
> --
> Thanks,
> Ilya
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-13 Thread Kevin Benton 
>What is left to be done in opt.2 is to handle situation when object is
trying to update the state and the revision is not the latest.


Once the revision counter is added, this will generate an error (StaleData
I think) that will be caught by the retry decorator so the process can be
started over by fetching the latest state.

On Fri, May 13, 2016 at 1:22 AM, Korzeniewski, Artur <
artur.korzeniew...@intel.com> wrote:

> Hi Ilya,
> Thanks for investigating the concurrency issue. It is indeed a problem
> from multithreaded neutron-server point of view.
>
> Regarding the opt.2, there is ongoing work to add the revision number[1]
> to main neutron resources (port, network, subnet...) This is connected to
> spec [2] and Launchpad bug [3]: push all object changes as AMQP
> notifications. I think [1] is implementing part of your idea about version
> number. If version number will be added to standard attributes table, it
> will automagically appear as a new field in object.
>
> What is left to be done in opt.2 is to handle situation when object is
> trying to update the state and the revision is not the latest. Then, the
> object should before update() fetch the current state and try to merge the
> requested changes to the current state of the object, and then apply it in
> the DB. Also it would send the newest object state via AMQP notification,
> obsoleting the previous configuration.
>
> We can also check how nova is handling the concurrency issue in their OVO
> usage model.
>
> Regards,
> Artur
> IRC: korzen
>
> [1] https://review.openstack.org/#/c/303966/11
> [2] https://review.openstack.org/#/c/225995
> [3] https://bugs.launchpad.net/neutron/+bug/1516195
>
> -Original Message-
> From: Ilya Chukhnakov [mailto:ichukhna...@mirantis.com]
> Sent: Thursday, May 12, 2016 8:26 PM
> To: OpenStack Development Mailing List 
> Subject: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues
>
> Hi everyone.
>
> I’ve recently found that straightforward use of NeutronDbObject is prone
> to concurrency-related problems.
>
> I’ve submitted a patch set [3] with some tests to show that without
> special treatment using NeutronDbObject could lead to unexpected results.
>
> Further patch sets will provide acquire_object/acquire_objects
> contextmanager methods to the NeutronDbObject class. These methods are to
> be used in place of get_object/get_objects whenever the user intends to
> make changes to the object.
> These methods would start an autonested_transaction.
>
> There are (at least) two potential options for the implementation:
>
> 1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy
> with_for_update).
>
>pros:
>  - the object is guaranteed to not be changed while within the context
>
>cons:
>  - prone to deadlocks ([1] and potentially when locking multiple
> objects)
>
> 2. Lock-free CAS based on object version counter. Can use SqlAlchemy
> version
>counter [2] or add our own. If conflicting changes are detected upon
> exiting
>the context (i.e. version counter held differs from the one in the DB),
> will
>raise OSLO RetryRequest exception.
>
>pros:
>  - does not require locking
>
>cons:
>  - require an additional field in the models
>
> While opt.2 only prevents the conflicting changes, but does not guarantee
> that the object does not change while within the context, opt.1 may seem
> preferential. But even with opt.1 the user should not expect that the
> changes made to the object while within the context will get to the
> database as the autonested_transaction could fail on flush/commit.
>
> So I’d like to hear others’ opinion on the problem and which of the two
> implementation options would be preferred? Or maybe someone has a better
> idea.
>
> [1]
> https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
> [2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html
>
> [3] https://review.openstack.org/#/c/315705/
>
> --
> Thanks,
> Ilya
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-13 Thread Korzeniewski, Artur 
Hi Ilya,
Thanks for investigating the concurrency issue. It is indeed a problem from 
multithreaded neutron-server point of view.

Regarding the opt.2, there is ongoing work to add the revision number[1] to 
main neutron resources (port, network, subnet...) This is connected to spec [2] 
and Launchpad bug [3]: push all object changes as AMQP notifications. I think 
[1] is implementing part of your idea about version number. If version number 
will be added to standard attributes table, it will automagically appear as a 
new field in object.

What is left to be done in opt.2 is to handle situation when object is trying 
to update the state and the revision is not the latest. Then, the object should 
before update() fetch the current state and try to merge the requested changes 
to the current state of the object, and then apply it in the DB. Also it would 
send the newest object state via AMQP notification, obsoleting the previous 
configuration.

We can also check how nova is handling the concurrency issue in their OVO usage 
model.

Regards,
Artur
IRC: korzen

[1] https://review.openstack.org/#/c/303966/11
[2] https://review.openstack.org/#/c/225995
[3] https://bugs.launchpad.net/neutron/+bug/1516195

-Original Message-
From: Ilya Chukhnakov [mailto:ichukhna...@mirantis.com] 
Sent: Thursday, May 12, 2016 8:26 PM
To: OpenStack Development Mailing List 
Subject: [openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

Hi everyone.

I’ve recently found that straightforward use of NeutronDbObject is prone to 
concurrency-related problems.

I’ve submitted a patch set [3] with some tests to show that without special 
treatment using NeutronDbObject could lead to unexpected results.

Further patch sets will provide acquire_object/acquire_objects contextmanager 
methods to the NeutronDbObject class. These methods are to be used in place of 
get_object/get_objects whenever the user intends to make changes to the object.
These methods would start an autonested_transaction.

There are (at least) two potential options for the implementation:

1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).

   pros:
 - the object is guaranteed to not be changed while within the context

   cons:
 - prone to deadlocks ([1] and potentially when locking multiple objects)

2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
   counter [2] or add our own. If conflicting changes are detected upon exiting
   the context (i.e. version counter held differs from the one in the DB), will
   raise OSLO RetryRequest exception.

   pros:
 - does not require locking

   cons:
 - require an additional field in the models

While opt.2 only prevents the conflicting changes, but does not guarantee that 
the object does not change while within the context, opt.1 may seem 
preferential. But even with opt.1 the user should not expect that the changes 
made to the object while within the context will get to the database as the 
autonested_transaction could fail on flush/commit.

So I’d like to hear others’ opinion on the problem and which of the two 
implementation options would be preferred? Or maybe someone has a better idea.

[1] 
https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
[2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html

[3] https://review.openstack.org/#/c/315705/

--
Thanks,
Ilya
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][ovo] NeutronDbObject concurrency issues

2016-05-12 Thread Ilya Chukhnakov 
Hi everyone.

I’ve recently found that straightforward use of NeutronDbObject is prone to
concurrency-related problems.

I’ve submitted a patch set [3] with some tests to show that without special
treatment using NeutronDbObject could lead to unexpected results.

Further patch sets will provide acquire_object/acquire_objects contextmanager
methods to the NeutronDbObject class. These methods are to be used in place of
get_object/get_objects whenever the user intends to make changes to the object.
These methods would start an autonested_transaction.

There are (at least) two potential options for the implementation:

1. Based on the DB locks (e.g. SELECT FOR UPDATE/SqlAlchemy with_for_update).

   pros:
 - the object is guaranteed to not be changed while within the context

   cons:
 - prone to deadlocks ([1] and potentially when locking multiple objects)

2. Lock-free CAS based on object version counter. Can use SqlAlchemy version
   counter [2] or add our own. If conflicting changes are detected upon exiting
   the context (i.e. version counter held differs from the one in the DB), will
   raise OSLO RetryRequest exception.

   pros:
 - does not require locking

   cons:
 - require an additional field in the models

While opt.2 only prevents the conflicting changes, but does not guarantee that
the object does not change while within the context, opt.1 may seem
preferential. But even with opt.1 the user should not expect that the changes
made to the object while within the context will get to the database as the
autonested_transaction could fail on flush/commit.

So I’d like to hear others’ opinion on the problem and which of the two
implementation options would be preferred? Or maybe someone has a better idea.

[1] 
https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
[2] http://docs.sqlalchemy.org/en/rel_0_9/orm/versioning.html

[3] https://review.openstack.org/#/c/315705/

--
Thanks,
Ilya
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev