Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Joshua Harlow
...@gmail.com] *Sent:* Monday, June 29, 2015 2:46 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.com mailto:michal.du...@intel.com wrote

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Dulko, Michal
: Monday, June 29, 2015 2:46 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.commailto:michal.du...@intel.com wrote: There’s also some

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Duncan Thomas
On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.com wrote: There’s also some similar situations when we actually don’t lock on resources. For example – a cgsnapshot may get deleted while creating a consistencygroup from it. From my perspective it seems best to have atomic

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Avishay Traeger
On Sun, Jun 28, 2015 at 1:16 PM, Duncan Thomas duncan.tho...@gmail.com wrote: We need mutual exclusion for several operations. Whether that is done by entity queues, locks, state based locking at the api later, or something else, we need mutual exclusion. Our current api does not lend itself

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Dulko, Michal
Development Mailing List Subject: Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot We need mutual exclusion for several operations. Whether that is done by entity queues, locks, state based locking at the api later, or something else, we need mutual exclusion. Our current

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Duncan Thomas
...@gmail.com] *Sent:* Monday, June 29, 2015 2:46 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.com mailto:michal.du...@intel.com

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Joshua Harlow
(not for usage questions) *Subject:* Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.com mailto:michal.du...@intel.com mailto:michal.du...@intel.com mailto:michal.du...@intel.com wrote: There’s also some similar situations

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Joshua Harlow
] *Sent:* Monday, June 29, 2015 2:46 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.com

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Clint Byrum
Excerpts from Duncan Thomas's message of 2015-06-29 07:54:27 -0700: Do we know what is so hated about the glance task API? Tasks and entity queues give the required exclusion, if you accept that tasks can fail if previous tasks in the queue can cause things to be pulled out from under it.

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Joshua Harlow
Clint Byrum wrote: Excerpts from Duncan Thomas's message of 2015-06-29 07:54:27 -0700: Do we know what is so hated about the glance task API? Tasks and entity queues give the required exclusion, if you accept that tasks can fail if previous tasks in the queue can cause things to be pulled out

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-29 Thread Gorka Eguileor
On Mon, Jun 29, 2015 at 03:45:56PM +0300, Duncan Thomas wrote: On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.com wrote: There’s also some similar situations when we actually don’t lock on resources. For example – a cgsnapshot may get deleted while creating a

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-28 Thread Duncan Thomas
We need mutual exclusion for several operations. Whether that is done by entity queues, locks, state based locking at the api later, or something else, we need mutual exclusion. Our current api does not lend itself to looser consistency, and I struggle to come up with a sane api that does -

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-27 Thread Duncan Thomas
We are working on some sort of distributed replacement for the locks in cinder, since file locks are limiting our ability to do HA. I'm afraid you're unlikely to get any traction until that work is done. I also have a concern that some backend do not handle load well, and so benefit from the

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-27 Thread Joshua Harlow
Duncan Thomas wrote: We are working on some sort of distributed replacement for the locks in cinder, since file locks are limiting our ability to do HA. I'm afraid you're unlikely to get any traction until that work is done. I also have a concern that some backend do not handle load well, and

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-27 Thread Joshua Harlow
Something that I would prefer is to get the following merged: https://github.com/python-zk/kazoo/pull/306 Then we can have tooz (http://docs.openstack.org/developer/tooz/) have a implementation of that (using the above PR #306 for the kazoo/zookeeper impl) and providing impls of it for the

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-27 Thread John Griffith
On Sat, Jun 27, 2015 at 11:47 AM, Joshua Harlow harlo...@outlook.com wrote: Duncan Thomas wrote: We are working on some sort of distributed replacement for the locks in cinder, since file locks are limiting our ability to do HA. I'm afraid you're unlikely to get any traction until that work

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-27 Thread Joshua Harlow
John Griffith wrote: On Sat, Jun 27, 2015 at 11:47 AM, Joshua Harlow harlo...@outlook.com mailto:harlo...@outlook.com wrote: Duncan Thomas wrote: We are working on some sort of distributed replacement for the locks in cinder, since file locks are limiting our

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-27 Thread Avishay Traeger
Do we really need any of these locks? I'm sure we could come up with some way to remove them, rather than make them distributed. On Sun, Jun 28, 2015 at 5:07 AM, Joshua Harlow harlo...@outlook.com wrote: John Griffith wrote: On Sat, Jun 27, 2015 at 11:47 AM, Joshua Harlow

[openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

2015-06-26 Thread niuzhenguo
Hi folks, Currently we use a lockfile to protect the create operations from concurrent delete the source volume/snapshot, we use exclusive locks on both delete and create sides which will ensure that: 1. If a create of VolA from snap/VolB is in progress, any delete requests for