Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
+1 to going back to file locks, too. We can keep the current scheme under a different API in the module for anyone that wants to use it explicitly, but I think at this point it’s better to have something that works reliably when configured properly as the default. I hope we can switch that to tooz/zookeeper in a future release, but we’ll need some more discussion before making that change. Doug On Aug 20, 2014, at 3:29 PM, Davanum Srinivas wrote: > Ben, +1 to the plan you outlined. > > -- dims > > On Wed, Aug 20, 2014 at 4:13 PM, Ben Nemec wrote: >> On 08/20/2014 01:03 PM, Vishvananda Ishaya wrote: >>> This may be slightly off-topic but it is worth mentioning that the use of >>> threading.Lock[1] >>> which was included to make the locks thread safe seems to be leading to a >>> deadlock in eventlet[2]. >>> It seems like we have rewritten this too many times in order to fix minor >>> pain points and are >>> adding risk to a very important component of the system. >>> >>> [1] https://review.openstack.org/#/c/54581 >>> [2] https://bugs.launchpad.net/nova/+bug/1349452 >> >> This is pretty much why I'm pushing to just revert to the file locking >> behavior we had up until a couple of months ago, rather than >> implementing some new shiny lock thing that will probably cause more >> subtle issues in the consuming projects. It's become clear to me that >> lockutils is too deeply embedded in the other projects, and there are >> too many implementation details that they rely on, to make significant >> changes to its default code path. >> >>> >>> On Aug 18, 2014, at 2:05 PM, Pádraig Brady wrote: >>> On 08/18/2014 03:38 PM, Julien Danjou wrote: > On Thu, Aug 14 2014, Yuriy Taraday wrote: > > Hi Yuriy, > > […] > >> Looking forward to your opinions. > > This looks like a good summary of the situation. > > I've added a solution E based on pthread, but didn't get very far about > it for now. In my experience I would just go with the fcntl locks. They're auto unlocked and well supported, and importantly, supported for distributed processes. I'm not sure how problematic the lock_path config is TBH. That is adjusted automatically in certain cases where needed anyway. Pádraig. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > Davanum Srinivas :: http://davanum.wordpress.com > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
On Wed, Aug 20 2014, Vishvananda Ishaya wrote: > This may be slightly off-topic but it is worth mentioning that the use of > threading.Lock[1] > which was included to make the locks thread safe seems to be leading to a > deadlock in eventlet[2]. > It seems like we have rewritten this too many times in order to fix minor > pain points and are > adding risk to a very important component of the system. Indeed it looks slightly off-topic as it actually looks like more the never ending nightmare of eventlet monkey patching we're trying to get rid off… :( -- Julien Danjou # Free Software hacker # http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
Ben, +1 to the plan you outlined. -- dims On Wed, Aug 20, 2014 at 4:13 PM, Ben Nemec wrote: > On 08/20/2014 01:03 PM, Vishvananda Ishaya wrote: >> This may be slightly off-topic but it is worth mentioning that the use of >> threading.Lock[1] >> which was included to make the locks thread safe seems to be leading to a >> deadlock in eventlet[2]. >> It seems like we have rewritten this too many times in order to fix minor >> pain points and are >> adding risk to a very important component of the system. >> >> [1] https://review.openstack.org/#/c/54581 >> [2] https://bugs.launchpad.net/nova/+bug/1349452 > > This is pretty much why I'm pushing to just revert to the file locking > behavior we had up until a couple of months ago, rather than > implementing some new shiny lock thing that will probably cause more > subtle issues in the consuming projects. It's become clear to me that > lockutils is too deeply embedded in the other projects, and there are > too many implementation details that they rely on, to make significant > changes to its default code path. > >> >> On Aug 18, 2014, at 2:05 PM, Pádraig Brady wrote: >> >>> On 08/18/2014 03:38 PM, Julien Danjou wrote: On Thu, Aug 14 2014, Yuriy Taraday wrote: Hi Yuriy, […] > Looking forward to your opinions. This looks like a good summary of the situation. I've added a solution E based on pthread, but didn't get very far about it for now. >>> >>> In my experience I would just go with the fcntl locks. >>> They're auto unlocked and well supported, and importantly, >>> supported for distributed processes. >>> >>> I'm not sure how problematic the lock_path config is TBH. >>> That is adjusted automatically in certain cases where needed anyway. >>> >>> Pádraig. >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
On 08/20/2014 01:03 PM, Vishvananda Ishaya wrote: > This may be slightly off-topic but it is worth mentioning that the use of > threading.Lock[1] > which was included to make the locks thread safe seems to be leading to a > deadlock in eventlet[2]. > It seems like we have rewritten this too many times in order to fix minor > pain points and are > adding risk to a very important component of the system. > > [1] https://review.openstack.org/#/c/54581 > [2] https://bugs.launchpad.net/nova/+bug/1349452 This is pretty much why I'm pushing to just revert to the file locking behavior we had up until a couple of months ago, rather than implementing some new shiny lock thing that will probably cause more subtle issues in the consuming projects. It's become clear to me that lockutils is too deeply embedded in the other projects, and there are too many implementation details that they rely on, to make significant changes to its default code path. > > On Aug 18, 2014, at 2:05 PM, Pádraig Brady wrote: > >> On 08/18/2014 03:38 PM, Julien Danjou wrote: >>> On Thu, Aug 14 2014, Yuriy Taraday wrote: >>> >>> Hi Yuriy, >>> >>> […] >>> Looking forward to your opinions. >>> >>> This looks like a good summary of the situation. >>> >>> I've added a solution E based on pthread, but didn't get very far about >>> it for now. >> >> In my experience I would just go with the fcntl locks. >> They're auto unlocked and well supported, and importantly, >> supported for distributed processes. >> >> I'm not sure how problematic the lock_path config is TBH. >> That is adjusted automatically in certain cases where needed anyway. >> >> Pádraig. >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
This may be slightly off-topic but it is worth mentioning that the use of threading.Lock[1] which was included to make the locks thread safe seems to be leading to a deadlock in eventlet[2]. It seems like we have rewritten this too many times in order to fix minor pain points and are adding risk to a very important component of the system. [1] https://review.openstack.org/#/c/54581 [2] https://bugs.launchpad.net/nova/+bug/1349452 On Aug 18, 2014, at 2:05 PM, Pádraig Brady wrote: > On 08/18/2014 03:38 PM, Julien Danjou wrote: >> On Thu, Aug 14 2014, Yuriy Taraday wrote: >> >> Hi Yuriy, >> >> […] >> >>> Looking forward to your opinions. >> >> This looks like a good summary of the situation. >> >> I've added a solution E based on pthread, but didn't get very far about >> it for now. > > In my experience I would just go with the fcntl locks. > They're auto unlocked and well supported, and importantly, > supported for distributed processes. > > I'm not sure how problematic the lock_path config is TBH. > That is adjusted automatically in certain cases where needed anyway. > > Pádraig. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
On 08/20/2014 01:40 AM, Angus Lees wrote: > On Mon, 18 Aug 2014 10:05:28 PM Pádraig Brady wrote: >> On 08/18/2014 03:38 PM, Julien Danjou wrote: >>> On Thu, Aug 14 2014, Yuriy Taraday wrote: >>> >>> Hi Yuriy, >>> >>> […] >>> Looking forward to your opinions. >>> >>> This looks like a good summary of the situation. >>> >>> I've added a solution E based on pthread, but didn't get very far about >>> it for now. >> >> In my experience I would just go with the fcntl locks. >> They're auto unlocked and well supported, and importantly, >> supported for distributed processes. >> >> I'm not sure how problematic the lock_path config is TBH. >> That is adjusted automatically in certain cases where needed anyway. >> >> Pádraig. > > I added a "C2" (fcntl locks on byte ranges) option to the etherpad that I > believe addresses all the issues raised.I must be missing something > because it seems too obvious to have not been considered before :/ > I've added some additional comments to the etherpad, but the only problem I'm aware of with using lock files was that it requires lock_path to be set and have secure permissions. Whether we lock entire files or byte ranges, that is still true. Also, we still need the ability to use different lock files because some locks use a specific file path that isn't in our configured lock_path, and this is necessary behavior until we can replace it with tooz or something. Also, I believe it has the same hash collision issue as sysv semaphores. While it's hugely unlikely that a hash would collide in the number of bits we're dealing with, an unlucky collision would be a nightmare to debug. All things being equal I'd prefer to eliminate the chance entirely. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
On Mon, 18 Aug 2014 10:05:28 PM Pádraig Brady wrote: > On 08/18/2014 03:38 PM, Julien Danjou wrote: > > On Thu, Aug 14 2014, Yuriy Taraday wrote: > > > > Hi Yuriy, > > > > […] > > > >> Looking forward to your opinions. > > > > This looks like a good summary of the situation. > > > > I've added a solution E based on pthread, but didn't get very far about > > it for now. > > In my experience I would just go with the fcntl locks. > They're auto unlocked and well supported, and importantly, > supported for distributed processes. > > I'm not sure how problematic the lock_path config is TBH. > That is adjusted automatically in certain cases where needed anyway. > > Pádraig. I added a "C2" (fcntl locks on byte ranges) option to the etherpad that I believe addresses all the issues raised.I must be missing something because it seems too obvious to have not been considered before :/ -- - Gus ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
On 08/18/2014 03:38 PM, Julien Danjou wrote: > On Thu, Aug 14 2014, Yuriy Taraday wrote: > > Hi Yuriy, > > […] > >> Looking forward to your opinions. > > This looks like a good summary of the situation. > > I've added a solution E based on pthread, but didn't get very far about > it for now. In my experience I would just go with the fcntl locks. They're auto unlocked and well supported, and importantly, supported for distributed processes. I'm not sure how problematic the lock_path config is TBH. That is adjusted automatically in certain cases where needed anyway. Pádraig. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
On Thu, Aug 14 2014, Yuriy Taraday wrote: Hi Yuriy, […] > Looking forward to your opinions. This looks like a good summary of the situation. I've added a solution E based on pthread, but didn't get very far about it for now. -- Julien Danjou # Free Software hacker # http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils
Hello. It started (for me) with a bug [1] that stated one problem with POSIX semaphores. I started implementing fix for that [2] and it appeared that there are other issues with lockutils. Discussion got spread over bug [1] and Gerrit [2] and got limited to those who get notifications on them. I'd like to move this discussion to this mailing list and recap all thoughts in etherpad [3] so that everyone will be on one page. I've written my view on possible options we have on etherpad [3], so please take a look at them and share your thoughts on it. I personally prefer doing options B and AB2 (SysV + fix for them) while we can fix file locks with C3 as well. Looking forward to your opinions. [1] https://launchpad.net/bugs/1327946 [2] https://review.openstack.org/108954 [3] https://etherpad.openstack.org/p/lockutils-issues -- Kind regards, Yuriy. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
