Re: [Async-sig] [python-tulip] asyncio.Lock equivalent for multiple processes

[Roberto Martínez]

Hi,

I don't know if there is a third party solution for this.

I think the closest you can get today using the standard library is using a
multiprocessing.manager().Lock (which can be shared among processes) and
call the lock.acquire() function with asyncio.run_in_executor(), using a
ThreadedPoolExecutor to avoid blocking the asyncio event loop.

Best regards,
Roberto


El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc ()
escribió:

> Hi,
>
> I'm looking for a equivalent of asyncio.Lock (
> https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
> shared between several processes on the same server, because I'm migrating
> a daemon from mono-worker to multi-worker pattern.
>
> For now, the closest solution in term of API seems aioredlock:
> https://github.com/joanvila/aioredlock#aioredlock
> But I'm not a big fan to use polling nor with a timeout because the lock I
> need is very critical, I prefer to block the code than unlock with timeout.
>
> Do I miss a new awesome library or do you have an easier approach ?
>
> Thanks for your responses.
> --
> Ludovic Gasc (GMLudo)
>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [python-tulip] asyncio.Lock equivalent for multiple processes

[Chris Jerdonek]

If you're already using PostgreSQL, you might also look at "advisory locks":
https://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS

--Chris

On Tue, Apr 17, 2018 at 4:34 AM, Ludovic Gasc  wrote:
> Hi Nickolai,
>
> Thanks for your suggestions, especially for the file system lock: We don't
> have often locks, but we must be sure it's locked.
>
> For 1) and 4) suggestions, in fact we have several systems to sync and also
> a PostgreSQL transaction, the request must be treated by the same worker
> from beginning to end and the other systems aren't idempotent at all, it's
> "old-school" proprietary systems, good luck to change that ;-)
>
> Regards.
> --
> Ludovic Gasc (GMLudo)
>
> 2018-04-17 12:46 GMT+02:00 Nickolai Novik :
>>
>> Hi, redis lock has own limitations and depending on your use case it may
>> or may not be suitable [1]. If possible I would redefine problem and also
>> considered:
>> 1) create worker per specific resource type to avoid locking
>> 2) optimistic locking
>> 3) File system lock like in twisted, but not sure about performance and
>> edge cases there
>> 4) make operation on resource idempotent
>>
>> [1]
>> http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
>> [2]
>> https://github.com/twisted/twisted/blob/e38cc25a67747899c6984d6ebaa8d3d134799415/src/twisted/python/lockfile.py
>>
>> On Tue, 17 Apr 2018 at 13:01 Ludovic Gasc  wrote:
>>>
>>> Hi Roberto,
>>>
>>> Thanks for the pointer, it's exactly the type of feedbacks I'm looking
>>> for: Ideas that are out-of-box of my confort zone.
>>> However, in our use case, we are using gunicorn, that uses forks instead
>>> of multiprocessing to my knowledge, I can't use multiprocessing without to
>>> remove gunicorn.
>>>
>>> If somebody is using aioredlock in his project, I'm interested by
>>> feedbacks.
>>>
>>> Have a nice week.
>>>
>>>
>>> --
>>> Ludovic Gasc (GMLudo)
>>>
>>> 2018-04-17 7:19 GMT+02:00 Roberto Martínez :


 Hi,

 I don't know if there is a third party solution for this.

 I think the closest you can get today using the standard library is
 using a multiprocessing.manager().Lock (which can be shared among 
 processes)
 and call the lock.acquire() function with asyncio.run_in_executor(), using 
 a
 ThreadedPoolExecutor to avoid blocking the asyncio event loop.

 Best regards,
 Roberto


 El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc ()
 escribió:
>
> Hi,
>
> I'm looking for a equivalent of asyncio.Lock
> (https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
> shared between several processes on the same server, because I'm 
> migrating a
> daemon from mono-worker to multi-worker pattern.
>
> For now, the closest solution in term of API seems aioredlock:
> https://github.com/joanvila/aioredlock#aioredlock
> But I'm not a big fan to use polling nor with a timeout because the
> lock I need is very critical, I prefer to block the code than unlock with
> timeout.
>
> Do I miss a new awesome library or do you have an easier approach ?
>
> Thanks for your responses.
> --
> Ludovic Gasc (GMLudo)
>>>
>>>
>
>
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [python-tulip] asyncio.Lock equivalent for multiple processes

[Ludovic Gasc]

Hi Nickolai,

Thanks for your suggestions, especially for the file system lock: We don't
have often locks, but we must be sure it's locked.

For 1) and 4) suggestions, in fact we have several systems to sync and also
a PostgreSQL transaction, the request must be treated by the same worker
from beginning to end and the other systems aren't idempotent at all, it's
"old-school" proprietary systems, good luck to change that ;-)

Regards.
--
Ludovic Gasc (GMLudo)

2018-04-17 12:46 GMT+02:00 Nickolai Novik :

> Hi, redis lock has own limitations and depending on your use case it may
> or may not be suitable [1]. If possible I would redefine problem and also
> considered:
> 1) create worker per specific resource type to avoid locking
> 2) optimistic locking
> 3) File system lock like in twisted, but not sure about performance and
> edge cases there
> 4) make operation on resource idempotent
>
> [1] http://martin.kleppmann.com/2016/02/08/how-to-do-
> distributed-locking.html
> [2] https://github.com/twisted/twisted/blob/e38cc25a67747899c6984d6ebaa8d3
> d134799415/src/twisted/python/lockfile.py
>
> On Tue, 17 Apr 2018 at 13:01 Ludovic Gasc  wrote:
>
>> Hi Roberto,
>>
>> Thanks for the pointer, it's exactly the type of feedbacks I'm looking
>> for: Ideas that are out-of-box of my confort zone.
>> However, in our use case, we are using gunicorn, that uses forks instead
>> of multiprocessing to my knowledge, I can't use multiprocessing without to
>> remove gunicorn.
>>
>> If somebody is using aioredlock in his project, I'm interested by
>> feedbacks.
>>
>> Have a nice week.
>>
>>
>> --
>> Ludovic Gasc (GMLudo)
>>
>> 2018-04-17 7:19 GMT+02:00 Roberto Martínez :
>>
>>>
>>> Hi,
>>>
>>> I don't know if there is a third party solution for this.
>>>
>>> I think the closest you can get today using the standard library is
>>> using a multiprocessing.manager().Lock (which can be shared among
>>> processes) and call the lock.acquire() function with
>>> asyncio.run_in_executor(), using a ThreadedPoolExecutor to avoid blocking
>>> the asyncio event loop.
>>>
>>> Best regards,
>>> Roberto
>>>
>>>
>>> El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc ()
>>> escribió:
>>>
 Hi,

 I'm looking for a equivalent of asyncio.Lock (
 https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
 shared between several processes on the same server, because I'm migrating
 a daemon from mono-worker to multi-worker pattern.

 For now, the closest solution in term of API seems aioredlock:
 https://github.com/joanvila/aioredlock#aioredlock
 But I'm not a big fan to use polling nor with a timeout because the
 lock I need is very critical, I prefer to block the code than unlock with
 timeout.

 Do I miss a new awesome library or do you have an easier approach ?

 Thanks for your responses.
 --
 Ludovic Gasc (GMLudo)

>>>
>>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [python-tulip] asyncio.Lock equivalent for multiple processes

[Nickolai Novik]

Hi, redis lock has own limitations and depending on your use case it may or
may not be suitable [1]. If possible I would redefine problem and also
considered:
1) create worker per specific resource type to avoid locking
2) optimistic locking
3) File system lock like in twisted, but not sure about performance and
edge cases there
4) make operation on resource idempotent

[1]
http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
[2]
https://github.com/twisted/twisted/blob/e38cc25a67747899c6984d6ebaa8d3d134799415/src/twisted/python/lockfile.py


On Tue, 17 Apr 2018 at 13:01 Ludovic Gasc  wrote:

> Hi Roberto,
>
> Thanks for the pointer, it's exactly the type of feedbacks I'm looking
> for: Ideas that are out-of-box of my confort zone.
> However, in our use case, we are using gunicorn, that uses forks instead
> of multiprocessing to my knowledge, I can't use multiprocessing without to
> remove gunicorn.
>
> If somebody is using aioredlock in his project, I'm interested by
> feedbacks.
>
> Have a nice week.
>
>
> --
> Ludovic Gasc (GMLudo)
>
> 2018-04-17 7:19 GMT+02:00 Roberto Martínez :
>
>>
>> Hi,
>>
>> I don't know if there is a third party solution for this.
>>
>> I think the closest you can get today using the standard library is using
>> a multiprocessing.manager().Lock (which can be shared among processes) and
>> call the lock.acquire() function with asyncio.run_in_executor(), using a
>> ThreadedPoolExecutor to avoid blocking the asyncio event loop.
>>
>> Best regards,
>> Roberto
>>
>>
>> El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc ()
>> escribió:
>>
>>> Hi,
>>>
>>> I'm looking for a equivalent of asyncio.Lock (
>>> https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
>>> shared between several processes on the same server, because I'm migrating
>>> a daemon from mono-worker to multi-worker pattern.
>>>
>>> For now, the closest solution in term of API seems aioredlock:
>>> https://github.com/joanvila/aioredlock#aioredlock
>>> But I'm not a big fan to use polling nor with a timeout because the lock
>>> I need is very critical, I prefer to block the code than unlock with
>>> timeout.
>>>
>>> Do I miss a new awesome library or do you have an easier approach ?
>>>
>>> Thanks for your responses.
>>> --
>>> Ludovic Gasc (GMLudo)
>>>
>>
>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [python-tulip] asyncio.Lock equivalent for multiple processes

[Ludovic Gasc]

Hi Roberto,

Thanks for the pointer, it's exactly the type of feedbacks I'm looking for:
Ideas that are out-of-box of my confort zone.
However, in our use case, we are using gunicorn, that uses forks instead of
multiprocessing to my knowledge, I can't use multiprocessing without to
remove gunicorn.

If somebody is using aioredlock in his project, I'm interested by feedbacks.

Have a nice week.


--
Ludovic Gasc (GMLudo)

2018-04-17 7:19 GMT+02:00 Roberto Martínez :

>
> Hi,
>
> I don't know if there is a third party solution for this.
>
> I think the closest you can get today using the standard library is using
> a multiprocessing.manager().Lock (which can be shared among processes) and
> call the lock.acquire() function with asyncio.run_in_executor(), using a
> ThreadedPoolExecutor to avoid blocking the asyncio event loop.
>
> Best regards,
> Roberto
>
>
> El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc ()
> escribió:
>
>> Hi,
>>
>> I'm looking for a equivalent of asyncio.Lock (https://docs.python.org/3/
>> library/asyncio-sync.html#asyncio.Lock) but shared between several
>> processes on the same server, because I'm migrating a daemon from
>> mono-worker to multi-worker pattern.
>>
>> For now, the closest solution in term of API seems aioredlock:
>> https://github.com/joanvila/aioredlock#aioredlock
>> But I'm not a big fan to use polling nor with a timeout because the lock
>> I need is very critical, I prefer to block the code than unlock with
>> timeout.
>>
>> Do I miss a new awesome library or do you have an easier approach ?
>>
>> Thanks for your responses.
>> --
>> Ludovic Gasc (GMLudo)
>>
>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/