[issue8800] add threading.RWLock

Andrew Svetlov added the comment: Thanks for sharing, your project looks viable! -- ___ Python tracker ___ ___ Python-bugs-list

[issue8800] add threading.RWLock

Éric Larivière added the comment: Hi, Sorry to wake up a 10 years old discussion But I think that you might be interested in the following Python package that I created and maintain since few years now: https://pypi.org/project/readerwriterlock/ 1- It implements the three type of

[issue8800] add threading.RWLock

Change by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing

[issue8800] add threading.RWLock

Nathaniel Smith added the comment: Side note, in case anyone else finds themselves looking for the MSDN magazine article referenced above: it appears to be in the June 2007 issue, titled "CONCURRENCY: Synchronization Primitives New To Windows Vista". For the moment at least, this link seems

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Seems to have fizzled out due to the intense amount of bikeshedding required. -- ___ Python tracker ___

[issue8800] add threading.RWLock

Ofek Lev added the comment: What is the status of the patch? -- nosy: +Ofekmeister ___ Python tracker ___ ___

[issue8800] add threading.RWLock

Changes by Dan O'Reilly oreil...@gmail.com: -- nosy: +dan.oreilly ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___ ___ Python-bugs-list

[issue8800] add threading.RWLock

Changes by Vladimir Rutsky rutsky.vladi...@gmail.com: -- nosy: +vrutsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___ ___

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Thanks Richard. I wonder if the issues with the multiprocessing tests can be fixed by making use of Barriers? One of the reasons I introduced Barriers into the lib originally was my alarm at seeing the various springklings of different _wait() calls

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: although I'd prefer a BSD errno example, such as ECONNRESET, instead of a winsock one Are you referring to the comment where I mention ECONNABORTED? That is a regular unix error (windows version is WSAECONNABORTED). This error occurs when the local

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: Kristjan: you seem to have attached socketserver.patch to the wrong issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___

[issue8800] add threading.RWLock

Changes by Kristján Valur Jónsson krist...@ccpgames.com: Removed file: http://bugs.python.org/file27430/socketserver.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___

[issue8800] add threading.RWLock

Changes by Kristján Valur Jónsson krist...@ccpgames.com: -- Removed message: http://bugs.python.org/msg172065 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: Le vendredi 05 octobre 2012 à 09:51 +, Kristján Valur Jónsson a écrit : Then I started to wonder if it were appropriate to use a Barrier in the Condition variable tests, particularly given that the former is implemented by way of the latter :) Indeed,

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Ok, here is a new patch. It takes into account various comments and suggestions: 1) The interface is java-like. a single RWLock instance only has attributes reader_lock and writer_lock. 2) Since the owning array needs only be process local, the same

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: Attached is a new version of Kristjan's patch with support for managers. (A threading._RWLockCore object is proxied and wrapped in a local instance of a subclass of threading.RWLock.) Also I made multiprocessing.RWLock.__init__() use

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: Fixed patch because I didn't test on Unix... -- Added file: http://bugs.python.org/file27422/rwlock-sbt.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800

[issue8800] add threading.RWLock

Changes by Richard Oudkerk shibt...@gmail.com: Removed file: http://bugs.python.org/file27421/rwlock-sbt.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: I admit that I kind of like Java's approach to this. First off, they define an interface, ReadWriteLock: http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html There, they also discuss the different choices an

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: Multiprocessing: Because there is no way I know to share a list of owning thread ids, this version is more limited Why do you need a *shared* list? I think it should be fine to use a per-process list of owning thread ids. So the current thread owns the

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Excellent point, I hadn't thought of that! Yes, it is is sufficient to test if _I_ am in the list. I'll make the necessary changes. That will make the thread/process implementation virtually identical. --

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: I don't like the API. Why should I write rwlock.acquire_read() instead of rwlock.reader.acquire()? There should be one obvious way to do it. The fact that reader_lock and writer_lock return a new object every time is a bit suboptimal, IMO. Also, RWLockBase

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: By the way, the fact that acquire_read() is documented as Acquire the lock in shared mode and acquire_write() as Acquire the lock in exclusive mode hints that perhaps RWLock is not the right name ;) -- ___ Python

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: You are not restricted to the context manager model. Just use selock.shared.acquire() or selock.exclusive.acquire(). The unlock operation is the same, so now you have to arbitrarily pick one of the lockd and chose release(). Why take a construct

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: We've already departed from that. Our Lock is nothing like a mutex, for example (it's more of a binary semaphore). This is not by nature of good design, but an accident. C python needed both mutex and signaling ability and decided that a single

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: I have implemented the simplest possible acquisition order. The lock acquired first will be granted first. Without that (or a more advanced policy) in applications with concurrent threads/processes that are heavily using the shared lock, the

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: I think you got that argument backwards. The simple greedy policy you implement works well provided there are not too many readers. Otherwise, the writers will be starved, since they have to wait for an oppertune moment when no readers are active to get

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: The unlock operation is the same, so now you have to arbitrarily pick one of the lockd and chose release(). That depends on the implementation. In the three implementations on http://en.wikipedia.org/wiki/Readers-writers_problem the unlock

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Here is a new patch. it is complete with: threading implementation and tests multiprocessing implementation and tests. Let's leave the naming bikeshedding a bit and focus on some practical aspects: 1) The threading version contains a RWLock and a

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Oh, I forgot to mention: Once one gets into the domain of allowing such niceties as writer priority, surely you can agree that the implementation of both locking modes belongs in the same class instance. That is just plain good coding practice,

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Exactly, with my implemantation the lock acquired first will be granted first. There is no way that either shared nor exclusive locks can starve, and therefore it should satisfy all use cases. Since you can only share simple datastructures like integers

[issue8800] add threading.RWLock

Sebastian Noack added the comment: @Kristján: Uhh, that is a huge amount of code, more than twice as much (don't counting tests) as my implementation, to accomplish the same. And it seems that there is not much code shared between the threading and multiprocessing implementation. And for

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: This amount of code provides recursion, context managers, condition variable compatibility, timeout functionality, error checking and conformance with the unit tests. The actual locking code is encapsulated in the three functions acquire_read(),

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Ah, you are implementing an FIFO lock. That should have been made clear. I see it now that you grant the lock in the order that the acquisition is attempted. Ok, this is fine, but with one important caveat: Explicit handoff such as that can suffer

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Yes, you could also look at the shared/exclusive lock as one lock with different states. But this approach is neither more common, have a look at Java's ReadWriteLock [1] for example, which works just like my patch does, except that a factory is returned

[issue8800] add threading.RWLock

Sebastian Noack added the comment: @richard: I'm sorry, but both of my patches contain changes to 'Lib/threading.py' and can be applied on top of Python 3.3.0. So can you explain what do you mean, by missing the changes to threading.py? -- ___

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Personally, I would prefer to make the shared and exclusive locks attributes of the same object, so one could do with selock.shared: ... with selock.exclusive: ... Please note, the same object could simply be a namedtuple

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: shared/exclusive - abstract description of what it is If you want to argue it this way, I counter that the attributes shared and exclusive apply to the type of access to the protected object you are talking about, and yet, the name suggest that they

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: @richard: I'm sorry, but both of my patches contain changes to 'Lib/threading.py' and can be applied on top of Python 3.3.0. So can you explain what do you mean, by missing the changes to threading.py? I was reading the Rietveld review page

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: With this, you are stuck with employing a context manager model only. You loose the flexibility to do explicit acquire_read() or acquire_write(). You are not restricted to the context manager model. Just use selock.shared.acquire() or

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Perhaps I should have pointed out, for Sebastian's benefit, that my second patch uses timeout rather than blocking since that is the new black in python 3. Also, I think the threading implementation shows clearly the problem of having two independent

[issue8800] add threading.RWLock

Sebastian Noack added the comment: If you want to argue it this way, I counter that the attributes shared and exclusive apply to the type of access to the protected object you are talking about, and yet, the name suggest that they are attributes of the lock itself. A lock's sole purpose is

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: I think Sebastian's algorithm does not correctly deal with the non-blocking case. Consider the following situation: * Thread-1 successfully acquires exclusive lock. Now num_got_lock == 1. * Thread-2 blocks waiting for shared lock. Will block until

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: My previous comment applied to Sebastian's first patch. The second seems to fix the issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___

[issue8800] add threading.RWLock

Christian Heimes added the comment: I wonder, why are you creating your own algorithm here? There must be plenty of reference implementations that are already used in production code. Don't be a shamed to copy a Java implementation! :) The entire threading module is a rip-off of the Java

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I would love to see how other people would implement a shared/exclusive lock that can be acquired from different processes. However it really seems that nobody did it before. If you know a reference implementation I would be more than happy. There are

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: On the naming front: shorthands like Shrd and Excl are a bit frown upon. Since SharedExclusiveLock is on the long side, I would suggest calling the API SELock. -- ___ Python tracker rep...@bugs.python.org

[issue8800] add threading.RWLock

Christian Heimes added the comment: A RW lock is part of POSIX threads [1]. It's usually a good idea to either use POSIX functions or to mimic their behavior. After all POSIX is an industry standard. Boost and Java have several lock and rw lock implementations. Wikipedia [2] is a good

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: A RW lock is part of POSIX threads [1]. It's usually a good idea to either use POSIX functions or to mimic their behavior. After all POSIX is an industry standard. We've already departed from that. Our Lock is nothing like a mutex, for example (it's more of

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Thanks, but as I already said there are a lot of implementations for shared/exclusive lock that can be acquired from different threads. But we need with threading as well as with multiprocessing. And by the way POSIX is the standard for implementing

[issue8800] add threading.RWLock

Charles-François Natali added the comment: The page also mentions a seqlock which looks interesting to me as it's fast for few writers with lots of readers. A seqlock is suitable for consistent views of simple data structures (e.g. a counter in the Linux kernel), but it won't fly for a

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Yes, any blocking on the internal lock is not semantic blocking due to the use of the Shared lock itself, but merely a technical aspect, not semantically visible to the program. It is equivalent to the operating system pausing the thread, a techical

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: I've added a new patch, that implements a shared/exclusive lock as described in my comments above, for the threading and multiprocessing module. The patch does not seem to touch the threading mode and does not come with tests. I doubt the

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: I can't say that I'm that familiar with multiprocessing to comment on that in particular. But I do find your approach strange, to create two lock-like objects, in stead of the more familiar construct of having a RWLock (this is known from other

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I was just waiting for a comment pointing out, that my patch comes without tests. :) Note that we are still discussing the implementation and this patch is just a proof of concept. And since the way it is implemented and the API it provides could still

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: A shared/exclusive lock isn't one lock but two locks, which are synchronized, but must be acquired separately. Similar to a pipe, which isn't one file, but one file connected to another file that reads whatever you have written into the first file. So

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: Ok, since we're bikeshedding, I like Richard's proposal best: Personally, I would prefer to make the shared and exclusive locks attributes of the same object, so one could do with selock.shared: ... with selock.exclusive: ... Please

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: @Sebastian: Both your patch sets are missing the changes to threading.py. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I would love to see a reader/writer lock implementation shipped with Python's threading (and multiprocessing) module. But I have some issues with the patch: 1. I would avoid the terms 'read' and 'write' as those terms are referring just to one of many use

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Excellent points. For 3. however, I don't consider access to synchronized state to be blocking. Blocking means to block while waiting for the lock. The internal lock is never held for any amount of time. Perhaps I'll cook up a new patch with these

[issue8800] add threading.RWLock

Changes by Christian Heimes li...@cheimes.de: -- versions: +Python 3.4 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___ ___

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Using a lock as context manager is the same as calling lock.acquire(blocking=True) and it will in fact block while waiting for an other thread to release the lock. In your code, the internal lock is indeed just hold for a very short period of time while

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I've added a new patch, that implements a shared/exclusive lock as described in my comments above, for the threading and multiprocessing module. -- Added file: http://bugs.python.org/file27350/Added-ShrdExclLock-to-threading-and-multiprocessing.patch

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: I should add that on Windows, the new SRW that is part of Vista and Windows 7, uses locking, that is it favors neither readers or writers. It appears that nowadays the complex semantics of RWLocks have not really proven worthwile. See

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: Perhaps this proposed patch is overly complex. I don't know. Do you think relaxing the semantics would make the implementation significantly faster? (interesting link, btw) -- ___ Python tracker

[issue8800] add threading.RWLock

Changes by Andrew Svetlov andrew.svet...@gmail.com: -- nosy: +asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___ ___ Python-bugs-list

[issue8800] add threading.RWLock

Changes by Matthew Lauber m...@mklauber.com: -- nosy: +mklauber ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___ ___ Python-bugs-list

[issue8800] add threading.RWLock

Changes by Antoine Pitrou pit...@free.fr: -- nosy: +jyasskin stage: - patch review versions: +Python 3.2 -Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8800 ___

[issue8800] add threading.RWLock

Jeffrey Yasskin jyass...@gmail.com added the comment: This patch doesn't apply cleanly against the py3k tree. Since Python 2.7 is in beta, and there's no 2.8, this can only go into python 3, so you should work against that tree. It's a bit annoying that the R in RWLock stands for a different

[issue8800] add threading.RWLock

Antoine Pitrou pit...@free.fr added the comment: I'm not sure it's a good idea to have a default context manager, and default acquire/release methods. In the face of ambiguity, refuse the temptation to guess. Is there any use in being able to use a RWLock inside a condition variable?

[issue8800] add threading.RWLock

Jeffrey Yasskin jyass...@gmail.com added the comment: In this case, acquire isn't ambiguous. All the other lock types actually acquire a write lock, so it makes sense to have the operation with the same name they use also acquire a write lock on this object. I realized that read/write locks

[issue8800] add threading.RWLock

Antoine Pitrou pit...@free.fr added the comment: In this case, acquire isn't ambiguous. All the other lock types actually acquire a write lock, so it makes sense to have the operation with the same name they use also acquire a write lock on this object. Well, it looks like a strange kind of

[issue8800] add threading.RWLock

Jeffrey Yasskin jyass...@gmail.com added the comment: You're definitely right that posix and java don't make these usable from the normal lock API. And I don't think it's essential that they be usable as RLocks, although it's nice for Conditions. I think what I'm actually saying is that, if