subject:"\[issue8800\] add threading.RWLock"


Kristján Valur Jónsson added the comment:

Thanks Richard.
I wonder if the issues with the multiprocessing tests can be fixed by making 
use of Barriers?

One of the reasons I introduced Barriers into the lib originally was my alarm 
at seeing the various springklings of different _wait() calls in the unittests, 
particularly in the threading module (there are Condition variable tests there 
still that have race conditions, that are cunnrently hidden by liberal use of 
_wait())

Then I started to wonder if it were appropriate to use a Barrier in the 
Condition variable tests, particularly given that the former is implemented by 
way of the latter :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Kristján Valur Jónsson added the comment:

 although I'd prefer a BSD errno example, such as ECONNRESET, instead of a 
 winsock one
Are you referring to the comment where I mention ECONNABORTED?  That is a 
regular unix error (windows version is WSAECONNABORTED).
This error occurs when the local stack decides to kill the connection, as 
opposed to ECONNRESET which occurs on account of receiving a RST packet.

New patch includes documentation change.

If 2.7 is still in bugfix mode, then this patch could probably be accepted.

--
Added file: http://bugs.python.org/file27430/socketserver.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-05 Thread Richard Oudkerk


Richard Oudkerk added the comment:

Kristjan: you seem to have attached socketserver.patch to the wrong issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Changes by Kristján Valur Jónsson krist...@ccpgames.com:


Removed file: http://bugs.python.org/file27430/socketserver.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Changes by Kristján Valur Jónsson krist...@ccpgames.com:


--
Removed message: http://bugs.python.org/msg172065

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-05 Thread Antoine Pitrou


Antoine Pitrou added the comment:

Le vendredi 05 octobre 2012 à 09:51 +, Kristján Valur Jónsson a
écrit :
 Then I started to wonder if it were appropriate to use a Barrier in
 the Condition variable tests, particularly given that the former is
 implemented by way of the latter :)

Indeed, the reason I wrote the lock tests that way is that I want the
tests to be independent of the primitives under test. Otherwise things
become very messy.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-04 Thread Kristján Valur Jónsson


Kristján Valur Jónsson added the comment:

Ok, here is a new patch.  It takes into account various comments and 
suggestions:
1) The interface is java-like. a single RWLock instance only has attributes 
reader_lock and writer_lock.
2) Since the owning array needs only be process local, the same semantics can 
be used for multiprocessing.  Multiprocessing version only does minimal 
patching of the threading classes, similarly to Barrier.  There is 
implementation coupling between threading and multiprocessing but that is 
probably ok.
3) Test are there, except that the final tests in the multiprocessing, those 
that used to rely on data gathered in lists, are nerfed.  Not sure how to 
gather data when multiprocessing
4) I also didn't get the manager tests to work.  I don't know what managers 
are or what they are supposed to do, and why they use proxy objects.  I 
attempted to create a proxy object in managers.py but it failed the tests.  
Also, I am unsure if it is sufficient to proxy only the RWLock object, or if 
the _ReaderLock and _WriterLock instances that are returned should be proxied 
too.

Documentation i stole from Java.
The locking policy is the same as the RecursiveReaderWiterLock from java, i.e. 
readers wait for writers, but otherwise no preference.  Recursion is provided 
for readers and writers, but not upgrade/downgrade.

Again, this is not intended to be final code, particularly I would like help 
and suggestions for the multiprocessing tests and the manager code.

Also, while I strongly prefer RWLock, or ReaderWriterLock, or something 
similar, basically because it fits with our already java-ized threading module 
and the terminology from everywhere else in the field today, I'm not going to 
be enormously difficult about it.

cheers!

--
Added file: http://bugs.python.org/file27412/rwlock.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-04 Thread Richard Oudkerk


Richard Oudkerk added the comment:

Attached is a new version of Kristjan's patch with support for managers.  (A 
threading._RWLockCore object is proxied and wrapped in a local instance of a 
subclass of threading.RWLock.)

Also I made multiprocessing.RWLock.__init__() use 
multiprocessing.util.register_after_fork() to clear self._owners.  Otherwise on 
Unix you can run into trouble if a forked processes starts a new thread whose 
id was in self._owners at the time of the fork.

I found that test_many_readers() and test_recursion() tended to fail when run 
with processes (on Windows).  This is because starting a process on Windows is 
slow, particularly with a debug build.  (Some of the buildbots running Windows 
in a VM can be crazily slow.)  Each reader only held the lock for 0.02 secs 
which is much less than the time to start a process on Windows.  This meant 
that it was easy to never have overlapping ownership, causing the tests to fail.

I fixed this by starting the readers while holding an exclusive, waiting for 
period, and then releasing the exclusive lock.  This makes it possible to change

self.assertTrue(max(nlocked)  1)

to

self.assertEqual(max(nlocked), N)

Choosing timeouts to keep the buildbots happy can be a pain:-(

--
Added file: http://bugs.python.org/file27421/rwlock-sbt.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-04 Thread Richard Oudkerk


Richard Oudkerk added the comment:

Fixed patch because I didn't test on Unix...

--
Added file: http://bugs.python.org/file27422/rwlock-sbt.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-04 Thread Richard Oudkerk


Changes by Richard Oudkerk shibt...@gmail.com:


Removed file: http://bugs.python.org/file27421/rwlock-sbt.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-03 Thread Kristján Valur Jónsson


Kristján Valur Jónsson added the comment:

I admit that I kind of like Java's approach to this.  First off, they define an 
interface, ReadWriteLock:
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html
There, they also discuss the different choices an implementation of the 
interface can make, regarding a) policy, b) reentrancy, c) upgrading or 
downgrading.
A concrete implemention is then presented in the form of the 
ReentrantReadWriteLock, with documented behaviour for the above.
The rest of threading is also, as previously pointed out, more or less a 
rip-off from Java.

Since there is no single correct choice for the above, and since the 
implementation restrictions are different between inter-process and 
inter-thread locks, it would make sense to adopt a similar model, where a 
RWLock() function is a factory function, taking an argument specify a desired 
class of locks.

The policies that have been seen in this thread are:
a) greedy policy (no policy)
b) writer preference
c) 'Fair' (or in-order) preference.
All have their benefits and disadvantages.

We have also seen recursive and nonrecursive.

The restrictions appear more serious in the inter-process case since I don't 
know if it is possible to maintain a shared dynamic array of thread ids across 
processes.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-03 Thread Richard Oudkerk


Richard Oudkerk added the comment:

 Multiprocessing:  Because there is no way I know to share a list of 
 owning thread ids, this version is more limited

Why do you need a *shared* list?  I think it should be fine to use a 
per-process list of owning thread ids.   So the current thread owns the lock if 
and only if it is in the current process's list of owners.

(On Unix you should probably clear the list when you fork by using 
multiprocessing.util.register_after_fork() in the initializer.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-03 Thread Kristján Valur Jónsson


Kristján Valur Jónsson added the comment:

Excellent point, I hadn't thought of that!
Yes, it is is sufficient to test if _I_ am in the list.  I'll make the 
necessary changes. That will make the thread/process implementation virtually 
identical.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-03 Thread Antoine Pitrou


Antoine Pitrou added the comment:

I don't like the API. Why should I write rwlock.acquire_read() instead of 
rwlock.reader.acquire()? There should be one obvious way to do it.

The fact that reader_lock and writer_lock return a new object every time is 
a bit suboptimal, IMO.

Also, RWLockBase should be private (_RWLockBase).

Ah, and the patch needs docs :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-03 Thread Antoine Pitrou


Antoine Pitrou added the comment:

By the way, the fact that acquire_read() is documented as Acquire the lock in 
shared mode and acquire_write() as Acquire the lock in exclusive mode hints 
that perhaps RWLock is not the right name ;)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Kristján Valur Jónsson added the comment:

 You are not restricted to the context manager model.  Just use 
 selock.shared.acquire() or selock.exclusive.acquire().

The unlock operation is the same, so now you have to arbitrarily pick one of 
the lockd and chose release().  Why take a construct which is essentially a 
lock that can be acquired in two different ways and force people to view it as 
separate objects?

I much prefer a simple RWLock primitve, such as is popular in other programming 
environments, and add your convenient pseudo-locks on top.
That way, we are not forcing a certain myopic view of what an RWLock is down 
people's throat.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Kristján Valur Jónsson added the comment:

 We've already departed from that. Our Lock is nothing like a mutex, for
 example (it's more of a binary semaphore).

This is not by nature of good design, but an accident.  C python needed both 
mutex and signaling ability and decided that a single non-recursive lock were 
good enough for that.  This is a debatable choice since all modern systems 
consider these two different needs and provide different primitives to satisfy 
them.  The Lock was then exposed to Python and RLock grafted on top to fix 
the non-recursiveness problem.  Then we added signaling in the form of Events, 
Semaphores and Condition variables.
Had this ben more purposefully designed, then there would be no Lock or RLock 
in threading.py, only a Mutex.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Kristján Valur Jónsson added the comment:

 I have implemented the simplest possible acquisition order.
 The lock acquired first will be granted first. Without that (or a more
 advanced policy) in applications with concurrent threads/processes
 that are heavily using the shared lock, the exclusive lock can never
 be acquired, because of there is always a shared lock acquired and
 before it is released the next shared lock will be acquired.

I think you got that argument backwards.  The simple greedy policy you 
implement works well provided there are not too many readers. Otherwise, the 
writers will be starved, since they have to wait for an oppertune moment when 
no readers are active to get a foot in the door, so to speak.
Your approach is similar to my SimpleSharableLock from my second patch in 
that respect, and also to Microsoft's SRW Locks 
(http://msdn.microsoft.com/en-us/magazine/cc163405.aspx).  They specifically 
state:

 This means that if your application requires that data updates take priority 
over data reads, you might want to consider a different reader/writer lock that 
favors writers.

While the test_threading.py showed ok results with the simple approach, my 
preliminary tests on multiprocessing show that writers need to be given 
priority if they are not to be starved.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-02 Thread Richard Oudkerk


Richard Oudkerk added the comment:

 I think you got that argument backwards.  The simple greedy policy you 
 implement works well provided there are not too many readers. Otherwise, 
 the writers will be starved, since they have to wait for an oppertune 
 moment when no readers are active to get a foot in the door, so to speak.

Actually, I think Sebastian's algorithm attempts to be fair to both readers and 
writers.

If there is a writer waiting then self._wait_count  self._granted_count.  
The writer cannot be prempted by a later reader because the reader would find 
waitno  self._granted_count until after writer has been granted the lock.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-02 Thread Richard Oudkerk


Richard Oudkerk added the comment:

 The unlock operation is the same, so now you have to arbitrarily pick one 
 of the lockd and chose release().

That depends on the implementation.  In the three implementations on

http://en.wikipedia.org/wiki/Readers-writers_problem

the unlock operateration is different for readers and writers.

 Why take a construct which is essentially a lock that can be acquired in two 
 different ways and force people to view it as separate objects?

I don't see why writing

lock.exclusive.acquire()

really requires a different way of thinking compared to writing

lock.exclusive_acquire()

or

lock.acquire_exclusive()

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Kristján Valur Jónsson added the comment:

Here is a new patch.  it is complete with:
threading implementation and tests
multiprocessing implementation and tests.

Let's leave the naming bikeshedding a bit and focus on some practical aspects:

1) The threading version contains a RWLock and a FairRWLock.  Both support 
recursion, although not upgrading from read to write access.  This is achieved 
with a list of 'owning' threads.
2) the Fair version provides 'write' priority, blocking further first 
acquisitions in read mode until no writers are waiting.

Multiprocessing:  Because there is no way I know to share a list of owning 
thread ids, this version is more limited:
a) The RWLock() only contains one owing thread ID.  This allows recursion but 
some error checking present in the threading version cannot be implemented.
b) the FairRWLock() again provides writer priority.  But to do that, we have to 
allow 'recursive' read if a writer is waiting, if we allow recursion at all.  
And for that, we need to know what threds own the lock in read mode.  Since we 
don't have that information, this version of the lock disallows recursion 
alltogether.

Discussion:
Recursion/No recursion?  The Windows SRW locks disallow recursion.  Their 
rationale is here:  http://msdn.microsoft.com/en-us/magazine/cc163405.aspx: 
First, regarding recursive acquires: if the locking policy you’ve designed for 
your application requires that synchronization objects be acquired recursively, 
this is very possibly a red flag telling you to re-examine your locking policy 
to eliminate the recursion. This is our opinion and results from the additional 
overhead of executing the lock acquisition and release code multiple times and, 
perhaps more importantly, because ensuring that the balance of lock releases 
with the lock acquisitions is often difficult to prove correct.

Of course the SRWLock is a slim lock and thus can be forgiven for not 
providing such functionality.

Fair vs Non-fair (Fair is not a good term, writer priority would be 
better).  The reason I'm coming back to this being useful is tests using the 
multiprocessing module.  The tests there show without doubt that a simple 
greedy locking algorithm fails miserably if there are more readers than 
writers.  The test test_writer_success has been adorned with a timer, and the 
simple version completes in 15 seconds, while the fair version completes in 
0.5 seconds.

Good times!

--
Added file: http://bugs.python.org/file27385/rwlock.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Kristján Valur Jónsson added the comment:

Oh, I forgot to mention:  Once one gets into the domain of allowing such 
niceties as writer priority, surely you can agree that the implementation of 
both locking modes belongs in the same class instance.  That is just plain good 
coding practice, allowing for class invariants, state assertions and such 
things.
Two class instances coupled by a bunch of common variables is IMHO neither good 
style nor practice.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-02 Thread Sebastian Noack


Sebastian Noack added the comment:

Exactly, with my implemantation the lock acquired first will be granted 
first. There is no way that either shared nor exclusive locks can starve, and 
therefore it should satisfy all use cases. Since you can only share simple 
datastructures like integers across processes, I also found that this seems to 
be the only policy (except ignoring the acquisition order at all), that can be 
implemented for multiprocessing.

I have also looked at the seqlock algorithm, which seems to be great for use 
cases where the exclusive lock is acquired rather rarely and where your 
reader code is in fact read-only and therefore can be repeated. But in any 
other case a seqlock would break your code. However the algorithm is ultra 
simple and can't be implemented as lock-like object anyway. Though you could 
implement it as context manager, but that would hide the fact that the reader 
code will be repeated. So if you find yourself that a seqlock is that what you 
need for your specific use case, you can just use the algorithm like below:

lock = multiprocessing.Value(0)
count = multiprocessing.Value(0)

def do_read():
  while True:
if count.value % 2:
  continue
data = ...
if count.value % 2:
  continue
return data

def do_write(data):
  with lock:
count.value += 1
# write data
count.value += 1

I have also experimented with implementing a shared/exclusive lock on top of a 
pipe and UNIX file locks (https://gist.github.com/3818148). However it works 
only on Unix and only with processes (not threads). Also it turned out that 
UNIX file locks don't implement an acquisition order. So exclusive locks can 
starve, which renders it useless for most use cases.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-02 Thread Sebastian Noack


Sebastian Noack added the comment:

@Kristján: Uhh, that is a huge amount of code, more than twice as much (don't 
counting tests) as my implementation, to accomplish the same. And it seems that 
there is not much code shared between the threading and multiprocessing 
implementation. And for what? Ah right, to make the API suck as much as the 
Windows API does. Please tell me more about good coding practice. ;)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Kristján Valur Jónsson added the comment:

This amount of code provides recursion, context managers, condition variable 
compatibility, timeout functionality, error checking and conformance with the 
unit tests.

The actual locking code is encapsulated in the three functions acquire_read(), 
acquire_write, release().

The requirements and possibilities between threading and multiprocessing are 
many and multiple.  Sharing the implementation has the drawback of imposing the 
shortcomings and preformance bottlenecks of the multiprocessing implementation 
on the threading implementation, for no good reason.

I't tell you about good programming practice, but I'm too busy trying to 
understand the actual locking policy in your patch.  Sometimes comments in code 
can be helpful.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment:

Ah, you are implementing an FIFO lock. That should have been made clear.
I see it now that you grant the lock in the order that the acquisition is
attempted.
Ok, this is fine, but with one important caveat: Explicit handoff such as that
can suffer from lock convoying. In contested situation this can result in
the protected resource being much less available than it ought to be.

In my dayjob, I write locking primitives for Stackless Python. We used to
employ handoff until we found out that this caused performance problems. All
the locking primitives now used by Eve Online and in Stacklesslib are 'greedy'
and don't attempt fairness. This has resulted in much improvement in resource
usage.

For this reason, I explicitly did not build any such mechanism into my RWLock
implementation. Any thread coming in can claim the lock, provided the policy
(writer priority policy) doesn't kick in.
In those rare cases where a special locking policy such as FIFO fairness is
_required_ it is always possible to construct such a thing in python using e.g.
a queue of condition variables.

For information on this, please see the following resource:
http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspx

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Sebastian Noack added the comment:

Yes, you could also look at the shared/exclusive lock as one lock with 
different states. But this approach is neither more common, have a look at 
Java's ReadWriteLock [1] for example, which works just like my patch does, 
except that a factory is returned instead of a tuple. Nor does it provide any 
of the benefits, I have mentioned before (same API as Lock and RLock, better 
compatibility with existing code an with statement, ability to pass the shared 
or exclusive lock separetly around). But maybe we could satisfy anybody, by 
following Richard's and Antoine's suggestion of returning a named tuple. So you 
could use the ShrdExclLock both ways:

# use a single object
lock = ShrdExclLock()

with lock.shared:
  ...

with lock.exclusive:
  ...

# unpack the the object into two variables and pass them separately around
shrd_lock, excl_lock = ShrdExclLock()

Thread(target=reader, args=(shrd_lock,)).start()
Thread(target=writer, args=(excl_lock,)).start)


The majority of us seems to prefer the terms shared and exclusive. However I 
can't deny that the terms read and write are more common, even though there are 
also notable exmples where the terms shared and exclusive are used [2] [3]. But 
let us ignore how other call it for now, and get to the origin of both set of 
terms, in order to figure out which fits best into Python:

shared/exclusive - abstract description of what it is
read/write   - best known use case

The reason why I prefer the terms shared and exculsive, is that it is more 
distinct and less likely to get misunderstand. Also naming a generic 
implementation after a specific use case is bad API design and I don't know any 
other case where that was done, in the Python core library.


[1] 
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/locks/ReadWriteLock.html
[2] http://www.postgresql.org/docs/9.2/static/explicit-locking.html
[3] http://www.unix.com/man-page/freebsd/9/SX/

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Sebastian Noack added the comment:

@richard: I'm sorry, but both of my patches contain changes to 
'Lib/threading.py' and can be applied on top of Python 3.3.0. So can you 
explain what do you mean, by missing the changes to threading.py?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-01 Thread Kristján Valur Jónsson

Kristján Valur Jónsson added the comment:

Personally, I would prefer to make the shared and exclusive locks
attributes of the same object, so one could do

with selock.shared:
...

with selock.exclusive:
...
Please note, the same object could simply be a namedtuple instance.
With this, you are stuck with employing a context manager model only. You
loose the flexibility to do explicit acquire_read() or acquire_write().
My latest patch has methods shared_lock(), exclusive_lock() that return proxy
lock objects that can be used like context managers like you describe, but you
still have the flexibility of using the lock manually.

As for the bikeshedding, let's look at the list of concrete implementations out
there:
Windows: SRW locks (slim reader writer)
http://msdn.microsoft.com/en-us/library/windows/desktop/aa904937(v=vs.85).aspx
Pthreads: rwlock_t (reader/writer)
http://publib.boulder.ibm.com/iseries/v5r2/ic2924/index.htm?info/apis/users_86.htm
Java: ReadWriteLock,
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html

Am I missing anything?
Don't see why we need to adopt a completly different name or idiom to what
people are used to. Also, note that the java version is quite similar to
Richard's proposal.

[issue8800] add threading.RWLock

2012-10-01 Thread Kristján Valur Jónsson


Kristján Valur Jónsson added the comment:

 shared/exclusive - abstract description of what it is

If you want to argue it this way, I counter that the attributes shared and 
exclusive apply to the type of access to the protected object you are 
talking about, and yet, the name suggest that they are attributes of the lock 
itself.

In that sense, reader lock and writer lock, describe attributes of the user 
of the lock, and the verbs readlock and writelock describe the operation 
being requested.

It's simply more difficult to use the more abstract concepts 'shared' and 
'exclusive' as convenient and transparent descriptors of what the thing does, 
and likely to just brew confusion.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Richard Oudkerk added the comment:

 @richard: I'm sorry, but both of my patches contain changes to 
 'Lib/threading.py' and can be applied on top of Python 3.3.0. So can you 
 explain what do you mean, by missing the changes to threading.py?

I was reading the Rietveld review page

http://bugs.python.org/review/8800/#ps6111

which only shows changes to multiprocessing/__init__.py and 
multiprocessing/synchronize.py.

The patch looks like it was produced using git rather than hg, so perhaps 
Rietveld got confused by this.  In that case it is a bug in Rietveld that it 
produced a partial review instead of producing no review.

 # unpack the the object into two variables and pass them separately around
 shrd_lock, excl_lock = ShrdExclLock()
 
 Thread(target=reader, args=(shrd_lock,)).start()
 Thread(target=writer, args=(excl_lock,)).start)

Although using namedtuple is probably a good idea, I don't think it really adds 
much flexibility.  This example could just as easily be written

  selock = ShrdExclLock()

  Thread(target=reader, args=(selock.shared,)).start()
  Thread(target=writer, args=(selock.exclusive,)).start)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Richard Oudkerk added the comment:

 With this, you are stuck with employing a context manager model only.  
 You loose the flexibility to do explicit acquire_read() or 
 acquire_write().

You are not restricted to the context manager model.  Just use 
selock.shared.acquire() or selock.exclusive.acquire().

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-01 Thread Kristján Valur Jónsson


Kristján Valur Jónsson added the comment:

Perhaps I should have pointed out, for Sebastian's benefit, that my second 
patch uses timeout rather than blocking since that is the new black in 
python 3.
Also, I think the threading implementation shows clearly the problem of 
having two independent objects that are only losely bound by some shared common 
variables:  The threading.py version has to rely on c_types ints for the common 
counters.
If the two locks were merely two different views of a common RWLock object, 
this problem could go away, and you could have the threading.RWLock and the 
multiprocessing.RWLock be different concrete classes derived from a common base 
class.

I also think it is time to drop the writer preference model, since it just 
adds complexity with doubtful benefits.  Sebastian's model also does that.

I'll provide a new example patch presently.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Sebastian Noack added the comment:

 If you want to argue it this way, I counter that the attributes
 shared and exclusive apply to the type of access to the
 protected object you are talking about, and yet, the name suggest
 that they are attributes of the lock itself.

A lock's sole purpose is to synchronize access to a protected object or 
context. So naming a lock after its type of protection absolutely makes sense. 
Those names are also not supposed to be attributes of the lock, rather two 
locks (a shared and an exclusive lock) should be created, that might be 
returned as a namedtuple for convenience.

 In that sense, reader lock and writer lock, describe attributes
 of the user of the lock, and the verbs readlock and writelock
 describe the operation being requested.

The user of the lock isn't necessarily a reader or writer. This is just one of 
many possible use cases. For example in a server application a shared/exclusive 
lock might be used to protect a connection to the client. So every time a 
thread wants to use the connection, a shared lock must be acquired and when a 
thread wants to shutdown the connection, the exclusive lock must be acquired, 
in order to ensure that it doesn't interrupt any thread still processing a 
request for that connection. In that case you clearly wouldn't call the users 
reader and writer.


 The patch looks like it was produced using git rather than hg, so
 perhaps Rietveld got confused by this.  In that case it is a bug
 in Rietveld that it produced a partial review instead of producing
 no review.

Yes, I have imported the Python 3.3.0 tree into a local git repository and 
created the patch that way. Since patches generated with git are still 
compatible with the 'patch' program in order to apply them, I hope that isn't a 
problem.


 Although using namedtuple is probably a good idea, I don't think it
 really adds much flexibility.  This example could just as easily be
 written

  selock = ShrdExclLock()

  Thread(target=reader, args=(selock.shared,)).start()
  Thread(target=writer, args=(selock.exclusive,)).start()

Yes, that is true, but in some cases it is more convenient to be able unpack 
the shared/exclusive lock into two variables, with a one-liner. And defining a 
namedtuple doesn't require any extra code compared to defining a class that 
holds both locks. In fact it needs less code to be implemented.

However the flexibility comes from having two lock objects, doesn't matter how 
they are accessed, instead as suggested by Kristján to have a single lock 
object, which just provides proxies for use with the with statement.


 I also think it is time to drop the writer preference model, since
 it just adds complexity with doubtful benefits.  Sebastian's model
 also does that.

I have implemented the simplest possible acquisition order. The lock acquired 
first will be granted first. Without that (or a more advanced policy) in 
applications with concurrent threads/processes that are heavily using the 
shared lock, the exclusive lock can never be acquired, because of there is 
always a shared lock acquired and before it is released the next shared lock 
will be acquired.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Richard Oudkerk added the comment:

I think Sebastian's algorithm does not correctly deal with the non-blocking 
case.  Consider the following situation:

* Thread-1 successfully acquires exclusive lock.
  Now num_got_lock == 1.

* Thread-2 blocks waiting for shared lock.
  Will block until (num_got_lock == 1 and excl_count == 0).
  Now num_got_lock == 1.

* Thread-3 does non-blocking acquire of shared lock but fails.
  Now num_got_lock == 2.

Now, since num_got_lock == 2, the predicate that Thread-2 is waiting for will 
not happen until num_got_lock overflows.

This is probably fixable if we just prevent a failed non-blocking acquire from 
modifying num_acq_lock and num_got_lock.  (But I don't see how to extend the 
algorithm to allow timeouts.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Richard Oudkerk added the comment:

My previous comment applied to Sebastian's first patch.  The second seems to 
fix the issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-01 Thread Christian Heimes


Christian Heimes added the comment:

I wonder, why are you creating your own algorithm here? There must be plenty of 
reference implementations that are already used in production code. Don't be a 
shamed to copy a Java implementation! :) The entire threading module is a 
rip-off of the Java threading API.

--
nosy: +christian.heimes

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock


Sebastian Noack added the comment:

I would love to see how other people would implement a shared/exclusive lock 
that can be acquired from different processes. However it really seems that 
nobody did it before. If you know a reference implementation I would be more 
than happy.

There are plenty of implementations for threading only, but they won't work 
with multiprocessing, due to the limitations in the ways you can share data 
between processes.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-01 Thread Antoine Pitrou


Antoine Pitrou added the comment:

On the naming front: shorthands like Shrd and Excl are a bit frown upon. 
Since SharedExclusiveLock is on the long side, I would suggest calling the 
API SELock.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-01 Thread Christian Heimes


Christian Heimes added the comment:

A RW lock is part of POSIX threads [1]. It's usually a good idea to either use 
POSIX functions or to mimic their behavior. After all POSIX is an industry 
standard.

Boost and Java have several lock and rw lock implementations. Wikipedia [2] is 
a good starting point for the various implementations. The page also mentions a 
seqlock which looks interesting to me as it's fast for few writers with lots of 
readers.

[1] http://linux.die.net/man/3/pthread_rwlock_init 
[2] http://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock

2012-10-01 Thread Antoine Pitrou


Antoine Pitrou added the comment:

 A RW lock is part of POSIX threads [1]. It's usually a good idea to
 either use POSIX functions or to mimic their behavior. After all POSIX
 is an industry standard.

We've already departed from that. Our Lock is nothing like a mutex, for
example (it's more of a binary semaphore).
We follow POSIX when exposing POSIX APIs (as in the os module), but
otherwise we have our own abstractions, for example the 3.x I/O stack.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8800
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8800] add threading.RWLock