Mike On Tuesday 11 December 2007, Michael Sparks wrote: > Hi Richard, > > > On Tuesday 11 December 2007 13:36, Richard Taylor wrote: > > I don't think that you can rely on the threadsafety of these functions. > > Even if they are threadsafe in C Python (which I doubt that 'set' is), the > > locking in Jython in more fine grained and would likely catch you out. > > It's perhaps worth noting in the version of the code I posted, in this thread, > where it said... > > """What key areas appear least threadsafe, and any general suggestions > around that.""" > > ...I knew that set and using were not threadsafe, but wondered about other > parts. I perhaps should've been more explicit on that point. (I wanted to > simply post some ideas which showed the core logic without locking. Perhaps a > mistake :) >
I did miss the subtlety :-) but at least it prompted me to reply! > Anyhow, the current version is here: > > https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Bindings/STM/Axon/STM.py > > In that version, "set" now looks like this: > def set(self, key, value): > success = False > if self.lock.acquire(0): > try: > if not (self.store[key].version > value.version): > self.store[key] = Value(value.version+1, > copy.deepcopy(value.value), > self, key) > value.version= value.version+1 > success = True > finally: > self.lock.release() > else: > raise BusyRetry > > if not success: > raise ConcurrentUpdate > > and "using" has changed to "usevar: (using now relates to a collection) > > def usevar(self, key): > try: > return self.get(key) > except KeyError: > if self.lock.acquire(0): > try: > self.store[key] = Value(0, None,self,key) > finally: > self.lock.release() > else: > raise BusyRetry > > return self.get(key) > > Since mutations of the store rely on acquiring the lock on the store, that > should be safe(r). User code doesn't have to worry about locks however - > which is course the point of the code :-) > I like to use explicit read locks for shared data structures, mainly because it makes it much safer when someone comes along and adds functionality to the methods later on. The next developer may not realise that the method needs to be threadsafe so the explicit locking code will help. I like to use decorators for the same reason, it is very easy to get the try/finally wrong end up with deadlocks. I think that I would split the usevar method in two and use decorators to acquire the read and write locks e.g. @read_lock() def usevar(self, key): try: return self.get(key) except KeyError: return self.make(key) @write_lock() def make(self,key): self.store[key] = Value(0, None,self,key) return self.get(key) > The reason for specifically using the acquire(0) call rather than acquire() > call is because I want it to fail hard if the lock can't be acquired. I know > it'd be nicer to have a finer grained lock here, but I'm personally primarily > going to be using this for rare operations rather than common operations. > Personally I am not sure that I would bother with the non-blocking acquire here. It will complicate the client code and the length of time that the lock will be held will be so small that almost all client code will simply retry the set. So I would go for a blocking acquire or maybe add a timeout based exception to catch a deadlock. > These locks above are of course in relation to write locking. I'll think about > the read locking you've suggested. > > Your locking looks incorrect on using since it both allows reading and writing > of the store. (retrieve value & if not present create & initialise) Yep, I did say that it was not tested :-) It is wrong on 'set' as well, as 'set' should acquire both the read and the write lock. > I also think the independent locks are a misnomer, but they're useful for > thinking about it. > > > I would suggest that you should routinely wrap shared datamodels like these > > in thread locks to be certain about things. > > Indeed. It makes the code look worse, so for this example I was really after > suggestions (like yours :-) of "OK, where does this break badly" as well as > "does the logic look sane?". Decorators help to hide the details from the main code path. I think that decorators are over used sometimes, but in this case I think that are very useful. > > I would also suggest that a small change to the Value class would make it > > possible for client code to subclass it, which might make it more flexible. > > I'm not convinced by the changes to Value - its there for storing arbitrary > values, rather than extending Value itself. It's probably worth noting > that .clone has changed in my version to this: > > def clone(self): > return Value(self.version, > copy.deepcopy(self.value),self.store,self.key) > > Which includes deepcopy on the value stored by Value. I'm beginning to think > that Value should be called "Variable" to make this clearer... > I just have a tendency to like to enable flexibility from client code where it is possible. I think I have fought with library code that was inflexible one too many times :-) > It's interesting though, after having developed large amounts of code of code > based on no-shared-data & read-only/write-only pipes with data handoff and > not having had any major concurrency issues (despite mixing threads and non > threads) switching to a shared data model instantly causes problems. > > The difference is really stark. One is simple, natural and easy and the other > is subtle & problematic. I'm not shocked, but find it amusing :-) I agree. Where possible we use 'pipe and filter' or 'message queue' patterns for inter-thread communication. We have a library based around the MASCOT design method that uses concepts of 'queues' and 'pools', which has proved very powerful for large multi-threaded applications. We have found that the best way to avoid thread related problems is to indulge in a little design work to explicitly limit all thread communication to a few well thought through mechanisms and then police their use rigidly. Regards Richard -- QinetiQ B009 Woodward Building St. Andrews Road Malvern Worcs WR14 3PS Jabber: [EMAIL PROTECTED] PGPKey: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xA7DA9FD9 Key fingerprint = D051 A121 E7C3 485F 3C0E 1593 ED9E D868 A7DA 9FD9
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
