On 21 March 2012 22:54, Claudiu Saftoiu <csaft...@gmail.com> wrote: > Hello ZODB List, > > (This is also a stackoverflow question - you might prefer the formatting > there: http://stackoverflow.com/questions/9810116/zodb-database-conflict-fail ) > > I have a server, and a client. > > A client sends a request. The request has a certain key associated with it, > e.g. `a-1`, `a-2`, `b-1`, `b-4`. > > If two requests for the same key come in at once, there will be a conflict > error, as the same data structures are being modified. > > I can adapt the client to simply not send two requests of the same key at > once. However, I'd like this system to work with multiple clients, too. It > seems silly to have the clients coordinate what they send to the server. > Instead, I'd like the server to simply block on a request of a certain key > if that key is already being modified, until the other requests with that > same key are done. > > To this end, I've created a locking system. At the beginning of the function > on the server, I do: > > key = ... > print "Acquiring %s lock..." % (key,) > KEY_LOCKS[key].acquire() > print "%s lock acquired." % (key,) > def after_commit_hook(success): > KEY_LOCKS[key].release() > print "(after %s commit): Released %s lock" % (('failed', > 'successful')[success], key) > transaction.get().addAfterCommitHook(after_commit_hook) > > where `KEY_LOCKS` is a dict mapping keys to `threading.Lock`s. Afterwards > follows the code that modifies the persistent data structures. > > What I assume would happen is that, if a request comes in for a key that's > already being processed, it would block when acquiring the lock. Only when > the earlier request **has already been committed** (thus being beyond any > conflict errors), would the new request resume. The requests do nothing that > would conflict until the lock is acquired. > > Most of the requests work fine: > > Acquiring a-b lock... > a-b lock acquired. > (after successful commit): Released a-b lock > Acquiring a-c lock... > a-c lock acquired. > (after successful commit): Released a-c lock > > However, there is _still_ an issue when the same key is sent, even though > the locking seems to work: > > Acquiring q-q lock... > q-q lock acquired. > Acquiring q-q lock... > (after successful commit): Released q-q lock > q-q lock acquired. > (after failed commit): Released q-q lock > repoze.retry retrying, count = 1 > Traceback (most recent call last): > ... > ConflictError: database conflict error (oid 0x13009b, class > persistent.list.PersistentList) > > And then the request retries. Note that the `q-q lock` was only acquired > after the successful commit. > > What gives? Why is this system not preventing conflict errors? Where is my > assumption incorrect? > > --- > > If, before the `transaction.get().addAfterCommitHook(after_commit_hook)` > line I put `transaction.begin()`, it works. For the life of me I can't > figure out why. Before the `transaction.begin()` line, the entirety of my > code is: > > post = request.params > if not post: return Response("No data!") > > data = eval(post['data']) > time_parsed = time.time() > my_app = request.context > > This solves my problem but I'm not putting that as an answer 'cause I still > want to know: Why does it give conflict errors if I don't start a fresh > transaction right before?
You shoudn't need to be building your own locking system on top of ZODB, this suggests that either ZODB is the wrong tool for your problem, or that you're using it wrong ;) ZODB has MVCC which means you get a consistent view of the database from the time the transaction was begun (see http://en.wikipedia.org/wiki/Snapshot_isolation.) In a web application using ZODB, each new request begins a transaction and at the end of the request the transaction is committed. If that commit raises a conflict error then the request is retried (usually up to three times.) Perhaps you could build on top of one of the existing frameworks that has this behaviour built in, e.g. http://pyramid.readthedocs.org/en/1.3-branch/tutorials/wiki/index.html It's difficult to offer any useful advice without knowing what you are trying to achieve here. Certain data structures (mostly those in the BTrees package) have conflict resolution built in. For instance, you can concurrently modify multiple values in the dict like BTrees.OOBTree.OOBTree (or any of the other variants) so long as each concurrent transaction is modifying different keys. Laurence _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
