[ZODB-Dev] Automating retry management
So I'm about to update the transaction package and this gives me an opportunity to do something I've been meaning to do for a while, which is to add support for the Python with statement: with transaction: ... transaction body ... and: with some_transaction_manager as t: ... transaction body, accesses current transaction as t ... This looks really great, IMO, but there's a major piece missing, which is dealing with transient transaction failures due to conflicts. If using an optimistic transaction mechanism, like ZODB's, you have to deal with conflict errors. If using a lock-based transaction mechanism, you'd have to deal with deadlock detection. In either case, you have to detect conflicts and retry the transaction. I wonder how other Python database interfaces deal with this. (I just skimmed the DBI v2 spec and didn't see anything.) What happens, for example, if there are conflicting writes in a Postgress or Oracle application? I assume that some sort of exception is raised. I also wonder how this situation could be handled elegantly. To deal with conflicts, (assuming transaction had with statement support) you'd end up with: tries = 0 while 1: try: with transaction: conn.root.x = 1 except ZODB.POSExeption.ConflictError: tries += 1 if tries 3: raise Yuck! (Although it's better than it would be without transaction with statement support.) In web applications, we generally don't see the retry management because the framework takes care of it for us. That is until we write a script to do something outside of a web application. This would be easier to automate if Python let us write custom looping structures or allowed full anonymous functions. The best I've been able to come up with is something like: t = ZODB.transaction(3) while t.trying: with t: ... transaction body ... Here the transaction function returns an object that: - keeps track of how many times it's tried and manages a trying attribute that is true while we haven't given up or suceeded, and - is a context manager that takes care of transaction boundaries and updates the trying attr depending on transaction outcome. This version is better than the one with the try/except version, but isn't entirely satisfying. :) Does anyone have any better ideas? I use a ZODB function, because ConflictError is ZODB specific. It would be nice if this could be standardized, so that a mechanism could be defined by the transaction package. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
I'm already using custom transaction/savepoint context managers in my code. I use them like with TransactionContext(): db.root['sp_test'] = 'init' with SavepointContext(): db.root['sp_test'] = 'saved' On of the context managers: class TransactionContext(object): def __init__(self, txn = None): if txn is None: txn = transaction.get() self.txn = txn def __enter__(self): return self.txn def __exit__(self, t, v, tb): if t is not None: self.txn.abort() else: self.txn.commit() Now you could probably extend this to look like class TransactionContext(object): def __init__(self, txn = None, retryCount = 3): if txn is None: txn = transaction.get() self.txn = txn self.retryCount = retryCount def __enter__(self): return self.txn def __exit__(self, t, v, tb): if t is not None: self.txn.abort() else: for i in range(self.retryCount): try: self.txn.commit() except ConflictError as exc2: exc = exc2 else: return raise exc The looping/except part could probably look nicer. Use case looks like: with TransactionContext(mytransaction, retryCount = 5): db.root['sp_test'] = 'init' Does this look similar to what you were looking for? -Matthias For completeness, here's my savepoint manager: class SavepointContext(object): def __enter__(self, txn = None): if txn is None: txn = transaction.get() self.savepoint = txn.savepoint() return self.savepoint def __exit__(self, type, value, traceback): if type is not None: self.savepoint.rollback() ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
def __exit__(self, t, v, tb): if t is not None: self.txn.abort() else: for i in range(self.retryCount): Oops, bug here. It should read range(1 + self.retryCount). It should probably have unittests anyway :-) -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
On Tue, May 11, 2010 at 7:34 AM, Jim Fulton j...@zope.com wrote: [...] The best I've been able to come up with is something like: t = ZODB.transaction(3) while t.trying: with t: ... transaction body ... I think you could get this to work: for transaction in ZODB.retries(3): with transaction: ... transaction body ... ZODB.retries would return an iterator that would raise StopIteration on the next go-round if the previously yielded context manager exited without a ConflictError. -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
On Tue, May 11, 2010 at 7:52 AM, Nitro ni...@dr-code.org wrote: I'm already using custom transaction/savepoint context managers in my code. I use them like ... Now you could probably extend this to look like class TransactionContext(object): def __init__(self, txn = None, retryCount = 3): if txn is None: txn = transaction.get() self.txn = txn self.retryCount = retryCount def __enter__(self): return self.txn def __exit__(self, t, v, tb): if t is not None: self.txn.abort() else: for i in range(self.retryCount): try: self.txn.commit() except ConflictError as exc2: exc = exc2 else: return raise exc The looping/except part could probably look nicer. Use case looks like: with TransactionContext(mytransaction, retryCount = 5): db.root['sp_test'] = 'init' Does this look similar to what you were looking for? This wouldn't work. You would need to re-execute the suite for each retry. It's not enough to just keep committing the same transaction. (There are other details wrong with the code above, but they are fixable.) Python doesn't provide a way to keep executing the suite. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
On Tue, May 11, 2010 at 8:38 AM, Benji York be...@zope.com wrote: On Tue, May 11, 2010 at 7:34 AM, Jim Fulton j...@zope.com wrote: [...] The best I've been able to come up with is something like: t = ZODB.transaction(3) while t.trying: with t: ... transaction body ... I think you could get this to work: for transaction in ZODB.retries(3): with transaction: ... transaction body ... ZODB.retries would return an iterator that would raise StopIteration on the next go-round if the previously yielded context manager exited without a ConflictError. This is an improvement. It's still unsatisfying, but I don't think I'm going to get satisfaction. :) BTW, if I do something like this, I think I'll add a retry exception to the transaction package and have ZODB.POSException.ConflictError extend it so I can add the retry automation to the transaction package. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
On Tue, May 11, 2010 at 10:08 AM, Jim Fulton j...@zope.com wrote: This is an improvement. It's still unsatisfying, but I don't think I'm going to get satisfaction. :) Given that PEP 343 explicitly mentions *not* supporting an auto retry construct, I should think not. :) -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
Am 11.05.2010, 16:01 Uhr, schrieb Jim Fulton j...@zope.com: This wouldn't work. You would need to re-execute the suite for each retry. It's not enough to just keep committing the same transaction. (There are other details wrong with the code above, but they are fixable.) Python doesn't provide a way to keep executing the suite. You are right. The only thing I could come up with was something like below, using a decorator instead of a context. -Matthias @doTransaction(count = 5) def storeData(): ... store data here ... def doTransaction(transaction = None, count = 3): def decorator(func): def do(): for i in range(1+count): try: func() except: transaction.abort() raise try: transaction.commit() except ConflictError: if i == count: raise else: return return do return decorator ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
Am 11.05.2010, 17:08 Uhr, schrieb Nitro ni...@dr-code.org: Am 11.05.2010, 16:01 Uhr, schrieb Jim Fulton j...@zope.com: This wouldn't work. You would need to re-execute the suite for each retry. It's not enough to just keep committing the same transaction. (There are other details wrong with the code above, but they are fixable.) Python doesn't provide a way to keep executing the suite. You are right. The only thing I could come up with was something like below, using a decorator instead of a context. -Matthias @doTransaction(count = 5) def storeData(): ... store data here ... def doTransaction(transaction = None, count = 3): def decorator(func): def do(): for i in range(1+count): try: func() except: transaction.abort() raise try: transaction.commit() except ConflictError: if i == count: raise else: return return do This should read return do(), i.e. the decorator should directly execute the storeData function. All in all I think Benji's proposal looks better :-) -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
On 11 May 2010 15:08, Jim Fulton j...@zope.com wrote: On Tue, May 11, 2010 at 8:38 AM, Benji York be...@zope.com wrote: On Tue, May 11, 2010 at 7:34 AM, Jim Fulton j...@zope.com wrote: [...] The best I've been able to come up with is something like: t = ZODB.transaction(3) while t.trying: with t: ... transaction body ... I think you could get this to work: for transaction in ZODB.retries(3): with transaction: ... transaction body ... ZODB.retries would return an iterator that would raise StopIteration on the next go-round if the previously yielded context manager exited without a ConflictError. This is an improvement. It's still unsatisfying, but I don't think I'm going to get satisfaction. :) BTW, if I do something like this, I think I'll add a retry exception to the transaction package and have ZODB.POSException.ConflictError extend it so I can add the retry automation to the transaction package. The repoze.retry package lets you configure a list of exceptions. http://pypi.python.org/pypi/repoze.retry http://svn.repoze.org/repoze.retry/trunk/repoze/retry/__init__.py Though it seems inspecting the error text is required for most sql database errors to know if they are retryable, as ZPsycoPGDA does: 188 except (psycopg2.ProgrammingError, psycopg2.IntegrityError), e: 189 if e.args[0].find(concurrent update) -1: 190 raise ConflictError (https://dndg.it/cgi-bin/gitweb.cgi?p=public/psycopg2.git;a=blob;f=ZPsycopgDA/db.py) For PostgreSQL it should be sufficient to catch these errors and raise Retry during tpc_vote. For databases which do not provide MVCC in the same way as PostgreSQL, concurrency errors could be manifested at any point in the transaction. Even Oracle can raise an error during a long running transaction when insufficient rollback space is available, resulting in what is essentially a read conflict error. Such errors could not be caught by a data manager and reraised as a Retry exception. I think it might be useful to add an optional method to data managers that is queried by the retry automation machinery to see if an exception should potentially be retried. Perhaps this would best be accomplished in two steps: 1. Add an optional property to data managers called ``retryable``. This is a list of potentially retryable exceptions. When a data manager is added to the transaction, the transaction's list of retryable exceptions is extended by the joining data managers list of retryable exceptions. t = transaction.begin() try: application() except t.retryable, e: t.retry(e): 2. t.retry(e) is then checks with each registered data manager if that particular exceptions is retryable, and if so raises Retry. def retry(self, e): for datamanager in self._resources: try: retry = datamanager.retry except AttributeError: continue if isinstance(e, datamanager.retryable): datamanager.retry(e) # dm may raise Retry here Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
